Search Results: "ag"

24 February 2009

Adrian von Bidder: New Toy

Last week, I couldn't resist and bought myself an Acer Aspire One (AOA 150Ab) netbook. It has 1G RAM, 120G HDD, unfortunately needs a fan, and is the model without 3G modem. It comes with Linpus Linux pre-installed. Looks quite nice, but is obviously ultimately the wrong OS ... Besides, it's locked down quite a bit, there's not even an easy way to start a terminal :-) I still have kept it, in a dual-boot configuration, to play around or show to people. Installing Lenny went very well, and to make things more fun I'm running quite a few things from experimental: KDE 4.2, xorg 7.4 (1:7.4~5 right now) and Oo.org 3. And, to get DRI2, also the 2.6.28 kernel from the newer-than-sid repository of the kernel team (2.6.28-2~snapshot.12850, but I hear 2.6.28 has now been uploaded.) While the whole thing is fun to use and didn't make any real problems, there are a few remaining issues (yes, this is a Dear Lazyweb posting, feel free to comment. I'll try to add updates to this article as this progresses): Ok, this has become rather a long list. But at least, as you can see, it's mostly minor issues, and hopefully a few where it's just missing configuration. One other issue is battery life: I see that there should be a large battery available for this thing. If it doesn't cost me as much as the whole netbook again, I'll seriously have to think about this... The 3-cell battery does last about 2.5h (and has, right now, uncovered a little bug where the battery monitoring applet tells me No AC adaptor plugged in, battery capacity: 50%, charging . Whee! I have a perpetuum mobile! I'm gonna be RICH!)

Dirk Eddelbuettel: R/Finance conference in Chicago in April: Registration now open

Regarding the aforementioned R/Finance conference that will take place at the end of April here in Chicago, we announced earlier today that the conference website is now available. It provides information about the program, speakers and other details as well as a link to registration details. See you in Chicago in April!

Matt Brown: The government listened!

I was very pleased to wake up this morning to the news that National has delayed the introduction of S92A via an order-in-council. It s a nice short-term victory, but I ll save the champagne until the law is fundamentally rewritten. The most pleasing aspect of the decision is simply that it was made at all. Within two weeks, a small band of protesters were able to harness the power of the Internet to direct international attention and place enough pressure on a Government, whose Prime Minister admitted to not having read the bill prior, that he then took the time to understand the issues and personally announce the delay in implementation of the law. We owe much thanks to the Creative Freedom Foundation for all the effort they put into co-ordinating the protest and ensuring that a single coherent message was presented. Just a little bit of my cynicism and belief that politicians never listen to public opinion outside of election campaigns was chipped away today. The reason I m not breaking out the champagne yet is that we ve only achieved a temporary reprieve in the commencement of the law. While those present at the press conference seem somewhat confident that John Key didn t like what he found in the law and would have repealed it if given the chance, all that has actually been done is delay it in the hopes of an agreement between the TCF and the rights holders (aka big media companies) on how to implement the still fundamentally broken law. The Government has given until late March for that to occur. To put this into a more global context. My happiness as I took the bus to work after reading about the decision to delay the law was short lived as the front page of the local paper declared that Eircom (Ireland s equivalent of Telecom) has voluntarily agreed to block sites such as The Pirate Bay upon request by the media companies (this comes a week after they also announced an agreement to, again voluntarily , implement a 3-strikes S92A style policy). Now, with the biggest ISP in their pocket (so to speak), the media companies have sent threatening letters to the remaining ISPs in the country demanding they implement the same procedure. To me, this illustrates one of the fundamental problems with S92. The concept that an ISP is liable for the conduct of its users, or for policing where on the Internet users should and shouldn t be able to connect to does not belong in our laws. Most ISPs already have provision to disconnect customers for illegal activity in their terms and conditions. If an end-user is doing something illegal, that is an issue between the rights holder and the end-user to take up in the courts just like every other sector of society must do when wronged, at which point the existing ISP terms and conditions can be invoked and access terminated. The big media companies, having decided that it is too expensive/hard/inconvenient to follow standard legal procedures to resolve their grievances are launching multi-pronged attacks to shift the playing field in their favour. In countries like New Zealand, where our politicians yearn for a Free Trade Agreement with America, they use their lobbyists to ensure that S92 style laws are part of the conditions. In other jurisdictions, like Ireland, they use strong-arm, divide and conquer style bully tactics outside of the political and legal process. I don t support copyright infringement. I rely on copyright to protect much of the work I place on the Internet, I want strong laws that protect me when my rights have been infringed. I don t believe that such laws should come at the expense of due process, our legal tradition and the basic principle of fairness! I don t believe that copyright infringement is such a heinous crime that it demands punishments stronger than those we deliver to paedophiles, stalkers or any other class of criminal who uses the Internet to enable their crimes. To me, today s (yesterday s - depending on your timezone) decision is only the first step in clawing New Zealand back from the dangerous path that the big media companies have been leading our law makers down. From here we need to press on and demonstrate to the Government over the next month that even if the TCF and rights holders are able to come up with some sort of workable code of practice, the law is still fundamentally flawed. It is based on premise that we are guilty by accusation. Even if guilt were to be proved by a competent legal body (eg. court or copyright tribunal) we don t need laws placing further liabilites onto ISPs (and remember the definition of ISP under this amendment act includes businesses who provide Internet access to staff, libraries, schools and hospitals) when their existing terms and conditions already prohibit illegal activity. Finally, and most importantly of all, we need to remember that laws exist to serve all sectors of society. Yes, copyright infringement is against the law and rights holders are reasonable in expecting the law to protect their content and allow them to make a fair profit. On the other side of the fence, average New Zealanders are not being unreasonable in their desire to have media available electronically, on demand and non-inhibited by DRM following a legal purchase. The failure of the media businesses to adequately cater to this change in market demand and usage of technology is obviously a contributing factor to the widespread copyright problems that they are facing today. Obviously, I m not condoning copyright infringement simply because the media companies are failing to address demand. Even stupid laws must be obeyed (and the concept of copyright is far from stupid). What I want to see is the Government acknowledging that the problem is not solely with consumers infringing copyright for malicious purposes, and therefore that the solutions do not lie solely in increasing the enforcement and punishments available. Copyright has always been a balancing act between the rights of content producers and consumers. S92 and the act it is contained within are taking us far too far down the road of catering to big business and their outdated business models with far too little concern for the rights of the individual consumer. Despite the many submissions made on this act last year when it was first passing through parliament, there was no comprehensive debate on what copyright means and how it should balance the rights of content producers and consumers in our digital century where copying is a zero-cost, zero-thought activity. Without such a debate we re doomed to continue wasting time arguing over the symptoms of the problem, like S92. So, I m saving my champagne for the day when we as a country address these issues and come up with a fair and workable interpretation of what copyright means today.

Bernd Zeimetz: why you should use gpsd over gypsy

While working on the Debian packages of navit I stumbled upon gypsy, as the libgypsy is used by navit to listen to notifications about location changes via D-Bus. As maintainer of the gpsd package I started to wonder what the opinion of the gpsd developers about gypsy is - especially after reading that "Gypsy was designed to fix the numerous design flaws found in GPSD". Well, here is their answer, which I support fully. I'm blogging about it as I think it is important to know about this details if you plan to do something useful with a gps device, for example on mobile phones like the OpenMoko Freerunner. gpsd works very well on embedded devices, and if there's a feature missing just contact the developers. They're very responsive and like to implement useful features, if they can get a spec to do so (and even better, the hardware to test it). So I'm really wondering if it wouldn't make sense to replace the minimal NMEA/UBX implementation in the FSO Framework Daemon by gpsd, even if that means to compile code instead of writing Python code, which is much faster for rapid prototyping, of course.

23 February 2009

Ingo Juergensmann: Performance Tweaking with Drupal

Isotopp just pointed me to the blog of Ronald Bradford, a MySQL consultant, writing about a Drupal installation and some performance observations.

He observes a large table "watchdog":

Presently, volume and content is of a low volume, but expecting to ramp up. I do however find 90% of disk volume in one table called watchdog ;


Well, yes, the watchdog table logs system messages. No big surprise so far, I guess. So, turning off database logging (Administer -> Site building -> Modules -> turn off Database logging module) or reduce the maximum number of entries in the database log at Administer -> Site Configuration -> Logging and alerts -> Database Logging will reduce the size of the watchdog table.

Robert continues with some "page not found" errors. Of course these have to be fixed somewhere, either by editing the content or fixing theme related bugs. Not really a Drupal issue, I guess.

Then he mentions repeated SQL queries:

Looking at just 1 random second of SQL logging shows 1200+ SELECT statements.
355 are SELECT changed FROM node
[...]
There is plenty of information regarding monitoring the Slow Queries in MySQL, but I have also promoted that s it not the slow queries that ultimately slow a system down, but the 1000 s of repeating fast queries.

MySQL of course has the Query Cache to assist, but this is a course grade solution, and a high volume read/write environment this is meaningless.

There is a clear need for either a application level caching, or a database redesign to pull rather then poll this information, however without more in depth review of Drupal I can not make any judgment calls.


Drupal offers some kind of application level caching at Administer -> Site configuration -> Performance, so turning it on might already help, although I'm not sure whether that minimizes the SQL queries or to what degree.

Anyway, there are some more Drupal performance tips here:

  1. http://drupal.org/node/97347

  2. http://wimleers.com/article/improving-drupals-page-loading-performance

  3. http://2bits.com/articles/drupal-performance-tuning-and-optimization-for-large-web-sites.html



Hope this helps... :-)

Jon Dowland: wicd

Just discovered 'wicd' - a wireless connection manager similar to network manager. I've never liked NM much (there's a couple of bugs open which explain why). So far wicd seems to work well, although it does not support VPNs. Time will tell if it suffers many of the problems that soured NM for me!

Matthew Garrett: Misc

Minor Fedora updates - I fixed up the FDI file in the wacom package, so tablet PCs should have a working stylus out of the box in rawhide. The eraser won't work right now - the driver needs some reworking to bind multiple X devices to a single logical input device. I've also added support for brightness control via smartdimmer to nouveau, which should increase the number of machines that have working brightness control. I don't think this has landed in the rawhide kernel yet, but should do soon. There's the potential for some conflict with the mbp_nvidia_bl driver. We may end up dropping that.

HP updates - The button bar on my 2510 got replaced last week. I now have working volume buttons again. However, the machine now reboots whenever the machine is suspended and I close the lid. Diagnosed to either a faulty switch assembly or system board, which will require an engineer visit. An engineer dropped round today to fix the touchpad. Despite the case notes clearly stating that the problem was with the cable assembly, he was sent a replacement top cover unit. Without any cables. So he's coming back at some point. HP's customer support system apparently does not allow these cases to be merged. Which means I now have two visits to look forward to.

Android - I'm gradually working my way through the code, replacing various custom interfaces with standard ones. const char * const LCD_BACKLIGHT = "/sys/class/leds/lcd-backlight/brightness"; is an interesting standout so far.

Michal Čihař: Gammu stable version 1.23.0

After two months of heavy development, new Gammu stable release is out. There are quite a lot of changes and I'm not really convinced it makes sense to include full list of almost 150 lines from ChangeLog, so here are the most important changes (I'm sure I missed something, but I think the list is already too big): You can download from usual place: http://cihar.com/gammu/, Debian users will find packages in unstable soon, SUSE, Fedora and Mandriva users can check repository in OpenSuse Build Service and Ubuntu in my PPA.

Theodore Ts'o: Reflections on a complaint from a frustrated git user

Last week, Scott James Remnant posted a series of Git Sucks on his blog, starting with this one here, with follow up entries here and here. His problem? To quote Scott, I want to put a branch I have somewhere so somebody else can get it. That s the whole point of distributed revision-control, collaboration. He thought this was a mind-numbingly trivial operation, and was frustrated when it wasn t a one-line command in git. Part of the problem here is that for most git workflows, most people don t actually use git push . That s why it s not covered in the git tutorial (this was a point of frustration for Scott). In fact, in most large projects, the number of people need to use the scm push command is a very small percentage of the developer population, just as very few developers have commit privileges and are allowed to use the svn commit command in a project using Subversion. When you have a centralized repository, only the privileged few will given commit privileges, for obvious security and quality control reasons. Ah, but in a distributed SCM world, things are more democratic anyone can have their own repository, and so everyone can type the commands git commit or bzr commit . While this is true, the number of people who need to be able to publish their own branch is small. After all, the overhead in setting up your own server just so people can pull changes from you is quite large; and if you are just getting started, and only need to submit one or two patches, or even a large series of patches, e-mail is a far more convenient route. This is especially true in the early days of git s development, before web sites such as git.or.cz, github, and gitorious made it much easier for people to publish their own git repository. Even for a large series of changes, tools such as git format-patch and git send-email are very convenient for sending a patch series, and on the receiving side, the maintainer can use git am to apply a patch series sent via e-mail. It turns out that from a maintainer s point of view, reviewing patches via e-mail is often much more convenient. Especially for developers who are just starting out with submitting patches to a project, it s rare that a patch is of sufficiently high quality that it can be applied directly into the repository without needing fixups of one kind or another. The patch might not have the right coding style compared to the surrounding code, or it might be fundamentally buggy because the patch submitter didn t understand the code completely. Indeed, more often than not, when someone submits a patch to me, it is more useful for indicating the location of the bug more than anything else, and I often have to completely rewrite the patch before it enters into the e2fsprogs mainline repository. Given that, publishing a patch that will require modification in a public repository where it is ready to be pulled just doesn t make sense for many entry-level patch submitters. E-mail is in fact less work, and more appropriate for review purposes. It is only when a mid-level to senior developer is trusted to create high quality patches that do not need review that publishing their branch in a pull-ready form really makes sense. And that is fairly rare, and why it is not covered in most entry-level git documentation and tutorials. Unfortunately, many people expect to see the command scm push in a distributed SCM, and since git pull is a commonly used command for beginning git users, they expect that they should use git push as well not realizing that in a distributed SCM, push and pull are not symmetric operations. Therefore, while most git users won t need to use git push , git tutorials and other web pages which are attempting to introduce git to new users probably do need to do a better job explaining why most beginning participants in a project probably don t need their own publically accessible repository that other people can pull from, and which they can push changes for publication. There is one exception to this, of course, and this is a developer who wants to get started using git for a new project which he or she is starting and is the author/maintainer, or someone who is interested in converting their project to git. And this is where bzr has an advantage over git, in that bzr is primarily funded by Canonical, which has a strong interest in pushing an on-line web service, Launchpad. This makes it easier for bzr to have relatively simple recipes for sharing a bzr repository, since the user doesn t need to have access to a server with a public IP address, or need to set up a web or bzr server; they can simply take advantage of Launchpad. Of course, there are web sites which make it easy for people to publish their git repositories; earlier, I had mentioned git.or.cz, github, and gitorious. Currently, the git documentation and tutorials don t mention them since they aren t formally affiliated with the git project (although they are used by many git users and developers and the maintainers of these sites have contributed a large amount of code and documentation to git). This should change, I think. Scott s frustrations which kicked off his git sucks complaints would have been solved if the Git tutorial recommended that the easist ways for someone to publicly publish their repository is via one of these public web sites (although people who want to set up their own server certainly free to do so). Most of these public repositories probably won t have much reason to exist, but they don t do much harm, and who knows? While most of the repositories published at github and gitoriuous will be like the hundreds of thousands of abandoned projects on Sourceforge, one or two of the new projects which someone starts experimenting on at github or gitorious could turn out to be the next Ruby on Rails or Python or Linux. And hopefully, they will allow more developers to be able to experiment with publishing commits on their own repositories, and lessen the frustrations of people like Scott who thought they needed their own repositories; whether or not a public repository is the best way for them to do what they need to do, at least this way they won t get as frustrated about git. :-) Related posts (automatically generated):
  1. Git and hg John Goerzen recently posted about Git, Mercurial and Bzr that...
  2. Batches of patched batches of patches I found the following from the Risks Digest, authored by...

Theodore Ts'o: Reflections on a complaint from a frustrated git user

Last week, Scott James Remnant posted a series of Git Sucks on his blog, starting with this one here, with follow up entries here and here. His problem? To quote Scott, I want to put a branch I have somewhere so somebody else can get it. That s the whole point of distributed revision-control, collaboration. He thought this was a mind-numbingly trivial operation, and was frustrated when it wasn t a one-line command in git. Part of the problem here is that for most git workflows, most people don t actually use git push . That s why it s not covered in the git tutorial (this was a point of frustration for Scott). In fact, in most large projects, the number of people need to use the scm push command is a very small percentage of the developer population, just as very few developers have commit privileges and are allowed to use the svn commit command in a project using Subversion. When you have a centralized repository, only the privileged few will given commit privileges, for obvious security and quality control reasons. Ah, but in a distributed SCM world, things are more democratic anyone can have their own repository, and so everyone can type the commands git commit or bzr commit . While this is true, the number of people who need to be able to publish their own branch is small. After all, the overhead in setting up your own server just so people can pull changes from you is quite large; and if you are just getting started, and only need to submit one or two patches, or even a large series of patches, e-mail is a far more convenient route. This is especially true in the early days of git s development, before web sites such as git.or.cz, github, and gitorious made it much easier for people to publish their own git repository. Even for a large series of changes, tools such as git format-patch and git send-email are very convenient for sending a patch series, and on the receiving side, the maintainer can use git am to apply a patch series sent via e-mail. It turns out that from a maintainer s point of view, reviewing patches via e-mail is often much more convenient. Especially for developers who are just starting out with submitting patches to a project, it s rare that a patch is of sufficiently high quality that it can be applied directly into the repository without needing fixups of one kind or another. The patch might not have the right coding style compared to the surrounding code, or it might be fundamentally buggy because the patch submitter didn t understand the code completely. Indeed, more often than not, when someone submits a patch to me, it is more useful for indicating the location of the bug more than anything else, and I often have to completely rewrite the patch before it enters into the e2fsprogs mainline repository. Given that, publishing a patch that will require modification in a public repository where it is ready to be pulled just doesn t make sense for many entry-level patch submitters. E-mail is in fact less work, and more appropriate for review purposes. It is only when a mid-level to senior developer is trusted to create high quality patches that do not need review that publishing their branch in a pull-ready form really makes sense. And that is fairly rare, and why it is not covered in most entry-level git documentation and tutorials. Unfortunately, many people expect to see the command scm push in a distributed SCM, and since git pull is a commonly used command for beginning git users, they expect that they should use git push as well not realizing that in a distributed SCM, push and pull are not symmetric operations. Therefore, while most git users won t need to use git push , git tutorials and other web pages which are attempting to introduce git to new users probably do need to do a better job explaining why most beginning participants in a project probably don t need their own publically accessible repository that other people can pull from, and which they can push changes for publication. There is one exception to this, of course, and this is a developer who wants to get started using git for a new project which he or she is starting and is the author/maintainer, or someone who is interested in converting their project to git. And this is where bzr has an advantage over git, in that bzr is primarily funded by Canonical, which has a strong interest in pushing an on-line web service, Launchpad. This makes it easier for bzr to have relatively simple recipes for sharing a bzr repository, since the user doesn t need to have access to a server with a public IP address, or need to set up a web or bzr server; they can simply take advantage of Launchpad. Of course, there are web sites which make it easy for people to publish their git repositories; earlier, I had mentioned git.or.cz, github, and gitorious. Currently, the git documentation and tutorials don t mention them since they aren t formally affiliated with the git project (although they are used by many git users and developers and the maintainers of these sites have contributed a large amount of code and documentation to git). This should change, I think. Scott s frustrations which kicked off his git sucks complaints would have been solved if the Git tutorial recommended that the easist ways for someone to publicly publish their repository is via one of these public web sites (although people who want to set up their own server certainly free to do so). Most of these public repositories probably won t have much reason to exist, but they don t do much harm, and who knows? While most of the repositories published at github and gitoriuous will be like the hundreds of thousands of abandoned projects on Sourceforge, one or two of the new projects which someone starts experimenting on at github or gitorious could turn out to be the next Ruby on Rails or Python or Linux. And hopefully, they will allow more developers to be able to experiment with publishing commits on their own repositories, and lessen the frustrations of people like Scott who thought they needed their own repositories; whether or not a public repository is the best way for them to do what they need to do, at least this way they won t get as frustrated about git. :-) Related posts (automatically generated):
  1. Git and hg John Goerzen recently posted about Git, Mercurial and Bzr that...
  2. Batches of patched batches of patches I found the following from the Risks Digest, authored by...

Andrew Pollock: [tech] Upgrading MythTV to Debian 5.0 (Lenny)

I've been dying to upgrade my MythTV setup to use Lenny for ages, ever since I got a HD HomeRun and discovered my primary MythTV front end couldn't play back the HD recordings for some reason (I just got a blue box and the audio, and a lot of errors in the X server log). I figured it was either the kernel or X, and I hadn't been able to get the lirc kernel module to build against anything newer than Etch's 2.6.18 kernel (as I discovered today, that was because of some contamination in module-assistant's build directory, so I could have tried a newer kernel ages ago - drat), so I was unable to determine if it was the kernel or X.org itself. An added bonus of upgrading X was it talks to the TV better. I've got the PC hooked up to the TV's VGA input, and previously, the best I could do was get it to talk 1024x768, with the TV set to a 4:3 aspect ratio. When I initially upgraded, for whatever reason, the TV was very unhappy with what it was receiving on the VGA input and wouldn't display anything, so in desperation, I tried ditching the xorg.conf entirely, and then everything just worked. X came up with a 1280x768 resolution, I think despite the TV thinking it's getting 1024x768, so now we can have the TV in 16:9 mode, and things don't look all stretched, even when the playback is only 4:3. And HD, oh it looks glorious. We just watched the Oscars, and it completely filled the TV and looked absolutely fantastic. Playing back HD content seems to really peg the CPU though. The playback was unbearably choppy until we paused the commercial flagging job that was madly trying to flag commercials in the same recording we were currently watching (i.e. the Oscars). The only thing that seems to be acting a bit weird is notification pop ups from the D-Bus notification daemon. I've got a Griffin PowerMate that runs a home-grown script to re-pair the Bluetooth keyboard and mouse, and that uses the D-Bus notification system to provide feedback while it's running. It seems that while the MythTV front end is running, those pop ups aren't displaying correctly. I haven't gotten to the bottom that one yet. I also had some problems with a couple of init scripts and splashy causing bootup to hang, but I'm not sure that's completely reproducible (I don't reboot this machine all that often) so it may have been operator error. Overall, very happy with the outcome. The upgrade was fairly smooth, and I've got the additional functionality I've been waiting for.

Russ Allbery: Lintian 2.2.6

This release is mostly bug fixes that have accumulated over the past couple of weeks. The most notable is a rewrite of the unpack script for source packages that should do a much better job of analyzing the upstream tar file and determining the common prefix. This is currently important so that the upstream index file is generated properly and other collect scripts that depend on it are happy, but in the future it will also be used to implement some checks that care about how the upstream tarball was packed (when it was repacked for DFSG reasons, for instance). There are a bunch of other bug fixes and some more work that lays the ground for future tags. This was a "keeping our heads above water" release more than anything else, taking care of new bugs that had come in since the previous release and not making much continued progress on the bug backlog. But I think we have some underlying infrastructure in place to make more progress in the next few releases. I haven't updated lintian.debian.org for a while. I may do that with this release, or wait for one more.

Bernd Zeimetz: awesome calendar widget

One of the things I was missing in my favourite windowmanager awesome was a calendar widget. So I spent some minutes and coded it. Moving the mouse over the time widget will show a naughty popup with the current month, use your mouse buttons (and the shift key) to scroll trough the months. The only annoying thing was to figure out how to change the text of an existing notification: notification.box.widgets[2].text = "new test" is neither easy to figure out nor to use. But here's the code for your rc.lua - in case you want to have a license for it - the code is published as public domain or under the ISC license, what ever you prefer.
function displayMonth(month,year,weekStart)
    local t,wkSt=os.time year=year, month=month+1, day=0 ,weekStart or 1
    local d=os.date("*t",t)
    local mthDays,stDay=d.day,(d.wday-d.day-wkSt+1)%7
    local lines =  
    for x=0,6 do
        lines[x+1] = os.date("%a ",os.time year=2006,month=1,day=x+wkSt )
    end
    lines[8] = "    "
    local writeLine = 1
    while writeLine < (stDay + 1) do
        lines[writeLine] = lines[writeLine] .. "   "
        writeLine = writeLine + 1
    end
    for x=1,mthDays do
        if writeLine == 8 then
            writeLine = 1
        end
        if writeLine == 1 or x == 1 then
            lines[8] = lines[8] .. os.date(" %V",os.time year=year,month=month,day=x )
        end
        if (#(tostring(x)) == 1) then
            x = " " .. x
        end
        lines[writeLine] = lines[writeLine] .. " " .. x
        writeLine = writeLine + 1
    end
    local header = os.date("%B %Y\n",os.time year=year,month=month,day=1 )
    header = string.rep(" ", math.floor((#(lines[1]) - #header) / 2 )) .. header
    return header .. table.concat(lines, '\n')
end
local calendar =  
function switchNaughtyMonth(switchMonths)
    if (#calendar < 3) then return end
    local swMonths = switchMonths or 1
    calendar[1] = calendar[1] + swMonths
    calendar[3].box.widgets[2].text = displayMonth(calendar[1], calendar[2], 2)
end
mytextbox.mouse_enter = function ()
    local month, year = os.date('%m'), os.date('%Y')
    calendar =   month, year, 
                naughty.notify( 
                    text = displayMonth(month, year, 2),
                    timeout = 0, hover_timeout = 0.5,
                    width = 200
                 )
                
end 
mytextbox.mouse_leave = function () naughty.destroy(calendar[3]) end
mytextbox:buttons( 
    button(   , 1, function()
        switchNaughtyMonth(-1)
    end),
    button(   , 3, function()
        switchNaughtyMonth(1)
    end),
    button(   , 4, function()
        switchNaughtyMonth(-1)
    end),
    button(   , 5, function()
        switchNaughtyMonth(1)
    end),
    button(  'Shift'  , 1, function()
        switchNaughtyMonth(-12)
    end),
    button(  'Shift'  , 3, function()
        switchNaughtyMonth(12)
    end),
    button(  'Shift'  , 4, function()
        switchNaughtyMonth(-12)
    end),
    button(  'Shift'  , 5, function()
        switchNaughtyMonth(12)
    end)
 )

Bernd Zeimetz: awesome calendar widget

One of the things I was missing in my favourite windowmanager awesome was a calendar widget. So I spent some minutes and coded it. Moving the mouse over the time widget will show a naughty popup with the current month, use your mouse buttons (and the shift key) to scroll trough the months. The only annoying thing was to figure out how to change the text of an existing notification: notification.box.widgets[2].text = "new test" is neither easy to figure out nor to use. But here's the code for your rc.lua - in case you want to have a license for it - the code is published as public domain or under the ISC license, what ever you prefer. Update: With thanks to a hint from madduck, the naughty popup is now aware of the screen your mouse is pointing to.
function displayMonth(month,year,weekStart)
    local t,wkSt=os.time year=year, month=month+1, day=0 ,weekStart or 1
    local d=os.date("*t",t)
    local mthDays,stDay=d.day,(d.wday-d.day-wkSt+1)%7
    local lines =  
    for x=0,6 do
        lines[x+1] = os.date("%a ",os.time year=2006,month=1,day=x+wkSt )
    end
    lines[8] = "    "
    local writeLine = 1
    while writeLine < (stDay + 1) do
        lines[writeLine] = lines[writeLine] .. "   "
        writeLine = writeLine + 1
    end
    for x=1,mthDays do
        if writeLine == 8 then
            writeLine = 1
        end
        if writeLine == 1 or x == 1 then
            lines[8] = lines[8] .. os.date(" %V",os.time year=year,month=month,day=x )
        end
        if (#(tostring(x)) == 1) then
            x = " " .. x
        end
        lines[writeLine] = lines[writeLine] .. " " .. x
        writeLine = writeLine + 1
    end
    local header = os.date("%B %Y\n",os.time year=year,month=month,day=1 )
    header = string.rep(" ", math.floor((#(lines[1]) - #header) / 2 )) .. header
    return header .. table.concat(lines, '\n')
end
local calendar =  
function switchNaughtyMonth(switchMonths)
    if (#calendar < 3) then return end
    local swMonths = switchMonths or 1
    calendar[1] = calendar[1] + swMonths
    calendar[3].box.widgets[2].text = displayMonth(calendar[1], calendar[2], 2)
end
mytextbox.mouse_enter = function ()
    local month, year = os.date('%m'), os.date('%Y')
    calendar =   month, year, 
                naughty.notify( 
                    text = displayMonth(month, year, 2),
                    timeout = 0, hover_timeout = 0.5,
                    width = 200, screen = mouse.screen
                 )
                
end 
mytextbox.mouse_leave = function () naughty.destroy(calendar[3]) end
mytextbox:buttons( 
    button(   , 1, function()
        switchNaughtyMonth(-1)
    end),
    button(   , 3, function()
        switchNaughtyMonth(1)
    end),
    button(   , 4, function()
        switchNaughtyMonth(-1)
    end),
    button(   , 5, function()
        switchNaughtyMonth(1)
    end),
    button(  'Shift'  , 1, function()
        switchNaughtyMonth(-12)
    end),
    button(  'Shift'  , 3, function()
        switchNaughtyMonth(12)
    end),
    button(  'Shift'  , 4, function()
        switchNaughtyMonth(-12)
    end),
    button(  'Shift'  , 5, function()
        switchNaughtyMonth(12)
    end)
 )

22 February 2009

Steve Kemp: I gotta motor if I wanna be ready for that party tonight.

Since I already shared it elsewhere here is my KVM-launcher, and the mercurial repository it lives in. I'll add my kvm-shell program later - the tools I've written so far is mostly standalone, rather than a package. This is almost a content-free post, but I can pretend it isn't because I'm testing a new theme on my blog. The theme is included in the new release of my chronicle blog compiler which was released yesterday. ObFilm: Heathers

Theodore Ts'o: Should Filesystems Be Optimized for SSD s?

In one of the comments to my last blog entry, an anonymous commenter writes:
You seem to be taking a different perspective to linus on the adapting to the the disk technology front (Linus seems to against having to have the OS know about disk boundaries and having to do levelling itself)
That s an interesting question, and I figure it s worth its own top-level entry, as opposed to a reply in the comment stream. One of the interesting design questions in any OS or Computer Architecture is where the abstraction boundaries should be drawn and which side of an abstraction boundary should various operations be pushed. Linus s arguments is that there a flash controller can do a better job of wear leveling, including detecting how worn a particular flash cell might be (for example, perhaps by looking at the charge levels at an analog level and knowing when the last time the cell was programmed), and so it doesn t make sense to try to do wear leveling in a flash file system. Some responsibilities of flash management, such as coalescing newly written blocks into erase blocks to avoid write amplification can be done either on the SSD or in the file system for example, by using a log-structured file system, or some other copy-on-write file system, instead of a rewrite-in-place style file system, you can essentially solve the write amplification problem. In some cases, it s necessary let additional information leak across the abstraction for example, the ATA TRIM command is a way for the file system to let the disk know that certain blocks no longer need to be used. If too much information needs to be pushed across the abstraction, one way or another, then maybe we need to rethink whether the abstraction barrier is in the right place. In addition, if the abstraction has been around for a long time, changing it also has costs, which has to be taken into account. The 512 byte sector LBA abstraction has been around long time, and therefore dislodging it is difficult and costly. For example, the same argument which says that because the underlying hardware details are changing between different generations of SSD is all of these details should be hidden in hardware, was also used to justify something that has been a complete commercial failure for years if not decades: Object Based Disks. One of the arguments of OBD s was that the hard drive has the best knowledge of how and where to store an contiguous stream of bytes, and so perhaps filesystems should not be trying to decide where on disk an inode should be stored, but instead tell the hard drive, I have this object, which is 134 kilobytes long; please store it somewhere on the disk . At least in theory the HDD or SSD could handle all of the details of knowing the best place to store the object on the spinning magnetic media or flash media, taking into account how worn the flash is and automatically move the object around in the case of an SSD, and in the case of the HDD, the drive could know about (real) cylinder and track boundaries, and store the object in the most efficient way possible, since the drive has intimate knowledge about the low-level details of how data is stored on the disk. This theory makes a huge amount of sense; but there s only one problem. Object Based Disks have been proposed in academia and advanced R&D shops of companies like Seagate et.al. have been proposing them for over a decade, with absolutely nothing to show for it. Why? There have been two reasons proposed. One is that OBD vendors were too greedy, and tried to charge too much money for OBD s. Another explanation is that the interface abstraction for OBD s was too different, and so there wasn t enough software or file systems or OS s that could take advantage of OBD s. Both explanations undoubtedly contributed to the commercial failure of OBD s, but the question is which is the bigger reason. And the reason why it is particularly important here is because at least as far as Intel s SSD strategy is concerned, its advantage is that (modulo implementation shortcomings such as the reported internal LBA remapping table fragmentation problem and the lack of ATA TRIM support) filesystems don t need to change (much) in order to take advantage of the Intel SSD and get at least decent performance. However, if the price delta is a stronger reason for its failure, then the X25-M may be in trouble. Currently the 80GB Intel X25-M has a street price of $400, so it costs roughly $5 per gigabyte. Dumb MLC SATA SSD s are available for roughly half the cost/gigabyte (64 GB for $164). So what does the market look like 12-18 months from now? If dumb SSD s are still available at 50% of the cost of smart SSD s, it would probably be worth it to make a copy-on-write style filesystem that attempts to do the wear leveling and write amplification reduction in software. Sure, it s probably more efficient to do it in hardware, but a 2x price differential might cause people will settle for a cheaper solution even if isn t the absolutely best technical choice. On the hand, if prices drop significantly, and/or dumb SSD s completely disappear from the market, then time spent now optimizing for dumb SSD s will be completely wasted. So for Linus to make the proclamation that it s completely stupid to optimize for dumb SSD s seems to be a bit premature. Market externalities for example, does Intel have patents that will prevent competing smart SSD s from entering the market and thus forcing price drops? could radically change the picture. It s not just a pure technological choice, which is what makes projections and prognostications difficult. As another example, I don t know whether or not Intel will issue a firmware update that adds ATA TRIM support to the X25-M, or how long it will take before such SSD s become available. Until ATA TRIM support becomes available, it will be advantageous to add support in ext4 for a block allocator option that aggressively reuses blocks above all else, and avoids using blocks that have never been allocated or used before, even if it causes more in-file system fragmentation and deeper extent allocation trees. The reason for this is at the moment, once a block is used by the file system, at least today, the X25-M has absolutely no idea whether we still care about the contents of that block, or whether the block has since been released when the file was deleted. However, if 20% of the SSD s blocks have never been used, the X25-M can use 20% of the flash for better garbage collection and defragmentation algorithms. And if Intel never releases a firmware update to add ATA TRIM support, then I will be out $400 out of my own pocket for an SSD that lacks this capability, and so adding a block allocator which works around limitations of the X25-M probably makes sense. If it turns out that it takes two years before disks that have ATA TRIM support show up, then it will definitely make sense to add such an optimization. (Hard drive vendors have been historically S-L-O-W to finish standardizing new features and then letting such features enter the market place, so I m not necessarily holding my breath; after all, the Linux block device layer and and file systems have been ready to send ATA TRIM support for about six months; what s taking the ATA committees and SSD vendors so long? <grin> On the other hand, if Intel releases ATA TRIM support next month, then it might not be worth my effort to add such a mount option to ext4. Or maybe Sandisk will make an ATA TRIM capable SSD available soon, and which is otherwise competitive with Intel, and I get a free sample, but it turns out another optimization on Sandisk SSD s will give me an extra 10% performance gain under some workloads. Is it worth it in that case? Hard to tell, unless I know whether such a tweak addresses an optimization problem which is fundamental, and whether or not such a tweak will either be unnecessary, or perhaps actively unhelpful in the next generation. As long as SSD manufacturers force us treat these devices as black boxes, there may be a certain amount of cargo cult science which may be forced upon us file system designers or I guess I should say, in order to be more academically respectable, we will be forced to rely more on empirical measurements leading to educated engineering estimations about what the SSD is doing inside the black box . Heh. Related posts (automatically generated):
  1. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...

Theodore Ts'o: Should Filesystems Be Optimized for SSD s?

In one of the comments to my last blog entry, an anonymous commenter writes:
You seem to be taking a different perspective to linus on the adapting to the the disk technology front (Linus seems to against having to have the OS know about disk boundaries and having to do levelling itself)
That s an interesting question, and I figure it s worth its own top-level entry, as opposed to a reply in the comment stream. One of the interesting design questions in any OS or Computer Architecture is where the abstraction boundaries should be drawn and which side of an abstraction boundary should various operations be pushed. Linus s arguments is that there a flash controller can do a better job of wear leveling, including detecting how worn a particular flash cell might be (for example, perhaps by looking at the charge levels at an analog level and knowing when the last time the cell was programmed), and so it doesn t make sense to try to do wear leveling in a flash file system. Some responsibilities of flash management, such as coalescing newly written blocks into erase blocks to avoid write amplification can be done either on the SSD or in the file system for example, by using a log-structured file system, or some other copy-on-write file system, instead of a rewrite-in-place style file system, you can essentially solve the write amplification problem. In some cases, it s necessary let additional information leak across the abstraction for example, the ATA TRIM command is a way for the file system to let the disk know that certain blocks no longer need to be used. If too much information needs to be pushed across the abstraction, one way or another, then maybe we need to rethink whether the abstraction barrier is in the right place. In addition, if the abstraction has been around for a long time, changing it also has costs, which has to be taken into account. The 512 byte sector LBA abstraction has been around long time, and therefore dislodging it is difficult and costly. For example, the same argument which says that because the underlying hardware details are changing between different generations of SSD is all of these details should be hidden in hardware, was also used to justify something that has been a complete commercial failure for years if not decades: Object Based Disks. One of the arguments of OBD s was that the hard drive has the best knowledge of how and where to store an contiguous stream of bytes, and so perhaps filesystems should not be trying to decide where on disk an inode should be stored, but instead tell the hard drive, I have this object, which is 134 kilobytes long; please store it somewhere on the disk . At least in theory the HDD or SSD could handle all of the details of knowing the best place to store the object on the spinning magnetic media or flash media, taking into account how worn the flash is and automatically move the object around in the case of an SSD, and in the case of the HDD, the drive could know about (real) cylinder and track boundaries, and store the object in the most efficient way possible, since the drive has intimate knowledge about the low-level details of how data is stored on the disk. This theory makes a huge amount of sense; but there s only one problem. Object Based Disks have been proposed in academia and advanced R&D shops of companies like Seagate et.al. have been proposing them for over a decade, with absolutely nothing to show for it. Why? There have been two reasons proposed. One is that OBD vendors were too greedy, and tried to charge too much money for OBD s. Another explanation is that the interface abstraction for OBD s was too different, and so there wasn t enough software or file systems or OS s that could take advantage of OBD s. Both explanations undoubtedly contributed to the commercial failure of OBD s, but the question is which is the bigger reason. And the reason why it is particularly important here is because at least as far as Intel s SSD strategy is concerned, its advantage is that (modulo implementation shortcomings such as the reported internal LBA remapping table fragmentation problem and the lack of ATA TRIM support) filesystems don t need to change (much) in order to take advantage of the Intel SSD and get at least decent performance. However, if the price delta is a stronger reason for its failure, then the X25-M may be in trouble. Currently the 80GB Intel X25-M has a street price of $400, so it costs roughly $5 per gigabyte. Dumb MLC SATA SSD s are available for roughly half the cost/gigabyte (64 GB for $164). So what does the market look like 12-18 months from now? If dumb SSD s are still available at 50% of the cost of smart SSD s, it would probably be worth it to make a copy-on-write style filesystem that attempts to do the wear leveling and write amplification reduction in software. Sure, it s probably more efficient to do it in hardware, but a 2x price differential might cause people will settle for a cheaper solution even if isn t the absolutely best technical choice. On the hand, if prices drop significantly, and/or dumb SSD s completely disappear from the market, then time spent now optimizing for dumb SSD s will be completely wasted. So for Linus to make the proclamation that it s completely stupid to optimize for dumb SSD s seems to be a bit premature. Market externalities for example, does Intel have patents that will prevent competing smart SSD s from entering the market and thus forcing price drops? could radically change the picture. It s not just a pure technological choice, which is what makes projections and prognostications difficult. As another example, I don t know whether or not Intel will issue a firmware update that adds ATA TRIM support to the X25-M, or how long it will take before such SSD s become available. Until ATA TRIM support becomes available, it will be advantageous to add support in ext4 for a block allocator option that aggressively reuses blocks above all else, and avoids using blocks that have never been allocated or used before, even if it causes more in-file system fragmentation and deeper extent allocation trees. The reason for this is at the moment, once a block is used by the file system, at least today, the X25-M has absolutely no idea whether we still care about the contents of that block, or whether the block has since been released when the file was deleted. However, if 20% of the SSD s blocks have never been used, the X25-M can use 20% of the flash for better garbage collection and defragmentation algorithms. And if Intel never releases a firmware update to add ATA TRIM support, then I will be out $400 out of my own pocket for an SSD that lacks this capability, and so adding a block allocator which works around limitations of the X25-M probably makes sense. If it turns out that it takes two years before disks that have ATA TRIM support show up, then it will definitely make sense to add such an optimization. (Hard drive vendors have been historically S-L-O-W to finish standardizing new features and then letting such features enter the market place, so I m not necessarily holding my breath; after all, the Linux block device layer and and file systems have been ready to send ATA TRIM support for about six months; what s taking the ATA committees and SSD vendors so long? <grin> On the other hand, if Intel releases ATA TRIM support next month, then it might not be worth my effort to add such a mount option to ext4. Or maybe Sandisk will make an ATA TRIM capable SSD available soon, and which is otherwise competitive with Intel, and I get a free sample, but it turns out another optimization on Sandisk SSD s will give me an extra 10% performance gain under some workloads. Is it worth it in that case? Hard to tell, unless I know whether such a tweak addresses an optimization problem which is fundamental, and whether or not such a tweak will either be unnecessary, or perhaps actively unhelpful in the next generation. As long as SSD manufacturers force us treat these devices as black boxes, there may be a certain amount of cargo cult science which may be forced upon us file system designers or I guess I should say, in order to be more academically respectable, we will be forced to rely more on empirical measurements leading to educated engineering estimations about what the SSD is doing inside the black box . Heh. Related posts (automatically generated):
  1. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...

Alastair McKinstry: Why did the frogs cross the road ?

While driving home from Letterfrack to Moycullen last night after a pleasant days hillwalking with the family, we saw frogs on the road. Hundreds of them. Over the course of over more than more than twenty kilometres, we saw frogs apparently crossing the road. Can anyone tell me what was going on? It didn't look as though they were going somewhere in particular. Has they all recently spawned (unlikely) or where they looking for somewhere to spawn ? This was in the early evening: its possible that they all got to the road at dusk. That part of Conemara is all boggy hillside, with individual widely spaced houses. From up on a hill, the linear feature of the road (the N59), especially in the wet, might have been mistaken for a river. Did they head to the road to spawn? Tags , ,

Antti-Juhani Kaijanaho: grep-dctrl is ten years old

My message Intent to package a Debian control file grepper to WNPP and debian-devel is today a decade old. Apparently, the message predates the invention of the acronym ITP for Intent to Package (the first instance I can find is from May 1999). The changelog reveals that the first upload was on March 1st, 1999. There is unfortunately no record of when the package hit unstable the first time, since dinstall did not send Installed (or even the later Accepted) mails to an archived mailing list at that time. I suppose I might have it in some old private email archive, but more likely it s just gone. Similarly, the first fixed bug (#35527) predates BTS archiving and is now lost. The grep-dctrl program came out of repeated awkward grepping of the dpkg available and status files. Eventually I decided there must be a better way, and failed to find any canned solutions (I was later pointed to sgrep, which didn t look useful enough, and even later to dpkg-awk, but I had already committed to my own solution by then). I wrote a simple C program that processed these files as a sequence of records and did simple substring searches in each record. I rapidly added support for field selection, regular expressions and output field selection. By version 1.3a of March 2000 (which was released with Debian 2.2 potato ), the program was as good as it was going to get with one exception. Make disjunctive searches possible, said my TODO file those days. Conjunctive searches (that is, AND-searches) were possible even then by using more than one grep-dctrl command in a pipeline. Disjunctive (OR) was not possible. The problem was not so much that it would be hard to program (although the program s internal structure wasn t very good, to be honest, making extensive changes difficult), it was more a problem of coming up with a good command line syntax. Another thing that bothered me with the old grep-dctrl was how to implement Ben Armstrong s feature request. Again, the programming part wasn t the problem, the problem was coming up with a good, clean semantics for the feature. It was finally the appearance of ara in 2003 that got me moving again. Ara s author proudly compared eir program to grep-dctrl, claiming that my program did not do disjunctive searches while ara does. Competition being good for the soul, I took it as a challenge. In April 2003 I announced a complete rewrite of grep-dctrl, which was completed in January 2004 (the 2.0 release). The rewrite changed the way the command line was handled even though the usual Unix switch style is still used, the command line is regarded as a language with a parser (first an operator-precedence parser, then a recursive-descent one). The command line is transformed into an interpreted stack-based language which drives the actual grepping. The rewrite also generalised the internal data structures into an internal library which could be used to write other tools. The first such tool was sort-dctrl (introcuded in 2.7, June 2005), which was soon followed by tbl-dctrl (2.8, July 2005). The later appearance of join-dctrl (2.11, August 2007) finally allowed me to close Ben Armstrong s longstanding feature request mentioned above. The unpronounceable part of the names, dctrl , is an abbreviation for Debian control , which I decided to call the file format used by dpkg. Some people call it a RFC-822 format, but that really is a misnomer, since the differences between dctrl and RFC-822 outweigh the mainly superficial similarities (and the historical connection). I did consider calling my program dpkg-grep, but I didn t feel like I had the right to invade the dpkg namespace. The later rename to dctrl-tools reflects the fact that there are now several tools, grep-dctrl being just the oldest. I have several plans for the dctrl-tools suite, but my time and energy are mostly claimed by other responsibilities. The suite is currently team-maintained, but unfortunately the team is not very active. I would love it if I weren t the most active one with my busy schedule! Feel free to pop in on the dctrl-tools-devel mailing list, and to look at the Git repository and the todo list. If you decide to participate, please follow the rules.

Rudy Godoy: software development practices for personal development

Often I happen to come with some pretty crazy ideas about anything in life. That s me. Some of them may work, that s why I adventure to share them with the world. Working with computers, software in particular, for almost 12 years have brought interesting views regarding life. For the past 3 years I ve progressively improved the human being am I, and also the improvement process. The other day I was heading to the uni and I ve just noted that some practices of my profession can also be applied to my personal improvement. So, here I m sharing some of them. Hope you find them useful.
  1. Spot your bugs and kill em. While it s important to want to improve yourself. It s more relevant to the fact of real improvement that you spot your personal bugs . This is a crucial part of it all. Once you are aware of you personal bugs you ll be able to take action on them. So, when you do, just kill them and don t let stink until you are not able to manage it at all and all your code taints and breaks.
  2. Let process run on the background. Life has a limitation, which is time. Time is a scarce resource. You must be aware of it, really. While we might want to do many different things life presents to us, is not really responsible to engage on all. However, we can hack this. Here technology is our ally. There are many different ways to be able to do some things in the background while on the foreground you focus on what really matters to you at the time.
  3. Outsource services that are not key for you. Delegation and distributed architecture come to mind. Professions and specialization are an evolution of humanity. Please take advantage of it. You don t need to know or do everything. Karen Sthepenson s connectivism axiom is illustrative: I store my knowledge in my friends .
  4. Focus on what matters. This is somehow like the unix tradition of doing one thing and doing it well. Identify what makes you unique, special and relevant on your ecosystem. What value you can offer to others, and work on that direction. Hard work pays.
  5. Release early and release often. Once you have spot bugs and managed to fix them, release a new version of yourself!. Tell the world by showing an improved version of yourself. Make this incremental and iterative, just like a software development framework. For instance I currently run a 3.1 version of myself. Just released a point version last week :)

Next.

Previous.