Search Results: "ana"

14 June 2010

Russell Coker: Should Passwords Expire?

It s widely regarded that passwords should be changed regularly. The Australian government declared last week the National Cyber Security Awareness Week [1] and has published a list of tips for online security which includes Get a stronger password and change it at least twice a year . Can a Password be Semi-Public? Generally I think of a password as being either secret or broken. If a password is known to someone other than the sysadmin and the user who is authorised to use the account in question then you have potentially already lost all your secret data. If a password is disclosed to an unauthorised person on one occasion then merely changing the password is not going to do any good unless the root cause is addressed, otherwise another anothorised person will probably get the password at some future time. Hitachi has published a good document covering many issues related to password management [2]. I think it does a reasonable job of making sysadmins aware of some of the issues but there are some things I disagree with. I think it should be used as a list of issues to consider rather than a list of answers. The Hitachi document lists a number of ways that passwords may be compromised and suggests changing them every 60 to 90 days to limit the use of stolen passwords. This seems to imply that a password is something that s value slowly degrades over time as it s increasingly exposed. I think that the right thing to do is to change a password if you suspect that it has been compromised. There s not much benefit in having a password if it s going to be known by unauthorised people for 89 days before being changed! Fundamentally a password is something that can have it s value rapidly drop to zero without warning. It doesn t wear out. Why are terms such as Three Months used for Maximum Password Ages? The Hitachi document gives some calculations on the probability of a brute-force attack succeeding against a random password with 90 days of attacking at a rate of 100 attempts per second [2]. I think that if a service is run by someone who wouldn t notice the load of 100 attempts per second then you have bigger security problems than the possibility of passwords being subject to brute-force attacks. Also it s not uncommon to have policies to lock accounts after as few as three failed login attempts. Rumor has it that in the early days of computing when the hashed password data was world readable someone calculated that more than 3 months of CPU time on a big computer would be needed to obtain a password by brute-force. But since then the power of individual CPUs has increased dramatically, computers have become cheap enough that anyone can easily gain legal access to dozens of systems and illegal access to a million systems, and it has become a design feature in every OS that hashed passwords are not readable by general users. So the limiting factor is to what degree the server restricts the frequency of password guesses. I don t think that specifying the minimum password length and maximum password age based on the fraction of the key space that could be subject to a brute-force attack makes sense. I don t think that any attempt to make an industry-wide standard for the frequency of password changes (as the government is trying to do) makes sense. Can there be a Delay between a Password being Compromised and being Used by an Attacker? Hypothetically speaking, if a password was likely to be compromised (EG by having the paper it was written on lost or stored insecurely) for some time before an attacker exploited it, then if the password was changed during that time period it could solve the problem. For example when a company moves office there is the possibility of notepaper with passwords to be lost. So if the sysadmin caused every user password to expire at the time of the move then a hostile party would be unable to gain access. Another possibility is the theft of backup tapes that contain the list of unencrypted passwords. If users change their passwords every three months then the theft of some four month old backup tapes will be less of a problem. Another possibility concerns the resale of old computers, phones, and other devices that may contain passwords. A reasonably intelligent user won t sell their old hardware as soon as the replacement device arrives, they will want to use the new device for some time to ensure that it works correctly. If passwords expire during this trial period with the new device then passwords stored in the old device won t have any value. The down-side to this idea is that people probably sell their old gear fairly quickly and making passwords expire every two weeks would not be accepted well by the users. It seems to me that having bulk password changes (all passwords for one user or for one company) based on circumstances that lead to potential insecurity would do more good than changing passwords at a fixed schedule. How are Passwords typically Compromised? Dinei Flor ncio and Cormac Herley of Microsoft Research and Baris Coskun of Brooklyn Polytechnic University wrote a paper titled Do Strong Web Passwords Accomplish Anything? [3] which discusses this issue. The first thing that they note is that nowadays passwords are most commonly compromised by phishing and keylogging. In those cases passwords are typically used shortly after they are stolen and the strength of a password never does any good. That paper suggests that banks should use stronger user-names rather than stronger passwords to combat the threat of bulk brute-force attacks. Can a Password Last Forever? If a password is entered in a secure manner, authenticated by a secure server, and all network links are encrypted or physically protected then there should never be a need to change it. Of course nothing is perfectly secure, so for some things with minimal short-term value or which can be used without anyone noticing there is a benefit in changing the password. But in the case of Internet banking if a hostile party gets your login details then you will probably know about it in a few days when the bank calls you about the unusual transactions from foreign countries long before a 90 day password change schedule would have done any good. Maybe one of the issues determining whether a password should be changed regularly is whether an attacker could use long-term read-only access to gain some benefit. Being able to read all the email someone received for a year could be a significant benefit if that person was a public figure, and there s usually no way for an ISP customer to know that someone else is downloading all their mail via POP or IMAP. Should a Password be the only Authentication Method? It is generally agreed that an authenitcation method should ideally involve something you have plus something you know. That means a password and a physical device such as a smart card, token with a changing sequential password, or a key such as a Yubikey [4]. If the physical device can t be cloned (through some combination of technical difficulty and physical access control) then it significantly improves security. When a physical device is used the purpose of the password is merely to stop someone who steals the physical device from being immediately exploit everything the password only has to be strong enough to keep the accounts secure until a new token can be issued. The combination of something you have and something you know is very strong. Having a physical token stored on the desk next to the PC that is used for logging in provides a significant benefit, then an attacker needs to break in to the house and can t sniff the password by compromising the PC remotely. Conclusion In all aspects of security you need to consider what threats you face. If an attacker is likely to launch an immediate noisy attack (such as transferring the maximum funds out of an Internet banking account) then changing the password regularly won t do any good. If a subtle long-term attack is expected then changing the password can do a lot of good but a physical token is the ideal if the account is valuable enough. But to put things in to perspective, it s technically possible to use a mobile phone camera at close range (or a SLR with a big lens at long range) to take a photo of keys that allow reproducing them. But this hasn t stopped people from carrying their house keys in an obvious manner that permits photography or leaving them on their desk at work. Also I ve never heard of anyone routinely changing the door locks in case a hostile party might have got a key although I m sure that such practices are common in highly secure locations. Few people even take their house keys off the key-ring when they have their car serviced! Related Posts Defense in Depth and Sudo when using sudo can increase security and when it can t.
Logging Shell Commands how to log what the sysadmin does and what benefits that provides you, it doesn t help if the sysadmin is hostile.
Logging in as Root should you login directly as root?

Christine Spang: Christine Spang: the end of an era

The world around me seems to whirl these days. One week ago, I graduated from MIT. People I've known during the last four years have been dispelling to various parts of the globe one by one, day by day. California, Canada, Indonesia, Seattle. Some will be back again. Some will not, or if so only to visit. pika is a continuous bustle of activity as the summer has commenced and it has filled with creative and adventurous MIT students who've suddenly found themselves having free time. A hammock being built on the roofdeck. Thrice-weekly icecream forays. Common areas overflowing with people playing musical instruments, chatting, and messing around on laptops. Summer's warmth has arrived, bringing with it farmer's markets, strawberry picking, and swimming expeditions. While it's wonderful to get to meet so many new people living in a college environment, I can't help but feel sadness thinking about everyone who's left. There are always more friends to be made as new people arrive, but old ones moving away leave bittersweet memories, and the new relationships are always a bit different as the age discrepancy between me and others changes. Or the me-the-ephemeral-collection-of-thoughts-which-when-regarding-other-people-sometimes-involve-the-mentor/mentee-distinctions-caused-by-one-party-being-older-or-more-knowledgeable-than-the-other-at-least-in-certain-areas changes. The end of a semester always feels like this, but this year even more so as the people I started university with start down new paths. For me, that was going to involve staying on at MIT to complete a one-year master's program, the "M.Eng." in electrical engineering and computer science. That plan, too, has changed. I've deferred the degree and accepted a full-time engineering position at Ksplice, an exciting early-stage Linux startup here in Cambridge. I'd been working at Ksplice part-time since January before joining full-time immediately following graduation. Ksplice is the realization of ideas I saw being born on the whiteboard at SIPB when I was a freshman, and it's fun to see that play out in a small, ever-changing, low-bullshit company. All in all, there are many more exciting things down the road, and, working at an MIT startup, I haven't even escaped the MIT/Cambridge reality-distortion bubble yet. Still, it's tempting to resist change and let myself romanticize the good old days, hoping to catch every person I've ever enjoyed spending time with and hold them down here forever. That's not the way life works, though. Change happens.

13 June 2010

Gunnar Wolf: World Naked Bike Ride 2010 Mexico

For the second time (First time was in 2008; I didn't join in 2009 as I travelled to Nicaragua on that date), I took part of the World Naked Bike Ride. The WNBR is a global effort, where people in ~150 cities all over the world go cycling nude on the streets of our towns, with varied demands, including: I love my bike! One of the things I most like about WNBR is its diversity. Not everybody goes for the same reasons. As people who read me often will know, I took part because I believe (and act accordingly!) that the bicycle is the best, most efficient vehicle in by far most of the situations we face day to day, but we need to raise awareness in everybody that the bicycle is just one more vehicle: On one side, we have the right to safely ride on the streets, like any other vehicle. On the other side, we must be responsible, safe drivers, just as we want car drivers to be. Ok, and I will recognize it before anybody complains that I sound too idealistic: I took part of the WNBR because it is _tons_ of fun. This year, we were between 300 and 500 people (depending on whom you ask). Compared to 2008, I felt less tension, more integration, more respect within the group. Of course, it is only natural in the society I live in that most of the participants were men, but the proportion of women really tends to even out. Also, many more people joined fully or partially in the nude (as nudity is not required, it is just an invitation). There was a great display of creativity, people painted with all kinds of interesting phrases and designs, some really beautiful. Oh, one more point, important to me: This is one of the best ways to show that we bikers are not athletes or anything like that. We were people ranging from very thin to quite fat, from very young to quite old. And that is even more striking when we show our whole equipment. If we can all bike around... So can you! Some links, with obvious nudity warnings in case you are offended by looking at innocent butts and similar stuff: As for the sad, stupid note: 19 cyclists were placed under arrest in Morelia, Michoac n because of faltas a la moral (trasgressions against morality), an ill-defined and often abused concept. Also, by far, most of the comments I have read from people on the media, as well a most questions we had by reporters before or after the ride were either why are you going nude (because that's the only way I'll get your attention!) or But many people were not nude! (nudity is not a requirement but only an option.

Petter Reinholdtsen: Lenny->Squeeze upgrades, removals by apt and aptitude

My testing of Debian upgrades from Lenny to Squeeze continues, and I've finally made the upgrade logs available from http://people.skolelinux.org/pere/debian-upgrade-testing/. I am now testing dist-upgrade of Gnome and KDE in a chroot using both apt and aptitude, and found their differences interesting. This time I will only focus on their removal plans. After installing a Gnome desktop and the laptop task, apt-get wants to remove 72 packages when dist-upgrading from Lenny to Squeeze. The surprising part is that it want to remove xorg and all xserver-xorg-video* drivers. Clearly not a good choice, but I am not sure why. When asking aptitude to do the same, it want to remove 129 packages, but most of them are library packages I suspect are no longer needed. Both of them want to remove bluetooth packages, which I do not know. Perhaps these bluetooth packages are obsolete? For KDE, apt-get want to remove 82 packages, among them kdebase which seem like a bad idea and xorg the same way as with Gnome. Asking aptitude for the same, it wants to remove 192 packages, none which are too surprising. I guess the removal of xorg during upgrades should be investigated and avoided, and perhaps others as well. Here are the complete list of planned removals. The complete logs is available from the URL above. Note if you want to repeat these tests, that the upgrade test for kde+apt-get hung in the tasksel setup because of dpkg asking conffile questions. No idea why. I worked around it by using 'echo >> /proc/pidofdpkg/fd/0' to tell dpkg to continue. apt-get gnome 72
bluez-gnome cupsddk-drivers deskbar-applet gnome gnome-desktop-environment gnome-network-admin gtkhtml3.14 iceweasel-gnome-support libavcodec51 libdatrie0 libgdl-1-0 libgnomekbd2 libgnomekbdui2 libmetacity0 libslab0 libxcb-xlib0 nautilus-cd-burner python-gnome2-desktop python-gnome2-extras serpentine swfdec-mozilla update-manager xorg xserver-xorg xserver-xorg-core xserver-xorg-input-all xserver-xorg-input-evdev xserver-xorg-input-kbd xserver-xorg-input-mouse xserver-xorg-input-synaptics xserver-xorg-input-wacom xserver-xorg-video-all xserver-xorg-video-apm xserver-xorg-video-ark xserver-xorg-video-ati xserver-xorg-video-chips xserver-xorg-video-cirrus xserver-xorg-video-cyrix xserver-xorg-video-dummy xserver-xorg-video-fbdev xserver-xorg-video-glint xserver-xorg-video-i128 xserver-xorg-video-i740 xserver-xorg-video-imstt xserver-xorg-video-intel xserver-xorg-video-mach64 xserver-xorg-video-mga xserver-xorg-video-neomagic xserver-xorg-video-nsc xserver-xorg-video-nv xserver-xorg-video-openchrome xserver-xorg-video-r128 xserver-xorg-video-radeon xserver-xorg-video-radeonhd xserver-xorg-video-rendition xserver-xorg-video-s3 xserver-xorg-video-s3virge xserver-xorg-video-savage xserver-xorg-video-siliconmotion xserver-xorg-video-sis xserver-xorg-video-sisusb xserver-xorg-video-tdfx xserver-xorg-video-tga xserver-xorg-video-trident xserver-xorg-video-tseng xserver-xorg-video-v4l xserver-xorg-video-vesa xserver-xorg-video-vga xserver-xorg-video-vmware xserver-xorg-video-voodoo xulrunner-1.9 xulrunner-1.9-gnome-support aptitude gnome 129
bluez-gnome bluez-utils cpp-4.3 cupsddk-drivers dhcdbd djvulibre-desktop finger gnome-app-install gnome-mount gnome-network-admin gnome-spell gnome-vfs-obexftp gnome-volume-manager gstreamer0.10-gnomevfs gtkhtml3.14 libao2 libavahi-compat-libdnssd1 libavahi-core5 libavcodec51 libbluetooth2 libcamel1.2-11 libcdio7 libcucul0 libcupsys2 libcurl3 libdatrie0 libdirectfb-1.0-0 libdvdread3 libedataserver1.2-9 libeel2-2.20 libeel2-data libepc-1.0-1 libepc-ui-1.0-1 libfaad0 libgail-common libgd2-noxpm libgda3-3 libgda3-common libgdl-1-0 libgdl-1-common libggz2 libggzcore9 libggzmod4 libgksu1.2-0 libgksuui1.0-1 libgmyth0 libgnomecups1.0-1 libgnomekbd2 libgnomekbdui2 libgnomeprint2.2-0 libgnomeprint2.2-data libgnomeprintui2.2-0 libgnomeprintui2.2-common libgnomevfs2-bin libgpod3 libgraphviz4 libgtkhtml2-0 libgtksourceview-common libgtksourceview1.0-0 libgucharmap6 libhesiod0 libicu38 libiw29 libkpathsea4 libltdl3 libmagick++10 libmagick10 libmalaga7 libmetacity0 libmtp7 libmysqlclient15off libnautilus-burn4 libneon27 libnm-glib0 libnm-util0 libopal-2.2 libosp5 libparted1.8-10 libpoppler-glib3 libpoppler3 libpt-1.10.10 libpt-1.10.10-plugins-alsa libpt-1.10.10-plugins-v4l libraw1394-8 libsensors3 libslab0 libsmbios2 libsoup2.2-8 libssh2-1 libsuitesparse-3.1.0 libswfdec-0.6-90 libtalloc1 libtotem-plparser10 libtrackerclient0 libxalan2-java libxalan2-java-gcj libxcb-xlib0 libxerces2-java libxerces2-java-gcj libxklavier12 libxtrap6 libxxf86misc1 libzephyr3 mysql-common nautilus-cd-burner openoffice.org-writer2latex openssl-blacklist p7zip python-4suite-xml python-eggtrayicon python-gnome2-desktop python-gnome2-extras python-gtkhtml2 python-gtkmozembed python-numeric python-sexy serpentine svgalibg1 swfdec-gnome swfdec-mozilla totem-gstreamer update-manager wodim xserver-xorg-video-cyrix xserver-xorg-video-imstt xserver-xorg-video-nsc xserver-xorg-video-v4l xserver-xorg-video-vga zip apt-get kde 82
cupsddk-drivers karm kaudiocreator kcoloredit kcontrol kde kde-core kdeaddons kdeartwork kdebase kdebase-bin kdebase-bin-kde3 kdebase-kio-plugins kdesktop kdeutils khelpcenter kicker kicker-applets knewsticker kolourpaint konq-plugins konqueror korn kpersonalizer kscreensaver ksplash libavcodec51 libdatrie0 libkiten1 libxcb-xlib0 quanta superkaramba texlive-base-bin xorg xserver-xorg xserver-xorg-core xserver-xorg-input-all xserver-xorg-input-evdev xserver-xorg-input-kbd xserver-xorg-input-mouse xserver-xorg-input-synaptics xserver-xorg-input-wacom xserver-xorg-video-all xserver-xorg-video-apm xserver-xorg-video-ark xserver-xorg-video-ati xserver-xorg-video-chips xserver-xorg-video-cirrus xserver-xorg-video-cyrix xserver-xorg-video-dummy xserver-xorg-video-fbdev xserver-xorg-video-glint xserver-xorg-video-i128 xserver-xorg-video-i740 xserver-xorg-video-imstt xserver-xorg-video-intel xserver-xorg-video-mach64 xserver-xorg-video-mga xserver-xorg-video-neomagic xserver-xorg-video-nsc xserver-xorg-video-nv xserver-xorg-video-openchrome xserver-xorg-video-r128 xserver-xorg-video-radeon xserver-xorg-video-radeonhd xserver-xorg-video-rendition xserver-xorg-video-s3 xserver-xorg-video-s3virge xserver-xorg-video-savage xserver-xorg-video-siliconmotion xserver-xorg-video-sis xserver-xorg-video-sisusb xserver-xorg-video-tdfx xserver-xorg-video-tga xserver-xorg-video-trident xserver-xorg-video-tseng xserver-xorg-video-v4l xserver-xorg-video-vesa xserver-xorg-video-vga xserver-xorg-video-vmware xserver-xorg-video-voodoo xulrunner-1.9 aptitude kde 192
bluez-utils cpp-4.3 cupsddk-drivers cvs dcoprss dhcdbd djvulibre-desktop dosfstools eyesapplet fifteenapplet finger gettext ghostscript-x imlib-base imlib11 indi kandy karm kasteroids kaudiocreator kbackgammon kbstate kcoloredit kcontrol kcron kdat kdeadmin-kfile-plugins kdeartwork-misc kdeartwork-theme-window kdebase-bin-kde3 kdebase-kio-plugins kdeedu-data kdegraphics-kfile-plugins kdelirc kdemultimedia-kappfinder-data kdemultimedia-kfile-plugins kdenetwork-kfile-plugins kdepim-kfile-plugins kdepim-kio-plugins kdeprint kdesktop kdessh kdict kdnssd kdvi kedit keduca kenolaba kfax kfaxview kfouleggs kghostview khelpcenter khexedit kiconedit kitchensync klatin klickety kmailcvt kmenuedit kmid kmilo kmoon kmrml kodo kolourpaint kooka korn kpager kpdf kpercentage kpf kpilot kpoker kpovmodeler krec kregexpeditor ksayit ksim ksirc ksirtet ksmiletris ksmserver ksnake ksokoban ksplash ksvg ksysv ktip ktnef kuickshow kverbos kview kviewshell kvoctrain kwifimanager kwin kwin4 kworldclock kxsldbg libakode2 libao2 libarts1-akode libarts1-audiofile libarts1-mpeglib libarts1-xine libavahi-compat-libdnssd1 libavahi-core5 libavc1394-0 libavcodec51 libbluetooth2 libboost-python1.34.1 libcucul0 libcurl3 libcvsservice0 libdatrie0 libdirectfb-1.0-0 libdjvulibre21 libdvdread3 libfaad0 libfreebob0 libgail-common libgd2-noxpm libgraphviz4 libgsmme1c2a libgtkhtml2-0 libicu38 libiec61883-0 libindex0 libiw29 libk3b3 libkcal2b libkcddb1 libkdeedu3 libkdepim1a libkgantt0 libkiten1 libkleopatra1 libkmime2 libkpathsea4 libkpimexchange1 libkpimidentities1 libkscan1 libksieve0 libktnef1 liblockdev1 libltdl3 libmagick10 libmimelib1c2a libmozjs1d libmpcdec3 libneon27 libnm-util0 libopensync0 libpisock9 libpoppler-glib3 libpoppler-qt2 libpoppler3 libraw1394-8 libsmbios2 libssh2-1 libsuitesparse-3.1.0 libtalloc1 libtiff-tools libxalan2-java libxalan2-java-gcj libxcb-xlib0 libxerces2-java libxerces2-java-gcj libxtrap6 mpeglib networkstatus openoffice.org-writer2latex pmount poster psutils quanta quanta-data superkaramba svgalibg1 tex-common texlive-base texlive-base-bin texlive-common texlive-doc-base texlive-fonts-recommended xserver-xorg-video-cyrix xserver-xorg-video-imstt xserver-xorg-video-nsc xserver-xorg-video-v4l xserver-xorg-video-vga xulrunner-1.9

12 June 2010

Matt Zimmerman: How to decide what to read (and what not to read)?

Like you, dear Internet readers, I have no shortage of reading material. I have ready access to more engaging, high quality, informative and relevant information than I can possibly digest. Every day, I have to choose what to read, and what to pass by. This seems like an important thing to do well, and I wonder if I do a good enough job of it. This is just one example of a larger breadth/depth problem, but I m finding the general problem difficult to stomach, so I m focusing on reading for the moment. These are my primary sources of reading material on a day-to-day basis: How do you decide what to read, and what not to read? How does your experience differ between your primary information sources? How have you tried to improve?

Joachim Breitner: bluetile in Debian

I just packaged and uploaded Jan Vornberger s window manager bluetile to Debian. This very nice piece of software brings the benefits of a tiling window manager to users who prefer to use the mouse and who don t want to learn a new programing language to configure their window manager. Bluetile uses the xmonad libraries and extends them with an easy to use and discoverable user interface.

Bradley Smith: Gitorious + Blog = Fun (+ Ruby)

Something I've wanted to do for a while now, is somehow combine my blog with all of my projects/git repositories, and now I've done it! Welcome to my new site :) So, I got bored a few nights ago, and started trying to find some kind of project management type system to host my git repositories in. After searching for a while I stumbled across Gitorious. I'd heard of Gitorious before, but I'd never really looked at it, nor did I know that it's codebase was infact completely open. So I thought I'd give it a try to see what it was like Now I have to admit, my first impressions of it weren't exactly great. Firstly, it uses Ruby, which I can't say I'm much of a fan of (but that's a different discussion :p), nor am I a fan of installing all of the 'packages' it needed, via gem. Deploying it is an absolute nightmare, it has 4 different daemons, all of which have hideous init scripts that don't really work properly, and low and behold you try and put the git repositories and the site itself under different users, figuring out what needs to run as what and what needs what permissions is very much a game of trial and error. In Gitorious' defence, there /are/ numerous guide lurking around on the internet explaining how to do these things. But once I'd got over these niggles, and had actually deployed it, it worked quite nicely, and really did exactly what I wanted, so I decided to stick with it. It then occurred to me that it probably wouldn't be too hard to stick a blog onto it, after all, it's written using Rails, which is supposed to be easy, right? So with no knowledge of Ruby, let alone Rails, I got stuck in. Fortunately, it turns out Rails is in fact pretty easy to use, (at least once you've got used to it implicitly doing everything behind the scenes..), I'm even starting to think I prefer it to Django... So there we have it, a brand new site, and one that looks considerably better than my old one.. even if it is just Gitorious in disguise _ The only thing I'm disappointed Gitorious doesn't have, is some kind of bug tracker, but then, I'm not sure I'd use one given most of my projects just go through Debian.

Russell Coker: Links June 2010

Seth Berkley gave an interesting TED talk about developing vaccines against the HIV and Influenza viruses [1]. The part I found most interesting was the description of how vaccines against viruses are currently developed using eggs and how they plan to use bacteria instead for faster and cheaper production. One of the problems with using eggs is that if the chickens catch the disease and die then you can t make a vaccine. Aigars Mahinovs wrote a really good review of Microsoft Azure and compared it to Amazon EC2 [2]. It didn t surprise me that Azure compared poorly to the competition. Johanna Blakley gave an insightful TED talk about IP lessons from the fashion industry [3]. She explained how an entire lack of IP protection other than trademark law was an essential part of the success of the fashion industry. She also compared the profits in various industries and showed that industries with little or no IP protection involve vastly larger amounts of money than industries with strong IP protection. Lisa D wrote an insightful post about whether Autism Spectrum Disorders (such as Asperger Syndrome) should be considered to be disabilities [4]. I don t entirely agree with her, but she makes some really good points. Sharmeen Obaid-Chinoy gave an interesting TED talk about the way the Taliban train young children to become suicide bombers [5]. Apparently the Taliban prey on large poor families, sometimes paying the parents for taking children away to school . At the Taliban schools the children are beaten, treated poorly, and taught theology by liars who will say whatever it takes to get a result. Then after being brain-washed they are sent out to die. Wired has an interesting article about Charles Komanoff s research into New York traffic problems [6]. He aims to track all the economic externalities of traffic patterns and determine incentives to encourage people to do things that impose less costs on the general economy. His suggestions include making all bus travel free as the externality of the time spent collecting fares is greater than the fare revenue. It s a really interesting article, his research methods should be implemented when analysing traffic in all large cities, and many of his solutions can be implemented right now without further analysis such as free buses and variable ticket pricing according to the time of day. William Li gave an interesting TED talk about starving cancer by preventing new blood vessels from growing to feed it [7]. Drugs to do this have been shown to increase the life expectancy of cancer patients by more than 100% on average. Also autopsies of people who died in car accidents show that half the women in their 40 s had breast cancer and half the men in their 50 s prostate cancer but those cancers didn t grow due to a natural lack of blood supply, so the aim here is to merely promote what naturally happens in terms of regulating cancers and preventing them from growing larger than 0.5mm^3. There are a number of foods that prevent blood vessels growing to cancers which includes dark chocolate! ;) Also drugs which prevent blood vessel growth also prevent obesity, I always thought that eating chocolate all the time prevented me from getting fat due to the central nervous system stimulants that kept me active Graham Hill gave an inspiring TED talk about becoming a weekday vegetarian [8]. Instead of making a commitment to being always vegetarian he s just mostly vegetarian (only eating meat on Sundays). He saves most of the environmental cost and doesn t feel guilty if he ever misses a day. It s an interesting concept. Cory Doctorow wrote an insightful article for the Guardian about the phrase Information Wants To Be Free [9]. He points out that really it s people who want to be free from the tyranny that is being imposed on us in the name of anti-piracy measures. He also points out that it s a useful straw-man for the MAFIAA to use when claiming that we are all pirates. The Atlantic has an interesting article about the way that Google is working to save journalistic news [10]. Adam Sadowsky gave an interesting TED talk about creating a Rube Goldberg machine for the OK Go video This Too Shall Pass [11]. At the end of the talk they include a 640*480 resolution copy of the music video. Brian Cox gave an interesting TED talk advocating increased government spending on scientific research [12]. Among other things he pointed out that the best research indicates that the amount of money the US government invested in the Apollo program was returned 14* to the US economy due to exports of new American products that were based on that research. It s surprising that any justification other than the return on investment for the Apollo program is needed! Moot gave an interesting TED talk about Anonymity [13]. I don t think that he made a good case for anonymity, he cited one person being identified and arrested for animal cruelty due to the efforts of 4chan people and also the campaign against the Cult of Scientology (which has not been very successful so far). Rory Sutherland gave an intriguing TED talk titled Sweat the Small Stuff [14]. He describes how small cheap changes can often provide greater benefits than huge expensive changes and advocates corporations having a Chief Detail Officer to take charge of identifying and implementing such changes. TED Hosted an interesting debate between pro and anti nuclear campaigners [15]. They agreed that global warming is a significant imminent problem but disagree on what methods should be implemented to solve it.

11 June 2010

Raphaël Hertzog: About the Debian Community Poll

While I find the idea interesting, several of the questions can t be correctly answered because the proposed choices are not realistic or too limited. On the question of the usage of money, I believe we should spend money to fund important projects but I don t want to fund people having important positions in Debian and doing important work . What should I reply? (Granted, there s the other item but that doesn t help getting a clear picture of the answers) On the question Do you prefer time based releases instead of the it s ready when it s ready releases? , it is putting two concepts in opposition when the release managers recently proposed a third way that combines both: time based freezes and release when it s ready . This is what I want and I can t adequately express it either in the current poll. Partagez cet article / Share This

10 June 2010

Russell Coker: Logging Shell Commands

In response to my previous post about logging in directly as root [1] it was suggested that using sudo is the only way to log the commands that are entered as root. One reason for doing this is if you don t trust the people who are granted root access and you want to log all commands to a remote server that is run by different people. I wonder whether it is really possible to run systems with untrusted sysadmins, if someone can apply patches etc then they can surely install a trojan and then wait a while before activating it to make things more difficult for anyone who is analysing the logs. One of the many issues is that even the restricted version of vim permits the :r and :w commands, so one could start vim from sudo with an innocuous file as the target of the edit operation and then read and write some critical file such as /etc/shadow. I expect that someone has written an editor which has a restricted mode that doesn t allow reading/writing files other than the one specified on the command-line, and if not it surely wouldn t be difficult to patch vim (or your favorite editor) to have such a mode of operation. But there are always other programs that can access files other than the ones specified on their command-line. It seems that using the auditctl interface to log access to certain critical files (EG read access to /etc/shadow and write access to everything under /etc, /bin, /sbin, and /usr) would be a necessary part of an effective auditing strategy and that sudo would only comprise a small part of a useful configuration. There are other viable ways of logging everything that is done as root which offer benefits over sudo. Ways of Logging Shell Commands The Korn shell supports doing all the logging you might desire as part of a shell function [2]. Bash can have a similar shell function to do the logging, but when a command is entered the previous command is logged [3], this means that any single bash command that unsets this will never be logged. It might be possible to solve this if you know more about Bash than I do. I wonder if the Korn shell function has the same issue. This is still probably useful for some situations when you want to track what honorable sysadmins do, but of little benefit for tracking hostile sysadmins (tracking hostile sysadmins is actually possible). You can put code in a file such as /etc/bash.bash_logout to log the commands elsewhere, but even trivial things such as kill -9 $$ can defeat that so it s only useful when the sysadmin is trusted. The Sudoshell project exists to log all data that is entered in a shell [3]. One deficiency of this for the people who don t trust the root user is that it logs the data to files on disk, but it shouldn t be difficult to rewrite sudoscriptd to write directly to another machine over the network. Also one benefit of this for auditing is that it captures all the output of the commands as well (which can be a little inconvenient to decipher when curses programs are run. The web site also describes some of the problems with trying to use sudo directly for everything (such as pipelines). If you compile Bash version 4.1 with the SYSLOG_HISTORY macro enabled (which can be done by editing the file config-top.h) then it will log all commands to syslog. RootShell.be has a short post about this which mentions the security issues, some commands take passwords as parameters and these passwords could be exposed to the network [5]. Of course the best option is to just avoid such commands. Thanks to Chris Samuel for pointing out the Bash logging feature. Conclusion If you use sudo for auditing root access then you lose some shell functionality. Sudo also only logs the commands that are executed you don t get logging of output. It seems that depending on the exact needs either a modified version of Sudoshell or the logging that can be compiled in to Bash would be the way to go depending on the exact requirements. The main benefit of using sudo for logging would be that some distributions of Linux are configured that way by default but it seems unlikely that someone would go to the effort of running a separate logging server that the regular sysadmin team can t touch and then configure their servers in a default manner.

9 June 2010

Ross Burton: Gypsy 0.8 Released

As acting release engineer of the Gypsy project (a GPS mux, if you didn't know) I'm proud to announce the release of Gypsy 0.8. So, what's new? Many thanks to Jussi Kukkonen for patch review, and Bastien Nocera for patch review and new features. The big question of course is what of the future? So far we've got some rough ideas. An overhaul of the device interaction layer is definitely required as actaully getting NMEA is becoming more complex: for integrated 3G/GPS chips you need to talk to oFono/ModemManager to get a socket, for some embedded GPS devices you need a proprietary binary that writes to a pipe, and so on. There are some new features we're considering too: server-side proximity detection and update rate limiting.

David Welton: Some difficult decisions?

I am, by nature, a tinkerer. I love to build things to see how they will work or because I have an idea that I think is fun. I am not so much of a natural business guy. I go build something "because it's there". Or rather, it's there in my mind, and I want to make it. I earn money, however, via consulting work. There are problems with this line of business though, namely that it doesn't scale very well unless, besides being good at a skill, you've also got the skills to build up a consulting business, which is very much about adding and managing people, something I'm not all that passionate about and would rather avoid. So what happens is that you go through slumps with less than full capacity, and then things pick up again, and before you know it, you hit your upper limit and can't take on any new work, putting a ceiling over your potential earnings. Sure, you can raise prices a bit, but that'll only get you so far. For a while now, I've been considering this problem, and I think some sort of product would be an ideal way out. Stories like Balsamiq's are inspirational, and along the lines of what I think might be nice. Don't get me wrong, I like my clients and like to help people solve problems a lot, but it's a stressful job in that you're never very far from having no income. I'm not much of an entrepreneur, though, in the sense of someone with a nose for making money. I tend to think about building something cool first, and not really worrying about making money with it. Predictably, some of the sites I have created are popular, but don't make any money. Sure, they have some ads, but that doesn't really make much unless you get really huge page views or have some sort of content that attracts users who will click on very lucrative advertisements. I've started to think that perhaps I should constrain myself in new initiatives to creating things that have a direct source of revenue: you want to use it, you pay. In my daydream world, I'd probably just hack on open source all day long, but there's no money in that for someone like me who is not the author of some huge, famous piece of software, nor a particularly great hacker in any case (I've met some, like Andrew Tridgell, and it's a really humbling experience). So that brings up the question of what to do with the aforementioned sites. One option would be to sell them off via a site like flippa.com, another would be simply to just let them sit there and stew, but they do take up some resources on my server, and require some maintainance now and then, and they're a bit of a distraction too (I start wondering about things to add or how to fiddle with them). Maybe there's a way to make money from some of them, but I'm too blind to see it. The ones I'm thinking about, are, in no particular order:

8 June 2010

Matt Zimmerman: DevOps and Cloud

DevOps I first heard about DevOps from Lindsay Holmwood at linux.conf.au 2010. Since then, I ve been following the movement with interest. It seems to be about cross-functional involvement in software teams, specifically between software development and system administration (or operations). In many organizations, especially SaaS shops, these two groups are placed in opposition to each other: developers are driven to deliver new features to users, while system administrators are held accountable for the operation of the service. In the best case, they maintain a healthy balance by pushing in opposite directions, but more typically, they resent each other for getting in the way, as a result of this dichotomy:
Development Operations
is responsible for creating products offering services
is measured on delivery of new features high reliability
optimizes by increasing velocity controlling change
and so is perceived as reckless and irresponsible obstructing progress
Of course, both functions are essential to a viable service, and so DevOps aims to replace this opposition with cooperation. By removing this friction from the organization, we hope to improve efficiency, lower costs, and generally get more work done.
So, DevOps promotes the formation of cross-functional teams, where individuals still take on specialist development or operations roles, but work together toward the common goal of delivering a great experience to users. By working as teammates, rather than passing work over the wall , they can both contribute to development, deployment and maintenance according to their skills and expertise. The team becomes a devops team, and is responsible for the entire product life cycle. Particular tasks may be handled by specialists, but when there s a problem, it s the team s problem. Some take it a step further, and feel that what s needed is to combine the two disciplines, so that individuals contribute in both ways. Rather than thinking of themselves as developers or sysadmins , these folks consider themselves devops . They work to become proficient in both roles, and to synthesize new ways of working by drawing on both types of skills and experience. A common crossover activity is the development of sophisticated tools for automating deployment, monitoring, capacity management and failure resolution. DevOps meets Cloud Like DevOps, cloud is not a specific technology or method, but a reorganization of the model (as I ve written previously). It s about breaking down the problem in a different way, splitting and merging its parts, and creating a new representation which doesn t correspond piece-for-piece to the old one. DevOps drives cloud because it offers a richer toolkit for the way they work: fast, flexible, efficient. Tools like Amazon EC2 and Google App Engine solve the right sorts of problems. Cloud also drives DevOps because it calls into question the traditional way of organizing software teams. A development/operations division just doesn t fit cloud as well as a DevOps model. Deployment is a classic duty of system administrators. In many organizations, only the IT department can implement changes in the production environment. Reaping the benefits of an IaaS environment requires deploying through an API, and therefore deployment requires development. While it is already common practice for system administrators to develop tools for automating deployment, and tools like Puppet and Chef are gaining momentum, IaaS makes this a necessity, and raises the bar in terms of sophistication. Doing this well requires skills and knowledge from both sides of the fence between development and operations, and can accelerate development as well as promote stability in production.
This is exemplified by infrastructure service providers like Amazon Web Services, where customers pay by the hour for black box access to computing resources. How those resources are provisioned and maintained is entirely Amazon s problem, while its customers must decide how to deploy and manage their applications within Amazon s IaaS framework. In this scenario, some operations work has been explicitly outsourced to Amazon, but IaaS is not a substitute for system administration. Deployment, monitoring, failure recovery, performance management, OS maintenance, system configuration, and more are still needed. A development team which is lacking the experience or capacity for this type of work cannot simply switch to an IaaS model and expect these needs to be taken care of by their service provider.
With platform service providers, the boundaries are different. Developers, if they build their application on the appropriate platform, can effectively outsource (mostly) the management of the entire production environment to their service provider. The operating system is abstracted away, and its maintenance can be someone else s problem. For applications which can be built with the available facilities, this will be a very attractive option for many organizations. The customers of these services may be traditional developers, who have no need for operations expertise. PaaS providers, though, will require deep expertise in both disciplines in order to build and improve their platform and services, and will likely benefit from a DevOps approach. Technical architecture draws on both development and operations expertise, because design goals like performance and robustness are affected by all layers of the stack, from hardware, power and cooling all the way up to application code. DevOps itself promotes greater collaboration on architecture, by involving experts in both disciplines, but cloud is a great catalyst because cloud architecture can be described in code. Rather than talking to each other about their respective parts of the system, they can work together on the whole system at once. Developers, sysadmins and hybrids can all contribute to a unified source tree, containing both application code and a description of the production environment: how many virtual servers to deploy, their specifications, which components run on which servers, how they are configured, and so on. In this way, system and network architecture can evolve in lockstep with application architecture. Cloudy promises such as dynamic scaling and fault tolerance call for a DevOps approach in order to be realized in a real-world scenario. These systems involve dynamically manipulating production infrastructure in response to changing conditions, and the application must adapt to these changes. Whether this takes the form of an active, intelligent response or a passive crash-only approach, development and operational considerations need to be aligned. So what? DevOps and cloud will continue to reinforce each other and gain momentum. Both individuals and organizations will need to adapt in order to take advantage of the opportunities provided by these new models. Because they re complementary, it makes sense to adopt them together, so those with expertise in both will be at an advantage.

Russell Coker: Is the PC Dying?

I just read an interesting article about the dispute between Microsoft and Apple about types of PC [1]. Steve Jobs predicted a switch from desktop PCs to portable devices, while Steve Ballmer of Microsoft claimed that the iPad is just a new PC. Defining a PC I think that the defining characteristic of the IBM Compatible PC was it s open architecture. Right from the start the PC could have it s hardware expanded by adding new circuit boards into slots on the motherboard (similar to other PC systems of that era such as the Apple 2 and the S-100 bus). The deal with IBM included Intel sharing all it s CPU designs with other manufacturers such as NEC and AMD from the 8086 until the mid-90 s. AMD specialised in chips that were close copies of Intel chips at low prices and higher clock rates while NEC added new instructions. Compaq started the PC clone market as well as the laptop market, and system software for the IBM compatible PCs was primarily available from IBM and Microsoft in the early days, along with less popular variants such as CP/M86, Novell Netware and others. In the late 80 s there was OS/2 as an alternate OS and Windows as one of several optional GUI environments to run on top of MS-DOS or PC-DOS. In the mid 90 s PCs were used for running protected mode OSs such as Linux and Windows/NT. Now if we look at a system such as a Netbook then it clearly misses some of the defining characteristics of the desktop PC. I can t upgrade a Netbook in any meaningful way changing a storage device or adding more RAM does not compare to adding an ISA/MCA/EISA/VL-Bus/PCI/PCIe expansion card. With my EeePC 701 I don t even have an option of replacing the storage as it is soldered to the motherboard! A laptop allows me to add a PCMCIA or PC-Card device to expand it, but with a maximum of two cards and a high price this isn t a great option. What is Best for Home Users? For a while now my parents have been using 3G net access for their home Internet use [2]. So it seems that a laptop provides greater benefits for their use now than it previously did when they used Cable and ADSL net access. My parents have been considering getting a new monitor (1920*1080 resolution monitors are getting insanely cheap nowadays) and driving such a monitor effectively might require a more capable PC. I recently bought myself a nice refurbished Thinkpad for $796 [3], it seems likely that I could find a refurbished Thinkpad at auction which is a little older and slower for a lower price, even buying an old T41p would be a reasonable option. This would give my parents not only the option of using the Internet when on holidays, but also in a different part of their house when they are at home. The Apple iPad would probably be quite a reasonable Internet platform for my parents if it wasn t for the fact that it uses DRM. While it s not a great platform for writing, my parents probably don t do enough that it would be a huge problem for them. So I might look for a less restrictive tablet platform for my parents. At the moment the best resolution for a tablet seems to be 1024*768, but I expect that some tablets (maybe with a hybrid tablet/laptop design like the Always Innovating Smartbook [4]) with a higher resolution will be released soon. I hope that the iPad and other closed devices don t get any serious market share, but it seems likely that OSs such as Android which are only slightly more open will have a significant market share. Ultra-Mobile Design vs PCs Design One significant problem with ultra-mobile devices is that they make significant engineering trade-offs to get the small size. For a desktop system there are lots of ways of doing things inefficiently, running the AMD64 or i386 architecture which is wasteful of energy and having lots of unused space inside the box in case you decide to upgrade it. But for a laptop there are few opportunities for being inefficient, and for a tablet or smart phone everything has to be optimised. When the optimisation of a device starts by choosing a CPU that s unlike most other systems (note that there is a significant range of ARM CPUs that are not fully compatible with each other) it makes it very difficult to produce free software to run it. I can salvage a desktop PC from a rubbish bin and run Linux on it (and I ve done that many times), but I wouldn t even bother trying to run Linux on an old mobile phone. It seems that in the near future my parents (and many other people with similar needs) will be best suited by having a limited device such as a tablet that stores all data on the Internet and not having anything that greatly resembles a PC. In many ways it would be easier for me to support my parents by storing their data in the cloud and then automatically backing it up to removable SATA disks than with my current situation of supporting a fully capable PC and backing it up to a USB device whenever I visit them. I m also considering what to do for some relatives who are about to go on a holiday in Europe, they want to be able to send email etc. It might not be possible just yet, but it seems like an ideal way of doing this would be to provide them with something like an iPad that they can use with a local 3G SIM for the country that they stay in and they could then upload all their best photos to some server that I can backup and send email to everyone they know. An iPad isn t good for this now as you don t want to go on holidays in another country while carrying something that is really desirable to thieves. Ultra Mobile Devices are Killing PCs It seems to me that Google Android and the Apple iPad/iPhone OS are taking over significant parts of the PC market. The people who are doing traditional PC things are increasingly using Laptops and Netbooks, and the number of people who get the freedom that a PC user did in the 80 s and 90 s is decreasing rapidly. I predict that by 2012 the majority of Linux systems will be running Google Android on hardware that doesn t easily allow upgrading to more open software. At the moment probably the majority of Linux systems are wireless routers and other embedded devices that people don t generally think about. But when iPad type devices running a locked-down Linux installation start replacing Ubuntu and Fedora desktop systems people will take notice. I don t think that the death of the PC platform as we know it will kill Linux, but it certainly won t do us any good. If there were smarter people at Microsoft then they would be trying to work with the Linux community on developing innovative new ways of using desktop PCs. Of all the attempts that Microsoft has made to leave the PC platform the only success has been the X-Box which is apparently doing well. Tablet devices such as the iPad could work really well in a corporate environment (where MS makes most of it s money). On many occasions I ve been in a meeting and we had to adjourn due to someone needing to go to their desk to look something up. If everyone had an iPad type device at their desk that used a wired network when it was available and encrypted wireless otherwise then for a meeting everyone could take their tablet without it s keyboard and be able to consult all the usual sources of data without any interruption. Could a high-resolution version of the iPad kill MS-Windows in the corporate environment?

7 June 2010

John Goerzen: Boys

Yesterday afternoon, it was nice outside. Terah, Jacob, Oliver, and I went outside. I had my camera to take photos while Jacob and Oliver mostly played. We did try to pose them a bit, but the overall plan was to just take some photos of them doing their own thing. So I wound up taking 824 photos. Hurray for digital cameras. We narrowed that down to a few of our favorites, which you can view as a slideshow. I ll highlight few of those below. IMG_3926.JPG
Jacob walking down the drive way, gazing at all the beans coming up off in the distance IMG_3910.JPG
Hmm, Oliver, let me see too IMG_3823.JPG
Running is fun! IMG_3818.JPG
Peek-a-boo IMG_3802.JPG
Goerzen s Law of Child Photography: If you attempt to get them to pose, a minimum of one boy will have a weird expression at any given time. IMG_3763.JPG
Also, chances are good that a cat will sneak in and also do something. IMG_3710.JPG
My imagined conversation for this one: Here Oliver, this is how you use a piece of bark. Quiet, I m trying to eat this grass. IMG_3718.JPG
Oliver decides that grass stuck on his teeth isn t fun IMG_3684.JPG
No explanation for this one. IMG_3644.JPG
Classic brothers. Jacob is using a piece of bark to mess around in some dirt, while Oliver quietly learns how to do it. He never said it, but I can easily imagine him saying, Here Oliver, here s what you do with bark: you dig up dirt like this! It s funny! IMG_3619.JPG
BONZAI!!! IMG_3580.JPG
Guess I better watch where I point that camera IMG_3477.JPG
Here kitty get ready for some more petting *thwack* Strangely the cat seemed to love it. Our other cat avoids Oliver. IMG_3404.JPG
Jacob deriving a great deal of joy from telling us that his shirt says big SPIDER . He came up with that joke himself and thought it was hilarious. IMG_3366.JPG
This is one of a few photos of Jacob wearing a suit that I once wore. IMG_3292.JPG
It s hard to stand still when the hammock has a great train whistle on it. IMG_3241.JPG
I have no idea what he s planning here. But really, I think I should be afraid. IMG_3191.JPG
It s a fridge door. Yay?

Chris Lamb: Distributing locking with Python and Redis

This is already possible with Memcache, but if you are already drinking the Koolaid...
import time
class Lock(object):
    def __init__(self, key, expires=60, timeout=10):
        """
        Distributed locking using Redis SETNX and GETSET.
        Usage::
            with Lock('my_lock'):
                print "Critical section"
        :param  expires     We consider any existing lock older than
                             expires  seconds to be invalid in order to
                            detect crashed clients. This value must be higher
                            than it takes the critical section to execute.
        :param  timeout     If another client has already obtained the lock,
                            sleep for a maximum of  timeout  seconds before
                            giving up. A value of 0 means we never wait.
        """
        self.key = key
        self.timeout = timeout
        self.expires = expires
    def __enter__(self):
        timeout = self.timeout
        while timeout >= 0:
            expires = time.time() + self.expires + 1
            if redis.setnx(self.key, expires):
                # We gained the lock; enter critical section
                return
            current_value = redis.get(self.key)
            # We found an expired lock and nobody raced us to replacing it
            if current_value and float(current_value) < time.time() and \
                redis.getset(self.key, expires) == current_value:
                    return
            timeout -= 1
            time.sleep(1)
        raise LockTimeout("Timeout whilst waiting for lock")
    def __exit__(self, exc_type, exc_value, traceback):
        redis.delete(self.key)
class LockTimeout(BaseException):
    pass
One common use case for distributed locks in web applications is to prevent clients dog-piling onto an expensive cache key:
def cache_without_dogpiling(key, cb, cache_expiry=None, *args, **kwargs):
    val = cache.get(key)
    if val is not None:
        return val
    # Cache miss; gain the lock to prevent multiple clients calling cb()
    with Lock(key, *args, **kwargs):
        # Check cache again - another client may have set the cache
        val = cache.get(key)
        if val is None:
            val = cb()
            cache.set(key, val, cache_expiry)
        return val
def slow():
    print "Inside slow()"
    return 1 + 1 # Python is slow
>>> cache_without_dogpiling('my_key', slow, 60 * 10)
Inside slow()
2
>>> cache_without_dogpiling('my_key', slow, 60 * 10)
2
As a bonus, if you don't want your test or development environment to rely on Redis, you can replace it with a no-op lock:
import contextlib
@contextlib.contextmanager
def Lock(*args, **kwargs):
    yield

6 June 2010

Gunnar Wolf: Spot the differences?

Spot the differences?
Left: British Petroleum's logo. British Petroleum gained international notoriety last April because of the catastrophic oil spill it is still unable to contain in the North of the Gulf of Mexico; so far, the oil has reached the coasts of Louisiana and Mississippi, and keeps growing and spreading, as probably the worst-ever oil spill. So far, the sea surface covered by oil is larger than several countries in the world. Some sites have very interesting maps that might help understand the importance and size of the spill: BP Gulf Oil Spill Maps in The Daily Green, Deepwater Horizon Incident, Office of Response and Restoration, USA National Ocean Service, Article on Slashdot about quantifying and dealing with the deepwater spill. Right: Mexico's de facto government's logo. One of the most pushed projects of Felipe Calder n's government (that was fortunately canceled) was to pursue the tesorito de las aguas profundas (little treasure that lies deep in the water); they strongly pushed for a reform in the oil legislation, which is 100% government-funded since 1938, to allow for private investment in orded to build platforms reaching oil deposits 3000m below the Gulf's surface. Yup, precisely like the one that produced this massive spill, although most would probably be bound to much less strict regulation and controls to what they have in the US. Is the similarity between the two logos just a strange coincidence?

Mart&iacute;n Ferrari: Loi 1901

After one year in France, there's still something I cannot completely understand; I don't know if it is something common in Europe, or a artifact of the French law, or if it is just that French people like to do things so formally all the time... I was taking a look at the website of the local LUG, and was amazed that they have association loi de 1901 status, an administration council with 11 members, statute, and written bylaws. Also, to be a part of the association, you need to send a paper form and at least 10. According to the Wikipedia page , an association loi de 1901 is just a non-profit organisation, and it has that status even if it is not registered. But it seems that is not mandatory to have any of the hierarchical structure, written rules and the such! So it seems that any group of more than 10 that gets together to do something go through all this trouble when it is not needed, there should be some explanation for all this... Tags: Planet Debian, Planet Lugfi, Francia

4 June 2010

Marco T&uacute;lio Gontijo e Silva: Summer of Code weekly report #4

My project. I m publishing my report earlier this week, because there was a lot to talk about. This week I started to make changes in the code going in the direction of what I want to do. I haven t started a final implementation, but I m studying about how what I want to do will affect the rest of the Garbage Collection (GC). I noticed the code in rts/sm/Sweep.c was simple and similar to what I m planning to do, so I started changing how it works. Sweep in the Glasgow Haskell Compiler (GHC) is done by a bitmap, which contains a bit for each word in a memory block, and is set to 1 when there s an object starting in the mapped area and 0 otherwise. When there s a block with no objects starting at it, that is, all bits of the bitmap are set to 0, the block is freed.
        if (resid == 0)
         
            freed++;
            gen->n_old_blocks--;
            if (prev == NULL)  
                gen->old_blocks = next;
              else  
                prev->link = next;
             
            freeGroup(bd);
         
The bits are analyzed in a group of BITS_IN(W_), where BITS_IN(W_) is the number of bits in a word.
        for (i = 0; i < BLOCK_SIZE_W / BITS_IN(W_); i++)
         
            if (bd->u.bitmap[i] != 0) resid++;
         
If more than of the groups are completely set to 0, the block is considered fragmented.
            if (resid < (BLOCK_SIZE_W * 3) / (BITS_IN(W_) * 4))  
                fragd++;
                bd->flags  = BF_FRAGMENTED;
             
Immix, the GC algorithm I plan to implement in GHC, divides the blocks of memory in lines. My initial plan was to identify free lines. I decided to consider a the size of a line fixed in BITS_IN(W_) words, because it will map to a word in the bitmap, and the code was already analyzing in groups of BITS_IN(W_) words. This was very easy with the current code.
            if (bd->u.bitmap[i] != 0) resid++;
            else printf("DEBUG: line_found(%p)\n", bd->start + BITS_IN(W_) * i);
This worked, and showed some free lines. I m sure there are other ways of logging in GHC, but printf was the simplest way I could thought of. I measured the occurrence of free lines using the bernouilli program from the NoFib benchmark suite, calling it with 500 +RTS -w, to make it uses sweep. In 782 calls to GarbageCollect(), sweep() was called 171 times, for 41704 blocks to be swept and found 230461 free lines. This gives us about 5.5 free lines per block, from the 8 lines in each block (on 64 bits systems). The problem is that the bitmap is marked only in the start of the objects allocated, so even in a line that all bits are marked with 0 we can t assume that it s completely free, because there may be an object that starts in the previous line that is using the space of the line. Checking only for the previous line doesn t work either, because a big object can span several lines. What we can do here is a variation of conservative marking, as proposed in the Immix paper, checking only the previous line and working only with objects smaller than a line. To make sure I was working only with objects smaller than a line, I had to mark the blocks that contains medium objects and avoid them when seeking free lines. The block flags are defined in includes/rts/storage/Block.h, so I included another flag in this file, BF_MEDIUM.
/* Block contains objects evacuated during this GC */
#define BF_EVACUATED 1
/* Block is a large object */
#define BF_LARGE     2
/* Block is pinned */
#define BF_PINNED    4
/* Block is to be marked, not copied */
#define BF_MARKED    8
/* Block is free, and on the free list  (TODO: is this used?) */
#define BF_FREE      16
/* Block is executable */
#define BF_EXEC	     32
/* Block contains only a small amount of live data */
#define BF_FRAGMENTED 64
/* we know about this block (for finding leaks) */
#define BF_KNOWN     128
/* Block contains objects larger than a line */
#define BF_MEDIUM    256
The GHC GC is generational, there is, objects are allocated in a generation and, after a time, the ones that are still being used are moved the next generation. This idea assumes the death probability of younger objects is higher, so few objects are moved to the next generation. Sweep and Immix work only in the last generation so, to mark blocks with medium objects we have to check the size of the objects that are moved to the next generation. This is done in the copy_tag function of rts/sm/Evac.c. I inserted a code that checks for the object size and marks the block when it s bigger than BITS_IN(W_).
STATIC_INLINE GNUC_ATTR_HOT void
copy_tag(StgClosure **p, const StgInfoTable *info,
         StgClosure *src, nat size, generation *gen, StgWord tag)
 
    StgPtr to, from;
    nat i;
    to = alloc_for_copy(size,gen);
    if(size > 8)  
        Bdescr(to)->flags  = BF_MEDIUM;
     
So I updated the code in rts/sm/Sweep.c to only inspect for free lines in blocks without BF_MEDIUM mark.
            if (bd->u.bitmap[i] != 0) resid++;
            else if(!(bd->flags & BF_MEDIUM))  
                printf("DEBUG: line_found(%p)\n", bd->start + BITS_IN(W_) * i);
             
This also worked. Now, in the 32012 blocks there were 189015 free lines, found in the same number of GCs, making about 5.9 free lines per block. We considered only blocks with small objects, but we didn t ignore the first line of each group of free lines. This can be achieved by checking if the previous line was also free.
            if (bd->u.bitmap[i] != 0) resid++;
            else if(!(bd->flags & BF_MEDIUM) && i > 0 && bd->u.bitmap[i] == 0)  
                printf("DEBUG: line_found(%p)\n", bd->start + BITS_IN(W_) * i);
             
Now, from the 32239 blocks, 165547 free lines were found, giving 5.1 free lines per block. But there are more things to improve. If the whole block is free, we want to free it, instead of marking it s lines as free. So it s better to mark the lines after we know that the blocks are not completely free. So I left the code that checks the bitmap as it was, and included a line check only for blocks that are not completely free. At this point, I also associated the fragmentation test with blocks with medium objects, because in blocks of small objects we plan to allocate in free lines, so fragmentation is not a (big) issue.
            if (resid < (BLOCK_SIZE_W * 3) / (BITS_IN(W_) * 4) &&
                (bd->flags & BF_MEDIUM))  
                fragd++;
                printf("DEBUG: BF_FRAGMENTED\n");
                bd->flags  = BF_FRAGMENTED;
             
            else if(!(bd->flags & BF_MEDIUM))  
                for(i = 1; i < BLOCK_SIZE_W / BITS_IN(W_); i++)
                 
                    if(bd->u.bitmap[i] == 0 && bd->u.bitmap[i - 1] == 0)  
                        printf("DEBUG: line_found(%p)\n", bd->start + BITS_IN(W_) * i);
                     
                 
             
The total ammount of blocks increased dramatically: the blocks that become fragmented and were not called again in sweep made a huge difference. From the 345143 blocks, 1633268 free lines were found, or about 4.7 free lines per block. 9434 blocks were free, so, from the remaining blocks, we have about 4.9 free lines per block. Something we ll need then is a way to access these lines latter. The simplest way I thought to achieve it is constructing a list of lines, in each the first word of each free line is a pointer to the next free line, and the first word of the last free line is 0. It s useful to keep reporting the lines to stdout, so that we can then follow the list and check if we went to the same lines.
                    if(bd->u.bitmap[i] == 0 && bd->u.bitmap[i - 1] == 0)  
                        StgPtr start = bd->start + BITS_IN(W_) * i;
                        printf("DEBUG: line_found(%p)\n", start);
                        if(line_first == NULL)  
                            line_first = start;
                         
                        if(line_last != NULL)  
                            *line_last = (StgWord) start;
                         
                        line_last = start;
                        *line_last = 0;
                     
                 
             
         
     
    for(line_last = line_first; line_last; line_last = (StgPtr) *line_last)  
        fprintf(stderr, "DEBUG: line_found(%p)\n", line_last);
     
I printed the inclusion of the lines on the list to stdout, and the walking on the list in stderr, so that it d be easy to diff. There was no difference between the lists. There re another improvements that can be made, like using a list of groups of free lines, but I think it ll be better to think about this after studying how the allocation in the free lines will be done. That s where I m going to now.
There are some minor things I learned, and thought they worth blogging. The current GHC uses three strategies for collecting the last generation: copying, mark-compact and mark-sweep. Copying is the default until the memory reaches 30% of the maximum heap size; after that, mark-compact is used. Sweep can be chosen by a Real Time System (RTS) flag, -w. To use mark-compact always, the flag is -c. I ve been submitting small patches to the cvs-ghc mailling list, mostly about outdated comments. Most of them were accepted, except for one which contained a lot of commentary, and that indeed was not completely correct. I corrected it and resend to the list, but the message is waiting for approval because the message header matched a filter rule. I believe this is because I replied the message generated from darcs. There s a very useful ghc option, specially for testing the compiler, because in this case you need to rebuild the source, even when there re no changes in it. It s -fforce-recomp, and it makes only sense when used with --make.

3 June 2010

Mehdi Dogguy: UbuntuDiff updates

Since the recent announce of http://ubuntudiff.debian.net, some people started using it and asked for a few features. Lately, I ve been working on it and tried to implement the following: I ll still have some details to fix like putting the download patch link on the left, fix the show/hide thing which is also activated when you click on download patch . Then, I ll consider it feature-complete (almost) and won t touch it again. My next game will be to write some tools to analyse or detect new transitions if you want to play, let me know :) As always, please test this new beta version of UbuntuDiff (and enjoy it :p) and maybe, you may also report errors/bugs, if any :) [1] http://ubuntudiff.debian.net/beta/

Next.

Previous.