Search Results: "pm"

20 November 2007

Russell Coker: Perfect Code vs Quite Good Code

Some years ago I worked on a project where software reliability should have been a priority (managing data that was sometimes needed by the police, the fire brigade, and the ambulance service). Unfortunately the project had been tainted by a large consulting company that was a subsidiary of an accounting firm (I would never have expected accountants to know anything about programming and several large accounting firms have confirmed my expectations). I was hired to help port the code from OS/2 1.2 to NT 4.0. The accounting firm had established a standard practice of never calling free() because “you might call free() on memory that was still being used”. This was a terribly bad idea at the best of times and on a 16 bit OS with memory being allocated in 64K chunks the problems were quite obvious to everyone who had any programming experience. The most amusing example of this was a function that allocated some memory and returned a pointer which was being called as if it returned a boolean, one function had a few dozen lines of code similar to if(allocate_some_memory()). I created a second function which called the first, free’d any memory which had been allocated and then returned a boolean. Another serious problem with that project was the use of copy and paste coding. A section of code would perform a certain task and someone would need it elsewhere. Instead of making it a function and calling it from multiple places the code would be copied. Then one copy would be debugged or have new features added and the other copy wouldn’t. One classic example of this was a section of code that displayed an array of data points where each row would be in a colour that indicated it’s status. However setting a row to red would change the colour of all it’s columns, setting a row to blue would change all except the last, and changing it to green would change all but the second-last. The code in question had been copied and pasted to different sections with the colours hard-coded. Naturally I wrote a function to change the colour of a row and made it take the colour as a parameter, the program worked correctly and was smaller too. The next programmer who worked on that section of code would only need to make one change - instead of changing code in multiple places and maybe missing one. Another example of the copy/paste coding was comparing time-stamps. Naturally using libc or OS routines for managing time stamps didn’t occur to them so they had a structure with fields for the year, month, day, hours, minutes, and seconds that was different from every other such structure that is in common use and had to write their own code to compare them, for further excitement some comparisons were only on date and some were on date and time. Many of these date comparisons were buggy and often there were two date comparisons in the same function which had different bugs. I created functions for comparing dates and the code suddenly became a lot easier to read, less buggy, and smaller. I have just read an interesting post by Theodore Ts’o on whether perfect code exists [1]. While I understand both Theodore’s and Bryan’s points of view in this discussion I think that a more relevant issue for most programmers is how to create islands of reasonably good code in the swamp that is a typical software development project. While it was impossible for any one person to turn around a badly broken software development project such as the one I describe, it is often possible to make some foundation code work well which gives other programmers a place to start when improving the code quality. Having the worst of the memory leaks fixed meant that memory use could be analysed to find other bugs and having good functions for comparing dates made the code more readable and thus programmers could understand what they were looking at. I don’t claim that my code was perfect, even given the limitations of the data structures that I was using there was certainly scope for improvement. But my code was solid, clean, commented, and accepted by all members of the team (so they would continue writing code in the same way). It might even have resulted in saving someone’s life as any system which provides data to the emergency services can potentially kill people if it malfunctions. Projects based on free software tend not to be as badly run, but there are still some nasty over-grown systems based on free software where no-one seems able to debug them. I believe that the plan of starting with some library code and making it reasonably good (great code may be impossible for many reasons) and then trying to expand the sections of good code is a reasonable approach to many broken systems. Of course the ideal situation would be to re-write such broken systems from scratch, but as that is often impossible rewriting a section at a time often gives reasonable results.

19 November 2007

Meike Reichle: Symptoms

The German language has an expression that is "Zeichen der Zeit". Literally translated this means "Signs of the time", the idea behind that expression is that sometimes there are certains things or events that are symptopmatic for a current development or a general change of times. One of these "Zeichen der Zeit" can currently be observed very nicely in our neighbourhood. In the last half year three shops in our direct vicinity closed down. One was a quite cute 60s cafe that was later rented out to a citizens' action committee, the second was a shop that sold wooden toys, the third was a health food shop. I don't know how long these shops where around, but all of them had lasted for at least a couple years if not longer. Now all three shops are rented out again, two are now temp work agencies, the third is a "consumer bank" that specialises on small consumer credits for things like cars, washing machines or entertainment electronics. Apparently they also cooperate with some mail order firms. Is this really where we want to go?

Benjamin Mako Hill: Affero General Public License Version 3

The Free Software Foundation sent out a press release today announcing a new addition to the FSF stable of licenses: the Affero General Public License or AGPL. The FSF has also published a set of answers to anticipated questions in the GPL FAQ. The first paragraph of the release explains what the AGPL is:
This is a new license; it is based on version 3 of the GNU General Public License (GNU GPLv3), but has an additional term to ensure that users who interact with the licensed software over a network can receive the source for that program. By publishing this license, the FSF aims to begin fostering user and development communities around free software web services and other network-oriented software.
The GPL is designed to ensure that users of software have access to the source code -- source is prerequisite to freedom and to the type of collaboration that has made free software successful. However, the GPL doesn't say "users" when it talks about who gets freedom; instead, it references people to whom the software is distributed. It doesn't say users for two reasons. The first is that, under copyright, "distribution" is a much more meaningful term and a powerful hook than "use" which is not, in most cases, one of the copyright holder's exclusive rights. The second is that, until very recently, having a copy of software was prerequisite to using it; possession was prerequisite to use. Things have changed. A large part of many people's computing experience involves running web applications. These include email clients (e.g., GMail or other webmails), office applications (e.g., Google Docs), social network systems, and others. These applications all run on servers -- i.e., on other people's computers. The providers of these services, the Googles and the FaceBooks, build upon, modify and improve GPL software without giving back to their users or the community that they took their software from. The AGPL was created several years ago by FSF board member Henri Poole as a way to address this issue. The license took the form of the GPLv2 with one extra clause. It was a first stab at a license and was imperfect. The language and methods were clunky and, most problematically, the license was incompatible with software under the GPL. The new AGPL is based on the GPLv3 and the extra clause has been rethought and rewritten. It has been vetted using the GPLv3 comment process and dozens of insightful comments from dozens of lawyers, hackers, and users of free software have been incorporated. The new license fixes the issues that many folks -- including myself -- had with the first version of the license. More importantly it can now be linked to GPLv3 code which makes the license a whole lot more practical. I am quoted in the release being excited about the license and I really am. I've got 2-3 major development projects (including Selectricity) which I've been waiting to distribute so that I could do so under the AGPLv3. The AGPL isn't a complete answer to the problem faced by disempowered users of web services. Without data or the capacity (in terms of servers, money, and expertise) to run web applications, the state and quality of these users' freedom remains far from clear. Thankfully, there are a whole bunch of folks thinking about what freedom for users of services might be -- it's a conversation that I'm going to push the FSF to participate in and pursue moving forward. The AGPLv3 marks a first solid contribution to the process of answering that question. If you'd like to help supporting or assisting the FSF in this effort, please consider becoming an associate member or donating.

Evan Prodromou: 28 Brumaire CCXVI

I'm glad to note that there's been a lot of movement over the last few weeks for the Montreal Coworking space. Patrick Tanguay, who's been organizing the project, let me know recently that we're closing in on a space. Which is just fantastic. Coworking is a loose term for a number of different ways of working together. The term is most commonly used for a kind of shared office where participants pay for either a fixed space (their own desk) by the month, or they pay for the use of a pool of free spaces (shared desks) on a daily or part-time basis. There are the kind of office resources, like WiFi, conference rooms, printers, copiers, etc. that you expect from an office, but unlike most shared-office situations, there's also an emphasis on community and communications. It's a great kind of work environment for people who would otherwise work at home or in caf s, such as freelancers, contractors, and startuppers -- or for people visiting from out-of-town. I've been to a couple of the Bay Area (California) coworking spaces -- the one at Citizen Space in San Francisco and the one at SocialText in Palo Alto. Both were convenient, fun, and social -- a good place to get some work done. Patrick's been working on a Montreal coworking space for a while -- talking to the City and neighbourhood development organizations -- and he's settled on a space in Mile End. It looks pretty fantastic. You can see the pictures and a map at the Montreal Coworking Web site. I've been working out of the house for about 5 years, and I really like the freedom and convenience it affords. That said, I'm really looking forward to having a more formal environment to work in than my home office. It's great here, and I feel very comfortable, but it would be nice to be out of the house. I also tend to meet with contractors and partners at local caf s, which can be distracting -- I'm looking forward to having meeting rooms. There's a 5 7 ("sank ah set" -- it's what we call a cocktail party here in Montreal) planned for people who'll be involved in the site planned for this Wednesday, November 21, at Bar Inc. (250 Mont-Royal Est). I won't be there -- we'll be in New Jersey by then -- but I hope others interested in the coworking take the chance to go. tags:

JS-Kit and data portability I've been a big fan of JS-Kit since I heard about it. I featured it in my list of 10 Web APIs for LinuxWorld, and I use it on my own personal site for comments. But one thing that concerns me, and that I've asked for clarification from JS-Kit about, is data portability. If they went out of business (heaven forfend), or if I figured out another cool way to do comments on my site, I'd like to be able to migrate old comments to the new system I'd build/use/buy. But as far as I can tell, there's no interface to download all comments on a site (either in HTML or some other structured format). I think this is going to be a key issue for people building Web sites with UI widgets in the future. It's a really, really useful way to make sites, but I don't think many developers are going to be willing to invest in a platform with no "insurance" that they can be flexible in the future. Sure, it's free of charge, and it seems kind of mean to use the poor JS-Kit folks and then move on to someone else -- but realistically it's an option you need for development. tags:

One-liners Some other stuff I'm thinking about today:
  • On the "upcoming events" front, we've got some plans to make BarCampCanada1 happen here in Montreal in May 2008. I'm not sure how that's going to shake out, but I'm excited about the possibility of a nationwide BarCamp.
  • I'm interested in Heather Ford's and Jimmy Wales's 50 parties idea - having fifty parties in one year to bring together local Free Culture people. I think we should have a party here in Montreal. Maybe in the Spring...? Maybe to coincide with BarCampCanada1?
tags:

Russell Coker: WTF - Let s write all the code twice

There is an interesting web site WorseThanFailure.com (with the slogan “Curious Perversions in Information Technology”) that documents amusingly failed projects. The name used to be TheDailyWTF.com but changed due to the idea that for some projects success (interpreted to mean limping along in production) is worse than failure (being scrapped and re-written). I’ve created a new category WTF [1] on my blog to document such projects, both ones that I have personally witnessed and ones that friends and colleagues have seen. In the 90’s I spent some time working on an OS/2 GUI program to be the front-end for a mainframe backed system used by call-center workers. The first thing that they did was to develop a file naming scheme, they decided that all source files should be in the same directory (not unreasonable), but that 8.3 should be the limit of file name lengths in case there was a need to port the system to Windows 95. The fact that porting a program which did little other than display a GUI and talk to a S/390 server to a different OS was difficult (given the large amount of platform specific GUI code) was lost on them. Then they decided that the first two letters would be “FE” in case the source was merged with another bank project (of course the bank had many “Front End” systems talking to mainframes - so even this didn’t guarantee that they would be free of name clashes). Characters 3 and 4 were to represent the section of the program, and as the 3 character extension was one of “.h” or “.cpp” that left exactly 4 characters to name the file - we may as well have numbered the source files. Eventually one of the guys started making jokes about the file names by trying to pronounce them as words and that convinced a majority of the developers that the files should have long human-readable names. As the NMAKE expert (they insisted on not using unsupported software such as GNU make) it was my job to fix all the file names. The bank had an internal users’ group for the C++ development environment, but the contractors were not invited to attend. You might think that it would make sense to have the most skillful C++ programmers attend the meetings and share their knowledge, but apparently attending such meetings was a perk of being a permanent employee. What do you do when you have a C++ development project running behind schedule and a permanent employee who has had no prior exposure to C++ and wants to learn? One manager believed that assigning the junior programmer to the project is a solution to both problems - it was good for teaching the programmer C++ in a hurry but not so good for the deadline. There was a lot of excitement related to the back-end development. The mainframe guys assured me that CICS could never handle bad data and it totally wasn’t their fault if my program sent bad data to them and the region crashed. The first CICS region crash occurred when the mainframe guys told me to make the unused space in text fields “empty” - to a C programmer this means filling them with the 0 character but it killed the CICS region (apparently “empty” means filled with spaces to CICS programmers). A later region crash came when they told me to change the field length for the account name everywhere that it occurred - they told me that they wanted the change immediately - so I changed a macro definition, ran “make”, and sent a transaction with the changed size. Apparently “immediately” really meant “after we have spent an hour of software development changing magic numbers throughout our system”. I’m assuming that COBOL has some facility that roughly compares to C macros for defining field lengths and that it was the programmers not the language that was at fault. When the project became seriously behind schedule they hired some expert programmers from another country to help with the development. The experts decided that we needed to redesign the system (not that it had been designed in the first place) to use a “model view controller” architecture. When the real business logic is all on a S/390 and the only logic in the front end is to combine the contents of some screen-scrapes from a virtual 3270 terminal into a single GUI dialogue (with the first screen occasionally getting data from the IVR system) the benefits of MVC seem rather small. The new MVC version of the system was dubbed “phase 2″ and there were a series of deadlines for changing over from “phase 1″ which were missed. So the original team of developers continued on their project without any help from the “experts”. The last report I heard (some time after leaving the project) was that Phase 1 had gone live in production while Phase 2 was still being developed (at that time Phase 2 had a year of development and Phase 1 had about 15 months). The lesson to be learned from this is that management should only hire people that they can control. If you hire a consulting company and then let them do whatever they want (as opposed to solving the problem that they were hired to solve) then at best they will do no good and they may just slow things down. One of the more memorable incidents was when the project manager was talking to his 2IC, he said “I think we’re really breaking ground, [pause] I’m breaking wind and we’re breaking ground”. My desk was a few meters from where the wind (and ground) were being broken, I unfortunately was able to verify that wind was being broken but was not as certain about the ground. I did learn some useful things during this project, one was the fact that sample code from library vendors should not be trusted and ideally should be read from a print-out not pasted into the project source tree. The CTI library vendor had buffer overflows in their sample code. Another useful thing was the coin-tossing method of dispute resolution. If two good programmers have a disagreement about how to implement something tossing a coin is a quick way of resolving the dispute - this only works with two good programmers (in which case either idea will work). The project in question was messed up in many ways. I may write another post in the WTF category about it.

Adam Rosi-Kessel: Smooth Velouria Espresso

I just discovered the recently-opened Velouria Espresso in Jamaica Plain via this subscribers-only article in the Atlantic. Velouria marks the latest arrival in the single-origin coffee movement pioneered by George Howell. In a nutshell, the idea of single-origin coffee beans is that if beans are identified with a particular region (or even particular grower), the farmers will have more incentive to distinguish themselves. Ultimately, coffee beans will be known like fine wines, and the best growers will earn premium prices (thus bringing even more money to underdeveloped areas than coffee sold as fair-trade). Flatblack Coffee Company, near my office, operates under similar principles. More details in this episode of On Point with George Howell. Velouria distinguishes itself with its focus on brewed coffee rather than espresso (although they have plenty of that as well). The shop features the “Clover” coffee machine (”Bring out the subtle nuances of all of your coffees through complete, independent control of all of the important brew parameters: grind size, dose, water temperature, and contact time.”) I got a cappucino and a single-origin Kenyan coffee made with the Clover Machine. (Actually, two distinct single-origin Kenyan coffees, which Steve surreptitiously swapped on me while I wasn’t looking). They were all excellent. My cappucino started with a little foam heart in the crema:
P1040782
Amazingly, even when it was finished, the heart was still faintly visible:
P1040784
Steve enjoyed his espresso as well:
P1040783
Velouria has no website that I could find. it’s at 389 Centre Street, Jamaica Plain, MA 02131. Beware the Yelp page as it has inaccurate hours information — just call them at (617) 522-2400 if you’re wondering if they’re open. Now if we could only get a place like this in one of the vacant storefronts in Roslindale… Technorati Tags: , , , , ,

Andrew Pollock: [life/americania] Two years in the US

So as it turns out, two years ago today, was when I arrived in the US with the intent of staying here for a bit. Aside from recent developments, it's been a good couple of years. We're both pretty settled, have made plenty of wonderful friends, and seen a bit (not as much as we'd like) of the country. Sarah would have liked to have had a better time with employment, but it's not like she's been sitting around at home climbing the walls in her unemployment, either. We keep talking about "when we move back", but I'm personally in no huge rush. I also don't have any strong desire to do anything relating to putting down roots over here. Buying a place just feels extremely unfeasible, and we can't really stay forever on the visa type I'm on, and I can't really see myself ever qualifying for a H-1B.

18 November 2007

Evan Prodromou: 27 Brumaire CCXVI

I've been getting really interested in the new crop of Linux-based operating systems for mobile devices. I got a Nokia N800 earlier this year, which I've found really useful -- although a little frustrating, too. And I've been following the developments on OpenMoKo with great interest, too. So last Monday, when Google Android came out, I offered to give an experienced developer's opinion of the development kit for Network World. I spent most of the afternoon trying out the SDK, checking out the documentation, watching the presentation videos, and working on sample applications to get a feel for it. I'm pretty impressed with Android, and I think it's got a good shot to become the Open Source mobile application platform. I think that whoever wins in this space has the chance to shape how all mobile phone platforms work in the future, and to take a good chunk of the market. Opening up the phone platform to allow a real marketplace of third-party apps is, I believe, going to be a huge advantage for whoever can make it work. I think the main two contenders in the area are well ahead of Android so far, but they're really going to have to capitalize on that lead in the next 8-10 months before the first Android handsets are set to hit the market (late 2008 -- although we'll see about that schedule). And I'm not sure they've got the means and commitment to do it. Nokia has been screwing around with the N series of Internet tablets for a while. Which is great -- the operating system is clever, the GUI ("Hildon") GTK-based, and development in Python on the platform is a breeze. The basic tools -- media player, RSS reader (ingenious), Web browser, email, etc. -- work well. However, for unexplained reasons, Nokia hasn't put a damn phone into the thing. If there is a similar device that does everything that the N800 (or N810) does, plus has a phone in it, which one are people going to buy? It seems obvious to me. The other interesting OS is OpenMoKo, which I think is a great project with exactly the right goals. However, there's been some disappointment in the enthusiastic community around the OS as schedules have slipped and the developer preview devices shipped without a functioning dialer app. These two projects will have to fight uphill against the fear, uncertainty and doubt generated by the Android announcement and SDK release, even to stay in the game. And, to be fair, Google's announcement was masterful: classic vaporware -- for a product that won't be shipping for at least a year. But they got a pretty solid list of partners for the platform to sign on (albeit with only vague requirements for support). Best of all, they released a one-two punch of a bunch of developer information videos with tantalizing views of a nice-looking potential device, and a good SDK with tons of documentation and developer tools. It even includes an emulator that looks like a little phone. Awww! Android is slick, it's deep, and it's well-designed. The architecture is smart and sophisticated without being too clever for its own good. It's built on conservative -- maybe a little too conservative -- twin pillars: Java and Linux. For hardware providers, there's Linux's tried-and-true driver framework. For applications developers, way up at the top of the stack, there's good ol' boring Java, with new and smart APIs targeted specifically for mobile device apps, all on top of a new virtual machine called Dalvik that (supposedly) optimizes the hell out of Java bytecode. I was really impressed that the emulator emulates much more than just the Java APIs -- you can build ARM binaries in C that run on the thing. Very nice. Am I still going to get an OpenMoKo device? Maybe. But I'm going to continue playing around with the Android SDK, and looking for early developer versions of Android-supporting devices? Oh, yeah. Oh, and the article I did the research for is here; strangely it ended up on ComputerWorld rather than NetworkWorld. OK for me, though. tags:

More about me and my cool life So, I talked yesterday (Journal/27 Brumaire CCXVI) about how much weight I've lost in the last few months, which has been great. Another thing that's been going on for me is that I've started using the Getting Things Done process for scheduling my tasks and my time. I've never been particularly good at managing my time, but at least I've always known that however late or irresponsible I might be, my brother Ted would always be later. But when I visited San Francisco in August, Ted was really doing great. Friends and family kept telling me how responsible and down-to-earth Ted seemed lately. And whenever he had an idea for something to do, he wrote it down in his little Sidekick device. Ted showed me how the GTD system worked for him, and so I went out and bought the book and started reading it. I have been doing GTD for a few months now, and I've found it really satisfying. I don't worry as much any more about little things that I should be doing -- they're all captured in my system. It's really gratifying. I don't know if I've been quite as successful as Ted in improving my personal productivity, but I do know that I'm feeling a lot better about things and about myself. Which is really all that matters. tags:

Skating away Maj, Amita June and I went ice-skating today. It was AJ's first time out on the ice, and she really enjoyed it (most of the time). We had a couple of spills, mostly because it was really my first time skating with a 2-year-old, and partly because it was her first time skating with anyone. We went to the indoor rink near our old house on rue Cartier at the corner of Marie-Anne. Pretty soon we'll be able to go outside on the pond at Parc Lafontaine. catching our breath Maj isn't really a big skating fan, but she seemed to have a good time this time around. Which made three of us. I think we've got the makings of a Sunday-afternoon family tradition. We'll see, at least. tags:

More one-liners
  • I'm glad to see that the Mister Wong logo contest I talked about a while ago (Journal/21 Thermidor CCXV) has borne fruit. The top 12 logos look pretty good.
  • Maj and I have been catching up on season 3 of Deadwood lately. Great show, nice to see that season three is as solid as previous ones.
  • We've got RecentChangesCamp coming up in San Francisco in February 2008 (I think). Anyone interested should get involved at http://aboutus.org/RCC2008 .
  • What's the best alternative to the Sitemaps protocol for people who are into RDF? I'm thinking you could just make a hugantic RSS 1.0 feed -- after all, it is the "RDF Site Summary" format. Best of all, most search engines consume RSS as a recommended feed format. I'm going to try it with Vinismo and let you all know. The downside: I'm not sure many processors will handle a 100,000-item RSS feed very well.
  • Norman Mailer died just as I got around to reading the copy of ''Oswald's Tale'' I stole from the Driskill Hotel last year at South by Southwest. I can't help feeling responsible.
tags:

Andreas Metzler: upload frenzy

I have busied myself with Debian a somehow this weekend. A little bit of snowshoe walking in fresh fallen snows really helped me to gather some energy.

16 November 2007

Mike Hommey: Laptop upgrade prices

I’m currently evaluating different laptops to decide what I’m going to buy to renew my current one. And while doing so, I came to the conclusion that you are often better off buying the cheapest model of a line, and upgrading it yourself, at least if you’re considering Mac or a Vaio. If you customize a Macbook on the french Apple web store, it costs you 140 to replace the default 2 x 512MiB SO-DIMM DDR2 PC5300 with 2 x 1GiB, or 810 for 2 x 2GiB. Such upgrade on other models have apparently the same price ranges. On the other hand, the most expensive 1GiB module I can find on materiel.net (a french web store selling computer parts, not necessarily the cheapest you can find) is priced 29.89 and branded Kingston. That’s a wooping 60 to upgrade to 2 x 1GiB (instead of 140), and you keep the original 2 x 512MiB ! And for a 2GiB module, it’s “only” 104.95 , which makes the whole 4GiB at roughly 210 instead of 600 more. And again, you keep the original 2 x 512 MiB. I don’t really know what Kingston memory modules are worth, but are the Apple ones made of gold ? As far as I know, they’re not even ECC, which could legitimate the difference. Now, on the same Macbook, it would cost you 140 to replace the standard 80GB SATA 5400rpm hard drive with a 160GB one and 280 for a 250GB disk. On the same materiel.net site, I see a Western Digital 160GB 5200rpm SATA drive at 102.99 , and a 250GB one for 152.49 . So again, it’s cheaper to buy the standard model and upgrade it yourself. And you keep the original disk as a bonus ! It works equally well on the american or the japanese Apple stores. And the same applies with Sony (though I couldn’t find how to customize a vaio laptop on the french site, I checked it was true on the japanese and american web stores). On the other hand, Dell and Lenovo seem to have much more reasonable upgrade prices. What the fsck ? Update: The more I look at it, the less the memory thing makes sense to me. If it happens to actualy be ECC, why don’t they advertise so, which is more important (and justifying the amazing pricing) than the memory being SO-DIMM DDR2 PC5300 (which they do advertise) ? Update 2: Looking for dmidecode output with google makes it clear that Apple and Sony laptops, at least the ones I checked price for, don’t use ECC memory.

Ondřej Čertík: SAGE Days 6

From November 9 till November 15 I attended SAGE Days 6 in Bristol, UK. It was a conference and a coding sprint for the SAGE project, which wants to build something comparable to Mathematica, Maple, Matlab and Magma built only from open source components without reinventing the wheel. Sage also provides excellent novel implementations for many mathematical algorithms (not found in Mathematica/Maple etc.).

I was invited to give a talk about SymPy, which is a Python library for symbolic mathematics, that we are doing.

Here are my notes from each day:

Saturday: one, two
Sunday: one, two
Monday, Tuesday, Wednesday: one

Impressions

Very positive. I started SymPy two years ago, because I wanted to play with symbolic mathematics in Python, see for example my presentation I gave at this conference for details. SAGE at that time was just able to do some mathematics things, but it was very weak in calculus, which is the SymPy's main domain. This has changed last half a year, when SAGE people managed to wrap Maxima in Python (which I thought to be completely impossible), so I started to follow SAGE development more closely. After the SD6, I must say I became very excited about the project.

One problem with mailinglists, IRC and other online interaction is that it's very difficult for me to get an impression about the people and the project. Being able to meet the developers and discuss with them face to face gives me the impression very quickly and very accurately.

By far the biggest guarantee, why it is worthy for me to contribute to SAGE, is the project leader, William Stein. He is very rational and pragmatic (I like these two properties) and after many discussions with him, I came to realize that he has basically identical views on the important things as I do and were I on his place, I would do the same decisions as he did and does. That's very nice, because I can concentrate my energy on things that I like to improve and don't have to worry about other things, because I know he will do it right.

The other SAGE developers are experts and with similar attitude as William has. It's enthusiastic to be among people who make things happen. For example one of the authors of Cython, Robert, implemented during SD6 a very nice HTML output, that shows Cython code, with colors according to how many Python API calls are called on that particular line, and by clicking that line it shows the corresponding C code.

SAGE project has a high aim and it stricly goes for it, without looking too much to the right or left, and that's how it should be. And it does produce a lot of very useful and high quality stuff along the way, for example Cython (probably the best wrapper for C/C++ things now, only pypy could possibly beat it, but that's still more a research project) or the SAGE notebook, which looks like a Mathematica notebook, but better and in a browser (together with a revision history, sharing, SSL encryption, etc.).

The only little problem is that currently SAGE developers are all mathematicians and as is well-known, mathematicians looks at mathematic from a very different prospective than physicists. :) And so I need calculus, advanced calculus and only when this is working, and working well, I can build on it some more advanced features. SAGE currently goes a little the other way - it has a lot of advanced features, from number theory, modular forms, elliptic curves, etc., but the basic calculus still needs a lot of improvements. SAGE wraps Maxima, because Maxima is quite fast, very well tested, so it works well. It's difficult to extend and written in LISP and that's very bad. That's where SymPy could help - it's in Python, very easy to extend, but currently slower than Maxima (rewriting parts of SymPy using Cython, or even C directly, will make it faster, hopefully as fast as Maxima or faster).

It's not yet in Debian, but SAGE people are working on it. It's not an easy task unfortunately.

Conclusion

SAGE is a very promising young project and I think it will succeed to provide an open source alternative to Maple, Matlab, Mathematica and Magma.

Russell Coker: Software vs Hardware RAID

Should you use software or hardware RAID? Many people claim that Hardware RAID is needed for performance (which can be true) but then claim that it’s because of the CPU use of the RAID calculations. Here is the data logged by the Linux kernel then the RAID-5 and RAID-6 drivers are loaded on a 1GHz Pentium-3 system: raid5: automatically using best checksumming function: pIII_sse
  pIII_sse  :  2044.000 MB/sec
raid5: using function: pIII_sse (2044.000 MB/sec)
raid6: int32×1    269 MB/s
raid6: int32×2    316 MB/s
raid6: int32×4    308 MB/s
raid6: int32×8    281 MB/s
raid6: mmxx1      914 MB/s
raid6: mmxx2    1000 MB/s
raid6: sse1×1    800 MB/s
raid6: sse1×2    1046 MB/s
raid6: using algorithm sse1×2 (1046 MB/s) There are few P3 systems that have enough IO capacity to support anywhere near 2000MB/s of disk IO and modern systems give even better CPU performance. The fastest disks available can sustain about 80MB/s when performing contiguous disk IO (which incidentally is a fairly rare operation). So if you had ten fast disks performing contiguous IO then you might be using 800MB/s of disk IO bandwidth, but that would hardly stretch your CPU performance. It’s obvious that CPU performance of the XOR calculations for RAID-5 (and the slightly more complex calculations for RAID-6) is not a bottleneck. Hardware RAID-5 often significantly outperforms software RAID-5 (in fact it should always outperform software RAID-5) even though in almost every case the RAID processor has significantly less CPU power than the main CPU. The benefit for hardware RAID-5 is in caching. A standard feature in a hardware RAID controller is a write-back disk cache in non-volatile RAM (RAM that has a battery backup and can typically keep it’s data for more than 24 hours without power). All RAID levels benefit from this but RAID-5 and RAID-6 gain particular benefits. In RAID-5 a small write (less than the stripe size) requires either that all the blocks other than the ones to be written are read or that the original content of the block to be written and the parity block are read - in either case writing less than a full stripe to a RAID-5 requires some reads. If the write-back cache can store the data for long enough that a second write is performed to the same stripe (EG to files being created in the same Inode block) then performance may double. There is one situation where software RAID will give better performance (often significantly better performance), that is for low-end hardware RAID devices. I suspect that some hardware RAID vendors deliberately cripple the performance of low-end RAID devices (by using an extremely under-powered CPU among other things) to drive sales of the more expensive devices. In 2001 I benchmarked one hardware RAID controller as being able to only sustain 10MB/s for contiguous read and write operations (software RAID on lesser hardware would deliver 100MB/s or more). But for random synchronous writes the performance was great and that’s what mattered for the application in question. Also there are reliability issues related to write-back caching. In a well designed system an update of an entire RAID-5 stripe (one block to each disk including the parity block) will first be performed to the cache and then the cache will be written back. If the power fails while the write is in progress then it will be attempted again when power is restored thus ensuring that all disks have the same data. With any RAID implementation without such a NVRAM cache a write to the entire stripe could be partially successful. This means that the parity block would not match the data! In such a situation the machine would probably work well (fsck would ensure that the filesystem state was consistent) until a disk failed. When the RAID-5 recovery procedure is used after a disk is failed it uses the parity block to re-generate the missing data, but if the parity doesn’t match then the re-generated data will be different. A disk failure may happen while the machine is online and this could potentially result in filesystem and/or database meta-data changing on a running system - this is a bad situation that most filesystems and databases will not handle well. A further benefit of a well designed NVRAM cache is that it can be used on multiple systems. For their servers HP makes some guarantees about which replacement machines will accept the NVRAM module. So if you have a HP server running RAID-5 with an NVRAM cache then you could have the entire motherboard die, have HP support provide a replacement server, then when the replacement machine is booted with the old hard drives and NVRAM module installed the data in the write-back cache will be written! This is a significant feature for improving reliability in bad corner cases. NB I’m not saying that HP is better than all other RAID vendors in this regard, merely that I know what HP equipment will do and don’t know about the rest. It would be good if there was a commodity standard for NVRAM on a PC motherboard. Perhaps a standard socket design that Intel could specify and that every motherboard manufacturer would eventually support. Then to implement such things on a typical PC all that would be required would be the NVRAM module, which while still being expensive would be significantly cheaper than current prices due to the increase in volume. If there was a significant quantity of PCs with such NVRAM (or which could be upgraded to it without excessive cost) then there would be an incentive for people to modify the Linux sotware RAID code to use it and thus give benefits for performance and reliability. Then it could be possible to install a NVRAM module and drives in a replacement server with Linux software RAID and have the data integrity preserved. But unless/until such things happen write-back caching that preserves the data integrity requires hardware RAID. Another limitation of Linux software RAID is expanding RAID groups. A HP server that I work on had two disks in a RAID-1 array, one of my colleagues added an extra disk and made it a RAID-5, the hardware RAID device moved the data around as appropriate while the machine was running and the disk space was expanded without any down-time. Some similar things can be done with Linux, for example here is documentation on converting RAID-1 to RAID-5 with Linux software RAID [1]. But that conversion operation requires some down-time and is not something that’s officially supported, while converting RAID-1 to RAID-5 with HP hardware RAID is a standard supported feature.

Michal Čihař: Lets automate the work

Fixing bugs in code and committing it to VCS usually also means that you need to interact somehow with BTS you use to let it know that the bug has been fixed. This is manual work which could be done by some clever commit scripts, right? All what is needed is just spend some time to set it up. Most of my development currently goes to Gammu/Wammu which uses own Mantis bug tracker. It is already a bit prepared for integration with VCS, but I never found their documentation sufficient. Fortunately somebody wrote tutorial on integrating Mantis and Subversion, which made it quite easy to set up. All what was needed was to put little commit hook into SVN. Now the logical step was to make it also for Debian packages. A bit of Googling revealed discussion on debian-devel. Unfortunately no example for SVN, but ripping out needed things from Manoj's script for Arch was quite easy and I made my own commit hook. While looking at this, I also found wonderful tool called debcommit, which commits changes to Debian package using changelog entries as commit message. I still wonder how many useful tools are hidden to me. The only thing I don't know is why Google didn't show me yesterday svnlog, which seems to have support for Debian BTS out of the box. Maybe I just entered slightly different keywords than today when trying to find the mentioned thread on debian-devel.

14 November 2007

Eddy Petrișor: Linux: plain weird network behaviour; Windows is OK

Update: problem fixed, thanks for the comments; it was MTU related issues.

Note: This is a long post, but I expect it to bring in some questions for many Linux people; I advise you to read this when you have enough time.


After the last events related to connectivity at home which have been lasting since Thursday evening and the weird fact that it seems that NAT doesn't work at home, but does at work for the same setup, I called the ISP's support and wasted 15 minutes trying to convince them at least to send a technical team with another modem just to test if that is the reason for the breakage.

Of course, talking to the ISP support and trying to convince them that there might be a problem on their side since NAT works with another provider but doesn't with them, was fruitless.

Tonight I was stuck and tried another approach, convinced I would confirm my suspicion that there is something wrong on ISP's side. Still I am not yet sure what to conclude from what happened.


So, to get a better view of what is going on, I'll describe the setup I have and what are its limitations and characteristics. People in a hurry can skip to the paragraph starting with "I tested" and stare in wonder at will.


So the connection I have is done through a DSL modem which gets the MAC of the network card connected to it and exposes that as its own to the ISP's network. This MAC seems to be quite persistent and special measures must be taken in order to be able to use another NIC to connect. The modem (or probably some machine in ISP's; the IP is 10.10.0.1) offers a DHCP address and everything should work fine.

Because of this "your MAC is my MAC" issue, when I connected the first time, I used an USB NIC since a broken router or some temporary failure would have allowed me to use the Internet connection directly from my laptop. I can say this decision has proven in time to be wise.

The router I used until now is a NSLU2 with Debian installed on it. The built in network interface always faced the internal network.

The router (which I call ritter) served as a NAT router for two machines inside the network, my laptop and my apartment mate's laptop. All until last Thursday, after which it never got back properly.



I tested (doing NAT on my laptop; the laptop shouldn't have been affected by the power problems):

Of course, I have n-checked that /proc/sys/net/ipv4/ip_forward was set to 1, the tables had policy ACCEPT and there were no extra rules, except the basic NAT-ting stuff, the routes were correctly set on both the clients and the machine doing the NAT.


All I could see is that:
  1. the machine doing the NAT was always working fine
  2. at work all NAT clients worked fine
  3. at home any of the NAT "clients" were
    1. able to resolve addresses while the DNS server was in ISP territory and another LAN
    2. ping the outside world (if ping was available - in D-I is not)
    3. hanging when trying to get a http page
    4. telnet-ting directly to the port 80 was ok (but I didn't try to "GET / HTTP/1.0")

So after all of this, I was thinking of trying to see if a new client (the laptop of my apartment mate, a Windows XP machine) would work using the connection at home. It didn't work.

Then I thought of trying to do "Internet Connection Sharing" as is called in Windows. Of course, there was some pain to find Windows XP drivers for the ASIX AX88172 network card (remember, the modem needed to see the MAC of that NIC), but I managed to find the proper one.


And, almost sure the NAT wouldn't work for this case either, I configured the new connection as a shared one. I didn't even disabled the firewall, as I was thinking I could take those down gradually.

I wasn't expecting this, not even by pure chance, but my laptop which was a NAT "client" now was able to browse, ping and do whatever was normal through NAT, while the Windows machine was doing the "Connection sharing".

I was utterly flabbergasted. And that was just the beginning.


I was expecting that the problem coincidentally went away, but after a minute I was proven otherwise. It still didn't work with Linux as the NAT-ting machine. I connected back the USB NIC to the Windows machine and I saw the same thing. NAT was just working.


At that point I was to observe an even more shocking fact: the IP that the Windows machine got was different from the one the Linux machine received, in spite of the fact that the network card was the same, so it would have made sense to get the same one. More than that, the IP that the Windows machine got was from an entirely different network, although it was a valid IP belonging to my ISP.

I was thinking that one reason why it works with Windows might be that there could be some TCP protocol twist that is differently implemented in Windows and the equipment from my ISP gets along better with the Windows network stack.


As a way to test that, I am thinking of forcing somehow the IP on the Linux machine to see if anything changes. But before doing that, I felt the urge to post these, maybe some kind soul will shed some light on this issue for me or drop a hint.

Another reason might be different DHCP servers answering, but I don't know how I can see in Windows who offered the lease.


If anyone has any clue why these weird things happen, please drop a line. I would greatly appreciate it. TIA.

Eddy Petrișor: Lesson relearned: when Linux networking weirdess occurs...

My relearned lesson for the day: when Linux networking weirdness occurs in a NAT environment, remember to try MTU clamping.

Thanks to the comments by Justin and Sesse, I was fast-tracked to the core of the problems I have been experiencing since Thursday, MTU issues. What's worse (from my pov) is that I have encountered this issue before with the provider I had in Timi oara, but, since that ISP was using PPPoE and my current ISP in Bucharest doesn't, I never really made the connection. I even had a commented out iptables rule for MTU clamping in my firewall script.

The rule I am talking about looks like this:

iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o $EXT_IF -j TCPMSS --clamp-mss-to-pmtu

or like the one I have been using (seems more logical to me):

iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu


Note that this is not a fix, but a workaround and the real problem is over-zealous admins or weird setups[1] which think that banning TCP fragmentation (or the entire ICMP traffic) is a way to secure networks.


Once again, thanks to everybody who read and/or commented about my issue.

[1] Sesse told me that in his case there was a transparent proxy involved when he exeprienced MTU weirdness.

Anthony Towns: Managing Debian Installs

For a while I’ve been trying to find some easy way to keep a few machines I admin behaving the way I want them too with minimal effort. They don’t really need much maintenance – but I would like something to help keep them all in sync. Basically, something like FAI, but much, much simpler – ideally something that takes five minutes to understand, and another five minutes to deploy; and leave the more complicated and powerful tools for when they’re actually needed. I figured what I really want was just a simple way to make a meta-package – one that doesn’t really provide any functionality, just tells apt/dpkg what I want installed (via Depends), and what I don’t want installed (via Conflicts) and adds any extra configuration stuff or local scripts that I decide I want. But doing that with a real Debian package is harder than I’m really comfortable with – I don’t want to have to worry about potential lintian errors, or rules files and debhelper commands, or writing a Makefile to get my files installed or whatever, I want something more trivial than that. Looking for meta-package creators, the only one I spotted that I thought looked likely was cdd-dev, described as “Custom Debian Distributions common files for developing meta packages”. Unfortunately it seems to just provide templates, which makes things quicker, but no less complex. Fortunately equivs (“Circumvent Debian package dependencies”) is actually used for metapackages these days, according to its maintainer on IRC and its long description:
This package provides a tool to create Debian packages that only contain dependency information. One use for this is to create a metapackage: a package whose sole purpose is to declare dependencies and conflicts on other packages so that these will be automatically installed, upgraded, or removed. Another use is to circumvent dependency checking. […]
That turned out to work much better than I remembered (from whenever I last tried it – back in ‘99 I guess?), with the only drawback being that I couldn’t add files easily. But that’s just a matter of creating a patch to equivs, which I then won’t have to worry about again. So having done that, I can now create a metapackage to do whatever I want by creating a file like:
Suite: client
Section: misc
Priority: standard
Package: ajs-client-stuff
Version: 20071114.1
Maintainer: Anthony Towns <aj@erisian.com.au>
Description: Metapackage for aj's client computers
 Depends on necessary packages, etc.
File: /etc/apt/sources.list.d/client.list
 deb http://mirror.localnet/debian etch main contrib non-free
 deb http://mirror.localnet/debian etch-proposed-updates main contrib non-free
 .
 deb http://security.debian.org/ etch/updates main contrib non-free
File: postinst
 #!/bin/sh -e
 .
 apt-key add - <<EOF
 [output of gpg --armour --export $KEY]
 EOF
 .
 ##DEBHELPER##
debhelper kindly takes care of getting the permissions right for me, and equivs will generate a full source package if I tell it to, which I can just upload to mini-dinstall and have a regular Debian repository just by writing a text file and running equivs-build. And my metapackage can add dependencies, conflicts, apt sources, cronjobs, scripts, configuration files, documentation, or whatever I happen to want – which means I can make it automatically update itself, and thus install any dependencies or remove any conflicts, which then means that modifying the config on all the machines is just a matter of updating the metapackage. And new installs is (hopefully) just a matter of doing a standard install and then adding the metapackage. Perfect. …even if it is really little more than a reinvention of rpm’s .spec files. :)

13 November 2007

Russell Coker: Restorecon Equivalent for Unix Permissions

SE Linux has a utility named restorecon to set (or reset) the security context. This is useful for many reasons, corrupted filesystems, users removing files or changing the context in inappropriate ways, and for re-creating files from tar files or backup programs that don’t restore SE Linux contexts. It can also be used to report the files that have different contexts to that which would be set by restorecon to verify the contexts of files. Restorecon determines the context from two sources of data, one is the policy that came with the system (including any policy modules from other sources which were loaded) and the other is the local file contexts that were created by semanage. It’s a pity that there doesn’t seem to be an equivalent program for Unix permissions. rpm has a -V option to verify the files from a package and dpkg doesn’t seem to have an option to perform a similar operation (/var/lib/dpkg/info/* doesn’t seem to have the necessary data). But even on an RPM based system this isn’t possible because there is no way to add local files into the list. I would like to be able to specify that an RPM system should have root:root as the owner and permission mode 0755 for all files matching /usr/local/bin/* and use a single command to check the RPM database as well as this extra data for the permissions of all files. Does anyone know of any work in this area? I’m going to file Debian and Fedora bug reports about this, but I would appreciate any comments first. Update: Here is an example of how this feature works in rpm:
# rpm -Vv nash
…….. /sbin/nash
…….. d /usr/share/man/man8/nash.8.gz
# chmod 700 /sbin/nash
# rpm -Vv nash
.M…… /sbin/nash
…….. d /usr/share/man/man8/nash.8.gz The “M” character indicates that the permission mode of the file does not match the RPM. There is no way to automatically correct it (AFAIK) but at least we know that something changed. With Debian AFAIK it’s only possible to verify file checksums not the permission.

12 November 2007

Benjamin Mako Hill: Code of Conduct

The Ubuntu Code of Conduct is probably the most widely read document I've written. Agreement to it is prerequisite to participation in the Ubuntu community in all official and many unofficial capacities. It is has successfully set a positive tone and helped turn Ubuntu into what is probably the most friendly and civil free software project I've worked in. Over time, quite a few free software projects have copied or adapted the CoC. Tired of giving folks permission, the project went ahead and licensed the CoC under the Creative Commons Attribution-Sharealike license to explicitly allow reuse as long as attribution to the Ubuntu project is given and derivatives are similarly modifiable. In a recent development, it was adapted by the Fort Erie, Ontario town council for use government interactions of their business improvement areas! It's amazing to see the document gain so much traction! Unfortunately, the person who repurposed the CoC did not attribute the document correctly and was publicly accused of plagiarism by another council member! Ubuntu is happy to have Fort Erie, and anyone else, use or adapt the CoC. Folks should just take care to be honest about where it came from and maintain the BY-SA license.

11 November 2007

Rob Bradford: Yes, no, maybe? Ja, nein, vielleicht?

My friend Martin has started a blog on usability, Yes, No, Cancel? (or in German, Ja, Nein, Abbrechen). He appears to have stalled after a good start so perhaps this mention will get him going again. He offers usability consulting through his company, Ept Computing. Martin Kleppmann
Martin, an early victim of the new camera

9 November 2007

Ben Armstrong: Debian-eeepc: the ideal mobile text editing device realized

The search for the ideal mobile text editing device is over. While the price was well over my original budget, we’ve since gone ahead and splurged on two new Eee PCs: one for me, one for my wife. In the end, I think it will be worth it because these systems are capable of far more than just editing text. Now comes the challenging part. Xandros is the default OS and we want Debian on it instead. I have started supplementing the system with packages from Etch and Etch-backports using apt pinning, but already I have noticed some cracks beginning to develop: with pure Xandros, an SD card was automounted when it was inserted. Now it fails. Also, the Network utility no longer launches. I can work around these glitches for now, but it is plain that in the long term this Xandros/Etch hybrid is going to be more grief than it is worth. To that end, enter Debian-eeepc. Building on the work started by timbobsteve and drawing from the collective experience of the Eeeuser.com community we will make a debian-live cd + debian-installer to install Debian (as pure as possible—obviously the kernel is going to be our most contentious issue) on the device. So stay tuned for more articles on that work-in-progress. And how does my new toy shape up for mobile text editing so far? I’m delighted! For starters, this article was drafted on the bus home from work. The keyboard, while small, is still quite usable for touch-typing, the display is crisp and bright, and the size is just perfect for the cramped quarters of a public transit bus seat. But beyond just editing text, I have loaded up the system with all of the tools that go around it: subversion, git, ruby, gcc, meld, etc. I’ll be able to manage a fair amount of development on the bus, with the exception being really large builds that will continue to be done on remote build systems.

Next.

Previous.