Search Results: "cas"

6 November 2007

Christian Perrier: Give credit where due...

I recently ranted about Ubuntu not giving much credit to Debian. I have to complete this by thanking the Merge-o-Matic feature which (for a few months now) sends to package maintainers a diff file with the Debian package when a new Ubuntu package is uploaded. This sometimes make it quite easy to spot modifications done by Ubuntu and eventually adopt them. This already bringed me, as maintainer, a few interesting patches for iso-codes, console-data and geneweb. In the case of iso-codes, samba and geneweb, the Ubuntu patch was even reported manually as a Debian bug by a Ubuntu contributor/developer (no later than today, Martin Pitt took care to suggest us to adopt bzip2 compression for iso-codes and Mathias Gug proposed a patch to enable usershares in samba). So, well, even if things can still be improved, many Good collaboration also often happens.

Christian Perrier: Saved some watts: thanks Julien and Powertop

Julien's blog entry and some spare time lead me to try this powertop utility which I never heard about before. As a consequence, I ended up tine tuning a few parameters on my laptop, mostly sound card and wireless card energy saving parameters. I also finally turned "noatime" ON on my mounted partitions when the laptop runs on battery (I use laptop-mode for such automation). I was prevented to do so util now because mutt relies on access times to define whether mailboxes got new mail or not. However, recent mutt version now have "set check_mbox_size" which make mutt rely on the mailboxes sizes I still have fine tuning to do, mostly trying to get my hard disk really spin down in some occasions (mail reading, mostly) as well as automate the screen brightness, but this powertop utility is really worth it in the hunt for wasted energy. And, of course, installing it is just a matter of "aptitude install powertop", thanks to Patrick Winnertz who packages it. And thanks to Julien who blogged about Powertop, even for ranting about it...:-)

Adam Rosi-Kessel: Please Leave Me Alone John Connolly

Up until recently, I’ve been supporting John Connolly as candidate for city council. In addition to the recent concerns about anonymous mailings, I’m now getting hammered by his auto-dialer. I’ve received four prerecorded phone calls today and yesterday reminding me to vote for Connolly. (By contrast, I got one real human phone call from a Michael Flaherty supporter.) Now with a few hours left to vote, I’m on the fence. I wish this guy would leave me alone. In the meantime, a WBUR interview claims that Felix Arroyo is the most vulnerable candidate (something I’ve never heard elsewhere). If that’s the case, I’m tempted to just bullet-vote for Felix. I hope this last-minute vacillation doesn’t make me an idgit voter. 5pm update: three more automated phone calls! One for Murphy from Consalvo; and two for Connolly (one from Tobin, the other I forget). They’re pulling out all the stops, only at the last minute. I wonder who the dismal weather (and poor turnout) favors. 7pm update: three or four more calls, now including real humans! My wife figured out part of the reason we’re getting so many — separate calls each for her and me, even though it’s all on the same phone number. In any case, we’ve all voted now. I did end up including Connolly in my votes despite these over-the-top tactics. I guess it shows, at least, that he’s well organized. Technorati Tags: , , ,

Manoj Srivastava: Manoj: Continuous Automated Build and Integration Environment

One of the things I have been tasked to do in my current assignment is to create a dashboard of the status of various software components created by different contractors (participating companies) in the program. These software components are built by different development groups, utilizing unlike toolsets, languages and tools though I was able to get an agreement on the VCS (subversion yuck). Specifically, one should be able to tell which components pass pre-build checks, compile, can be installed, and pass unit and functional tests. There should be nightly builds, as well as builds whenever someone checks in code on the release branches. And, of course, the dashboard should be HTTP accessible, and be bright and, of course, shiny. My requirements were that since the whole project is not Java, there should be no dependencies on maven or ant or eclipse projects (or make, for that matter); that it should be able to do builds on multiple machines (license constraints restrict some software to Solaris or Windows), not suck up too much time from my real job (this is a service, if it is working well, you get no credit, if it fails, you are on the hot seat). And it should be something I can easily debug, so no esoteric languages (APL, haskell and Python :P) So, using continuous integration as a google search term, I found the comparison matrix at Damage Control I looked at anthill, and cruisecontrol, and the major drawback people seemed to think it had was that configuration was done by editing an XML file, as opposed to a (by some accounts, buggy) UI is not much of a factor for me. (See this). I also like the fact that it seems easier to plug in other components. I am uncomfortable with free software that has a commercial sibling; we have been burned once by UML software with those characteristics. Cruisecontrol, Damagecontrol, Tinderbox1 & tinderbox2, Continuum, and Sin match. I tried to see the demo versions; Sin s link led me to a site selling Myrtle Beach condo s, never a good sign. Continuum and Damagecontrol were currently down, so I could not do an evaluation. So, here are the ones I could get to with working demo pages: http://cclive.thoughtworks.com/ and http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey Cruisecontrol takes full control, checking things out of source control; and running the tests; which implies that all the software does build and run on the same machine this is not the case for me. Also, CC needs to publish the results/logs in XML; which seems to be a good fit for the java world; but might be a constraint for my use case. I like the tinderbox dashboard better, based on the information presented; but that is not a major issue. It also might be better suited for a distributed, open source development model; cruisecontrol seems slightly more centralized more on this below. cruisecontrol is certainly more mature; and we have some experience with it. Tinderbox has a client/server model, and communicates via EMAIL to a number of machines where the actual build/testing is done. This seems good. Then there is flamebox nice dashboard, derivative of tinderbox2; and seems pretty simple (perhaps too simple); and easily modifiable. However, none of these seemed right. There was too much of an assumption of a build and test model and few of them seemed to be a good fit for a distributed, Grid-based software development; so I continued looking. Cabie screen shot. I finally decided CABIE
Continuous Automated Build and Integration Environment. Cabie is a multi-platform, multi-cm client/server based application providing both command line and web-based access to real time build monitoring and execution information. Cabie builds jobs based upon configuration information stored in MySQL and will support virtually any build that can be called from the command line. Cabie provides a centralized collection point for all builds providing web based dynamic access, the collector is SQL based and provides information for all projects under Cabie s control. Cabie can be integrated with bug tracking systems and test systems with some effort depending on the complexity of those systems. With the idea in mind that most companies create build systems from the ground up, Cabie was designed to not have to re-write scripted builds but instead to integrate existing build scripts into a smart collector. Cabie provides rapid email notification and RSS integration to quickly handle build issues. Cabie provides the ability to run builds in parallel, series, to poll jobs or to allow the use of scripted nightly builds. Cabie is perfect for agile development in an environment that requires multiple languages and tools. Cabie supports Perforce, Subversion and CVS. The use of a backend broker allows anyone with perl skills to write support for additional CM systems.
The nice people at Yo Linux have provided a Tutorial for the process. I did have to make some changes to get things working (mostly in line with the changes recommended in the tutorial, but not exactly the same. I have sent the patches upstream, but upstream is not sure how much of it they can use, since there has been major progress since the last release. The upstream is nice and responsive, and have added support in unreleased versions for using virtual machines to run the builds in (they use that to do the solaris/windows builds), improved the web interface using (shudder) PHP, and and all kinds of neat stuff.

Gunnar Wolf: Lucas killed ries!

Is there a causality relation between Lucas' posting and ries.debian.org's (a.k.a. ftp-master.debian.org) sudden demise? Grmbl...

5 November 2007

Lucas Nussbaum: NEW processing

NEW is currently nearly empty (only 3 packages), and there are only 2 NMs waiting for DAM approval. Some people deserve beers (… or milk ;) )!

4 November 2007

MJ Ray: Dear Blogger - the truth about your CAPTCHA

Dear Blogger, Your CAPTCHA (word verification or whatever) doesn't test whether someone is spamming. It tests whether someone has good eyesight, hearing, literacy, numeracy or browser-configuration, which is correlated in one way (few spambots include captcha-crackers yet, but there are some and they already have an 80% success rate IIRC) but it is not strongly correlated for humans (are blind people more likely to spam? I doubt it) and now some spammers are using porno-trojans to get humans to crack the eyetests for them - see http://www.schneier.com/blog/archives/2007/11/spammers_using.html http://news.bbc.co.uk/1/hi/technology/7067962.stm I block spam by premoderating all comments (they're well over 95% spam), but that does mean you need to check your site dashboard a lot to avoid cooling discussions too much. Other anti-spam tactics include OpenID, URL blacklists and so on - I don't use them because I've not added support for them to my site yet. I suspect Blogger.com users can't use them because Google hasn't added support for them. They've bought captcha's false sense of security. http://www.w3.org/TR/turingtest#security As for encouraging non-spam comments, I'll generally defer to this top 10 from Darren Rowse: http://www.problogger.net/archives/2006/10/12/10-techniques-to-get-more-comments-on-your-blog/ On my sites, I find that inviting questions doesn't make much difference, asking questions has limited use because readers are suffering from "question-fatigue" these days and I have difficulty appearing to be humble or gracious (people often read it as sarcasm! Hopefully that's just my audience), but the rest of the techniques seem to work for me and I use them when I can (not all of the sites I write are under my full control). Please let me know if you see other successful ways to reduce spam or encourage good comments. Hope that helps, MJR/slef

Lucas Nussbaum: Some numbers about Debian s history

There’s an interesting article on LWN.net about Gentoo: Who made Gentoo Linux, and when? A commit analysis I was wondering how Debian compared, so I made some stats using the debian-devel-changes archives. First, the number of source uploads per quarter:
It’s interesting to note that, while the number of packages in Debian increased a lot, the number of uploads basically stayed the same since 2001. Then, I wondered about the number of active developers in Debian. So I counted: I can provide the raw data on demand, if someone else want to work on this. There’s probably a lot of other interesting things to do!

Daniel Burrows: Lessig at the University of Washington

On Friday 2007-11-02, I watched Larry Lessig give a presentation at the University of Washington entitled Is Google (2008) Microsoft (1998)? Since it was Lessig, the talk was articulate and thought-provoking, and he used his slides very well (unlike many presenters who just read bullet points).

The argument he made is that Google is like Microsoft in some ways (and so are other companies like Facebook). In his analysis, the problem with Microsoft was that so many companies made themselves Microsoft-dependent, which both forced other companies to follow suit (due to network effects) and gave Microsoft leverage they could use against everyone in their ecosystem (e.g., Netscape). According to Lessig, Microsoft was not evil to use this leverage, but you have to recognize that Microsoft will do what is best for Microsoft, and this may not be what is best for you. He then suggested that we should be aware of the potential for the same thing to happen; in support of this point, he quoted excerpts from the terms of service from the APIs of Google and Facebook, which contain typical statements like we reserve the right to terminate your use of this service for any reason or for no reason, and asked: What if Microsoft had written this license agreement?

But it seemed to me that this wasn't his main point. He was using this observation as a springboard to talk about issues revolving around competition and public policy. Specifically, he feels that we should work towards a more just model of interaction between companies and users whose work makes the company more valuable (by contributing to a software ecosystem, by posting content on the Web site, etc). But the correct way to do this (he says) is not via competition, after-the-fact litigation, or voluntary codes of conduct: competition may not be around to enforce good behavior (as we've seen with Microsoft), litigation may be too late and ineffective (again, see Microsoft), and voluntary codes of conduct will disappear as soon as they get in the way of the bottom line. The correct approach, he argued, is across-the-board regulation so that all the companies have to play by the same rules.

From there he went on to talk about governmental corruption. He said (quoting Robert Reich's book Supercapitalism) that the reason we consistently look for solutions in the market and in voluntary compliance is that our governmental system is broken and does not effectively regulate corporations in the public interest. But Lessig is optimistic that we can change things (he joked that his publisher was unhappy with this point of view, because his brand has been built on pessimism). In his view, the politicians in Washington, by and large, want to be honest and do good, but they aren't able to within the current political system. For instance, on the few occassions that he managed to get access to lawmakers to discuss copyright issues, it was often the first time they had heard that there was more than one side to the argument. He thinks we need a national political movement that will shame politicians into being less corrupt.

This is all a simplification of what he said, of course; unfortunately, I wasn't able to find a recording of this talk on Lessig's Web site, or I'd point you at primary source material. I don't know if this is because he chose not to post it, or because it just hasn't been made available yet.

My view

In general, I thought that the speech was very interesting and well thought out. But as much as I'd like to buy into his optimism that we can fix our system, I don't think it's well-founded, for two reasons.

First, he argued that competition will resolve some of these problems (that was before and after he said we can't rely on competition, which I found confusing and may indicate that I misunderstood something). His evidence for this seemed to be that when Microsoft was dominating the world, users responded by reducing their dependence on Microsoft products and by switching to alternatives such as Linux. But, in fact, although more people may be using Linux than in the past, virtually everyone is still only using Windows, it's still almost impossible to find a programming job that isn't Windows-only, and Microsoft is still raking in money hand over fist; indeed, they're doing so well that they're planning to expand their workforce by a third. To take just one data point of many: I ride the bus to Seattle and back, and I regularly see other people using laptops as they commute. Virtually all the laptops run Windows. Occasionally I'll see a Mac, and every few months I'll see a guy running Linux on his laptop; I say "a guy" because it's the same guy every single time. So by my estimation, use of Linux on laptops in buses on the 545 route from Redmond to Seattle is way below 1%, with Macintoshes somewhere around 5%.

To broaden my point, I think Lessig has fallen into a trap that highly intelligent people, particularly those in academia, tend to fall into. In academia, you are surrounded by people who value learning and thinking deeply about the world. These are people who will give serious and unbiased consideration to questions like what is the social effect of my acceptance of the Facebook Terms of Service? and If I become dependent on this service, will that possibly affect me in five years if the company decides to act against my interests? The problem with this is that these people are not representative of the population at large. The population at large thinks why is this check-box getting in the way of me sending pictures to my friends? And I have no doubt that any kid who, e.g., refuses to sign up for Facebook as a protest against their TOS will be roundly mocked in their social circles for being a weird antisocial nonconformist. (of course, this may not apply to the small minority of people like myself whose social circles consist of weird antisocial nonconformists)

Second, Lessig's optimism about the political system seemed to be based on his observation that the people in government are, by and large, not venally corrupt: they want to do good, but the system is constructed in a way that makes this impossible. To me, that's a tremendously disheartening statement. If our problem were simply a few, or even many, corrupt politicians, this would be a solvable problem: even nowadays, Americans have some ability to choose who gets elected, and I believe that a sufficiently well-coordinated campaign could replace bad politicians with good ones.

But if the problem is that the system is corrupt even when the individuals are honest, it's a much deeper problem and frankly one that I don't know how to solve. While individuals have some direct control over who gets elected, we have no way to directly change the system that produces the corruption; the levers to do this are in the hands of the politicians elected through that broken system. It's unlikely that the politicians themselves will fix the problem, because they've benefited deeply from it: if the system is left as it is, well over 90% of Congress members will keep
their jobs
in the next election cycle, and any effective change seems likely (by definition) to disrupt this cushy arrangement.

On the other hand, running new candidates for office will not fix the problem. After all, if the problem is that the system is corrupt even when the participants are honest, then putting more honest people into government will have no effect. In order to get elected, these new politicians will have to become just as corrupt as the old ones, because that's how the system is set up. So we might change the faces, but we'll have the same old problems.

I recall two specific concrete proposals that Lessig made to make the system more responsive. First, he suggested shaming politicians who engage in corrupt behavior -- presumably taking large donations from interested parties. I don't see how this will help. Americans generally have a very low regard for the political class as it is; will pointing out particular examples of politicians taking money really make a difference when it comes to votes? And if it doesn't, will it really produce any change in their behavior? Politicians are motivated by votes the same way corporations are motivated by money; unless they stand a chance of losing their job (which is virtually impossible in any event), there's no reason for them to change how they behave.

His second proposal was for a system of contributions where it's impossible for any individual to prove they had donated a particular amount of money to a particular candidate. Even assuming it could be implemented perfectly, I don't think this will really solve the problem, for two reasons. First, it's not generally a secret what moneyed interests want. Politicians who want to attract large donations can just take positions that they know will appeal to donors of that sort. The donors themselves can make this easier on the candidates by, e.g., posting public position statements on issues of the day. Secondly, and this is far more insidious, even if politicians honestly represent their positions, then because success in running for office is so tied to the amount of money the candidate can raise, only candidates who hold opinions favorable to large corporations and wealthy individuals will be able to get elected. In fact, for all I know this is what happens already!

But with all that said, it was a really interesting talk and Lessig tied together some things you might not think would go together. If a video or Flash presentation becomes available, I would recommend watching through it. And hopefully my pessimistic predictions will be wrong; it's certainly happened before.

3 November 2007

Simon Richter: XUL

in case my last post was not cynic enough yet: "XUL is the Java of the new millenium"

Biella Coleman: On Bats, The Suckage of American Health Insurance and Paperwork Warfare

Two nights ago I was having dinner with a colleague who recounted a truly horrible experience he had with the modern-day American health care “system.” While in Montana over the summer, he got bit by a bat and well, as a result, he and his entire family had to get rabies shots. These shots are hard to get and priceeeeeeeeeeeeeeeeeeey. The final bill was $ 20,000 and his NYU based health insurance, United Health Care is claiming they are not responsible!### Like getting bit by a bat and the possibility of rabies are not bad enough. Isn’t the point of health insurance to take care of the unexpected?? Someone also pointed me to The Daily Kos’ wrangle with BCBS of California and well, they picked the wrong guy to mess with given there are so many many many many eyeballs tuned to his blog, everyday (just check out the number of comments). I wonder if the public relations director will contact him as they did with me after I blogged about my trials and tribulations. I certainly don’t have the audience and readers he does, so I am always surprised they ever contacted me in the first place. The Daily Kos also provided a link for a site that I was thinking of creating if my problems with BCBS were not solved (glad to see someone else has) called Sick of Blue Cross. I personally like the name I came up with a little better Blue Cross No Shield but the important thing is that a site like that exists and it is very good site, design and content wise. There should be a site created for every health insurance company and they should federate and pool resources to gain more visiblity. While I am not hopeful that the health insurance industry will change within, I think mounting and steady consumer pressure can do a lot to force the industry to change their awful ways. And while I think the bad publicity is a key part to putting pressure on the health insurance to change, I actually think that the power of the consumers also can come from elsewhere, and this elsewhere is overwhelming the administrative channels within and outside of the health insurance. The gist is if they send us a lot of (useless) paperwork, send them A LOT of paperwork back. On the Daily Kos, a lot of people noted that things got moving when they complained to the Department of Banking and Insurance. While my own complaint to that department did not produce any immediate fruit, it did get the ball moving for the appeal process, which eventually panned out for me. That is, I learned a lot and started to get decent customer service. Doors opened. Now, it took me weeks to find out that I could complain to the Department of Banking and Insurance and I am sure many find themselves in that situation of walking in a no man’s land of confusion and frustration. If more and more people know that you can file a complaint to these Departments (and why are doctor’s offices not relaying this information??!!), they and the health insurance companies will get inundated with that which they love to inflict on us, paperwork. This may slow things down for you but at a certain point, state governments will get the message that something is deeply wrong. So, people MUST use these channels and overwhelm these agencies with complaints, faxes, emails as this is a powerful and very material message that something is deeply wrong and broken and must be fixed. Also, send faxes and letters to various departments in your health insurance company (appeals, customer service, medical director’s office, public relations office…) get people’s email’s and send them emails. Inundate them with a barrage of administrative paperwork. If they are not being helpful, send a fax or two a day concerning your case until you get clear answers. While individual faxes won’t make much of a difference, imagine if there are 300 a day coming in or 3000…. This can work with the numbers are there and I have a feeling that the numbers Are There. So spread the word for housebreaking your health insurance and push the paperwork back on them.

2 November 2007

Wouter Verhelst: Vim vs Emacs

John Goerzen blogs about his experience of using vim for about 8 months. Having been an emacs user a few years back, I understand both editors on a basic level, and can relate to some of the things he says. I have to disagree with one particular bit, though:
It starts faster. I'm not sure if that really was true even when I switched, but it certainly isn't true on any of my machines today. Both Vim and Emacs have had major version upgrades (v7 and v22, respectively) since I started using Vim. People seem to say that Emacs 22 feels faster, though I don't know if that's true. The startup times of the two, if they're different, are imperceptible.
Sorry, but that just isn't true. Using my very non-scientific method of measuring startup times ("click on the icon, watch the clock, and start counting until the editor is ready to use"), I can tell that gvim on my hardware takes less than a second to start, whereas emacs requires approximately 5 seconds to do the same. For reference, this is on my laptop, a 1.3Ghz PowerPC G4. In any case, I'm not a fan of either editor. There are things that vi does well, and there are things that emacs does well. On emacs' end, I like M-x gdb, I slightly prefer its M-x compile over vim's :make, and I just love psgml. I've also occasionally used gnus for news reading (never for mail, though). On vim's end, I find regex search & replace easier (as you mention in your blog post), and I prefer its syntax highlighting (I've seen emacs get confused because the first line in the output window started in the middle of a string. Such things will never happen to vim: if it gets confused once on a given file, it will always be confused on the same file. While annoying, at least you know it's there). All in all, I don't think it matters much. I've seen people who use an editor as an operating system, both in the vim and in the emacs camp, by having a ~/.vimrc or ~/.emacs that's larger than my collection of pictures[1]. I'm not one of them; to me, an editor that doesn't properly work in its defaults shouldn't exist. [1] okay, that's an exaggeration. Still.

Wouter Verhelst: Beid updates

There've been some posts on Planet Grep about using the Belgian electronic ID card in Ubuntu, recently. As the one who's responsible for making that possible, I felt I had to comment. First, yes, it's possible to do crypto stuff with plain OpenSC rather than the belpic-patched versions; in fact, the tools required to sign plain text files aren't shipped anymore as part of belpic since (I believe) version 2.5.9, which is currently in Debian. However, if you're going to write your own software that somehow uses the eID, I strongly recommend you use the beid-patched versions, since they have support for a feature of which I can't remember the name currently but which means that you need to send your PIN to the card on every signature operation; plain OpenSC is not aware of this possiblity, so does not have support for it, may end up in a confused state when the card refuses to sign, and you may end up locking your card if you keep going on. You have been warned. Second, there is a README.Debian file in libbeidlibopensc2 which explains some of the idiosynchracies in the packaging, and tells you which packages you need and which ones you don't. If you're interested in using the eID, I suggest you read it. Finally, I'm currently working on packages of 2.6.0, which was released on 2007-08-30. They're mostly ready; I have a rather large changelog currently and a two-item TODO list—one of which reads "test&upload"; if all goes well, I should be able to upload them this weekend. Occasionally, this upload includes a small patch that was communicated to me through some mandriva developer which will make one of the aforementioned idiosynchracies—the fact that you have to install the -dev packages if you want to use the mozilla plugin—obsolete.

Adam Rosi-Kessel: Catch-22 Poor Organization, and City Council Endorsements

I was quoted in a recent Needham Times article regarding my complaints with the T:
But Adam Rosi-Kessel, a Roslindale resident who takes the Needham Line to his job in the financial district, said 15- to 20-minute delays have kept him from getting to work before 9 a.m. since May. Now Rosi-Kessel, a lawyer who biked to work when he lived in Jamaica Plain, is thinking about trading in his Charlie Card again. It used to be, living here in Roslindale, the train was always the fastest way to get in, he said. Now, it s slowed down to the point where it s starting to get competitive with biking again.
I’m glad they got my biking quote. Here’s the part I don’t understand (emphasis added):
Massachusetts Bay Transportation Authority spokeswoman Lydia Rivera said Needham Line delays have been caused by construction work and speed restrictions on tracks that don t go anywhere near Needham. In order to maximize train use, the T often transfers trains between tracks, rather than leaving underused trains empty in the MBTA yard. The method helps the T get trains where they re needed most, but it also means a delay on one track can slow service on a seemingly unrelated track. It s a Catch-22, Rivera said.
I don’t think the T spokesperson knows what a “Catch-22″ is. I refer those interested to the Wikipedia article on the subject, but it is nicely summarized there as “heads I win, tails you lose.” The T’s problem is not a logical paradox, but some combination of inadequate resources and poor organization. Can someone please help them out? Also on Boston local topics: I’ve received several unattributed mailings lately related to the city council election that do not facially promote any particular candidate. Today’s mailing attacked Stephen Murphy for repeatedly trying (and failing) to win some other office or get some other job than City Councilor. The return address was 31 Milk Street, which is the address of many different businesses. A few days ago, I received another unattributed mailing bemoaning how long it has been since an at-large city councilor came from the Parkway Area (my neighborhood), but not mentioning any candidate in particular. My guess is these mailings are all meant to support John Connolly, a West Roxbury resident, attorney, and ostensibly good guy. I was feeling pretty happy about the possibility of Connolly replacing Murphy on the Council (I have to admit some unfair prejudice in that the only house in our neighborhood I’ve ever seen prominently posting a sign in support of Bush also features a Murphy billboard). But these questionable campaign tactics are giving me pause. Does anyone know anything more about this? In any case, here are my endorsements for next week’s election, notwithstanding the concern outlined above: (1) Felix Arroyo, (2) John Connolly, and (3) Sam Yoon. I have no pick for a fourth candidate. I’ll also throw in a vote for Matt Geary. I really wouldn’t want the socialists running the City, but a broader spectrum of political opinion within the council wouldn’t hurt. Incidentally, it is remarkably hard to find any information summarizing all of the candidates’ positions and records. It’s almost as if the election isn’t even happening. (A commenter properly points me to Brighton Centered as a good resource.) 11/3/07 Update: More discussion here. Technorati Tags: , , , , , , , , , , , ,

1 November 2007

Andrew McMillan: DAViCal and support for Apple's CalDAV client in OS 10.5

Quite a few people seem impressed with the new release of Leopard, and are now looking for a CalDAV server to use with their shiny new iCal app. Unfortunately it seems that Apple wrote this primarily to work with their own (free, open-source) calendar server, which has the side effect that it doesn't work with DAViCal. Part of the "doesn't work" is due to DAViCal not implementing some areas of the CalDAV specification, which is fair enough. Part of it is due to DAViCal not implementing some draft extensions to the CalDAV specification, which I can also understand, since it allows them to provide some useful features that those extensions are designed to support. There also seem to be some parts of the "doesn't work" which are due to a dependence on extensions beyond either of these cases, which is a little more disappointing - and quite a bit harder to implement. So far I have made some fixes to the first point, and some additions towards parts of the second, but as of today it still does not work. This is complicated by my not having access to a Mac. Things are looking up, however, because Tom Robinson has kindly agreed to loan me a Mac running Leopard from next week. In order to "clear the slate" for that, I will be releasing a 0.9.2 over the weekend with the various minor enhancements and fixes that have been applied over the last week. So although this upcoming release will let you add your DAViCal account to iCal 3, it still won't actually work with it. I'm hoping that ready access to the application will enable me to correct that fairly quickly. Also waiting in the wings (and which unfortunately won't be in 0.9.2 either) Maxime Delorme has been working on SyncML support, and is nearly ready with a patch, so we can look forward to that addition fairly soon also.

Russell Coker: Load Average

Other Unix systems apparently calculate the load average differently to Linux. According to the Wikipedia page about Load(computing) [1] most Unix systems calculate it based on the average number of processes that are using a CPU or available for scheduling on a CPU while Linux also includes the count of processes that are blocked on disk IO (uninterruptible sleep). There are three load average numbers, the first is for the past minute, the second is for the past 5 minutes, and the third is for the past 15 minutes. In most cases you will only be interested in the first number. What is a good load average depends on the hardware. For a system with a single CPU core a load average of 1 or greater from CPU use will indicate that some processes may perform badly due to lack of CPU time - although a long-running background process with a high “nice” value can increase the load average without interfering with system performance in most cases. As a general rule if you want snappy performance then the load average component from CPU use should be less than the number of CPU cores (not hyper-threads). For example a system with two dual-core CPUs can be expected to perform really well with a load average of 3.5 from CPU use but might perform badly with a load average of 5. The component of the load average that is due to disk IO is much more difficult to interpret in a sensible manner. A common situation is to have the load average increased by a NFS server with a network problem. A user accesses a file on the NFS server and gets no response (thus giving a load average of 1), they then open another session and use “ls” to inspect the state of the file - ls is blocked and gives a system load average of 2. A single user may launch 5 or more processes before they realise that they are not going to succeed. If there are 20 active users on a multi-user system then a load average of 100 from a single NFS server that has a network problem is not uncommon. While this is happening the system will perform very well for all tasks that don’t involve the NFS server, the processes that are blocked on disk IO can be paged out so they don’t use any RAM or CPU time. For regular disk IO you can have load average incremented by 1 for each non-RAID disk without any significant performance problems. For example if you have two users who each have a separate disk for their home directory (not uncommon with certain systems where performance is required and cooperation between users is low) then each could have a single process performing disk IO at maximum speed with no performance problems for the entire system. A system which has four CPU cores and two hard drives used for separate tasks could have a load average slightly below 6 and the performance for all operations would be quite good if there were four processes performing CPU intensive tasks and two processes doing disk intensive tasks on different disks. The same system with six CPU intensive programs would under-perform (each process would on average get 2/3 of a CPU), and if it had six disk intensive tasks that all use the same disk then performance would be terrible (especially if one of the six was an interactive task). The fact that a single load average number can either mean that the system is busy but performing well, under a bit of load, or totally overloaded means that the load average number is of limited utility in diagnosing performance problems. It is useful as a quick measure, if your server usually has a load average of 0.5 and it suddenly gets a load average of 10 then you know that something is wrong. Then the typical procedure for diagnosing it starts with either running “ps aux grep D” (to get a list of D state processes - processes that are blocked on disk IO) or running top to see the percentages of CPU time idle and in IO-wait states. Cpu(s): 15.0%us, 35.1%sy,  0.0%ni, 49.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
7331 rjc      25  0  2868  640  312 R  100  0.0  0:21.57 gzip Above is a section of the output of top showing a system running gzip -9 < /dev/urandom > /dev/null. Gzip is using one CPU core (100% CPU means 100% of one core - a multi-threaded program can use more than one core and therefore more than 100% CPU) and the overall system statistics indicate 49.9% idle (the other core is almost entirely idle). Cpu(s):  1.3%us,  3.2%sy,  0.0%ni, 50.7%id, 44.4%wa,  0.0%hi,  0.3%si,  0.0%st   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
7425 rjc      17  0  4036  872  588 R    4  0.1  0:00.20 find Above is a section of the output of top showing the same system running find /. The system is registering 44% IO wait and 50.7% idle. The IO wait is the percentage of time that CPU core is waiting on IO, so 44% of the total system CPU time (or 88% of one CPU core) is idle while the system is waiting for disk IO to complete. A common mistake is to think that if the IO was faster then more CPU time would be used, in this case with the find program using 4% of one CPU core if all the IO was instantaneous (EG in cache) then the command would complete 25 times faster with 100% CPU use. But if the disk IO performance was doubled (a realistic possibility given that the system has a pair of cheap SATA disks in a RAID-1) then find would probably use 8% of CPU time. Really the only use for load average is for getting an instant feel for whether there are any performance problems related to CPU use or disk IO. If you know what the normal number is then a significant change will stand out. Dr. Neil Gunther has written some interesting documentation on the topic [2], which goes into more technical detail including kernel algorithms used for calculating the load average. My aim in this post is to educate Unix operators as to the basics of the load average. His book The Practical Performance Analyst gives some useful insights into the field. One thing I learned from his book is the basics of queueing theory. One important aspect of this is that as the rate at which work arrives approaches the rate at which work can be done the queue length starts to increase exponentially, and if work keeps arriving at the same rate when the queue is full and the system can’t perform the work fast enough the queue will grow without end. This means that as the load average approaches the theoretical maximum the probability of the system dramatically increasing it’s load average increases. A machine that’s bottlenecked on disk IO for a task where there is a huge number of independent clients (such as a large web server) may have it’s load average jump from 3 to 100 in a matter of one minute. Of course this won’t mean that you actually need to be able to serve 30 times the normal load, merely slightly more than the normal load to keep the queues short. I recommend reading the book, he explains it much better than I do.

Steve Kemp: It's been seven hours and fifteen days

I made a new release of the Chronicle blog compiler the other day, which seems to be getting a suprising number of downloads from my apt repository. The apt repository will be updated shortly to drop support for Sarge, since in practise I've not uploaded new things there for a while. In other news I made some new code for the Debian Administration website! The site now has the notion of a "read-only" state. This state forbids new articles from being posted, new votes being cast, and new comments being posted. The read-only state is mostly designed for emergencies, and for admin work upon the host system (such as when I'm tweaking the newly installed search engine). In more coding news I've been updating the xen-shell a little recently, so it will shortly have the ability to checksum the filesystem of Xen guests - and later validate them. This isn't a great security feature because it assumes you trust dom0 - and more importantly to checksum files your guest must be shutdown. However as a small feature I believe the suggestion was an interesting one. Finally I've been thinking about system exploitation via temporary file abuse. There are a couple of cases that are common: Exploiting the first to go from user to root access is trivial. But how would you exploit the last two? Denial Of Service attacks are trivial via the creation/truncation of /etc/nologin, /etc/shadow, (or even /boot/grub/menu.lst! But gaining privileges? I can't quite see how. Comments welcome!

31 October 2007

Andrew McMillan: CRM114 Awesomeness

I hate spam! Which probably puts me in the same camp as 99.99999% of the world. The other 1 in 10 million are, of course, the spammers, who seem to take the space invaders approach to sending e-mail: we'll keep sending you more until you die. A few years ago I used to only receive perhaps 1 every 100 seconds, which was pretty annoying, but Spamassassin was quite able to filter out 99% of those and let through about 1-2 each day, which I could deal with. My spam levels increased to maybe 1 every 20 seconds, and late in 2005 I implemented a second layer of spam filtering on my laptop using DSpam. This worked quite effectively, but DSpam is really not the tool for the job - it's much more appropriate as a company-wide antispam solution, and potentially as a replacement for Spamassassin. It drove me nuts on my laptop because it's resource usage slowed down the interactive response. When I got my new laptop at the beginning of the year I decided against continuing with my rather baroque mail setup and to leave the spam filtering on the server. What I didn't realise is that my spam rate had increased again to around 1 every 8 seconds, and it has been slowly driving me to distraction ever since. It seems to have cranked up another notch recently, to perhaps 1 every 3 seconds now, so that 1% making its way through Spamassassin was getting to a very annoying several hundred each day. The longer I took to resolve it, the more time I would be wasting dealing with it every day. What I chose to apply on this occasion was CRM114, which I had some vague idea might be able to help. I was fairly impressed by the relatively simple install, but what completely blew me away was the speed with which it was able to learn to be useful. Starting from scratch, it seems to be correctly classifying over 90% of my incoming mail after about 12 hours of training, on a total of only 75 'Unsure' messages. Even after only an hour it was getting over 50% (I'll describe my actual CRM114 installation process in a comment below). So far there have been no false positives. Now that CRM114 is installed I will be able to look into some of it's other mail classifying features too, and I'm really looking forward to that too.

MJ Ray: Current Issues / Tesco wants even more car parking!

Tesco in WsM centre has applied to change their car park layout and fit in more cars!  See http://212.158.113.30/MULTIWAM/showCaseFile.do;jsessionid=3211D08BDB306F441561341603089E41?action=show&appType=planning&appNumber=07/P/2670/F

I don't really like that store.  It's typical Tesco: it faces away from the town, to discourage shoppers visiting anywhere else, and is a bit of a pain to access except by car, with its narrow pedestrian ramps and botched cycle parking.

Actually, it's a fairly sensible planning application apart from the net gain of six spaces, but I'm objecting on the grounds that they're not improving the cycle parking in line with the car parking.  If you shop in Weston-super-Mare, leave your own comment.

Luciano Bello: captchas: The Good, the Bad and the Ugly

Some months ago, I commented about a weak implementation in a fancy captcha. Today I would like to comment about other bad implementations, but in other ways.
The goodA captcha should have big Shannon entropy, finite, but big. The session ID and the challenge must not be reused. The images must be resistant to OCR but should be understandable by a human.
The badHere is the first example:

Believe it or not.. This is a real case. So incredible eh?

The uglyThe victim, in this case, is this one:
This is an implementation of captchanumbers, by Hadar Porat. This captcha and many others generated by captchanumbers can be read with this script.

The idea is simple. As the numbers are nearly in the same place, they can be cut. Those parts can compared independently, reducing the entropy. May be the script and this image would be more enlightening than my limited English:

The 10,000 possibilities was reduced to 159. No OCR, 100% deterministic.

Second moral: Understand the fundamentals first, write code later.

Next.

Previous.