Search Results: "tar"

16 April 2007

Matt Brubeck: 15 Apr 2007

A hacker remembers This story by David Humphrey from last week really hit home:
She told about how my grandfather made his living repairing radios, toasters, fans–anything and everything electrical. He’d only received a 4th grade education in Scotland, before he had to start earning a living to support the family. She described how their entire house, with its mechanical and other systems, was kept working through his creativity and persistence. When the well pump went, he made a new one out of parts from three others…
David could have been describing my own grandfather, who died last month. I knew him as Ojiichan ("Grandpa"). A child of immigrant farmers, his formal education stopped after the sixth grade. As a young man during the internment, he took a correspondence course in radio repair, and after the war started a life-long career in electronics. He worked on electronics at home too, building projects like a cruise control for his own car. Almost up until his death at 93, he lived alone, still drove, and still flew every year to keep his pilot's license up to date. At his memorial service, his family remembered his love of tricky, hard-to-solve problems. Many of Ojiichan's children and grandchildren have grown up to be engineers, but few of us could do some the things he did. He was a lively embodiment of the hacker spirit.y embodiment of the hacker spirit.

15 April 2007

Steve McIntyre: Stay on target...

It looks like some of us might be a little busy this weekend. If all goes well, then it looks like next weekend may be busy too. Yay! UPDATE: The party happened yesterday, and we had more than 20 people for the BBQ. Yay! :-) I was too busy cooking and drinking to be bothered with photos, but I expect others will oblige at some point - check the wiki page above for updates.

Daniel Burrows: How *not* to impress your customers.

So, today I tried to sign up for a service online (unidentified for reasons that will become obvious briefly). This is a service that, had I successfully signed up, would be withdrawing monthly subscription fees from my bank account and handling some data that's fairly important to me. But when I hit the button to start the registration process, I got back...


[TCX][MyODBC]You have an error in your SQL syntax. Check the manual that corresponds to your MySQL server version for the right syntax to use near ''7GDiSp=E8-Mgvgo\', Now() )' at line 9

SQL = "INSERT INTO [REDACTED] ([REDACTED], FieldName, FieldValue, update_dt ) VALUES ( 10724, 'PASSWORD', '7GDiSp=E8-Mgvgo\', Now() )"


The password I tried to enter, for the record, was 7GDiSp=E8-Mgvgo\#dEwb8m7Ec_e~z0myAAj. This is at best inexcusable sloppiness and at worst a security hole, and I don't want a company that does either anywhere near my money and data. (sadly, there probably are already many such companies near my money and data, but I try to avoid the ones I know about)

14 April 2007

Ben Hutchings: ACCU Conference

I've been meaning to attend the ACCU Conference for some years and finally decided to pay my own way for a couple of days this year. It was an opportunity to hear world experts on C++ and other programming topics and to a smaller extent to meet and socialise with other professional programmers. I attended the following sessions: Keynote: Improving Collaboration in Open Source Projects, Mark Shuttleworth This was a somewhat surprising choice of speaker and subject since I think most attendees work on proprietary software, but I think any sensible developer in the proprietary world will try to build on free libraries under non-copyleft licences, where appropriate. Mark talked about governance and leadership in "open source" projects (actually praising Debian for having a clear constitution) and about the need for good communication between projects, for example about release schedules. He seems to have accepted that Launchpad cannot be a single hub, and called for standard formats for, for example, exchanging information about related bugs between bug trackers. But it doesn't sound like he's ready to take the lead on that. Choose your Poison: Exceptions or Error Codes, Andrei Alexandrescu Andrei looked at the strengths and weaknesses of various error reporting mechanisms in C++ and introduced a technique for making error handling more flexible. Normally a function is designed either to indicate errors through its return value or to throw an exception on error. If a function returns an error indicator, it's easy for callers to ignore it. If it throws an exception on error, then a caller that provokes errors more often than the called function's author anticipated pays a high cost in processing time for those errors. Andrei's solution involves a class template Likely<typename T> whose instances hold either a return value of type T or an exception. A function that can fail has a specialisation of this template as its return type. The caller can check for failure if it knows how to handle it, or it can ignore it - in which case the conversion to T or the destructor will throw the exception. This is clever but doesn't seem to be entirely satisfactory. The destructor which throws does first check that no exception is already being processed, avoiding collision of the two exceptions and consequent abrupt termination of the program, but this can result in errors being unexpectedly ignored. Introduction to Component-Level Testing, John Lakos A marathon 3-hour talk on designing and testing software, particularly C++ programs (the subject of his well-known book) as a set of well-defined components. Lakos first defined these components as being "physical" i.e. sets of source files - normally a header file defining an interface, an implementation file, and a test driver file. They may also include metadata for the build system. He talked about the value of reusable components (as if this wasn't obvious!) and the risk of introducing circular dependencies - which essentially turn many components into one and reduce reusability - if we fail to keep track of dependencies. Software built out of components can be viewed as a hierarchy or stack defined by dependencies. In a large system, components with similar dependencies and dependents can be grouped together into packages or package groups to simplify a view of these dependencies. He explained that the test driver's dependencies count just as much as the implementation's, because we cannot reasonably test a component until we have tested its dependencies. Test drivers should also be written to avoid depending on environmental settings and configuration, so far as possible. Finally, and perhaps most importantly, he spent a long time talking about how to construct test cases and test data. I might try to summarise this in a later entry if anyone's interested. The Appliance of Science: Things in Computer Science that every practitioner should know, Andrei Alexandrescu Andrei began by complaining that OpenOffice Impress was even worse than PowerPoint, but being able to use Linux (specifically Ubuntu) more than made up for the pain. He briefly presented four key ideas that he believes programmers should understand and be ready to make use of: Writing for ACCU (BoF) I think I've been convinced to write an article about standardisation of multithreading in C++. C++ Threads, Lawrence Crowl Crowl gave an overview (similar to this) of the extensive work that's been done on standardisation of multithreading in the C++ language and library. This is one of the most important features in C++0x due to the increased parallelism mentioned above. The current semantics of multithreaded programs are not standardised, resulting in subtle but potentially serious differences between current C++ implementations that support multiple threads, and in the need for assembly-language code in programs that use lockless synchronisation. Much of this is also relevant to C and is being considered by the C standard committee. C++0x Initialisation: Lists and a Uniform Syntax, Bjarne Stroustrup Stroustrup recapped the 4 forms of syntax for initialising objects in C++, some inherited from C and some necessarily invented for C++, none of which are entirely generic. He noted the lack of a simple syntax to initialise the elements of containers, which violates the general principle that class types should have the same status as built-in types in C++. He then introduced a new initialiser syntax that he and others have defined for C++0x, which is a brace-enclosed comma-separated list of values, similar to that used for arrays and C-style structures (aggregates). Container types can define a constructor with a parameter of type const std::initializer_list<T> & and when constructed with this new form of initialiser they will receive this wrapper for a constant array of element values. For types that do not, the values in the list are treated as arguments to some other constructor, if one matches. Otherwise, for compatibility, this may be treated as aggregate initialisation. C++ Standard Library report, Alisdair Meredith There was far too much information here for me to even begin to summarise. Suffice to say there are lots of good things coming bu the library committee will have its work cut out to get them all properly specified in time for the Committe Draft coming late this year.
I also met old friends and new, put faces to names from newsgroups and books, and got too little sleep.

Adam Rosi-Kessel: Google Reader is the Solution

I was a long-time hold-out user of standalone RSS aggregators (SharpReader for Windows and Straw on GNU/Linux, both of which have been somewhat stagnant over the last couple of years). I’ve finally thrown in the towel and switched to Google Reader. It was a great decision. The key factor is mobility. I check email and blogs from my personalGoogle Mobile Image laptop; our kitchen computer; my office workstation; and, perhaps most significantly for Google Reader, from my cell phone. The Google Reader cell phone app is quite good (handy “keyboard” shortcuts, efficient presentation), and allows me to catch up during time that is difficult to use for other purposes: waiting for the train, standing in line at Starbucks, on the elevator. The only real downside is the lack of offline support (although I don’t think it would take a rocket scientist to create it), but this is far outweighed by the mobile functionality. As “access anywhere” becomes more important than “offline access,” I predict we’ll see more users of all applications make this sort of switch.

Mike Hommey: Init policy

There has been quite some talk about init systems on the planet, recently, but I saw noone talk about the policy stuff we have. It annoys me that, while we have an existing policy-based system to enable or disable services, yet all rc scripts I know of that disable themselves do it through a pref in /etc/default/service_name. It also annoys me there is near to zero documentation on how you are supposed to write your policy script, and that there is no bundled policy that would help getting rid of the NO_START or whatever variables in /etc/default/*.

Jacobo Tarr&#237;o Barreiro: Freestylin

Today I found out that I can ad-lib a 45 minute talk about l10n in Debian.

13 April 2007

Ingo Juergensmann: Should m68k drop the Debian distribution?

Debian Etch has been released lately, as you may know, and the work on Debian Lenny is already starting. For some of the DDs the work is way much different than for the rest of the DDs: the good ol' brave porters of m68k.
Stephen Marenka asked on debian-release what is needed to get back and be able to release with Lenny?
The answers are shaking:

1. To get back to testing: Basically, the release team didn't mind the
state we had with m68k in testing recently - basically, we just ignored
it, but it was still there. The release team didn't asked for full
dropping from testing, we just didn't mind either way. We also wouldn't
mind for m68k to be back in testing - as long as we can just continue to
ignore it.

2. To get released (or a realistic chance to): Basically, make sure that
the toolchain works (see the other mails), and get really fast buildds
(otherwise, the security team would probably veto the inclusion). The
second might happen with Coldfire, but - the how is your part, not ours
(i.e. if you do it some other way, that's ok as well of course).

Having said that, I can understand if the ftp-masters don't want to have
an architecture be in testing unless there is reason to believe it is
possible to include it in the next stable release, but that's not the
domain of the release team.


ad 1)
Ah, ok... as long as we are ignored, everything is fine. But how will testing m68k get updated when the testing scripts ignore m68k? Pardon me, but isn't this a little bit strange, at least?

ad 2)
Getting really fast buildds - good joke, indeed! Or is this a Friday, 13th habit like April Fools Day?

Conclusion)
Let's summarize: we can release with Lenny, when we get back in shape - but testing is needed for this, where we're ignored for. And without faster buildds for an arch such as m68k, there's simply no way back to make it into a release again - because of this.
Describing Coldfire as such a possibility for faster machine seems a little bit strange to me. CF is totally new, there's not progress yet on the CF front to my knowledge and a combined m68k/CF port may introduce other drawbacks, which nobody can predict for sure today. Do you really believe that ftp-masters or release team will allow such a port to get released? I don't.
Getting faster buildd in a reasonable amount of time is illusory for m68k. We can get more buildds online, but not "really fast buildds". Despite one older blog post of me, I'll receive just another '060 m68k soon (to be hosted by a fellow DD). A few days ago, two other m68k buildds were added to the m68k buildd rotation list. That will make 3 more buildds for the m68k.

Anyway, in most cases your first impression is the correct one and my first impression of the Vancouver Massacre/Proposal was: this is a plan to drop m68k!
Well, the lastest incidents confirm this impression, IMHO.
After all, I think that the m68k port would be best served by dropping Debian instead of being dropped by the project. The m68k port - de facto - has been dropped already. I'm disappointed.

Erich Schubert: DBus init changes

Just yesterday I mentioned that DBus contains an own set of init-like scripts. From todays new DBus package in unstable:
  * debian/dbus.init
    + Add functionality to start/stop dbus dependent services based on their
      LSB header.
      Instead of installing an init script into /etc/dbus-1/event.d, services
      depending on dbus should from now on install a regular sysv init script
      and add an LSB header with "Required-Start: dbus".
So the dbus init script now adds a 'workaround' (likely, since this is done in DBus, not in sysvinit) for the missing dependency handling in sysv-style init, by taking care of the restarting itself. This is good, because init scripts aren't distributed into two places anymore (soon).It probably highlights why we need to work on our init.

Sven Mueller: Init HackFest / New init systems in Debian

In his “Init followup” post, Erich posted a reference to his Debian-Wiki post about a proposed Debian HackFest after Etch was released (which would be about 5 days ago<smile />). Anyway, since that page at the wiki is protected, which means I can’t edit it, I’m posting my thoughts here. I agree with Erich and some other commentors at Gunnar Wolf’s blog post that it doesn’t make sense to add X init-script-only-packages for any daemon to support X init systems. Instead, each package containing one or more daemons should contain the meta information needed to create all init scripts. Basically, this would probably include script snippets which do the following operations: Additionally, some control information would be needed including, but probably not limited to: The script snippets listed above should allow basically any operation on a daemon, including the force-reload and try-restart commands not yet implemented by all current sysv-initscripts.

MJ Ray: Blog Tools 1: the Aggregator

I just rewrote my aggregator (schcyrssmerge2) which still works roughly on this algebra from 2004 (still the basic set theory rule of blog aggregate generation: aggregate_i+1 = (blogs/aggregate_i) u (aggregate_inblogs)) but now calculates agg1 a different way. First, there's agg0 - the items from the input RSS file - and I store only the post ids of agg0 as agg0i for speed. Previously, I was reading in all the blogs and doing the above set theory calculation in one go. With a couple of ill-behaved blogs (full-content feeds of a few hundred kb), mzscheme's memory usage was just getting silly, topping out at hundreds of megabytes to merge 4Mb of feeds into a 3Mb aggregate. I think some of the problems are PLT's - reading a 3Mb RSS file in with
"(define a (call-with-input-file "friends.rss" read-xml))"
seems to result in mzscheme (v352) holding 100Mb. What's with that? Anyway, now I've written a newitems function which extracts new items not listed in agg0i and old items which still appear in a blog and adds them to lists. Then, there's a function which tail-recurses over all the blog filenames, parsing the RSS, calling newitems on each one and building the lists of old and new items. Finally, agg1 is the new items appended to the old items. That is only calculated as it gets written out as a RSS file. I'll give it a few days of testing, then I'll publish a tarball. I use this to build my all.rss feed from my various source feeds, in case you didn't realise, as well as to run some planets for public and private use. Anyone got a suggestion how best to reduce the read-xml memory consumption? Upgrade? Switch dialect? I'd like some XML read/write functions and the SRFI list functions.

John Goerzen: And we're off!

Yesterday afternoon, we started our information meetings with employees about our Linux on the desktop project. We're underway on our migration.

But before I talk about that, I need to back up and describe what the project is.

We are converting approximately 80% of our 150 or so PC users to Linux desktops. They're Debian etch (4.0) running Gnome, Firefox (Iceweasel), Evolution, NFSv4, and SystemImager. Over the coming days and weeks, I'll be writing about why we're doing this, how we're making it happen, things we've run into along the way, and the technology behind it.

Today I'd like to start with a high-level overview of the reasons we started investigating this option.

It became apparent that Vista was going to be a problem for us. Most of our desktop PCs are not very old, but Vista meant a significant degradation in performance from the Windows XP Pro that most people were running. A performance dip so significant, in fact, that it would have created a significant negative impact on employee productivity.

We tend to buy PCs with Windows licenses from the vendor (Windows preinstalled). As such, we knew it wouldn't be long before XP-based machines would be hard to find. If we stuck with Windows, we'd be running a mixed-OS network -- which we knew from experience we did NOT want to do. The other option would be to replace all those old PCs. The direct costs of doing that, with the associated Vista and Office licenses, would have been more than $200,000.

So we started to look at other options -- changing the way we license Windows, sticking with XP for awhile, or switching away from Windows. This last option sounded the most promising.

I took a spare desktop-class machine, representative of the hardware most end users would have, and installed etch (then testing) on it. I spent a bit of time tweaking the desktop settings, making things as transparent to the user as possible. We liked what we saw and started pursuing it a bit more. We knew we had some Windows apps we couldn't discard, so we tested running them off a Windows terminal server with the Linux rdesktop client. That worked well -- and the appropriate Server 2003 licenses plus CALs would still be far cheaper than a mass migration to Vista.

To make a long story short, we are getting quite a few benefits out of all this. One of the most important is a single unified system image. Excepting a few files like /etc/fstab, every system gets a bit-for-bit identical installation from the server, updated using rsync. /home is mounted from the network using NFS (v4). So our users can sit down at any PC, log in, and have all their programs, settings, email, etc. available. A side benefit is that hardware problems become minor annoyances rather than major inconveniences; if your hard disk dies, we can just bring up a different PC. We had tried numerous times to make roaming profiles work in Windows, but never really achieved a reliable setup there -- perhaps because it seemed virtually impossible to assure that each Windows PC had the exact same set of software, in the exact same versions, installed.

More to come.

Erich Schubert: Init followup

Gunnar Wolf commented on my recent series of init script posts. Thanks for the feedback.The thing is that I've been doing a lot of Debian stuff these days (such as our GSoC coordination, and lots of stuff you saw in this blog). But I shouldn't... instead I should be trying to finally get my diploma thesis done.I had my exams last july. If I had been focused, I could have been finished with my degree last christmas. Now it will likely by this fall. :-( All that is remaining is my final thesis. It should be written in 6 months, but I still havn't submitted my topic yet. So if you are wondering what took me 8 months already: I investigated two topics, but they didn't work out. Then I started researching for my 'current' topic, and I have a vision for it, but I still havn't managed to convince my professor (which of course is largely due to me not being able to convey this idea, and that the topic is in information management and representation, where it's probaly often hard to find a solid theoretical part. I have examples I'd like to do, and I hope they are actually convincing, but I have no idea how to 'prove' something here; it's too much about social aspects and human perception I guess, and I'm not really qualified or capable to do solid research in that field.)(And as you can probably imagine, I'm pretty fed up with studying now. A year ago, I'd have considered to go for a PhD afterwards, but now I just want to get out, and work on my projects.)Anyway, I got distracted. Back to the init topic.I already posted this link: Init Hackfest, as well as a pointer to the initscripts-ng mailing list. I'd really like to get this topic on it's way for lenny. So please, join this effort. While I'd like to push it, I shouldn't; on contrary, I should suspend my Debian efforts.As for your (Gunnar) suggestion on how to handle multiple inits: what's wrong with just including multiple init scripts in one package? They are not particularly large and compress well. A single init script package might already be bigger than all the init scripts.The problem is more on how to write these init scripts; by the time we're done with this project, we might actually have hit 1000 init scripts. Yes; I didn't just make up this number: packages.debian.org has 890 hits for /etc/init.d; the init-like scripts in /etc/dbus-1/event.d not included. So if just want to add support to 5 alternate init systems, this means we would need to write 5000 new init scripts. Ouch. So we maybe should convert them to 1000 meta-init-scripts instead, and generate the 6000 real init scripts from them.Please join the discussion, and lets get this project going, despite having to face such scaringly high numbers...

Evan Prodromou: 22 Germinal CCXV

A lot of people have asked me about the Tim Wu piece in Slate about Wikitravel. I've commented already about the article in Wikitravel's Logbook: 8 April 2007. The upshot: Wu got a lot of things really wrong about Wikitravel (like saying we didn't have any hotel listings for Bangkok). But if our readers get things wrong, that means Wikitravellers need to do some more work to help them get things right. User errors are developer errors writ large; we can do better. But I think Tim Wu missed the main reason Wikitravel guides will always be better than proprietary guides: they are Free Cultural Works. Where I come from, that means a lot. I wrote to Tim about his article, and he responded really kindly. The thing I liked most is that when he got home, he updated Wikitravel with the information he picked up on his trip to Thailand. That is pretty cool. I'm not taking others' responses to the article too seriously. Wikitravel got a Webby Award nomination for Best Travel Site the same week. As Tom Wolfe said, "The lead dog is the one they always try to bite in the ass." And Oscar Wilde: "The only thing worse than being talked about is not being talked about." (Thanks to Hugh!) tags:

Off to San Juan We're getting prepared this morning to head out to Puerto Rico for the first annual World-wide Wikitravel Get-together. Our flight leaves at 2:20PM, so we've got kind of a leisurely morning to get ready. As usual, we're pretty tense: what are we forgetting? And I usually get hung up on unnecessary stuff to deal with. I should probably print out the Spanish phrasebook from Wikitravel, because my Spanish is pretty rusty. And I should download all the WT pages about Puerto Rico destinations to my laptop, just to know where we should fill in. And we can't find the charger for our camera's battery. And I want to bring a certain pair of cargo pants, but there's a button missing. Do I have time to sew on another? Of course, all these anxieties melt once we get out of the house and into a taxi to the airport. Once you can't do things, you stop worrying about what you should do. I think. The more I learn about Puerto Rico, the more fascinated I am. San Juan (Puerto Rico)'s Old Town is a UNESCO World Heritage site. The giant telescope at Arecibo is world-renowned as the SETI site. The islands of Vieques and Culebra are isolated and fun, and Fajardo and Rincon has world-class surfing. My friend Biella Coleman gave me a great list of travel tips for San Juan and Culebra, which have been a big help in our planning. OK, enough blogging: gotta get to work sewing on that button. (I've been stalling long enough for Maj to have found the camera charger under a baseball hat. Yay!) tags:

12 April 2007

Biella Coleman: On My Aunt Enid, Cars, and History

Those who know me even moderately well (as well as every doctor I have ever seen), knows that my real first name as “Enid.” That is the mysterious E. that sometimes precedes the Gabriella. My mom’s intention was to call me Gabriella, the first name of some famous Italian Cabaret singer, Gabriella Ferranti. But when Enid, my aunt on my fathers side, passed away a few months before I was born, I was given the name Enid Gabriella. We have always used Biella or Gabriella but I like that I have Enid in there and have fancied from time to time to start using the name Enid. By all accounts, Enid was an amazing and energetic woman, who reared her four kids with passion, was very open to friends and family, and who, despite living a solidly upper middle class Jewish life, was also involved in interesting political work(like helping Americans doge the draft by escaping to Canada). My mom held a special fondness for her because, well, honestly I think she was one of the only members of my dad’s family who she deeply loved. And Enid always went out of her way to show her care and concern, as when she immediately went to Caracas after my mothers first child unexpectedly died of a high fever at the age of five months. Today as I was writing away, I received an email from my father that he sent to me and a bunch of cousins and other siblings where he attached a 9 page document of remembrances and memories of Enid. My dad, though in no way as bad off as my mother (health wise), is no spring chicken. And I think as he fast approaches 80 years old, he is committing a lot to writing and thus, keeping his memory alive for us. In fact, this reminded me of a really beautiful quote I recently came across on memory:
“You have to begin to lose your memory, if only in bits and pieces, to realize that memory is what makes our lives. Life without memory is no life at all, just as an intelligence without the possibility of expression is not really an intelligence. Our memory is our coherence, our reason, our feeling, even our action. Without it, we are nothing.” Luis Bu uel
But what was a little odd (but nonetheless very endearing) about most of the account is that it was not organized by stories, or date, but by car models. Yes, by technology. My dad organized the stories about Enid by virtue of the cars they had, which struck me a little strange because it is not like he was a car buff or anything (growing up we always had the most unsexy cars, like Ford Tauruses). The cars he covers are the following: The Model A Ford Convertible
The 1941 Oldsmobile
The 1948 Buick Convertible
A Chevy Convertible Once you read his little tribute, it becomes immediately clear that his choice of talking about cars is also way for him to talk about history which is my dad’s great love (he can talk to you straight for 5 hours about some odd event in WW II and why he should have become a historian) as it is about Enid. Case in point here:
This was the first General Motors car with automatic drive. It was called Hydromatic and even though this car had a powerful 8-cylinder engine, this first automatic drive was very sluggish. Forget about 0 to 60 miles an hour in 7 seconds. This was in minutes. It was the car that Abe used to teach both Enid and I to drive. We both had to have additional lessons with a stick shift car, because the driving test was only given on stick shift cars. I learned to drive in 1944, when I was 16. Enid took lessons from Abe and learned to drive in 1945 when she was fifteen. This was against the law, but the law in the name of Tom, the policeman that covered our neighborhood, liked Enid and looked the other way.
For those who care to read more (it may only be amusing to me and those few people that are into American car history), I have included all the cars and here are the main cast of characters. Ruth = My Grandmother
Abe = Grandfather
David = Dad
Enid = My Aunt, David’s sister (more…)

Erich Schubert: System init - part #3

Today's post (previous parts) tries to show what benefits other init systems can offer.I'm picking 'minit' as an example; not because it's the ultimate solution or anything, but because I happen to know it quite well. I used to be the package maintainer for some time, and I've also used it on an embedded system (where it was a perfect match). However, it's not very distribution-friendly; it's great when you can invest some time into configuring it (such as when building an embedded system) but not when it should ideally work automatically out of the box. The Debian package is still there, but it's orphaned. Still it's well written, clean code; there shouldn't be any bigger bugs in there.The first thing to notice is that minit is tiny. It's statically linked, but only about 6.5 kb. Regular init is 86 kb plus some libraries (despite actually not doing much during init!). This makes it very well-suited for embedding.But now for the benefits of minit over regular init:1. Service monitoring. This is the biggest point: minit will keep an eye on the services and restart them when needed. (Another thing very useful in embedded situations, btw.)Minit doesn't use the start/stop paradigm of sysv/shell-style-init. Instead it has a "run" script for every service. If a service is flagged as 'respawn', it will automatically be restarted when the run script quits; therefore the applications should not background themselves. This offers some benefits, for example the init process will be notified by the kernel of a termination, and can immediately respawn the service; in contrast to other monitoring systems suchas monit it doesn't rely on 'busy polling'.2. Dependency handling. Minit handles startup dependencies. If a service lists another service as a dependency, it will bring that service up first.3. Parallel startup. Unless an ordering is given via dependencies, minit will start the services in parallel.4. Control apps. In minit, you don't start a service by running some random shell script. Instead you call "msvc -u servicename", which will signal minit to bring up the service (and it's dependencies). This would (there currently is no policy for minit) be especially useful in SELinux, but it also allows to e.g. give a user the permission of running "sudo msvc -u" without introducing a huge security hole, since this cannot be used to execute arbitrary scripts. "msvc" can also be used to check if a service is up. (Another benefit for embedded systems, since that removes the need for pid files, which then need a ram (writeable) filesystem)I'll also give an example for a minit service:A service is represented by a directory in /etc/minit which can contain various files. I'm going to look at getty/1 example; which will bring up a console login on the first terminal. There are three files: If the service would need it, there could also be a file called "depends" listing service dependencies.This allows minit to start the getty with no overhead. There is no shell script being executed or anything (note: sysvinit also does this for gettys, but it's not being used much for other services).Of course minit isn't perfect. I've already said that it isn't particularly distribution friendly. This is because it uses a lot of magic files (e.g. an empty file named 'respawn' to flag a service as needing to be respawned), and it's configuration files aren't easy to handle automatically. The often frowned-upon 'symlink farms' of sysv-shell-style init are much better here.Another thing that is missing from minit is an automatic rollback of dependencies. If we told minit to bring up service 'a', which lists dependencies 'b' and 'c', minit will bring these three up. If we tell it to shutdown 'a', it will however not shutdown 'b' or 'c'. And unfortunately, even if we shutdown 'b', it won't shutdown 'a'. If you want a more complex dependency handling (or things such as runlevels), you need to do this externally. (An example is my 'enitdir' app, it adds runlevel support to minit, and allows switching of runlevels by replacing a symlink; Services are added and removed from runlevels by adding a symlink to the service into the runlevels directory. By using dnotify, this also takes virtually no CPU time.)Minit also lacks a differentiation between a 'started' and a 'running' service. It has support for 'sync' services, such as e.g. mounting a filesystem. It will wait for this operation to complete before it starts services depending on it. However, there are services that will continue to run, while not being instantly available when the exec() was called. Sometimes it would be nice if a service could go into a 'started' state, then do some cleanup work, and then when actually running the real service signal minit that it's now 'running'.As a last malus, minit won't print progress messages to the console. A minit boot is pretty much silent, directing it's output to some logging processes. There are good reasons for this (since services are started in parallel!), but it makes debugging harder, and users might be wondering what their system is actually doing (though some users prefer to see a colored 'busy' bar)...I guess this concludes this series of blog posts. Maybe someone can follow up with some details on upstart (which seems to be the most promising init replacement)? I hope to have given an impression on where there is room for improvement in our current init to one or the other. Now we just need some people to actually work on a better support for other inits in Debian...

Carlos Villegas: Etch boots as fast as Woody...

With the Post-etch frenzy, I installed Debian 4.0 stable as soon as I could. I used the same system used a few months ago for the google Summer of Code. Under the same conditions, the system required 32 seconds until KDE started. Compared with the previous stable releases:
Nevertheless, Etch has significantly more process started at boot time and the CPU usage is more intensive --with still some room for improvement --. Follow this link for the bootcharts and more detailed information.

As I starting point, using dash instead of bash, the boot time is reduced to 30 seconds and it could be further improved but there is some "hald" process (maybe part of dbus) that doesn't seem to want to run in parallel. See bootcharts. Any ideas?

Russell Coker: presentations about SE Linux

I have just read the Presentation Zen blog post about PowerPoint.

One of the interesting suggestions was that it's not effective to present the same information twice, so you don't have notes covering what you say. Having a diagram that gives the same information is effective though because it gives a different way of analyzing the data. I looked at a couple of sets of slides that I have written and noticed that the ratio of text slides to diagram slides was 6:1 and 3:1 in favor of text, and that wasn't counting the first and last slides that have the title of the talk and a set of URLs respectively.

So it seems that I need more and better diagrams. I'll include most of the diagrams I use in my current SE Linux talks in this post with some ideas on how to improve them. I would appreciate any suggestions that may be offered (either through blog comments or email).

The above diagram shows how the SE Linux identity limits the roles that may be selected, and how the role limits the domains that may be entered. Therefore the identity controls what the user may do and in this example the identity "root" means that the user has little access to the machine (a Play Machine configuration). I think that the above is reasonably effective and have been using it for a few years. I have considered a more complex diagram with the "staff_r" role included as well and possibly including the way that "newrole" can be used to change between roles. So I could have the above as slide #1 about identities and roles with a more detailed diagram following to replace a page of text about role transition.

The above diagram shows the domain transitions used in a typical system boot and login process. It includes the names of the types and a summary of the relevant policy rules used to implement the transitions. I also have another diagram that I have used which is the same but without the file types and policy. In the past I have never used both in the one talk - just used one of the two and had text to describe the information content of the other. To make greater use of diagrams I could start with the simple diagram and then have the following slide have all the detail.

The above diagram simply displays the MCS security model with ellipses representing processes and rectangles representing files.

The above diagram shows a simplified version of the MMCS policy. With MMCS each process has a range with the low level representing the minimum category set of files to which it is permitted to write and the high level representing the files that it may read and write. So to write to a file with the "HR" category the process must have a low level that's no higher than "HR" and a high level that is equal or greater than "HR". The full set of combinations of two categories with low and high levels means 10 different levels of access for processes which makes for a complex diagram. I need something other than plain text for this but the above diagram is overly complex and a full set is even more so. Maybe a table with process contexts on one axis, file contexts on another and access granted being one of "R", "RW" or nothing?

I also have a MLS diagram in the same manner, but I now think it's too awful to put on my blog. Any suggestions on how to effectively design a diagram for MLS? For those of you who don't know how MLS works the basic concept is that every process has an "Effective Clearance" (AKA low level) which determines what it can write, it can't write to anything below that because it might have read data from a file at it's own level and it can't read from a level higher than it's own level. MLS also uses a high level for ranged processes and filesystem objects (but that's when it gets really complex).

This last one is what I consider my most effective diagram. It shows the benefits of SE Linux in confining daemons in a clear and effective manner. Any suggestions for improvement (apart from fixing the varying text size which is due to a bug in Dia) would be appreciated.

The above diagrams are all on my SE Linux talks page, along with the Dia files that were used to create them. They may be used freely for non-commercial purposes.

If anyone has some SE Linux diagrams that they would like to share then please let me know, either through a blog comment, email, or a blog post syndicated on Planet SE Linux.

11 April 2007

Kai Hendry: Webconverger 2.4

Korean Webconverger split 2.4 I’ve just released Webconverger (Web kiosk) USB&ISO versions built with stable Debian etch. Webconverger has received some good feedback and press. I would like to thank Daniel Bauman for his rather excellent Debian Live work on which Webconverger is based. I was hoping that Webconverger could support every locale, now it looks like I need to produce a separate ISO/IMG (or at least apt-get hooks) for every locale. One reason is that Iceweasel/Firefox extensions don’t scale very well. Try install every locale and dictionary extension. I dare you. :) Sidenote: I wonder why no one has packaged the dictionaries for Iceweasel. In other news, I’ve been playing around with scim (keyboard switcher) and Korean locales. It seems to work though the title bar didn’t render hangul characters right in dwm. After some investigation I discovered there isn’t such a thing as a single “generic” Unicode font. This came as a shock to me. :) Typical ttf only have 65536 glyphs, so no one font will do. Jeff Zhang also informed me about free Unicode fonts. Though I think especially in CJK environments proprietary fonts are unfortunately more common. Or I don’t know why microsoft-gulim isn’t in msttcorefonts. Anyway the Korean version of Webconverger turned out OK, though I doubt any Koreans will be using it. It’s all sadly ActiveX out there. My package wordpress 2.0.10 didn’t quite make it to etch in time as I was waiting for CVEs! 2.0.10 will be coming soon though… It always amazes me how much work can get done in breaks like Easter. It’s great that etch is out. And whoa, the weather is great.

Erich Schubert: System init - part #2

Yesterday, I wrote about our init actually being /bin/sh, since all sysvinit does it start a single shell script. Today, I'm going a bit into how services are brought up with sysv-style init. I won't be writing much about runlevels, since most users will not make use of multiple runlevels anyway. So what does a typical init script in /etc/init.d do? If you want to find out yourself, consider looking at /etc/init.d/skeleton, which is a template for init scripts. A modern init script should (as required by the LSB) start with a formatted comment with metadata such as dependencies and description. This is already trying to address some shortcomings of sysv-style init, but no actual code. The first thing most init scripts do is to check if their application is actually still installed. Init scripts are sometimes created by users, or are not automatically uninstalled since being treated as 'configuration files'; so it occurs often that the init script is still around even when the service was uninstalled. So they test if they're installed and executable. Next they probably read a configuration file, usually named /etc/default/packagename, with user settings (people used to do such changes in the init script itself, but for upgrading it proved to be a lot better to put all settings the user might want to modify there). The third step is to do the requested user action, which usually is one of 'start', 'stop', 'reload' and 'restart'. Some init scripts also have extra actions such as 'status'; not all init scripts support 'reload'. Doing such an action usually means to print out a status message and launching the service in the background (usually using the 'start-stop-daemon' daemon helper, which offers functionality to e.g. record the process PID in a file or forcing applications into the background that don't do so on their own), or locating the service (usually via a PID file, again this is commonly done using start-stop-daemon) and reloading or stopping it. Upon success, they again print a status message. An init script could be as simple as:
#!/bin/sh
if [ "$1" = "start" ]; then
  /usr/local/sbin/mycheapdaemon &
fi
There are many good reasons not to do it that way, but it will work. Some characteristics of sysv-style init scripts: Keep your eyes open for the next part of this series, where I'll show how some init systems improve on these issues.

Next.

Previous.