Over my time working with Debian packages, I've always been concerned that I have been missing catchable mistakes by not running all the static checking tools I could run. As a result, I've been interested in writing some code that automates this process, a place where I can push a package and come back a few hours later to check on the results. This is great, since it provides a slightly less scary interface to new packagers, and helps avoid thinking they've just been told off by a Developer. I've spent the time to actually write this code, and I've called it debuild.me. The code itself is in its fourth iteration, and is built up from a few core components. The client / server code (lucy and ethel) are quite interconnected, but firehose works great on it's own, and is a single, unified (and sane!) spec that is easy to hack with (or even on!). Hopefully, this means that our wrappers will be usable outside of debuild.me, which is a win for everyone. Backend Design The backend (lucy) was the first part I wanted to design. I made the decision (very early on) that everything was going to be 100% Python 3.3+. This lets me use some of the (frankly, sweet) tools in the stdlib. Since I've written this type of thing before (i've tried to write this tool many, many, many, many times before), so I had a rough sense of how I wanted to design the backend. Past iterations had suffered from and overly complex server half, so I decided to go ultra minimal with the design of debuild.me. <aside class="left"> You can find the code for the server (lucy) on my GitHub </aside> The backend watches a directory (using a simple inotify script) and processes .changes files as they come in. If the package is a source package, a set of jobs are triggered (such as lintian, build and desktop-file-validate), as well as a different set for binary packages (such as lintian, piuparts and adequite). Only people may upload source packages (without any debs) and only builders can upload binary packages (without source). The client and server talk using XML-RPC with BASIC HTTP auth. I'm going to (eventually) SSL secure the transport layer, but for now, this will work as a proof of concept. Since I tend to like to keep my codebase simple and straightforward, I've used MongoDB as Lucy's DB. This lets me move between documents in Mongo to Python objects without any trouble. In addition, I evaluated some of the queue code out there (ZMQ, etc), and they all seemed like overkill for my problem, and had a hard time keeping track of jobs that (must never!) get lost. As a result, I wrote my own (very simple) job queue in Mongo, which has no sense of scheduling (at all), but can do it's job (and do it well). Jobs describe what's to be built with a link to the package document that the job relates to, and it's arch and suite (don't worry about the rest just yet). Jobs get assigned via natural sort on it's UUID based _id, and assigned to the first builder that can process it's arch / suite. Source packages are considered arch:all / suite:unstable (so they always get the most up-to-date linters on any arch that comes along). Lucy also allows for uploads to be given an X-Lucy-Group tag to manage which set of packages they're a part of. This comes in handy for doing partial archive rebuilds, or eventually using it to manage what jobs should be run on which uploads. This will allow me to run much more time-consuming tools for packages I want to review versus rebuilding to ensure packages don't FTBFS or aren't adequite. Client Design The buildd client (ethel) talks with lucy via XML-RPC to get assigned new jobs, release old jobs, close finished jobs, and upload package report data. When the etheld requests a new job, it also passes along what suites it knows of, which arches it can build, as well as what types it can run (stuff like lintian, build or cppcheck.) Lucy then assigns the builder to that job (so that we don't allocate the same job twice), and what time it was assigned at. <aside class="right"> You can find the code for the client (ethel) on my GitHub </aside> Ethel then takes the result of the job (in the form of a firehose.model tree) and transmits it over the line back to the Lucy server as a report (which also contains information on if the build failed or not), at which point lucy hands back a location (on the lucy host) that the daemon can write the log to. If the job was a binary build, the etheld process will dput the package to the server, with a special X-Lucy-Job tag to signal which job that build relates to, so that future lint runs can fetch the deb files that the build produced. Tooling Ethel runs a set of static checkers on the source code, which are basically fancy wrappers around the tools we all know and love (like lintian, desktop-file-validate, or piuparts) which output Firehose in place of home-grown stdout. This allows us to programmatically deal with the output of these tools in a normal and consistent way. <aside class="left"> You can read more about Firehose over in the Firehose README.rst </aside> Some of the more complex runners are made of 3 parts - a runner, wrapper and command. The server invokes the command routine, which invokes the runner (the command just provides a unified interface to all the runners), who's output gets parsed by the wrapper to turn it into a Firehose model tree. The goal here is that tons of very quick-running tools get run over a distributed network, and machine-readable reports get filed in a central location to aid in reviewing packages. Ricky In addition to the actual code to run builds, I've worked on a few tools to aid with using debuild.me for my DD related life. I have some uncommon use-cases that are nice to support. One such use-case is the ability to rebuild packages from the archive (unmodified) to check that they rebuild OK against the target. This is handy for things like arch:all packages that get uploaded (since they never get rebuilt on the buildd machines, and FTBFSs are sadly common) or packages that have had a Build-Dependency change on them. Ricky is able to create a .dsc url to your friendly local mirror, and fetch that exact version of the package. Ricky can then also use the .dsc (in a monumental hack) to forge a package_version_source.changes file, and sign it with an autobuild key and upload it to the debuild.me instance. Since it can also modify the .changes's target distribution, you can also use this to test if a package will build on stable or testing, unmodified. Fred Fred is a wrapper around Ricky, to help with fetching packages that may not exist yet. Fred also contains an email scraper to read off such lists as debian-devel-changes, and add an entry to fetch that upload when it becomes available on the local mirror, pass it to ricky, and allow debuild.me to rebuild new packages that match a set of criteria. I'm currently playing around with the idea of rebuilding all incoming Python packages to ensure they don't FTBFS in a clean chroot. Loofah Loofah is also another wrapper around Ricky, but for use manually. Loofah is able to sync down the apt Sources list, and place it in Mongo for fast queries. This than allows me to manually run rebuilds on any Source package that fits a set of critera (written in the form of a Mongo query), which get pulled and uploaded by Ricky. An example script to rebuild any packages that Build-Depend on python3-all-dev in Debian unstable / main would look like: <aside class="right"> You can find more queries in the Loofah examples </aside> Or, a script to rebuild any package that depends on CDBS: You can use anything that exists in the Sources.gz file to query off of ( including Maintainer!) Future Work The future work on debuild.me will be centered around making it easier for buildd nodes to be added to the network, with more and more automation in that process (likely in the form of debs). I also want to add better control over the jobs, so that packages I upload only go to my personal servers. I'd also very much like to get better EC2 / Virtualization support integrated into the network, so that the buildd count grows with the queue size. This is a slightly hard problem that I'm keen to fix. I'm also considering moving the log parsing code out of the workers, so that the parsing code can be fixed without upgrading all the workers. This would also drop the Firehose dep on the client code, which would be nice. Migration from a debuild.me build into a local reprepro repo is something that would be fairly easy to do as well, likely to be done remotely via the XML-RPC interface, which calls a couple of reprepro commands (such as includedsc and includedeb) and publishes it to the user's repo. This is a nice use of the debs that get built, and could also allow debuild.me to be used like a PPA system, but this allows the user to not migrate packages that may contain piuparts issues.

I know, I'm starting to repeat myself but: this week was again characterized more be RC bug prevention than by actual RC bug fixing. in other words: I was mostly dealing with not-yet release-critical bugs around the perl 5.18 transition. actual RC bugs I've worked on (also mostly pkg-perl related):

• #666799 libapache2-reload-perl: "libapache2-reload-perl: sourceful transition towards Apache 2.4"
  upload to unstable after review (adaption to apache 2.4) (pkg-perl)
• #666800 libapache-singleton-perl: "libapache-singleton-perl: sourceful transition towards Apache 2.4"
  upload to unstable after review (adaption to apache 2.4) (pkg-perl)
• #666807 libapache-authenhook-perl: "libapache-authenhook-perl: sourceful transition towards Apache 2.4"
  upload to unstable after review (adaption to apache 2.4) (pkg-perl)
• #707528 libpackage-stash-perl: "libpackage-stash-perl: Package::Stash::* used only once; causes FTBFS in other packages"
  some investigation (pkg-perl)
• #710873 libapache2-mod-perl2: "libapache2-mod-perl2: FTBFS with newer libhttp-message-perl: t/api/err_headers_out.t failures"
  prepare a patch in git and upload after some tweaks and ntyni's help (pkg-perl)
• #711314 libmockito-java: "Missing jars in /usr/share/maven-repo"
  re-upload built against recent maven-*-helper packages to DELAYED/1
• #711419 src:libcgi-application-plugin-devpopup-perl: "libcgi-application-plugin-devpopup-perl: FTBFS: Got copyright stanza with unrecognised field"
  fix debian/copyright and debian/components/copyright.in (pkg-perl)
• #711631 src:libstring-toidentifier-en-perl: "libstring-toidentifier-en-perl: FTBFS with perl 5.18: test failure"
  upload new upstream release (pkg-perl)
• #711632 src:libtest-www-mechanize-mojo-perl: "libtest-www-mechanize-mojo-perl: FTBFS: pod coverage test failure"
  upload new upstream release (pkg-perl)

This blog post is the first of a series of posts dealing with the results of the Debian systemd survey. I intend to give a presentation at DebConf 2013, too, so you could either read my posts, or watch the talk, or both :-). The top concern shared by most people is:

systemd has too many dependencies, or it is bloated, or it does too many things, or is too complex

Now this concern actually has a lot of different facets, and I am trying to share my opinion on each of them. systemd has too many dependencies First, let s start with too many dependencies , because that is easy to check and reason about. I have created a document which lists all dependencies of the systemd binary itself (pid 1) and all the binaries which are currently shipped by the systemd Debian package. If you don t want to take my word for granted, please read that document. Have you read the document? Very nice! As you can see, the systemd binary itself has 10 dependencies (excluding libc). Now, the question is, what is bad about dependencies? Why do people list dependencies as a top concern?

1. Cyclic dependencies. When you hear that your init system depends on DBus, you might argue that there is a cyclic dependency here, because DBus needs to be started by the init system. However, systemd does NOT depend on dbus-daemon (!) to boot your machine. Instead of using the system bus, it uses a private UNIX socket. Therefore, systemd uses DBus merely as a serialization format for IPC between its different processes. Only when you want to access systemd via its API as a user (non-root), you actually use the system bus. Since we are talking about DBus: DBus provides a well-tested serialization format and IPC mechanism so that systemd doesn t have to reinvent the wheel and instead benefits from wide support within languages.
2. Complicated code. I feel like there is the implicit assumption that lots of dependencies correlate with complicated code that is easy to break. I encourage you to have a look at systemd s source code: look for the places where specific libraries are used, e.g. enforce_user which uses libcap. You ll notice that the code is not complex and usage of the libraries is clear.
3. Software dragging in lots of library packages. The libraries which systemd uses are already in widespread use (e.g. DBus, udev, selinux, libcap, pcre, ). On a typical Debian installation, only very few of them will be dragged in by systemd, if at all. As an example, on a fresh Debian Wheezy installation, less than 10 packages will end up on your machine when running apt-get install systemd .
4. More memory use. The Linux kernel maps libraries into memory only once, no matter how many processes use them. As stated in the dependency list, on machines where the libraries are not already loaded, systemd brings in about 500 KiB of additional memory-mapped libraries in the worst case. On the machines we have these days, this is a reasonable cost to pay for all the benefits systemd gets us. This holds true on embedded systems with only a few MiB of RAM and especially on typical workstations with 8+ GiB of RAM.

systemd is bloated Now, let s talk about bloat. Again, this is a point which has many facets. I d like to quote the Wikipedia definition of software bloat:

Software bloat is a process whereby successive versions of a computer program become perceptibly slower, use more memory or processing power, or have higher hardware requirements than the previous version whilst making only dubious user-perceptible improvements.

The first part of the definition certainly does not match systemd it is measurably faster than sysvinit. As for memory usage: systemd s RSS is 1.8 MiB, whereas sysvinit uses 0.8 MiB. As I argued on the More memory use point in the dependencies section, I think the additional resource cost is well worth the benefits. Also note that systemd s features are NOT all implemented in the binary which is PID 1. As explained in the dependency list, systemd consists of many cleanly separated binaries. So if a new version of systemd gathers an additional feature, this does not mean that your PID 1 will be bigger. While systemd runs on any hardware, it has an indirect hardware requirement: it requires some Linux kernel features (which are all enabled in Debian kernels). That might rule out usage of systemd on really old embedded hardware where you don t have a chance to update the kernel. While it is sad that those machines cannot profit from systemd, switching to systemd as a default has no downside either: Debian continues to support sysvinit for quite some time, so these machines will continue to work even with upcoming Debian versions. systemd does too many things The Wikipedia definition continues:

[ ] perceived bloat can occur from the software servicing a large, diverse marketplace with many differing requirements. Most end users will feel they only need some limited subset of the available functions and will regard the others as unnecessary bloat, even if people with different requirements do use them.

I think the last part of the Wikipedia definition applies to systemd: it does service a large and diverse marketplace . That marketplace is the entirety of existing software which is started by an init system. Also, systemd can be used on a wide range of hardware (embedded devices, tablets, phones, notebooks, desktops, servers) which requires different features. As an example: on a desktop system you typically don t care strongly about a watchdog feature, but on embedded or servers that feature is very handy. Similarly, on a tablet, forward secure sealing of logfiles is not as important as on a server. Therefore, I can understand if you feel that you don t need many of the features systemd provides. But please think of other users and maintainers who are very happy with systemd s benefits. Also note that while systemd supports many things (in separate binaries!), you don t have to use them all. It still makes sense to ship them all in the same package. Take coreutils as another example in that area. The binaries belong together, even though you most likely haven t used all of them (e.g. od, pr, ptx, :-)). systemd is too complex The remaining concern is that systemd is too complex. In my experience, complexity is often inherent to a specific area and one cannot simply make it go away. Instead, there are different models of how that complexity is represented. Think of the monolithic Linux kernel versus the MINIX microkernel. The latter has a very small amount of lines of source in the kernel, but puts the complexity into userspace. The former uses a different approach with more source in the kernel. The arguments between both camps show that neither is clearly right or clearly wrong. In a way, sysvinit represents the MINIX model: it has a small core (the init binary itself), but a lot of complexity in shell scripts and external programs. The fact that solutions are copied from one init script to another leads to lots of subtle errors and makes code reuse really hard. systemd however has more source code in the binaries, but requires only very simple, descriptive, textual configuration instead of complex init scripts. To me, it seems preferable to have the complexity in a single place instead of distributed across lots of people and projects. Conclusion In a way, you are right. systemd centralizes complexity from tons of init scripts into a single place. However, it therefore makes it very easy for maintainers to write service files (equivalent of an init script) and provides a consistent and reliable interface for service management. Furthermore, it is different than sysvinit, and different solutions often seem complex at first. While systemd consumes more resources than sysvinit, it uses them to make more information available about services; its finer-grained service management requires more state-keeping, but in turn offers you more control over your services.

I ll be spending the week after next (June 17-23) in London for the annual meeting of the International Communication Association where I ll be presenting a paper. This will be my first ICA and I m looking forward to connecting with many new colleagues in the discipline. If you re one of them, reading this, and would like to meet up in London, please let me know! Starting June 24th, I ll be in Ann Arbor, Michigan for four weeks of the ICPSR summer program in applied statistics at the Institute for Social Research. I have been wanting to sign up for some of their advanced methods classes for years and am planning to take the opportunity this summer before I start at UW. I ll be living with my friends and fellow Berkman Cooperation Group members Aaron Shaw and Dennis Tennen. I would love to make connections and meet people in both places so, if you would like to meet up, please get in contact.

I ve finally gotten around to listing all Waypoints (Geocaches, Opencaches, Closedcaches, Earthcaches, Terracaches including Locationless, Navicaches, etc.) I ve found a box, enjoyful, educating, a good place to hide one myself, etc. and putting up a list and, of course, generate my own statpic.

I ll put them up for the other project members, too (already made a picture for gecko2@ but bsiegert@ still needs one; we also need to collect offline lists of found, owned and attended waypoints) A bit of background story: I decided, years ago, to have an offline list of cache finds in case something would happen. Just, I had found way too many already, so this was a huge bit of work. Oh well I of course procrastinated, and then something did happen (Opencaching wanting to force a Restricted Commons licence; me disagreeing and suggesting a change; some trigger-happy person immediately deleting my account without waiting for the discussion or the decision period to end; weeks of forum discussions; Opencaching allowing dual-licencing; them telling me they can t restore my data probably never heard of databa sorry, MySQL backups). And I still didn t have the list. Now I do; recreated even the OC information from what was still accessible and with help from one OC supporter ( mic@ , thanks); merged caches that are co-listed on several platforms, etc. (still need to put in the FTF/STF/TTF/4TF/LTF and voting/favourites information) and a statpic, all in Open Source and Open Data, in cvs(1) with mksh(1) and a frontend for libgd2 I admit, but we had been using that for the MirWebsite for a while already. I suggest every geocacher keep an offline or local record of all their finds (and hides and attended logs) for things like this, in case some platform decides to let s say, put your data into the cloud where it is? I don t know .

Recently I wanted to cleanup my "main" remote machine. It is a system I've had for many years, which started off as a i386 KVM-guest and was later migrated in place to an AMD64 installation. These days the host runs:

• Mail for many domains, via QPSMTPD and ms-lite.
• Website hosting for many domains. Each site running under a dedicated per-UID thttpd instance, behind a node reverse proxy.
• IRC for kirsi.

The plan was originally to move the "mail stuff" to a new (wheezy) guest. I aborted that after discovering that the mutt-patched package has (IMHO) regressed. So today I spun up a new virtual machine, and configured it to host websites. Thus far I've migrated steve.org.uk, and lumail.org to the new host. Both simple sites that are built via my static-site-generator. Migration mostly involved configuring the proxy and the thttpd instances - then using rsync to migrate the content. I've renamed my current host, which was previously www.steve.org.uk and is now ssh.steve.org.uk, and pushed DNS changes. If all is smooth and happy I'll slowly migrate the rest of the sites. Fingers crossed this will be painless and I'll have a clean split between "login + mail" and "websites".

Yesterday I uploaded a big package and got multiple timeouts. I then figured out that DDs can also upload using SFTP (i.e. SSH s file transfer thingie) instead of traditional FTP, which seems like a more modern alternative. So let s give that a try. With dput-ng, the following configuration leads to using sftp by default:

mkdir -p ~/.dput.d/profiles/
cat > ~/.dput.d/profiles/ftp-master.json <<EOT
 
    "fqdn": "ssh.upload.debian.org",
    "incoming": "/srv/upload.debian.org/UploadQueue/",
    "method": "sftp"
 
EOT

Note that uploading via SFTP will lead to debianqueued uploading the files via FTP for you. But maybe that is more reliable than doing it yourself. We ll see :-).

Yes! It is possible! I ve done it! On my production linode! It was a bit hard, and I made quite a few mistakes, but I m convinced that a minimal system can be sanely crossgraded. I did it and only had an hour or so of server downtime In the process. However, most of the stack gets stripped down and back out, so I fear what happens with the X stack (nevermind GNOME Or anything that depends on it) A more detailed post to follow! Thanks to the dpkg team for making this happen!

Will that be enough? Let s load some VMs in that beast! :)

For email extensions such as SPF, DKIM and DMARC, I think the most flexible and portable system is the milter protocol. Originally developed for Sendmail, it is now also supported by Postfix, and it allows to plug specific filters in the mail server without the hassle of the previous systems like SMTP proxies.

SPF milters in Debian OpenDKIM provides a good milter for checking DKIM. OpenDMARC provides a similar milter for DMARC. But the situation is more difficult for SPF in Debian which is a requirement for DMARC! :

spf-milter

It was based on the buggy libspf0 and was never updated to libspf2, and was finally removed from Debian;

spf-milter-python

This one seems a bit Sendmail-centric it uses a dbm configuration file for instance and not very documented.

Packaging a new SPF milter? There are other milters for SPF, which have not been packaged for Debian yet, but I am ready to package one. Has anybody tried another SPF milter?

spfmilter

Still in beta test, still to be ported to libspf2 which may never happen since it was not updated since 2005.

smf-spf

Based on libspf2, seems rather neat, with a clean configuration, but inactive since 2007.

milter-spiff

Proprietary.

Today I was lucky enough to be able to visit the Telecom Italia telecommunications tower located in Rozzano, just south of Milano, and took some photos. This tower, with its 187 meters, is one of the tallest man-made structures in Italy. It was built by Telecom Italia in 1990 to create high capacity radio links to Genova and Torino and nowadays it contains radio transmitters for a TV station and many kinds of radio networks. It is an impressive monument to an age when telcos had no optical fibers, but plenty of money.

I ll be presenting Authoritative linked data descriptions of debian source packages using ADMS.SW at OSS 2013. Here s the abstract :

The Debian Package Tracking System is a Web dashboard for Debian contributors and advanced users. This central tool publishes the status of subsequent releases of source packages in the Debian distribution. It has been improved to generate RDF meta-data documenting the source packages, their releases and links to other packaging artifacts, using the ADMS.SW 1.0 model. This constitutes an authoritative source of machine-readable Debian facts and proposes a reference URI naming scheme for Linked Data resources about Debian packages. This should enable the interlinking of these Debian package descriptions with other ADMS.SW or DOAP descriptions of FLOSS projects available on the Semantic Web also using Linked Data principles. This will be particularly interesting for traceability with upstream projects whose releases are packaged in Debian, derivative distributions reusing Debian source packages, or with other FLOSS distributions.

If you are interested, a previous paper which describes the same experiment is available online. See also previous installments on ADMS.SW in this blog.

One of the replies to the post about Debian's last groupware meeting was from Patrick Ohly of syncevolution fame pointing out that syncevolution already implements calendar autodetection for CalDAV calendars as described in draft-daboo-srv-caldav-10. While looking at the code I noticed that there's a backend for the N900s calendar by Ove K ven as well. When I tried Ove's latest package on my N900 it lead to an immediate crash when doing a:

syncevolution --print-items target-config@webdav calendar

According to Patrick the bug was supposed to be fixed in recent versions so I set up scratchbox and built a newer git snapshot for maemo (sources). This wouldn't crash but didn't show up any items either. It turned out to be a minor bug in calypso returning no content type for REPORT queries which resulted in libneon discarding the whole reply (now already fixed in calypso upstream). With this out of the way setting up synchronisation is quiet simple:

# Configuration
CALDAV_SERVER=192.168.0.10
syncevolution --configure username=<username> password=<password> \
              calendar/backend=caldav calendar/database=https://$ CALDAV_SERVER :5233/private/my_calendar \
              target-config@webdav calendar
syncevolution --configure --template SyncEvolution_Client sync=none syncURL=local://@webdav username= password= webdav
syncevolution --configure sync=two-way backend=calendar webdav calendar

You should then be able to print the items on the local (N900) and from the remote (CalDAV server) end:

# This lists the current calendar items on the server
syncevolution --print-items target-config@webdav calendar
# This lists the current calendar items on the N900
syncevolution --print-items @default calendar

And from there on sync away:

# initial slow sync
syncevolution --sync slow webdav
# from there on
syncevolution webdav

The syncevolution source code has great documentation about debugging problems (e.g. src/backends/webdav/README). So check that in case you run into problems. The tl;dr version is

SYNCEVOLUTION_DEBUG=1 src/syncevolution loglevel=10 --print-items target-config@webdav calendar

to debug CalDAV related problems. In case you need to run syncevoluton from source be sure to set these beforehand:

export SYNCEVOLUTION_TEMPLATE_DIR=$PWD/src/templates/
export SYNCEVOLUTION_XML_CONFIG_DIR=$PWD/src/syncevo/configs/

On the CalDAV side I used current Calypso git which (with some additional minor fixes) now also interoperates nicely with Iceowl/Icowl-Extension aka Sunbird/Lightning on the desktop side. There's also an ITP for it. So it'll hopefully end up in Debian soon.

A Armadillo release 3.900.0 was provided by Conrad yesterday. It has been rolled into a new RcppArmadillo release 0.3.900.0 which is now on CRAN and in Debian. It has a number of nice changes, mostly on the performance side of things (see below) and a minor cleanup/improvement in our Kalman filter example (which, as I just realized, I forgot to note in the \code NEWS.Rd in the released tarball):

Changes in RcppArmadillo version 0.3.900 (2013-06-04)

• Upgraded to Armadillo release Version 3.900.0 (Bavarian Sunflower)
  • added automatic SSE2 vectorisation of elementary expressions (eg. matrix addition) when using GCC 4.7+ with -O3 optimisation
  • added support for saving & loading of cubes in HDF5 format, contributed by Szabolcs Horvat
  • faster median(), contributed by Ruslan Shestopalyuk
  • faster handling of compound expressions with transposes of submatrix rows
  • faster handling of compound expressions with transposes of complex vectors
• Kalman filter example switched from inline to \code sourceCpp , which simplifies / streamlines the C++ side a little

Courtesy of CRANberries, there is also a diffstat report for the most recent release As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

As part of writing a new mail client I'm wondering about how to change my email-life, and how other people process/handle their incoming email. I sort my incoming email into folders at delivery-time using procmail. Mail is generally filtered into mailboxes on the basis of the company that sent it, the person that sent it, or the machine which generated it. Because I manage a lot of machines personally I've split things up so that I have a folder per host. So on a morning I might have unread mail in the following folders:

machines.steve.org.uk/
machines.da-db1/
machines.da-db2/
machines.da-web1/
machines.da-web2/
machines.da-web3/
machines.da-web4/

The per-machine mailboxes usually contain a single mail every day from LogWatch, along with output from any cron-jobs. For example today I received the mail:

From: Cron Daemon
To: steve@steve.org.uk
Subject: Cron steve@steve.org.uk /home/steve/bin/download-check
URL http://nodejs.org/ - no longer matches v0.10.9

Generally speaking I don't need to read the per-machine messages. I'll keep the most recent 100 for reference, but only need to look if something seems "off" on a machine. But if I don't look I'd not see the node upgrade notice, so find that I do read them after all. This suggest to me that email isn't the right way to handle this kind of thing. Instead I should use a notification system - at work we have a central service called MauveAlert (yes, Red Dwarf reference). Mauve receives "alerts" of various kinds, via UDP. The alerts are then fanned out to appropriate people via XMPP, Email, or SMSs. I have a similarly-inspired system I use on my Debian Administration cluster. A (node) service runs non-stop collecting UDP messages and showing them on a dashboard. I look at it throughout the day to see when slaughter runs, etc. Anyway in conslusion I get a lot of mail. Some of it is related to random projects, and all ends up in the steve.org.uk/ mailbox, some of it relates to machines, and gets filed away, and I have regular conversions with folk so I have a .people.kirsi/ folder which receives a lot of attention, for example. ObRandom:daily() - Mark ~/Maildir/.machines.*, etc, read.

One of the various ways to measure an open source project is the way it encourages and accepts patches from community members (or future members). The various open source projects I m involved with have many ways to send patches: mailing list, bug tracking systems and more recently pre-commit tools for peer review (such as gerrit). Another popular option is to use github s pull request feature for this. Personally, I find it too complex for simple patches and would prefer gerrit over it. As part of my work at Zend, I try to make sure patches are sent back upstream. Comparing to other open source projects I know, it seems to me that PHP isn t open enough for contributions. This might be just my private case, but if not, it would be discouraging people to contribute and blocking the community from expanding. 6 Months ago I submitted patch to #36103 (patch is at #63767), but no response. A month ago I submitted a simple patch to FPM s shell script (#64764), and no response there as well. To ease things up, I ve created a github fork of the php-src repository and sent pull requests with each of the patches (as suggested at http://php.net/git-php.php). Even edited the bugs reports with the pull request details (as they provide such integration). I m waiting to see if this will trigger a more rapid review process from the project. For the simple patch, I got a response very fast from a user, but still waiting for a core member to check it. Going over the pull requests (both open and closed), I m not too optimistic about having the patches merged, judging by past merged patches (most of them are at least 2 months) and by the open patches (e.g. a typos fix patch is waiting for month).
Filed under: PHP