Search Results: "pm"

17 April 2012

Russell Coker: ZFS vs BTRFS on Cheap Dell Servers

I previously wrote about my first experiences with BTRFS [1]. Since then I ve been using BTRFS on more systems and have had good results. The main problem I want to address is with the reliability of RAID [2]. Requirements for a File Server Now one of my clients has a need for a new fileserver. They need to reliably store terabytes of data (currently 6TB and growing) which is mostly comprised of data files in the 10MB 15MB size range. The data files will almost never be re-written and I anticiapte that the main bottleneck will be the latency of NFS and other network file sharing protocols. I would hope that saturating a GigE network when sending 10MB data files from SATA disks via NFS, AFS, or SMB wouldn t be a technical challenge. It seems that BTRFS is the way of the future. But it s still rather new and the lack of RAID-5 and RAID-6 is a serious issue when you need to store 10TB with today s technology (that would be 8*3TB disks for RAID-10 vs 5*3TB disks for RAID-5). Also the case of two disks entirely failing in a short period of time requires RAID-6 (or RAID-Z2 as the ZFS variant of RAID-6 is known). With BTRFS at it s current stage of development it seems that to recover from two disks failing you need to have BTRFS on another RAID-6 (maybe Linux software RAID-6). But for filesystems based on concepts similar to ZFS and BTRFS you want to have the filesystem run the RAID so that if a block has a filesystem hash mismatch then the correct copy can be reconstructed from parity. ZFS seems to be a lot more complex than BTRFS. While having more features is a good thing (BTRFS seems to be missing some sysadmin friendly features at this stage) complexity means that I need to learn more and test more before going live. But it seems that the built in RAID-5 and RAID-6 is the killer issue. Servers start becoming a lot more expensive if you want more than 8 disks and even going past 6 disks is a significant price point. As 3TB disks are available an 8 disk RAID-6 gives something like 18TB usable space vs 12TB on a RAID-10 and a 6 disk RAID-6 gives about 12TB vs 9TB on a RAID-10. With RAID-10 (IE BTRFS) my client couldn t use a 6 disk server such as the Dell PowerEdge T410 for $1500 as 9TB of usable storage isn t adequate and the Dell PowerEdge T610 which can support 8 disks and costs $2100 would be barely adequate for the near future with only 12TB of usable storage. Dell does sell significantly larger servers such that any of my clients needs could be covered by RAID-10, but in addition to costing more there are issues of power use and noise. When comparing a T610 and a T410 with a full set of disks the price difference is $1000 (assuming $200 per disk) which is probably worth paying to delay any future need for upgrades. Buying Disks The problem with the PowerEdge T610 server is that it uses hot-swap disks and the biggest disks available are 2TB for $586.30! 2TB*8 in RAID-6 gives 12TB of usable space for $4690.40! This compares poorly to the PowerEdge T410 which supports non-hot-swap disks so I can buy 6*3TB disks for something less than $200 each and get 12TB of usable space for $1200. If I could get hot-swap trays for Dell disks at a reasonable price then the T610 would be worth considering. But as 12TB of storage should do for at least the next 18 months it seems that the T410 is clearly the better option. Does anyone know how to get cheap disk trays for Dell servers? Implementation In mailing list discussions some people suggest using Solaris or FreeBSD for a ZFS server. ZFS was designed for and implemented on Solaris, and FreeBSD was the first port. However Solaris and FreeBSD aren t commonly used systems so it s harder to find skilled people to work with them and there is less of a guarantee that the desired software will work. Among other things it s really convenient to be able to run software for embedded Linux i386 systems on the server. The first port of ZFS to Linux was based on FUSE [3]. This allows a clean separation of ZFS code from the Linux kernel code to avoid license issues but does have some performance problems. I don t think that I will have any performance issues on this server as the data files are reasonably large, are received via an ADSL link, and which require quite a bit of CPU time to process them when they are accessed. But ZFS-FUSE doesn t seem to be particularly popular. The ZFS On Linux project provides source for a ZFS kernel module which you can compile and load [4]. As the module isn t distributed with or statically linked to the kernel the license conflict of the CDDL ZFS code and the GPL Linux kernel code is apparently solved. I ve read some positive reports from people who use this so it will be my preferred option. Related posts:
  1. ECC RAM in a Cheap Machine In a comment on my previous post about ECC RAM...
  2. Starting with BTRFS Based on my investigation of RAID reliability [1] I have...
  3. Dell PowerEdge T105 Today I received a Dell PowerEDGE T105 for use by...

15 April 2012

Richard Hartmann: Choo choo two

Now that our departure is looming ahead of us, it's time to give a short update. (Turns out it's not so short, after all...) Preparation Visa Mongolian and Russian visas are done; we will apply for the Chinese ones next Monday. Travel All train tickets, hotels, hostels, and tents have been booked and both a horse-back tour through Mongolia and scuba diving in Lake Baikal have been pre-arranged. The Russian railway seems to do maintenance and development on the live website all the time which meant we were forced to book through a local travel agent. We paid a hefty extra charge for that privilege but still, that was less than half of what we would have paid if we had booked through a German agent. With the help of #china-lug on irc.freenode.net, we found accommodation in a traditional Siheyuan in an old Hutong. Hopefully, this will be a interesting as it sounds. Locals By the power of attending conferences and this blog, I now have local contacts in Moscow, Beijing, and Shanghai. Everyone who contacted me was extremely nice and helpful. I have been getting hints and tips galore; I doubt our preparations would be as neat as they are without this invaluable help. Thanks; you know who you are :) Packing list Apparently there's an interest in what we will be bringing; also, this will be a nice reminder for the future. Information and note-keeping Luggage Clothing Personal and computer security Fastening, storage, and tools Health Photography Minimal scuba gear Food Convenience Leisure That's it. Re-reading this list, I wonder how we managed to fit all of this into an 80-liter travel bag and a 15-liter backpack, but we did and with room to spare.

Petter Reinholdtsen: Debian Edu interview: Andreas Mundt

Behind Debian Edu and Skolelinux there are a lot of people doing the hard work of setting together all the pieces. This time I present to you Andreas Mundt, who have been part of the technical development team several years. He was also a key contributor in getting GOsa and Kerberos set up in the recently released Debian Edu Squeeze version. Who are you, and how do you spend your days? My name is Andreas Mundt, I grew up in south Germany. After studying Physics I spent several years at university doing research in Quantum Optics. After that I worked some years in an optics company. Finally I decided to turn over a new leaf in my life and started teaching 10 to 19 years old kids at school. I teach math, physics, information technology and science/technology. How did you get in contact with the Skolelinux/Debian Edu project? Already before I switched to teaching, I followed the Debian Edu project because of my interest in education and Debian. Within the qualification/training period for the teaching, I started contributing. What do you see as the advantages of Skolelinux/Debian Edu? The advantages of Debian Edu are the well known name, the out-of-the-box philosophy and of course the great free software of the Debian Project! What do you see as the disadvantages of Skolelinux/Debian Edu? As every coin has two sides, the out-of-the-box philosophy has its downside, too. In my opinion, it is hard to modify and tweak the setup, if you need or want that. Further more, it is not easily possible to upgrade the system to a new release. It takes much too long after a Debian release to prepare the -Edu release, perhaps because the number of developers working on the core of the code is rather small and often busy elsewhere. The Debian LAN project might fill the use case of a more flexible system. Which free software do you use daily? I am only using non-free software if I am forced to and run Debian on all my machines. For documents I prefer LaTeX and PGF/TikZ, then mutt and iceweasel for email respectively web browsing. At school I have Arduino and Fritzing in use for a micro controller project. Which strategy do you believe is the right one to use to get schools to use free software? One of the major problems is the vendor lock-in from top to bottom: Especially in combination with ignorant government employees and politicians, this works out great for the "market-leader". The school administration here in Baden-Wuerttemberg is occupied by that vendor. Documents have to be prepared in non-free, proprietary formats. Even free browsers do not work for the school administration. Publishers of school books provide software only for proprietary platforms. To change this, political work is very important. Parts of the political spectrum have become aware of the problem in the last years. However it takes quite some time and courageous politicians to 'free' the system. There is currently some discussion about "Open Data" and "Free/Open Standards". I am not sure if all the involved parties have a clue about the potential of these ideas, and probably only a fraction takes them seriously. However it might slowly make free software and the philosophy behind it more known and popular.

14 April 2012

Sandro Tosi: Mercurial: what's the tag containing this changeset?

In software development, you often need to know: when was this feature released? in which releases was this changeset included?

Typical example is: you're debugging a problem, you find out what's the changeset introducing it, now you want to know when that changeset was firstly released (i.e. included in a tag) - how to do that?

hg log -r "cset:: and tag()"

where cset is the changeset you're looking for.

So know I know that Sphinx new way to load searchindex.js was introduced in the this changeset, and released for the first time in 1.0b2:

$ hg log -r "423faa03c908:: and tag()"
changeset: 2261:b494009dccf1
tag: 1.0b2
user: Georg Brandl <georg@python.org>
date: Sun May 30 19:52:08 2010 +0200
summary: Release preparation for 1.0b2.
...

awesome!

PS: why did I need to know that? because Matplotlib is still using the old way, but Debian needs the new one.

Wouter Verhelst: LOAD: tutorial on Debian Packaging

Two weeks ago, I was at LOAD, where I did a tutorial on Debian Packaging. Unfortunately, I wasn't doing very well physically, and as a result my preparation wasn't what I had hoped for. It ended up being not so much a tutorial as a walking the audience through packaging a piece of software in casu NBD with debhelper in dh mode. Not very difficult, but since there's nothing for people to fall back on afterwards, it may help if they have a writedown of what I said during the session; and I promised to put it on my blog. So, here goes. However, note that the canonical written tutorial on Debian Packaging is here. With apologies to readers of Planet Debian, most of whom for which all this is probably old hat. Unlike RPM packaging, in Debian, the data that tells the debian packaging system how to package a piece of software is functionally split among a number of files. All these files go in a toplevel directory in the source package, (unsurprisingly) called 'debian'. There are four files that are required for every Debian package: the control, changelog, copyright, and rules files. Without these files, dpkg-dev will fail to produce a package any package. The first, the control file, contains metadata on packages: the name of the package, its description, the dependencies, etc. Basically, almost all the information you see when you run 'apt-cache show <package>' (with one exception) is in the control file. If you have a source package that builds multiple binary packages, then you should have multiple 'Package' stanzas in the control file. In my opinion, creating a control file is easiest done by copying a control file from another (similar?) package, and modifying it to suit the software you're dealing with. The one exception, the one bit of metadata that is not found in the control file is the version number: this data is contained in the changelog file. This file looks like a free-form file for the most part, but it really is a machine-parsable format; as such, it's best to edit it with specialized tools, such as the debian-changelog-mode in the emacs editor, or the debchange script, also available as dch, from the devscripts package. In the changelog, you should document any changes you make to the package, making sure the version number (top line), distribution (unstable, experimental, stable, ...; top line), urgency (top line; used mainly when testing needs to be updated urgently for things like security updates), author (bottom line) and date and time (also bottom line) are correct. The copyright file, unsurprisingly, should contain the copyrights statements of the original software, and the license (or a reference to a copy of the same license text under /usr/share/common-licenses). This file is still free-format in most packages today, although a machine-readable format for this file has recently been defined. Of the four required files, this is probably the most boring one, but hey, we can't like everything we do. The last file, the rules file, is where all the action happens. This file is defined as a Makefile, of which the targets are called for various parts of the build system. The rules file has a number of required targets (such as build, clean, and others), but if you're using debhelper in dh mode, you don't need to worry about those; instead, you can use the following simple rules file: 
#!/usr/bin/make -f
%:
	dh $@
With the first line, the shebang, we make clear that this is a makefile, and that it should be called by make. The next is a generic target (the % is a wildcard for make), with just one command: 'call dh with the name of the target being called'. Since dh implements all required rules targets, that immediately gets you a working package. Go ahead, try! Run 'dpkg-buildpackage -rfakeroot', and see what happens. Did that work? Maybe, maybe not. First, you'll have seen loads of warnings about something involving a "compatibility level". This is from debhelper; this compatibility level, or 'API level' if you wish, allows debhelper to move forward and make incompatible changes without breaking all the existing packages out there; whenever the compat level is bumped, some new functionality will only be made available if you also raise the compat level in your source package (and then you'll know you may have some changes to do in your package). Since the original debhelper, over a decade ago, did not have a compat level, the absense of a compat level signals compat level 1, which is far outdated now, and about to be unsupported. Hence the warning. The fix is simple: create a file debian/compat containing a single number: the compatibility level you're working with. It's best to use the most recent level which debhelper supports when you create your package, which as of this writing is level 9. Ignoring the compatibility level, if you're building a software package which uses a well-established build system, and you only build one binary package out of that, chances are pretty high that everything worked as expected. If not, you'll have more work. Building multiple packages requires that you tell debhelper somehow which file goes in which package. You can do this with a file debian/package.install, where package is the name of the binary package you're building. The install file is read by dh_install, one of many tools in the debhelper suite. You see, in the old days (before debhelper 7), debhelper was just a suite of tools, which required that you wrote a debian/rules file containing all the individual tools to be called in the correct order. This mode is still supported; in fact, even if you do use dh, you do, since all that tool really does is call the right tools in the right order; the real work is done by the invidual tools. They all have their own man page; to understand how dh_install chooses which file to put in which package, go read man dh_install. Go ahead, do it now; I'll wait. Back? Good. You've now learned that dh_install can install files either from the source directory (useful for packages containing only scripts or so), or from a directory debian/tmp. If you use an autotools-based software package, this is what dh will do for you. When you called dpkg-builpackage above, you may also have noticed that the output contained many lines starting with dh_, one of which said dh_install. As you may have guessed, dh echoes every debhelper command just before it will execute them. This allows you to look at the output, and see what's happening. For more detail, set the environment variable DH_VERBOSE to a non-zero value. I'm sure you'll have seen one or two debhelper tools that could make your package better. Go ahead, go and read their man pages to see what they do, and how they do it. Most of these will require you to create a file debian/package.toolname to specify details. In some cases, you may need to specify command-line arguments to the tool to get it to do what you want. In yet other cases, the tool won't support what you need it to do, and you'll have to do something manual. What to do now? The dh command line doesn't support adding extra commands easily. Does this mean you'd have to revert to old-style long, non-dh debian/rules files? Luckily, no. You can create an override target. If dh detects that you have such a target for a particular tool, it will call that target instead of the tool. This target's rules can then call the tool in question (or not), and add any command line arguments, or extra commands, as needed. An override target is a normal Makefile target with a name of the form override_dh_something, where dh_something is the name of the tool you wish to override. At this point, I'd reached almost the end of my two allotted hours, and a member of the audience asked how to make dpkg deal with configuration files. The answer is, you don't need to do anything! Not if you use debhelper, anyway; all you need to do is install the file in /etc. Since debian policy specifies that all files in /etc need to be configuration files and that no conffiles may be placed anywhere but in /etc, debhelper will automatically mark any file installed in /etc as a conffile, which will cause dpkg to ask the user what to do with changes to such files. Note that in Debian parlance, a conffile is not the same thing as a configuration file: a conffile is a configuration file that is part of the binary package (i.e., if you call 'dpkg --contents' on the .deb file, you'll see it in the output), whereas a configuration file is a file (any file, including files generated during or after installation of the .deb file on a system). Occasionally, this is also why some differences to configuration files are managed through debconf, while others aren't: the ones managed through debconf are non-conffile configuration files managed through ucf, whereas the others are conffiles. So in the simplest of cases, if you want to install a configuration file and you want to make sure it's protected against accidental overwrites by package upgrades, all you need to do is make sure it's installed to /etc; debhelper will do the rest. And that concludes this introduction. Please note it's only an introduction, not a full-blown tutorial; while this will allow you to get started, you may have to learn a bit more if you wish to eventually upload a package to Debian.

13 April 2012

David Welton: Mr. Blank, we're outside the building, and we want eBooks!

Steve Blank is known for his teachings on the Silicon Valley type of entrepreneurship, with his ideas forming the basis for the "lean startup movement" amongst other things. He writes frequently on entrepreneurship, and with a great deal of credibility, having been involved in various startups in a number of roles. He has, without a doubt, walked the walk in terms of startups, and now seems to be spending his time helping other people learn how to walk the same path. That's a noble thing to be doing when, with the money he's made, he could probably be off doing pretty much whatever he wants. If you've heard of Steve Blank, you've probably also heard his famous phrase: "get out of the building", an admonition to startup founders to get out and talk with their customers to validate their ideas, rather than huddling in their offices building something that may or may not have a market. With that in mind, when I saw he had a new book out, The Startup Owner's Manual, I thought "great, that's one I'll get without hesitating!". Unfortunately, though, an eBook wont' be out until "2nd half of 2012"! Ouch. To me, his ignoring eBooks is indicative of a need to get a bit further outside the building, though. "I want an eBook" was probably the biggest request on his blog post announcing the new book, along side messages of thanks for writing the book. After reading, on Blank's blog about the availability of the book from BookDepository Ltd, who offer free worldwide shipping, I went ahead and ordered it even if I would have prefered the eBook. Since they're in the UK, and I'm in Italy, I figure it can't take that long, right? Wrong. I ordered on March 15th, and as of April 13th, it still isn't here. Compare and contrast with the other books I'm currently reading which I was able to order and start looking at in just a few minutes on my Kindle. Granted, Steve Blank surely isn't doing this for the money, and from that point of view has little real need to listen to his customers - it's not wrong to say he's doing the world a favor by writing the book in the first place. If he thinks a paper version is far superior, that's his perogative. However, I think he's doing a lot of his readers a disservice by not making the eBook available sooner. I know I would have liked to start reading what he had to say last month, rather than waiting for a paper book to make its way (by mule train?) down here to Italy. The crux of the matter is that while he may well be right in thinking a paper book is "better", for some people, an eBook is the only option, and for them, an "inferior" eBook is a heck of a lot better than no book at all. Also, on a more constructive note, with eBooks, you can get pretty creative. For instance, if you have a tabular worksheet, you can simply hyperlink to it in, say, Google Docs, so that those with more advanced devices like iPads can open up the link and start working with a real, live spreadsheet immediately, rather than a chart in a printed book. Granted, that means 'giving away' the worksheet, but presumably it's not that valuable on its own, and makes for great advertising if it gets a lot of attention. Finally, since I actually run a business that does eBook conversions , on the blog post announcing the book, I offered to donate our services, so he'd get his book done for free, so you can't accuse me of just complaining! Mr. Blank, get out of that building and make an eBook available, please!

Peter Eisentraut: Enabling core files for PostgreSQL on Debian

The other day, I was a bit puzzled over a seemingly simple task: Enable core files to be generated from a PostgreSQL instance running on Debian. That instance has unfortunately been segfaulting on occasion, but never left a core file.

Now in principle it is clear that
ulimit -c unlimited
is the incantation to get this done. But where do you put this? You could hack it into the init script, but that seemed a bit ugly, and I wanted a sustainable solution.

A useful thing in the meantime is to check the current settings. That information is available in /proc/$PID/limits with the PID of the postmaster process (or any child process, really), and it looked like this to begin with:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            ms        
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
...
Use sudo grep core /proc/$(sudo cat /var/run/postgresql/8.4-main.pid)/limits if you want it automated.

So it's good to know that we only need to set the soft limit.

One way to configure this properly would appear to be in /etc/security/limits.conf. There you can add a line like
*               soft    core            unlimited
to enable core dumps globally. I'm not actually sure whether that would work if the service is started during the boot without PAM. In any case, I didn't want to enable core files globally; who knows what that would lead to.

One could replace the * by a user name, such as postgres, and then enable pam_limits.so in /etc/pam.d/su. But the postgresql init script in Debian is nested about four levels deep, so it wasn't clear whether it called su at all.

Now as it turns out, the init script ends up changing the user using this Perl code:
$) = $groups;
$( = $gid;
$> = $< = $uid;
(see change_ugid in /usr/share/postgresql-common/PgCommon.pm), so the whole PAM line of thought wasn't going to work anyway. (Other packages such as pgbouncer and slony1 do got through su, so that would be a solution for those.)

The way to solve this is the pg_ctl -c option, which sets the soft limit for core files to unlimited. And the way to pass this option through the init script maze is the file /etc/postgresql/8.4/main/pg_ctl.conf, which should contain a line like this:
pg_ctl_options = '-c'
Then restart postgresql, and check /proc/$PID/limits again:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            ms        
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes
OK.

Another thing that's recommendable in this context is to change the core file names to have a unique element, so that if multiple backends crash before you can take a look, they don't overwrite each other's core files. The core(7) man page explains the configuration options; I went with this sysctl setting:
kernel.core_pattern = core.%e.%p
which includes process name and PID. The PID file still ends up in the data directory of the PostgreSQL instance, which could also be changed, but I didn't find it necessary.

Stick the above line in /etc/sysctl.d/local.conf and reload with
service procps force-reload
I actually use a setting like that on all machines now; it's just nicer.

OK, and now I'll wait for the next core file. Or not.

Martin F. Krafft: Mouse on Mars

The atmosphere in Munich s Backstage Werk just before the opening act to the Mouse on Mars was very chilled. People sat on the stairs or scattered themselves over the dance floor while low-fi ambient tunes came from the speakers. It wasn t loud, you had to try hard to hear the people mumble. I have no idea who the opening act was, and their first tune was very nice and groovy. Then ensued a noise explosion, one could only pity the electronic equipment that was being asked to perform in ways that may be described as everything else than you expect , and of course, the base beat shook the building; I am quite sure they didn t use treble at all, but I may also simply have been unable to hear it. Plus, it seemed to us that the musicians catered for what may be a widespread decrease of attention span: it was noticable how they jumped from one thing to the next, not leaving them (or their listeners) any time to get in the groove. My brother and I went outside for a bit and talked about today s music and its simplicity. We postulated the repetitiveness as the basis of a mass movement, considered scene clubs that played heavy techno to an audience that is so entirely different to who historically frequented such musical performances, and in general tried to avoid assuming a position between simplifying society and accepting that individual freedom is as eclectic as can be. When MoM opened, they continued pretty much in line with their openers and half way through the first tune, I started to wonder how long I would last, or when it would be reasonable to step outside again. I had been a little afraid this would happen, having bought and listened to their latest album Parastrophics in preparation of the concert and not being able to get into it. However, what then followed blew us away. Still heavy, still all over the place, but now they were developing sound scenes, ripping them apart, having fun playing with and teasing the audience, while putting on a groove that inevitably made your muscles twitch with the beat. David Bowie called MoM the next big thing and I have to give it to them: MoM have always had a certain aura of that s what your music is like? we can improve on that! to them, and yesterday, they continued along those lines with astounding consistency, and it felt fresh. It also felt real. They weren t just pushing buttons and computers making music, they were making music and the computers were their instruments. Between the two founding members of MoM sat Dodo Nkishi, drummer and microphone artist, and if you don t believe, fast, big breakbeat can be performed live, well, you re wrong. Most everyone in the room was dancing. And while I was more swaying in awe, watching and wondering how the heck they are doing what they are doing, I couldn t contain the bouncing any longer. They came back for an encore and there was no more stopping the crowd, Thomas or me. Three tracks later, they waved goodbye and left, but a bunch of us simply continued to dance. Thomas questioned who would last longer and I started yelling loudly for another encore. The lights turned on, I considered it a slap in the face, but I did not stop yelling. Others tuned in. And then the lights went off and the band came back. Following their 2.5 hour show, gosh was I exhausted. It was a magnificent show. If you aren t afraid of big beat electronica and you take pleasure in nonstandard art, I heartily recommend you ensure that MoM aren t soon playing near you without you there. PS: MoM will play at the (D sseldorf Open-Source Festival)[http://www.open-source-festival.de/en/] on 30 June 2012! PPS: Now I listen to Parastrophics and I am really enjoying it. NP: Mouse on Mars: Parastrophics

12 April 2012

Michal &#268;iha&#345;: phpMyAdmin needs you

phpMyAdmin is very popular tool, but for whatever reason number of contributors is not that high as you would expect in such famous project. We're not looking only for developers, basically anybody can help! Working with phpMyAdmin will give you great opportunity to participate in one of most popular free software projects (we're constantly ranked in 20 most downloaded projects on SourceForge.net for many years). You can learn a lot and extend your knowledge by cooperating with other skilled people. And of course you can use this reference in your resume :-). So how you can help? You can help with translating, writing documentation, creating video tutorials, bug screening or coding. You can see all possibilities in more details on our website.

Filed under: English Phpmyadmin 3 comments Flattr this!

11 April 2012

Tim Retout: Weekend of change

Agile processes harness change for the customer's competitive advantage. Principles behind the Agile Manifesto
After two and a half years at Smoothwall, I'm moving on - Friday is my last day. Since I joined the development team, we have adopted Agile development, set up a pretty nifty Gerrit/Jenkins code review + integration system, and introduced dpkg for package management. Along the way, I helped with a bunch of important features for the business, like a ground-up rewrite of the web filter, and time-based browsing usage quotas. I will be starting at CV-Library on Monday, for a whole new set of challenges. They're based in Fleet, so I'll have an hour-long commute each way on the train. I've been assured by an expert in these matters that facilitating wage slavery is a comparatively more ethical pursuit than facilitating internet censorship. :) To make the commute somewhat more bearable, I'm moving house on Saturday. So far, the packing's going quite well...

Dirk Eddelbuettel: RcppArmadillo 0.3.0 is now on CRAN

Conrad Sanderson has released a new major version 3.0.0 of his excellent Armadillo C++ template library for linear algebra. A corresponding new release 0.3.0 of RcppArmadillo is now on CRAN. This follows four pre-releases of Armadillo which we packaged as before, but for reasons detailed in another post did not send to CRAN. All that said, here are the NEWS entries for all release since the last non-test release: 
0.3.0   2012-04-10
    o   Upgraded to Armadillo release 3.0.0 "Antarctic Chilli Ranch"
          * added non-contiguous submatrix views
          * added shorthand for inverse: .i()
          * added hist() and histc()
          * faster repmat()
          * faster handling of submatrix views with one row or column 
          * faster generation of random numbers
          * faster element access in fixed size matrices
          * better detection of vector expressions by sum(), cumsum(),
            prod(), min(), max(), mean(), median(), stddev(), var() 
          * expressions X=A.i()*B and X=inv(A)*B are automatically converted
            to X=solve(A,B) 
0.2.40  2012-04-04
    o   Upgraded to Armadillo release 2.99.4 "Antarctic Chilli Ranch (Beta 4)"
          * fixes for handling expressions with fixed size matrices
0.2.39  2012-04-02
    o   Upgraded to Armadillo release 2.99.3 "Antarctic Chilli Ranch (Beta 3)"
          * faster repmat()
          * workarounds for braindead compilers (eg. Visual Studio)
0.2.38  2012-03-28
    o   Upgraded to Armadillo release 2.99.2 "Antarctic Chilli Ranch (Beta 2)"
          * added .i()
          * much faster handling of .col() and .row()
          * expressions X=A.i()*B and X=inv(A)*B are automatically converted
            to X=solve(A,B) 
0.2.37  2012-03-19
    o   Upgraded to Armadillo release 2.99.1 "Antarctic Chilli Ranch (Beta 1)"
          * added non-contiguous submatrix views
          * added hist() and histc()
          * faster handling of submatrix views
          * faster generation of random numbers
          * faster element access in fixed size matrices
          * better detection of vector expressions by sum(), cumsum(),
            prod(), min(), max(), mean(), median(), stddev(), var()
Courtesy of CRANberries, there is also a diffstat report for 0.3.0 relative to 0.2.36 As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

10 April 2012

Michal &#268;iha&#345;: phpMyAdmin in GSoC 2012

As student application period of Google Summer of Code 2012 is over, it's time to look at proposals we got for phpMyAdmin. First of all we got slightly less proposals than in past years (38 this year, 52 in 2011 and 68 in 2010). This is probably related to the fact that we heavily focus on code cleanup this year and this is definitely not that interesting topic as adding new features. I think quality has also slightly improved, but still there were few completely bogus applications for totally unrelated projects. Also as usual, people tend to leave submission for very last date: Number of applications over time (and then complain, that they don't have enough time to submit patch as described in our applicant guide). Quite unsurprisingly the most interesting topic seemed to be frames removal, where we got few really good proposals. You can see how other topics were frequent in following chart: Number of applications for various types Please note that the numbers are not 100% accurate as some proposals really did not fit into above categories. Anyway we're just waiting for last people submitting patches today and evaluation will finalize in upcoming days. Of course you will know the results from Google on April 23rd.

Filed under: English Phpmyadmin 0 comments Flattr this!

8 April 2012

Richard Hartmann: Richuck

Q: How much wood would a Richuck chuck if a Richuck could chuck wood? A: Quite a bit. The few times in the past when I was able to get my hands onto a tree or three, I thoroughly enjoyed turning them into firewood. It's good workout, you get fresh air, and you can actually see what you accomplished with your hands. Back then, I had access to a hatchet, a felling axe and a single wedge. Splitting anything larger than 20 or 30 cm in diameter was tedious work, though still enjoyable. Two weeks ago, I was presented with several cubic meters of spruce; log diameters ranging in between 10 and 50 cm, pre-cut to two meters in length. Plus, access to a chainsaw. Yay. After some research, I ordered two helicoidal splitting wedges (they turn themselves when driven into wood), a splitting axe and a splitting maul. The rest of the equipment handed to me is probably older than I am, but still working perfectly fine. You have to admire the simple yet efficient design of the cant hook. Something I could not find an English name for is the German Sapie, in itself loosely defined as "any tool you can use to pick up, turn, drag or otherwise move logs". There's a surprising variety of these tools, but then, moving wood around has been a necessity of life for millennia so people are bound to come up with good designs. Last weekend, the chainsaw wasn't running very smoothly or much at all, resulting in hours spent coaxing it to cut at least enough for some hacking (the wood, not the computer kind). Other than our chainsaw troubles, that weekend was very pleasant and I ended up splitting tons of logs. We had the chainsaw overhauled and resharpened during the week and picked it up again on Friday, planning to resume cutting early Saturday, a plan that was spoiled by torrential downpour and hail. In the afternoon, I started splitting some left-overs around the house and used a short break from the rain to head out into the woods to where the logs are piled. The place I chose for splitting is somewhat exposed and even though the mix of rain, sleet, snow, hail, and everything in between resumed soon enough turns out you simply stop caring at some point. While temperatures were around freezing, I ended up being drenched in sweat in no time at all. There is something primordial and deeply satisfying about simply working away at these logs, ending with piles and piles of firewood. Of course, using my favourite new toy, 4.6 kg of forged stainless steel fuck you, increased said satisfaction immensely. Between the two wedges and that maul, splitting any middle-European conifers is a breeze. Add in the splitting axe for slim trunks and splitting will take less than 60 seconds per cut, no matter how many branch knots or pockets of shock-absorbing partially decayed wood you encounter. If only I've had those tools a decade and a half ago... While I am feeling muscles I forgot I had, I am planning to pursue this new-found old hobby for at least the foreseeable future. Especially since I sourced more logs and a few trees, already.

6 April 2012

Petter Reinholdtsen: Why the KDE menu is slow when /usr/ is NFS mounted - and a workaround

Recently I have spent time with Skolelinux Drift AS on speeding up a Debian Edu / Skolelinux Lenny installation using LTSP diskless workstations, and in the process I discovered something very surprising. The reason the KDE menu was responding slow when using it for the first time, was mostly due to the way KDE find application icons. I discovered that showing the Multimedia menu would cause more than 20 000 IP packages to be passed between the LTSP client and the NFS server. Most of these were NFS LOOKUP calls, resulting in a NFS3ERR_NOENT response. Because the ping times between the client and the server were in the range 2-20 ms, the menus would be very slow. Looking at the strace of kicker in Lenny (or plasma-desktop i Squeeze - same problem there), I see that the source of these NFS calls are access(2) system calls for non-existing files. KDE can do hundreds of access(2) calls to find one icon file. In my example, just finding the mplayer icon required around 230 access(2) calls. The KDE code seem to search for icons using a list of icon directories, and the list of possible directories is large. In (almost) each directory, it look for files ending in .png, .svgz, .svg and .xpm. The result is a very slow KDE menu when /usr/ is NFS mounted. Showing a single sub menu may result in thousands of NFS requests. I am not the first one to discover this. I found a KDE bug report from 2009 about this problem, and it is still unsolved. My solution to speed up the KDE menu was to create a package kde-icon-cache that upon installation will look at all .desktop files used to generate the KDE menu, find their icons, search the icon paths for the file that KDE will end up finding at run time, and copying the icon file to /var/lib/kde-icon-cache/. Finally, I add symlinks to these icon files in one of the first directories where KDE will look for them. This cut down the number of file accesses required to find one icon from several hundred to less than 5, and make the KDE menu almost instantaneous. I'm not quite sure where to make the package publicly available, so for now it is only available on request. The bug report mention that this do not only affect the KDE menu and icon handling, but also the login process. Not quite sure how to speed up that part without replacing NFS with for example NBD, and that is not really an option at the moment. If you got feedback on this issue, please let us know on debian-edu (at) lists.debian.org.

5 April 2012

Michal &#268;iha&#345;: Why weblate?

I still get lot of questions why I wrote Weblate instead of staying with some existing solution. Most of the reasons were already addressed in previous posts, where I looked for Pootle alternative or announced Weblate, but still it might be worth to summarize reasons once more. Git integration None of existing tools integrate with distribute version control, while I find this integration somewhat obvious and necessary. Having translation system as one of remote branches makes things really easy to handle for developers. Merging translations is nothing more than git merge. Attribution to translator I think it's fair to request that translator gets appropriate attribution for his contribution. Once we rely on Git, it is quite obvious that he will be author for Git commit doing changes. This way you can find author purely with Git and don't need to look for other tools. Handling of several branches In phpMyAdmin we heavily use branches to maintain stable releases. On the other side we want to be able to push translations to both stable and testing branches without translators having to do the work twice. Weblate does automatically push changes to all branches. Well this is not limited to branches but to subprojects as they don't have to be only branches, they can be also just related projects like with Gammu and Wammu. Displaying context Most of tools I tried fail to provide way to show related source code even when Gettext po files contain such references. With Weblate this is just matter of configuring URL of viewer, what nicely integrates with existing services like Github. Sane merging of uploaded translations With Pootle, it is possible to upload incomplete po file and it will luckily take it and replace existing one. This is quite dangerous as you would not notice it is incomplete unless you do some translation validation before release. Weblate does merge uploaded translations unit by unit, what makes it slower, but generates minimal diff and works also across branches. Consistency checks There are always things which can go easily wrong in the translation like format strings and translation tool should catch these as early as possible. Weblate semi-enforces checks - if check has failed, you will be redirected back to previous translation to possibly correct it (or mark check for ignoring). At current state checks are not as good as I would like to, but they are completely modular and can be easily extended and improved. Bonus points While working on Weblate I realized that some things could be done even better than I originally anticipated. For example with Github web hooks, it is possible to receive notifications on upstream repository updates and merge changes as they appear, leaving to really live translation process. Git allows to write custom merge drivers and this can be utilized for less merge conflicts on po files (they are usually caused by timestamps in headers), so that's another handy you get with Weblate. Of course there is much more built in, just check the documentation.

Filed under: English Weblate 3 comments Flattr this!

Dominique Dumont: My activities during Perl/QA hackathon in Paris

Hello Last week-end, I ve participated Perl/QA hackathon in Paris. Here s a short summary of the main activities during this week-end. I was not in the best of shape due to a nasty (biological) bug that wore me down before the hackathon. First of all, I discussed with rjbs about the bunch of debian patches I ve written for Software::License. These patches (and hacks) are used in Debian to provide licenses summaries in Debian::Dpkg::Copyright model to help write Debian copyright files. License management is boring but necessary. cme and Software::License help in making this task slightly less boring. Ricardo kindly suggested some modifications and agreed to include my patches. I implemented these modifications directly during the hackathon. The other tasks during this week-end were (not necessarily in that order): Many thanks to FreeSide who sponsored my travel, to the hackathons organisers and to all the participants for their welcome. All the best

3 April 2012

Michal &#268;iha&#345;: Weblate 0.8

After quite big delay, Weblate 0.8 is finally ready. There are many fixes and improvements like grouping of Git commits, improved checks or activity logging. All in all this should be feature complete for 1.0 release. Now it's time to stabilize all the features and release 0.9 (AKA 1.0rc) and then we'll be ready to fire 1.0. Full list of changes for 0.8: You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others. PS: If you're about to use Weblate, rather use Git version than tagged 0.8, you will get bunch of bug fixes for free :-).

Filed under: English Weblate 0 comments Flattr this!

2 April 2012

Michal &#268;iha&#345;: phpMyAdmin translations status

phpMyAdmin 3.5 should be released this weekend, so let's look at translations status for last time before release. Here comes seventh round of translation status update. Since last update we have one more translation and one less (due to some not translated strings in it) summing up to 11 translations at 100%: There are few which are really close to 100% and I hope they will get there till release: As you can see, there is still lot of languages missing, this might be your opportunity to contribute to phpMyAdmin. Also you are welcome to translate phpMyAdmin 3.5 using translation server.

Filed under: English Phpmyadmin 0 comments Flattr this!

1 April 2012

Kai Wasserb ch: kvpm in Debian now

Just a short announcement: kvpm (short for "KDE volume and partition manager") is now a part of Debian's unstable distribution, Sid, and should enter testing (Wheezy) soon. The big thank you for this goes to Benjamin, upstream author of kvpm who has become also the Debian Maintainer of his software! The story behind packaging kvpm started with Benjamin writing an e-mail to pkg-kde-extras, in which he asked for inclusion of kvpm into Debian. After an initial answer by myself about filing a RFP bug, in case he wouldn't like to package kvpm himself, a discussion in private ensued. There I explained that Debian is a do-ocracy, where things get packaged if some maintainer uses something herself or if the requesting party does the work. Benjamin took up the challenge and packaged kvpm, I helped only with feedback and finally sponsoring the upload. Why do I tell this story? Because I hope it encourages other people to do the same as Benjamin did: package the software they'd like to see become a part of Debian. We, the Debian Developers around the world, are more than happy to assist you. Just ask us (e.g. by e-mail or through IRC (#debian-mentors)).

Russ Allbery: Review: Agile Project Management with Scrum

Review: Agile Project Management with Scrum, by Ken Schwaber
Publisher: Microsoft
Copyright: 2004
ISBN: 0-7356-1993-X
Format: Trade paperback
Pages: 153
Ken Schwaber is one of the two original developers of Scrum and one of the signatories to the Agile Manifesto. Agile Project Management with Scrum is an introduction to Scrum by him that feels aimed primarily at business people (project managers, executives, line managers) rather than at technical people. There's very little about agile software development in general; the emphasis is on business process, team organization, and how the Scrum process looks to the rest of the organization. After an introductory chapter laying out the basics of Scrum, the remainder of the book is primarily case studies. Schwaber presents some principle and then discuss an anonymized case study that illustrates the principle. Each major Scrum role (ScrumMaster, Product Owner, and Team Member) gets their own chapter, interspersed with chapters on planning, reporting, and scaling. Finally, there are some concluding appendices laying out the rules of Scrum in more detail, adding a brief (and very disappointing) discussion of fixed-term, fixed-price contracts, and providing some other supplemental information. This is only the second regular book that I've read about Scrum, but my reading has been a bit out of order. We've been using Scrum for various projects for about a year and a half, I've played both team member and product owner roles, and we've been thinking hard about how to integrate Scrum more into the way we do development. My impression of this book is that it's intended as more of an introduction for people who aren't yet comfortable with Scrum or who are still struggling to make the pieces fit together. I'm therefore possibly not the target audience. Much of this book is a rehash of things someone who has studied Scrum will already know. I'm of two minds about the case study presentation style of this book. On one hand, it does help to make the points Schwaber discusses more concrete, and it's difficult to talk about business process in any other way. They're also more interesting and engaging than Schwaber's straight rule descriptions. But I had difficulty digging into most of the ones here. They're very simple and a little shallow, and I had trouble applying many of them to practical situations I might be in. I would have liked somewhat more detailed stories with more analysis and possible alternatives within the situation, rather than simple illustrations of how Scrum solves the problem that Schwaber had just said it would solve. What I did like about this book is that it provides the background and theory of Scrum as empirical process control much more clearly than any material I'd previously read. Schwaber is good at explaining what empirical process control is and why it's a superior approach, and he gave me new vocabulary and basic concepts to use to talk about Scrum. I also liked the high-level look from a management perspective without getting into the details of story points and velocity estimation (Schwaber doesn't use them at all in this book, relying instead on traditional "days of work" estimates). Much of my previous experience with Scrum was tightly focused on its advantages for estimation and responsiveness to changes, and it's nice to see some of the other advantages of Scrum (developer job satisfaction, improved focus, incremental deployment) highlighted more. Another part of this book I appreciated was the discussion of sprint length. Schwaber doesn't go into the possible alternatives, largely assuming that a month is the best interval, but the discussion of the impact and implications of a month sprint length helped me see the advantages of longer sprints. (To date, we've been doing either one week or two week sprints.) I came away with the feeling that a hybrid model where longer sprints are used for business purposes and then subdivided into shorter sprints for developer coordination might be particularly effective, although that's not an idea that Schwaber presents. This is not a particularly compelling read, particularly if, like me, you're not a huge fan of genericized case studies. Schwaber's writing is clear, but wooden and occasionally awkward, and the book is often dry. But it talks about Scrum from a management angle instead of a developer angle, which is an additional perspective that I found valuable. Possibly worth taking a look if you want more detail about Scrum from a high-level management perspective, although I don't think you're missing a lot by not reading it. Rating: 6 out of 10

Next.

Previous.