Talking about Secure Boot again, I'm afraid. One of the things that's made discussion of this difficult is that, while the specification isn't overly complicated, some of the outcomes aren't obvious at all until you spend a long time thinking about it. So here's some clarification on a few points.
Secure Boot provides no additional securityUntrue. Attacks against the pre-boot environment are real and increasingly common - this is a recent proof of concept, and this has been seen in the wild. Once something has got into the MBR, all bets are off. It can modify your bootloader or kernel, inserting its own code to return valid results whenever any kind of malware checker scans for it. The only way to reliably identify it is to either move the disk to another system or to cold boot off verified media. By restricting pre-OS code to something that's been signed, Secure Boot does provide enhanced security.
Secure Boot is just another name for Trusted BootUntrue. Trusted Boot requires the ability to measure the entire boot process, which gives the OS the ability to figure out everything that's been run before OS startup. The root of trust is in the hardware and a TPM is required. Secure Boot is simply a way to limit the applications that are run before the OS. Once booted, there is no way for the OS to identify what was previously booted, or even if the system was booted securely.
Microsoft are just requiring that vendors implement the specificationUntrue. Quoting from the Windows 8 Hardware Certification Requirements:

MANDATORY: No in-line mechanism is provided whereby a user can bypass Secure Boot failures and boot anyway Signature verification override during boot when Secure Boot is enabled is not allowed. A physically present user override is not permitted for UEFI images that fail signature verification during boot. If a user wants to boot an image that does not pass signature verification, they must explicitly disable Secure Boot on the target system.

Section 27.7.3.3 of version 2.3.1A of the UEFI spec explicitly permits implementations to provide a physically present user override. Whether this is a good thing or not is obviously open to argument, but the fact remains that Microsoft forbid behaviour that the specification permits.
Secure Boot can be used to implement DRMUntrue. The argument here is that Secure Boot can be used to restrict the software that a machine can run, and so can limit a system to running code that implements effective copy protection mechanisms. This isn't the case. For that to be true, there would need to be a mechanism for the OS to identify which code had been run in the pre-OS environment. There isn't. The only communication channel between the firmware and the OS is via a single UEFI variable called "SecureBoot", which may be either "1" or "0". Additionally, the firmware may provide a table to the OS containing a list of UEFI executables that failed to authenticate. It is not required to provide any information about the executables that authenticated correctly.

In both these cases, the OS is required to trust the firmware. If the firmware has been compromised in any way (such as the user turning off Secure Boot), the data provided by the firmware can be trivially modified and the OS told that everything is fine. As long as machines exist where users are permitted to disable Secure Boot, Secure Boot is not any kind of DRM scheme.
Secure Boot provides physical securityUntrue. Secure Boot does not in any way claim to improve security against attackers who have physical access, for the same reasons as the DRM case. The OS has no way to determine that the firmware's behaviour has been modified. A physically-present attacker can simply disable Secure Boot and install a piece of malware that lies to the OS about platform security. The "Evil Maid" attack still exists.
Secure Boot only defines the interaction between the firmware and the bootloader. It doesn't specify any higher policyMisleading. It's true that Secure Boot only specifies the authentication of pre-OS code. However, if you implement Secure Boot it's because you want to ensure that only authenticated code has run before your OS. If there is any way for unauthenticated code to touch the hardware before your OS starts, you can't ensure that. If an authenticated Linux kernel is booted and then loads an unsigned driver, that unsigned driver can fabricate a fake UEFI environment and then launch Windows from it. Windows would falsely believe that it has booted securely. If that authenticated Linux kernel were widely distributed, attackers could use it as an attack vector for Windows. The logical response from Microsoft would be to blacklist that kernel.

The inevitable outcome of implementing Secure Boot is that every component that can touch hardware must be signed. Anyone who implements Secure Boot without requiring that is adding no additional security whatsoever.
Only machines that want to boot Windows need to carry Microsoft's keysAgain, misleading. Microsoft only require one signing key to be installed, and the Windows bootloader will be signed with a key that chains back to this one. However, the bootloader is not the only component that must be signed. Any drivers that are carried on ROMs on plug-in cards must also be signed. One approach here would have been for all hardware vendors to have their own keys. This would have been unscalable - any shipped machine would have to carry keys for every vendor who produces PCI cards. If a machine carried an nvidia key but not an AMD one, swapping a geforce for a radeon would have resulted in the firmware graphics driver failing to load. Instead, Microsoft are providing a signing service. Vendors will be able to sign up for WHQL membership and have their UEFI drivers signed by Microsoft.

This leads to the problem. The Authenticode format used for signing UEFI objects only allows for a single signature. If a driver is signed by Microsoft, it can't be signed by anybody else. Therefore, if a system vendor wants to support off-the-shelf PCI devices with Microsoft-signed drivers, the system must carry Microsoft's key. If the same key is used as the root of trust for the driver signing and for the bootloader signing, that also means that the system will boot Windows.
This is only a problem for clients, not serversNot strictly true. While Microsoft's current requirements don't mandate the presence of Secure Boot on server hardware, if present it must be enabled and locked down as it is for clients. It's not valid for servers to ship with disabled Secure Boot support, or for it to be shipped in a mode allowing users to make automated policy modifications at OS install time.
Secure Boot is required by NISTThis is one that I've heard from multiple people working on Secure Boot. It's amazingly untrue. The document they're referring to is NIST SP800-147, which is a document that describes guidelines for firmware security - that is, what has to be done to ensure that the firmware itself is secure. This includes making sure that the OS can't overwrite the firmware and that firmware updates must be signed. It says absolutely nothing about the firmware only running signed software. Secure Boot depends upon the firmware being trusted, so these guidelines are effectively a required part of Secure Boot. But Secure Boot is not within the scope of SP800-147 at all.
It's easy for Linux to implement Secure BootMisleading. From a technical perspective, sure. From a practical perspective, not at all. I wrote about the details here.
It's only a problem for hobbyist Linux, not the real Linux marketUntrue. It's unclear whether even the significant Linux vendors can implement Secure Boot in a way that meets the needs of their customers and still allows them to boot on commodity hardware. A naive implementation removes many of the benefits of Linux for enterprise customers, such as the ability to use local modifications to micro-optimise systems for specific workloads. One of the key selling points of Linux is the ability to make use of local expertise when adapting the product for your needs. Secure Boot makes that more difficult.

ConclusionMuch reporting on the issues surrounding Secure Boot so far has been inaccurate, leading to misunderstandings about the (genuine) benefits and the (genuine) drawbacks. Any useful discussion must be based around an accurate understanding of the specification rather than statements from analysts with no real understanding of the Linux market or security.

comments

but I ve already forseen the unwanted meet for some time . My elder daughter does need to run educational software from school. And my wife wants to run education software for our younger son. And making all that running under different emulation environments does eat my time, and result is not always good. And also children start to want games Issue 1. Need hardware to run more or less modern windows on. For last 6 years, my home network consists of a more or less powerful headless server, and 3 terminals to access that. Terminals are definitely too weak for windows. And server is needed in 24/7 mode so sorry no windows there. Likely solution for this is one of 3 terminals is actually dying (it is doing so already for months but still works, just hangs sometimes) and needs a replacement, so I can just replace it now and use more powerful hardware. Issue 2. Windows host or windows VM? While windows VM feels preferable for a such a deep-linux person as myself, I tend to think that dual-boot host is better here. If the very reason of allowing windows into this house is make it easy to install and run random windows software, then doing so in VM environment is explicitly against the goal. Issue 3. Use unlicensed windows or buy a license? Although absolute majority of windows users here is Russia use unlicensed windows, it is somewhat unfair to promote free software and use unlicensed windows at the same time. And windows home basic license is not that expensive after all. The setup that makes me less uncomfortable than others is:
- obtain powerful enough hardware, and install that instead of failing terminal,
- obtain windows 7 home basic license
- install dual boot linux + windows, and under linux install windows VM
- when needed, try windows VM first, and in case of any issues, immediately reboot into windows host. What I can t find out for some time already is does microsoft retail license allow to have both host and VM installation on the same computer? These will be physically different installations, but on the same computer, and will never run at the same time. License wording does not give exact answer on this. I wanted to ask on Microsoft s support forum but failed to register there. After asking me quite a few things it claimed that there is some temporary problem and suggested to retry later. I retried later and got the same. I then asked here but looks like nobody knows. At least I want to know, what will happen if I will try to activate windows in VM after I ve activated windows (with the same key) on host. Will this render both installations non-genuine ? Also I d appreciate any comments on the whole situation

With dpkg from experimental and the new zlib upload in unstable, I've now got a partial Multi-Arch install. There are more packages necessary, particularly related to how -dev packages can exist and how a cross compiler gets built/installed in Multi-Arch world. One bug #659588 in libglib2.0-0 so far but that's quite good seeing as it's been all but impossible to test the Multi-Arch changes in existing packages until now.

$ dpkg --print-foreign-architectures

armel

i386

$ dpkg -l   cut -c -80   grep armel grep -v cross

ii  gcc-4.6-base:armel                   4.6.2-14                     

ii  libc6:armel                          2.13-26                    

ii  libdatrie1:armel                     0.2.5-3                

ii  libffi5:armel                        3.0.10-3                 

ii  libgcc1:armel                        1:4.6.2-14                 

ii  libgmp10:armel                       2:5.0.4+dfsg-1         

ii  libgmpxx4ldbl:armel                  2:5.0.4+dfsg-1        

ii  libgomp1:armel                       4.6.2-14         

ii  libmpc-dev:armel                     0.9-4              

ii  libmpc2:armel                        0.9-4          

ii  libmpfr-dev:armel                    3.1.0-3         

ii  libmpfr4:armel                       3.1.0-3       

ii  libpcre3:armel                       8.12-4                

ii  libpixman-1-0:armel                  0.24.4-1    

ii  libpng12-0:armel                     1.2.46-4     

ii  libpopt0:armel                       1.16-3        

ii  libpwl5:armel                        0.11.2-6    

ii  libselinux1:armel                    2.1.0-4.1      

ii  libstdc++6:armel                     4.6.2-14       

ii  libthai0:armel                       0.1.16-3     

ii  linux-libc-dev:armel                 3.2.4-1         

ii  zlib1g:armel                         1:1.2.6.dfsg-1   

There are more packages which can be installed on amd64 for i386 as #659588 only affects Multi-Arch versions which cannot execute compiled binaries from the foreign architecture.

Further progress, inside a test chroot, involves using gcc-4.7 from experimental but even then, libc6-dev is not installable as a Multi-Arch package. That's the current blocker for toolchain stuff.

ii  gcc-4.7-base:i386               4.7-20120210-1

ii  libgcc1:i386                    1:4.7-20120210-1

Once we have libc6-dev:armel installable on i386 and amd64, work on cross-building Emdebian can be considered again. It's been a long time.

You know ACTA and related topics have reached the mainstream when the German news show devotes a segment of several minutes(!) to ACTA, complete with background information about the sleazy backroom dealings, and the various political sunday talk shows debate controversially about it, as well. In a round of five 60-plus aged people, only one defenced ACTA; the rest was sceptic or dead against it. The several tens of thousands of people who walked the streets of Germany yesterday (I couldn't; I am ill) really managed to stir up Germany's political landscape. This. is. awesome. Now if only the population at large would understand the inherent differences between physical counterfeiting, physical theft, and copyright infrigement. That's where we all come in: Inform people about the background and actual issues. Now, more than ever, there's a chance that they will be willing to listen.

When Jacob was just born, I wondered how I might introduce them to computing. I thought over various things, but that wasn t really the most pressing thing right then. I don t suppose that I could have predicted installing an XMPP IM server (Prosody) for the boys. And I certainly couldn t have predicted creating accounts named: jacob, oliver, butterfly, bear. Because, as Jacob pointed out to me, if (Jacob s favorite toy) butterfly is typing with his wings, then he shouldn t be logged in as Jacob. I admire my 5-year-old s security consciousness Anyhow, as I mentioned yesterday, Jacob and Oliver enjoy their computer, which I recently put on the LAN. The firewall does not pass any of its traffic to the Internet, though, with very limited exceptions. Jacob can read, and is starting to enjoy typing as well. So I thought he would enjoy sending IMs to me. As his computer has no GUI, I needed a text-mode client. Something with an IRC-like interface that could be scripted to open up a window with me directly sounded perfect. Initially I tried irssi s XMPP plugin, but it proved to be too buggy (wanting to always latch on to a particular resource on the remote end, not having very predictable window behavior, etc.) So I switched to mcabber. With a couple of quick configuration bits to get him automatically logged in, remove superflous windows, and connect him directly to a chat with me, it was set. And well-loved. He sent me a mix of real words and random things he created by replacing letters in Jacob or by holding down keys. In the mcabberrc, besides the obvious setting of username and password, there is:
set log_win_height = 1
set hook-post-connect = source ~/.mcabber/post-connect.rc
The hook is simply:
roster search Dad
roster hide
After awhile, Jacob wanted to switch computers. He wanted to use my laptop, and me use his computer. He refused to switch back. I asked him why. Because on your computer, my name is red. I should have known. I set it to bright white on his computer, but I think tomorrow we may need to upgrade him to the color monitor I ve been saving for just such an occasion It will be a whole new set of discoveries, I m sure. Update: I also tried out freetalk, which looked like it would meet my goals nicely. The problem was it didn t have a dedicated everything typed goes to this person mode. It did have a mode where it put the person s JID on the command line by default, but excessive use of backspace key by a 5-year-old could wipe that out and leave it in a state where he d be confused.

If you are working with Socket.IO and Express Framework while developing your applications you must know they work together so Socket.IO will listen on the same port that your HTTP/S server and will serve the requests by that way, but Socket.IO doesn t works on the same paradigm where you pass a middleware to the route and you could get authentication on this manner. Instead, you must configure Socket.IO with a global authorization method or a handshake authorization manner. Since in my applications I serve cookies already where I save session info and this is already stored in Redis with help of the module connect-redis then I should be able to get this info onto Socket.IO authorization method and with this allow only to authorized users to get info from my websockets. I found how to do it with the default session-store in-memory from Express, and adapted that so it could work with Redis. Here s the snippet: <script src="https://gist.github.com/1805743.js"> </script><noscript>You don t have JavaScript enabled, check it out at gist.github.com </noscript> Hope anyone can found it useful!

I've been spending a lot more time recently in meetings. Mostly things I should actually be at. And in general if it's something I think is reasonable I'll try to be there. In an effort to help with this I actually keep my work calendar up to date. Given that I'm running Linux on my laptop and the corporate standard is Exchange this requires a little bit of effort on my part (the Thunderbird Provider for Microsoft Exchange and Android support for talking to Exchange are helpful with this). Sometimes it seems like I shouldn't bother. I spent this week at a conference, and marked my calendar to indicate I was out of the office. I think I had at least 3 meeting requests, all for things that would actually have been appropriate for me to go to. Last week I managed to be booked for 7 hours of meetings from 7am until noon. That included a 30 minute window where I was triple booked. The thing is, I'm really not that busy in terms of meetings - you can usually find a spot when I'm free on any given day unless I'm actually not in the office. If you bother to check my calendar, that is. Another problem I have is the times people like to book meetings at. Booking a technical meeting at 9am isn't going to get me at my best. Equally doing so at 5pm is likely to have me clock watching to make sure I don't miss my bus and/or train. Also I seem to work with far too many people who don't eat lunch and book hour long meetings at midday or 1pm. I understand sometimes that's the only time you can get everyone into a room together, but at least bloody ask and explain the need rather than just sending out a meeting invite. Finally, book meetings of a realistic length. There are some people who invite me to things and cause me to add another 30 minutes on the end, because I know it always overruns. It's not all bad. I have a VP who always runs a meeting to time, and never seems to call one for spurious reasons. I've also worked with a program manager who will organize the meeting so that if you're only there for one point on the agenda that'll get dealt with near the beginning so it doesn't take up more of your time that it needs to. Funnily enough I'm much more likely to go to things both of these people arrange. Disclaimer: In the unlikely event anyone I work with who invites me to meetings is reading this, I might be talking about you, but everything I mention has been done by more than one person, so I'm not thinking about anyone in particular for each point.

I experienced situations of writing substantial amounts of code (feature branches) for hours, days and even weeks only to find later that the written code can be thrown to the bin, either because of hidden design problem(s) or too much negative implications outweighting the positive implications of the change.

After reading some recent multiarch threads on debian-devel@ and seeing what hacks are seriously being proposed to implement only to keep the thing going, I now think that the low-level part of the Debian multiarch implementation proposal is no less broken than its high-level part, and the whole proposal is a one big hack which requires and will require more subhacks. Some will benefit from the added functionality, but all, both maintainers and users will suffer from drawbacks.

How two paragraphs above are related? We still have the time to revert the changes and say "sorry, it was a nice idea but the software world isn't ready".

I ve just discovered that the magic database for the file(1) command in Debian/Unstable has no support for Linux Software RAID and that it s support for BTRFS is lacking (no reporting of space used, number of devices, or the UUID). Below is my first draft of a change to fix these problems. I would appreciate it if someone with a big-endian system could test these out and let me know how they go, I suspect that I will have to change the lelong types to long but I m not sure. 4096 lelong 0xa92b4efc Linux Software RAID
>4100 lelong x version 1.2 (%d)
>4112 belong x UUID=%8x:
>4116 belong x \b%8x:
>4120 belong x \b%8x:
>4124 belong x \b%8x
>4128 string x name=%s
>4168 lelong x level=%d
>4188 lelong x disks=%d 0 lelong 0xa92b4efc Linux Software RAID
>4 lelong x version 1.1 (%d)
>16 belong x UUID=%8x:
>20 belong x \b%8x:
>24 belong x \b%8x:
>28 belong x \b%8x
>32 string x name=%s
>72 lelong x level=%d
>92 lelong x disks=%d # BTRFS
0 10040 string _BHRfS_M BTRFS Filesystem
>0x1012b string >\0 label "%s",
>0 10090 lelong x sectorsize %d,
>0 10094 lelong x nodesize %d,
>0 10098 lelong x leafsize %d,
>0 10020 belong x UUID=%8x-
>0 10024 beshort x \b%4x-
>0 10026 beshort x \b%4x-
>0 10028 beshort x \b%4x-
>0x1002a beshort x \b%4x
>0x1002c belong x \b%8x,
>0 10078 lequad x %lld/
>0 10070 lequad x \b%lld bytes used,
>0 10088 lequad x %lld devices Related posts:

1. Label vs UUID vs Device Someone asked on a mailing list about the issues related...
2. Starting with BTRFS Based on my investigation of RAID reliability [1] I have...
3. Software vs Hardware RAID Should you use software or hardware RAID? Many people claim...

It probably comes as no surprise to anybody that Jacob has had a computer since he was 3. Jacob and I built it from spare parts, together. It may come as something of a surprise that it has no graphical interface, and Jacob uses the command line and loves it and did even before he could really read. A few months ago, I wrote about the fun Jacob had with speakers and a microphone, and posted a copy of the cheat sheet he has with his computer. Lately, Jacob has really enjoyed playing with the speech synthesizer both trying to make it say real words and nonsense words. Sometimes he does that for an hour. I was asked for a copy of the scripts I wrote. They are really simple. I gave them names that would be easy for a preschooler to remember and spell, even if they conflicted with existing Unix/Linux commands. I put them in /usr/local/bin, which occurs first on the PATH, so it doesn t matter if they conflict. First, for speech systhesis, /usr/local/bin/talk:
#!/bin/bash
echo "Press Ctrl-C to stop."
espeak -v en-us -s 150
espeak comes from the espeak package. It seemed to give the most consistenly useful response. Now, on to the sound-related programs. Here s /usr/local/bin/ssl, the sound steam locomotive . It starts playing a train sound if one isn t already playing:
#!/bin/bash
pgrep mpg321 > /dev/null mpg321 -q /usr/local/trainsounds/main.mp3 &
sl "$@"
And then there s /usr/local/bin/record:
#!/bin/bash
cd $HOME/recordings
echo "Now recording. Press Ctrl-C to stop."
DATE= date +%Y-%m-%dT%H-%M-%S
FILENAME="$DATE-$$.wav"
chmod a-w *.wav
exec arecord -c 1 -f S16_LE -c 1 -r 44100 "$FILENAME"
This simply records in a timestamped file. Then, its companion, /usr/local/bin/play. Sorry about the indentation; for whatever reason, it is being destroyed by the blog, but you get the idea.
#!/bin/bash
case "$1" in
train)
mpg321 /usr/local/trainsounds/main.mp3
;;
song)
/usr/bin/play /usr/local/trainsounds/traindreams.flac
;;
*)
cd $HOME/recordings
exec aplay ls -tr tail -n 1
;;
esac
So, Jacob can run just play , which will play back his most recent recording. As something of a bonus, the history of recordings is saved for us to listen to later. If he types play train , there is the sound of a train passing. And, finally, play song plays Always a Train in My Dreams by Steve Gillette (I heard it on the radio once and bought the CD). Some of these commands kick off sound playing in the background, so here is /usr/local/bin/bequiet:
#!/bin/bash
killall mpg321 &> /dev/null
killall play &> /dev/null
killall aplay &> /dev/null
killall cw &> /dev/null


zenbook ~ # uname -r
3.3.0-rc3-fenio
zenbook ~ # Let s see how many sudden shutdowns will be on that kernel

If you met Ana, you ll easily remember her. She has a great and pronounced Spanish accent I m glad that the existence of the Debian Women project helped her to join Debian because she has been doing a great job. From KDE packaging to publicity/marketing work, her interests shifted over the years but this allowed her to stay very involved. As she explains it very well, Debian is big enough so that you can stop doing something which is no longer fun for you, and still find something new to do in another part of Debian! Read on to learn more about Ana, the KDE team, Debian s participation to the Google Summer of Code, and more. Raphael: Who are you? Ana: I m Ana Guerrero L pez and I m in my early 30s. I was born and raised in the wonderful city of Sevilla, Spain and I live in Lyon, France. I share my life with another Debian Developer and my paid work is doing Debian support and integration, so you won t be surprised to read that Debian is a big part of my life. Raphael: How did you start contributing to Debian? Ana: Although I knew about the existence of Linux since 1997 or so, I didn t really start using Linux until the summer 2001 when I finally got a computer on my own and an Internet link at home. In the beginning, I was using Mandrake in a dual boot with Windows and later around 2003, I happily moved to only using Debian and ditching the Windows partition. Once settled as a Debian user, I knew anybody could help improve the distribution but I hesitated to join mostly due to two reasons, my perception of Debian was the one of a very elitist and aggressive club and who wants to join this kind of cult^wproject? And even if I wanted to join, I did not know how to get started. By the summer of 2004, the Debian Women project started, it made me seeing Debian as a more welcoming project, and I started maintaining my first packages. The following summer 2005, I attended akademy 2005 (the annual KDE conference) where I had the pleasure to meet there some of the people from the KDE team and this really made a difference for me. Christopher Martin and Adeodato Sim , with the help of other people, have started the maintenance of KDE as a team a few months before and by that time most of the KDE modules where under the maintenance umbrella of the team. This was a very good move since it allowed easily to share the KDE maintenance in a more coordinated way and also eased having non-DDs, like me at that time, to join in and help.

The Debian Women project started, it made me seeing Debian as a more welcoming project.

Raphael: You re part of the Debian KDE team. What s your role in the team and what are your plans for Wheezy? Ana: Nowadays, I am not as active in the KDE team as I used to be in the past. The KDE 3 to KDE 4 transition was quite tiring and changes on the KDE side like the successive marketing renames, the shorter 6 months schedule (it used to be at least 9) or the uncoordinated KDE releases mostly burnt me out. Currently, I am mostly working in helping others to get started within the team, some small fixes here and there, and helping with the uploads: an upload of the full KDE suite to the archive requires some building power and upload bandwidth not everybody have. For Wheezy, with the tentative freeze date in June, the plan is to try to ship the latest possible point release of the KDE 4.8 series. The first release of the series, 4.8.0 was released a couple of weeks ago and while writing these lines, the packaging work for 4.8 hasn t started yet. The next move for the team is getting 4.7.4 in unstable, currently sitting in experimental.

For Wheezy, [ ] the plan is to try to ship the latest possible point release of the KDE 4.8 series.

Besides the KDE packages, there is some software which users perceive as KDE, such as amarok, digikam, etc., which are not part of KDE but fall under its umbrella. These other programs have their own maintainers and their updates depend greatly in the availability of them. For the KDE office suite, we have right now KOffice in the archive. KOffice got a fork some time ago named Calligra and we should replace KOffice by Calligra in the archive before the release of Wheezy. Sadly there isn t yet a final release of Calligra to use. My personal goal for Wheezy was to finish the removal of all the remaining packages depending on KDE 3 and Qt 3 that Squeeze still contained. The removal of the KDE 3 libraries and all the packages using them was quickly achieved after the release of Squeeze. The removal of Qt 3 soon showed that it was task harder than expected since some popular packages (sometimes not in the Debian archive, e.g. third-party scientific software) depend on it, and also Qt 3 is a requirement for LSB compatibility. Right now, Qt 3 has been orphaned for 9 months and nobody has shown any interest in adopting it. Raphael: KDE, much like GNOME, has been forked by people who were unhappy by the direction that the project has taken since version 4 (cf Trinity). What s your personal opinion on KDE 4.x and what s the position of the Debian KDE team concerning this fork? Ana: I use KDE 4 on my laptop and I think it is a solid desktop environment and platform. However I am finding it less and less attractive for me. On one side, my usage of the computer has been slightly changing and on the other side, I do not like how the new developments in KDE are evolving, things like plasmoids or activities are not attractive for me. I have switched my other 2 systems to awesome although I continue to use mainly a bunch of KDE applications: dolphin, konsole, kate, juk, kmix, etc. So you might say my desktop environment is an awesome KDE. Regarding the Trinity project, a lot of users complained very loudly when KDE 3 got replaced by KDE 4 in testing/unstable, so I find quite laudable the decision of some users to act instead and try to continue with a forked development of KDE 3. However the Trinity team seems to be about 3 persons (funny for a project named Trinity ) while KDE 3 is big. In perspective, it does not look that big because KDE 4 is even larger, but it is still too much for such small team. In addition those developers need to maintain Qt3 that has been end-of-lifed years ago by Nokia/Trolltech . So my guess is that sooner or later the project will fade away. Nobody from the KDE team is interested in Trinity and in case someone wants to package it for Debian, they would have to make a new team. For the reasons mentioned above: Qt3 maintenance and reduced upstream group, this would be a bad idea. My advice if you do not like KDE 4 and you miss KDE 3, would be taking a look at razor-qt based on Qt4 and quite similar to KDE 3. I read they have plans to port it to Qt4, but frankly that could take some years same it took to the KDE project for KDE 4.0.0 Raphael: You used to maintain news.debian.net, a WordPress blog dedicated to Debian, but you stopped a while ago. A few months later you started to maintain a Debian page on Google+. Why did you stop the blog and what s your goal with the Google+ page? Ana: I blogged about the reasons I started news.debian.net. In short, I thought Debian needed a better system to publish news, something like a blog. I first tried to suggest the idea to the press/publicity team but they weren t interested, so I started the project alone. IMHO the blog worked quite well and I was feeling like it should be made official. I talked about this with some people but at the time I wasn t pushing it because I had other priorities and I knew pushing it to become official would need some extra time and energy. Stefano decided to start the discussion about making news.debian.net official (that s moving it to a debian.org domain) in its own initiative. After the public discussion and some private exchange of emails with DSA, the situation became frustrating and I decided to close news.debian.net after the release of Squeeze. Later, during DebConf, an officer from the press team announced they were launching a blog and I asked Stefano if he could try to have a discussion about this to see if it could still somehow fit my ideas, and maybe contributing myself, but nobody from the press team answered Stefano s email and the blog hasn t started yet either. Irony that communication didn t work when wanting to improve communication. About the Google+ page, everyday I follow what is going in Debian and quite often I find things I want to share. I do not want to clutter my own profiles with Debian stuff or have people following me because of that, so I decided to create the Debian page when Google+ made them available. I like the fact that people can follow that without having an account in Google+ although they can not comment anonymously. I am not happy about the fact that Google+ is a closed platform but hopefully the data will become easier to export in the near future. Right now, there are some services that provides RSS feeds of Google+ pages if you want to follow the page and you are not in Google+ (or I could setup one if several people ask me). Raphael: Last year you helped to manage Debian s participation to the Google Summer of Code. How did it went? Is there something that you can improve for this year? Ana: I think last year we managed to have people in Debian more aware about what the students were doing. That also helped students to get more feedback and therefore get to know more people in the project and get more integrated. Students were sending periodic public reports available to everybody interested in the status of the projects and some of them also held their own sessions in DebConf. We still failed to start looking for mentors early enough and to give them information about how the GSoC worked and how they could have a successful project. Having good projects in Debian is harder than in other projects because the GSoC mostly promotes having students started in Open Source *coding* for a project, while Debian is more a project about integrating software and we overall do not have so many parts that has to be coded. My personal goal for this year is to try getting the projects earlier to attract good students from the very beginning, even if that means we have less projects than in other years. Raphael: What motivates you to continue to contribute year after year? Ana: Three things. I like improving the OS I use, I like the friends I have made while working in Debian through the years and because I have fun. Also Debian is quite a big project, so if you become tired or burn out working in some area, you always can easily find interesting things to do somewhere else. Raphael: Is there someone in Debian that you admire for their contributions? Ana: Adeodato Sim , he is now in a long leave from the project, but it is one of those persons who made a difference in the project in his job in the release team some years ago. Aur lien Jarno because of his tireless work in (e)glibc and porting of several architectures. I also have special admiration for all those people who have been very active in the project for more than 7-8 years because I know it is not always easy to combine it with real life.

---

Thank you to Ana for the time spent answering my questions. I hope you enjoyed reading her answers as I did. Note that older interviews are indexed on wiki.debian.org/PeopleBehindDebian.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook .

7 comments Liked this article? Click here. My blog is Flattr-enabled.

Hello In Debian package, the debian/control file has a specific syntax where each line must begin with a space and paragraphs are separated by a single . . If you want to follow DEP-5 syntax for debian/copyright files, you have a similar problem when comes the time to fill the license text for creative-common license: it s easy to miss a space or a dot. Let s say I want to add a new section in the description of my package. Here s the current description:

Description: module to automate definition of a DBIx::Class::Schema
 DBIx::Class::Schema::Loader is an extension to DBIx::Class that automates the
 definition of a DBIx::Class::Schema by scanning table schemas and setting up
 columns and primary keys appropriately. It supports MySQL, PostgreSQL, SQLite
 and DB2.
 .
 Bare table definitions are fairly straightforward, but relationship creation
 is somewhat heuristic, especially with respect to choosing relationship types
 and names, as well as join types. The relationships generated by this module
 will probably never be as well-defined as hand-generated ones.

I want to add This package is awesome but I don t want to be bothered by leading dots and white spaces. First, I go in the package directory, then check the syntax with cme:

$ cme check dpkg

Then create a loop directory, where the dpkg content will be mapped to a fuse directory:

$ mkdir loop
$ cme fusefs dpkg -fuse-dir loop/
Mounting config on loop/ in background.
Use command 'fusermount -u loop/' to unmount
$ cat loop/control/binary/libdbix-class-schema-loader-perl/Description
DBIx::Class::Schema::Loader is an extension to DBIx::Class that automates the
definition of a DBIx::Class::Schema by scanning table schemas and setting up
columns and primary keys appropriately. It supports MySQL, PostgreSQL, SQLite
and DB2.
Bare table definitions are fairly straightforward, but relationship creation
is somewhat heuristic, especially with respect to choosing relationship types
and names, as well as join types. The relationships generated by this module
will probably never be as well-defined as hand-generated ones.

Then edit loop/control/binary/libdbix-class-schema-loader-perl/Description with your favorite editor. For the sake of this blog, I will use echo to edit this file:

$ echo -e "\nThis module is awesome" >> loop/control/binary/libdbix-class-schema-loader-perl/Description

Now umount the loop (note that the actual debian/control is not changed before this step):

fusermount -u loop

Now the result:

$ cat debian/control
[...]
Description: module to automate definition of a DBIx::Class::Schema
 DBIx::Class::Schema::Loader is an extension to DBIx::Class that automates the
 definition of a DBIx::Class::Schema by scanning table schemas and setting up
 columns and primary keys appropriately. It supports MySQL, PostgreSQL, SQLite
 and DB2.
 .
 Bare table definitions are fairly straightforward, but relationship creation
 is somewhat heuristic, especially with respect to choosing relationship types
 and names, as well as join types. The relationships generated by this module
 will probably never be as well-defined as hand-generated ones.
 .
 This module is awesome

Here s another example to fill the license text for a license while respecting DEP-5 syntax, (done after the cme command):

# creating this dir will create a new DEP-5
# standalone license section with name CC-BY-30
mkdir loop/copyright/License/CC-BY-30
# fill the text part of the  license
wget http://creativecommons.org/licenses/by/3.0/legalcode -O - \
   html2text -style pretty > loop/copyright/License/CC-BY-30/text
$ fusermount -u loop
$ cat debian/copyright
[...]
License: CC-BY-30
 .
 .
 .
                                Creative_Commons
 .
 .
 Creative Commons Legal Code
 .
 .
 Attribution 3.0 Unported
[...]

And that s it. (note that cme command is new and is available only from version 2.004 of Config::Model) Of course, there s more than a awy to do it: you can also run cme edit dpkg and add or edit license text in the GUI (or in an editor launched by the GUI). cme will take care of the syntax details. All the best

Based on my investigation of RAID reliability [1] I have determined that BTRFS [2] is the Linux storage technology that has the best potential to increase data integrity without costing a lot of money. Basically a BTRFS internal RAID-1 should offer equal or greater data protection than RAID-6. As BTRFS is so important and so very different to any prior technology for Linux it s not something that can be easily deployed in the same way as other filesystems. It is possible to easily switch between filesystems such as Ext4 and XFS because they work in much the same way, you have a single block device which the filesystem uses to create a single mount-point. While BTRFS supports internal RAID so it may have multiple block devices and it may offer multiple mountable filesystems and snapshots. Much of the functionality of Linux Software RAID and LVM is covered by BTRFS. So the sensible way to deploy BTRFS is to give it all your storage and not make use of any other RAID or LVM. So I decided to do a test installation. I started with a Debian install CD that was made shortly before the release of Squeeze (it was first to hand) and installed with BTRFS for the root filesystem, I then upgraded to Debian/Unstable to get the latest kernel as BTRFS is developing rapidly. The system failed on the first boot after upgrading to Unstable because the /etc/fstab entry for the root filesystem had the FSCK pass number set to 1 which wasn t going to work as no FSCK program has been written. I changed that number to 0 and it then worked. The initial install was on a desktop system that had a single IDE drive and a CD-ROM drive. For /boot I used a degraded RAID-1 and then after completing the installation I removed the CD-ROM drive and installed a second hard drive, after that it was easy to add the other device to the RAID-1. Then I tried to add a new device to the BTRFS group with the command btrfs device add /dev/sdb2 /dev/sda2 and was informed that it can t do that to a mounted filesystem! That will decrease the possibilities for using BTRFS on systems with hot-swap drives, I hope that the developers regard it as a bug. Then I booted with an ext3 filesystem for root and tried the btrfs device add /dev/sdb2 /dev/sda2 again but got the error message btrfs: sending ioctl 5000940a to a partition! which is not even found by Google. The next thing that I wanted to do was to put a swap file on BTRFS, the benefits for having redundancy and checksums on swap space seem obvious and other BTRFS features such as compression might give a benefit too. So I created a file by using dd to take take from /dev/zero, ran mkswap on it and then tried to run swapon. But I was told that the file has holes and can t be used. Automatically making zero blocks into holes is a useful feature in many situations, but not in this case. So far my experience with BTRFS is that all the basic things work (IE storing files, directories, etc). But the advanced functions I wanted from BTRFS (mirroring and making a reliable swap space) failed. This is a bit disappointing, but BTRFS isn t described as being ready for production yet.

• [1] http://etbe.coker.com.au/2012/02/06/reliability-raid/
• [2] http://en.wikipedia.org/wiki/Btrfs

Related posts:

1. Discovering OS Bugs and Using Snapshots I m running Debian/Unstable on an EeePC 701, I ve got an...
2. Reliability of RAID ZDNet has an insightful article by Robin Harris predicting the...
3. How I Partition Disks Having had a number of hard drives fail over the...

SNMP stands for Simple Network Management Protocol. It allows a manager to query information from an agent. A popular use of SNMP is to retrieve network interface counters to plot a bandwidth graph. While the S in SNMP stands for simple, SNMP can be quite difficult to deal with. However, it is still the de facto standard for retrieving metrics in an heterogeneous environment. Net-SNMP is a suite of SNMP applications including an agent. Out of the box, this agent exports a lot of information but if something is missing, there are several ways to extend it. Which ones? tl;dr: in my opinion, extending Net-SNMP should be done with either of those methods:

• using extend directive;
• using pass_persist directive;
• using AgentX protocol.

SNMP in a nutshell The variables accessible with SNMP through the agent are organized in a tree. Each variable is typed and associated to an object identifier (or OID). For example, .1.3.6.1.2.1.31.1.1.1.1.2 is the OID of the name of the second network card. Usually, SNMP is operated over UDP, port 161. For the purpose of this article, we consider a manager can issue three kinds of requests:

GET

retrieve a variable.

GETNEXT

retrieve the next variable, in lexical order of their OID. This operation enables an SNMP manager to walk through all available variables, for example to discover the list of interfaces of an agent. This operation is what makes an SNMP agent difficult to implement but this is also a critical feature.

SET

request the modification of a variable.

Since such an OID may be difficult to handle by humans, available variables are described by a MIB module which is a file that should be parsed to get a mapping between OID and variable names and types. For example, the above OID can also be referred as IF-MIB::ifName.2. Those MIB modules are described using a subset of ASN.1 defined in RFC 2578. A MIB module allows to group several variables into a conceptual table. For example, IF-MIB::ifDescr, IF-MIB::ifType, IF-MIB::ifSpeed are columns of IF-MIB::ifTable. Each row is associated to an index. In the case of IF-MIB::ifTable, it is the interface index. Net-SNMP includes snmptable allowing one to display such a table in a natural manner¹:

$ snmptable localhost IF-MIB::ifTable
SNMP table: IF-MIB::ifTable
 ifIndex ifDescr           ifType ifMtu   ifSpeed
       1      lo softwareLoopback 16436  10000000
       2    eth0   ethernetCsmacd  1500 100000000
       3    eth1   ethernetCsmacd  1500  10000000
       4     br0   ethernetCsmacd  1500         0

Variables inside a table are called columnar objects while those outside are scalar objects. SNMP, as a protocol, does not care of this distinction since it does not know the concept of MIB modules. Other useful tools from Net-SNMP includes snmpget to get a specific variable, snmpwalk to discover available variables (with GETNEXT operation) from an OID and snmptranslate to translate between object names and OID.

$ snmpget localhost IF-MIB::ifDescr.2
IF-MIB::ifDescr.2 = STRING: eth0
$ snmpwalk localhost IF-MIB::ifDescr
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
IF-MIB::ifDescr.4 = STRING: br0
$ snmpwalk -On public localhost .1.3.6.1.2.1.2.2.1.2
.1.3.6.1.2.1.2.2.1.2.1 = STRING: lo
.1.3.6.1.2.1.2.2.1.2.2 = STRING: eth0
.1.3.6.1.2.1.2.2.1.2.3 = STRING: eth1
.1.3.6.1.2.1.2.2.1.2.4 = STRING: br0
$ snmptranslate -On IF-MIB::ifDescr.3
.1.3.6.1.2.1.2.2.1.2.3

Extending Net-SNMP ethtool -S allows us to access various statistics from a network card, like the number of octets received and transmitted or the number of collisions. While some of those statistics are defined in IF-MIB, in RMON-MIB and in EtherLike-MIB, some of them are not available and very specific. Therefore, we would like to export those statistics, unmodified, through SNMP. To the best of my knowledge, no MIB module exists for such an usage. The first task is to write one. It can be called ETHTOOL-MIB. It only contains one table indexed by the interface index and the name of the statistic. Here is the expected output:

$ snmpwalk localhost ETHTOOL-MIB::ethtoolStat.2
ethtoolStat.2.'align_errors' = Counter64: 0
ethtoolStat.2.'broadcast' = Counter64: 25
ethtoolStat.2.'multicast' = Counter64: 345
ethtoolStat.2.'rx_errors' = Counter64: 0
ethtoolStat.2.'rx_missed' = Counter64: 0
ethtoolStat.2.'rx_packets' = Counter64: 262011
ethtoolStat.2.'tx_aborted' = Counter64: 0
ethtoolStat.2.'tx_errors' = Counter64: 0
ethtoolStat.2.'tx_multi_collisions' = Counter64: 0
ethtoolStat.2.'tx_packets' = Counter64: 296170
ethtoolStat.2.'tx_single_collisions' = Counter64: 0
ethtoolStat.2.'tx_underrun' = Counter64: 0
ethtoolStat.2.'unicast' = Counter64: 261641

I have coded various implementations of this MIB module and made them available in a GitHub repository.

With arbitrary commands The extend directive allows the agent to execute an arbitrary command and to provide its output and status through NET-SNMP-EXTEND-MIB module. If we want to export the statistics for eth0, we could add something like this in snmpd.conf, using a custom script to format the output of ethtool -S in a way that is more suitable for our use:

# Export eth0 statistics
extend eth0-names  /full/path/to/ethtool-stats eth0 names
extend eth0-values /full/path/to/ethtool-stats eth0 values

The result can be retrieved through NET-SNMP-EXTEND-MIB:: nsExtendOutput2Table:

$ snmpwalk localhost NET-SNMP-EXTEND-MIB::nsExtendOutput2Table
nsExtendOutLine."eth0-names".1 = STRING: tx_packets
nsExtendOutLine."eth0-names".2 = STRING: rx_packets
nsExtendOutLine."eth0-names".3 = STRING: tx_errors
nsExtendOutLine."eth0-names".4 = STRING: rx_errors
nsExtendOutLine."eth0-names".5 = STRING: rx_missed
nsExtendOutLine."eth0-names".6 = STRING: align_errors
nsExtendOutLine."eth0-names".7 = STRING: tx_single_collisions
nsExtendOutLine."eth0-names".8 = STRING: tx_multi_collisions
nsExtendOutLine."eth0-values".1 = STRING: 246309
nsExtendOutLine."eth0-values".2 = STRING: 223941
nsExtendOutLine."eth0-values".3 = STRING: 0
nsExtendOutLine."eth0-values".4 = STRING: 0
nsExtendOutLine."eth0-values".5 = STRING: 0
nsExtendOutLine."eth0-values".6 = STRING: 0
nsExtendOutLine."eth0-values".7 = STRING: 0
nsExtendOutLine."eth0-values".8 = STRING: 0

It is also possible to configure such a command using SET requests. For example, if we want to add statistics for eth2, we could issue the following command:

$ snmpset -m +NET-SNMP-EXTEND-MIB localhost \
>    'nsExtendStatus."eth2-names"'  = createAndGo \
>    'nsExtendCommand."eth2-names"' = /full/path/to/ethtool-stats \
>    'nsExtendArgs."eth2-names"'    = 'eth2 names' \
>    'nsExtendStatus."eth2-values"'  = createAndGo \
>    'nsExtendCommand."eth2-values"' = /full/path/to/ethtool-stats \
>    'nsExtendArgs."eth2-values"'    = 'eth2 values'
nsExtendStatus."eth2-names" = INTEGER: createAndGo(4)
nsExtendCommand."eth2-names" = STRING: /full/path/to/ethtool-stats
nsExtendArgs."eth2-names" = STRING: eth2 names
nsExtendStatus."eth2-values" = INTEGER: createAndGo(4)
nsExtendCommand."eth2-values" = STRING: /full/path/to/ethtool-stats
nsExtendArgs."eth2-values" = STRING: eth2 values
$ snmpgetnext -m +NET-SNMP-EXTEND-MIB localhost \
>    'nsExtendOutLine."eth2-names"'
nsExtendOutLine."eth2-names".1 = STRING: tx_packets

This can be very convenient but also a huge security risk. You should disable this functionality or enable it only for some SNMPv3 user. Using extend does not allow to implement an arbitrary MIB module. If the cache is not disabled, extend is a good solution for various simple needs, like providing the content of a one-line file or the output of some Nagios plugin. However, exporting anything tabular, like interface statistics, is not a good fit.

With pass-through scripts Net-SNMP allows one to extend the agent by delegating some sub-tree to a script defined by the pass_persist directive. Such a script will receive requests on its standard input with a very simple line-oriented protocol described in the manual page of snmpd.conf. Here is an example of interaction:

   PING
   PONG
   get
   .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116
   .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116
   counter64
   223941
   getnext
   .1.3.6.1.4.1.39178.100.1.1.1.2.2
   .1.3.6.1.4.1.39178.100.1.1.1.2.2.97.108.105.103.110.95.101.114
   counter64
   0

The protocol is simple enough to implement it from scratch in any scripting language. However, in Perl, you may prefer to use SNMP:: ExtensionSNMP:: PassPersist extension from S bastien Aperghis-Tramoni. You can find the complete example on Github.

use SNMP::Extension::PassPersist;
my $extsnmp = SNMP::Extension::PassPersist->new(
    backend_collect => \&update_tree
    );
$extsnmp->run;
sub update_tree  
    my @interfaces = </sys/class/net/*>;
    foreach my $interface (@interfaces)  
        $interface =~ s/^.*\///;
        open(ETHTOOL, "ethtool -S $interface 2>/dev/null  ") or next;
        while (<ETHTOOL>)  
            /^\s+(\w+): (\d+)$/ or next;
            my $name  = $1;
            my $value = int($2);
            my $oid   = oid_compute($interface, $name);
            $extsnmp->add_oid_entry($oid, "counter", $value);
         
        close(ETHTOOL);
     
 

With Python, snmp-passpersist module, from Nicolas Agius, provides a similar interface. Look at the complete example on GitHub. pass_persist allows you to implement an arbitrary MIB module with honest performances. Here are some important things to know about this directive:

• Unless you have a recent version of Net-SNMP, 64bit types are converted to 32bit equivalents. The counters may overflow quickly.
• While handling a request, you are not allowed to pause (for example, to fetch a remote information): the agent would become unresponsive. If you need slow-to-get data or data from the agent, grab them in a separate thread and ensure you have them beforehand.

With AgentX protocol AgentX is a protocol defined in RFC 2741 allowing a master agent to be extended by independent sub-agents. For snmpd to become a master agent, you just need to add master agentx in snmpd.conf. Here are some of the benefits of an AgentX sub-agent over pass_persist:

1. No configuration is needed for the master agent to accept an additional sub-agent. A sub-agent registers to the master agent the MIB modules (or part of them) it wants to take care of.
2. A sub-agent is decoupled from the master agent. It can run with a different identity or be integrated into another daemon to export its internal metrics, send traps or allow remote configuration through SNMP.
3. AgentX protocol can be carried over TCP. Sub-agents can therefore run on a foreign host or in a jailed environment.
4. 64bit types are fully supported. Traps are also supported.

Perl and Python Net-SNMP provides a Perl module, NetSNMP::agent, to write a sub-agent using AgentX protocol. This module is not as convenient as the module for pass_persist described above since it does not provide a convenient layer to put required information into some datastore. Basically, it mimics the available C API. Here are a few examples of its use:

• An implementation of ETHTOOL-MIB I have done in the same way as the pass_persist version. My Perl skills are a bit rusty and the code has many rough edges.
• An implementation of a sub-agent for Redis servers.
• An implementation of BRIDGE-MIB shipped with Net-SNMP.

There is no similar binding for Python but there is an agentx extension using ctypes. It provides a more high-level view. I did not write an example for this one but you can look at the example provided by the author.

C with new API To write a sub-agent in C, you have to choose between two API:

1. the API inherited from UCD-SNMP also known as the traditional API;
2. the new API developed for Net-SNMP 5.x.

With both API, an handler is registered to take care of one or several sub-trees. With the traditional API, the handler is invoked with very little information. It is then up to you to locate the appropriate variable by mapping the index part of the OID to the objects you want to export. It is quite simple for scalar objects but columnar objects with complex indexes are a pain with such an API. The new API provides various helpers. Some of them allow you to copy objects into an array or a list and let Net-SNMP library do the hard work of serving them. Another helper will provide methods to serve objects without putting them in a special structure but by providing an iterator. Some of those helpers can also help you to cache some variables. On top of this new API, Net-SNMP comes with mib2c, a tool that will help you convert a MIB module into C code: after a few questions, you will get a skeleton C code you will have to complete with the logic to retrieve real objects. A generated file also provides directions on what to do (with some magic command to have a step-by-step cookbook). I was able to implement ETHTOOL-MIB by just running it through mib2c, choosing the table container helper with cache and adding some code in ethtoolStatTable_data_access.c. You can look at the result on Github. Here is a stripped-down version² of what has been modified:

/* Socket for ioctl */
skfd = socket(AF_INET, SOCK_DGRAM, 0);
/* Iterate through all interfaces */
getifaddrs(&ifap);
for (ifa = ifap; ifa != NULL; ifa = ifa->ifa_next)  
    /* Grab statistics name and values */
    strcpy(ifr.ifr_name, ifa->ifa_name);
    drvinfo.cmd         = ETHTOOL_GDRVINFO;
    ifr.ifr_data        = (caddr_t) &drvinfo;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    n_stats             = drvinfo.n_stats;
    strings->cmd        = ETHTOOL_GSTRINGS;
    strings->string_set = ETH_SS_STATS;
    strings->len        = n_stats;
    ifr.ifr_data        = (caddr_t) strings;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    stats->cmd          = ETHTOOL_GSTATS;
    stats->n_stats      = n_stats;
    ifr.ifr_data        = (caddr_t) stats;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    ifIndex = if_nametoindex(ifa->ifa_name);
    /* Iterate through statistics */
    for (i = 0; i < n_stats; i++)  
        /* Declare the appropriate index */
        strncpy(ethtoolStatName,
                (char *)&strings->data[i * ETH_GSTRING_LEN],
                ETH_GSTRING_LEN);
        ethtoolStatName[sizeof(ethtoolStatName) - 1] = '\0';
        ethtoolStatName_len = strlen(ethtoolStatName);
        /* Append this new variable to the container. */
        rowreq_ctx = ethtoolStatTable_allocate_rowreq_ctx();
        ethtoolStatTable_indexes_set(rowreq_ctx,
                    ifIndex,
                    ethtoolStatName, ethtoolStatName_len);
        rowreq_ctx->data.ethtoolStat.high = stats->data[i] >> 32;
        rowreq_ctx->data.ethtoolStat.low = (uint32_t)stats->data[i];
        CONTAINER_INSERT(container, rowreq_ctx);
     
 

See? It is pretty easy. I have only enumerated all the objects I wanted to expose. I didn t even have to convert statistics names to an OID, mib2c built a ethtoolStatTable_indexes_set() for this purpose. OpenHPI project provides an extensive documentation to write a sub-agent this way. Now, on the downside, to implement a single table, I have 14 generated files whose code style is usually unfit to integrate into an existing project : some people, me included, do not like automatically generated code, all the more when it is so verbose. The solution would be to use the new API without mib2c; unfortunately, while the documentation exists, all provided examples rely exclusively on mib2c tool. Unless you are able to grasp the new API without it, I would restrict its use to two cases:

• Writing a new module for inclusion into Net-SNMP. Since most modules are now written with the help of mib2c, there is no coding style problem.
• Writing a standalone sub-agent for some MIB. As a standalone project, you won t run into coding style problems either.

For other uses, let s have a look at the traditional API.

C with traditional API The traditional API is way simpler and even if it is quite old, it is easier to find associated examples. Here is how to initialize the agent, again, in a stripped-down version (the complete version is on GitHub):

static oid ethtool_oid[] =  1, 3, 6, 1, 4, 1, 39178, 100, 1 ;
static struct variable3 ethtool_vars[] =  
   1, ASN_COUNTER64, RONLY, ethtool_stat, 3,  1, 1, 2 
 ;
int main()
 
  netsnmp_enable_subagent();
  snmp_disable_log();
  snmp_enable_stderrlog();
  init_agent("ethtoolAgent");
  REGISTER_MIB("ethtoolStatTable", ethtool_vars,
               variable3, ethtool_oid);
  init_snmp("ethtoolAgent");
  while (1)
    agent_check_and_process(1);
 

The REGISTER_MIB() macro is used to register the provided root OID (ethtool_oid) to the master agent. It also registers handlers associated to various sub-OID through the ethtool_vars[] array. Each member of this array features the following fields:

• magic is an integer allowing one to discriminate OID handled by the same handler. For a table, this allows to handle several columns with the same handler since they share a common index.
• type is the type of the variable that will be returned.
• acl tells if the object is read-only (RONLY) or read-write (RWRITE).
• findVar is the handler associated to the object. Its task will be to find the appropriate object and return its value.
• The two last fields are the relative OID this entry is responsible for along with its length.

The prototype of ethtool_stat() handler function is the following:

static u_char*
ethtool_stat(struct variable *vp, oid *name, size_t *length,
             int exact, size_t *var_len, WriteMethod **write_method);

This function should return the value of the requested variable or NULL if no appropriate object was found. The value can be statically stored since it will be copied. Here is the description of the arguments:

• struct variable *vp is the structure the handle was associated with, except that vp->name is now a full OID, not a relative one. You can access vp->magic to determine which column or which scalar to handle if this handler is common to several objects.
• oid *name and size_t *length describe the OID requested. If you want to return another OID (for a GETNEXT request for example), you can modify them. name points to an OID array large enough to fit any OID you want to return.
• int exact tells if the requested OID should be exactly matched (case of GET or SET) or not (GETNEXT).
• size_t *var_len is used to tell the size of the returned object.
• WriteMethod **write_method is only used if the variable is modifiable. When a SET request is received, the API will call the handler with exact set to 1 and expects to get a pointer to the function that will handle the write operation³.

To implement this function, you are on your own. When relying on externally fetched data, you need to handle a cache yourself. I have used a red-black tree for this purpose (with code stolen from OpenBSD). This also enables an easy implementation of the GETNEXT operation. Here are three projects using this API to provide SNMP support:

• Keepalived is a VRRP daemon and a monitoring daemon for LVS clusters. I have added a complete SNMP support. The most interesting files are core/snmp.c, check/check_snmp.c and vrrp/vrrp_snmp.c. It features a tight integration into Keepalived event loop that should be reusable by other projects. It includes read-only support of all data structures (configuration, state and statistics), ability to send traps on events and a write support for some values.
• lldpd is an implementation of 802.1AB (LLDP) which is a protocol allowing an equipment to advertise itself to its neighbors on Ethernet level. 802.1AB comes with a huge MIB module and many extensions. lldpd provides a fairly complete read-only implementation of this MIB module. Some tables have complex indexes. Have a look at agent.c. There is currently no write support and no traps but lldpd features privilege separation and some code allows it to still use an Unix socket to speak with the master agent.
• Asterisk, an open source PBX, includes an AgentX sub-agent allowing one to grab some metrics. Look at res/snmp/agent.c.

Other solutions The manual page for snmpd.conf explores other ways to extend Net-SNMP agent under the section Extending agent functionality :

• exec and sh are deprecated. Use extend instead.
• pass is a bit simpler than pass_persist. The command is run once for each request. Such a command can easily be turned into a pass_persist script. Moreover, Net-SNMP will not answer any request while a pass command is running.
• proxy redirects requests to another SNMP agent. Unless you have some code that can only act as a complete SNMP agent, it is better to use AgentX instead. For programs using Net-SNMP API, turning an agent into an AgentX sub-agent is just one line of code. Since a sub-agent involves less code than a complete agent, it is more efficient and more reliable.
• smux has been superseeded by AgentX.
• perl and dlmod allow one to load and execute external code inside the running agent. Unless you have important performance requirements, it is better to use a sub-agent instead: if some bug happens, only the sub-agent will be affected, not the master agent. A sub-agent sticks more closely to the Unix philosophy: write programs that do one thing and do it well . However, Net-SNMP FAQ says:

Implementing the module in C within the main agent (directly or via dlmod) is probably the most efficient and reliable, closely followed by embedded perl (or python) extensions. These have the advantage of minimal overheads between the code implementing the MIB module, and the agent framework, and no inter-process communication issues.

---

1. Additional arguments for snmp* are usually required to specify a version and a community (for SNMPv1 and SNMPv2c) or a user (for SNMPv3). In all examples, I do not specify anything. I assume that ~/.snmp/snmp.conf contains the appropriate bits. Moreover, some outputs are truncated.
2. No allocations, no variable declarations and no error checkings. Do not do this at home!
3. If you want to look at an example of how to handle a SET request, look at the
   SNMP support for Keepalived. A SET operation is more complex because SNMP ensures that the
   SET operation is atomic across all provided variables. It uses a three-staged commit. The traditional API is very inefficient for this kind of operation because the object has to be located again for each step.

Software in the Public Interest, the mass-membership association that supports some great Free and Open Source Software projects, will hold a public board of directors meeting today, Thursday 9th February 2012 at 21:00 UTC. The day and time of SPI meetings has changed recently, so maybe different people can get to them now. They re held online, on irc.spi-inc.org (the OFTC network). The agenda for the meeting is open and available at http://www.spi-inc.org/meetings/agendas/2012/2012-02-09/ and there s been a bit of discussion of back office support on the SPI email list. I ll link to a meeting summary from the comments in this blog post after it happens.

I ve previously written Some Notes on DRBD [1] and a post about DRBD Benchmarking [2]. Previously I had determined that replication protocol C gives the best performance for DRBD, that the batch-time parameters for Ext4 aren t worth touching for a single IDE disk, that barrier=0 gives a massive performance boost, and that DRBD gives a significant performance hit even when the secondary is not connected. Below are the results of some more tests of delivering mail from my Postal benchmark to my LMTP server which uses the Dovecot delivery agent to write it to disk, the rates are in messages per minute where each message is an average of 70K in size. The ext4 filesystem is used for all tests and the filesystem features list is has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize .

	p4-2.8
Default Ext4	1663
barrier=0	2875
DRBD no secondary al-extents=7	645
DRBD no secondary default	2409
DRBD no secondary al-extents=1024	2513
DRBD no secondary al-extents=3389	2650
DRBD connected	1575
DRBD connected al-extents=1024	1560
DRBD connected al-extents=1024 Gig-E	1544

The al-extents option determines the size of the dirty areas that need to be resynced when a failed node rejoins the cluster. The default is 127 extents of 4M each for a block size of 508MB to be synchronised. The maximum is 3389 for a synchronisation block size of just over 13G. Even with fast disks and gigabit Ethernet it s going to take a while to synchronise things if dirty zones are 13GB in size. In my tests using the maximum size of al-extents gives a 10% performance benefit in disconnected mode while a size of 1024 gives a 4% performance boost. Changing the al-extents size seems to make no significant difference for a connected DRBD device. All the tests on connected DRBD devices were done with 100baseT apart from the last one which was a separate Gigabit Ethernet cable connecting the two systems. Conclusions For the level of traffic that I m using it seems that Gigabit Ethernet provides no performance benefit, the fact that it gave a slightly lower result is not relevant as the difference is within the margin of error. Increasing the al-extents value helps with disconnected performance, a value of 1024 gives a 4% performance boost. I m not sure that a value of 3389 is a good idea though. The ext4 barriers are disabled by DRBD so a disconnected DRBD device gives performance that is closer to a barrier=0 mount than a regular ext4 mount. With the significant performance difference between connected and disconnected modes it seems possible that for some usage scenarios it could be useful to disable the DRBD secondary at times of peak load it depends on whether DRBD is used as a really current backup or a strict mirror. Future Tests I plan to do some tests of DRBD over Linux software RAID-1 and tests to compare RAID-1 with and without bitmap support. I also plan to do some tests with the BTRFS filesystem, I know it s not ready for production but it would still be nice to know what the performance is like. But I won t use the same systems, they don t have enough CPU power. In my previous tests I established that a 1.5GHz P4 isn t capable of driving the 20G IDE disk to it s maximum capacity and I m not sure that the 2.8GHz P4 is capable of running a RAID to it s capacity. So I will use a dual-core 64bit system with a pair of SATA disks for future tests. The difference in performance between 20G IDE disks and 160G SATA disks should be a lot less than the performance difference between a 2.8GHz P4 and a dual-core 64bit CPU.

• [1] http://etbe.coker.com.au/2011/12/17/drbd-notes/
• [2] http://etbe.coker.com.au/2012/01/05/drbd-benchmarking/

Related posts:

1. DRBD Benchmarking I ve got some performance problems with a mail server that s...
2. Some Notes on DRBD DRBD is a system for replicating a block device across...
3. Ethernet bonding Bonding is one of the terms used to describe multiple...

As you might have noticed from my previous posts, we've moved phpMyAdmin code to github. Now I'm going to describe some things which might be useful for you if you are about do to similar switch. While using git, moving to another hosting provider should be pretty straightforward. Just add another remote, push code there and it should be done. On the other side you probably have dozens of things in your infrastructure which you need to adjust. So the first thing to do is write down what all services are connected to your current git repositories. Let me name some which might be worth checking:

• continuous integration server
• snapshots generating
• demo server (in case you're providing something like we do)
• commit notifications
• various statistics services such as cia.vc
• website generating
• references in wiki, website and documetation

Once you think you have remembered all important things (the less important will probably show up anyway, but majority of things should work), you're ready to make the move. I've decided to make the move in few steps. First push all data to new location, what can take some time. I'll get in more details to that later. In the same time I asked all contributors to give me their login information, so that I can allow them access to new repositories. Once all recently active developers were migrated, it was time to push all remaining commits to new git repositories and make the switch for real. Pushing git repo to another location, should be pretty easy. On the other side if you have many branches, it get's slightly more complex, I've ended up with following shell snippet (pushing all branches present in origin to github remote):

git branch -r   grep origin/   grep -v HEAD   sed 's@.*/@@'   while read b ; do git checkout $b ; git push github $b:$b ; done

Please ensure that you check output of this, because you may hit network problems somewhere in the middle and you end up with few branches than you expect. As the code is pretty much idempotent, you can safely run it several times until there is nothing to push. You should also push all tags to new location:

git push --tags github

Okay, we've all data on right place, so let's switch all our users to new location:

git remote set-url origin git@github.com:phpmyadmin/phpmyadmin.git # read/write
git remote set-url origin git://github.com/phpmyadmin/phpmyadmin.git # read only

Of course everybody has to do this manually. Next good thing is to let people know when they are using wrong repo (which will stay there for some time). Unfortunately there is AFAIK no way to warn them on pull, so let's warn at least on push:

$ cat > hooks/update
#!/bin/sh
echo "phpMyAdmin git repositories have moved to https://github.com/phpmyadmin"
exit 1

I think this is pretty much all. You can find some more bits in our Git migration wiki page. PS: Thanks to github for offering us hosting and sorry for breaking their branch displaying page by too many divergent branches.

Filed under: Coding English Phpmyadmin 4 comments Flattr this!

Recently I've discovered a subtle consequence of how the order in which dependencies are specified in debian actually matters. While re-factoring the code of dose3, I changed the order in which dependencies are considered by our sat solver (of edos-fame) . I witnessed a twofold performance loss just by randomizing how variables were presented to our sat solver. This highlights, on one hand, how our solver is strongly dependent on the structure of the problem and, on the other hand the standard practice of debian maintainers to assign an implicit priority in the disjunctive dependencies where the first is the most preferred packages (and maybe the most tested, at least dependency-wise). The basic idea of distcheck is to encode the dependencies information contained in a Packages file in CNF format and then to feed them to a sat solver to find out if a package has broken dependencies or if its dependencies are such that no matter what, it would be impossible to install this package on a user machine. Conflicts are encoded as binary clauses. So if package A conflicts with package B, I add a constraint they says "not (A and B)" , that is A and B cannot be considered together. The dependencies encoding associates to each disjunction of the depends field a clause that says "A implies B". If a package foo depends on A,B C,D , I'll add "foo implies A and B" & "foo implies C and D" . This encoding is pretty standard and it is easy to understand. The problem is how the sat solver will search for a solution to the problem "Is is possible to install package foo in an empty repository". The solver we use is very efficient and can easily deal with 100K packages or more. But in general is not very good at dealing with random CNF instances. The reason because edos-debcheck is so efficient lies in the way it exploits the structure of the sat problems. The goal of a sat solver is to find a model (that is a variable assignment list) that is compatible with the given set of constraints. So if my encoding of the debian repository is a set of constraints R, the installability problem boils down to add an additional constraint to R imposing that the variable associated to the package foo must be true, and then ask the solver to find a model to make this possible. This installation, in sat terms, would be just an array of variables that must be true in order to satisfy the given set of constraints. If you look at the logic problem as a truth table, the idea is to find a row in this table. This is the solution of your problem. Brute force of course is not an option and modern sat solvers use a number of strategies and heuristic to guide the search in the most intelligent way possible. Some of them try to learn from previous attempts, some of them, when they are lost try to pick a random variable to proceed. If we consider problems that have a lot of structure, award winning sat solver do not back track very much. By exploiting the structure of the problem, their algorithm allows them to considerably narrow down the search only to those variables that are really important to find a solution. All this long introduction was to talk about the solver that is currently used in edos-debcheck and distcheck (that is a rewrite of the edos-debcheck). So why dependency order matters ? If we consider any package, even if the policy does not specify any order in the dependencies, it's common practice to write disjunctive dependencies specifying the most probable and tested alternative first and all other, more esoteric choices later. Moreover real packages are considered *before* virtual packages. Since every developer seems be doing the same, some kind of structure might be hidden in the order in which dependencies are specified. Part of the efficiency of the the solver used in our tools is actually due to the fact that its search strategy is strongly dependent on the order in which literal are specified in each clause. Saying the package foo depends on A and B is "different" then saying it depends on B and A, even if semantically equivalent. In my tests, I found about a twofold performance loss if the order of literals is either randomized or inverted. This is clearly a specific problem related to our solvers, while other solvers might not be susceptible to such small structural changes. Sat competitions often employs some form of obfuscation strategies of well known problems with well known structures in order to make useless to encode a search strategy that exploits the specific structure of a problem. Since here we're not trying to win a sat competition, but to provide tool to solve a specific problem, we are of course very happy to exploit this structure.

This is the second entry in my series about squeeze release critical bug squashing. In response to my last blog post it was asked whether this is proper release critical bug squashing. Indeed there haven't been any patches or upload involved in this, only BTS handling, but this doesn't mean that these bugs weren't considered to be affecting squeeze. You can see this effort currently as weeding out the "wrong" bugs so that the list gets more useful and actually be able to ask maintainers to address the real issues. You can at least see in this graph that the blue line is going down constantly since the year change instead of rising up like before. And I hope I will be able to keep it below the green line for a while still. Also thanks to the release-team and ftpmasters that it was possible to keep the massbugs about waf binary blob not being preferred source for modification out of affecting squeeze and ignore it for the current stable release the required changes for those would rather be a fair bit intrusive for a stable update.

• 637213: doesn't fail in squeeze
• 648989: doesn't fail in squeeze
• 584568: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 624464: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 632106: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 632108: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 632262: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 632264: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 632265: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 632266: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 595298: squeeze has linux 2.6.32
• 613732: doesn't fail in squeeze
• 616694: doesn't fail in squeeze
• 620264: doesn't fail in squeeze
• 620573: doesn't fail in squeeze
• 621440: libdb4.8 not obsolete in squeeze
• 624593: doesn't fail in squeeze
• 626868: doesn't fail in squeeze
• 628475: libdb4.8 not obsolete in squeeze
• 629838: boost1.46 not in squeeze
• 628475: boost1.46 not in squeeze
• 631623: doesn't fail in squeeze
• 639990: doesn't fail in squeeze
• 632763: no libevent 2.0 in squeeze
• 627000: not affecting squeeze
• 658216: doesn't fail in squeeze
• 654870: doesn't fail in squeeze
• 655115: this is explicitly about testing
• 657092: working in squeeze
• 655710: doesn't fail in squeeze
• 644691: no nautilus 3 in squeeze

I am glad that I managed to keep it up and even have a nice margin in case I can't put any effort into it some day but still have more bugs squashed than days there are in the year so far. Currently I am at 60 bugs in 39 days. This gives a warm feeling. :) Enjoy!

/debian permanent link Comments: 3 Flattr this