Search Results: "will"

13 February 2014

Craig Small: Google doesn t get SPF

Someone has decided to use my email address for a spam source. They have even used google to relay it which, given Googles current policies seems like a winning idea. I keep getting emails from Google s servers with header lines like this: 
X-Original-Authentication-Results: mx.google.com; spf=hardfail (google.com: domain of csmall@small.dropbear.id.au does not designate 66.80.26.66 as permitted sender)
You don t say? You mean even though my SPF records do not include some dodgy server in California, even though Google knows I don t include this in my SPF records well we will let the email go through anyhow. SPF records mean that s where my email comes from. If the record has a -all at the end of it, like mine do, then it means don t accept it from anywhere else. The hardfail means Google sees the -all and still does nothing about it.
Related articles

12 February 2014

Mario Lang: Roughly 1500 source packages have possibly broken links in debian/control

There are currently roughly 1500 source packages in Debian which possibly (very likely actually) do have broken URLs in debian/control. While it is quite useful that we have VCS information and Homepage URLs in the Packages file these days, we also created a rather big source of bitrot. These URLs are typically paste-and-forget. Sure, people occasionally catch the fact that a homepage or VCS has moved, especially if they are active and in good contact with their upstreams. However, there are also other cases...
DUCK to the rescue! My coworker Simon Kainz has worked on a service that helps at least to track which URLs are currently broken. We've initially discussed some way to make this a part of Lintian, which is what I would have prefered. However, for good reasons, Lintian doesn't want to call out to the net by default, so these checks would likely not get run by many developers anyway. So Simon ended up creating DUCK - the Debian Url ChecKer which actually goes out to the net and verifies that all the Vcs-* and Homepage fields of debian/control are actually reachable. The frontend allows developers to search for packages they maintain, to quickly see if they have any URLs which are possibly broken. There is a slight chance of temporary network problems of course, so what DUCK does is to show the status of checks in the last few days, so that you quickly see if you are dealing with a typical false positive. First of all, thanks to Simon! I think DUCK is an excellent project for a future NM candidate. I actually already wanted to advocate him for DD, but we ended up on a website which suggested that new contributors should start by applying for DM these days, and only later go for DD. I find that actually quite strange, especially in Simons case, but well, we did not feel like argueing. Secondly, and thats also very important: whats needed to improve the overall quality of URLs in the package system is your attention! You can easily search for email addresses of maintainers or uploaders. Team members can create a bookmark entry that checks for problems in all packages maintained by the teams they are a member of. You just need to actually visit these pages from time to time. It would probably not be well received if we filed all these bugs at once :-). So we need you to care, since we don't want to generate too much noise regarding "just these broken URLs". Or are they? If your vcs fields are broken, debcheckout will not work properly. Which defeats the purpose of debcheckout. If your homepage URL is broken, packages.d.o will also have a wrong link. 1500 packages with broken URLs. Don't you think we can do better then that?

Don Armstrong: Working with Org-mode: Committing Changes Everywhere

I'm a huge fan of Org-mode, and I keep all of my org-mode files in git repositories which are under myrepos control. However, because I often make lots of changes to my agenda and notes, I hate having to manually visit each individual project and make changes to it. [And it's also annoying when I forget to commit a specific change and then have to try to get my laptop and desktop back into sync.] Luckily, myrepos can easily run a command in parallel in all of the repositories! The following "update_org_files" command will update all of my org-file containing repositories in parallel: 
#!/bin/bash
ORG_GREP='-e .org$ -e .org_archive$ -e .org_done$'
if [ "x$1" == "xdoit" ]; then
    if git status --porcelain -z   grep -z '^ M'   grep -zq $ORG_GREP; then
        git status --porcelain -z   grep -z '^ M'   grep -z $ORG_GREP   \
            sed -z 's/^ M//g'   \
            xargs -0 git commit -m'update org files'
        git push;
    fi; 
else
    emacsclient -n -e '(org-save-all-org-buffers)' >/dev/null 2>&1
    mr -d ~ -j5 run update_org_files doit;
fi;
An updated version of this lives in my git repository

11 February 2014

Erich Schubert: Debian chooses systemd as default init

It has been all over the place. The Debian CTTE has chosen systemd over upstart as default init system by chairman call. This decision was overdue, as it was pretty clear that the tie will not change, and thus it will be up to chairman. There were no new positions presented, and nobody was being convinced of a different preference. The whole discussion had been escalating, and had started to harm Debian. Some people may not want to hear this, but another 10 ballots and options would not have changed this outcome. Repating essentially the same decision (systemd, upstart, or "other things nobody prefers") will do no good, but turn out the same result, a tie. Every vote counting I saw happen would turn out this tie. Half of the CTTE members prefer systemd, the other half upstart. The main problems, however, are these: So please, everybody get back to work now. If there ever is enough reason to overturn this decision, there are formal ways to do this, and there are people in the CTTE (half of the CTTE, actually) that will take care of this. Until then, live with the result of a 0.1 votes lead for systemd. Instead of pursuing a destructive hate campaign, why not improve your favorite init system instead. Oh, and don't forget about the need to spell out a policy for init system support requirements of packages.

Sylvestre Ledru: Hello Planet Mozilla

Hello Planet Mozilla I started as a new employee a few weeks ago. I am working as Release Manager for the next releases of Firefox (both on desktop + android).
As a start, Lukas and I will be working on Firefox 29 (no pressure at all to start with Australis + Firefox account).
I grew up with the first versions of Netscape, followed the Phoenix/Firebird adventures and I have been a supporter of Firefox since then.
I am glad to join such an important project, especially in the perspective of the recent privacy scandals. I think that Mozilla is in the best position to communicate to the public, to denounce certain activities and provides technologies to bypass them. Besides that, let me introduce myself. I have been in free software for almost a decade.
My previous work as working on Scilab (from Inria to Scilab Enterprises) where I stayed for almost 8 years. Over the years, I worked on various development (C, C++, Java, Python, Scilab, etc), user experience, roadmap definition, release management, R and D / customer project management, management, board member, etc. During this period, I learnt tons of things from coding, release and product management, QA, etc. In parallel, I worked part time at IRILL, a free software research center (hosted by Inria). I was involved as community management ie organizing event like GNU Hacker meeting, FOSDEM, Euro LLVM, etc and recording them. I had the chance to meet plenty of people from the FLOSS world (including folks from Mozilla who suggested me to join the adventure). I am also involved in two communities: Debian and LLVM. The first one is Debian. I started to contribute in Debian to get Scilab properly packaged. 7 years
after, I am now maintaining
important packages like LLVM, Clang, libedit, emscripten (!) or BLAS/LAPACK, treasurer of the French representative association and organizing the French mini Debian Conference (Debconf) / Meetup. For the last couple of years, I contributed to LLVM, Clang and LLDB. I participated to the release 3.3 and 3.4. My technical contributions were mainly driven by the packaging, the nightly Debian/Ubuntu packages and also by the rebuild of the Debian archive using Clang (instead of gcc). And when I am not behind my laptop, I am teaching scuba diving with my girlfriend or hiking, mountaineering, ski touring, climbing, etc. Yeh, I am a bit hyperactive.
Original post blogged on b2evolution.

Andrew Pollock: [life] Day 14, Kindergarten and a make up swim class

Zoe woke up pretty early today, at around 5:30am. I also officially became a Xoogler at some point overnight. I used this transition as the opportunity to move from my Nexus 4 to my Nexus 5, so I could have a clean start on a phone that had never had work credentials on it. I started that process before breakfast, and mostly had a usable phone by the time we biked to Kindergarten. Zoe was a bit more like her usual clingy self at drop off today, so I spent a bit longer getting her settled. I got home and pretty much spent the rest of the day sorting out my phone. I can say many good things about Android, but seamless device migration isn't one of them. I'll probably factory reset my Nexus 4 tomorrow. I biked back to Kindergarten to pick up Zoe, and sure enough, she was fast asleep again. It turned out that she'd gone down about 30 minutes later today, so she was deeply asleep, and didn't take too kindly to being woken up. She was very sad, and it took forever just to leave the Kindergarten. We were well and truly the last ones out. Then she saw Megan having a trial tennis lesson at the tennis courts on the way to the bike, and wanted to do that too, and we burned some more time, while I tried to talk her out of that. Zoe had a make up swim class today to cover the one she'd missed on Saturday when she went down to Canberra with Sarah. That was at 3:15pm. We were a tad late for it. Weekday swim classes are much quieter than the weekend ones, so I used the opportunity to talk to one of the swim school staff about progressing Zoe to the next level. They conducted an assessment on the spot, and graduated her! She's moving from "Platypus" to "Octopus", starting next week. The class time options were pretty crappy though, something like 11:45am on Saturday, or a late afternoon weekday. We've opted for a weekday class after Kindergarten for the rest of this quarter and will try and get an earlier Saturday class next quarter. There's a reasonable gap between the end of Kindergarten and the start of swim class, so I'll have to figure out how best to use that time. I might make our library visit in between, even though it means a bit of back and forth. The upside is Saturday mornings are now wide open. We had a celebratory ice cream when we got home from swim class, and went out for a babyccino after dinner. I think it's important to celebrate achievements. I managed to get Zoe to bed slightly early tonight in light of her reduced nap time, but she just woke up briefly at 10:30pm. She seems to be particularly sensitive to mosquitoes, and has a bite on her forehead, right at the hairline, and it was giving her some grief. I gave her some antihistamine and she's gone back to bed fairly easily. Fingers crossed she sleeps okay for the rest of the night.

10 February 2014

Bits from Debian: skyrocketing how-can-i-help popcon count

This is a repost from Stefano Zacchiroli's post how-can-i-help by Lucas Nussbaum is one of the best things that happened in the area of attracting contributions to Debian in quite a while. It can be used both as a standalone tool to list opportunities for contributing to Debian which are related to your installed packages, and as an APT hook (which is also the default configuration) that at each upgrade will inform you of new contribution opportunities. how-can-i-help is great for newbies who are looking for ways to give back to Debian which are a good match for their skills: among other things, how-can-i-help shows bugs tagged "gift" related to packages you use. how-can-i-help is also great for experienced developers, as it allows them to find out, in a timely manner, that packages they use are in dire need of help: RC bugs, pending removals, adoptions needed, requests for sponsor, etc. (As highly unscientific evidence: I've noticed a rather quick turnover of RFA/O/ITA bugs on packages installed on my machine. I suspect how-can-i-help is somehow responsible for that, due to the fact that it increases awareness of ongoing package issues directly with the people using them.) So, if you haven't yet, please apt-get install how-can-i-help RIGHT NOW. I daresay that we should aim at installing how-can-i-help by default on all Debian machines, but that might be an ambitious initial goal. In the meantime I'll settle for making how-can-i-help's popcon count skyrocket. As of today, it looks like this: Alt how-can-i-help popularity contest graph 10/02/2014 which is definitely too low for my taste. Please spread the word about how-can-i-help. And let's see what we can collectively do to that graph. how-can-i-help is just a tiny teeny helper, but I'm convinced it can do wonders in liberating dormant contributions to the Debian Project.

Matthew Garrett: IBM's remote firmware configuration protocol

I spent last week looking into the firmware configuration protocol used on current IBM system X servers. IBM provide a tool called ASU for configuring firmware settings, either in-band (ie, running on the machine you want to reconfigure) or out of band (ie, running on a remote computer and communicating with the baseboard management controller - IMM in IBM-speak). I'm not a fan of using vendor binaries for this kind of thing. They tend to be large (ASU is a 20MB executable) and difficult to script, so I'm gradually reimplementing them in Python. Doing that was fairly straightforward for Dell and Cisco, both of whom document their configuration protocol. IBM, on the other hand, don't. My first plan was to just look at the wire protocol, but it turns out that it's using IPMI 2.0 to communicate and that means that the traffic is encrypted. So, obviously, time to spend a while in gdb.

The most important lesson I learned last time I did this was "Check whether the vendor tool has a debug option". ASU didn't mention one in its help text and there was no sign of a getopt string, so this took a little longer - but nm helpfully showed a function called DebugLog and setting a breakpoint on that in gdb showed it being called a bunch of the time, so woo. Stepping through the function revealed that it was checking the value of a variable, and looking for other references to that variable in objdump revealed a -l argument. Setting that to 255 gave me rather a lot of output. Nearby was a reference to --showsptraffic, and passing that as well gave me output like:

BMC Sent: 2e 90 4d 4f 00 06 63 6f 6e 66 69 67 2e 65 66 69
BMC Recv: 00 4d 4f 00 21 9e 00 00

which was extremely promising. 2e is IPMI-speak for an OEM specific command, which seemed pretty plausible. 4d 4f 00 is 20301 in decimal, which is the IANA enterprise number for IBM's system X division (this is required by the IPMI spec, it wasn't an inspired piece of deduction on my behalf). 63 6f 6e 66 69 67 2e 65 66 69 is ASCII for config.xml. That left 90 and 06 to figure out. 90 appeared in all of the traces, so it appears to be the command to indicate that the remaining data is destined for the IMM. The prior debug output indicated that we were in the QuerySize function, so 06 presumably means query the size. And this is supported by the response - 00 (a status code), 4d 4f 00 (the IANA enterprise number again, to indicate that the responding device is speaking the same protocol as you) and 21 9e 00 00 - or, in rational endianness, 40481, an entirely plausible size.

Once I'd got that far the rest started falling into place fairly quickly. 01 rather than 06 indicated a file open request, returning a four byte file handle of some description. 02 was read, 03 was write and 05 was close. Hacking this together with pyghmi meant I could open a file and read it. Success!

Well, kind of. I was getting back a large blob of binary. The debug trace showed calls to an EfiDecompress function, so on a whim I tried just decompressing it using the standard UEFI compression format. Shockingly, it worked and now I had a 345K XML blob and presumably many more problems than I'd previously had. Parsing the XML was fairly straightforward, and now I could retrieve the full set of config options, along with the default, current and possible values for them.

Making changes was pretty much just the reverse of this. A small XML blob containing the new values is compressed and written to asu_update.efi. One of the elements is a random identifier, which will then appear in another file called config_log along with a status. Just keep re-reading this until the value changes to CM_DONE and we're good.

The in-band configuration appears to be identical, but rather than sending commands over the wire they're send through the system's IPMI controller directly. Yes, this does mean that it's sending compressed XML through a single io port. Yes, I'd rather be drinking.

I've documented the protocol here and hope to be able to release this code before too long - right now I'm trying to nail down the interface a little, but it's broadly working.

comment count unavailable comments

Mario Lang: Neurofunkcasts

I have always loved Drum and Bass. In 2013 I rediscovered my love for Darkstep and Neurofunk, and found that these genres have developed quite a lot in the recent years. Some labels like Black Sun Empire and Evol Intent produce mixes/sets on a regular basis as podcasts these days. This article aggregates some neurofunk podcasts I like a lot, most recent first. Enjoy 33 hours and 57 minutes of fun with dark and energizing beats. Thanks to BSE Contrax and Evol Intent for providing such high quality sets. You can also see the Python source for the program that was used to generate this page.

Stefano Zacchiroli: apt-get install how-can-i-help

skyrocketing how-can-i-help popcon count how-can-i-help by Lucas Nussbaum is one of the best things that happened in the area of attracting contributions to Debian in quite a while. It can be used both as a standalone tool to list opportunities for contributing to Debian which are related to your installed packages, and as an APT hook (which is also the default configuration) that at each upgrade will inform you of new contribution opportunities. how-can-i-help is great for newbies who are looking for ways to give back to Debian which are a good match for their skills: among other things, how-can-i-help shows bugs tagged "gift" related to packages you use. how-can-i-help is also great for experienced developers, as it allows them to find out, in a timely manner, that packages they use are in dire need of help: RC bugs, pending removals, adoptions needed, requests for sponsor, etc. (As highly unscientific evidence: I've noticed a rather quick turnover of RFA/O/ITA bugs on packages installed on my machine. I suspect how-can-i-help is somehow responsible for that, due to the fact that it increases awareness of ongoing package issues directly with the people using them.) So, if you haven't yet, please apt-get install how-can-i-help RIGHT NOW. I daresay that we should aim at installing how-can-i-help by default on all Debian machines, but that might be an ambitious initial goal. In the meantime I'll settle for making how-can-i-help's popcon count skyrocket. As of today, it looks like this:
how-can-i-help popularity contest graph, 10/02/2014
which is definitely too low for my taste. Please spread the word about how-can-i-help. And let's see what we can collectively do to that graph. how-can-i-help is just a tiny teeny helper, but I'm convinced it can do wonders in liberating dormant contributions to the Debian Project.

Russell Coker: Fingerprints and Authentication

Dustin Kirkland wrote an interesting post about fingerprint authentication [1]. He suggests using fingerprints for identifying users (NOT authentication) and gives an example of a married couple sharing a tablet and using fingerprints to determine who s apps are loaded. In response Tollef Fog Heen suggests using fingerprints for lightweight authentication, such as resuming a session after a toilet break [2]. I think that one of the best comments on the issue of authentication for different tasks is in XKCD comic 1200 [3]. It seems obvious that the division between administrator (who installs new device drivers etc) and user (who does everything from playing games to online banking with the same privileges) isn t working, and never could work well particularly when the user in question installs their own software. I think that one thing which is worth considering is the uses of a signature. A signature can be easily forged in many ways and they often aren t checked well. It seems that there are two broad cases of using a signature, one is to enter into legally binding serious contract such as a mortgage (where wanting to sign is the relevant issue) and the other is cases where the issue doesn t matter so much (EG signing off on a credit card purchase where the parties at risk can afford to lose money on occasion for efficient transactions). Signing is relatively easy but that s because it either doesn t matter much or because it s just a legal issue which isn t connected to authentication. The possibility of serious damage (sending life savings or incriminating pictures to criminals in another jurisdiction) being done instantly never applied to signatures. It seems to me that in many ways signatures are comparable to fingerprints and both of them aren t particularly good for authentication to a computer. In regard to Tollef s ideas about lightweight authentication I think that the first thing that would be required is direct user control over the authentication required to unlock a system. I have read about some Microsoft research into a computer monitoring the office environment to better facilitate the user s requests, an obvious extension to such research would be to have greater unlock requirements if there are more unknown people in the area or if the device is in a known unsafe location. But apart from that sort of future development it seems that having the user request a greater or lesser authentication check either at the time they lock their session or by policy would make sense. Generally users have a reasonable idea about the risk of another user trying to login with their terminal so user should be able to decide that a toilet break when at home only requires a fingerprint (enough to keep out other family members) while a toilet break at the office requires greater authentication. Mobile devices could use GPS location to determine unlock requirements, GPS can be forged, but if your attacker is willing and able to do that then you have a greater risk than most users. Some users turn off authentication on their phone because it s too inconvenient. If they had the option of using a fingerprint most of the time and a password for the times when a fingerprint can t be read then it would give an overall increase in security. Finally it should be possible to unlock only certain applications. Recent versions of Android support widgets on the lock screen so you can perform basic tasks such as checking the weather forecast without unlocking your phone. But it should be possible to have different authentication requirements for various applications. Using a fingerprint scan to allow playing games or reading email in the mailing list folder would be more than adequate security. But reading the important email and using SMS probably needs greater authentication. This takes us back to the XKCD cartoon.
Related posts:
  1. SASL Authentication and Debian/Wheezy After upgrading a mail server to Debian/Unstable (which will soon...
  2. biometrics and passwords In a comment on my post more about securing an...

9 February 2014

Jan Dittberner: Going to Chemnitzer Linux-Tage 2014

This year I take care of organizing of the Debian booth at Chemnitzer Linux-Tage 2014 which has been approved a few days ago. The CLT is a yearly (mostly) german speaking Free Software community event which takes place on the weekend of 15th/16th march in Chemnitz (Germany). On the Linux-Live pages you find a lot of projects that will have a booth there and the talk schedule contains many interesting topics. There will also be a key signing event for which you can register until 11th of march. The Wiki page for the Event is already in a good shape. Many things are already organized, but we still have some items left. A lot of people from the Debian community have already told me that they will be there. We will have a Debian Wheezy BabelBox demonstration running on a VirtualBox host that Jan Wagner will provide as well as merchandising (Thanks to Alexander Wirt). Two talks from people on our Wiki page have been accepted by the CLT organizers too: I am happy to meet many nice people from the Free Software community in Chemnitz soon.

Erich Schubert: Minimizing usage of debian-multimedia packages

How to reduce your usage of debian-multimedia packages: As you might be aware, the "deb-multimedia" packages have seen their share of chaos. Originally, they were named "debian-multimedia", but it was then decided that they should better be called "deb-multimedia" as they are not an official part of Debian. The old domain was then grabbed by cybersquatters when it expired. While a number of packages remain indistributable for Debian due to legal issues - such as decrypting DVDs - and thus "DMO" remains useful to many desktop users, please note that for many of the packages, a reasonable version exists within Debian "main". So here is a way to prevent automatically upgrading to DMO versions of packages that also exist in Debian main: We will use a configuration option known as apt pinning. We will modify the priority of DMO package to below 100, which means they will not be automatically upgraded to; but they can be installed when e.g. no version of this package exists within Debian main. I.e. the packages can be easily installed, but it will prefer using the official versions. For this we need to create a file I named /etc/apt/preferences.d/unprefer-dmo with the contents: 
Package: *
Pin: release o=Unofficial Multimedia Packages
Pin-Priority: 123
As long as the DMO archive doesn't rename itself, this pin will work; it will continue to work if you use a mirror of the archive, as it is not tied to the URL. It will not downgrade packages for you. If you want to downgrade, this will be a trickier process. You can use aptitude for this, and start with a filter (l key, as in "limit") of ?narrow(~i,~Vdmo). This will only show packages installed where the version number contains dmo. You can now patrol this list, enter the detail view of each package, and check the version list at the end for a non-dmo version. I cannot claim this will be an easy process. You'll probably have a lot of conflicts to clear up, due to different libavcodec* API versions. If I recall correctly, I opted to uninstall some dmo packages such as ffmpeg that I do not really need. In other cases, the naming is slightly different: Debian main has handbrake, while dmo has handbrake-gtk. A simple approach is to consider uninstalling all of these packages. Then reinstall as needed; since installing Debian packages is trivial, it does not hurt to deinstall something, does it? When reinstalling, Debian packages will be preferred over DMO packages. I prefer to use DFSG-free software wherever possible. And as long as I can watch my favorite series (Tatort, started in 1970 and airing episode 900 this month) and the occasional DVD movie, I have all I need. An even more powerful aptitude filter to review installed non-Debian software is ?narrow(~i,!~ODebian), or equivalently ?narrow(?installed,?not(?origin(Debian))). This will list all package versions, which cannot be installed from Debian sources. In particular, this includes versions that are no longer available (they may have been removed because of security issues!), software that has been installed manually from .deb files, or any other non-Debian source such as Google or DMO. (You should check the output of aptitude policy that no source claims to be o=Debian that isn't Debian though). This filter is a good health check for your system. Debian packages receive a good amount of attention with respect to packaging quality, maintainability, security and all of these aspects. Third party packages usually failed the Debian quality check in at least one respect. For example, Sage isn't in Debian yet; mostly because it has so many dependencies, that making a reliable and upgradeable installation is a pain. Similarly, there is no Hadoop in Debian yet. If you look at Hadoop packaging efforts such as Apache Bigtop, they do not live up to Debian quality yet. In particular, the packages have high redundancy, and re-include various .jar copies instead of sharing access to an independently packaged dependency. If a security issue arises with any of these jars, all the Hadoop packages will need to be rebuilt. As you can see, it is usually not because of Debian being too strict or too conservative about free software when software is not packaged. More often than not, it's because the software in question itself currently does not live up to Debian packaging quality requirements. And of course sometimes because there is too little community backing the software in question, that would improve the packaging. If you want your software to be included in Debian, try to make it packaging friendly. Don't force-include copies of other software, for example. Allow the use of system-installed libraries where possible, and provide upgrade paths and backwards compatibility. Make dependencies optional, so that your software can be pacakged even if a dependency is not yet available - and does not have to be removed if the dependency becomes a security risk. Maybe learn how to package yourself, too. This will make it easier for you to push new features and bug fixes to your userbase: instead of requiring your users to manually upgrade, make your software packaging friendly. Then your users will be auto-upgraded by their operating system to the latest version. Update: If you are using APT::Default-Release or other pins, these may override above pin. You may need to use apt-cache policy to find out if the pins are in effect, and rewrite your default-release pin into e.g. 
Package: *
Pin: release a=testing,o=Debian
Pin-Priority: 912
for debugging, use a different priority for each pin, so you can see which one is in effect. Notice the o=Debian in this pin, which makes it apply only to Debian repositories.

Neil Williams: Debian ARMMP in LAVA using iMX53

The home lab is my LAVA development environment and I ve recently got two iMX53 Quick Start Boards working with a typical LAVA master image based on a Linaro build of Linux 3.1.0 for mx5 with a Ubuntu Oneiric Ocelot 11.10 rootfs: 
 3.1.0-1002-linaro-lt-mx5 (buildd@hubbard) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #13-Ubuntu PREEMPT Fri Dec 16 01:21:07 UTC 2011
As part of my Debian work, it was clearly time to look at a current, Debian, kernel and rootfs and as I m developing and testing on Debian unstable, this would necessarily mean testing the Debian ARMMP (multi-platform) kernel which replaces the mx5 build used in Wheezy. 
Linux version 3.12-1-armmp (debian-kernel@lists.debian.org) (gcc version 4.8.2 (Debian 4.8.2-10) ) #1 SMP Debian 3.12.9-1 (2014-02-01)
I will be scripting the creation of a suitable image for these tests and there are other changes planned in LAVA to make it easier to build suitable images, but it is useful to document just how I worked out how to build the first image. Manual build steps First, I ve already discovered that the u-boot on the iMX53 doesn t like ext4, so the first step was to prepare an ext3 filesystem for the image. With a SATA drive attached, it was also much better to use that than the SD card, at least for generating the image. I m also doing this natively, so I am working inside the booted master image. This is fine as the master is designed to manipulate test images, so the only package I needed to install on the LAVA master image was debootstrap. I had an empty SATA drive to play with for these tests, so first prepare an ext3 filesystem: 
# mkfs.ext3 /dev/sda1
# mkdir /mnt/sata
# mount /dev/sda1 /mnt/sata
# mkdir /mnt/sata/chroots/
Start the debootstrap: 
# apt-get update
# apt-get install debootstrap
# debootstrap --arch=armhf --include=linux-image-armmp \
 --verbose unstable \
 /mnt/sata/chroots/unstable-armhf http://ftp.uk.debian.org/debian
Various actions in this chroot will need proc, so mount it here: 
# chroot /mnt/sata/chroots/unstable-armhf
# mount proc -t proc /proc
# mount devpts -t devpts /dev/pts
# exit
You may also have to edit the apt sources the LAVA master image doesn t have editors installed, so either use echo or download an edited file. I m using: 
deb http://ftp.uk.debian.org/debian sid main
flash-kernel needs changes For the initial tests, I ve got to get this image to boot directly from u-boot, so flash-kernel is going to be needed inside the chroot and to get the iMX53 to work with the ARMMP kernel and Device Tree, flash-kernel will need an update which will mean a patch: 
# chroot /mnt/sata/chroots/unstable-armhf
# apt-get update
# apt-get install patch flash-kernel
# cp /usr/share/flash-kernel/db/all.db /home
# cd /home
# patch -p2
The patch itself goes through a couple of iterations. Initially, it is enough to use: 
 Machine: Freescale MX53 LOCO Board
-Kernel-Flavors: mx5
+Kernel-Flavors: armmp
+DTB-Id: imx53-qsb.dtb
+DTB-Append: yes
Later, once the device has booted with the ARMMP kernel, the Machine Id can be updated to distinguish it from the mx5 flavour (from /proc/cpuinfo) and use the model name from the Device Tree (/proc/device-tree/model): 
diff --git a/db/all.db b/db/all.db
index fab3407..41f6c78 100644
--- a/db/all.db
+++ b/db/all.db
@@ -62,6 +62,18 @@ U-Boot-Initrd-Address: 0x00800000
 Required-Packages: u-boot-tools
 Bootloader-Sets-Incorrect-Root: yes
 
+Machine: Freescale i.MX53 Quick Start Board
+Kernel-Flavors: armmp
+DTB-Id: imx53-qsb.dtb
+DTB-Append-From: 3.12
+Boot-DTB-Path: /boot/dtb
+U-Boot-Kernel-Address: 0x70008000
+U-Boot-Initrd-Address: 0x0
+Boot-Kernel-Path: /boot/uImage
+Boot-Initrd-Path: /boot/uInitrd
+Required-Packages: u-boot-tools
+Bootloader-Sets-Incorrect-Root: no
+
 Machine: Freescale MX53 LOCO Board
 Kernel-Flavors: mx5
 U-Boot-Kernel-Address: 0x70008000
(I will be filing this patch in a bug report against flash-kernel soon.) With that patched, update and run flash-kernel: 
# mv all.db /usr/share/flash-kernel/db/all.db
# flash-kernel
flash-kernel: installing version 3.12-1-armmp
Generating kernel u-boot image... done.
Taking backup of uImage.
Installing new uImage.
Generating initramfs u-boot image... done.
Taking backup of uInitrd.
Installing new uInitrd.
Installing new dtb.
# exit
LAVA overlays This will be a LAVA test image and it needs an overlay if you want a vanilla image, set up a passwd inside the chroot instead. 
# cd /mnt/sata/chroots/unstable-armhf/
# wget --no-check-certificate https://launchpad.net/~linaro-maintainers/+archive/overlay/+files/linaro-overlay-minimal_1112.2_all.deb
# wget --no-check-certificate https://launchpad.net/~linaro-maintainers/+archive/overlay/+files/linaro-overlay_1112.2_all.deb
# chroot /mnt/sata/chroots/unstable-armhf/
# dpkg -i linaro-overlay-minimal_1112.2_all.deb linaro-overlay_1112.2_all.deb
# rm linaro-overlay-minimal_1112.2_all.deb linaro-overlay_1112.2_all.deb
# exit
Changes to allow the chroot to boot Now the chroot needs setting up as a boot image: 
# chroot /mnt/sata/chroots/unstable-armhf/
# echo T0:23:respawn:/sbin/getty -L ttymxc0 115200 vt102 >> ./etc/inittab
# echo auto lo eth0 > ./etc/network/interfaces.d/mx53loco
# echo iface lo inet loopback >> ./etc/network/interfaces.d/mx53loco
# echo iface eth0 inet dhcp >> ./etc/network/interfaces.d/mx53loco
# apt-get clean
# umount /dev/pts
# umount /proc
# exit
Partitioning as a LAVA test image This would be enough as a single partition test image but, currently, LAVA expects a much more hard-wired image. Depending on the history of the device and the need for LAVA to be able to put fresh kernel builds together with a known rootfs, LAVA has used a separate /boot and / partition in the test image for nearly all boards. Standard LAVA test images for many boards (like the iMX53) also have a small unallocated space at the start of the SD card, so until I can get LAVA upstream to handle test images of arbitrary design, I m adapting the image to suit the expectations inside LAVA. Yes, I know but it s better to get something working before spending time fixing things to make it work better. It will be fixed, in time. So I needed to separate out the /boot contents from the rest of the rootfs whilst keeping the chroot itself in a state which I can easily update and use to create other images. 
# cd /mnt/sata/chroots/unstable-armhf/boot/
# tar -czf ../../boot.tar.gz ./*
# cd ..
# tar -czf ../root.tar.gz ./*
Now the test image file and its partitions: 
# dd if=/dev/zero of=/mnt/sata/images/empty/debian.img bs=1M count=1024
# cp /mnt/sata/images/empty/debian.img /mnt/sata/images/debian-unstable-armhf-armmp.img
# losetup /dev/loop0 /mnt/sata/images/debian-unstable-armhf-armmp.img
# parted /dev/loop0 -s unit mb mktable msdos
# parted /dev/loop0 -s unit mb mkpart primary 1 10
# parted /dev/loop0 -s unit mb mkpart primary 11 110
# parted /dev/loop0 -s unit mb mkpart primary 111 1024
I did make the mistake of using kpartx at this stage but there are many areas of confusion when translating the kpartx output to offsets for mount when parted is easier: 
# parted /dev/loop0 unit B -s print
Model:  (file)
Disk /dev/loop0: 1073741824B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start       End          Size        Type     File system  Flags
 1      1048576B    10485759B    9437184B    primary
 2      10485760B   110100479B   99614720B   primary
 3      110100480B  1024458751B  914358272B  primary
Use the Start numbers and use losetup to create the loop devices for each
partition 
# losetup -o 10485760 /dev/loop1 /dev/loop0
# losetup -o 110100480 /dev/loop2 /dev/loop0
# mkfs.vfat /dev/loop1
# mkfs.ext3 /dev/loop2
Now clean up the loop mountpoints: 
# losetup -d /dev/loop2
# losetup -d /dev/loop1
# losetup -d /dev/loop0
# losetup -a
losetup -a should return nothing. If it doesn t, investigate the contents of /dev/mapper and use dmsetup remove until losetup -a does report empty. Otherwise the subsequent stages will fail. Now deploy the /boot contents into the empty image: 
# mount -oloop,offset=10485760 /mnt/sata/images/debian-unstable-armhf-armmp.img /mnt/boot/
# pushd /mnt/boot/
# tar -xzf /mnt/sata/chroots/boot.tar.gz
# popd
# sync
# umount /mnt/boot/
and the / contents (removing the duplicate ./boot contents) 
# mount -oloop,offset=110100480 /mnt/sata/images/debian-unstable-armhf-armmp.img /mnt/root/
# pushd /mnt/root
# tar -xzf /mnt/sata/chroots/root.tar.gz
# rm ./boot/*
# popd
# sync
# umount /mnt/root
Check the image 
# mount -oloop,offset=10485760 debian-unstable-armhf-armmp.img /mnt/boot
# ls /mnt/boot
config-3.12-1-armmp  initrd.img-3.12-1-armmp  System.map-3.12-1-armmp  uImage  uInitrd  vmlinuz-3.12-1-armmp
# umount /mnt/boot
# mount -oloop,offset=110100480 debian-unstable-armhf-armmp.img /mnt/root
# ls /mnt/root
bin  boot  dev  etc  home  initrd.img  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vmlinuz
# ls /mnt/root/bin/auto-serial-console
/mnt/root/bin/auto-serial-console
# umount /mnt/root
# md5sum debian-unstable-armhf-armmp.img
Downloads Yes, I have put that file online, if you are interested. Do read the readme first though. Getting the image off the device 
# ip a
# python -m SimpleHTTPServer 80 2>/dev/null
then wget (just http://, IP address / and the filename), md5sum , finish with Ctrl-C. Depending on setup, it may be quicker to transfer the uncompressed
file over a LAN than to let the device compress it. Your main machine
will do the compression much faster, even with a larger download. (It also helps to not compress the image on the device, you can
test the mount offsets more easily and do check that the image
can be mounted with the offsets from parted.) Results 
# cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 5 (v7l)
Features	: swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc08
CPU revision	: 5
Hardware	: Freescale i.MX53 (Device Tree Support)
Revision	: 0000
Serial		: 0000000000000000

# uname -a
Linux imx53-02 3.12-1-armmp #1 SMP Debian 3.12.9-1 (2014-02-01) armv7l GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux unstable (sid)
Release:	unstable
Codename:	sid
Next! Yes, there is a lot to do from here on. Then there is the whole issue of actually making this multi-platform. After all, it is far from ideal that a multi-platform kernel package has to have platform-specific operations using flash-kernel at each update. So Grub on ARM is going to be on the list of things to investigate .

Thomas Goirand: Removal of XCP / Xen-API from Jessie and Sid

This is sad, but no choice, due to the lack of upstream support: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=738322 I just hope xenserver-core will work soon in Debian, so we can fix this unfortunate situation.

8 February 2014

Paul Tagliamonte: Upcoming Talks

Hello, World! I ve got two upcoming talks - one at LibrePlanet 2014 about OpenPGP, and one at PyCon US 2014 (in Canada) on Hy. Slides will be posted as I give the talks. Hope to see folks there!

Sam Hartman: IETF Takes on Challenge of Internet Monitoring

At IETF 88, we held a plenary discussion of how we could harden the Internet against ongoing monitoring and survalence. There were no significant surprises in what people said about monitoring. So, we had an opportunity to focus on what the IETF as a standards organization responsible for technical standards that drive the Internet can do about this problem. I think we made amazing progress. The IETF works by consensus. We discuss issues, and see if there s some position that can gain a rough consensus of the participants in the discussion. After a couple of hours of discussion, Russ Housley asked several consensus questions. The sense of the room is that we should treat these incidents of monitoring as an attack and include them in the threats we try and counter with security features in our protocols. The room believes that encryption, even without authentication has value in fighting these attacks. There was support for the idea of end-to-end encryption is valuable even when there are middle boxes. IETF decisions made in meetings are confirmed on public mailing lists, so the sense of the room is not final. Also, note that I did not capture the exact wording of the questions that were posed. This is huge. There is very strong organizational agreement that we re going to take work in this space as seriously. Now that we ve decided pervasive monitoring is an attack, anyone can ask how a proposed protocol (or change to a protocol) counters that attack. If it doesn t handle the attack and there is a way to address the attack, then we will be in a stronger position arguing the threat could be addressed. In addition, the commitment to encryption providing value without authentication will be useful in providing privacy and minimizing fingerprinting by passive attackers. The IETF is only one part of the solution. We can work on the standards that describe how systems interact. However, implementations, policy makers, operators and users will also play a role in securing the Internet against pervasive attacks.

Enrico Zini: original-sin

Original sin I feel that I was somewhat born innocent, an animal with sound, primal instincts. But if I remained that way, I wouldn't be able to function in a complex society, so I got education. Education taught me what is expected from me in order to be accepted by my peers[1]. Education taught me more than that: it also taught me to enjoy what a complex society can give me: art, science, history, philosophy, adding depth of meaning and correlations to my perceptions and memories. Education wasn't perfect, though. Some of my educators obsessed about some of the expectations, and gave me rules to follow that aren't really needed to interact with a society. Sit down for hours in silence without complaining. Don't talk back to figures of authority, even when they are abusing me. Don't ever feel that my efforts in a task have been enough, because there is always something more that I could have done. Do what people expect me to do, regardless of what I wish to do. Do what I need to do, not what I want to do. So I grew up with a set of arbitrary expectations that weren't needed to function in a society, and weren't in any way meeting any of my needs, yet I still felt them as a part of me, putting every effort into meeting them that I would put in making myself happy. I like to call this set of learned arbitrary, unneeded expectations "neurosis". I like this as an interpretation of the "original sin" myth: in order to go from an innocent animal to a member of a complex society, I acquired a set of neuroses that make me behave in a meaningless way. Or, rephrased going along with the myth, that rob me of my innocence. In some environments like BDSM, many have a name for an unnegotiated practice forced on a person: they call it "abuse". It made sense to accept that I have been abused many times while being educated, because my educators also had their own baggage of parasite expectations, of neuroses. And since I don't currently know how I can ever be sure that I freed myself from all my neuroses, I feel like I should accept that it is possible that I do and will abuse others; by accepting it as a possibility, I hope at least to be able to realise as soon as possible that I am doing it, and try to stop. I have now realised that my life is far simpler and more rewarding when expectations are negotiated between all the parties involved. I have recently spent a substantial amount of my energy in recognising and renegotiating many of the expectations and the neuroses that I had learnt while growing up, and, who knows, perhaps this effort will continue throughout my life. So there you go, an agnostic atheist freeing himself of his original sin, and regaining his paradise lost one step at a time, alive, on earth. And enjoying every moment of it. [1] I suspect that is why people educated in a high class society tend to be accepted more easily by a high class society: they conform to the right set of expectations. But I digress.

Mario Lang: boost::python and boost::variant

I have unsuccessfully tried to find a solution for the following problem on the internet several times. Now that I have come at least closer to a usable approach, I thought I'd document what I have found so that others trying to achieve a similar thing can use this as a starting point. Boost.Python offers a very nice and flexible way to interface C++ data types with Python. With just a few lines of code, and the proper linker flags, you get a Python importable shared object from your C++ compiler. This can be very productive. However, there is one aspect of C++ data types that I couldn't figure out how to interface with Python, which are C++ discriminated unions, or more specifically, heterogeneous containers. While Python has no problems with containers containing objects of different types, C++ does not make this very easy by default. Usually the problem is solved with a container of pointers to a base class, and various subclasses with virtual functions. However, this approach is not always practical, especially if the different types of objects in an heterogeneous container dont have many things in common. This is where discriminated unions come to the rescue. They basically behave like a normal union in C, but have an additional field which indicates the type of object currently stored in the union. Boost.Variant does exactly that, with a nice visitor interface added on top of it.
Heterogeneous containers in C++ If we put the boost::variant<> template inside a STL container like std::vector<>, the result is a heterogeneous container. For the purpose of illustration, lets implement such a container. The example below is deliberately simple. In reality, the various types allowed in your variant will probably have more fields then just one. 
#include <boost/variant.hpp>
#include <vector>
struct a   int x;  ;
struct b   std::string y;  ;
typedef boost::variant<a, b> variant;
typedef std::vector<variant> vector;
To ease creation of these two types of objects, we are going to write a few factory functions. We are going to wrap them in Python later on. 
variant make_variant()   return variant();  
vector make_vector()   return vector a(), b(), a() ;
Boost.Python Now lets create a Python module which exports the above functionality to Python. 
#include <boost/python/class.hpp>
#include <boost/python/def.hpp>
#include <boost/python/implicit.hpp>
#include <boost/python/init.hpp>
#include <boost/python/module.hpp>
#include <boost/python/object.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
vector_indexing_suite apparently needs operator== defined on the value_type of the container. In our case, this is our boost::variant<a, b> type. Luckily, boost::variant<> already provides operator==. However, that operator== relies on operator== being defined for the underlying types. Since equality comparison is probably useful for other things as well, lets just create operator== for our two classes a and b. 
bool operator==(a const &lhs, a const &rhs)   return lhs.x == rhs.x;  
bool operator==(b const &lhs, b const &rhs)   return lhs.y == rhs.y;
Convert a boost::variant<> to PyObject * Boost.Python needs a way to convert our discriminated union to a Python object. This code relies on Python class definitions being present for all underlying variant types. We will define them later. 
struct variant_to_object : boost::static_visitor<PyObject *>  
  static result_type convert(variant const &v)  
    return apply_visitor(variant_to_object(), v);
   
  template<typename T>
  result_type operator()(T const &t) const  
    return boost::python::incref(boost::python::object(t).ptr());
   
 ;
And finally, lets create our Python module. 
BOOST_PYTHON_MODULE(bpv)  
  using namespace boost::python;
  class_<a>("a", init<a>()).def(init<>()).def_readwrite("x", &a::x);
  class_<b>("b", init<b>()).def(init<>()).def_readwrite("y", &b::y);
  to_python_converter<variant, variant_to_object>();
  implicitly_convertible<a, variant>();
  implicitly_convertible<b, variant>();
  def("make_variant", make_variant);
  class_<vector>("vector").def(vector_indexing_suite<vector, true>());
  def("make_vector", make_vector);
Compiling Lets create a shared object for Python. 
$ g++ -std=c++11 -fPIC -shared $(python-config --includes) -o bpv.so file.cpp -lboost_python
Running We can load the module into Python and see what it does. 
>>> import bpv
>>> variant=bpv.make_variant()
>>> variant
<bpv.a object at 0x7f06bb2130c0>
>>> variant.x
0
>>> variant.x=2
>>> variant.x
2
Nice. We can access the underlying type, and even modify it. Lets see how our heterogeneous container wrapping code behaves. 
>>> vector=bpv.make_vector()
>>> vector
<bpv.vector object at 0x7f20693289d0>
>>> len(vector)
3
>>> list(vector)
[<bpv.a object at 0x7f20693190c0>, <bpv.b object at 0x7f20693193d0>, <bpv.a object at 0x7f2069319440>]
So far, so good. This will at least make it possible to convert heterogeneous containers from C++ to Python, which was my initial goal. Unfortunately, contained objects are not treated as references. Whenever retrieved, we get a copy. So in-place modification does not work. 
>>> vector[0].x
0
>>> vector[0].x=2
>>> vector[0].x
0
However, we can override an existing element with a modified copy. 
>>> e0=vector[0]
>>> type(e0)
<class 'bpv.a'>
>>> e0.x = 2
>>> vector[0] = e0
>>> vector[0].x
2
And we can also use the append and extend methods of Python containers. 
>>> len(vector)
3
>>> vector.extend(vector)
>>> vector.append(bpv.a())
>>> len(vector)
7
>>> len(filter(lambda x: type(x)==bpv.b, vector))
2
>>> len(filter(lambda x: type(x)==bpv.a, vector))
5
>>> map(lambda x: x.x, filter(lambda x: type(x)==bpv.a, vector))
[2, 0, 2, 0, 0]
All that is missing for a perfect world is reference semantics for container elements. If anyone has a hint on how to achieve this, please let me know.

Sam Hartman: Debian: Init Interfaces Our Users *and* Free Software

Recently, I ve been watching the Debian Schadenfreude related to init systems. For those who do not know, Debian is trying to decide whether Systemd or Upstart will be used to start software on Debian. There are two other options, although I think Systemd and Upstart are the main contenders. Currently the Technical Committee is considering the issue, although there s some very real chance that the entire project will get dragged through this swamp with the general resolution process. This is one of those discussions that proves that Debian truly is a community: if we didn t have something really strong holding us together we d never put up with something this painful. Between the accusations that our users now know the persecution European pagans faced at the hands of Christians as the Systemd priests drive all before them, a heated discussion of how the Debian voting system interacts with this issue, two failed calls for votes, a discussion of conflicts of interests, and a last-minute discussion of whether the matter had been appropriately brought before the Technical committee (and if so, under what authority), there s certainly schadenfreude to go around if you re into that sort of thing. However, through all this, the Technical Committee has been doing great work understanding the issues; it has been a pleasure to watch their hard work from the side lines. I d like to focus on one key question they ve found: how tightly can other software depend on the init system. Each init system offers some nice features. Upstart has an event triggering model. Systemd manages login sessions and at least in my opinion has a really nice service file format. Can I take advantage of these in my software. If I do, then users of my software might need to use a particular init system. Ian Jackson argues that we should give our users the choice of what init system to run. He reminds us that Debian is a community that supports diversity and diverse goals. We support multiple desktop systems, web browsers, etc. Wherever we can, we support people working on their goals and give developers the opportunity to make it work. When developers succeed, we give our users the opportunity to choose from among the options that have been developed. I think of Ian s argument as an appeal to part of our Social Contract. Clause 4 of the social contract begins:
Our priorities are our users and free software.
We will be guided by the needs of our users and the free software community. We will place their interests first in our priorities. We will support the needs of our users for operation in many different kinds of computing environments.
Ian is right that our users will be served by choice in init systems. However, this trade off needs to be balanced against the needs of the free software community. Diversity is an important goal but it should not come at the price of stagnation and innovation. If I want to avoid using init scripts because they don t provide restart on failure, because they are hard to write correctly, and because they don t provide access to modern security isolation techniques, I should be able to do that. If Systemd service files provide a superior solution, I should be able to work toward using them. If the desire for init system diversity shuts down my ability to find like-minded people who want to take advantage of improvements in init systems and work towards that goal, then we ve significantly damaged Debian as a forum for technical cooperation. Early in my work as a Debian Developer, Anthony Towns taught me an interesting principle. Those who do the work get significant influence over how and what gets done. Ian s right that we need to enable people to work towards choice in init systems. Those like-minded people need to be given a chance to find each other and pursue their goal. However the cost of success should be on the shoulders of those who value choice in init systems. It should not come at the cost of preventing people from depending on improvements in init systems. The best proposal I ve seen for balancing this is to enumerate stable interfaces that people want to use. Things like the logind interface or my favorite the service file interface. It needs to be possible to make another implementation of the interface. The interface needs to be stable enough that a dedicated team could have a chance of catching up with an implementation. At some point during the release cycle such interfaces would need to be frozen. However, I don t think it is reasonable to mandate that there are multiple implementations of the interface, only that there could be. The point is to give people a chance to work towards diversity in init systems, not to force it down peoples throats kicking and screaming until they leave the project or ignore our processes. Steve Langasek and Colin Watson seem to be working towards this. It s possible there s another approach besides interfaces. My main concern is the same as Ian s: maintain Debian as a forum for people to pursue their goals and work together. I suspect we see the conflict in these goals differently. I hope that we as a project can explore this conflict and figure out where we have common ground. I commit to exploring how we can work towards init choice in my frameworks; I ask those who prioritize init choice to commit to explore how we can take advantage of new features in their framework.

Next.

Previous.