Search Results: "ema"

17 February 2014

Martin Zobel-Helas: Have YOU registered Services at Debian Services Census?

Enrico and me have just started a census on Debian Services floating around, and we want YOU to participate!

The Debian Services Census is an attempt to gather detailed information about software services for the Debian community. It doesn't matter where these services live or who provides them. For now we are only interested in their existence.

Services can be anything from the mail forwarding service for @debian.org addresses that lives on master.debian.org operated by DSA, web applications, email bots, command line scripts you can run on debian.org machines, automated package checkers that report bugs to people, Debian-related services that people run on their own infrastructure like screenshots.debian.net and son on. Have a look at https://wiki.debian.org/Services for more examples.

If you think it's a service, then we think it's a service too and we want to know about it. Feel free to contribute data whether you are the maintainer of that service or not.

Here is how to let us know about the service:
  • please visit https://wiki.debian.org/Services and verify if the service is already listed there
  • if the service you had in mind is *not* listed, please create an entry for it using the "Add a new service" button.
If you are the maintainer of a service, we also encourage you to subscribe to the debian-services-admin@lists.debian.org mailing list, which is a low-traffic list and should be the contact point for inter-service communication and coordination.

Update: Up to now, 51 services have been registered (most of them within 24 hours after the d-d-a mail, WOW), but we are sure that there are a lot of more services, that still need to be registered. Help us to get a complete list!

Andrew Pollock: [life] Day 20, Kindergarten, rocket building

Today was another Kindergarten day. Zoe was complaining about being sweaty before we even left the house this morning, so I gave her the choice of going by car or by bike, and she chose car, which was fine by me. It's been stinking hot the last few days. Summer is going out with a bang. Drop off went pretty smoothly. It was another morning where she was happy to have me hang out with her for a little bit, and then I was allowed to leave without any issues. I decided to use this Monday as a Debian day as planned. I mostly ran into a snag with my email setup, but I did manage to do some bug triage and upload one package. It had been hot all day, and I was feeling sapped by it, so I picked up Zoe in the car as well. I was surprised to find she wasn't asleep and hadn't had a nap at all. She seemed in good spirits despite this. Megan's Dad has quit his job, so I don't need to pick her up any more on Mondays and Wednesdays. We dropped by the post office to check my PO box on the way home. Zoe enjoyed peering through the mailbox into the post office. Today's TV diversion was to build a rocket ship out of one of the big moving boxes I've kept aside from the move back. Zoe had great fun blinging up the inside and outside with various craft "gems", and drew a bunch of aliens on the outside. I could have quite easily strung this out until Sarah arrived to pick her up, but I let her watch a little bit of TV right at the end. It was a pretty good afternoon. I liked how Zoe ran with the rocket ship idea and added her own embellishments all by herself. After Sarah picked up Zoe I went for a run. I'd intended to do 10km, but the heat took it out of me, and I lacked the willpower, so I stopped at 5km. After that I had the sweatiest yoga class yet.

16 February 2014

Andrew Pollock: [tech] Crypto is hard, let's go shopping

(Or "Today's yak shaving exercise") I'm trying to get back into some Debian work today, which involved trying to send an email from my laptop, relayed through daedalus, which is in a colo facility. Then I started shaving my yak. daedalus rejected the email, because the relay attempt wasn't authenticated. It wasn't authenticated because Exim on my laptop wasn't happy about the Diffie-Hellman parameters Sendmail on daedalus was advertising, and had failed STARTTLS, and like a sensible person, I only allow encrypted authentication. Along the way, I figured I'd need to address the fact that the host certificate has expired long ago (truth is, I probably just needed to fix the Diffie-Hellman parameters file), but my whole CA certificate expired over a year ago. I couldn't find the encrypted USB key with my CA on it, so I decided to cut my losses and start from scratch. It's good to see OpenSSL hasn't got any more user friendly with time. I futzed around and generated a new CA key and certificate, a new CSR for Sendmail, signed it, did the appropriate sacrifices, and confirmed that STARTTLS now worked using OpenSSL s_client. Still couldn't send an email. Still had some bleating in the Exim logs: 
2014-02-17 11:29:26 1WFD1M-00086p-KI TLS error on connection to daedalus.andrew.net.au [203.33.60.161] (gnutls_handshake): The Diffie-Hellman prime sent by the server is not acceptable (not long enough).
Back on daedalus, /etc/mail/tls/sendmail-common.prm has been an empty file since time immemorial. I managed to find something in Google's cache suggesting I should put something in it with openssl dhparam -out /etc/mail/tls/sendmail-common.prm -2 1024 I initially went for 4096 bits, because it must be better, right? I'm still waiting for the key generation to come back, while I'm writing this. At least it warned me in advance it was going to take a long time. How thoughtful. What would be more thoughtful was verifying the write permission to the file before the long operation so it could fail fast, rather. Sigh. I ended up giving up on 4096 and sticking with 1024, because I was getting old waiting. Yak shaved, I can send mail from my laptop now.

Evgeni Golov: diffing configuration files made easy

Let s assume you are a sysadmin and have to debug a daemon giving bad performance on one machine, but not on the other. Of course, you did not setup either machine, have only basic knowledge of the said daemon and would really love to watch that awesome piece of cinematographic art with a bunch of friends and a couple of beers. So it s like every day, right? The problem with understanding running setups is that you often have to read configuration files. And when reading one is not enough, you have to compare two or more of them. Suddenly, a wild problem occurs: order and indentation do not matter (unless they do), comments are often just beautiful noise and why the hell did that guy smoke/drink/eat while explicitly setting ALL THE OPTIONS to their defaults before actually setting them as he wanted. If you are using diff(1), you probably love to read a lot of differences, which are none in reality. Want an example? [foo]
bar = bar
foo = foo and # settings for foo
[foo]
# foo is best
foo = foo
# bar is ok here, FIXME?
bar = bar and [foo]
foo = x
bar = x
[foo]
foo = foo
bar = bar are actually the same, at least for some parsers. XTaran suggested using something like wdiff or dwdiff, which often helps, but not in the above case. Others suggested vimdiff, which is nice, but not really helpful here either. As there is a problem, and I love to solve these, I started a small new project: cfgdiff. It tries to parse two given files and give a diff of the content after normalizing it (merging duplicate keys, sorting keys, ignoring comments and blank lines, you name it). Currently it can parse various INI files, JSON, YAML and XML. That s probably not enough to be the single diff tool for configuration files, but it is quite a nice start. And you can extend it, of course ;)

0 comment(s) this blog is flattr enabled

Craig Small: psmisc 22.21 Released

Today as it was raining and I couldn t do much gardening, psmisc version 22.21 was released. The files are located up on sourceforge at https://sourceforge.net/projects/psmisc/files/latest/download or at your favorite distro repository soon. Once again, thanks to all patch submitters, bug reports and translators for all their help in getting this out. Apologies to the translation teams for having two alpha versions. So what does psmisc 22.21 bring you? Amongst a lot of minor bug fixes it has: The Debian psmisc package should be out in the next few hours.

14 February 2014

Gergely Nagy: I love Free Software

I love Free Software! I love Free Software, and the people behind it. Even though I have been a free software user and developer for more than a decade now, there's still more to be amazed at, still more to contribute, and still more to learn. I love the Free Software community, because it never ceases to amaze me. It's not just about the technical side, but the social, human side of it, too!During the past year, I came to another milestone in my career: as of july first, I am a Free Software developer, and enjoying every moment of it. This is a feat that could not have been done without there being demand for what I do, without the free software community as a whole. For that, I owe you all a big thank you, you make it possible for me to make my dream come true. I can only hope that through my work, and through my example, I can contribute back enough.To this day, I take delight in contributing to free software, be it a Lisp on Python (or projects built in it), any of my own projects, syslog-ng, Google Summer of Code mentoring, Debian work - any and all of that are great fun. Even processing NEW is (I really should do that more often)! It's an endless source of not only entertainment and knowledge, but inspiration too.Whenever I read something as amazing as Russ Allbery's mail on the Debian technical committee impasse, I realize just how amazing people we have in our community, and how much we all can learn from them, and how much we owe them. This is yet another reason why I feel that contributing to Free Software is a passion many can share, because it teaches us good values too, atop of being technically excellent.I love Free Software, because without it, I would be lost. Thank you.

13 February 2014

Craig Small: Google doesn t get SPF

Someone has decided to use my email address for a spam source. They have even used google to relay it which, given Googles current policies seems like a winning idea. I keep getting emails from Google s servers with header lines like this: 
X-Original-Authentication-Results: mx.google.com; spf=hardfail (google.com: domain of csmall@small.dropbear.id.au does not designate 66.80.26.66 as permitted sender)
You don t say? You mean even though my SPF records do not include some dodgy server in California, even though Google knows I don t include this in my SPF records well we will let the email go through anyhow. SPF records mean that s where my email comes from. If the record has a -all at the end of it, like mine do, then it means don t accept it from anywhere else. The hardfail means Google sees the -all and still does nothing about it.
Related articles

12 February 2014

Mario Lang: Roughly 1500 source packages have possibly broken links in debian/control

There are currently roughly 1500 source packages in Debian which possibly (very likely actually) do have broken URLs in debian/control. While it is quite useful that we have VCS information and Homepage URLs in the Packages file these days, we also created a rather big source of bitrot. These URLs are typically paste-and-forget. Sure, people occasionally catch the fact that a homepage or VCS has moved, especially if they are active and in good contact with their upstreams. However, there are also other cases...
DUCK to the rescue! My coworker Simon Kainz has worked on a service that helps at least to track which URLs are currently broken. We've initially discussed some way to make this a part of Lintian, which is what I would have prefered. However, for good reasons, Lintian doesn't want to call out to the net by default, so these checks would likely not get run by many developers anyway. So Simon ended up creating DUCK - the Debian Url ChecKer which actually goes out to the net and verifies that all the Vcs-* and Homepage fields of debian/control are actually reachable. The frontend allows developers to search for packages they maintain, to quickly see if they have any URLs which are possibly broken. There is a slight chance of temporary network problems of course, so what DUCK does is to show the status of checks in the last few days, so that you quickly see if you are dealing with a typical false positive. First of all, thanks to Simon! I think DUCK is an excellent project for a future NM candidate. I actually already wanted to advocate him for DD, but we ended up on a website which suggested that new contributors should start by applying for DM these days, and only later go for DD. I find that actually quite strange, especially in Simons case, but well, we did not feel like argueing. Secondly, and thats also very important: whats needed to improve the overall quality of URLs in the package system is your attention! You can easily search for email addresses of maintainers or uploaders. Team members can create a bookmark entry that checks for problems in all packages maintained by the teams they are a member of. You just need to actually visit these pages from time to time. It would probably not be well received if we filed all these bugs at once :-). So we need you to care, since we don't want to generate too much noise regarding "just these broken URLs". Or are they? If your vcs fields are broken, debcheckout will not work properly. Which defeats the purpose of debcheckout. If your homepage URL is broken, packages.d.o will also have a wrong link. 1500 packages with broken URLs. Don't you think we can do better then that?

Don Armstrong: Working with Org-mode: Committing Changes Everywhere

I'm a huge fan of Org-mode, and I keep all of my org-mode files in git repositories which are under myrepos control. However, because I often make lots of changes to my agenda and notes, I hate having to manually visit each individual project and make changes to it. [And it's also annoying when I forget to commit a specific change and then have to try to get my laptop and desktop back into sync.] Luckily, myrepos can easily run a command in parallel in all of the repositories! The following "update_org_files" command will update all of my org-file containing repositories in parallel: 
#!/bin/bash
ORG_GREP='-e .org$ -e .org_archive$ -e .org_done$'
if [ "x$1" == "xdoit" ]; then
    if git status --porcelain -z   grep -z '^ M'   grep -zq $ORG_GREP; then
        git status --porcelain -z   grep -z '^ M'   grep -z $ORG_GREP   \
            sed -z 's/^ M//g'   \
            xargs -0 git commit -m'update org files'
        git push;
    fi; 
else
    emacsclient -n -e '(org-save-all-org-buffers)' >/dev/null 2>&1
    mr -d ~ -j5 run update_org_files doit;
fi;
An updated version of this lives in my git repository

10 February 2014

Matthew Garrett: IBM's remote firmware configuration protocol

I spent last week looking into the firmware configuration protocol used on current IBM system X servers. IBM provide a tool called ASU for configuring firmware settings, either in-band (ie, running on the machine you want to reconfigure) or out of band (ie, running on a remote computer and communicating with the baseboard management controller - IMM in IBM-speak). I'm not a fan of using vendor binaries for this kind of thing. They tend to be large (ASU is a 20MB executable) and difficult to script, so I'm gradually reimplementing them in Python. Doing that was fairly straightforward for Dell and Cisco, both of whom document their configuration protocol. IBM, on the other hand, don't. My first plan was to just look at the wire protocol, but it turns out that it's using IPMI 2.0 to communicate and that means that the traffic is encrypted. So, obviously, time to spend a while in gdb.

The most important lesson I learned last time I did this was "Check whether the vendor tool has a debug option". ASU didn't mention one in its help text and there was no sign of a getopt string, so this took a little longer - but nm helpfully showed a function called DebugLog and setting a breakpoint on that in gdb showed it being called a bunch of the time, so woo. Stepping through the function revealed that it was checking the value of a variable, and looking for other references to that variable in objdump revealed a -l argument. Setting that to 255 gave me rather a lot of output. Nearby was a reference to --showsptraffic, and passing that as well gave me output like:

BMC Sent: 2e 90 4d 4f 00 06 63 6f 6e 66 69 67 2e 65 66 69
BMC Recv: 00 4d 4f 00 21 9e 00 00

which was extremely promising. 2e is IPMI-speak for an OEM specific command, which seemed pretty plausible. 4d 4f 00 is 20301 in decimal, which is the IANA enterprise number for IBM's system X division (this is required by the IPMI spec, it wasn't an inspired piece of deduction on my behalf). 63 6f 6e 66 69 67 2e 65 66 69 is ASCII for config.xml. That left 90 and 06 to figure out. 90 appeared in all of the traces, so it appears to be the command to indicate that the remaining data is destined for the IMM. The prior debug output indicated that we were in the QuerySize function, so 06 presumably means query the size. And this is supported by the response - 00 (a status code), 4d 4f 00 (the IANA enterprise number again, to indicate that the responding device is speaking the same protocol as you) and 21 9e 00 00 - or, in rational endianness, 40481, an entirely plausible size.

Once I'd got that far the rest started falling into place fairly quickly. 01 rather than 06 indicated a file open request, returning a four byte file handle of some description. 02 was read, 03 was write and 05 was close. Hacking this together with pyghmi meant I could open a file and read it. Success!

Well, kind of. I was getting back a large blob of binary. The debug trace showed calls to an EfiDecompress function, so on a whim I tried just decompressing it using the standard UEFI compression format. Shockingly, it worked and now I had a 345K XML blob and presumably many more problems than I'd previously had. Parsing the XML was fairly straightforward, and now I could retrieve the full set of config options, along with the default, current and possible values for them.

Making changes was pretty much just the reverse of this. A small XML blob containing the new values is compressed and written to asu_update.efi. One of the elements is a random identifier, which will then appear in another file called config_log along with a status. Just keep re-reading this until the value changes to CM_DONE and we're good.

The in-band configuration appears to be identical, but rather than sending commands over the wire they're send through the system's IPMI controller directly. Yes, this does mean that it's sending compressed XML through a single io port. Yes, I'd rather be drinking.

I've documented the protocol here and hope to be able to release this code before too long - right now I'm trying to nail down the interface a little, but it's broadly working.

comment count unavailable comments

Mario Lang: Neurofunkcasts

I have always loved Drum and Bass. In 2013 I rediscovered my love for Darkstep and Neurofunk, and found that these genres have developed quite a lot in the recent years. Some labels like Black Sun Empire and Evol Intent produce mixes/sets on a regular basis as podcasts these days. This article aggregates some neurofunk podcasts I like a lot, most recent first. Enjoy 33 hours and 57 minutes of fun with dark and energizing beats. Thanks to BSE Contrax and Evol Intent for providing such high quality sets. You can also see the Python source for the program that was used to generate this page.

Russell Coker: Fingerprints and Authentication

Dustin Kirkland wrote an interesting post about fingerprint authentication [1]. He suggests using fingerprints for identifying users (NOT authentication) and gives an example of a married couple sharing a tablet and using fingerprints to determine who s apps are loaded. In response Tollef Fog Heen suggests using fingerprints for lightweight authentication, such as resuming a session after a toilet break [2]. I think that one of the best comments on the issue of authentication for different tasks is in XKCD comic 1200 [3]. It seems obvious that the division between administrator (who installs new device drivers etc) and user (who does everything from playing games to online banking with the same privileges) isn t working, and never could work well particularly when the user in question installs their own software. I think that one thing which is worth considering is the uses of a signature. A signature can be easily forged in many ways and they often aren t checked well. It seems that there are two broad cases of using a signature, one is to enter into legally binding serious contract such as a mortgage (where wanting to sign is the relevant issue) and the other is cases where the issue doesn t matter so much (EG signing off on a credit card purchase where the parties at risk can afford to lose money on occasion for efficient transactions). Signing is relatively easy but that s because it either doesn t matter much or because it s just a legal issue which isn t connected to authentication. The possibility of serious damage (sending life savings or incriminating pictures to criminals in another jurisdiction) being done instantly never applied to signatures. It seems to me that in many ways signatures are comparable to fingerprints and both of them aren t particularly good for authentication to a computer. In regard to Tollef s ideas about lightweight authentication I think that the first thing that would be required is direct user control over the authentication required to unlock a system. I have read about some Microsoft research into a computer monitoring the office environment to better facilitate the user s requests, an obvious extension to such research would be to have greater unlock requirements if there are more unknown people in the area or if the device is in a known unsafe location. But apart from that sort of future development it seems that having the user request a greater or lesser authentication check either at the time they lock their session or by policy would make sense. Generally users have a reasonable idea about the risk of another user trying to login with their terminal so user should be able to decide that a toilet break when at home only requires a fingerprint (enough to keep out other family members) while a toilet break at the office requires greater authentication. Mobile devices could use GPS location to determine unlock requirements, GPS can be forged, but if your attacker is willing and able to do that then you have a greater risk than most users. Some users turn off authentication on their phone because it s too inconvenient. If they had the option of using a fingerprint most of the time and a password for the times when a fingerprint can t be read then it would give an overall increase in security. Finally it should be possible to unlock only certain applications. Recent versions of Android support widgets on the lock screen so you can perform basic tasks such as checking the weather forecast without unlocking your phone. But it should be possible to have different authentication requirements for various applications. Using a fingerprint scan to allow playing games or reading email in the mailing list folder would be more than adequate security. But reading the important email and using SMS probably needs greater authentication. This takes us back to the XKCD cartoon.
Related posts:
  1. SASL Authentication and Debian/Wheezy After upgrading a mail server to Debian/Unstable (which will soon...
  2. biometrics and passwords In a comment on my post more about securing an...

9 February 2014

Jan Dittberner: Going to Chemnitzer Linux-Tage 2014

This year I take care of organizing of the Debian booth at Chemnitzer Linux-Tage 2014 which has been approved a few days ago. The CLT is a yearly (mostly) german speaking Free Software community event which takes place on the weekend of 15th/16th march in Chemnitz (Germany). On the Linux-Live pages you find a lot of projects that will have a booth there and the talk schedule contains many interesting topics. There will also be a key signing event for which you can register until 11th of march. The Wiki page for the Event is already in a good shape. Many things are already organized, but we still have some items left. A lot of people from the Debian community have already told me that they will be there. We will have a Debian Wheezy BabelBox demonstration running on a VirtualBox host that Jan Wagner will provide as well as merchandising (Thanks to Alexander Wirt). Two talks from people on our Wiki page have been accepted by the CLT organizers too: I am happy to meet many nice people from the Free Software community in Chemnitz soon.

Erich Schubert: Minimizing usage of debian-multimedia packages

How to reduce your usage of debian-multimedia packages: As you might be aware, the "deb-multimedia" packages have seen their share of chaos. Originally, they were named "debian-multimedia", but it was then decided that they should better be called "deb-multimedia" as they are not an official part of Debian. The old domain was then grabbed by cybersquatters when it expired. While a number of packages remain indistributable for Debian due to legal issues - such as decrypting DVDs - and thus "DMO" remains useful to many desktop users, please note that for many of the packages, a reasonable version exists within Debian "main". So here is a way to prevent automatically upgrading to DMO versions of packages that also exist in Debian main: We will use a configuration option known as apt pinning. We will modify the priority of DMO package to below 100, which means they will not be automatically upgraded to; but they can be installed when e.g. no version of this package exists within Debian main. I.e. the packages can be easily installed, but it will prefer using the official versions. For this we need to create a file I named /etc/apt/preferences.d/unprefer-dmo with the contents: 
Package: *
Pin: release o=Unofficial Multimedia Packages
Pin-Priority: 123
As long as the DMO archive doesn't rename itself, this pin will work; it will continue to work if you use a mirror of the archive, as it is not tied to the URL. It will not downgrade packages for you. If you want to downgrade, this will be a trickier process. You can use aptitude for this, and start with a filter (l key, as in "limit") of ?narrow(~i,~Vdmo). This will only show packages installed where the version number contains dmo. You can now patrol this list, enter the detail view of each package, and check the version list at the end for a non-dmo version. I cannot claim this will be an easy process. You'll probably have a lot of conflicts to clear up, due to different libavcodec* API versions. If I recall correctly, I opted to uninstall some dmo packages such as ffmpeg that I do not really need. In other cases, the naming is slightly different: Debian main has handbrake, while dmo has handbrake-gtk. A simple approach is to consider uninstalling all of these packages. Then reinstall as needed; since installing Debian packages is trivial, it does not hurt to deinstall something, does it? When reinstalling, Debian packages will be preferred over DMO packages. I prefer to use DFSG-free software wherever possible. And as long as I can watch my favorite series (Tatort, started in 1970 and airing episode 900 this month) and the occasional DVD movie, I have all I need. An even more powerful aptitude filter to review installed non-Debian software is ?narrow(~i,!~ODebian), or equivalently ?narrow(?installed,?not(?origin(Debian))). This will list all package versions, which cannot be installed from Debian sources. In particular, this includes versions that are no longer available (they may have been removed because of security issues!), software that has been installed manually from .deb files, or any other non-Debian source such as Google or DMO. (You should check the output of aptitude policy that no source claims to be o=Debian that isn't Debian though). This filter is a good health check for your system. Debian packages receive a good amount of attention with respect to packaging quality, maintainability, security and all of these aspects. Third party packages usually failed the Debian quality check in at least one respect. For example, Sage isn't in Debian yet; mostly because it has so many dependencies, that making a reliable and upgradeable installation is a pain. Similarly, there is no Hadoop in Debian yet. If you look at Hadoop packaging efforts such as Apache Bigtop, they do not live up to Debian quality yet. In particular, the packages have high redundancy, and re-include various .jar copies instead of sharing access to an independently packaged dependency. If a security issue arises with any of these jars, all the Hadoop packages will need to be rebuilt. As you can see, it is usually not because of Debian being too strict or too conservative about free software when software is not packaged. More often than not, it's because the software in question itself currently does not live up to Debian packaging quality requirements. And of course sometimes because there is too little community backing the software in question, that would improve the packaging. If you want your software to be included in Debian, try to make it packaging friendly. Don't force-include copies of other software, for example. Allow the use of system-installed libraries where possible, and provide upgrade paths and backwards compatibility. Make dependencies optional, so that your software can be pacakged even if a dependency is not yet available - and does not have to be removed if the dependency becomes a security risk. Maybe learn how to package yourself, too. This will make it easier for you to push new features and bug fixes to your userbase: instead of requiring your users to manually upgrade, make your software packaging friendly. Then your users will be auto-upgraded by their operating system to the latest version. Update: If you are using APT::Default-Release or other pins, these may override above pin. You may need to use apt-cache policy to find out if the pins are in effect, and rewrite your default-release pin into e.g. 
Package: *
Pin: release a=testing,o=Debian
Pin-Priority: 912
for debugging, use a different priority for each pin, so you can see which one is in effect. Notice the o=Debian in this pin, which makes it apply only to Debian repositories.

8 February 2014

Enrico Zini: original-sin

Original sin I feel that I was somewhat born innocent, an animal with sound, primal instincts. But if I remained that way, I wouldn't be able to function in a complex society, so I got education. Education taught me what is expected from me in order to be accepted by my peers[1]. Education taught me more than that: it also taught me to enjoy what a complex society can give me: art, science, history, philosophy, adding depth of meaning and correlations to my perceptions and memories. Education wasn't perfect, though. Some of my educators obsessed about some of the expectations, and gave me rules to follow that aren't really needed to interact with a society. Sit down for hours in silence without complaining. Don't talk back to figures of authority, even when they are abusing me. Don't ever feel that my efforts in a task have been enough, because there is always something more that I could have done. Do what people expect me to do, regardless of what I wish to do. Do what I need to do, not what I want to do. So I grew up with a set of arbitrary expectations that weren't needed to function in a society, and weren't in any way meeting any of my needs, yet I still felt them as a part of me, putting every effort into meeting them that I would put in making myself happy. I like to call this set of learned arbitrary, unneeded expectations "neurosis". I like this as an interpretation of the "original sin" myth: in order to go from an innocent animal to a member of a complex society, I acquired a set of neuroses that make me behave in a meaningless way. Or, rephrased going along with the myth, that rob me of my innocence. In some environments like BDSM, many have a name for an unnegotiated practice forced on a person: they call it "abuse". It made sense to accept that I have been abused many times while being educated, because my educators also had their own baggage of parasite expectations, of neuroses. And since I don't currently know how I can ever be sure that I freed myself from all my neuroses, I feel like I should accept that it is possible that I do and will abuse others; by accepting it as a possibility, I hope at least to be able to realise as soon as possible that I am doing it, and try to stop. I have now realised that my life is far simpler and more rewarding when expectations are negotiated between all the parties involved. I have recently spent a substantial amount of my energy in recognising and renegotiating many of the expectations and the neuroses that I had learnt while growing up, and, who knows, perhaps this effort will continue throughout my life. So there you go, an agnostic atheist freeing himself of his original sin, and regaining his paradise lost one step at a time, alive, on earth. And enjoying every moment of it. [1] I suspect that is why people educated in a high class society tend to be accepted more easily by a high class society: they conform to the right set of expectations. But I digress.

Mario Lang: boost::python and boost::variant

I have unsuccessfully tried to find a solution for the following problem on the internet several times. Now that I have come at least closer to a usable approach, I thought I'd document what I have found so that others trying to achieve a similar thing can use this as a starting point. Boost.Python offers a very nice and flexible way to interface C++ data types with Python. With just a few lines of code, and the proper linker flags, you get a Python importable shared object from your C++ compiler. This can be very productive. However, there is one aspect of C++ data types that I couldn't figure out how to interface with Python, which are C++ discriminated unions, or more specifically, heterogeneous containers. While Python has no problems with containers containing objects of different types, C++ does not make this very easy by default. Usually the problem is solved with a container of pointers to a base class, and various subclasses with virtual functions. However, this approach is not always practical, especially if the different types of objects in an heterogeneous container dont have many things in common. This is where discriminated unions come to the rescue. They basically behave like a normal union in C, but have an additional field which indicates the type of object currently stored in the union. Boost.Variant does exactly that, with a nice visitor interface added on top of it.
Heterogeneous containers in C++ If we put the boost::variant<> template inside a STL container like std::vector<>, the result is a heterogeneous container. For the purpose of illustration, lets implement such a container. The example below is deliberately simple. In reality, the various types allowed in your variant will probably have more fields then just one. 
#include <boost/variant.hpp>
#include <vector>
struct a   int x;  ;
struct b   std::string y;  ;
typedef boost::variant<a, b> variant;
typedef std::vector<variant> vector;
To ease creation of these two types of objects, we are going to write a few factory functions. We are going to wrap them in Python later on. 
variant make_variant()   return variant();  
vector make_vector()   return vector a(), b(), a() ;
Boost.Python Now lets create a Python module which exports the above functionality to Python. 
#include <boost/python/class.hpp>
#include <boost/python/def.hpp>
#include <boost/python/implicit.hpp>
#include <boost/python/init.hpp>
#include <boost/python/module.hpp>
#include <boost/python/object.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
vector_indexing_suite apparently needs operator== defined on the value_type of the container. In our case, this is our boost::variant<a, b> type. Luckily, boost::variant<> already provides operator==. However, that operator== relies on operator== being defined for the underlying types. Since equality comparison is probably useful for other things as well, lets just create operator== for our two classes a and b. 
bool operator==(a const &lhs, a const &rhs)   return lhs.x == rhs.x;  
bool operator==(b const &lhs, b const &rhs)   return lhs.y == rhs.y;
Convert a boost::variant<> to PyObject * Boost.Python needs a way to convert our discriminated union to a Python object. This code relies on Python class definitions being present for all underlying variant types. We will define them later. 
struct variant_to_object : boost::static_visitor<PyObject *>  
  static result_type convert(variant const &v)  
    return apply_visitor(variant_to_object(), v);
   
  template<typename T>
  result_type operator()(T const &t) const  
    return boost::python::incref(boost::python::object(t).ptr());
   
 ;
And finally, lets create our Python module. 
BOOST_PYTHON_MODULE(bpv)  
  using namespace boost::python;
  class_<a>("a", init<a>()).def(init<>()).def_readwrite("x", &a::x);
  class_<b>("b", init<b>()).def(init<>()).def_readwrite("y", &b::y);
  to_python_converter<variant, variant_to_object>();
  implicitly_convertible<a, variant>();
  implicitly_convertible<b, variant>();
  def("make_variant", make_variant);
  class_<vector>("vector").def(vector_indexing_suite<vector, true>());
  def("make_vector", make_vector);
Compiling Lets create a shared object for Python. 
$ g++ -std=c++11 -fPIC -shared $(python-config --includes) -o bpv.so file.cpp -lboost_python
Running We can load the module into Python and see what it does. 
>>> import bpv
>>> variant=bpv.make_variant()
>>> variant
<bpv.a object at 0x7f06bb2130c0>
>>> variant.x
0
>>> variant.x=2
>>> variant.x
2
Nice. We can access the underlying type, and even modify it. Lets see how our heterogeneous container wrapping code behaves. 
>>> vector=bpv.make_vector()
>>> vector
<bpv.vector object at 0x7f20693289d0>
>>> len(vector)
3
>>> list(vector)
[<bpv.a object at 0x7f20693190c0>, <bpv.b object at 0x7f20693193d0>, <bpv.a object at 0x7f2069319440>]
So far, so good. This will at least make it possible to convert heterogeneous containers from C++ to Python, which was my initial goal. Unfortunately, contained objects are not treated as references. Whenever retrieved, we get a copy. So in-place modification does not work. 
>>> vector[0].x
0
>>> vector[0].x=2
>>> vector[0].x
0
However, we can override an existing element with a modified copy. 
>>> e0=vector[0]
>>> type(e0)
<class 'bpv.a'>
>>> e0.x = 2
>>> vector[0] = e0
>>> vector[0].x
2
And we can also use the append and extend methods of Python containers. 
>>> len(vector)
3
>>> vector.extend(vector)
>>> vector.append(bpv.a())
>>> len(vector)
7
>>> len(filter(lambda x: type(x)==bpv.b, vector))
2
>>> len(filter(lambda x: type(x)==bpv.a, vector))
5
>>> map(lambda x: x.x, filter(lambda x: type(x)==bpv.a, vector))
[2, 0, 2, 0, 0]
All that is missing for a perfect world is reference semantics for container elements. If anyone has a hint on how to achieve this, please let me know.

John Goerzen: Migrated from Hetzner to OVH hosting

Since August 2011, my sites such as complete.org have been running on a Xen-backed virtual private server (VPS) at Hetzner Online, based in Germany. I had what they called their VQ19 package, which included 2GB RAM, 80GB HDD, 100Mb NIC and 4TB transfer. Unlike many other VPS hosts, I never had performance problems. However, I did sometimes have hardware problems with the host, and it could take hours to resolve. Their tech support only works business hours German time, which was also a problem. Meanwhile, OVH, a large European hosting company, recently opened a datacenter in Canada. Although they no longer offer their value-line Kimsufi dedicated servers there starting at $11.50/mo they do offer their midrange SoYouStart servers there. $50/mo gets a person a 4-core 3.2GHz Xeon server with 32GB RAM, 2x2TB SATA HDD, 200Mbps bandwidth. Not bad at all! The Kimsufi options are still good for lower-end needs as well. I signed up for one of the SoYouStart servers. I ve been pleased with my choice to migrate, and at the possibilities that having hardware like that at my disposal open up, but it is not without its downside. The primary downside is lack of any kind of KVM console. If the server doesn t boot, I can t see the Grub error message (or whatever) behind it. They do provide hardware support and automatic technician dispatching when the server isn t pingable, but they state they have no KVM access at all. They support many OS flavors, and have a premade image for them, but there is no using a custom ISO to install; if you want ZFS on Linux, for instance, you can t very easily build it into root. My server was promised within 72 hours, but delivered much quicker: within about 1. I had two times they said they had to replace a motherboard within the first day; once they did it in 30 minutes, and the other took them 2.5 hours for some reason. They do have phone support, which answers almost immediately, but the people there are not the people actually in the datacenter. It was frustrating with a server down for hours and nobody really commenting on what was going on. The server performs quite well, and after the initial issues, I ve been happy. I was initially planning an all-ZFS installation. SoYouStart does offer a rescue environment, but it doesn t support ZFS, so I figured I better stick with an ext4 root at least. The default Debian install uses RAID1 on md-raid, with a 20GB root partition and the rest of the 2TB drive in /home, and then a swap partition on each drive (mysteriously NOT in the RAID!) So I broke the mirror on /home and converted those into the two legs of a mirrored vdev for a zpool. I run all of the real work inside KVM VMs, so that should minimize the number of times I have to do anything to the root filesystem that could cause trouble. SoYouStart includes 100GB of space on a separate FTP server for backup purposes. I have scripts that upload nightly tarballs of the root filesystem, plus full zfs send streams of everything else. Every hour, it uploads an incremental zfs send stream as well. This all works quite nicely; even if the machine is a complete loss, I d never lose more than an hour s work, and could restore it completely from a rescue environment. Very nice! I ll write more in a few days about the ZFS setup I m using, and some KVM discoveries as well.

7 February 2014

Jo Shields: Dear Fake Debian Developers, shoo.

Another post about the Valve/Collabora free games thing. This time, the bad bit people trying to scam free games from us. Before I start, I want to make one thing clear there are people who have requested keys who don t meet the criteria, but are honest and legitimate in their requests. This blogspam is not about them at all. If you re in that category, you re not being complained about. So. Some numbers. At time of writing, I ve assigned keys to 279 Debian Developers or Debian Maintainers almost 25% of the total eligible pool of about 1200. I ve denied 22 requests. Of these 10 were polite requests from people who didn t meet the conditions stated (e.g. Ubuntu developers rather than Debian). These folks weren t at all a problem for us, and I explained politely that they didn t meet the terms we had agreed at the time with Valve. No problem at all with those folks. Next, we have the chancers, 7 of them, who pretended to be eligible when they clearly weren t. For example, two people sent me signed requests pointing to their entry on the Debian New Maintainers page when challenged over the key not being in the keyring. The NM page showed that they had registered as non-uploading Debian Contributors a couple of hours previously. A few just claimed I am a DD, here is my signature when they weren t DDs at all. Those requests were also binned.
Papers, Please screenshot - denied entry application

DENIED

And then we move onto the final category. These people tried identity theft, but did a terrible job of it. There were 5 people in this category: 
From: Xxxxxxxx Xxxxxx <xxxxxxxx.xxxxxx@ieee.org>
Subject: free subscription to Debian Developer
8217 A205 5E57 043B 2883 054E 7F55 BB12 A40F 862E
This is not a signature, it s a fingerprint. Amusingly, it s not the fingerprint for the person who sent my mail, but that of Neil McGovern a co-worker at Collabora. Neil assured me he knew how to sign a mail properly, so I shitcanned that entry. 
From: "Xxxxx, Xxxxxxxxx" <x.xxxxx@bbw-bremen.de>
Subject: Incoming!
Hey dude,
I want to have the redemption code you are offering for the Valve Games
mQGiBEVhrscRBAD4M5+qxhZUD67PIz0JeoJ0vB0hsLE6QPV144PLjLZOzHbl4H3N
...snip...
Lz8An1TEmmq7fltTpQ+Y1oWhnE8WhVeQAKCzh3MBoNd4AIGHcVDzv0N0k+bKZQ=3D=3D
=3Du/4R
Wat? Learn to GPG! 
From: Xxxxxx-Xxxx Le Xxxxxxx Xxxx <xx.xxxxxxxxx@gmail.com>
Subject: pass steam
Hey me voila
Merci beaucoup
valve
2069 1DFC C2C9 8C47 9529 84EE 0001 8C22 381A 7594
Like the first one, a fingerprint. This one is for S bastien Villemot. Don t scammers know how to GPG sign? 
From: "Xxxxxxxxx Xxxxxxx" <xxxxxxxx@web.de>
Subject: thanks /DD/Steam gifts us finally something back
0x6864730DF095E5E4
Yet again, a fingerprint. This one is for Marco Nenciarini. I found this request particularly offensive due to the subject line the haughty tone from an identity thief struck me as astonishingly impertinent. Still, when will scammers learn to GPG? 
From: Sven Hoexter <svenhoexter@gmail.com>
Subject: Valve produced games
I'm would like to get the valve produced games
My keyring: 0xA6DC24D9DA2493D1 Sven Hoexter <hoexter> sig:6
Easily the best scam effort, since this is the only one which both a) registered an email address under the name of a DD, and b) used a fingerprint which actually corresponds to that human. Sadly for the scammer, I m a suspicious kind of person, so my instinct was to verify the claim via IRC. 
31-01-2014 16:52:48 > directhex: Hoaxter, have you started using gmail without updating your GPG key? (note: significantly more likely is someone trying to steal your identity a little to steal valve keys from collabora)
31-01-2014 16:54:51 < Hoaxter!~sh@duckpond6.stormbind.net: directhex: I do not use any Google services and did not change my key
So yeah. Nice try, scammer. I m not listing, in all of this, the mails which Neil received from people who didn t actually read his mail to d-d-a. I m also not listing a story which I ve only heard second ha actually no, this one is too good not to share. Someone went onto db.debian.org, did a search for every DD in France, and emailed every Jabber JID (since they look like email addresses) asking them to forward unwanted keys. All in all, the number of evildoers is quite low, relative to the number of legitimate claims 12 baddies to 279 legitimate keys issued. But still, this is why the whole key issuing thing has been taking me so long and why I have the convoluted signature-based validation system in place. Enjoy your keys, all 279 of you (or more by the time some of you read this). The offer has no explicit expiry on it Valve will keep issuing keys as long as there is reason to, and Collabora will continue to administer their allocation as long as they remain our clients. It s a joint gift to the community thousands of dollars worth of games from Valve, and a significant amount of my time to administer them from Collabora.

6 February 2014

Johannes Schauer: botch updates

My last update about ongoing development of botch, the bootstrap/build ordering tool chain, was three months ago with the announcement of bootstrap.debian.net. Since then a number of things happened, so I thought an update was due.

New graphs for port metrics By default, a dependency graph is created by arbitrarily choosing an installation set for binary package installation or source package compilation. Installation set vertices and source vertices are connected according to this arbitrary selection. Niels Thykier approached me at Debconf13 about the possibility of using this graph to create a metric which would be able to tell for each source package, how many other source packages would become uncompilable or how many binary packages would become uninstallable, if that source package was removed from the archive. This could help deciding about the importance of packages. More about this can be found at the thread on debian-devel. For botch, this meant that two new graph graphs can now be generated. Instead of picking an arbitrary installation set for compiling a source package or installing a binary package, botch can now create a minimum graph which is created by letting dose3 calculate strong dependencies and a maximum graph by using the dependency closure.

Build profile syntax in dpkg With dpkg 1.17.2 we now have experimental build profile support in unstable. The syntax which ended up being added was: 
Build-Depends: large (>= 1.0), small <!profile.stage1>
But until packages with that syntax can hit the archive, a few more tools need to understand the syntax. The patch we have for sbuild is very simple because sbuild relies on libdpkg for dependency parsing. We have a patch for apt too, but we have to rebase it for the current apt version and have to adapt it so that it works exactly like the functionality dpkg implements. But before we can do that we have to decide how to handle mixed positive and negative qualifiers or whether to remove this feature altogether because it causes too much confusion. The relevant thread on debian-dpkg starts here.

Update to latest dose3 git Botch heavily depends on libdose3 and unfortunately requires features which are only available in the current git HEAD. The latest version packaged in Debian is 3.1.3 from October 2012. Unfortunately the current dose3 git HEAD also relies on unreleased features from libcudf. On top of that, the GraphML output of the latest ocamlgraph version (1.8.3) is also broken and only fixed in the git. For now everything is set up as git submodules but this is the major blocker preventing any packaging of botch as a Debian package. Hopefully new releases will be done soon for all involved components.

Writing and reading GraphML Botch is a collection of several utilities which are connected together in a shell script. The advantage of this is, that one does not need to understand or hack the OCaml code to use botch for different purposes. In theory it also allows to insert 3rd party tools into a pipe to further modify the data. Until recently this ability was seriously hampered by the fact that many botch tools communicated with each other through marshaled OCaml binary files which prevent everything which is not written in OCaml from modifying them. The data that was passed around like this were the dependency graphs and I initially implemented it like that because I didnt want to write a GraphML parser. I now ended up writing an xmlm based GraphML parser so as of now, botch only reads and writes ASCII text files in XML (for the graphs) and in rfc822 packages format (for Packages and Sources files) which can both easily be modified by 3rd party tools. The ./tools directory contains many Python scripts using the networkx module to modify GraphML and the apt_pkg module to modify rfc822 files.

Splitting of tools To further increase the ability to modify program execution without having to know OCaml, I split up some big tools into multiple smaller ones. Some of the smaller tools are now even written in Python which is probably much more hackable for the general crowd. I converted those tools to Python which did not need any dose3 functionality and which were simple enough so that writing them didnt take much time. I could convert more tools but that might introduce bugs and takes time which I currently dont have much of (who does?).

Gzip instead of bz2 Since around January 14, snapshot.debian.org doesnt offer bzip2 compressed Packages and Sources files anymore but uses xz instead. This is awesome for must purposes but unfortunately I had to discover that there exist no OCaml bindings for libxz. Thus, botch is now using gzip instead of bz2 until either myself or anybody else finds some time to write a libxz OCaml binding.

Self hosting Fedora Paul Wise made me aware of Harald Hoyer's attempts to bootstrap Fedora. I reproduced his steps for the Debian dependency graph and it turns out that they are a little bit bigger. I'm exchanging emails with Harald Hoyer because it might not be too hard to use botch for rpm based distributions as well because dose3 supports rpm. The article also made me aware of the tred tool which is part of graphviz and allows to calculate the transitive reduction of a graph. This can help making horrible situations much better.

Dose3 bugs I planned to generate such simplified graphs for the neighborhood of each source package on bootstrap.debian.net but then binutils stopped building binutils-gold and instead provided binutils-gold while libc6-dev breaks binutils-gold (<< 2.20.1-11). This unfortunately triggered a dose3 bug and thus bootstrap.debian.net will not generate any new results until this is fixed in dose3. Another dose3 bug affects packages which Conflicts/Replaces/Provides:bar while bar is fully virtual and are Multi-Arch:same. Binaries of different architecture with this property can currently not be co-installed with dose3. Unfortunately linux-libc-dev has this property and thus botch cannot be used to analyze cross builds until that bug is fixed in dose3. I hope I get some free time soon to be able to look at these dose3 issues myself.

More documentation Since I started to like the current set of tools and how they work together I ended up writing over 2600 words of documentation in the past few days. You can start setting up and running botch by reading the first steps and get more detailed information by reading about the 28 tools that botch makes use of as of now. All existing articles, thesis and talks are linked from the wiki home.

Steve Kemp: External Comments, updated

The simple external-comments code is now complete enough for me to stop poking it on a daily basis: All in all I'm pretty happy with the way it works, and the server-code. The client-side Javascript is less good, but I'm probably done poking that too. In an ideal world the client-side code should be a jQuery plugin, but I've not worked out how to make a static method (the JSONP callback) be a member of a jQuery plugin-object. So without that I have to re-pass the options around too many places, rather than making them a member of "this". Meh, pull requests welcome for adding new storage back-ends (redis and sqlite are supported by default), and similarly for cleanups. Links:

Next.

Previous.