Search Results: "zed"

5 February 2014

Wouter Verhelst: "Nuclear precision weapons"

Now that the madness is mostly over, I have some time to catch up on reading my newspaper. Last week, one article in De Standaard talks about the nuclear weapons on Belgian soil which officially didn't exist (until wikileaks proved otherwise) and which are now apparently going to be modernized. The newer model would be "nuclear precision weapons". Ignoring the question of whether today's world still requires nuclear bombs (this may or may not be true, I don't care), I question the logic which leads to that phrase. A nuclear weapon is a weapon of mass destruction. By definition, a weapon of mass destruction causes collateral damage. By definition, a precision weapon is a weapon that does not cause collateral damage or, at the very lease, where every effort is made to limit the amount of collateral damage. Even the very first nuclear bombs were capable of destroying entire cities. Today's nuclear weapons, even the smaller ones, are far more powerful than those. Don't get me wrong: I'm not a peace activist. In fact, I have been contracted by companies who produce military equipment, and don't feel bad about that. But to claim that it is possible to create "nuclear precision weapons" is to deceive oneself. A nuclear weapon is not very precise.

Ricardo Mones: Fixing partridge eggs with industrial duct tape

Human nature is hard to change. Very hard. We can talk about it for ages, but mistakes repeat again and again. In the end it's mostly by mistakes how we learn, so I doubt this could ever be changed without losing our own nature. One of these is trying to fix some social issue with a technical measure. Unfortunately, given the technical orientation of most of the developers, this appears from time to time in our Debian private mailing list, and yesterday I realized it's our own version of Godwin's law:

As a social problem discussion grows longer in debian-private the probability of some developer proposing a technical solution aproaches one.

Not discussing about this problems in debian-private would be a good start, but of course that would only change the name of the list in the above sentence ;-).

Andrew Pollock: [life] Day 9, Kindergarten and cleaning

Another busy day today. I saw my girlfriend off in the morning, and then got stuck into cleaning the bathrooms. I ducked out to Jaycar to have another attempt at getting components. Got home, did some more cleaning, and had my chiropractic adjustment. Then I had my exit interview for work via video conference, and did some administrivia while I still had my work computer. Then the guy from Pack & Send came to take all the work equipment away. Now I feel very disconnected. Had some lunch, did more cleaning. Mum and Dad came around to go with me for pick up today, so they could see the new Kindergarten. Zoe was again fast asleep at pick up, and the kids who were awake were all in the other room eating birthday cake, so we let Zoe wake up as gently as possible and then collected Megan and squeezed everyone into the car and headed home. Zoe and Megan had a great time playing with each other again, and Megan's Dad picked her up at about 4:30pm and I started on dinner. We all had dinner and then Mum and Dad headed off, and I gave Zoe a bath and put her to bed. She was a bit emotional at bedtime, so it took a while to get her settled. Here's hoping she sleeps okay tonight.

4 February 2014

Enrico Zini: debops

Debops What I like the most about being a Developer is building tools to (hopefully) make someone's life better. I like it when my software gets used, and people thank me for it, because there was a need they had that wasn't met before, and thanks to my software now it is being met. I am maintaining software for meteorological research that is soon going to be 10 years old, and is still evolving and getting Real Work done. I like to develop software as if it is going to become a part of human cultural heritage, developing beyond my capacity, eventually surviving me, allowing society to declare that the need, small as it was, is now met, and move on to worry about some other problem. I feel that if I'm not thinking of my software in that way, then I am not being serious. Then I am not developing something fit for other people to use and rely on. This involves Development as much as it involves Operations: tracking security updates for all the components that make up a system. Testing. Quality assurance. Scalability. Stability. Hardening. Monitoring. Maintenance requirements. Deployment and upgrade workflows. Security. I came to learn that the requirements put forward by sysadmins are to be taken seriously, because they are the ones whose phone will ring in the middle of the night when your software breaks. I am also involved in more than one software project. I am responsible for about a dozen web applications deployed out there in the wild, and possibly another dozen of non-web projects, from terabyte-sized specialised archival tools to little utilities that are essential links in someone's complex toolchain. I build my software targeting Debian Stable + Backports. At FOSDEM I noticed that some people consider it uncool. I was perplexed. It provides me with a vast and reasonably recent set of parts to use to build my systems. It provides me with a single bug tracking system for all of them, and tools to track known issues in the systems I deployed. It provides me with a stable platform, with a well documented upgrade path to the next version. It gives me a release rhythm that allows me to enjoy the sweet hum of spinning fans thinking about my next mischief, instead of spending my waking time chasing configuration file changes and API changes deep down in my dependency chain. It allows me to rely on Debian for security updates, so I don't have to track upstream activity for each one of the building blocks of the systems I deploy. It allows me not to worry about a lot of obscure domain specific integration issues. Coinstallability of libraries with different ABI versions. Flawless support for different versions of Python, or Lua, or for different versions of C++ compilers. It has often happened to me to hear someone rant about a frustrating situation, wonder how come it had never happened to me, and realise that someone in Debian, who happens to be more expert than I can possibly be, had thought hard about how to deal with that issue, years before. I know I cannot be an expert of the entire stack from bare iron all the way up, and I have learnt to stand on the shoulders of giants. 'Devops' makes sense for me in that it hints at this cooperation between developers and operators, having constructive communication, knowing that each side has their own needs, trying their best to meet them all. It hints at a perfect world where developers and operators finally come to understand and trust each other's judgement. I don't know that perfect world, but I, a developer, do like to try to understand and trust the judgement of sysadmins. I sympathise with my sysadmin friends who feel that devops is turning into a trend of developers thinking they can do without sysadmins. Reinventing package managers. Bundling dependencies. Building "apps" instead of components. I wish that people who deploy a system built on such premises, have it become so successful that they end up being paid to maintain them for their whole career. That is certainly what I wish and strive for, for me and my own projects. In my experience, a sustainable and maintainable system won't come out of the startup mindset of building something quick&dirty, then sell it and move on to something else. In my experience, the basis for having sustainable and maintainable systems have been well known and tested in Debian, and several other distributions, for over two decades. At FOSDEM, we thought that we need a name for such a mindset. Between beers, that name came to be "debops". (It's not just Debian, though: many other distributions get it right, too)

2 February 2014

Gunnar Wolf: CuBox-i4Pro

CuBox-i4Pro
Somewhere back in August or September, I pre-ordered a CuBox-i A nicely finished, completely hackable, and reasonably powerful ARM system, nicely packaged and meant to be used to hack on. A sweet deal! There are four models (you can see the different models' specs here) I went for the top one, and bought a CuBox-i4Pro. That means, I have a US$130 nice little box, with 4 ARM7 cores, 2GB RAM, WiFi, and... well, all of its basic goodies and features. For some more details, look at the CuBox-i block diagram. I got it delivered by early January, and (with no real ARM experience on my side) I finally got to a point where I can, I believe, contribute something to its adoption/usage: How to get a basic Debian system installed and running in it. The ARM world is quite different to the x86 one: Compatibility is much harder, the computing platform does not self-describe properly, and a kernel must first understand how a specific subarchitecture is before being able to boot on it. Somewhere in the CuBox forums (or was it the IRC channel?) I learnt that the upstream Linux kernel does not yet boot on the i.MX6 chip (although support is rumored to be merged for the 3.14 release), so I am using both a kernel and an uBoot bootloader not built for (or by) Debian people. Besides that, the result I will describe is a kosher Debian install. Yes, I know that my orthodox friends and family will say that 99% kosher is taref... But remember I'm never ever that dogmatic. (yeah, right!) Note that there is a prebuilt image you can run if you are so inclined: In the CuBox-i forums and wiki, you will find links to a pre-installed Debian image you can use... But I cannot advise to do so. First, it is IMO quite bloated (you need a 4GB card for a very basic Debian install? Seriously?) Second, it has a whole desktop environment (LXDE, if I recall correctly) and a whole set of packages I will probably not use in this little box. Third, there is a preinstalled user, and that's a no-no (user: debian, password: debian). But, most importantly, fourth: It is a nightly build of the Testing (Jessie) suite... Built back in December. So no, as a Debian Developer, it's not something we should recommend our users to run! So, in the end and after quite a bit of frustration due to my lack of knowledge, here goes the list of steps I followed:
Using the CuBox
On the i2 and i4 models, you can use it either with a USB keyboard and a HDMI monitor, or by a serial consoles (smaller models do not have a serial console). I don't have a HDMI monitor handy (only a projector), so I prefer to use the serial terminal. Important details to avoid frustration: The USB keyboard has to be connected to the lower USB port, or it will be ignored during the boot process. And make sure your serial terminal is configured not to use hardware flow control. Minicom is configured by default to use hardware flow control, so it was not sending any characters to the CuBox. ^A-O gets you to the Minicom configuration, select Serial port setup, and disable it.
Set up the SD card
I created a 2GB partition, but much less can suffice; I'd leave it at least to 1GB to do the base install, although it can be less once the system is set up (more on this later). Partition and format using your usual tools (fdisk+mke2fs, or gparted, or whatever suits your style).
Install the bootloader
I followed up the instructions on this CuBox-i forums thread to get the SPL and uBoot bootloader running. In short, from this Google Drive folder, download the SPL-U-Boot.img.xz file, uncompress it (xz --decompress SPL-U-Boot.img.xz), and write it to the SD card just after the partition map: As root,
# dd if=SPL-U-Boot.img of=/dev/mmcblk0 bs=1024 seek=1.
Actually, to be honest: As I wanted something basic to be able to debug from, I downloaded (from the same Google Drive) the busybox.img.gz file. That's a bit easier to install from: xz --decompress busybox.img.xz, and just dump it into the SD from the beginning (as it does already include a partition table):
# dd if=busybox.img of=/dev/mmcblk0
This card is already bootable and minimal, and allows to debug some bits from the CuBox-i itself (as we will see shortly).
After this step, I created a second partition, as I said earlier. So, my mmcblk0p1 partition holds Busybox, and the second will hold Debian. We are still working from the x86 system, so we mount the SD card in /media/mmcblk0p2
Installing the base system
Without debian-installer to do the heavy lifting, I went for debootstrap. As I ran it from my PC, debootstrap's role will be for this first stage only to download and do a very initial pre-unpacking of the files: Bootstrapping a foreign architecture implies, right, using the --foreign switch:
debootstrap --foreign --arch=armhf wheezy /media/mmcblk0p2 http://http.debian.net/debian
You can add some packages you often use by specifying --include=foo,bar,baz
So, take note notes: This board is capable of running the armhf architecture (HF for Hardware Float). It can also run armel, but I understand it is way slower.
First boot (with busybox)
So, once debootstrap finishes, you are good to go to the real hardware! Unmount the SD card, put it in the little guy, plug your favorite console in (I'm using the serial port), and plug the power in! You should immediately see something like:
  1. U-Boot SPL 2013.10-rc4-gd05c5c7-dirty (Jan 12 2014 - 02:18:28)
  2. Boot Device: SD1
  3. reading u-boot.img
  4. Load image from RAW...
  5. U-Boot 2013.10-rc4-gd05c5c7-dirty (Jan 12 2014 - 02:18:28)
  6. CPU: Freescale i.MX6Q rev1.2 at 792 MHz
  7. Reset cause: POR
  8. Board: MX6-CuBox-i
  9. DRAM: 2 GiB
  10. MMC: FSL_SDHC: 0
  11. In: serial
  12. Out: vga
  13. Err: vga
  14. Net: phydev = 0x0
  15. Phy not found
  16. PHY reset timed out
  17. FEC
  18. (Re)start USB...
  19. USB0: USB EHCI 1.00
  20. scanning bus 0 for devices... 1 USB Device(s) found
  21. scanning usb for storage devices... 0 Storage Device(s) found
  22. scanning usb for ethernet devices... 0 Ethernet Device(s) found
  23. Hit any key to stop autoboot: 3

Let it boot (that means, don't stop autoboot), and you will soon see a familiar #, showing you are root in the busybox environment. Great! Now, mount the Debian partition:
# mount /dev/mmcblk0p2 /mnt
Finishing debootstrap's task
With everything in place, it's time for debootstrap to work. Chroot into the Debian partition:
# chroot /mnt
And ask Debootstrap to finish what it started:
# debootstrap --second-stage
Be patient, as this step takes quite a bit to be finished.
Some extra touches...
After this is done, your Debian system is almost ready to be booted into. Why almost? Because it still does not have any users, does not know its own name nor knows I want to use it via a serial terminal. Three very simple tasks to fix. First two:
  1. # passwd
  2. Enter new UNIX password:
  3. Retype new UNIX password:
  4. passwd: password updated successfully
  5. # echo cubox-i.gwolf.org > /etc/hostname

For the second one, add a line to /etc/inittab specifying the details of the serial console. You can just do this:
# echo 'T0:23:respawn:/sbin/getty -L ttymxc0 115200 vt100' >> /etc/inittab
Boot into Debian!
So, ready to boot Debian? Ok, first exit the chroot shell, to go back to the Busybox shell, unmount the Debian partition, and set the root partition read-only:
  1. # exit
  2. # umount /mnt
  3. # mount / -o remount,ro

Disconnect and connect power, and now, do interrupt the boot process when you see the Hit any key to stop automount prompt. To see the configuration of uboot, you can type printenv We will only modify the parameters given to the kernel:
  1. CuBox-i U-Boot > setenv root /dev/mmcblk0p2 rootfstype=ext3 ro rootwait
  2. CuBox-i U-Boot > boot

So, the kernel will load, and a minimal Debian system will be initialized. In my case, I get the following output:
  1. ** File not found /boot/busyEnv.txt **
  2. 4703740 bytes read in 390 ms (11.5 MiB/s)
  3. ## Booting kernel from Legacy Image at 10000000 ...
  4. Image Name: Linux-3.0.35-8
  5. Image Type: ARM Linux Kernel Image (uncompressed)
  6. Data Size: 4703676 Bytes = 4.5 MiB
  7. Load Address: 10008000
  8. Entry Point: 10008000
  9. Verifying Checksum ... OK
  10. Loading Kernel Image ... OK
  11. Starting kernel ...
  12. Unable to get enet.0 clock
  13. pwm-backlight pwm-backlight.0: unable to request PWM for backlight
  14. pwm-backlight pwm-backlight.1: unable to request PWM for backlight
  15. _regulator_get: get() with no identifier
  16. mxc_sdc_fb mxc_sdc_fb.2: NO mxc display driver found!
  17. INIT: version 2.88 booting
  18. [info] Using makefile-style concurrent boot in runlevel S.
  19. [....] Starting the hotplug events dispatcher: udevd. ok
  20. [....] Synthesizing the initial hotplug events...done.
  21. [....] Waiting for /dev to be fully populated...done.
  22. [....] Activating swap...done.
  23. [....] Cleaning up temporary files... /tmp. ok
  24. [....] Activating lvm and md swap...done.
  25. [....] Checking file systems...fsck from util-linux 2.20.1
  26. done.
  27. [....] Mounting local filesystems...done.
  28. [....] Activating swapfile swap...done.
  29. [....] Cleaning up temporary files.... ok
  30. [....] Setting kernel variables ...done.
  31. [....] Configuring network interfaces...done.
  32. [....] Cleaning up temporary files.... ok
  33. [....] Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix. ok
  34. INIT: Entering runlevel: 2
  35. [info] Using makefile-style concurrent boot in runlevel 2.
  36. [....] Starting enhanced syslogd: rsyslogd. ok
  37. [....] Starting periodic command scheduler: cron. ok
  38. Debian GNU/Linux 7 cubox-i.gwolf.org ttymxc0
  39. cubox-i login:

And that's it, the system is live and ready for my commands!
So, how big is this minimal Debian installed system? I cheated a bit on this, as I had already added emacs and screen to the system, so yours will be a small bit smaller. But anyway Lets clear our cache of downloaded packages, and see the disk usage information:
  1. root@cubox-i:~# apt-get clean
  2. root@cubox-i:~# df -h
  3. Filesystem Size Used Avail Use% Mounted on
  4. rootfs 1008M 347M 611M 37% /
  5. /dev/root 1008M 347M 611M 37% /
  6. devtmpfs 881M 0 881M 0% /dev
  7. tmpfs 177M 132K 177M 1% /run
  8. tmpfs 5.0M 0 5.0M 0% /run/lock
  9. tmpfs 353M 0 353M 0% /run/shm

So, instead of a 4GB install, we have a 350MB one. Great improvement! Now, lets get it to do something useful, in a most Debianic way!

31 January 2014

Andrew Pollock: [life] Day 4, Brazilian jiu jitsu, Science Friday and the Lunar New Year

I want Zoe to do one "extra curricular" activity per term this year. Something dance-related, something gymnastics-related and maybe some other form of sport (I'm thinking soccer). My girlfriend and I were wandering through Westfield Carindale on Saturday, and we happened upon a guy from Infinity Martial Arts touting his wares. I thought I'd suss it out, and they had an introductory offer where the sign-up fee and uniform fee were significantly reduced, and they had a pretty flexible timetable. They were just starting up their East Brisbane location, and the close proximity to home along with the reduced price sealed the deal. Long-term, I'd like for Zoe to learn Tae Kwon Do, for self-defense, but BJJ seemed as good as anything to get her introduced to the idea of martial arts. The class for 2-4 year olds was billed as "Fun and Fitness 4 Kids" so it's really a combination of listening to the instructor, some basic gymnastics-style stuff and a little bit of martial arts. We biked over this morning (going up Hawthorne Road is a slog) and got there in about 15 minutes via the direct route. It's in the upstairs of a gym in the middle of an industrial area, but it was pretty easily accessible by bike. They're still waiting on some of the equipment, so the space was a little spartan. It was just Zoe and I and a mother of two not quite 2 year old twin girls taking a trial class. First up, we got her uniform, and she looked so cute. There were pants and a jacket and a belt. I'm going to have to video the instructor tying up the belt next week so I can learn how to do it the right way. The class started with the kids standing on these flat coloured circles ("mushrooms") and effectively playing "Simon Says" ("instructor says") without being caught out. It was a pretty sneaky way of doing a bunch of warm up exercises like rotating the knees and ankles. Zoe did very well, but there were a few that she just point blank refused to do. Next the instructor set up a bunch of "stations" around the room. The first station involved me crouching in a fetal position with a football and Zoe had to try and tip me over to get the football. That was a load of fun. The next station was a few steps and foam-filled vinyl ramp, and Zoe just had to do a somersault down that. The next station was pretty much the same but taller, and Zoe had to do a "sausage roll" on her side down that. The next station was just a small exercise ball and Zoe had to do some "donkey kicks" on it. The final station involved me waving a couple of cut-off pool noodles at arm's length, and Zoe had to run in covering her head and give me a bear hug. We did a few rotations of these stations. It was heaps of fun. Next, the instructor got a whole bunch of ball pit balls of different colours, and scattered them over the floor, and put a basket at each end of the room. The kids were then instructed to retrieve specific colours as fast as possible. Zoe started out trying to get as many as possible in her arms before returning to the basket, but the idea was to do it one ball at a time. A lot of running back and forth. Finally, we did some actual BJJ (I think). It was called the "sleeping crocodile hold" or something like that. I had to lie on my back, and Zoe had to sneak up to me from my side and grapple me with one arm behind my head and the other around my waist and a knee in my side. I have no interest in Zoe learning mixed martial arts, but this class was so much fun. I was feeling a bit tired this morning before we headed out, but by the end of it I was so pumped. It was just the right combination of daddy/daughter rough and tumble, with a bit of gymnastics and following instruction. I'm pretty certain Zoe enjoyed it. I liked that the instructor stopped for a water break between each activity, so the kids were kept well hydrated throughout. We took the "scenic route" home, because we had no particular time constraints. It was more like 25 minutes and involved the Norman Park Greenway. I was so glad we went that way. It was a beautiful ride that I didn't know existed. Very indirect, it involved going through Woolloongabba, Coorparoo and Norman Park, around the back of Coorparoo State High School along the side of Norman Creek. It was semi-wetland conditions.The only part that was a bit annoying was where Norman Avenue met Wynnum Road. It was quite steep and the green light didn't last very long. I had wanted to do story time at Bulimba Library at 10:30am on Fridays, but I'd rather bike home via the Norman Park Greenway instead, because it's a nice ride. I've since decided that I'll just use the story time at the library during wet weather, when we'd be driving to BJJ anyway, and be able to make it to the library in time after class. I've also got to figure out where I'm going to fit doing some Science into the schedule. Fridays are going to be busy I think. I managed to get Zoe down for a nap by a bit after 12:30pm today. She was pretty knackered after the class (as was I from biking home) so I let her watch a bit of TV while I prepared lunch. She was funny, she saw how sweaty I was when we got home, and suggested I take a shower while she watch some TV. I was a bit unprepared for my first Science Friday, though. I'd been considering going to the Sir Thomas Brisbane Planetarium after BJJ, but I discovered it's closed until February 7 for maintenance and an upgrade, so that cunning plan was thwarted. I used her nap time to order 365 Science Experiments and 50 Dangerous Things (you Should Let Your Children Do). I should get plenty of inspiration out of those two. After she woke up, we did the old "vinegar and sodium bicarbonate" trick as our science experiment. I've finally got a use for my Google Labs Founders' Award lab coat. I need to try and find some child-sized safety glasses. The adult ones barely stay on her little nose. After that, we went for a walk to our local toy store to see if they sold child-sized safety glasses (they didn't) and then Zoe watched a little bit of TV and then we walked to the CityCat to go to Teneriffe to catch a bus to Chinatown. I couldn't have timed it better if I tried. Just as we got to Chinatown, they started doing a thing with the Chinese dragons, and I hoisted Zoe up on my shoulders so she could watch. I wasn't sure how she was going to take it, with all the noise from the drums and the dragons themselves, but she was enthralled. We watched other acts and then my girlfriend joined us, and we went to a Chinese restaurant for dinner. I bought Zoe these training chopsticks a while ago, and she's taken to them like a duck to water. I brought them with us so she could use them at the restaurant, and she ate the biggest dinner I've ever seen her eat. After dinner, we caught the tail end of a procession and had some ice cream. We were starting to leave, and there was another dragon, and Zoe was brave enough to go and touch it on her own, for good luck. We then made our way back to the bus. By the time the bus arrived and got us back to Teneriffe, and a CityCat finally arrived, it was very late, but Zoe was pretty good the whole time. It was probably the latest night she's ever had with me while we've been out on the go, and she only got a bit ratty once we were home. It didn't help that she'd forgotten that she'd put Cowie in a cupboard this morning and we burned some more time tracking her down. Today was a fantastic (and very full) day. I enjoyed it a lot, and I think Zoe did too. Fortunately, Fridays won't always be this full on.

29 January 2014

Petter Reinholdtsen: A fist full of non-anonymous Bitcoins

Bitcoin is a incredible use of peer to peer communication and encryption, allowing direct and immediate money transfer without any central control. It is sometimes claimed to be ideal for illegal activity, which I believe is quite a long way from the truth. At least I would not conduct illegal money transfers using a system where the details of every transaction are kept forever. This point is investigated in USENIX ;login: from December 2013, in the article "A Fistful of Bitcoins - Characterizing Payments Among Men with No Names" by Sarah Meiklejohn, Marjori Pomarole,Grant Jordan, Kirill Levchenko, Damon McCoy, Geoffrey M. Voelker, and Stefan Savage. They analyse the transaction log in the Bitcoin system, using it to find addresses belong to individuals and organisations and follow the flow of money from both Bitcoin theft and trades on Silk Road to where the money end up. This is how they wrap up their article:
"To demonstrate the usefulness of this type of analysis, we turned our attention to criminal activity. In the Bitcoin economy, criminal activity can appear in a number of forms, such as dealing drugs on Silk Road or simply stealing someone else s bitcoins. We followed the flow of bitcoins out of Silk Road (in particular, from one notorious address) and from a number of highly publicized thefts to see whether we could track the bitcoins to known services. Although some of the thieves attempted to use sophisticated mixing techniques (or possibly mix services) to obscure the flow of bitcoins, for the most part tracking the bitcoins was quite straightforward, and we ultimately saw large quantities of bitcoins flow to a variety of exchanges directly from the point of theft (or the withdrawal from Silk Road). As acknowledged above, following stolen bitcoins to the point at which they are deposited into an exchange does not in itself identify the thief; however, it does enable further de-anonymization in the case in which certain agencies can determine (through, for example, subpoena power) the real-world owner of the account into which the stolen bitcoins were deposited. Because such exchanges seem to serve as chokepoints into and out of the Bitcoin economy (i.e., there are few alternative ways to cash out), we conclude that using Bitcoin for money laundering or other illicit purposes does not (at least at present) seem to be particularly attractive."
These researches are not the first to analyse the Bitcoin transaction log. The 2011 paper "An Analysis of Anonymity in the Bitcoin System" by Fergal Reid and Martin Harrigan is summarized like this:
"Anonymity in Bitcoin, a peer-to-peer electronic currency system, is a complicated issue. Within the system, users are identified by public-keys only. An attacker wishing to de-anonymize its users will attempt to construct the one-to-many mapping between users and public-keys and associate information external to the system with the users. Bitcoin tries to prevent this attack by storing the mapping of a user to his or her public-keys on that user's node only and by allowing each user to generate as many public-keys as required. In this chapter we consider the topological structure of two networks derived from Bitcoin's public transaction history. We show that the two networks have a non-trivial topological structure, provide complementary views of the Bitcoin system and have implications for anonymity. We combine these structures with external information and techniques such as context discovery and flow analysis to investigate an alleged theft of Bitcoins, which, at the time of the theft, had a market value of approximately half a million U.S. dollars."
I hope these references can help kill the urban myth that Bitcoin is anonymous. It isn't really a good fit for illegal activites. Use cash if you need to stay anonymous, at least until regular DNA sampling of notes and coins become the norm. :) As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

27 January 2014

Kees Cook: -fstack-protector-strong

There will be a new option in gcc 4.9 named -fstack-protector-strong , which offers an improved version of -fstack-protector without going all the way to -fstack-protector-all . The stack protector feature itself adds a known canary to the stack during function preamble, and checks it when the function returns. If it changed, there was a stack overflow, and the program aborts. This is fine, but figuring out when to include it is the reason behind the various options. Since traditionally stack overflows happen with string-based manipulations, the default (-fstack-protector), only includes the canary code when a function defines an 8 (--param=ssp-buffer-size=N, N=8 by default) or more byte local character array. This means just a few functions get the checking, but they re probably the most likely to need it, so it s an okay balance. Various distributions ended up lowering their default --param=ssp-buffer-size option down to 4, since there were still cases of functions that should have been protected but the conservative gcc upstream default of 8 wasn t covering them. However, even with the increased function coverage, there are rare cases when a stack overflow happens on other kinds of stack variables. To handle this more paranoid concern, -fstack-protector-all was defined to add the canary to all functions. This results in substantial use of stack space for saving the canary on deep stack users, and measurable (though surprisingly still relatively low) performance hit due to all the saving/checking. For a long time, Chrome OS used this, since we re paranoid. :) In the interest of gaining back some of the lost performance and not hitting our Chrome OS build images with such a giant stack-protector hammer, Han Shen from the Chrome OS compiler team created the new option -fstack-protector-strong, which enables the canary in many more conditions: This meant we were covering all the more paranoid conditions that might lead to a stack overflow. Chrome OS has been using this option instead of -fstack-protector-all for about 10 months now. As a quick demonstration of the options, you can see this example program under various conditions. It tries to show off an example of shoving serialized data into a non-character variable, like might happen in some network address manipulations or streaming data parsing. Since I m using memcpy here for clarity, the builds will need to turn off FORTIFY_SOURCE, which would also notice the overflow.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
struct no_chars  
    unsigned int len;
    unsigned int data;
 ;
int main(int argc, char * argv[])
 
    struct no_chars info =    ;
    if (argc < 3)  
        fprintf(stderr, "Usage: %s LENGTH DATA...\n", argv[0]);
        return 1;
     
    info.len = atoi(argv[1]);
    memcpy(&info.data, argv[2], info.len);
    return 0;
 
Built with everything disabled, this faults trying to return to an invalid VMA: Built with FORTIFY_SOURCE enabled, we see the expected catch of the overflow in memcpy: So, we ll leave FORTIFY_SOURCE disabled for our comparisons. With pre-4.9 gcc, we can see that -fstack-protector does not get triggered to protect this function: However, using -fstack-protector-all does trigger the protection, as expected: And finally, using the gcc snapshot of 4.9, here is -fstack-protector-strong doing its job: For Linux 3.14, I ve added support for -fstack-protector-strong via the new CONFIG_CC_STACKPROTECTOR_STRONG option. The old CONFIG_CC_STACKPROTECTOR will be available as CONFIG_CC_STACKPROTECTOR_REGULAR. When comparing the results on builds via size and objdump -d analysis, here s what I found with gcc 4.9: A normal x86_64 defconfig build, without stack protector had a kernel text size of 11430641 bytes with 36110 function bodies. Adding CONFIG_CC_STACKPROTECTOR_REGULAR increased the kernel text size to 11468490 (a +0.33% change), with 1015 of 36110 functions stack-protected (2.81%). Using CONFIG_CC_STACKPROTECTOR_STRONG increased the kernel text size to 11692790 (+2.24%), with 7401 of 36110 functions stack-protected (20.5%). And 20% is a far-cry from 100% if support for -fstack-protector-all was added back to the kernel. The next bit of work will be figuring out the best way to detect the version of gcc in use when doing Debian package builds, and using -fstack-protector-strong instead of -fstack-protector. For Ubuntu, it s much simpler because it ll just be the compiler default.

2014, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

Daniel Pocock: A quick look at ENUM mapping telephone numbers to DNS

With my talk about Lumicall at FOSDEM coming this Sunday, I'm going to have a quick look at some of the technologies used in Lumicall over this week. The first is ENUM Introduction to ENUM ENUM, as defined by the IETF, is a convenient and powerful way to map telephone numbers to the latest communications technologies such as SIP, XMPP (Jabber) and web sites. ENUM resolution operates much like reverse DNS lookup:
  • Take the number you want to call and normalize it into E.164 format - in other words, by removing any leading zero and punctuation and adding the country code. For example, for a British number "020 7123 4567", the country code is 44 and so the normalized form of the number is +442071234567
  • Remove the leading plus symbol, reverse the digits and insert a dot between each digit. For our sample number +442071234567, it becomes 7.6.5.4.3.2.1.7.0.2.4.4
  • Choose the ENUM suffix. The official ENUM tree defined by the IETF uses the e164.arpa ENUM suffix. There are several third-party suffixes as well, such as e164.org and for the sip5060.net service, use the suffix e164-addr.sip5060.net. For our example, the full string, with suffix, is 7.6.5.4.3.2.1.7.0.2.4.4.e164.arpa
  • Perform a DNS lookup, typically looking for NAPTR records
Here is an example using dig. The subject number we are looking up is 01865 332244 from the Nominet contact page:
$ dig -t naptr 4.4.2.2.3.3.5.6.8.1.4.4.e164.arpa ; > DiG 9.8.4-rpz2+rl005.12-P1 > -t naptr 4.4.2.2.3.3.5.6.8.1.4.4.e164.arpa
;; global options: +cmd
;; Got answer:
;; ->>HEADER ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 3, ADDITIONAL: 4 ;; QUESTION SECTION:
;4.4.2.2.3.3.5.6.8.1.4.4.e164.arpa. IN NAPTR ;; ANSWER SECTION:
4.4.2.2.3.3.5.6.8.1.4.4.e164.arpa. 86400 IN NAPTR 100 20 "u" "E2U+pstn:tel" "!^(.*)$!tel:\\1!" .
4.4.2.2.3.3.5.6.8.1.4.4.e164.arpa. 86400 IN NAPTR 100 10 "u" "E2U+sip" "!^\\+441865332(.*)$!sip:\\1@nominet.org.uk!" . Ready to use today dlz-ldap-enum is a plugin for bind9 nameservers that can read telephone numbers from LDAP and make them available in real time for DNS lookup Most SIP proxies and related products like Asterisk also support ENUM lookups during the process of routing a call. ENUM dialing from Lumicall When dialing any number from your address book on Android, Lumicall can perform an ENUM lookup against the public ENUM tree and any private ENUM trees you configure. This blog is based on a page I published at sip5060.net

25 January 2014

Russell Coker: Links January 2014

Fast Coexist has an interesting article about the art that Simon Beck creates by walking in snow [1]. If you are an artist you can create art in any way, even by walking in patterns in the snow. Russ Altman gave an interesting TED talk about using DNA testing before prescribing drugs [2]. I was surprised by the amount of variation in effects of codeine based on genetics, presumably many other drugs have a similar range. Helen Epstein wrote an interesting article about Dr. Sara Josephine Baker who revolutionised child care and saved the lives of a huge number of children [3]. Her tenacity is inspiring. Also it s interesting to note that the US Republican party was awful even before the Southern Strategy . The part about some doctors opposing child care because it s the will of God for children to die and keep them in employment is chilling. Jonathan Weiler wrote an insightful article about the problems with American journalism in defending the government [4]. He criticises the media for paying more attention to policing decorum than to content. Tobias Buckell wrote an interesting post about the so-called socialised health-care in the US [5]. He suggests that Ronald Reagan socialised health-care by preventing hospitals from dumping dying people on the street. I guess if doing nothing for people until they have a medical emergency counts as socialised health-care then the US has it. Kelvin Thomson MP made some insightful comments about climate change, the recent heat-wave in Australia, and renewable energy [6]. Iwan Baan gave an interesting TED talk about ways that people have built cheap homes in unexpected places [7], lots of good pictures. Racialicious has an interesting article by Arturo R. Garc a about research into the effects of concussion and the way the NFL in the US tried to prevent Dr. Bennet Omalu publicising the results of his research [8]. Stani (Jan Schmidt) wrote an interesting post about how they won a competition to design a commemerative Dutch 5 Euro coin [9]. The coin design is really good (a candidate for the geekiest coin ever), I want one! Seriously if anyone knows how to get one at a reasonable price (IE close to face value for circulated or not unreasonably expensive for uncirculated) then please let me know. When writing about Edward Snowden, Nathan says Imagine how great a country would be if if it were governed entirely by people who Dick Cheney would call Traitor [10]. That s so right, that might make the US a country I d be prepared to live in. Andrew Solomon gave an interesting TED talk Love No Matter What about raising different children [11]. Aditi Shankardass gave an interesting TED talk about using an ECG to analyse people diagnosed wit severe Autism and other developmental disorders [12]. Apparently some severe cases of Autism have a root cause that can be treated with anti-seizure medication. George Monbiot wrote an insightful article about the way that Bono and Bob Geldoff promote G8 government intervention in Africa and steal air-time that might be given to allow Africans to represent themselves in public debates [13]. Daniel Pocock wrote an informative article about racism in Australian politics and how it is bad for job-seekers and the economy (in addition to being horribly wrong) [14]. Aeon Magazine has an interesting article by Anne Buchanan about the difference between scientists and farmers [15]. She has some interesting points about the way that the lack of general knowledge impacts research, but misses the point that in most fields of study there is a huge problem of people not knowing about recent developments in their own field. I don t think it s a pipe dream to be well educated in humanities and science, but I guess that depends on the definition of well educated . Brian Cox gave an interesting TED talk titled Why We Need the Explorers about the benefits of scientific research [16]. Yupu Zhang, Abhishek Rajimwale, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau from the University of Wisconsin-Madison wrote an interesting paper about ZFS corruption in the face of disk and memory errors [17]. One thing to note is that turning off atime can reduce the probability of a memory error leading to corrupt data being written to disk, run zfs set atime=off tank to fix this. The comedian Solomon Georgio celebrated Martin Luther King day by tweeting I love you to racists [18]. It s an interesting approach and appears to have worked well.

24 January 2014

Erich Schubert: Definition of Data Science

Everything is "big data" now, and everything is "data science". Because these terms lack a proper, falsifiable definition.
A number of attempts to define them exist, but they usually only consist of a number of "nice to haves" strung together. For Big Data, it's the 3+ V's, and for Data Science, this diagram on Wikipedia is a typical example.
This is not surprising: effectively these term are all marketing, not scientific attempts at definiting a research domain.
Actually, my favorite definition is this, except that it should maybe read pink pony in the middle, instead of unicorn.
Data science has been called "the sexiest job" so often, this has recently led to an integer overflow.
The problem with these definitions is that they are open-ended. They name some examples (like "volume") but they essentially leave it open to call anything "big data" or "data science" that you would like to. This is, of course, a marketers dream buzzword. There is nothing saying that "picking my nose" is not big data science.

If we ever want to get to a usable definition and get rid of all the hype, we should consider a more precise definition; even when this means making it more exclusive (funnily enough, some people already called above open-ended definitions "elitist" ...).
Big data:
  • Must involve distributed computation on multiple servers
  • Must intermix computation and data management
  • Must advance over the state-of-the-art of relational databases, data warehousing and cloud computing in 2005
  • Must enable results that were unavailable with earlier approaches, or that would take substantially longer (runtime or latency)
  • Must be disruptively more data-driven
Data science:
  • Must incorporate domain knowledge (e.g. business, geology, etc.).
  • Must take computational aspects into account (scalability etc.).
  • Must involve scientific techniques such as hypothesis testing and result validation.
  • Results must be falsifiable.
  • Should involve more mathematics and statistics than earlier approaches.
  • Should involve more data management than earlier approaches (indexing, sketching&hashing etc.).
  • Should involve machine learning, AI or knowledge discovery algorithms.
  • Should involve visualization and rapid prototyping for software development.
  • Must satisfy at least one of these shoulds in a disruptive level.
But this is all far from a proper definition. Partially because these fields are so much in flux; but largely because they're just too ill-defined.
There is a lot of overlap, that we should try to flesh out. For example, data science is not just statistics. Because it is much more concerned with how data is organized and how the computations can be made efficiently. Yet often, statistics is much better at integrating domain knowledge. People coming from computation, on the other hand, usually care too little about the domain knowledge and falsifiability of their results - they're happy if they can compute anything.
Last but not least, nobody will be in favor of such a rigid definition and requirements. Because most likely, you will have to strip that "data scientist" label off your business card - and why bite the hand that feeds? Most of what I do certainly would not qualify as data science or big data anymore with an "elitist" definition. While this doesn't lessen my scientific results, it makes them less marketable.
Essentially, this is like a global "gentlemans agreement". Buzz these words while they last, then move on to the next similar "trending topic".

Maybe we should just leave these terms to the marketing folks, and let them bubble them till it bursts. Instead, we should just stick to the established and better defined terms...
Thank you.
Of course, sometimes you will have to play Buzzword Bingo. Nobody is going to stop you. But I will understand that you are doing "playing buzzword bingo", unless you get more precise.
Once you then have results that are so massively better, and really disrupted science, then you can still call it "data science" later on.
You have been seeing, I've been picking on the word "disruptive" a lot. As long as you are doing "business as usual", and focusing on off-the-shelf solution, it will not be disruptive. And it then won't be big data science, or a big data approach that yields major gains. It will be just "business as usual" with different labels, and return results as usual.
Let's face it. We don't just want big data or data science. What everybody is looking for is disruptive results, which will require a radical approach, not a slight modification involving slightly more computers of what you have been doing all along.

22 January 2014

Clint Adams: December 2013 debian-keyring analysis

Total keys: 996
Key versions:
    996 4
Primary key pubkey algorithms:
    623 "DSA"
    373 "RSA"
Primary key pubkey sizes:
    624 1024
     27 2048
      2 3072
    340 4096
      2 8192
      1 10240
Total number of UIDs + UAts: 4394
Hash algorithm used for most recent self-sig:
      1 "RIPEMD160"
   3188 "SHA1"
   1041 "SHA256"
      1 "SHA384"
    163 "SHA512"
Judgment on preferred hash algorithms:
   1776 null
   2618 "weak hash with higher preference"
Judgment on expiration times:
     53 "expiration passed"
    111 "expiration too far in future"
   3887 "no expiration set"
    343 null
This was generated with debian-keyring 2013.12.13, hopenpgp-tools 0.4-1, jshon 20131010-3, and the following inefficient script:
#!/bin/zsh
infile=/usr/share/keyrings/debian-keyring.gpg
tempfile=$(mktemp)
trap 'rm $ tempfile ' EXIT
hokey lint --output-format JSON <$ infile  >$ tempfile 
print -n "Total keys: "
jshon -a -e keyFingerprint <$ tempfile    wc -l
print "Key versions: "
jshon -a -e keyVer -e val <$ tempfile    sort   uniq -c
print "Primary key pubkey algorithms: "
jshon -a -e keyAlgorithmAndSize -e pubkeyalgo -e val <$ tempfile    sort   uniq -c
print "Primary key pubkey sizes: "
jshon -a -e keyAlgorithmAndSize -e pubkeysize -e val <$ tempfile    sort -n   uniq -c
print -n "Total number of UIDs + UAts: "
jshon -a -e keyUIDsAndUAts -k <$ tempfile    wc -l
print "Hash algorithm used for most recent self-sig: "
jshon -a -e keyUIDsAndUAts -a -e uidSelfSigHashAlgorithms -a -e val <$ tempfile    sort   uniq -c
print "Judgment on preferred hash algorithms: "
jshon -a -e keyUIDsAndUAts -a -e uidPreferredHashAlgorithms -a -e explanation <$ tempfile    sort   uniq -c
print "Judgment on expiration times: "
jshon -a -e keyUIDsAndUAts -a -e uidKeyExpirationTimes -a -e explanation <$ tempfile    sort   uniq -c
This represents incorrect handling of revoked UIDs and user attributes, and possibly unknown bugs. "Judgments" are based on this document and are not generalized per key.

Gunnar Wolf: Finally: A student once again

Formally, today is my first day as a student on a formal, scholarized institution Basically for the first time in almost twenty years! Yes, those that know me know that I aspire to live the life of academia. I have worked at public universities for almost all of my adult life (between 1997 and 1999 I worked at a local ISP and at a private school), and have had a minor academic position ( T cnico Acad mico ) for almost ten years. And not having a proper degree limited me from pursuing anything further. Then, in early 2010 I presented the written exam. By late 2010, the corresponding oral exam. That allowed me to get my formal diploma in December 2010. By the end of 2011, I requested to be a teacher in the Engineering Faculty of UNAM, and started teaching Operating Systems a year ago, in January 2012. So, a good advance in the last few years... But I know that if I just sit here, I won't be able to advance my position towards really entering the Sacred Halls of Academia. And there are some rituals I have to comply with. One of those rituals is... Devoting some long time to studying under the formal structures. Ok, so I'm finally a postgraduate student I have enrolled in Especialidad en Seguridad Inform tica y Tecnolog as de la Informaci n, a short (one year) postgraduate program in ESIME Culhuac n, of Instituto Polit cnico Nacional (a small campus of Mexico's second-largest university). Some friends have asked me, why am I starting with a Specialization and not a Masters degree. Some simple reasons: Just as when I went to Tijuana in 2010 to do my written exam, once I got and started with the paperwork, I didn't want to let it go If I postpone it, I will probably lose the push to do it by May-July, when the Masters admission process starts. Also, this specialization can be linked with the masters degree on the same topic given at the same campus. This program is one year long, and the masters two But having them both takes 2.5 years. So, not such a bad deal after all. And finally, because, after such a long time without being scholarized, I fear not having an easy time getting to grips with the discipline. I can commit to overworking myself for a year If it's too much for me, I'll just stay with that degree and give up. I expect to like it and continue... But it's also a safe bet :-) Now, there has to be a downside to picking up this path: Of course, my free time will be harshly reduced. I have reduced my Debian involvement in the last year, as I devoted a huge chunk of my time to teaching and book-writing... This year... We shall see what happens. I can for now only confirm what I have said publicly but inside our team only: I have requested to my peers and to our DPL to step down as a DebConf chair. I love organizing DebConf, but I don't want to be formally committed to a position I just cannot fulfill as I did when I started with it. As for package maintenance, by far most of my packges are team maintained, and those that are not are relatively easy to keep track of. And of course, I'll keep an eye on my keyring-maint duties as well Will even try to link that work with what I do at school! Anyway, lets see what comes now!

20 January 2014

Dirk Eddelbuettel: RProtoBuf 0.4.0: A whole lot of goodies and Windoze support

A new major release 0.4.0 of RProtoBuf, is now on CRAN. RProtoBuf provides GNU R bindings for the Google Protocol Buffers ("Protobuf") data encoding library used and released by Google, and deployed as a language and operating-system agnostic protocol by numerous projects. With this release, we are welcoming Jeroen Ooms to the team. Jeroen had already worked on some RProtoBuf extensions in the context of his OpenCPU project; we have now integrated those Protocol Buffers functions. And Jeroen pushed all the right buttons to finally get RProtoBuf built on everybody's least-favourite operating system by provding a static library for use by win-builder and CRAN. Murray once again did a lot of work on internals. His use of the LLVM tool llvm-format was particular helpful to make our coding style a little more consistent. The complete NEWS file entry for this release follows:
Changes in RProtoBuf version 0.4.0 (2014-01-14)
  • Changes to support CRAN builds for MS Windows.
  • Added functions serialize_pb, unserialize_pb, and can_serialize_pb plus documentation from Jeroen Ooms RProtoBufUtils package.
  • New dir inst/python with some Python examples.
  • Added Jeroen Ooms as author.
  • Vignettes have been converted to the R 3.0.0 or later use of external vignette builders, no longer need a Makefile
  • Added missing methods to dollar completion list for Message, Descriptor, EnumValueDescriptor, and FileDescriptor classes.
  • Add missing export for .DollarNames EnumValueDescriptor to allow completion on that class.
  • Add more than 15 additional pages to the main Intro vignette documenting better all of the S4 classes implemented by RProtoBuf, updating the type mapping tables to take into account 64-bit support, and documenting advanced features such as Extensions.
  • Added better error checking in EnumDescriptors to catch the case when wrong types are provided.
  • Updated the FileDescriptor name() method to accept a boolean for full paths just like the generic name() method.
  • Correct a bug that incorrectly dispatched as.character() when as.list() was called on Descriptor objects.
  • Update FileDescriptor $ dispatch to work properly for the names of fields defined in the FileDescriptor, instead of just returning NULL even for types returned by $ completion.
  • Added a reservation for extension fields in the example tutorial.Person schema.
  • Support setting int32 fields with character representations and raise an R-level stop() error if the provided string can not be parsed as a 32-bit integer, rather than crashing the R instance.
  • Update the project TODO file.
  • Add better documentation and tests for all of the above.
  • Corrected the handling of uint32 and fixed32 types in protocol buffers to ensure that they work with numbers as large as 2^32 - 1, which is larger than an integer can hold in R since R does not have an unsigned integer class. These values are stored as doubles internally now to avoid losing precision.
  • Added unit tests to verify behavior of RProtoBuf with extreme values for uint32 types.
  • Removed old exception handling code and instead rely on the more modern Rcpp::stop method maintained in Rcpp.
  • Add better error messages when setting a repeated field of messages to inform the user which element index was of the wrong type and what the expected type was.
  • Add an optional 'partial' argument to readASCII allowing uninitialized message fragments to be read in.
  • (internal) Added const qualifiers in more places throughout the C++ code for type safety.
  • (internal) Standardize coding conventions of the C++ files and run them through clang-format for consistency. A STYLE file has been submitted to R-Forge with details about the coding standards and how they are enforced with Emacs and clang-format.
  • Applied changes suggested by Kevin Ushey to the S4 class handling to support both the currently released Rcpp as well as the currently pending next version.
CRANberries also provides a diff to the previous release 0.3.2. More information is at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

17 January 2014

Michael Stapelberg: debmirror 6x speed-up, or how to fill a 1 GBps Rackspace link

As explained in more detail in my my last blog post, Rackspace is providing hosting for Debian Code Search. For those of you who don t know, Rackspace is a cloud company that provides (among other services) a public cloud based on OpenStack. That means you can easily (and programmatically, if you want) bring up virtual servers, block storage volumes, configure the network between them, etc. As part of my initial performance experiments, I was running debmirror to clone a full Debian source mirror, and I noticed this took about one hour. Given that the peak network and storage write rates I have observed in my benchmarks are much higher, I wondered why it took so long. About 43 GB (that s how big the Debian sid sources are) in 60 minutes means 12 MB/s download rate, which is a bit more than a sustained 100 MBit/s connection. Luckily, at least the Rackspace servers I looked at are connected with 1 GBit/s to the internet, so more should be possible. Note that these are the old Rackspace servers. There is a new generation of servers, which I have not yet tried, that apparently offer even higher performance. After a brief look at the debmirror source code, I concluded that it can only use a single mirror and downloads files sequentially. There is some obvious potential for improvement here, and the fact that I could come up with a proof of concept written in Go to determine the files to download in a couple of minutes encouraged me to spend my saturday on this problem :-). About 7 hours later, my prototype had gone through various iterations and the code could sustain about 115 MB/s incoming bandwidth for most of the time. Here is a screenshot of dstat measuring the performance: Another 3 hours later, the most obvious bugs were weeded out and the code successfully cloned an entire mirror for the first time. I verified the correctness by running debmirror afterwards, and no files were downloaded that had not actually arrived on the mirror in the meantime. Features
  1. Parallel downloading from multiple mirrors.
    Of course, it is not guaranteed that the mirrors are consistent, but that is a solvable problem: whenever a non-200 HTTP response is encountered, the file is put back into the queue and gets rescheduled onto any mirror. Eventually, it will be rescheduled onto the mirror from which the sources.gz was downloaded, and that mirror has to have it.
  2. HTTP keep-alive and pipelining.
    debmirror also does HTTP keep-alive (I read that in the source, did not verify it), but it downloads files sequentially, i.e. it downloads one file, then sends the request for the next file. This means that after one file was received, no data is being received until another round-trip finished. For a lot of small files (think of .dsc and .debian.tar.gz files or small orig tarballs) that really adds up.
    Therefore, I use HTTP pipelining: I send 99 requests and then just receive all the responses at full speed. The magic number 99 comes from the fact that this is the smallest common denominator that all mirrors I have tested allow. nginx by default allows 100 requests.
  3. Request ordering.
    Instead of sending requests at random, I sort the download queue by size and make sure that the first request on each connection is for a big file. The intention is that TCP will adapt to that quickly and use a bigger window size as soon as possible instead of ramping up slower later on. Note that I have not actually measured the window sizes, so this is just a hunch.
    Another nice side-effect of the ordering is that the amount of data that is being sent through each connection is roughly equal. This means we don t waste TCP connections by sending 99 requests for small files only.
  4. (Goroutines).
    I m a bit hesitant to even write about it, so let s make it quick: Perl (and thus debmirror) is typically single-threaded, whereas Goroutines scale nicely on multi-core systems. I don t think it makes a big difference on the machines I am running this on as they have only two cores and are not maxed out on CPU usage at all. But maybe this will be more important for saturating 10 GBit/s? :-)
  5. (Mirror selection).
    It s not a feature of the code, but certainly relevant: of course I selected mirrors that are connected to the internet with at least a Gigabit link, serve files reasonably fast and have low latency to Rackspace s Chicago presence.
    Interestingly, while I used to think that universities have access to big uplinks and beefy machines, none of the mirrors that I use are located at a university. In fact, every mirror ending in .edu that I tried was really slow most of them providing something like 2-4 MB/s.
Results With the latest iteration of the code, I can clone an entire Debian source mirror with 43 GB of data in about 11.5 minutes, which corresponds to a 63 MB/s download rate:
GOMAXPROCS=20 ./dcs-debmirror -num_workers=20
68,24s user 384,90s system 66% cpu 11:25,97 total
It should be noted that the download rate is very low in the first couple of seconds since the sources.gz file is downloaded from a single mirror, then unpacked and analyzed. The peak download rate is about 115 MB/s (= 920 MBit/s) which is reasonably close at what you can achieve with a Gigabit link, I think. If the entire uplink was available to me at all time, the Rackspace hardware would be able to saturate that easily, both in terms of reading from the network and in terms of writing to block storage. I tested this on an SSD volume, but I see about 113 MB/s throughput with the internal hard disk, so I think that should work, too. There is another dstat screenshot of the final version (writing to disk). Perhaps even more interesting to some readers is the time for an incremental update:
GOMAXPROCS=20 ./dcs-debmirror -tcp_conns=20
2013/10/13 11:09:55 Downloading 307 files (447 MB total)
 
2013/10/13 11:10:04 All 307 files downloaded in 9.105605931s. Download rate is 49.186567 MB/s
4,91s user 3,99s system 67% cpu 13,205 total
The wall clock time is higher than the time reported by the code because the code does not count the time for downloading and parsing the sources.gz file. The entire program is about 400 lines (not SLOC) of Go code. It s part of the Debian Code Search source. If you re interested, you can read dcs-debmirror.go in your browser. Conclusions The outcome of this experiment is that I now know (and have shown!) that there are significantly more efficient ways of cloning a Debian mirror than what debmirror does. Furthermore, I have a good grasp on what kind of performance the Rackspace cloud offers and I am fairly happy with the numbers :-). My code is useful to me in context of Debian Code Search, but unless you need a sid-only source-only mirror, it will not be useful to you directly. Of course you can take the ideas that I implemented and implement them elsewhere personally, I don t plan to do that. If you have hardware, bandwidth and a use-case for 10 GBit/s+ mirroring, I d like to hear about it! :-)

Sylvestre Ledru: Debian & LLVM events

Being a bit hyperactive, I have been involved in the organization of two events. I am the main organizer with Alexandre Delano of the Mini Debconf 2014 in Paris, January 18 & 19th. The (great) planning is available here:
https://france.debian.net/events/minidebconf2014/
Saturday morning presentations will be general public, the beginning of Saturday afternoon will be used by the Debian France association to vote the new status (1901 law and Debian Trusted Organization).
Sunday will be more focused on Debian itself.
During the week end, I will be talking about the Debile project, the finance of Debian France and be part of the round table on compiler selection for Debian.
The (mandatory) registration should be done on the Wiki or meetup.com In parallel, with Tobias Grosser, we organized the LLVM devroom track at FOSDEM (Bruxelles), February 2nd (Sunday).
The schedule is a mix between core developers, third party software using LLVM / Clang and academic users.
https://fosdem.org/2014/schedule/track/llvm/
I will be talking on how to become a LLVM contributor. Both events should be recorded.

Steve Kemp: So I found a job.

Just to recap my life since December:
I had worked with Bytemark for seven years and left for reasons which made sense. I started working for "big corp" with a job that on-paper sounded good, but ultimately turned out to be a poor fit for my tastes. I spent a month trying to decide "Is this bad, or is this just not what I'm used to?", because I was aware that there would obviously be big differences as well as little ones. At the point I realized some of the niggles could be fixed but most couldn't then I resigned, rather than prolong the initial probationary training period - because I knew I wouldn't stay, and it seemed unfair and misleading to stay for the full duration of the probationary period knowing full well I'd leave the moment it concluded - and the notice period switched from seven days to one month.
A couple of people were kind enough to get in touch and discuss potential offers, both locally, remotely in the UK, and from abroad (the latter surprised me, but pleased me too). I spent a couple of days "contracting", by which I really mean doing a few favours for friends, some of whom paid me in Amazon vouchers, and some of whom paid me in beer.
e.g. I tweaked the upcoming death Knight site to handle 3000 simultaneous HTTP connections, then I upgraded some servers from Squeeze to Wheezy for some other folk.
That aside I've largely been idle for about 10 days and have now picked the company to work for - so I'm going to be a contractor with a day-rate for an American firm for the next couple of months. If that goes well then I'll become a full-time employee, hopefully.

16 January 2014

Daniel Pocock: Get ready to try Debian SIP

Before going on to talk about SIP, there is one really important thing that must be emphasized: XMPP/Jabber is also in the roadmap. Some of the infrastructure that has already been deployed (the new RTC passwords in ud-ldap, the TURN server and the RADIUS server) is good for both SIP and XMPP use cases. All to be revealed on Sunday As mentioned in my recent blog about the Debian.org NAPTR record, more details will be coming in my talk at mini-DebConf Paris this weekend. Today I'm just going to describe some things to install for those who want a head start. Jitsi Jitsi has been around for many years (originally as SIP Communicator) and it is probably the most advanced desktop softphone in the free software world. Jitsi packages are in jessie and sid - why not install them now. If you don't want to pull in dependencies from jessie on your wheezy system, you can also download from upstream and run it standalone. WebRTC browsers I've prepared a JSCommunicator implementation based on the jscommunicator-web-phone package. To use it, you need the latest version of Iceweasel/Firefox or Chromium. As HTML5 WebRTC / RTCPeerConnection is an evolving standard, using a recent browser is essential. Jitsi configuration for sip:debian.org It is not open for users just yet, but this is how you will configure it when it is available. More details to come...
AttachmentSize
step-4-setup-connection-scaled.png59.66 KB

Andrew Pollock: [life] And now for something completely different...

After 8 years, February 6 will be my last day with Google. The hardest pill to swallow with the end of my marriage was the reduction in my time with Zoe. It was one thing to lose my wife, but to lose 50% of my daughter as well has been, and will continue to be, a struggle for me. 2014 is the last year before Zoe starts in Prep (Queensland's school grade before "one"). This is my last opportunity to be able to have a big chunk of time with her before she gets swallowed up by the school system. It's ridiculous to be sticking her in full-time long day care if I have an alternative. And I do have an alternative. I'm in the fortunate financial position of not having a mortgage, or any debt to speak of, and a modest amount of savings. I shall be Daddy Day Care. It had always been my career goal in life to be a stay at home Dad. I'd wanted to have another child and be the one to stay at home and look after the kids, and allow Sarah to resume her career. Obviously none of that worked out, but there's no reason why I can't be a stay at home Dad for at least this year. Zoe's at a wonderful age now. We can have a lot of fun together this year. She's going to be in a 5 day a fortnight Kindergarten program (Monday, Tuesday and every other Wednesday) between 8:20am to 2:30pm, so I'm planning on using one of those days for Debian contributions, one of those days for exploring some of the crazy startup ideas I have rattling around in my head, and every second Wednesday to keep the home tidy. It's going to be a fun, different year for Zoe and I, and I'm so looking forward to it. I'm planning on doing lots of activities from Play at Home Mom, doing lots of baking and we'll be doing Science Friday's. If anyone knows where I can get a toddler-sized lab coat, drop me a line. I'm hoping I can make time to write up our activities for posterity as well.

14 January 2014

Daniel Leidert: Debian Sid and GNOME 3 - Restoring automount function of USB devices

I recently found, that the automount feature of my recent Debian Sid / GNOME 3 system didn't work anymore. For example: I plugged in an USB stick. Nothing happened, although dmesg clearly showed, that the system recognized it. After manually mounting and unmounting the device, nautilus automatically offered to automount it. But I always had to manually mount and unmount it first. Seems the culprit was a pmount installation. JFTR: In some cases also usbmount might cause trouble.

Next.

Previous.