Search Results: "cas"

17 December 2012

Ian Campbell: Foreign Chroots with schroot and qemu

Since I've been working on qcontrol I've had need of a way to build packages in an ARM environment. Since all my native ARM hardware is "production" (at least so far as something can be production in a home environment) I prefer to avoid installing development tools on them. When I started looking into this all my ARM systems ran from relatively small flash devices which also ruled out the possibility of a development chroot. After search the web a bit I discovered that it is possible to use qemu-user-static together with [[!binfmt-support ]] to create cross architecture chroots which can be entered mostly transparently. It is also possible to combine this with schroot and sbuild which makes the day to day use pretty much the same as a native or biarch chroot. Since I've now had to rediscover how to do this twice (once for Sid and then again when I realized I also needed a Wheezy environment) I thought I should write it down for the benefit of my future self. First things first lets install the various packages which we need. As well as debootstrap to create the chroot we need qemu-user-static and binfmt-support to allow us to run ARM binaries:
$ sudo apt-get install debootstrap qemu-user-static binfmt-support
I'm using schroot's lvm-snapshot mode so we need an LVM volume, formatted with a filesystem and mounted in a temporary location:
$ sudo lvcreate -L8G -n sbuild-wheezy-armel /dev/VG
$ sudo mkfs.ext3 /dev/VG/sbuild-wheezy-armel
$ sudo mount /dev/VG/sbuild-wheezy-armel /mnt/
Next we need to create the chroot. For this we use debootstrap in foreign mode which does a 2 stage setup, firstly by using the --foreign option:
$ sudo debootstrap --variant=buildd --include=fakeroot,build-essential \
        --arch=armel --foreign \
        wheezy /mnt http://<DEBIAN-MIRROR>/debian/
Now we (temporarily) copy the qemu-arm-static binary into the chroot (at the place where binfmt-support expects it), change into the chroot and run debootstrap's second stage to finish the basic chroot setup:
$ sudo cp /usr/bin/qemu-arm-static /mnt/usr/bin/
$ sudo chroot /mnt
(chroot)# ./debootstrap/debootstrap --second-stage
(chroot)# exit
Now that we have a basic chroot in place it needs a little bit of tailoring. I've found the easiest way to do this is to use sbuild-createchroot and then tweak the result:
$ sudo sbuild-createchroot --arch=armel --foreign --setup-only \
    wheezy /mnt http://<DEBIAN-MIRROR>/debian/
If we were building a native chroot then we could have skipped the debootstrap stuff and let sbuild-createchroot handle things. Unfortunately this doesn't seem to play nicely with --foreign. Now we can remove the qemu-arm-static binary from the chroot. schroot will automatically bind mount the version from the host into the chroot which ensures it remains up to date. We can also unmount the chroot since we will let schroot manage that from here on:
$ sudo rm /mnt/usr/bin/qemu-arm-static
$ sudo umount /mnt
Next we need to tweak the resulting schroot/sbuild configuration since sbuild-createchroot assumes the use of schroot's directory mode. First remove the symlink to /mnt which it created:
$ sudo rm /etc/sbuild/chroot/wheezy-armel-sbuild
Lastly we need to edit /etc/schroot/chroot.d/wheezy-armel-sbuild-<SUFFIX> (where is randomly generated) and reconfigure it to use lvm-snapshot mode, by apply the following changes:
 [wheezy-armel-sbuild]
-type=directory
-directory=/mnt
+type=lvm-snapshot
+device=/dev/VG/sbuild-wheezy-armel
+lvm-snapshot-options=--size 2G
 description=Debian wheezy/armel autobuilder
 groups=root,sbuild
 root-groups=root,sbuild
 profile=sbuild
That's it, we now have a chroot which we can trivially invoke via sbuild to build armel packages:
$ sbuild --arch=armel --dist=wheezy <... other options...>
We can also use sbuild-update to keep the chroot up to date:
$ sudo sbuild-update -udcar wheezy-armel
You can also use schroot to login to the chroot directly, although for that case you'll likely want to create a different chroot using schroot's directory or block-device mode instead lvm-snapshot and drop some of the sbuild specific setup. I expect I'll blog about this configuration in some more detail at some point in the future since I'll want to document it to give a more convenient build environment for Xen on ARM than the build farm which we have at work.

14 December 2012

Joachim Breitner: Pl tzchentetris

What happens if the same person (in this case, my girlfriend) is both a bit geeky and likes to bake? She has ideas leading to this:
Tetris Biscuits
And what is the obvious thing to do with such biscuit (or are they cookies?). Play tetris:
<video controls="true" height="720" poster="http://www.joachim-breitner.de/various/Plaetzchentetris-poster.jpg" tabindex="0" width="960"> <source src="http://www.joachim-breitner.de/various/Plaetzchentetris.webm" type="video/webm"> <source src="http://www.joachim-breitner.de/various/Plaetzchentetris.avi" type="video/x-msvideo"> Your browser does not support the video tag. Try youtube instead! </video>
(Also on YouTube if your browser does not play the video above.) Oh, and to refute common stereotypes about me: The baking was done by me, but she helped me glazing the biscuits.

13 December 2012

Andrea Veri: The future is Cloudy

Have you ever heard someone talking extensively about Cloud Computing or generally Clouds? and have you ever noticed the fact many people (even the ones who present themselves as experts) don t really understand what a Cloud is at all? That happened to me multiple times and one of the most common misunderstandings is many see the Cloud as something being on the internet. Many companies add a little logo representing a cloud on their frontpage and without a single change on their infrastructure (but surely with a price increment) they start calling their products as being on the Cloud. Given the lack of knowledge about this specific topic people tend to buy the product presented as being on the Cloud without understanding what they really bought. But what Cloud Computing really means? it took several years and more than fifteen drafts to the National Institute of Standards and Technology s (NIST) to find a definition. The final accepted proposal:

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

The above definition requires a few more clarifications specifically when it comes to understand where should we focus on while checking for a Cloud Computing solution. A few key points:
  1. On-demand self-service: every consumer will be able to unilaterally provision multiple computing capabilities like server time, storage, bandwidth, dedicated RAM or CPU without requiring any sort of human interaction from their respective Cloud providers.
  2. Rapid elasticity and scalability: all the computing capabilities outlined above can be elastically provisioned and released depending on how much demand my company will have in a specific period of time. Suppose the X company is launching a new product today and it expects a very large number of customers. The X company will add more resources to their Cloud for the very first days (where they suppose the load to be very high) and then it ll scale the resources back as they were before. Elasticity and scalability permit the X company to improve and enhance their infrastructure when they need it with an huge saving in monetary terms.
  3. Broad network access: capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, tablets, laptops, and workstations).
  4. Measured service: Cloud systems allow maximum transparency between the provider and the consumer, the usage of all the resources is monitored, controlled, and reported. The consumer knows how much will spend, when and in how long.
  5. Resource pooling: each provider s computing resources are pooled to serve multiple consumers at the same time. The consumer has no control or knownledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).
  6. Resources price: when buying a Cloud service make sure the cost for two units of RAM, storage, CPU, bandwidth, server time is exactly the double of the price of one unit of the same capability. An example, if a provider offers you one hour of bandwitdh for 1 Euro, the price of two hours will have to be 2 Euros.
Another common error I usually hear is people feeling Cloud Computing just as a place to put their files online as a backup or for sharing them with co-workers and friends. That is just one of the available Cloud features , specifically the Cloud Storage , where typical examples are companies like Dropbox, Spideroak, Google Drive, iCloud and so on. But let s make a little note about the other three features :
  1. Infrastructure as a Service (IaaS): the capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. In this specific case the consumer has still no control or management over the underlying Cloud infrastructure but has control over operating systems, storage, and deployed applications. A customer will be able to add and destroy virtual machines (VMs), install an operating system on them based on custom kickstart files and eventually manage selected networking components like firewalls, hosted domains, accounts.
  2. Platform as a Service (PaaS). the capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages, libraries, services, and tools (like Mysql + PHP + PhpMyAdmin or Ruby on Rails) supported by the provider. In this specific case the consumer has still no control or management over the Cloud infrastructure itself (servers, OSs, storage, bandiwitdh etc.) but has control over the deployed applications and configuration settings for the application-hosting environment.
  3. Software as a Service (SaaS): the capability provided to the consumer is to use the provider s applications running on a Cloud infrastructure. The applications are accessible through various client devices, such as a browser, a mobile phone or a program interface. The consumer doesn t not manage nor control the Cloud infrastructure (servers, OSs, storage, bandwidth, etc.) that allows the applications to run. Even the provided applications aren t customizable by the consumer, which should rely on limited configuration settings.
The Cloud Computing technology is reasonably the future but can we trust Cloud providers? Are we sure that no one will ever have access to our files except us? and what about governments interested in acquiring a specific customer data hosted on the Cloud? I always suggest to read deeply both the Privacy Policy and Terms of Use of a certain service before signing in especially when it comes to choose a Cloud storage provider. Many providers have the same aspect, they seem to provide the same resources, the same amount of storage for the same price but legally they may present different problems, and that is the case of Spideroak vs Dropbox. Quoting from the Dropbox s Privacy Policy:
Compliance with Laws and Law Enforcement Requests; Protection of DropBox s Rights. We may disclose to parties outside Dropbox files stored in your Dropbox and information about you that we collect when we have a good faith belief that disclosure is reasonably necessary to (a) comply with a law, regulation or compulsory legal request; (b) protect the safety of any person from death or serious bodily injury; (c) prevent fraud or abuse of DropBox or its users; or (d) to protect Dropbox s property rights. If we provide your Dropbox files to a law enforcement agency as set forth above, we will remove Dropbox s encryption from the files before providing them to law enforcement. However, Dropbox will not be able to decrypt any files that you encrypted prior to storing them on Dropbox.
It s evident that Dropbox employees can access your data or be forced by legal process to turn over your data unencrypted. On the other side, Spideroak on its latest update to its Privacy Policy states that data stored on their Cloud is encrypted and inaccessible without user s key, which is stored locally on user s computers. And what about the latest research paper, titled Cloud Computing in Higher Education and Research Institutions and the USA Patriot Act written by the legal experts of the University of Amsterdam s Institute for Information Law stating the anti-terror Patriot Act could be theoretically used by U.S. law enforcement to bypass strict European privacy laws to acquire citizen data within the European Union without their consensus? The only requirement for the data acquisition is the provider being an U.S company or an European company conducting systematic business in the U.S. For example an Italian company storing their documents (protected by the European privacy laws and under the general Italian jurisdiction) on a provider based in Europe but conducting systematic business in the United States, could be forced by U.S. law enforcement to transfer data to the U.S. territory for inspection by law enforcement agencies. Does someone really care about the privacy of companies, consumers and users at all? or better does privacy exists at all for the millions of the people that connect to the internet every day?

12 December 2012

Vincent Sanders: What man's mind can create, man's character can control.

I have a project that required me to programmatically control power to several devices. I have done this before using a Velleman vm8090 board which is relatively easy to control. However they are relatively expensive.

I turned to ebay and found a similar module at a substantially reduced cost. Upon receipt however I discovered that instead of being a simple serial USB interface it presented USB HID and the Debian system I was running it on has loaded the hiddev driver for me but it did not implement any of the standard HID Usage Pages leaving me with no way to control the device.
I did the obligatory
sudo lsusb -d 12bf:ff03 -vvv

Bus 003 Device 019: ID 12bf:ff03  
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 1.10
bDeviceClass 0 (Defined at Interface level)
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 8
idVendor 0x12bf
idProduct 0xff03
bcdDevice 1.00
iManufacturer 1 Matrix Multimedia Ltd.
iProduct 2 Flowcode USB HID
iSerial 0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 41
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0x80
(Bus Powered)
MaxPower 50mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 2
bInterfaceClass 3 Human Interface Device
bInterfaceSubClass 0 No Subclass
bInterfaceProtocol 0 None
iInterface 0
HID Device Descriptor:
bLength 9
bDescriptorType 33
bcdHID 1.10
bCountryCode 0 Not supported
bNumDescriptors 1
bDescriptorType 34 Report
wDescriptorLength 54
Report Descriptors:
** UNAVAILABLE **
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0008 1x 8 bytes
bInterval 5
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x01 EP 1 OUT
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0008 1x 8 bytes
bInterval 5
Device Status: 0x0980
(Bus Powered)
This simply showed me what I already knew and surprised me that lsusb did not dump HID report descriptor items. Some searching revealed that teh device had to be unbound so lussb could access the descriptor.

Thus a simple
echo -n 3-1.1.4:1.0   sudo dd of=/sys/bus/usb/drivers/usbhid/unbind
resulted in lussb dumping the descriptor items:

Item(Global): Usage Page, data= [ 0xa0 0xff ] 65440
(null)
Item(Local ): Usage, data= [ 0x01 ] 1
(null)
Item(Main ): Collection, data= [ 0x01 ] 1
Application
Item(Local ): Usage, data= [ 0x02 ] 2
(null)
Item(Main ): Collection, data= [ 0x00 ] 0
Physical
Item(Global): Usage Page, data= [ 0xa1 0xff ] 65441
(null)
Item(Local ): Usage, data= [ 0x03 ] 3
(null)
Item(Local ): Usage, data= [ 0x04 ] 4
(null)
Item(Global): Logical Minimum, data= [ 0x00 ] 0
Item(Global): Logical Maximum, data= [ 0xff 0x00 ] 255
Item(Global): Physical Minimum, data= [ 0x00 ] 0
Item(Global): Physical Maximum, data= [ 0xff ] 255
Item(Global): Report Size, data= [ 0x08 ] 8
Item(Global): Report Count, data= [ 0x08 ] 8
Item(Main ): Input, data= [ 0x02 ] 2
Data Variable Absolute No_Wrap Linear
Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Local ): Usage, data= [ 0x05 ] 5
(null)
Item(Local ): Usage, data= [ 0x06 ] 6
(null)
Item(Global): Logical Minimum, data= [ 0x00 ] 0
Item(Global): Logical Maximum, data= [ 0xff 0x00 ] 255
Item(Global): Physical Minimum, data= [ 0x00 ] 0
Item(Global): Physical Maximum, data= [ 0xff ] 255
Item(Global): Report Size, data= [ 0x08 ] 8
Item(Global): Report Count, data= [ 0x08 ] 8
Item(Main ): Output, data= [ 0x02 ] 2
Data Variable Absolute No_Wrap Linear
Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Main ): End Collection, data=none
Item(Main ): End Collection, data=none

By consulting the device class definitions document I determined the device was using the "Vendor defined" Usage page (0xff00 to 0xffff) so I would definitely have to write a program to control the device.

Linux provides a really easy interface to deal with HID devices called hiddev (gosh, such adventurous naming) which I already had to unbind to get my descriptors decoded so I am fairly sure it works ;-)

The kernel documentation and header for hiddev provide the absolute basic mechanics of the interface but no example code or guidance. The obligatory web search turned up very little and even that had to be retrieved from the internet archive. So It seems I would be forced to work it through myself.

It seems the hiddev interface is orientated around HID devices generating reports which the program is expected to read. Numerous ioctl() are provided so the program can obtain the descriptor information necessary to control and process the received reports.

However in this case we need to be able to send reports to the device, all the descriptor information revealed was that there were eight (Item Report Count = 8) values with eight bits each (Item Report Size = 8) with logical and physical values representing the whole range of the octets.

Fortunately the seller provided a website with some control programs and even source. After some time rummaging through the Visual Basic program I finally found (in FrmMain.vb:2989) that the eight bytes were largely unused and the first was simply a bitmask of the eight relays coil status, set for energised clear for off. With bit 0 controlling relay labelled 1 through to bit 7 for relay 8.

To send a report to a HID device the hiddev interface uses the HIDIOCSREPORT ioctl where the report data is first set using HIDIOCSUSAGE .

The HIDIOCSUSAGE ioctl is passed a hiddev_usage_ref structure which must be initialised with information about the report descriptor identifier (constructed from the Usage Page and Usage as set by the items in the descriptor), the index of the item (named usage) we wish to set in the report (in this case the first which is 0) and the value we actually want to set.

After a great deal of debugging the final program is very short indeed but does the job, my main problem now is that if I switch too many (more than one) relays at once the whole device resets. The scope says the supply rails are behaving very badly when this happens, looks like I need to add a load of capacitance to the power well to stabilise it during the switching events.

Oh and add in the fact Relay 1 LED doest work unless you push on it and I do wonder about the wisdom of the economy in this case. Though yet again Linux makes the software side easy.

Sylvestre Ledru: Mini Debconf 2012 - videos and feedbacks

A bit more than two weeks after the Mini Debconf in Paris, I am glad to say that the videos of the event are finally published (the sound is not very good for the 4 first presentations, sorry about that).
They will be also available on the new IRILL website with a video player when ready.
All slides are also available on the page of the event. I believe that there is a consensus about the quality of the event. We had around 150 people attending to the event, many interesting and various talks.
As usual, it was nice to meet some old and new friends from Debian. Mini debconf - group picture
Group picture by Frederic Lehobey
For those who wonder, I am confident there will be a 2013 Parisian Mini Debconf. Various feedbacks about the event:
Lucas Nussbaum
Stefano Zacchiroli
Vincent Untz
Raphael Hertzog
Pietro Abate
Logilab (Julien Cristau)
The 'official' Debian news And, once more, many thanks to the sponsors!
Logilab SmartJog
Bearstech Evolix
IRILL

Raphael Geissert: A bashism a week: $RANDOM numbers

Commonly used to sleep a random amount of time or to create unique temporary file names, $RANDOM is one of those bashisms that you are best avoiding it altogether.

It is not uncommon to see scripts generating a "unique" temporary file name with code that goes like: tempf="/tmp/foo.$RANDOM", or tempf="/tmp/foo.$$.$RANDOM".

Under some shells the "unique" temporary file name will be "/tmp/foo." for the first example code. So much for randomness, right?

Even if you go around it by defining $RANDOM to the output of cksum after reading some bytes from /dev/urandom, please: don't do that. Use the mktemp command instead.
When creating temporary files there's more than just generating a file name. Just don't do it on your own: use mktemp. Really, use it, the list of those who weren't using mktemp (or similar) is large enough as it is.

Don't even dare to mention the linux kernel-based protection against symlink attacks. There's no excuse for not using mktemp.

Tip: If you are going to use multiple temporary files, create a temporary directory instead. Use mktemp -d.
Tip: Don't reuse a temporary file's name, even if you unlink/remove it. Generate a new one with mktemp.
Tip: Reusing also means doing things like tmp="$(mktemp)"; some_command > "$tmp.stdout" 2> "$tmp.stderr"
Tip: Even if $RANDOM is not empty, don't use it. It could have been exported as an environment variable. Again, just use mktemp.

For the remaining cases where you may want a pseudo random number such as for sleeping a random number of seconds: you can use something as simple as $$. Use shell arithmetic to adjust it as needed: use the modulo operator, multiply it, etc.

If you think you need something more "random" than the process' id, then you should probably not be using $RANDOM in the first place.

11 December 2012

Francesca Ciceri: Welcome Malatesta, Goodbye Kasbah!

Kasbah, my old laptop, is almost dead. It was my first laptop, an Asus X50C, which worked like a charm and was robust like a tank. But then the screen hinges broke. Time for a new one. So, ladies and gentlemen, I'm glad to present you malatesta. Malatesta is a Lenovo Thinkpad E335. Sadly, while kasbah worked with free drivers, malatesta requires proprietary drivers both for the video card and the wifi. Which sounds a bit ironic for a laptop named after a famous anarchist. The only two problems I encountered with it were related to the video card (00:01.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Device 9808) and the special keys. Malatesta worked well with the kernel of Squeeze, but freezed during the boot with Wheezy. Thankfully, I stumbled upon this blogpost by Blars Blarson suggesting to use the fglrx-driver with Sid. And that solved the problem. Another little odissey was the brightness of the monitor. With kernels < 3.2.0-4-amd64 the thinkpad_acpi didn't recognize my model and I was stuck with a brightness level probably visible from the outer space. Neither xrandr nor xbacklight could help, and I was seriously considering to use sunglasses to work (a slight unusual approach to the brightness problem, I know, but I suffer of migraines and couldn't bear an hyper-bright monitor), when the upgrade to Sid solved the problem. On the bright side (pun intended) this laptop was born without Windows. (Both malatesta and kasbah were shipped with FreeDOS). Now, if you're wondering about the names: my first namescheme was "references to songs by the Clash" (and kasbah is obviously for "Rock the Casbah", with a typo). But it wasn't really scalable. Since last year, I've decided to name my systems after famous anarchists. The server is named after one of my all time favourite anarchists, Buenaventura Durruti.
madamezou@durruti:~$ cat /etc/motd
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Llevamos un mundo nuevo en nuestros corazones;
y ese mundo est  creciendo en este instante
                        Buenaventura Durruti
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
While the new laptop's name comes from Errico Malatesta.
madamezou@malatesta:~$ cat /etc/motd 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We anarchists do not want to emancipate the people;
We want the people to emancipate themselves.
            Errico Malatesta, L'Agitazione (18 June 1897)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Russell Coker: Dependencies in Online Ordering

I have just ordered two Samsung Galaxy S3 phones and matching cases from Kogan. The price was good and Kogan gave me 30 cents discount as part of a verification process. Instead of billing the full amount for a large order (for which the cutoff is somewhere between $25 and $1014) Kogan will deduct a random number of cents and demand that you confirm the amount of money on your credit card statement, if you don t know the amount that was billed then it s a fraudulent transaction. But now Kogan have decided that my phones and cases are separate things, they have dispatched the cases but not the phones. This is really annoying, I will have to arrange to receive two separate parcels and the first of which won t be of any immediate use to me. I can use a phone without a case, but a case without a phone isn t useful. If I had wanted to receive two parcels then I would have gone through the checkout process twice! It should have been really obvious to Kogan that I didn t want two parcels, they suggested that I buy cases after I selected the phones, so while it s theoretically possible that I might want to buy two new phones at the same time as buying two cases for entirely unrelated phones I don t think that the people who programmed the Kogan web server expected that to be the case.

Russell Coker: Links December 2012

Steven Johnson gave an interesting TED talk about where good ideas come from [1]. He starts by attributing coffee replacing alcohol as a standard drink for some good ideas and then moves on to how ideas develop. Erez Lieberman Aiden and Jean-Baptiste Michel gave an interesting and amusing TED talk about the ngram analysis of books that Google scanned [2]. Here is the link for the Google Books ngram search [3]. Clay Shirky gave an insightful TED talk about how the Internet is changing the world [4]. He cites Open Source programmers as the modern day equivalent to the Invisible College based on our supposed great ability to get along with other people on the Internet. If we really are so much better than the rest of the Internet then things must be bad out there. He ends with ways of using Git to draft legislation. Hans Rosling gave an interesting TED talk about religion and the number of babies that women have [5]. His conclusion is that it s more about income and social stability and that the world s population can stabilise at 10 billion if we provide family planning to everyone. Alexis C. Madrigal wrote an interesting interview with Genevieve Bell about her work at Intel and the way people use technology [6]. Indigogo is raising funds for the Cuddle Mattress , it s a mattress with foam slats and a special fitted sheet to allow your arm to slide between the slats [7]. So you could have your arm underneath your SO for an extended period of time without risking nerve damage. They also show that when sleeping on your side your shoulder can go between the slats to avoid back problems. Nate Silver (who is famous for predicting US elections gave an interesting TED talk about racism and politics [8]. One of his main points is to show the correlation between racism and lack of contact of members of other races. Sociological Images has an interesting article by Lisa Wade about whether marriage is a universal human value [9]. In regard to historical marriage she says women were human property, equivalent to children, slaves, servants, and employees . The general trend in the comments seems to be that there are so many types of marriage that it s difficult to make any specific claims to traditional marriage unless you count a tradition of a short period in a single geographic region. Plurality is an excellent sci-fi short movie on youtube [10]. TED has an interesting interview with Hakeem Oluseyi about his research about astrophysics and how he achieved a successful career after being a gangster as a teenager [11]. He has some good ideas about helping other children from disadvantaged environments become successful. Paul Dwerryhouse wrote an interesting blog post about his work in designing and implementing a filesystem based on a Cassandra data store with FUSE [12]. Paul also wrote a post about using Apache Zookeeper to lock metadata to permit atomic operations [13]. The documentary Monumental Myths provides an interesting and insightful analysis of the processes of creating, maintaining, and explaining monuments [14]. It focusses on some significant monuments in the US and explains both sides to each story. Howard Zinn makes the insightful point that when people present a certain point of view of history it s not controversial, as soon as you present the other side they call it controversial . That happens even in debates about current issues. Howard also says to criticise whatever the government does is not anti-America, it s anti-government, it s pro-America, it s pro the people, it s pro the country . The song that plays during the closing credits is interesting too. The music video Same Love is one of the best presentations of the argument for marriage equality [15]. Chris Samuel wrote an interesting post about systems locked down for Windows 8 and options for purchasing PCs for running Linux [16]. His solution is to buy from ZaReason. I saw his laptop in action at the last LUV meeting and it looks really nice. Unfortunately a byproduct of the extremely thin form factor is the fact that it lacks a VGA port, this meant that Chris had to use my Thinkpad T61 (which is rather clunky by comparison) for his presentation.

10 December 2012

Russell Coker: Returning the Aldi Tablet

I have decided to return the 7 Android tablet I bought from Aldi last week [1]. I still think that the tablet isn t all bad, but the main problem I had is the short battery life. The battery can be entirely exhausted in less than an hour s use. Another problem is the fact that most of the Angry Birds games just abort on startup, as my plans for the tablet involved some recreational use this is a significant problem. The final factor that made me decide to return it is the fact that Kogan offered a special deal on a Samsung Galaxy S3 [2] for $449. That ended up as $507 per phone for two phones that have PowerCases (which more than double the battery life as well as protecting the phone from being dropped). So a Galaxy S3 increments my Android device count and with a 4.8 720*1280 display will compete well with a low-end 7 tablet that only has a 800*480 display (the Galaxy S3 has 2.4* as many pixels). Aldi offers 60 days money back, so I ll return the tablet when my Galaxy S3 arrives.

8 December 2012

Eddy Petri&#537;or: Consent is needed, but not the point

Russ makes a good point about a nuance of practical jokes that makes them special, the characteristic of involving a third party which is the target of the joke, and how that relates to consent. He is right, consent is an important fact, I acknowledge that. Russ also says at the end of his post that being an asshole doesn't make you responsible for somebody else's decisions, I ackowledge that, too. I even somewhat agree to the position that the employer might fire you on assholery reasons, but in this particular case, I think that's a hypocrite position. If assholery was an issue, it should have been before, too, and I don't think the employer thought of it before this sad even, as long as it meant more money for them. I think this only shows how cynical they are, the cost analysis for them showed them they now might lose more money by not firing, than by keeping them. It's just cynical.

In spite of that, my initial post, just as I wrote, was not about the DJs, the people involved, the assholery, the taste or lack thereof of the joke. That's why I was intentionally leaving out names or specific context. It was about the bigger picture, irrational reactions and how in the face of a tragedy we surrender our thinking capabilities and react in an irrational manner to the least significant things.

I have had quite recent unfortunate events happening to people close to me, and, invariably, I observe this loss of rational faculties in the face of sad events. Even when this kind of disproportionate response is shown for what it is and how it is harmful in itself, people still refuse to give up on being irrational or they do after a long struggle with the facts.

I'll finish my article with a similar question: can we, as a species, go beyond our condition and make a conscious effort to keep calm and think things through before making decisions in the face of tragedy, or are we incapable to do that?

And no, I am not advocating to drop feelings and affection, I find it sad that I have to say that again, but I am almost sure that in spite of it, I will be wrongly understood by somebody.

Russ Allbery: Consent: Do you speak it?

Yes, humor is a matter of taste. Does this mean that practical jokes, pranks, and social engineering are just a matter of taste? No. What's different, and what's missing, is consent. A key characteristic of practical jokes is that they involve a third party in the joke without getting specific prior consent because to get prior consent would ruin the joke. There are situations in which that's fine: if the people involved know each other well, they can give implicit consent to that sort of joke, for example. If the victim isn't known to the pranksters, they are outside of the set of situations in which this is okay. Consent is something that human society in general seems to be struggling with right now. Consent to sex. Consent to have one's personal information used by a corporation. Consent to have one's pictures posted on-line. And in every case, I think the default position should be the same as the gold standard for sex: enthusiastic consent. If you want to make someone else part of your business, your art work, your joke, your sex act, or anything else that happens outside of your head in a way that affects them, the required step is very simple: you need to get their enthusiastic consent. If that's too much trouble for you, if that makes it less fun for you, if that means you can't make money, if you think your great idea overrides their rights over their own life, body, ideas, information, or participation, then you fail. That simple. If you want to use someone else, you have to ask first. It's not a hard concept. One might even say that it's basic politeness. Now, the role of governments in this is limited and often not practical. For example, I don't think one can write laws about radio performers who like prank calling, at least without causing so many negative side effects that the effort isn't really practical. Doing that doesn't make you a criminal. It just makes you an asshole. Also, doing things to other people without their consent doesn't mean that you're then morally responsible for anything that subsequently happens that may or may not be related. But you're still an asshole. And if your employer decides that you being an asshole as part of your job has suddenly stopped being funny and fires you, hey, works for me.

Eddy Petri&#537;or: Faulty logic - confusing correlation with causation

I want to start this post by saying two things:
  1. It is sad when a good person decides to commit suicide
  2. It is even sadder anyone writing on a subject like this MUST add this kind of disclaimers in the hope the message of the writing isn't distorted because we are unable to listen to each other without suspecting/blaming the other of pure evilness/lack of compassion.
Imagine you are working in an office with other colleagues and one of them makes a print screen of your desktop screen and then displays it full-screen on your computer. You struggle a little and get frustrated when nothing works, the colleagues laugh at your expense. Pretty childish, but no harm, right? You laugh it off.

Imagine a similar scenario, but your shoelaces are tied to the desk without your notice, and the colleagues tied your laces so you can't stand, so the risk of injury is null. You stumble a little when you want to leave the desk. Again, childish, funny, but no harm, right? Again, you laugh it off.

Imagine even a scenario where you are the target of a prank that doesn't injure you physically, but it happens in front of an audience of 100 people. You might feel ashamed, embarrassed, depending on the prank. But that would go away with time, with your laughs or by leaving that audience, right?

So now imagine you are just a person that unknowingly contributed to the success of a harmless prank in which somebody was duped into thinking they were talking to a very public figure about some unimportant and inconsequential stuff related to their job. You'd brush it off, wouldn't you? Most likely, you'd probably laugh at the prank and you wouldn't think the pranksters should be prosecuted or fired. Even if the famous public figure would be upset, taking into account the fact that the things discussed were inconsequential, you wouldn't worry about it. I wouldn't worry even if that famous person was my direct boss and I was the one talking about my boss.

At least, that's how I am thinking, but for some reason, many people don't think this way. And yes, I am talking about the recent sad news about the apparent suicide of a nurse who was involved in the care of the person called Kate Middleton.

I don't want to talk about names, or man-made titles, man-made concepts, or how some people forget that even the most influential people in the world are still a member of this same species that all of us are. I don't even want to talk about this particular sad situation, I just want to talk about the totally irrational reaction many people had to it.


Everybody knows that jokes are a matter of taste. What I might consider funny, some people would take offence at, but, generally speaking, there's an area of jokes that even kids would consider funny. Also, if we are honest to ourselves, a joke is not funnier or less funnier if from the total of people in the room instead of 2 people not liking it, 5 people don't. Not even if some of them are so offended by the joke they leave the room. The joke is equally funny or in poor taste with 2 or with 100 people in the room.

And there's the issue of confusing correlation with causation. If I tell somebody some good news about my aunt and the next day that somebody happens to had bought a car, it doesn't mean my good news made that person decide they want to buy a car. It could be, but it's not necessary, it could be a coincidence.

The same happens in the sad case of the nurse. She decided to take her life, as far as it seems. But that's the extent to which we know anything about the situation. We don't know if the suicide had anything to do with the prank or not. Just the fact that it happened the next day doesn't mean AT ALL the prank call is the cause of the suicide. It doesn't even mean that is "the straw that broke the camel's back", it might be totally unrelated. Or it might not. Taking into account the inconsequential and harmless nature of the call, it seems even less likely. Yet, I might be mistaken.

What I am sure of now is that jumping to the conclusion that the prank call had anything to do with the death, is wrong, as it is jumping to the opposing conclusion. I am also sure that in the absence of other information, the latter, the lack of connection, is more likely, because we know that such drastic sad decisions are not generally caused by trivial stuff.

Time proximity doesn't prove causation. Geographical proximity doesn't either. There needs to be a lot more into it than simple temporal sequence.



Which gets me to the second part of irrational reactions: terminating the show and twitter accounts of the two pranksters. What the hell? Talk about hypersensitivity and paranoid reactions! Are we this blind and willing to blame the least insignificant thing for the gravest consequences that we forget how we all make decisions and how to use our brains, as soon as some tragedy happens?

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="315" src="http://www.youtube.com/embed/tbhsdlJ_MiQ" width="560"></iframe>
Australian Prime-minister Julia Gillard making fun of doomsday believers


Imagine some person who thinks the 21st of December 2012 will bring the end of the earth. Imagine that person taking as a serious declaration the obvious joke from the Australian prime-minister Julia Gillard, and would end their lives to avoid the mayhem, panic and violence they think would follow the 21st of December. I wonder, would as many people blame Julia Gillard for the suicide? I doubt it. And the irony is that for such an hypothetical case, there would be a stronger case for causation than what we now know about the previous case.


I still wonder, when will we learn to think a little more rationally than the people walking the Earth 100.000 years ago?

7 December 2012

Thorsten Glaser: Collision resolution in open addressing hashtables

Before we begin, everyone should read up on hashtables and what open addressing / closed hashing is. The context is lines 111 190 of Python s Objects/dictobject.c as of today (so we get the line numbers straight). (I ve reworded this wlog entry a bit; I originally wrote it too late at night for it to read coherent.) Basically, I ve got an application where I d like to use a hashtable for a number of things not as generic as Python, and with focus on small footprint. I d like to offer associative arrays in a scripting language, where the keys are always arbitrary byte strings excluding NUL. Also, I d like to use the hashtable as backend for indexed arrays, where the keys are uint32_t and the usual use case is sequential. Finally, I m using it for several internal tables, such as a list of keywords, one of builtins, one of special variables, etc. which is a reason for me to not use a self-balancing binary tree as data structure (reading further below might suggest that, but getting a sorted list of hashtable keys is not the focus, though not unimportant).
My questions on this are:

Why is the shift on perturb done after its first use? In my experiments (using 32-bit width everywhere), for the pathological case of an 8-element (i = 3) table with three entries 0, 0x40000000 and 0x800000000, the second round yields 1 for all three, so it cannot have to do with the upper bits. My lookup looks like:

	mask = 2  - 1;
	j = perturb = hash(key);
	goto find_first_slot;
	 find_next_slot:
	j = (j << 2) + j + perturb + 1;
	perturb >>= PERTURB_SHIFT;
	/* FALLTHROUGH */
	 find_first_slot:
	entry = table[j & mask];
	if (!match(entry)) goto find_next_empty_slot;
 

This means that my first check is always the bare hash (so only do it if needed is no reason) and, since I m using gotos, I could just move the perturb >>= PERTURB_SHIFT; line before the line recalculating the next j to use. This seems to make more sense, even in the face of Python. (I actually looked at the Python file s comments again today because I thought to use a different resolution, but they have a good rationale for using the multiplication by 5.) Why can t we just use i as the PERTURB_SHIFT? Sure, this changes a shift-right by a constant, which can possibly be encoded as immediate value in assembly (unless you re on a pre-80186, which can only do SHR AX,1 and SHR AX,CL but not SHR AX,4, but that s outside of mksh s scope) into a right-shift by a variable, but i is already known, and I think the behaviour is better (it wouldn t eat any bits; assume the same 8-entry hashtable and pathologic keys 0, 8 and 16). Again: who do I think I am to go against the wisdom of the Python people, who seem to have shed more thought on this than everyone else I saw, asked, read about (including Spammipedia). That s why I m asking here. On that reference: I don t support spammers or people nagging for donations or premium accounts, like Xing and Groundspeak/Geocaching.COM, at all. In fact, I urge others to do the same, so it really hurts them; it may be their business model, but not if they spam me. Besides, OpenCaching.DE exists. Another thing is: to avoid CVE-2011-4815, I m randomising the hash used, with one seed value per hashtable, changed before a resize operation. I originally thought to seed it with nonzero, but then I have to rehash on hashtable resize, so I ll be XORing the final hash value instead (thanks ciruZ for the idea). I m thinking of omitting that for indexed arrays, as an attacker almost certainly cannot determine the keys there. (To directly use the indexed array keys, which are already uint32_t, as hashes makes using i from even more important.) The hash I m using is a modified Jenkins one-at-a-time called NZAAT: it s my new generic standard n n-cryptographic hash, and the changes are thus: while adding a byte, another increment of the hash is done (so NUL counts), and the finaliser got prefixed with the shift-left-add+shift-right-xor sequence of the adder (but not adding any value or the +1), to get best avalanche for all bytes. I actually compiled several versions of Hash.cs on a Windows VM at work to analyse the original one-at-a-time and all of my modifications; these turned out to be the simplest ones (I originally had added 0x100 instead of 1, but the effect was the same, and +1 is usually cheaper on most CPUs). Also, to avoid people being able to get to the seed, a user will always get only a sorted list of hashtable keys (numeric for indexed arrays, ASCIIbetically otherwise; see also my thoughts on JSON from the previous wlog entry). What algorithm do I use? For strings, comparisons are much more expensive, so I d like to keep them low. Memory use is also a factor; allocating one large(r) block is better than many small ones due to the pool allocator overhead and due to portability to ancient Unic s (which is another reason for me to use a hashtable which is a small struct plus an array of pointers, and then pass the list of keys as array of string pointers, instead of a tree). For both reasons, I m thinking a relatively simple MergeSort: I need to allocate the result array anyway, so I can just get two and free the one that isn t the end result, and it s AFAICT the cheapest on comparisons other than Tree Sort (which nobody really seems to use, and which would effect to using a balanced binary tree again). Since keys are unique, stability and duplicate handling is never an issue. I d like to use only one algorithm and one data structure, not a combination, as compactness is a design goal. Please drop your thoughts on Freenode, e.g. by /msg MemoServ send mirabilos your text here or per eMail to the domains debian, freewrt or mirbsd, which are organisations, with the localpart tg. Or just contact me as usual, if you re already acquainted. Or lookup 0xE99007E0. Thanks in advance! (Especially, Python Developers thoughts are welcome.)

Joachim Breitner: Calculating the internal rate of return with hledger

For more than ten years now, I am a regular and happy user of GnuCash, the free accountancy program. Especially the integrated online banking support for HBCI that allows me not only to fetch my transactions, but also to transfer money using a cryptographic makes it a great tool. Nevertheless, it has shortcomings, especially when it comes to reports. And for some reason I never felt the urge to hack on that C/Guile code (although I recently hacked a bit on the UI).
Joey Hess blog post about Simon Michael s hledger made me look at that command line based accountancy tool. It certainly won t replace Gnucash for me, but luckily ledger, of which hledger is a re-implementation, can read GnuCash s file format and convert it to a ledger file that hledger can read as well. So now I can do analysis on my financial data in Haskell what would you want more? One report that I was missing was the internal rate of investment of some account. This is the annual rate of a hypothetical fixed interest-bearing savings account that would, given the same deposits and payouts, result in the same final value. I now implemented this as the hledger-irr command and as you would expect you can find it on hackage. See the package description for documentation and example output. It is very fresh, hardly tested code, so don t base important decisions on it yet. Also, it needs work to work correctly with multiple currencies or things with varying prices, and I am not yet sure what to do there. Update: Simon asked me to elaborate the difference to hledger-interest by Peter Simons (on which my code was initial based, but eventually I rewrote everything but the option parsing code). The difference is that hledger-interest answers the question I have this investment with this yearly interest rate (which may even change), how large are the interest payments going to be. I imagine a usecase is that you have borrowed some money to a friend at a certain (surely amicable) rate and need to figure out what he owes you, even when you gave him the money mid year and he payed you some back shortly after easter. hledger-irr serves a different purpose: Here you already know your investment payments, gains and losses and you want to have a number describing how good your investment was. One usecase might be that, at the end of a live of paying into a pension fund, you ask yourself: Given all the feeds and commissions, was this a good idea or should I just have invested the money myself? Then the internal rate of investment tells you how lucrative an alternative fixed-rate investment would have had to been to match your pension fund investment. (Or course this is not the only aspect that you should look at when comparing investments, as it totally ignores risk and possible insurance elements of the investment.)

6 December 2012

Leo 'costela' Antunes: Silly Wal(l Cloc)k 2.0

So I finally got the first laser-cut version of the Silly Wal(l Cloc)k done: The back plate was cut from a 1mm plexiglass sheet using this SVG. The black parts were spray painted with matte black and inserted back into the hollowed out sections (I should have made pictures of the pieces before assembling; too lazy to do it now). This proved out to be somewhat problematic, since keeping the small pieces (the chin and the white part of the suitcase handle) in place isn t really easy without gluing everything together, which I wanted to avoid because I m not sure what effect the glue might have on plexiglass in the long term. Another problem was the fact that the plexiglass sheet is white, but slightly translucent. Since the back piece of the clock is black, this caused the white parts to look dirty . As a workaround for both problems I ended up using a thin white cardboard sheet as additional background, where the small hollowed out pieces could be affixed and the translucency of the plexiglass could be hidden. On the other hand, the 1mm thickness for the plexiglass turned out to be perfect for the clock hands (or feet). I think it would still have worked with 2mm, but that might be too heavy for the puny clock torque and they might start snagging on each other. Careful eyes will also note that I made a stupid mistake when writing down the dimensions for the back plate and it ended up being slightly smaller than the available space. Not my brightest moment, but it worked out alright with the additional cardboard background. The next iteration will probably be made with a different two-color plexiglass sheet that can be engraved. Let s see how that works out. Related: Silly Wal(l Cloc)k

Kurt Roeckx: iDEAL and ING

For the e-webshops site I run I started implementing iDEAL because we have a lot customers in the Netherlands. iDEAL is a standard used in the Netherlands to do online payments. We needed to open a bank account for that in the Netherlands, so we did that with ING in October. It first took them a month to reject us, stating we needed to register our company in Belgium. Of course we're already registered, we couldn't be in business in the first place if we weren't. After pointing them to the government website it seems to have sorted itself out a few days later. So once we were approved I could actually start to test the software for iDEAL. It uses XML sig which I think is a horrible standard. Once I started testing it I directly ran into a problem. They give me an error back saying that the signature wasn't valid. So I contacted them about 2 weeks ago and have yet to get answer back that's useful. So far it went like this:
So I'm now back to step 1.

5 December 2012

Raphael Geissert: Introducing: a bashism a week

No matter how many scripting programming languages exist, it appears that shell programming is here to stay around. In many cases it is fast, it "does the job", and best of all: it is available "everywhere". The shell is used by makefiles, on every call to system(), and whatnot.

However, it is a real pain, implementations differ from the standards, some implementations still in use pre-date them, they leave room for undefined behaviour, and bugs in the implementations are nothing but unknown. You can't just specify a given shell interpreter and think you've dealt with the problem. Writing shell scripts that are portable among many platforms is a nightmare, if even possible.

Surprisingly, in spite of all that, a great amount of shell scripts appear to work flawlessly in many systems.

The switch from bash to dash as the default shell interpreter in Debian wasn't done without quite some work (more if you list archived bug reports), and the work ain't over.

For the following months I will be writing about different "bashisms" every Wednesday, hopefully helping people write slightly-more-portable shell scripts. The posts are going to be focused on widely-seen bashisms, probably ignoring those that Debian's policy defines as required to be implemented.

The term "bashism" must be understood as any feature or behaviour not required by SUSv3 (aka POSIX:2001), no matter what its origins are or even if the behaviour is not exhibited by the bash shell.

One of the key points is documenting the script's requirements, starting by specifying the right shell interpreter in the shebang.

Let's see what comes out of this experiment.

As a matter of fact, I have a few months worth of posts written already. All posts are going to be published at scheduled dates, just like this very post.

4 December 2012

Andrea Veri: My favorite WordPress Plugins

It took me a while to build a complete WordPress blog with all the things I needed, from modifying the default Twenty Eleven theme to broadcasting my posts directly on Twitter. WordPress has a nice selection of plugins and given the fact I spent a few days evaluating all the possibilities, I decided to share my own setup to speed up the process in case you are willing to build a WordPress powered blog. The plugins: And a few modifications I made on my Child theme for the Twenty Eleven theme to suit my needs: Remove the navigation buttons on the header. On the Child theme s style.css:
#access, #branding .only-search #s, .entry-header .comments-link  
display: none;
 
Remove the gray line on the top of the header s banner. On the Child theme s style.css:
#branding  
border-top: none;
 
.and a little tip about WordPress itself: have you ever wondered how you could install/update/remove a WordPress theme or plugin without FTP access? Here s how! On the config.php file:
/*** Updates WordPress without FTP credentials. ***/
define('FS_METHOD','direct');
If you know any other handy WordPress Tip or Plugin, please share!

Russell Coker: Recruiting at a LUG Meeting

I m at the main meeting of Linux Users of Victoria (my local LUG). A couple of recruiting agents from Interpro [1] are here and have been working the crowd, one of them is on each side of the room and it seems that their plan is to speak to every person at the meeting and ask about whether they are looking for work. It is apparently difficult for them to find good Linux candidates and they hope to find people here (they are mainly looking for a senior programmer/team leader and an experienced sysadmin). One of my friends is looking for work but he s got two interviews for arranged for this week so they will have to be quick if they want to get him. I guess this means that the economy must be going well, or at least it s not too difficult for Linux people to find work (which is what matters the most to me). Attending the meeting and talking to people is a good business idea for the recruiters and is generally good for members of the group. Before the meeting starts and during the intermission people just hang out and talk, asking them if they are looking for work generally won t harm anyone and can really help some people. I wouldn t want to see multiple agencies doing this at every meeting, but I think that having it happen occasionally is a good thing.

Next.

Previous.