Search Results: "beh"

4 November 2015

Raphaël Hertzog: My Free Software Activities in October 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 13.25 hours on Debian LTS. During this time I worked on the following things:

I prepared and released DLA 330-1 fixing two CVE on unzip.
I prepared a bouncycastle update fixing CVE-2015-7940 (after having requested that CVE assignment since nobody had done it yet) but I have not yet released the corresponding DLA yet since I m waiting for a review by the upstream author. This is hairy cryptographic Java code that was non-trivial to backport and I d rather make sure that I do not mess anything. The patches are available in the bug report #802671 that I opened.
I tested the update to MySQL 5.5 with multiple packages and sent back my findings to the debian-lts mailing list.

I also started a conversation about what paid contributors could work on if they have some spare cycle as the current funding level might allow us to invest some time on work outside of just plain security updates. The Debian Administrator s Handbook I spent quite some time finalizing the Jessie book update, both for the content and for the layout of the printed book.

Debian Handbook: cover of the jessie edition

Misc Debian work GNOME 3.18. I uploaded a new gnome-shell-timer working with GNOME Shell 3.18 and I filed bugs #800660 and #802480 about an annoying gnome-keyring regression I did multiple test rounds with the Debian maintainers (Dmitry Shachnev, kudos to him!) and the upstream developers (see here and here). Apart from those regressions, I like GNOME 3.18! Python-modules team migration to Git. After the Git migration, and since the team policy now imposes usage of git-dpm on all members, I made some tries with it on the python-django package while pushing version 1.8.5 to experimental. And the least I can say is that I m not pleased with the result. I thus filed 3 bugs summarizing the problems I have with git-dpm: #801666 (no way to set the upstream branch names from within the repository), #801667 (no clean way to merge between packaging branches), #801668 (does not create upstream tag immediately on tarball import). That is on top of other randomly stupid bugs that were already reported like #801548 (does not work with perfectly valid pre-existing upstream tags). Django packaging. I filed bugs on all packages build-depending on python-django that fail to build with Django 1.8 and informed them that I would upload Django 1.8 to unstable in early November (it s done already). Then I fixed python-django-jsonfield myself since Distro Tracker relies on this package. Following this small mass-bug filing, I filed a wishlist bug on devscripts to improve the mass-bug helper script (see #801926). And since I used ratt to rebuild the packages, I filed a wishlist issue on this new tool as well. Tryton 3.6 upgrade. I upgraded my own Tryton installation to version 3.6 and filed bug #803066 because the SysV init script was not working properly. That also reminded me that the DD process of Matthias Behrle (the tryton package maintainer) was stalled due to a bug in the NM infrastructure so I pinged the NM team and we sorted out a way for me to advocate him and get his process going Distro Tracker. I continued my work to refactor the way we handle incoming mail processing (branch people/hertzog/mailprocessing). It s now mostly finished and I need to deploy it in a test environment before being able to roll it out on tracker.debian.org. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

3 November 2015

Daniel Pocock: How much of Linux will be illegal in the UK?

This week I've been in the UK again, giving a talk about Lumicall and JSCommunicator in Manchester last night and a talk about Free Real-Time Communications at the mini-DebConf in Cambridge on the weekend of 7-8 November. An interesting backdrop to these activities has been a national debate about Internet privacy. The UK Government and police are demanding laws to mandate back doors in all communications products and services. It leaves me wondering about a range of issues:

Will overzealous UK police, reknowned for singling out and bullying people who don't conform with their idea of normality, start taking a more sinister attitude to people using software like Linux? For example, if airport security asks to inspect a laptop and doesn't see the familiar Windows or Mac OS desktop, will the owner of the laptop be delayed or told to leave it behind? Some people may feel this is extreme, but workers in these roles are known for taking initiative in their own special way, such as the infamous baby pat-down. If the owner of a Linux laptop is a Muslim, like the Texas schoolboy recently arrested because his clock looks suspicious to the untrained eye of a policeman, the chances of a rough encounter with authority probably rise even further.
Will developers still be able to use technologies like PGP and ZRTP in the UK? Will PGP key-signing parties become illegal or have to be held 20 miles offshore on a boat like the legendary pirate radio stations of the sixties?
Will Linux distributions such as Debian and Fedora have to avoid distributing packages such as Enigmail?
Will updates to Android and iOS on smartphones seek to automatically disable or remove apps like Lumicall?
Even if a user chooses a secure app like Lumicall for communication, will the vendor of the operating system be required to provide alternative ways to monitor the user, for example, by intercepting audio before it is encrypted by the app?
Without strong encryption algorithms, digital signatures will no longer be possible either and it will be impossible for software vendors to securely distribute new versions of their software.
Why should the police be the only workers to have their job done for them by Internet snooping? Why shouldn't spouses have a right to all their partner's communications to periodically verify they are not cheating and putting themselves at risk of diseases? Why shouldn't employers be able to check on employee's private communications and home computers to help prevent leaks of customer data? Why shouldn't the NHS be able to go through people's garbage to monitor what they eat given the WHO warning that bacon is more likely to kill you than a terrorist?
While the authorities moan about the internet being a "safe" place for terrorists and paedophiles, what is their real motivation for trying to bring in these new laws, even when their best technical advisors must surely be telling them about the risks and negative consequences for compatibility of UK systems in a global Internet? If the terrorist scare story is not so credible, is it more likely they are seeking to snoop on people who may not be paying taxes or to maintain the upper hand over rival political parties like the Greens and the UKIP in a time of prolonged and increasingly punitive austerity?
Australia already introduced similar laws a few weeks ago, despite widespread criticism from around the world. With cricket and rugby now over, is the UK just looking to go one up on Australia in the game of snooping?

Island mentality in the Internet age Politics aside, what would this mean from a technical perspective? The overwhelming consensus among experts is that secure technology that people use and expect in many other parts of the world, including the US, simply won't be compatible with the products and services that UK residents will be permitted to use. Bigger companies like Google and Apple may be able to offer differentiated versions of their services for the UK but smaller companies or companies who have built their reputation on technical excellence simply won't be able or willing to offer crippled versions of their products with backdoors for the UK. The UK's island geography will become a metaphor for its relationship with the global marketplace. The first thing to take note of is that encryption and authentication are closely related. Public-key cryptography, for example, simply swaps the public key and private key when being used to authenticate instead of encrypt. An effective and wide-reaching legal ban on encryption would also potentially prohibit the algorithms used for authentication. Many methods of distributing software, including packages distributed through Linux distributions or apps distributed through the Google Play store are authenticated with such algorithms. This is often referred to as a digital signature. Digital signatures help ensure that software is not corrupted, tampered with by hackers or infected by viruses when it is transmitted and stored in the public Internet. To correctly implement these mechanisms for installing software safely, every device running an operating system such as Debian, Ubuntu, Fedora or Android needs to include some software modules implementing the algorithms. In Linux, for example, I'm referring to packages like GnuPG, OpenSSL and GnuTLS. Without these components, it would be hard or even impossible for developers in the UK to contribute or publish new versions of their software. Users of the software would not be able to securely receive vital updates to their systems. An opportunity for free software? Some people say that any publicity can be good publicity. Now the Government has put the ball into play, people promoting secure solutions based on free software have an opportunity to participate in the debate too. While laws may or may not change, principles don't. It is a perfect time to remind users that many of the principles of software freedom were written down many years ago, before the opportunity for mass surveillance came into existence. These principles remain relevant to this day. The experts who developed these principles back then are also far more likely to offer insights and trustworthy solutions for the road ahead. If you'd like to discuss these issues or ask questions, please join the Free-RTC mailing list.

31 October 2015

Joey Hess: a tiling region manager for the console

Building on top of concurrent-output, and some related work Joachim Breitner did earlier, I now have a kind of equivilant to a tiling window manager, except it's managing regions of the console for different parts of a single program. Here's a really silly demo, in an animated gif:

Not bad for 23 lines of code, is that? Seems much less tedious to do things this way than using ncurses. Even with its panels, ncurses requires you to think about layout of various things on the screen, and many low-level details. This, by contrast, is compositional, just add another region and a thread to update it, and away it goes. So, here's an apt-like download progress display, in 30 lines of code.

Not only does it have regions which are individual lines of the screen, but those can have sub-regions within them as seen here (and so on). And, log-type messages automatically scroll up above the regions. External programs run by createProcessConcurrent will automatically get their output/errors displayed there, too. What I'm working on now is support for multiline regions, which automatically grow/shrink to fit what's placed in them. The hard part, which I'm putting the finishing touches on, is to accurately work out how large a region is before displaying it, in order to lay it out. Requires parsing ANSI codes amoung other things. STM rules There's so much concurrency, with complicated interrelated data being updated by different threads, that I couldn't have possibly built this without Software Transactional Memory. Rather than a nightmare of locks behind locks behind locks, the result is so well behaved that I'm confident that anyone who needs more control over the region layout, or wants to do funky things can dive into to the STM interface and update the data structures, and nothing will ever deadlock or be inconsistent, and as soon as an update completes, it'll display on-screen. An example of how powerful and beuatiful STM is, here's how the main display thread determines when it needs to refresh the display:

data DisplayChange
        = BufferChange [(StdHandle, OutputBuffer)]
          RegionChange RegionSnapshot
          TerminalResize (Maybe Width)
          EndSignal ()
    ...
                change <- atomically $
                        (RegionChange <$> regionWaiter origsnapshot)
                                 orElse 
                        (RegionChange <$> regionListWaiter origsnapshot)
                                 orElse 
                        (BufferChange <$> outputBufferWaiterSTM waitCompleteLines)
                                 orElse 
                        (TerminalResize <$> waitwidthchange)
                                 orElse 
                        (EndSignal <$> waitTSem endsignal)
                case change of
                        RegionChange snapshot -> do
                ...
                        BufferChange buffers -> do
                ...
                        TerminalResize width -> do
                ...

So, it composes all these STM actions that can wait on various kinds of changes, to get one big action, that waits for all of the above, and builds up a nice sum type to represent what's changed. Another example is that the whole support for sub-regions only involved adding 30 lines of code, all of it using STM, and it worked 100% the first time.

Available in concurrent-output 1.1.0.

29 October 2015

Daniel Pocock: Lumicall rapid provisioning, Opus support and other enhancements

Thanks to feedback from many people at DebConf15, Lumicall has had a range of improvements over the last couple of months and it is now easier to use and more reliable than ever. Rapid provisioning for Debian.org, FedRTC.org and other SIP accounts 1. Setup your SIP password in the Debian LDAP or the FedRTC account setup (using Fedora OpenID) or any other SIP service that supports rapid provisioning (see the FAQ). 2. Install the Lumicall app from F-Droid, Google Play or by direct download. 3. When the app starts the first time, you can verify your phone number (so other users can call you using the number they know) or just click the "Manual account setup" link underneath the verification button. If you do verify your number, the "Add account" link is also available in the menu from the main screen.

Test calls If a friend installs Lumicall too and you dial their number, the Lumicall popup should appear and give you the option to make the call over SIP. You can also make test calls using this link for a recorded announcement. Many other improvements

Support for the OPUS codec, which brings us one step closer to direct WebRTC interoperability
Fixing a bug in ice4j that caused NAT traversal to fail intermittently
Tweaks to make it more tolerant of IPv6 networks
The rapid account setup form (described above) for existing SIP accounts
Conversion to the Gradle build system (for Android Studio)
Breaking out embedded library code from the Lumicall repository into standalone projects so they can be maintained independently
Use of the compatibility library to display the ActionBar, consistent with the current design guidelines, while still maintaining compatibility with older devices
Use of Spongy Castle instead of Bouncy Castle crypto library for more consistent behavior across different Android devices

Share your experiences with Lumicall All feedback is welcome. If you can come in person to the workshop in Manchester (2 November) or the mini-DebConf in Cambridge (I'll be there Saturday, 7 November) then we can look at any problems in person and access logs with the Android tools if necessary. Discussion is also welcome on the Lumicall users mailing list or the Free RTC mailing list.

Joey Hess: concurrent output library

concurrent-output is a Haskell library I've developed this week, to make it easier to write console programs that do a lot of different things concurrently, and want to serialize concurrent outputs sanely. It's increasingly easy to write concurrent programs, but all their status reporting has to feed back through the good old console, which is still obstinately serial. Haskell illustrates problem this well with this "Linus's first kernel" equivilant interleaving the output of 2 threads:

> import System.IO
> import Control.Concurrent.Async
> putStrLn (repeat 'A')  concurrently  putStrLn (repeat 'B')
BABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABA
BABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABA
...

That's fun, but also horrible if you wanted to display some messages to the user:

> putStrLn "washed the car"  concurrently  putStrLn "walked the dog"
walwkaesdh etdh et hdeo gc
ar

To add to the problem, we often want to run separate programs concurrently, which have output of their own to display. And, just to keep things interesting, sometimes a unix program will behave differently when stdout is not connected to a terminal (eg, ls cat). To tame simple concurrent programs like these so they generate readable output involves a lot of plumbing. Something like, run the actions concurrently, taking care to capture the output of any commands, and then feed the output that the user should see though some sort of serializing channel to the display. Dealing with that when you just wanted a simple concurrent program risks ending up with a not-so-simple program. So, I wanted an library with basically 2 functions:

outputConcurrent :: String -> IO ()
    
createProcessConcurrent :: CreateProcess -> IO whatever

The idea is, you make your program use outputConcurrent to display all its output, and each String you pass to that will be displayed serially, without getting mixed up with any other concurrent output. And, you make your program use createProcessConcurrent everywhere it starts a process that might output to stdout or stderr, and it'll likewise make sure its output is displayed serially. Oh, and createProcessConcurrent should avoid redirecting stdout and stderr away from the console, when no other concurrent output is happening. So, if programs are mostly run sequentially, they behave as they normally would at the console; any behavior changes should only occur when there is concurrency. (It might also be nice for it to allocate ttys and run programs there to avoid any behavior changes at all, although I have not tried to do that.) And that should be pretty much the whole API, although it's ok if it needs some function called by main to set it up:

import Control.Concurrent.Async
import System.Console.Concurrent
import System.Process
main = withConcurrentOutput $
    outputConcurrent "washed the car\n"
         concurrently 
    createProcessConcurrent (proc "ls" [])
         concurrently 
    outputConcurrent "walked the dog\n"

$ ./demo
washed the car
walked the dog
Maildir/  bin/  doc/  html/  lib/  mail/  mnt/  src/  tmp/

I think that's a pretty good API to deal with this concurrent output problem. Anyone know of any other attempts at this I could learn from? I implemented this over the past 3 days and 320 lines of code. It got rather hairy:

It has to do buffering of the output.
There can be any quantity of output, but program memory use should be reasonably small. Solved by buffering up to 1 mb of output in RAM, and writing excess buffer to temp files.
Falling off the end of the program is complicated; there can be buffered output to flush and it may have to wait for some processes to finish running etc.
The locking was tough to get right! I could not have managed to write it correctly without STM.

It seems to work pretty great though. I got Propellor using it, and Propellor can now run actions concurrently!

26 October 2015

Simon Josefsson: Combining Dnsmasq and Unbound

For my home office network I have been using Dnsmasq for some time. Dnsmasq provides me with DNS, DHCP, DHCPv6, and IPv6 Router Advertisement. I run dnsmasq on a Debian Jessie server, but it works similar with OpenWRT if you want to use a smaller device. My entire /etc/dnsmasq.d/local configuration used to look like this:

dhcp-authoritative
interface=eth1
read-ethers
dhcp-range=192.168.1.100,192.168.1.150,12h
dhcp-range=2001:9b0:104:42::100,2001:9b0:104:42::1500
dhcp-option=option6:dns-server,[::]
enable-ra

Here dhcp-authoritative enable DHCP. interface=eth1 says to listen on eth1 only, which is my internal (IPv4 NAT) network. I try to keep track of the MAC address of all my devices in a /etc/ethers file, so I use read-ethers to have dnsmasq give stable IP addresses for them. The dhcp-range is used to enable DHCP and DHCPv6 on my internal network. The dhcp-option=option6:dns-server,[::] statement is needed to inform the DHCP clients of the DNS resolver s IPv6 address, otherwise they would only get the IPv4 DNS server address. The enable-ra parameter enables IPv6 router advertisement on the internal network, thereby removing the need to run radvd too useful since I prefer to use copyleft software. Recently I had a desire to use DNSSEC, and enabled it in Dnsmasq using the following statements:

dnssec
trust-anchor=.,19036,8,2,49AAC11D7B6F6446702E54A1607371607A1A41855200FD2CE1CDDE32F24E8FB5
dnssec-check-unsigned

The dnssec keyword enable DNSSEC validation in dnsmasq, using the indicated trust-anchor (get the root-anchors from IANA). The dnssec-check-unsigned deserves some more discussion. The dnsmasq manpage describes it as follows:

As a default, dnsmasq does not check that unsigned DNS replies are legitimate: they are assumed to be valid and passed on (without the authentic data bit set, of course). This does not protect against an attacker forging unsigned replies for signed DNS zones, but it is fast. If this flag is set, dnsmasq will check the zones of unsigned replies, to ensure that unsigned replies are allowed in those zones. The cost of this is more upstream queries and slower performance.

For example, this means that dnsmasq s DNSSEC functionality is not secure against active man-in-the-middle attacks between dnsmasq and the DNS server it is using. Even if example.org used DNSSEC properly, an attacker could fake that it was unsigned to dnsmasq, and I would get potentially incorrect values in return. We all know that the Internet is not a secure place, and your threat model should include active attackers. I believe this mode should be the default in dnsmasq, and users should have to configure dnsmasq to not be in that mode if they really want to (with the obvious security warning). Running with this enabled for a couple of days resulted in frustration about not being able to reach a couple of domains. The behaviour was that my clients would hang indefinitely or get a SERVFAIL, both resulting in lack of service. You can enable query logging in dnsmasq with log-queries and enabling this I noticed three distinct form of error patterns:

jow13gw dnsmasq 460 - -  forwarded www.fritidsresor.se to 213.80.101.3
jow13gw dnsmasq 460 - -  validation result is BOGUS

jow13gw dnsmasq 547 - -  reply cloudflare-dnssec.net is BOGUS DNSKEY
jow13gw dnsmasq 547 - -  validation result is BOGUS

jow13gw dnsmasq 547 - -  reply linux.conf.au is BOGUS DS
jow13gw dnsmasq 547 - -  validation result is BOGUS

The first only happened intermittently, the second did not cause any noticeable problem, and the final one was reproducible. To be fair, I only found the last example after starting to search for problem reports (see post confirming bug). At this point, I had a confirmed bug in dnsmasq that affect my use-case. I want to use official packages from Debian on this machine, so installing newer versions manually is not an option. So I started to look into alternatives for DNS resolving, and quickly found Unbound. Installing it was easy:

apt-get install unbound
unbound-control-setup

I created a local configuration file in /etc/unbound/unbound.conf.d/local.conf as follows:

server:
	interface: 127.0.0.1
	interface: ::1
	interface: 192.168.1.2
	interface: 2001:9b0:104:42::2
	access-control: 127.0.0.1 allow
	access-control: ::1 allow
	access-control: 192.168.1.2/24 allow
	access-control: 2001:9b0:104:42::2/64 allow
	outgoing-interface: 155.4.17.2
	outgoing-interface: 2001:9b0:1:1a04::2
#	log-queries: yes
#	verbosity: 2

The interface keyword determine which IP addresses to listen on, here I used the loopback interface and the local address of the physical network interface for my internal network. The access-control allows recursive DNS resolving from those networks. And outgoing-interface specify my external Internet-connected interface. log-queries and/or verbosity are useful for debugging. To make things work, dnsmasq has to stop providing DNS services. This can be achieved with the port=0 keyword, however that will also disable informing DHCP clients about the DNS server to use. So this has to be added in manually. I ended up adding the two following lines to /etc/dnsmasq.d/local:

port=0
dhcp-option=option:dns-server,192.168.1.2

Restarting unbound and dnsmasq now leads to working (and secure) internal DNSSEC-aware name resolution over both IPv4 and IPv6. I can verify that resolution works, and that Unbound verify signatures and reject bad domains properly with dig as below, or use online DNSSEC resolver test page although I m not sure how confident you can be in the result from that page.

$ host linux.conf.au
linux.conf.au has address 192.55.98.190
linux.conf.au mail is handled by 1 linux.org.au.
$ host sigfail.verteiltesysteme.net
;; connection timed out; no servers could be reached
$

I use Munin to monitor my services, and I was happy to find a nice Unbound Munin plugin. I installed the file in /usr/share/munin/plugins/ and created a Munin plugin configuration file /etc/munin/plugin-conf.d/unbound as follows:

[unbound*]
user root
env.statefile /var/lib/munin-node/plugin-state/root/unbound.state
env.unbound_conf /etc/unbound/unbound.conf
env.unbound_control /usr/sbin/unbound-control
env.spoof_warn 1000
env.spoof_crit 100000

I run munin-node-configure --shell sh to enable it. To work unbound has to be configured as well, so I create a /etc/unbound/unbound.conf.d/munin.conf as follows.

server:
	extended-statistics: yes
	statistics-cumulative: no
	statistics-interval: 0
remote-control:
	control-enable: yes

The graphs may be viewed at my munin instance.

Craig Small: ps standards and locales

I looked at two interesting issues today around the ps program in the procps project. One had a solution and the other I m puzzled about. ps User-defined Format Issue #9 was quite the puzzle. The output of ps changed depending if a different option had a hyphen before it or not. First, the expected output

$ ps p $$ -o pid=pid,comm=comm
 pid comm
31612 bash

Next, the unusual output.

$ ps -p $$ -o pid=pid,comm=comm
pid,comm=comm
 31612

The difference being the second we have -p not p. The second unexpected thing about this is, it was designed that way. Unix98 standard apparently permits this sort of craziness. To me it is a useless feature that will more likely than not confuse people. Within ps, depending on what sort of flags you start with, you use a sysv parser or a bsd parser. One of them triggered the Unix98 compatibility option while the other did not, hence the change in behavior. The next version of procps will ship with a ps that has the user-defined output format of the first example. I had a look at the latest standard, IEEE 1003.1-2013, doesn t seem to have anything like that in it. Short Month Length This one has got me stuck. A user has reported in issue #5 that when they use their locale columns such as start time get mis-aligned because their short month is longer than 3 characters. They also mention some other columns for memory etc are not long enough but that s a generic problem that is impossible to fix sensibly. OK, for the month problem the fix would be to know what the month length is and set the column width for those columns to that plus a bit more for the other fixed components, simple really. Except; how do you know for a specific locale what their short month length is? I always assumed it was three! I haven t found anything that has this information. Note, I m not looking for strlen() but some function that has the maximum length for short month names (e.g. Jan, Feb, Mar etc). This also got me thinking how safe some of those date to string functions are if you have a static buffer as the destination. It s not safe to assume they will be DD MMM YYYY because there might be more Ms. So if you know how to work out the short month name length, let me know!

23 October 2015

Joey Hess: propelling disk images

Following up on Then and Now ... In quiet moments at ICFP last August, I finished teaching Propellor to generate disk images. With an emphasis on doing a whole lot with very little new code and extreme amount of code reuse. For example, let's make a disk image with nethack on it. First, we need to define a chroot. Disk image creation reuses propellor's chroot support, described back in propelling containers. Any propellor properties can be assigned to the chroot, so it's easy to describe the system we want.

 nethackChroot :: FilePath -> Chroot
    nethackChroot d = Chroot.debootstrapped (System (Debian Stable) "amd64") mempty d
        & Apt.installed ["linux-image-amd64"]
        & Apt.installed ["nethack-console"]
        & accountFor gamer
        & gamer  hasInsecurePassword  "hello"
        & gamer  hasLoginShell  "/usr/games/nethack"
      where gamer = User "gamer"

Now to make an image from that chroot, we just have to tell propellor where to put the image file, some partitioning information, and to make it boot using grub.

 nethackImage :: RevertableProperty
    nethackImage = imageBuilt "/srv/images/nethack.img" nethackChroot
        MSDOS (grubBooted PC)
        [ partition EXT2  mountedAt  "/boot"
             setFlag  BootFlag
        , partition EXT4  mountedAt  "/"
             addFreeSpace  MegaBytes 100
        , swapPartition (MegaBytes 256)
        ]

The disk image partitions default to being sized to fit exactly the files from the chroot that go into each partition, so, the disk image is as small as possible by default. There's a little DSL to configure the partitions. To give control over the partition size, it has some functions, like addFreeSpace and setSize. Other functions like setFlag and extended can further adjust the partitions. I think that worked out rather well; the partition specification is compact and avoids unecessary hardcoded sizes, while providing plenty of control. By the end of ICFP, I had Propellor building complete disk images, but no boot loader installed on them.

Fast forward to today. After stuggling with some strange grub behavior, I found a working method to install grub onto a disk image. The whole disk image feature weighs in at: 203 lines to interface with parted
88 lines to format and mount partitions
90 lines for the partition table specification DSL and partition sizing
196 lines to generate disk images
75 lines to install grub on a disk image
652 lines of code total Which is about half the size of vmdebootstrap 1/4th the size of partman-base (probably 1/100th the size of total partman), and 1/13th the size of live-build. All of which do similar things, in ways that seem to me to be much less flexible than Propellor.

One thing I'm considering doing is extending this so Propellor can use qemu-user-static to create disk images for eg, arm. Add some u-boot setup, and this could create bootable images for arm boards. A library of configs for various arm boards could then be included in Propellor. This would be a lot easier than running the Debian Installer on an arm board. Oh! I only just now realized that if you have a propellor host configured, like this example for my dialup gateway, leech --

 leech = host "leech.kitenet.net"
        & os (System (Debian (Stable "jessie")) "armel")
        & Apt.installed ["linux-image-kirkwood", "ppp", "screen", "iftop"]
        & privContent "/etc/ppp/peers/provider"
        & privContent "/etc/ppp/pap-secrets"
        & Ppp.onBoot
        & hasPassword (User "root")
        & Ssh.installed

-- The host's properties can be extracted from it, using eg hostProperties leech and reused to create a disk image with the same properties as the host! So, when my dialup gateway gets struck by lightning again, I could use this to build a disk image for its replacement:

 import qualified Propellor.Property.Hardware.SheevaPlug as SheevaPlug
    laptop = host "darkstar.kitenet.net"
        & SheevaPlug.diskImage "/srv/images/leech.img" (MegaBytes 2000)
            (& propertyList "has all of leech's properties"
                (hostProperties leech))

This also means you can start with a manually built system, write down the properties it has, and iteratively run Propellor against it until you think you have a full specification of it, and then use that to generate a new, clean disk image. Nice way to transition from sysadmin days of yore to a clean declaratively specified system.

20 October 2015

Ritesh Raj Sarraf: Controlling ill behaving applications with Linux Cgroups

For some time, I have been wanting to read more on Linux Cgroups to explore possibilities of using it to control Ill behaving applications. At this time, while I'm stuck in travel, it has given me some time to look into it. In our Free Software world, most of the things are do-o-cracy, i.e. when your use case is not the common one, it is typically you who has to explore possible solutions. It could be Bugs, Feature Requests or as is in my case, performance issues. But that is not to assume that we do not have better quality software in Free Software world. Infact, in my opinion, some of the tools available are far much more better than the competition in terms of features, and to add a sweetener (or nutritional facts) to it is the fact that Free Software liberates the user. One of my favorite tool, for photo management, is Digikam. Digikam is a big project, very featureful, and has some functionalities that may not be available in the competition. But as is with most Free Software projects, Digikam is a tool which underneath consumes many more subprojects from the Free Software ecosystem. For anyone who has used Digikam, may know some of the bugs that surface on it. Not necessarily a bug in Digikam, but maybe in one of the underneath libraries/tools that it consumes (Exiv, libkface, marble, OpenCV, libPGF etc). But the bottom line is that the overall Digikam experience (and if I may say: the overall GNU/Linux experience) takes a hit. Digikam has pretty powerful features for annotation, tagging, facial recognition. These features, together with Digikam, make it a compelling product. But the problem is that many of these projects are independent. Thus tight integration is a challenge. And at times, bugs can be hard to find, root cause and fix. Let's take a real example here. If you were to use Digikam today (version 4.13.0) with annotation, tagging and facial recognition as some of the core features for your use case, you may run into frustrating overall experience. Not just that, the bugs would also effect your overall GNU/Linux experience. The facial recognition feature, if triggered, will eat up all your memory. Thus leading you to uncover Linux's long old memory bug. The tagging feature, if triggered, again will lead to frequent I/O. Thus again leading to a stalled Linux system because of blocked CPU cycled, for nothing. So one of the items on my TODO list was to explore Linux Cgroups, and see if it was cleanly possible to tame a process to a confinement, so that even if it was ill behaving (for whatever reasons), your machine does not take the beating. And now that the cgroups consumer dust has kinda settled down, systemd was my first obvious choice to look at. systemd provides a helper utility, systemd-run, for similar tasks. With systemd-run, you could apply all the resource controller logic to the given process, typically cpu, memory and blkio. And restrict it to a certain set. You can also define what user to run the service as.

rrs@learner:/var/tmp/Debian-Build/Result$ systemd-run -p BlockIOWeight=10 find /
Running as unit run-23805.service.
2015-10-20 / 21:37:44

rrs@learner:/var/tmp/Debian-Build/Result$ systemctl status -l run-23805.service
  run-23805.service - /usr/bin/find /
   Loaded: loaded
  Drop-In: /run/systemd/system/run-23805.service.d
            50-BlockIOWeight.conf, 50-Description.conf, 50-ExecStart.conf
   Active: active (running) since Tue 2015-10-20 21:37:44 CEST; 6s ago
 Main PID: 23814 (find)
   Memory: 12.2M
      CPU: 502ms
   CGroup: /system.slice/run-23805.service
            23814 /usr/bin/find /
Oct 20 21:37:45 learner find[23814]: /proc/3/net/raw6
Oct 20 21:37:45 learner find[23814]: /proc/3/net/snmp
Oct 20 21:37:45 learner find[23814]: /proc/3/net/stat
Oct 20 21:37:45 learner find[23814]: /proc/3/net/stat/rt_cache
Oct 20 21:37:45 learner find[23814]: /proc/3/net/stat/arp_cache
Oct 20 21:37:45 learner find[23814]: /proc/3/net/stat/ndisc_cache
Oct 20 21:37:45 learner find[23814]: /proc/3/net/stat/ip_conntrack
Oct 20 21:37:45 learner find[23814]: /proc/3/net/stat/nf_conntrack
Oct 20 21:37:45 learner find[23814]: /proc/3/net/tcp6
Oct 20 21:37:45 learner find[23814]: /proc/3/net/udp6
2015-10-20 / 21:37:51

But, out of the box, graphical applications do not work. I haven't looked, but it should be doable by giving it the correct environment details.

Underneath, systemd is using the same Linux Control Groups to limit resources for individual applications. So, in cases where you have a requirement and do not have systemd, or you directly want to make use of cgroups, it could be easily done with basic cgroups tools like cgroup-tools.

With cgroup-tools, I now have a simple cgroups hierarchy set for my current use case, i.e. Digikam

rrs@learner:/var/tmp/Debian-Build/Result$ ls /sys/fs/cgroup/memory/rrs_customCG/
cgroup.clone_children           memory.kmem.tcp.limit_in_bytes      memory.numa_stat
cgroup.event_control            memory.kmem.tcp.max_usage_in_bytes  memory.oom_control
cgroup.procs                    memory.kmem.tcp.usage_in_bytes      memory.pressure_level
digikam/                        memory.kmem.usage_in_bytes          memory.soft_limit_in_bytes
memory.failcnt                  memory.limit_in_bytes               memory.stat
memory.force_empty              memory.max_usage_in_bytes           memory.swappiness
memory.kmem.failcnt             memory.memsw.failcnt                memory.usage_in_bytes
memory.kmem.limit_in_bytes      memory.memsw.limit_in_bytes         memory.use_hierarchy
memory.kmem.max_usage_in_bytes  memory.memsw.max_usage_in_bytes     notify_on_release
memory.kmem.slabinfo            memory.memsw.usage_in_bytes         tasks
memory.kmem.tcp.failcnt         memory.move_charge_at_immigrate
2015-10-20 / 21:45:38         
rrs@learner:/var/tmp/Debian-Build/Result$ ls /sys/fs/cgroup/memory/rrs_customCG/digikam/
cgroup.clone_children           memory.kmem.tcp.max_usage_in_bytes  memory.oom_control
cgroup.event_control            memory.kmem.tcp.usage_in_bytes      memory.pressure_level
cgroup.procs                    memory.kmem.usage_in_bytes          memory.soft_limit_in_bytes
memory.failcnt                  memory.limit_in_bytes               memory.stat
memory.force_empty              memory.max_usage_in_bytes           memory.swappiness
memory.kmem.failcnt             memory.memsw.failcnt                memory.usage_in_bytes
memory.kmem.limit_in_bytes      memory.memsw.limit_in_bytes         memory.use_hierarchy
memory.kmem.max_usage_in_bytes  memory.memsw.max_usage_in_bytes     notify_on_release
memory.kmem.slabinfo            memory.memsw.usage_in_bytes         tasks
memory.kmem.tcp.failcnt         memory.move_charge_at_immigrate
memory.kmem.tcp.limit_in_bytes  memory.numa_stat
2015-10-20 / 21:45:53

rrs@learner:/var/tmp/Debian-Build/Result$ cat /sys/fs/cgroup/cpu/rrs_customCG/cpu.shares 
1024
2015-10-20 / 21:48:44         
rrs@learner:/var/tmp/Debian-Build/Result$ cat /sys/fs/cgroup/cpu/rrs_customCG/digikam/cpu.shares 
512
2015-10-20 / 21:49:05

rrs@learner:/var/tmp/Debian-Build/Result$ cat /sys/fs/cgroup/memory/rrs_customCG/memory.limit_in_bytes 
9223372036854771712
2015-10-20 / 22:20:14         
rrs@learner:/var/tmp/Debian-Build/Result$ cat /sys/fs/cgroup/memory/rrs_customCG/digikam/memory.limit_in_bytes 
2764369920
2015-10-20 / 22:20:27

rrs@learner:/var/tmp/Debian-Build/Result$ cat /sys/fs/cgroup/blkio/rrs_customCG/blkio.weight
500
2015-10-20 / 21:51:43         
rrs@learner:/var/tmp/Debian-Build/Result$ cat /sys/fs/cgroup/blkio/rrs_customCG/digikam/blkio.weight
10
2015-10-20 / 21:51:50

The base group, $USER_customCG needs super admin privileges. Which once set appropriately, allows the user to further self-define sub-groups. And users can then also define separate limits per sub-group.

With the resource limitations set in place, my overall experience on very recent hardware (Intel Haswell Core i7, 8 GiB RAM, 500 GB SSHD, 128 GB SSD) has improved considerably. It still is not perfect, but it definitely is a huge improvement over what I had to go through ealire: A stalled machine for hours.

top - 21:54:38 up 1 day,  6:46,  1 user,  load average: 7.22, 7.51, 7.37
Tasks: 299 total,   1 running, 298 sleeping,   0 stopped,   0 zombie
%Cpu0  :  7.1 us,  3.0 sy,  1.0 ni, 11.1 id, 77.8 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  6.0 us,  4.0 sy,  2.0 ni, 49.0 id, 39.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  5.0 us,  2.0 sy,  0.0 ni, 24.8 id, 68.3 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  5.9 us,  5.0 sy,  0.0 ni, 21.8 id, 67.3 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 7908.926 total,   96.449 free, 4634.922 used, 3177.555 buff/cache
MiB Swap: 8579.996 total, 3454.746 free, 5125.250 used. 2753.324 avail Mem 
PID to signal/kill [default pid = 8879] 
  PID  PPID nTH USER        PR  NI S %CPU %MEM     TIME+ COMMAND                           UID 
 8879  8868  18 rrs         20   0 S  8.2 31.2  37:44.64 digikam                          1000 
10255  9960   4 rrs         39  19 S  1.0  0.8  19:47.73 tracker-miner-f                  1000 
10157  9960   7 rrs         20   0 S  0.5  3.0  32:29.76 gnome-shell                      1000 
    7     2   1 root        20   0 S  0.2        0:53.48 rcu_sched                           0 
  401     1   1 root        20   0 S  0.2  1.3   0:54.93 systemd-journal                     0 
10269  9937   4 rrs         20   0 S  0.2  0.4   2:34.50 gnome-terminal-                  1000 
15316     1  14 rrs         20   0 S  0.2  3.7  30:05.96 evolution                        1000 
23777     2   1 root        20   0 S  0.2        0:05.73 kworker/u16:0                       0 
23814     1   1 root        20   0 D  0.2  0.0   0:02.00 find                                0 
24049     2   1 root        20   0 S  0.2        0:01.29 kworker/u16:3                       0 
24052     2   1 root        20   0 S  0.2        0:02.94 kworker/u16:4                       0 
    1     0   1 root        20   0 S       0.1   0:18.24 systemd                             0

The reporting tools may not be correct here. Because from what is being reported above, I should be having a machine stalled, and heavily paging, while the kernel scanning its list of processes to find the best process to kill. From this approach of jailing processes, the major side effect I can see is that the process (Digikam) is now starved of resources and will take much much much more time than what it would have been usually. But in the usual cases, it takes up all, and ends up starving (and getting killed) for consuming all available resources. So I guess it is better to be on a balanced resource diet. :-)

Categories:
Debian-Blog
Tools

Keywords:
systemd
cgroups
desktop
digikam

RHUT

16 October 2015

Vincent Sanders: Raspberries are not the only fruit

I have worked with ARM based systems for longer than I care to admit to myself. From the Acorn Archimedes 305 in 1987 through to modern 64bit systems I have seen many many changes in the ARM community. One big change has been the rise of the inexpensive single board computer (SBC).

Arguably the Raspberry Pi (RPi) was responsible for starting this trend. Before RPi there were small development boards available, I was even involved in producing some of them, none of these really became a big thing and were principally vehicles for silicon vendors to showcase their SoC in an accessible way. When I say accessible I mean for silicon vendors who were previously used to charging many thousands of pounds for their development boards now only charging hundreds.

In my opinion the RPi was a complete disruption to the SBC market. In early 2012 a complete ARM computer system could now be purchased for $35 ( 25) which was substantially cheaper than the best contemporary competitor the Beaglebone $89 ( 60)

To be clear, the reason the RPi succeeded (millions sold, household name) was not on price alone but also the large amount of good supporting software and how easy it was made for teachers and makers to use.

There were several areas that the RPi managed to change perceived issues into opportunities for using what was already available or third parties to provide. There were also several issues raised about the original RPi:

no case
no integrated storage
not having an x86 processor
being a slow processor
limited peripheral support
no real time clock
not supplying keyboards, mice
not providing displays.

Some of these have been addressed since release as the foundation now sells cases, hardware revisions with much more powerful processors, pre-configured storage, cameras and displays. The important thing to note here though is the RPi has made the bare SBC a much more widely accepted product where all the non critical parts are considered "extras" and a lot is forgiven because of the price.

With that acceptance there have been many, many new SBC coming to market with better peripherals and increasingly competitive pricing. These are technically not clones as none of them use the Broadcom processor of the RPi but they often share many features and possibly even a compatible expansion header.

Banana pi in my case with a 2.5inch drive bay

The Banana Pi was one of these copycats which I acquired for a similar price as an RPi in 2014. The main processor of this system was a 1Ghz dual core Allwinner A20 processor (a considerable advance on the 700MHz single core of the original RPi) coupled to a gigabyte of memory. Additionally the board benefited from having SATA and gigabit Ethernet MAC which made for a much more versatile system. Various third parties filled in the missing peripherals including my own attempt at a case.

I acquired a cubietruck for the NetSurf project to use as a build node in their CI system this is again based on the A20 but with more memory and somewhat better peripheral support but at a substantial cost over the Banana Pi.

The most recent addition to this form factor is the introduction by Xunlong Software of the Orange Pi PC. This little board is the same footprint as the RPi2 B+ design but with differing connector placement. The processor is a quad core 1.6GHz Allwinner H3 with a gigabyte of memory and has has Ethernet and USB but no SATA.

The big news about this board though is the price, at $15 ( 10) it resets the price expectations just as the RPi did before it. I was initially sceptical of the quality of the product (or if it would arrive at all) but I have acquired five of these boards and every one of them came well packaged, boxed and in a static bag, just like the RPi does, and they all worked.

I created a case design based on my RPi slimline case so they would be protected when piled up with all the other boards. The use of a DC barrel jack instead of micro USB for power is better in that the connector is more robust and intended for higher current draws but does mean additional leads are needed. There are, however, two flies in the ointment, neither are showstoppers but make the board a little more difficult to use.

One is a simple necessity of a substantial heatsink on the H3 processor. Initially I used a small 20 x 20mm copper heatsink (around 800mm square surface area) but this was insufficient under full load. I did not want to have to use a fan so I milled some slots into copper round bar, then cut off sections and faced them on a lathe. The completed design had more than 2200 square mm surface area and cost around $2.5 ( 1.5) in material (and a couple of hours in the workshop but that was fun and I made something)

The second issue and arguably much more serious is that of software. Let me be honest, it is dreadful, I mean very bad indeed. The images provided from the Orange Pi website are some of the worst examples of "do something quick" I have experienced.

Fortunately a user on the forums named loboris decided to create scripts that generate a Debian (and Ubuntu and Fedora) distribution images that can be installed from SD card. He relies on a somewhat patched 3.4 kernel full of Allwinner vendor changes and the inevitable binary blobs for the Marli GPU but the result does work.

I have had a few units acting as distcc compiler slaves for two weeks now at 100% CPU loading and they are still running. The processor does not get overly warm with the heatsink installed and Debian behaves just fine. The main caveat being that it is definitely not going to work if you try and update the kernel through packaging.

The state of software support and Xunlong Software relying on a forum user to complete their product does tarnish an otherwise impressive and possibly market changing SBC.

Perhaps I expect too much for fifteen bucks? I guess when the cost of the case, heatsink, cables and memory storage card is similar to the rest of the computer there is simply no margin left for anything else.

In conclusion hopefully this brief overview has provided some insight into what is available in this market and that the Raspberry Pi is indeed not the only option.

I finish with an image is of my ARM build farm consisting of every SBC I mention here (including a couple of RPi)

14 October 2015

Norbert Preining: On Lars Wirzenius, Mart n Ferrari, and Debian

Now that the tsunami that followed my recent blog post has passed, I can read through all the nice comments and blogs of other people. I didn t expect that just writing up some links to posts on LKML, plus adding my personal opinion on what has happened and about my interpretation concerning Debian could gather so much attention. Anyway, let us look at the blog posts written in the aftermath. debian-newspeak-coc

In particular do I want to answer to two blogs: Lars Wirzenius blog and Mart n Ferrari s blog, both of which are excellent examples of insults. I also want to mention that although the previous two are bad examples, there are fortunately also those who are able to resist primitive insults, in particular Gunnar Wolf (direct link to the blog does not work), and Miriam Ruiz blog, which contradict my blog and propose a different view, without pulling the knife. Lars Wirzenius Let us start with Lars (ah sorry, I have to call him Wirzenius here, strange custom has suddenly popped up in Debian!):

I think Preining mis-characterises what happens on the Linux kernel mailing list, and in Debian, in free software development in general, and what Sarah Sharp has said and done. I continue to think he is wrong about everything on this topic.

I am very surprised that Lars^WWirzenius is in possession of a crystal ball that allows him to see and evaluate my attitude towards all these items, considering that I have presented links to posts on the LKML, and yes, my interpretation of the matter. Mind that the whole text amounts to about 130 lines on my screen, while my personal opinion was stated in two paragraphs of total 9 lines plus some interspersed comments. And in these maybe 20 lines I didn t make general statements about open source, kernel development etc etc. Lars, I know you have closed your blog for comments so I couldn t ask you but please, can you send me one of your crystal balls if you have more of them? At least he has managed to keep a bit of proper writing and disagreed with my statement on Debian (whether there is fun or not in Debian). He is of course free to do that, but please, don t rob me the right to state that I think Debian has changed. All this dispute centers around people not being capable to distinguish two things: One, being against the Code of Conduct due to the inclusion of administrative actions without clear definitions, and Two, being pro offensive behavior and and insults. Now, dear Lars^WWirzenius, please listen: I never advocated abusive behavior or insults, nor do I defend it. (Did you hear that!) I simply opposed the Code of Conduct as ruling instrument. And what kind of emails I got due to my opposition was far outside the Code of Conduct you are so strongly defending. So please, stay at the facts, and stop insulting me. Thanks. Mart n Ferrari Concerning Mart n I don t have much to say but please, stop spreading lies. You stated:

Once again, he s complaining about how the fun from Debian has been lost because making sexist jokes, or treating other people like shit is not allowed any more.

Could you please come up with a reference to this? Or are you just interpreting? I am very disappointed about this level of discussion between Debian Developers. You not even cared to answer my comment on your blog. Should I say something clear here you should be happy that this has not been written on a Debian mailing list, otherwise I hope the Code of Conduct hammer would hit you.

So yes, it seems that at least these two blogs underline exactly what my opinion is: Communication culture in Debian has changed to be more company-like. Probably due to the ever increasing amount of developers that are paid for their work on Debian (Ubuntu), the pressure to follow US company codes has taken a firm grip, so firm that even stating my diverging opinion is already enough to get branded. Good Future!

Russ Allbery: Review: Magic's Price

Review: Magic's Price, by Mercedes Lackey

Series:	Last Herald Mage #3
Publisher:	DAW
Copyright:	July 1990
ISBN:	0-88677-426-8
Format:	Mass market
Pages:	351

Magic's Price is the conclusion of the Last Herald Mage trilogy (Lackey is from an era of fantasy in which trilogies stopped at three books), but those books are widely spaced and mostly stand alone. You might miss some context around Vanyel's personal life and the politics of the kingdom, but there's a lot of somewhat-awkward in-line summary, so you could read these books out of order if you wanted to. As with the previous two, this is a re-read of a book I've read several times before, but not for more than fifteen years. This is the book that finally introduces Stefan. On this re-read, that surprised me I thought of him as a large part of Vanyel's overall arc but he's only in the third book. He is, of course, key to the grand conclusion of Vanyel's story, beginning as the Bard who has the ability to blunt the pain of the badly-suffering king of Valdemar and then becoming more than that to Van. It's also a book about Vanyel trying desperately to hold things together as the country falls on hard times and far too much weight is put on his shoulders. And a book about a climactic fight against an arch-villain (awkwardly and retroactively inserted). And Vanyel coming to terms with his own foresight. It's a book about rather a lot of things, and therein lies a problem. I thought Magic's Promise had a better story than Magic's Pawn, but both had reasonably coherent stories and some internal flow. Magic's Price has three or four interwoven stories and some odd stops and starts. This is still early in Lackey's writing career (three years after her first novel), and I don't think she was entirely successful in what she was trying to do. The climaxes of the different stories don't quite line up and don't get enough separate attention, leaving a book that lacks clear dramatic shape, an engrossing build-up, or a satisfying climax. The biggest problem I had is that the main threat is not adequately supported. The story is building towards a rather drastic conclusion, one that has been foreshadowed from early in the series, but the threat that creates that conclusion comes out of nowhere. Lackey tries to tie it back to events from earlier in the series, and maybe it was planned all along, but there's little concrete support and no build-up. It feels like a retcon. And the key character in that threat receives almost no characterization whatsoever. I'm a bit grateful for that, since Lackey tends towards total evil in her villains, but to have the cause of all the drama in this story be a complete cipher takes a lot out of the story. (Also, I truly could have done without the rape.) The other irritation is that Vanyel spends quite a lot of the book being a jackass. This is another occasional problem with Lackey's books: her love stories, at least in her early books, involve a bit too much of people being stupid at each other. But I hadn't remembered just how much Van jerked Stef around and treated him miserably, straight to the end of the book. Love is a bit too ordained and imbued with magical powers in Valdemar, which is lucky for Vanyel: extensive plot support is required to explain why Stef didn't find someone else who was less of an ass. In the real world, when someone insists they have no emotional space for you, the correct reaction is "gotcha, have a nice life" and finding someone else to spend time with. This gets partly sorted out in the end, but I didn't find the reconciliation entirely believable, and certainly not fair to Stef. (And the age difference is just a little creepy, although thankfully Lackey doesn't put much focus on that.) On my first read of the Valdemar corpus, this was the book during which I started getting interested in the lore and politics of this universe. There's much more of that to come, thankfully, as I think it's my favorite part of the series. At this point in her writing career, Lackey is getting better at avoiding infodumps and making her magic system interesting. The politics is still a bit black and white, but the Tayledras are a lot of fun (and increasingly less like elves, even if that's the obvious initial inspiration), and the areas beyond Valdemar are developing some more coherence. The conclusion loses a lot from the lack of build-up and emotional support, but it still adds an interesting bit of magical history that will be important later. I wish I'd enjoyed Magic's Price more on re-read, but it was much spottier and more exasperating than I had remembered. That said, it certainly has its moments, particularly around the lore. This is the first book where one starts to realize just how powerful the Companions are, and just how much they're capable of that they're not showing. When I first read it, I loved it. Now, the flaws are too obvious, and Vanyel's behavior is too irritating, for me to unreservedly recommend it, but Magic's Price, and the whole trilogy, is important for building the lore of later books. When I get to Elspeth's part of the series, the politics and magical system will be developed at much greater length. Next, though, my favorite stories in this universe: Tarma and Kethry. Rating: 6 out of 10

12 October 2015

Michael Stapelberg: ratt: Rebuild All The Things!

When uploading a new library package which changes its API/behavior in a subtle way, typically you will only hear about the downstream breakage after you ve uploaded the new library package (via bug reports telling you that your package FTBFS, fails to build from source). I prefer quality-assurance to happen proactively rather than reactively whenever possible, so I set out to write a tool which automates the rebuild of reverse-build-dependencies, i.e. all packages whose build process could be affected by the package one is about to upload. The result is a tool called ratt, which is short for Rebuild All The Things! . It injects the newly-built Debian package you provide it, figures out all the reverse-build-dependencies, invokes sbuild for all of them, and finally presents you with a list of packages that failed to build. To demonstrate how the tool works, let s assume we want to upload a new version of the Go library github.com/jacobsa/gcloud. To keep this example brief, we don t actually do anything to the package but bump its version number:

midna /tmp $ debcheckout golang-github-jacobsa-gcloud-dev
declared git repository at git://anonscm.debian.org/pkg-go/packages/golang-github-jacobsa-gcloud.git
git clone git://anonscm.debian.org/pkg-go/packages/golang-github-jacobsa-gcloud.git golang-github-jacobsa-gcloud-dev ...
Cloning into 'golang-github-jacobsa-gcloud-dev'...
remote: Counting objects: 84, done.
remote: Compressing objects: 100% (73/73), done.
remote: Total 84 (delta 19), reused 0 (delta 0)
Receiving objects: 100% (84/84), 87.70 KiB   0 bytes/s, done.
Resolving deltas: 100% (19/19), done.
Checking connectivity... done.
midna /tmp $ cd golang-github-jacobsa-gcloud-dev
midna /tmp/golang-github-jacobsa-gcloud-dev $ dch -i -m 'dummy new version'
midna /tmp/golang-github-jacobsa-gcloud-dev $ git commit -a -m 'dummy new version'
midna /tmp/golang-github-jacobsa-gcloud-dev $ gbp buildpackage --git-pbuilder
[ ]

So far, so good. Now let s invoke ratt to rebuild all reverse-build-dependencies:

midna /tmp/golang-github-jacobsa-gcloud-dev $ ratt golang-github-jacobsa-gcloud_0.0\~git20150709-2_amd64.changes         
2015/08/16 11:48:41 Loading changes file "golang-github-jacobsa-gcloud_0.0~git20150709-2_amd64.changes"
2015/08/16 11:48:41  - 1 binary packages: golang-github-jacobsa-gcloud-dev
2015/08/16 11:48:41  - corresponding .debs (will be injected when building):
2015/08/16 11:48:41     golang-github-jacobsa-gcloud-dev_0.0~git20150709-2_all.deb
2015/08/16 11:48:41 Loading sources index "/var/lib/apt/lists/ftp.ch.debian.org_debian_dists_sid_contrib_source_Sources"
2015/08/16 11:48:41 Loading sources index "/var/lib/apt/lists/ftp.ch.debian.org_debian_dists_sid_main_source_Sources"
2015/08/16 11:48:43 Loading sources index "/var/lib/apt/lists/ftp.ch.debian.org_debian_dists_sid_non-free_source_Sources"
2015/08/16 11:48:43 Building golang-github-jacobsa-ratelimit_0.0~git20150723.0.2ca5e0c-1 (commandline: [sbuild --arch-all --dist=sid --nolog golang-github-jacobsa-ratelimit_0.0~git20150723.0.2ca5e0c-1 --extra-package=golang-github-jacobsa-gcloud-dev_0.0~git20150709-2_all.deb])
2015/08/16 11:49:19 Build results:
2015/08/16 11:49:19 PASSED: golang-github-jacobsa-ratelimit_0.0~git20150723.0.2ca5e0c-1

Thanks for reading this far, and I hope ratt makes your life a tiny bit easier. As ratt just entered Debian unstable, you can install it using

apt-get install
ratt

. If you have any feedback, I m eager to hear it.

8 October 2015

Lars Wirzenius: On Norbert Preining, Sarah Sharp, and Debian

I disagree with pretty much everything Norbert Preining says in his blog post about Sarah Sharp in response to Sharp's blog post Closing a door about leaving Linux kernel development. I think Preining mis-characterises what happens on the Linux kernel mailing list, and in Debian, in free software development in general, and what Sarah Sharp has said and done. I do not wish to participate in yet another discussion on the Debian Code of Conduct, respect, appropriate behaviour, or whether these things result in worse software. I won't spend any further of my time on debating, point-by-point, the arguments and straw-men Preining makes. Those discussions have happened often enough, without any effect on Preining and those who think like him. Likewise, despite the many times he makes the arguments, I continue to think he is wrong about everything on this topic. I do feel it is important to make it clear to the people reading Planet Debian, where both Preining's and my blogs are published, that his opinions are not mainstream in the Debian project, and that despite what he says, Debian development continues to be fun.

Jonathan McDowell: The sad state of home entertainment software platforms

I got home from a law lecture[1] today and took apart my TV. As you do. I d a couple of motivations for this. The TV is an LG 55LM8600-UC. That -UC is important - it means it s the US model. I shipped it back to the UK along with the rest of my belongings at the start of the year. It s a lovely TV, and I m very happy with the picture quality. Up until now I ve been running it through a 240v->120v transformer, because the back clearly states 120v 50/60Hz. Having had success converting my Roomba PSU to UK voltage I was wondering if it might be easy enough to do the same for the TV. So the first reason for taking it apart was to look at the PSU board and work out if I could replace it with a UK version, or fix up a handful of components myself. Turns out its rated for 100-240v 50/60Hz so it looks like it should be fine to plug into UK mains directly. And, having done so, it all seems fine. However I don t take any responsibility if you try this yourself and it all goes up in smoke. The other reason was to have a general poke at trying to get access to the onboard CPU - the firmware updates are signed and encrypted so I m a bit stuck working from just those. My TV is from the US; it therefore has an ATSC tuner, and is no good for picking up the DVB-T/T2 broadcast in the UK. It also has a range of smart apps built in. Of course are also tailored to the US, so the Amazon Instant Video app connects to amazon.com and is no use with my Amazon UK account. Netflix is smarter about this and detects where you re located rather than requiring logging in to the appropriate geographic site. However the Netflix app is old and doesn t support user profiles. Which causes a problem when I spend an afternoon watching Unbreakable Kimmy Schmidt and it screws up the recommendations for my girlfriend. In an ideal world I d easily be able to update the apps on my TV to recent versions appropriate for my geographic location. Instead, the current state of home entertainment software platforms is at least 10 years behind mobile phones. Some devices allow you to get hold of SDKs even as an end user, but you re stuck with a multiple ecosystems even between different models from the same manufacturer. LG has no interest in updating my 2012-era TV. They d rather I buy a nice new 2015 TV. Which means at present my TV is mostly a really nice monitor for my XBMC/Kodi box. I can get away with using my Blu-ray player for Netflix, as it does user profile, but it s also a US model and again Amazon Instant Video doesn t work with a UK account. I should point out /that/ box is from Panasonic, just in case anyone thinks I m needlessly picking on LG. My amp is an Onkyo and it would be nice if it could stream music from Amazon/Google as well (it supports Spotify and a couple of others already), but again no joy there. I understand there are DRM issues around video and audio content, but these problems seem to have largely been solved in the Android world while still providing a fairly standard development environment and allowing end users to install their own software. It would seem to make more sense if there was a standard TV / Blu-ray platform that the likes of Netflix and Amazon could write apps once for, reducing the effort required on both sides to get the latest and greatest services on end user equipment. I guess this is some of what Google was trying to achieve with Google TV, but it s probably less interesting to them than mobile devices as I certainly don t tend to do things like search on my TV; I just want to view content and would be mighty pissed off if it started advertising things at me unexpectedly. While I m not planning to buy a new TV any time soon, I could do with a UK spec 3D-capable Blu-ray player. Is there anything out there with some commitment to open apps / proper updates? I haven t found anything, but I d love to be proved wrong. [1] It was on the duty of care and negligence, if you care.

6 October 2015

Matthew Garrett: Going my own way

Reaction to Sarah's post about leaving the kernel community was a mixture of terrible and touching, but it's still one of those things that almost certainly won't end up making any kind of significant difference. Linus has made it pretty clear that he's fine with the way he behaves, and nobody's going to depose him. That's unfortunate, because earlier today I was sitting in a presentation at Linuxcon and remembering how much I love the technical side of kernel development. "Remembering" is a deliberate choice of word - it's been increasingly difficult to remember that, because instead I remember having to deal with interminable arguments over the naming of an interface because Linus has an undying hatred of BSD securelevel, or having my name forever associated with the deepthroating of Microsoft because Linus couldn't be bothered asking questions about the reasoning behind a design before trashing it.

In the end it's a mixture of just being tired of dealing with the crap associated with Linux development and realising that by continuing to put up with it I'm tacitly encouraging its continuation, but I can't be bothered any more. And, thanks to the magic of free software, it turns out that I can avoid putting up with the bullshit in the kernel community and get to work on the things I'm interested in doing. So here's a kernel tree with patches that implement a BSD-style securelevel interface. Over time it'll pick up some of the power management code I'm still working on, and we'll see where it goes from there. But, until there's a significant shift in community norms on LKML, I'll only be there when I'm being paid to be there. And that's improved my mood immeasurably.

(Edited to add a context link for the "deepthroating of Microsoft" reference)

comments

Norbert Preining: Craft Beer Kanazawa 2015

Last weekend the yearly Craft Beer Kanazawa Festival took place in central Kanazawa. This year 14 different producers brought about 80 different kind of beers for us to taste. Compared with 6 years ago when I came to Japan and Japan was still more or less Kirin-Asahi-Sapporo country without any distinguishable taste, the situation as improved vastly, and we can now enjoy lots of excellent local beers!

Returning from a trip to Swansea and a conference in Fukuoka, I arrived at Kanazawa train station and went directly to the beer festival. A great welcome back in Kanazawa, but due to excessive sleep deprivation and the feeling of finally I want to come home , I only enjoyed 6 beers from 6 different producers. In the gardens behind the Shinoki Cultural Center lots of small tents with beer and food were set up. Lots of tables and chairs were also available, but most people enjoyed flocking around in the grass around the tents. What a difference to last year s rainy and cold beer festival!

This year s producers were (in order from left to right, page links according to language):

Swanlake Beer ( )
Ise Kadoya Brewery ( )
Yoho Brewing ( )
Wakuwaku Farm Kawakita ( )
Minoh Brewery ( )
Ohya Brasserie ( )
Shiga Kogen Beer ( )
Harvest Moon ( )
Ushitora Brewery ( )
Johana Brewery ( )
Fujizakura Brewery ( )
North Island Brewery ( )
Minami Shinshu Brewery ( )
Heart&Beer Nihonkai Club ( )

With only 6 beers to my avail (due to ticket system), I choose the ones I don t have nearby. Mind that the following comments are purely personal and do not define a quality standards

I just say what I like from worst to best:

Ohya Brasserie, Kitokito Hop : A desaster, I was close to throw this beer away, but then thought (what a waste!). Strange and disturbing taste.
Ushitora Brewery, Pure Street Session IPA: Ok, but nothing special. Too light and not taste a bit unclear to me.
Minoh Brewery, Momo Weizen: Typical Weizen Beer, light and refreshing taste. Good.
Swanlake Brewery, some seasonal IPA: good, not so extremely bitter, nice taste.
Ise Kadoya Brewery, Pale Ale: very good, full taste
Yoho Brewery, Red Ale: my absolute favorite I don t know why, but this brewery simply produces absolutely stunning ales. Their Aooni IPA is my day-in-day-out beer, world class. Their Yona Yona Ale, less bitter than the Aooni, is already famous, and this Red Ale was a perfect addition.

A great beer festival and I am looking for next years festival to try a few more. In the mean time I will stock up beers at home, so that I have always a good Yoho Brewery beer at hand! Enjoy

5 October 2015

Julien Danjou: Gnocchi talk at OpenStack Paris Meetup #16

Last week, I've been invited to the OpenStack Paris meetup #16, whose subject was about metrics in OpenStack. Last time I spoke at this meetup was back in 2012, during the OpenStack Paris meetup #2. A very long time ago!

I talked for half an hour about Gnocchi, the OpenStack project I've been running for 18 months now. I started by explaining the story behind the project and why we needed to build it. Ceilometer has an interesting history and had a curious roadmap these last year, and I summarized that briefly. Then I talk about how Gnocchi works and what it offers to users and operators. The slides where full of JSON, but I imagine it offered a interesting view of what the API looks like and how easy it is to operate. This also allowed me to emphasize how many use cases are actually really covered and solved, contrary to what Ceilometer did so far. The talk has been well received and I got a few interesting questions at the end. The video of the talk (in French) and my slides are available on my talk page and below. I hope you'll enjoy it.

30 September 2015

Ben Armstrong: Halifax Mainland Common: Early Fall, 2015

A friend and I regularly meet to chat over coffee and then usually finish up by walking the maintained trail in the Halifax Mainland Common Park, but today we decided to take a brief excursion onto the unmaintained trails criss-crossing the park. The last gasp of a faint summer and early signs of fall are evident everywhere. Some mushrooms are dried and cracked in a mosaic pattern:

Ferns and other brush are browning amongst the various greens of late summer:

A few late blueberries still cling to isolated bushes here and there:

The riot of fall colours in this small clearing, dotted with cotton-grass, burst into view as we round a corner, set behind by a backdrop of nearby buildings:

The ferns here are vivid, like a slow burning fire that will take the rest of fall to burn out:

We appreciate one last splash of colour before we head back under the cover of woods to rejoin the maintained trail:

So many times we ve travelled our usual route on automatic . I m happy today we left the more travelled trail to share in these glimpses of the changing of seasons in a wilderness preserved for our enjoyment immediately at hand to a densely populated part of the city.

25 September 2015

Dirk Eddelbuettel: RcppEigen 0.3.2.5.1

A bugfix release of RcppEigen is now on CRAN and in Debian. The NEWS file entry follows.

Changes in RcppEigen version 0.3.2.5.1 (2015-09-23)

Corrected use of kitten() thanks to Grant Brown (#21)

Applied upstream change to protect against undefined behaviour with null pointers

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.