Search Results: "ana"

4 February 2015

Charles Plessy: News of the package mime-support.

The package mime-support is installed by default on Debian systems. It has two roles: first to provide the file /etc/mime.types that associates media types (formerly called MIME types) to suffixes of file names, and second to provide the mailcap system that associates media types with programs. I adopted this package at the end of the development cycle of Wheezy. Changes since Wheezy. The version distributed in Jessie brings a few additions in /etc/mime.types. Among them, application/vnd.debian.binary-package and text/vnd.debian.copyright, which as their name suggest describe two file formats designed by Debian. I registered these types to the IANA, which is more open to the addition of new types since the RFC 6838. The biggest change is the automatic extraction of the associations between programs and media types that are declared in the menu files in FreeDesktop format. Before, it was the maintainer of the Debian package who had to extract this information and translate it in mailcap format by hand. The automation is done via dpkg triggers. A big thank you to Kevin Ryde who gave me a precious help for the developments and corrections to the run-mailcap program, and to all the other contributors. Your help is always welcome! Security updates. In December, Debian has been contacted by Timothy D. Morgan, who found that an attacker could get run-mailcap to execute commands by inserting them in file names (CVE-2014-7209). This first security update for me went well, thanks to the help and instructions of Salvatore Bonaccorso from the Security team. The problem is solved in Wheezy, Jessie and Sid, as well as in Squeeze through its long term support. One of the consequences of this security update is that run-mailcap will systematically use the absolute path to the files to open. For harmless files, this is a bit ugly. This will perhaps be improved after Jessie is released. Future projects The file /etc/mime.types is kept up to date by hand; this is slow and inefficient. The package shared-mime-info contains similar information, that could be used to autogenerate this file, but that would require to parse a XML source that is quite complex. For the moment, I am considering to import Fedora's mailcap package, where the file /etc/mime.types is very well kept up to date. I have not yet decided how to do it, but maybe just by moving that file from one package to the other. In that case, we would have the mime-support package that would provide mailcap support, and the package whose source is Fedora's mailcap package who would provide /etc/mime.types. Perhaps it will be better to use clearer names, such as mailcap-support for the first and media-types for the second? Separating the two main functionalities of mime-support would have an interesting consequence: the possibility of not installing the support for the mailcap system, or to make it optional, and instead to use the FreeDesktop sytem (xdg-open), from the package xdg-utils. Something to keep in mind...

Vincent Bernat: Directory bookmarks with Zsh

There are numerous projects to implement directory bookmarks in your favorite shell. An inherent limitation of those implementations is they being only an enhanced cd command: you cannot use a bookmark in an arbitrary command. UPDATED: My initial implementation with Zsh was using dynamic named directories. I have been pointed on Twitter that there is a simpler way to implement bookmarks. The article has been updated to reflect that. As a side note, it is also possible to just use shell variables1. Zsh comes with a not well-known feature called static named directories. They are declared with the hash builtin and can be refered by prepending ~ to them:
$ hash -d -- -lldpd=/home/bernat/code/deezer/lldpd
$ echo ~-lldpd/README.md
/home/bernat/code/deezer/lldpd/README.md
$ head -n1 ~-lldpd/README.md
lldpd: implementation of IEEE 802.1ab (LLDP)
Because ~-lldpd is substituted during file name expansion, it is possible to use it in any command like a regular directory, like shown above. The - prefix is only here to avoid collision with home directories. Bookmarks are kept into a dedicated directory, $MARKPATH. Each bookmark is a symbolic link to the target directory: for example, ~-lldpd should be expanded to $MARKPATH/lldpd which points to the appropriate directory. Assuming that you have populated $MARKPATH with some links, here is how bookmarks are registered:
for link ($MARKPATH/*(N@))  
    hash -d -- -$ link:t =$ link:A 
 
You also automatically get completion and prompt expansion:
$ pwd
/home/bernat/code/deezer/lldpd/src/lib
$ echo $ (%):-%~ 
~-lldpd/src/lib
The last step is to manage bookmarks without adding or removing symbolic links manually. The following bookmark() function will display the existing bookmarks when called without arguments, will remove a bookmark when called with -d or add the current directory as a bookmark otherwise.
bookmark()  
    if (( $# == 0 )); then
        # When no arguments are provided, just display existing
        # bookmarks
        for link in $MARKPATH/*(N@); do
            local markname="$fg[green]$ link:t $reset_color"
            local markpath="$fg[blue]$ link:A $reset_color"
            printf "%-30s -> %s\n" $markname $markpath
        done
    else
        # Otherwise, we may want to add a bookmark or delete an
        # existing one.
        local -a delete
        zparseopts -D d=delete
        if (( $+delete[1] )); then
            # With  -d , we delete an existing bookmark
            command rm $MARKPATH/$1
        else
            # Otherwise, add a bookmark to the current
            # directory. The first argument is the bookmark
            # name.  .  is special and means the bookmark should
            # be named after the current directory.
            local name=$1
            [ $name == "." ] && name=$ PWD:t 
            ln -s $PWD $MARKPATH/$name
        fi
    fi
 
Find the complete version on GitHub.

Dynamic named directories Another (more complex) way to achieve the same thing is using dynamic named directories. I was initially using this solution but it is far more complex. This section is only here for historical reason. You can find the complete implementation in my GitHub repository. During file name expansion, a ~ followed by a string in square brackets is provided to the zsh_directory_name() function which will eventually reply with a directory name. This feature can be used to implement directory bookmarks:
$ cd ~[@lldpd]
$ pwd
/home/bernat/code/deezer/lldpd
$ echo ~[@lldpd]/README.md
/home/bernat/code/deezer/lldpd/README.md
$ head -n1 ~[@lldpd]/README.md
lldpd: implementation of IEEE 802.1ab (LLDP)
Like for static named directories, because ~[@lldpd] is substituted during file name expansion, it is possible to use it in any command like a regular directory.

Basic implementation Bookmarks are still kept into a dedicated directory, $MARKPATH and are still symbolic links. Here is how the core feature is implemented:
_bookmark_directory_name()  
    emulate -L zsh #  
    setopt extendedglob
    case $1 in
        n)
            [ [ $2 != ( #b)"@"(?*) ]] && return 1 #  
            typeset -ga reply
            reply=($ $ :-$MARKPATH/$match[1] :A ) #  
            return 0
            ;;
        *)
            return 1
            ;;
    esac
    return 0
 
add-zsh-hook zsh_directory_name _bookmark_directory_name
zsh_directory_name() is a function accepting hooks2: instead of defining it directly, we define another function and register it as a hook with add-zsh-hook. The hook is expected to handle different situations. The first one is to be able to transform a dynamic name into a regular directory name. In this case, the first parameter of the function is n and the second one is the dynamic name. In , the call to emulate will restore the pristine behaviour of Zsh and also ensure that any option set in the scope of the function will not have an impact outside. The function can then be reused safely in another environment. In , we check that the dynamic name starts with @ followed by at least one character. Otherwise, we declare we don t know how to handle it. Another hook will get the chance to do something. (#b) is a globbing flag. It activates backreferences for parenthesised groups. When a match is found, it is stored as an array, $match. In , we build the reply. We could have just returned $MARKPATH/$match[1] but to hide the symbolic link mechanism, we use the A modifier to ask Zsh to resolve symbolic links if possible. Zsh allows nested substitutions. It is therefore possible to use modifiers and flags on anything. $ :-$MARKPATH/$match[1] is a common trick to turn $MARKPATH/$match[1] into a parameter substitution and be able to apply the A modifier on it.

Completion Zsh is also able to ask for completion of a dynamic directory name. In this case, the completion system calls the hook function with c as the first argument.
_bookmark_directory_name()  
    # [...]
    case $1 in
        c)
            # Completion
            local expl
            local -a dirs
            dirs=($MARKPATH/*(N@:t)) #  
            dirs=("@"$ ^dirs ) #  
            _wanted dynamic-dirs expl 'bookmarked directory' compadd -S\] -a dirs
            return
            ;;
        # [...]
    esac
    # [...]
 
First, in , we create a list of possible bookmarks. In *(N@:t), N@ is a glob qualifier. N allows us to not return nothing if there is no match (otherwise, we would get an error) while @ only returns symbolic links. t is a modifier which will remove all leading pathname components. This is equivalent to use basename or $ something##*/ in POSIX shells but it plays nice with glob expressions. In , we just add @ before each bookmark name. If we have b1, b2 and b3 as bookmarks, $ ^dirs expands to b1,b2,b3 and therefore "@"$ ^dirs expands to the (@b1 @b2 @b3) array. The result is then feeded into the completion system.

Prompt expansion Many people put the name of the current directory in their prompt. It would be nice to have the bookmark name instead of the full name when we are below a bookmarked directory. That s also possible!
$ pwd
/home/bernat/code/deezer/lldpd/src/lib
$ echo $ (%):-%~ 
~[@lldpd]/src/lib
The prompt expansion system calls the hook function with d as first argument and the file name to transform.
_bookmark_directory_name()  
    # [...]
    case $1 in
        d)
            local link slink
            local -A links
            for link ($MARKPATH/*(N@))  
                links[$  #link:A $'\0'$ link:A ]=$ link:t  #  
             
            for slink ($ (@On)$ (k)links )  
                link=$ slink#*$'\0'  #  
                if [ [ $2 = ( #b)($ link )( /*) ]]; then
                    typeset -ga reply
                    reply=("@"$ links[$slink]  $(( $  #match[1]  )) )
                    return 0
                fi
             
            return 1
            ;;
        # [...]
    esac
    # [...]
 
OK. This is some black Zsh wizardry. Feel free to skip the explanation. This is a bit complex because we want to substitute the most specific bookmark, hence the need to sort bookmarks by their target lengths. In , the associative array $links is created by iterating on each symbolic link ($link) in the $MARKPATH directory. The goal is to map a target directory with the matching bookmark name. However, we need to iterate on this map from the longest to the shortest key. To achieve that, we prepend each key with its length. Remember, $ link:A is the absolute path with symbolic links resolved. So, $ #link:A is the length of this path. We concatenate the length of the target directory with the target directory name and use $'\0' as a separator because this is the only safe character for this purpose. The result is mapped to the bookmark name. The second loop is an iteration on the keys of the associative array $links (thanks to the use of the k parameter flag in $ (k)links ). Those keys are turned into an array (@ parameter flag) and sorted numerically in descending order (On parameter flag). Since the keys are directory names prefixed by their lengths, the first match will be the longest one. In , we extract the directory name from the key by removing the length and the null character at the beginning. Then, we check if the extracted directory name matches the file name we have been provided. Again, (#b) just activates backreferences. With extended globbing, we can use the or operator, . So, when either the file name matches exactly the directory name or is somewhere deeper, we create the reply which is an array whose first member is the bookmark name and the second member is the untranslated part of the file name.

Easy typing Typing ~[@ is cumbersome. Hopefully, Zsh line editor can be extended with additional bindings. The following snippet will substitute @@ (if typed without a pause) by ~[@:
vbe-insert-bookmark()  
    emulate -L zsh
    LBUFFER=$ LBUFFER "~[@"
 
zle -N vbe-insert-bookmark
bindkey '@@' vbe-insert-bookmark
In combination with the autocd option and completion, it is quite easy to jump to a bookmarked directory.

  1. For example:
    $ lldpd=/home/bernat/code/deezer/lldpd
    $ echo $lldpd/README.md
    /home/bernat/code/deezer/lldpd/README.md
    $ head -n1 $lldpd/README.md
    lldpd: implementation of IEEE 802.1ab (LLDP)
    
    The drawback is that you don t have a separate namespace for your bookmarks. You can still use a special prefix for that. Also, no prompt expansion.
  2. Other functions accepting hooks are chpwd() or precmd().

1 February 2015

Ritesh Raj Sarraf: Debian GNU/Hurd on VirtualBox

One of the great things about Debian is the wide range of kernels it supports can run. This gives the user the flexibility to not spend time on the common userland stuff. For example, most apps, package management and system admin tasks are common across all Debian platforms. These platforms may not be optimal at par to Linux, but still, choice is good. For long, I had used Debian GNU/Hurd, only on a KVM hypervisor. Recently being involved in VirtualBox maintenance, I've almost beeing using VirtualBox for all my virtualization tasks. So it was time to try out Hurd on VirtualBox. All praises to the work the Debian Hurd team has put, it just works wonderfully. To try out Hurd on VirtualBox:
  1. You could download ready to use VirtualBox images from here.
  2. These are raw images. For VirtualBox, I'd recommend you convert the image to VDI format.
    1. rrs@learner:~/VirtualBox VMs$ VBoxManage convertdd debian-hurd-20150105.img debian-hurd-20150105.vdi --format VDI
      Converting from raw image file="debian-hurd-20150105.img" to file="debian-hurd-20150105.vdi"...
      Creating dynamic image with size 3146776576 bytes (3001MB)...
    2. The VDI format may help in with some of the goodies VirtualBox has to offer, like efficient snapshots.
  3. The rest remains the same. You need to configure the VM just as else. Currently supported arch is x86 only, so make the selections accordingly.
  4. The final VM config may look something like the following screenshot.
  5. And here's the result in a video captured on VirtualBox running Debian GNU/Hurd.

Categories:

Keywords:

Images:

30 January 2015

Raphaël Hertzog: My Free Software Activities for January 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 12 hours on Debian LTS. I did the following tasks: I want to expand on two cases that I stumbled upon in my CVE triage work and that took quite long to investigate each. While my after-the-fact description is rather straightforward, the real process involved more iterations and data gathering that I do not mention here. First I was investigating CVE-2012-6685 on libnokogiri-ruby and the upstream bug discussion revealed that libxml2 could also be part of the problem. Using the tests cases submitted there, I confirmed that libxml2 was also affected by an issue of its own then I started to analyze the history of CVE of libxml2 to find out whether that issue got a CVE assigned: yes, that was CVE-2014-0191 (although the CVE description is unrelated). But this CVE was marked as fixed in all releases. Why? It turns out that the upstream fix for this CVE is just the complement of another commit that was merged way earlier (and that was used as a basis for the commit as the copy/paste of the comment shows). When the security teams integrated the upstream patch in wheezy/squeeze, they were probably not aware that a full fix required to also include something else. In the end, I thus reopened CVE-2014-0191 on our tracker (commit here). The second problematic case was pound. Thijs Kinkhorst added pound related data on the multiple (high profile) SSL related issues. So it appeared on my radar of new vulnerable package in Squeeze because it was marked that CVE-2009-3555 was fixed in version 2.6-2 while Squeeze has 2.5-1. There was no bug reference in the security tracker and the Debian changelog for that version only mentioned an anti_beast patch which is yet another issue (CVE-2011-3389). I had to dig a bit deeper in the end I discovered that the above patch also has provisions for the CVE that was of interest to me, except that Brian May recently reported in #765649 that the package was still vulnerable to this issue I tried to understand where the above patch was failing and thus submitted my findings to the bug. And I updated the tracker data with my newly gained knowledge (commit 31751 and 31752). Tryton For me, January is always the month where I try to close the accounting books of Freexian. This year is no exception except that it s the first year where I do this with Tryton. I first upgraded to Tryton 3.4 to have the latest version. Despite this I discovered multiple problems while doing so since I don t want to have those problems next year, I reported them and prepared fixes for those related to the French chart of accounts: Saltstack I mentioned this idea last month setting up and maintaining a lot of sbuild chroots can be tiresome so I wanted to automate this as much as possible. To achieve this I created three Salt formulas and got them added to the official Saltstack repository: Each one builds on top of the former. debootstrap-formula creates chroots with debootstrap or cdebootstrap. schroot-formula does the same and registers those chroots in schroot. And sbuild-formula does the same as schroot-formula but with different defaults that are more suited to sbuild chroots (and obviously ensures that sbuild is installed and that generated chroots are buildd chroots). With the sbuild formula I can put this in pillar data:
sbuild:
  chroots:
    wheezy:
      architectures: [amd64, i386]
      extra_dists:
        - wheezy-backports
        - wheezy-security
      extra_aliases:
        - wheezy-backports
        - stable-security
        - wheezy-security
    jessie:
    [...]
And then a simple salt-call state.highstate (I m running in standalone mode) will ensure that I have all the chroots properly setup. Misc packaging I packaged new upstream releases of Django in experimental and opened a pre-approval request to get the latest 1.7.x in jessie (#775892). It seems to be a difficult sell for the release team, which is a pity because we have active Debian developers, active upstream developers, and everybody is well aware of the no-new features rule to avoid regressions. Where is the risk? I also filed an unblock request for Dolibarr (on the request of the security team which wants to see the CVE fix reach Jessie). I did small contributions to two bugs that were of special interest to some of my donators (#751339 and #774811), they were not under my responsibility but I tried to get them moving by pinging the relevant people. I prepared a security upload for Django in Wheezy (python-django_1.4.5-1+deb7u9) and sent it to the security team. While doing this I discovered a small problem in their backported patch that I reported upstream in Django s ticket #24239. Debian France With the new year, it s again time to organize a general assembly with the election of a third of its board. So we solicited candidacies among the members and I m pleased to see that we got 6 candidacies for the 3 seats. It s a good sign that we still have enough persons caring about the association. One of them is even speaking of Debconf 17 in France great plans! On my side, I announced that I would not candidate to be president for the next year. I will stay on the board though to ensure we have a smooth transition. Thanks See you next month for a new summary of my activities.

One comment Liked this article? Click here. My blog is Flattr-enabled.

Laura Arjona: Going selfhosting: Installing Debian Wheezy in my home server

It was in my mind to open a new series of articles with topic selfhosting , because I really believe in free software based network services and since long time I want to plug a machine 24 7 at home to host my blog, microblog, MediaGoblin, XMPP server, mail, and, in conclusion, all the services that now I trust to very kind third parties that run them with free software, but I know I could run myself (and offer them to my family and friends). Last September I bought the domain larjona.net (curious, they say buy but it s a rent, for 1,2,3 years never yours. Pending another post about my adventures with the domain name, dynamic DNS, and SSL certs!) and I bought an HP Microserver G7 N54L, with 2 GB RAM. It had a 250GB SATA harddisk and I bought 2 more SATA harddisks, 1 TB each, to setup a RAID 1 (mirror). Total cost (with keyboard and mouse), 300 . A friend gave me a TFT monitor that was too old for him (1024 768) but it serves me well, (it s a server, no graphical interface, and I will connect remotely most of the times). Installing Debian stable (wheezy) I decided to install Debian stable. Jessie was not frozen yet, and since it was my first non-LAMP server install, I wanted to make sure that errors and problems would be my errors, not issues of the non-released-yet distro. I thought to install YunoHost or some other distro prepared for selfhosting, but I ve never tried them, and I have not much free time, so I decided to stick on Debian, my beloved distro, because it s the one that I know best and I m part of its awesome community. And maybe I could contribute back some bug reports or documentation. I wanted to try a crypto setup (just for fun, just for learn, for its benefits, and to be one more freecrypto-tester in the world) so after reading a bit: https://wiki.debian.org/DebianInstaller/SataRaid
https://wiki.archlinux.org/index.php/disk_encryption
http://madduck.net/docs/cryptdisk/
http://linuxgazette.net/140/kapil.html
http://smcv.pseudorandom.co.uk/2008/09/cryptroot/
http://www.linuxquestions.org/questions/linux-security-4/lvm-before-and-after-encryption-871379/ and some other pages, and try some different things, this is the setup that I managed to configure: Everything went well. Yay! Some doubts and one problem Everything went quite well except some doubts: After talking about this issues with friends (and in debian-women IRC channel), I decided to install the non-free driver, just in case, with the same reasoning as with the RAID: let the card do the job, so the CPU can care about other things. Again, I notice that learning a bit about benchmarking (and having some time to do some tests) would be nice And now, the problem: I left this problem apart and go on installing the software. I would think later what to do. Installing MediaGoblin The most urgent selfhosting service, for me, was GNU MediaGoblin, because I wanted to show my server to my family in Christmas, and upload the pictures of the babies and kids of the family. And it s a project where I contribute translations and I am a big fan, so I would be very proud of hosting my own instance. I followed the documentation to setup 2 instances of GNU MediaGoblin 0.7 (the stable release in the moment), with their corresponding PostgreSQL databases. Why two instances? Well, I want an instance to host and show my videos, images, and replicate videos that I like, and a private one for sharing photos and videos with my family. MediaGoblin has no privacy settings yet, so I installed separate instances, and the private one I put it in a different port, with a self-signed SSL cert, and enabled http-authorization in Nginx, so only authorized Linux users of my machine can accesss the website. Installing MediaGoblin was easier than what I thought. I only had some small doubts about the documentation, and they were solved in the IRC channel. You can access, for example, my user profile in my public instance, and see some different files that I already uploaded. I m very happy!! Face to face with the bug, again I had to solve the problem of the password not accepted in reboots. I began to think that it could be a bug in cryptsetup. Should I upgrade the package to the version in wheezy-backports? Jessie was almost frozen, maybe it was time to upgrade the whole system, to see if the problem was solved (and to see how my MediaGoblin was working or not in Jessie. It should work, it s almost packaged! But who knows). And if it didn t work, maybe it was time to file a bug So I upgraded my system to Debian Jessie. And after upgrade, the system didn t boot. But that s the story of another blog post (that I still need to finish to write don t worry, it has happy end, as you could see accessing my Mediagoblin site!). Comments? You can comment in this pump.io thread.
Filed under: My experiences and opinion Tagged: Debian, encryption, English, libre software, MediaGoblin, Moving into free software, N54L, selfhosting, sysadmin

27 January 2015

Laura Arjona: Upgrading my computers to Debian Jessie: Husband s laptop (Acer Aspire 5250)

This is an old laptop, with AMD E-300 processor, 6 GB RAM, Radeon HD 6310 VGA and Atheros AR9485 wireless network adapter. It was running Windows 7 (preinstalled). The hard disk failed, and I put the hard disk of another laptop (a broken Acer Aspire One D255) on it. Surprisingly, the Windows 7 on it booted (after some self-configuration that took quite long), but it was a Windows 7 Home 32bits, so it was only recognizing 4 GB RAM. That was the perfect excuse to convince my husband to install Debian in the laptop and begin his transition to a free OS. Yay! I installed Debian Jessie from scratch last summer. Everything went well (the installer went fine, 8 months before than its RC1 release, congrats Debian-boot team!). I needed the non-free radeon driver for the graphical display :/ Jessie is running GNOME3 desktop, and I ve been seeing all these months the transition to 3.14 version, and later, the integration of the Lines theme (by Juliette Belin), which I like very much. I have problems to watch high quality videos, in every player that I tried (VLC, Totem, mplayer) the audio and video are not synced, and video sometimes freezes. I m almost sure that the problem is what mplayer says: Your system is too SLOW to play this! . I tried to install the ATI non-free driver for better performance, but after successfully install it and reboot, GNOME was not starting (I got a black screen, no gdm greeting me). I could log in tty2, though. I don t know if I did something wrong, how to solve the problem, and I don t wanted to waste time, so I uninstalled it and returned to the non-free firmware that goes to the Linux kernel. For now, when I need to watch a video that gives those problems, I upload the file to my GNU MediaGoblin site, or use WinFF to reduce size/quality. Overall impression Fine! Both my husband and me are very happy. The installation went really well. I m not a GNOME expert user but I find it easy, intuitive, and he found it easy too. My husband uses the computer to surf the web, watch some videos and online series (we had to install non-free flash plugin from Adobe #grr), read mail from the browser, write something in LibreOffice and print it (hey! we just plug the printer/scanner and it works, no need to install drivers!), scan some image and send it by email I set Debian as default in GRUB, and the switch from Windows has been very natural for him (he was already using Firefox and LibreOffice in Windows. He still says I m a Windows user although he is just using Debian for months!). He bought an IPhone 4S (#grr!) and I tried to connect it as shown in the corresponding Debian wiki page, but it didn t work (I got segmentation fault when connecting the phone). However, it is recognized by Shotwell and we can copy all the photos and videos to the computer, which is what we wanted to do. So no problem on that side, either. In conclussion, one more computer at home running Debian ( future stable ), and we don t run Windows at home anymore :)
Filed under: My experiences and opinion Tagged: Debian, English, Moving into free software

Matthias Klumpp: AppStream 0.8 released!

Yesterday I released version 0.8 of AppStream, the cross-distribution standard for software metadata, that is currently used by GNOME-Software, Muon and Apper in to display rich metadata about applications and other software components. What s new? The new release contains some tweaks on AppStreams documentation, and extends the specification with a few more tags and refinements. For example we now recommend sizes for screenshots. The recommended sizes are the ones GNOME-Software already uses today, and it is a good idea to ship those to make software-centers look great, as others SCs are planning to use them as well. Normal sizes as well as sizes for HiDPI displays are defined. This change affects only the distribution-generated data, the upstream metadata is unaffected by this (the distro-specific metadata generator will resize the screenshots anyway). Another addition to the spec is the introduction of an optional <source_pkgname/> tag, which holds the source package name the packages defined in <pkgname/> tags are built from. This is mainly for internal use by the distributor, e.g. it can decide to use this information to link to internal resources (like bugtrackers, package-watch etc.). It may also be used by software-center applications as additional information to group software components. Furthermore, we introduced a <bundle/> tag for future use with 3rd-party application installation solutions. The tag notifies a software-installer about the presence of a 3rd-party application bundle, and provides the necessary information on how to install it. In order to do that, the software-center needs to support the respective installation solution. Currently, the Limba project and Xdg-App bundles are supported. For software managers, it is a good idea to implement support for 3rd-party app installers, as soon as the solutions are ready. Currently, the projects are worked on heavily. The new tag is currently already used by Limba, which is the reason why it depends on the latest AppStream release. How do I get it? All AppStream libraries, libappstream, libappstream-qt and libappstream-glib, are supporting the 0.8 specification in their latest version so in case you are using one of these, you don t need to do anything. For Debian, the DEP-11 spec is being updated at time, and the changes will land in the DEP-11 tools soon. Improve your metadata! This call goes especilly to many KDE projects! Getting good data is partly a task for the distributor, since packaging issues can result in incorrect or broken data, screenshots need to be properly resized etc. However, flawed upstream data can also prevent software from being shown, since software with broken data or missing data will not be incorporated in the distro XML AppStream data file. Richard Hughes of Fedora has created a nice overview of software failing to be included. You can see the failed-list here the data can be filtered by desktop environment etc. For KDE projects, a Comment= field is often missing in their .desktop files (or a <summary/> tag needs to be added to their AppStream upstream XML file). Keep in mind that you are not only helping Fedora by fixing these issues, but also all other distributions cosuming the metadata you ship upstream. For Debian, we will have a similar overview soon, since it is also a very helpful tool to find packaging issues. If you want to get more information on how to improve your upstream metadata, and how new metadata should look like, take a look at the quickstart guide in the AppStream documentation.

25 January 2015

Chris Lamb: Recent Redis hacking

I've done a bunch of hacking on the Redis key/value database server recently:
I also made the following changes to the Debian packaging:

Joey Hess: making propellor safer with GADTs and type families

Since July, I have been aware of an ugly problem with propellor. Certain propellor configurations could have a bug. I've tried to solve the problem at least a half-dozen times without success; it's eaten several weekends. Today I finally managed to fix propellor so it's impossible to write code that has the bug, bending the Haskell type checker to my will with the power of GADTs and type-level functions. the bug Code with the bug looked innocuous enough. Something like this:
foo :: Property
foo = property "foo" $
    unlessM (liftIO $ doesFileExist "/etc/foo") $ do
        bar <- liftIO $ readFile "/etc/foo.template"
        ensureProperty $ setupFoo bar
The problem comes about because some properties in propellor have Info associated with them. This is used by propellor to introspect over the properties of a host, and do things like set up DNS, or decrypt private data used by the property. At the same time, it's useful to let a Property internally decide to run some other Property. In the example above, that's the ensureProperty line, and the setupFoo Property is run only sometimes, and is passed data that is read from the filesystem. This makes it very hard, indeed probably impossible for Propellor to look inside the monad, realize that setupFoo is being used, and add its Info to the host. Probably, setupFoo doesn't have Info associated with it -- most properties do not. But, it's hard to tell, when writing such a Property if it's safe to use ensureProperty. And worse, setupFoo could later be changed to have Info. Now, in most languages, once this problem was noticed, the solution would probably be to make ensureProperty notice when it's called on a Property that has Info, and print a warning message. That's Good Enough in a sense. But it also really stinks as a solution. It means that building propellor isn't good enough to know you have a working system; you have to let it run on each host, and watch out for warnings. Ugh, no! the solution This screams for GADTs. (Well, it did once I learned how what GADTs are and what they can do.) With GADTs, Property NoInfo and Property HasInfo can be separate data types. Most functions will work on either type (Property i) but ensureProperty can be limited to only accept a Property NoInfo.
data Property i where
    IProperty :: Desc -> ... -> Info -> Property HasInfo
    SProperty :: Desc -> ... -> Property NoInfo
data HasInfo
data NoInfo
ensureProperty :: Property NoInfo -> Propellor Result
Then the type checker can detect the bug, and refuse to compile it. Yay! Except ... Property combinators There are a lot of Property combinators in propellor. These combine two or more properties in various ways. The most basic one is requires, which only runs the first Property after the second one has successfully been met. So, what's it's type when used with GADT Property?
requires :: Property i1 -> Property i2 -> Property ???
It seemed I needed some kind of type class, to vary the return type.
class Combine x y r where
    requires :: x -> y -> r
Now I was able to write 4 instances of Combines, for each combination of 2 Properties with HasInfo or NoInfo. It type checked. But, type inference was busted. A simple expression like
foo  requires  bar
blew up:
   No instance for (Requires (Property HasInfo) (Property HasInfo) r0)
      arising from a use of  requires'
    The type variable  r0' is ambiguous
    Possible fix: add a type signature that fixes these type variable(s)
    Note: there is a potential instance available:
      instance Requires
                 (Property HasInfo) (Property HasInfo) (Property HasInfo)
        -- Defined at Propellor/Types.hs:167:10
To avoid that, it needed ":: Property HasInfo" appended -- I didn't want the user to need to write that. I got stuck here for an long time, well over a month. type level programming Finally today I realized that I could fix this with a little type-level programming.
class Combine x y where
    requires :: x -> y -> CombinedType x y
Here CombinedType is a type-level function, that calculates the type that should be used for a combination of types x and y. This turns out to be really easy to do, once you get your head around type level functions.
type family CInfo x y
type instance CInfo HasInfo HasInfo = HasInfo
type instance CInfo HasInfo NoInfo = HasInfo
type instance CInfo NoInfo HasInfo = HasInfo
type instance CInfo NoInfo NoInfo = NoInfo
type family CombinedType x y
type instance CombinedType (Property x) (Property y) = Property (CInfo x y)
And, with that change, type inference worked again! \o/ (Bonus: I added some more intances of CombinedType for combining things like RevertableProperties, so propellor's property combinators got more powerful too.) Then I just had to make a massive pass over all of Propellor, fixing the types of each Property to be Property NoInfo or Property HasInfo. I frequently picked the wrong one, but the type checker was able to detect and tell me when I did. A few of the type signatures got slightly complicated, to provide the type checker with sufficient proof to do its thing...
before :: (IsProp x, Combines y x, IsProp (CombinedType y x)) => x -> y -> CombinedType y x
before x y = (y  requires  x)  describe  (propertyDesc x)
onChange
    :: (Combines (Property x) (Property y))
    => Property x
    => Property y
    => CombinedType (Property x) (Property y)
onChange = -- 6 lines of code omitted
fallback :: (Combines (Property p1) (Property p2)) => Property p1 -> Property p2 -> Property (CInfo p1 p2)
fallback = -- 4 lines of code omitted
.. This mostly happened in property combinators, which is an acceptable tradeoff, when you consider that the type checker is now being used to prove that propellor can't have this bug. Mostly, things went just fine. The only other annoying thing was that some things use a [Property], and since a haskell list can only contain a single type, while Property Info and Property NoInfo are two different types, that needed to be dealt with. Happily, I was able to extend propellor's existing (&) and (!) operators to work in this situation, so a list can be constructed of properties of several different types:
propertyList "foos" $ props
    & foo
    & foobar
    ! oldfoo    
conclusion The resulting 4000 lines of changes will be in the next release of propellor. Just as soon as I test that it always generates the same Info as before, and perhaps works when I run it. (eep) These uses of GADTs and type families are not new; this is merely the first time I used them. It's another Haskell leveling up for me. Anytime you can identify a class of bugs that can impact a complicated code base, and rework the code base to completely avoid that class of bugs, is a time to celebrate!

24 January 2015

Dirk Eddelbuettel: Rcpp 0.11.4

A new release 0.11.4 of Rcpp is now on the CRAN network for GNU R, and an updated Debian package will be uploaded in due course. Rcpp has become the most popular way of enhancing GNU R with C++ code. As of today, 323 packages on CRAN depend on Rcpp for making analyses go faster and further; BioConductor adds another 41 packages, and casual searches on GitHub suggests dozens mores. This release once again adds a large number of small bug fixes, polishes and enhancements. And like the last time, these changes were made by a group of seven different contributors (counting code commits) plus three more providing concrete suggestions. This shows that the Rcpp development and maintenance rests a large number of (broad) shoulders. See below for a detailed list of changes extracted from the NEWS file.
Changes in Rcpp version 0.11.4 (2015-01-20)
  • Changes in Rcpp API:
    • The ListOf<T> class gains the .attr and .names methods common to other Rcpp vectors.
    • The [dpq]nbinom_mu() scalar functions are now available via the R:: namespace when R 3.1.2 or newer is used.
    • Add an additional test for AIX before attempting to include execinfo.h.
    • Rcpp::stop now supports improved printf-like syntax using the small tinyformat header-only library (following a similar implementation in Rcpp11)
    • Pairlist objects are now protected via an additional Shield<> as suggested by Martin Morgan on the rcpp-devel list.
    • Sorting is now prohibited at compile time for objects of type List, RawVector and ExpressionVector.
    • Vectors now have a Vector::const_iterator that is 'const correct' thanks to fix by Romain following a bug report in rcpp-devel by Martyn Plummer.
    • The mean() sugar function now uses a more robust two-pass method, and new unit tests for mean() were added at the same time.
    • The mean() and var() functions now support all core vector types.
    • The setequal() sugar function has been corrected via suggestion by Qiang Kou following a bug report by S ren H jsgaard.
    • The macros major, minor, and makedev no longer leak in from the (Linux) system header sys/sysmacros.h.
    • The push_front() string function was corrected.
  • Changes in Rcpp Attributes:
    • Only look for plugins in the package's namespace (rather than entire search path).
    • Also scan header files for definitions of functions to be considerd by Attributes.
    • Correct the regular expression for source files which are scanned.
  • Changes in Rcpp unit tests
    • Added a new binary test which will load a pre-built package to ensure that the Application Binary Interface (ABI) did not change; this test will (mostly or) only run at Travis where we have reasonable control over the platform running the test and can provide a binary.
    • New unit tests for sugar functions mean, setequal and var were added as noted above.
  • Changes in Rcpp Examples:
    • For the (old) examples ConvolveBenchmarks and OpenMP, the respective Makefile was renamed to GNUmakefile to please R CMD check as well as the CRAN Maintainers.
Thanks to CRANberries, you can also look at a diff to the previous release As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

23 January 2015

Jaldhar Vyas: Mini-Debconf Mumbai 2015

Last weekend I went to Mumbai to attend the Mini-Debconf held at IIT-Bombay. These are my impressions of the trip. Arrival and Impressions of Mumbai Getting there was a quite an adventure in itself. Unlike during my ill-fated attempt to visit a Debian event in Kerala last year when a bureaucratic snafu left me unable to get a visa, the organizers started the process much earlier at their end this time and with proper permissions. Yet in India, the wheels only turn as fast as they want to turn so despite their efforts, it was only literally at the last minute that I actually managed to secure my visa. I should note however that Indian government has done a lot to improve the process compared to the hell I remember from, say, a decade ago. It's fairly straightforward for tourist visas now and I trust they will get around to doing the same for conference visas in the fullness of time. I didn't want to commit to buying a plane ticket until I had the visa so I became concerned that the only flights left would be either really expensive or on the type of airline that flies you over Syria or under the Indian Ocean. I lucked out and got a good price on a Swiss Air flight, not non-stop but you can't have everything. So Thursday afternoon I set off for JFK. With only one small suitcase getting there by subway was no problem and I arrived and checked in with plenty of time. Even TSA passed me through with only a minimal amount of indignity. The first leg of my journey took me to Zurich in about eight hours. We were only in Zurich for an hour and then (by now Friday) it was another 9 hours to Mumbai. Friday was Safala Ekadashi but owing to the necessity of staying hydrated on a long flight I drank a lot of water and ate some fruit which I don't normally do on a fasting day. It was tolerable but not too pleasant; I definitely want to try and make travel plans to avoid such situations in the future. Friday evening local time I got to Mumbai. Chhattrapati Shivaji airport has improved a lot since I saw t last and now has all the amenities an international traveller needs including unrestricted free wifi (Zurich airport are you taking notes?) But here my first ominous piece of bad luck began. No sign of my suitcase. Happily some asking around revealed that it had somehow gotten on some earlier Swiss Air flight instead of the one I was on and was actually waiting for me. I got outside and Debian Developer Praveen Arimbrathodiyil was waiting to pick me up. Normally I don't lke staying in Mumbai very much even though I have relatives there but that's because we usually went during July-August the monsoon season when Mumbai reverts back to the swampy archipelago it was originally built on. This time the weather was nice, cold by local standards, but lovely and spring-like to someone from snowy New Jersey. There have been a lot of improvements to the road infrastructure and people are actually obeying the traffic laws. (Within reason of course. Whether or not a family of six can arrange themselves on one Bajaj scooter is no business of the cops.) The Hotel Tuliip (yes, two i's. Manager didn't know why.) Residency where I was to stay while not quite a five star establishment was adequate for my needs with a bed, hot water shower, and air conditioning. And a TV which to the bellhops great confusion I did not want turned on. (He asked about five times.) There was no Internet access per se but the manager offered to hook up a wireless router to a cable. Which on closer inspection turned out to have been severed at the base. He assured me it would be fixed tomorrow so I didn't complain and decided to do something more productive thank checking my email like sleeping. The next day I woke up in total darkness. Apparently there had been some kind of power problem during the night which tripped a fuse or something. A call to the front desk got them to fix that and then the second piece of bad luck happened. I plugged my Thinkpad in and woke it up from hibernation and a minute later there was a loud pop from the power adapter. Note I have a travel international plug adapter with surge protector so nothing bad ought to have happened but the laptop would on turning on display the message "critical low battery error" and immediately power off. I was unable to google what that meant without Internet access but I decided not to panic and continue getting ready. I would have plenty of opportunity to troubleshoot at the conference venue. Or so I thought... I took an autorickshaw to IIT. There also there have been positive improvements. Being quite obviously a foreigner I was fully prepared to be taken along the "scenic route." But now there are fair zones and the rickshaws all have (tamperproof!) digital fare meters so I was deposited at the main gate without fuss. After reading a board with a scary list of dos and don'ts I presented myself at security only to be inexplicably waved through without a second glance. Later I found out they've abandoned all the security theatre but not got around to updating the signs yet. Mumbai is one of the biggest, densely populated cities in the world but the IIT campus is an oasis of tranquility on the shores of Lake Powai. It's a lot bigger than it looked on the map so I had to wander around a bit before I reached the conference venue but I did make for the official registration time. Registration I was happy to meet several old friends (Such as Kartik Mistry and Kumar Appiah who along with Praveen and myself were the other DDs there,) people who I've corresponded with but never met, and many new people. I'm told 200+ people registered altogether. Most seemed to be students from IIT and elsewhere in Mumbai but there were also some Debian enthusiasts from further afield and most hearteningly some "civilians" who wanted to know what this was all about. With the help of a borrowed Thinkpad adapter I got my laptop running again. (Thankfully, despite the error message, the battery itself was unharmed.) However, my streak of bad luck was not yet over. It was that very weekend that IIT had a freak campus-wide network outage something that had never happened before. And as the presentation for the talk I was to give had apparently been open when I hibernated my laptop the night before, the sudden forced shutdown had trashed the file. (ls showed it as 0 length. An fsck didn't help.) I possibly had a backup on my server but with no Internet access I had no way to retrieve it. I still remained cool. The talk was scheduled for the second day so I could recover it at the hotel. Keynotes Professor Kannan Maudgalya of the FOSSEE (Free and Open Source Software for Education) Project which is part of the central government Ministry for Human Resource Development spoke about various activities of his project. Of particular interest to us are: FOSSEE is well funded, backed by the government and has enthusiastic staff so we should be seeing a lot more from them in the future. Veteran Free Software activist Venky Hariharan spoke about his experiences in lobbying the government on tech issues. He noted that there has been a sea change in attitudes towards Linux and Open source in the bureacracy of late. Several states have been aggressively mandating the use of it as have several national ministries and agencies. We the community can provide a valuable service by helping them in the transition. They also need to be educated on how to work with the community (contributing changes back, not working behind closed doors etc.) Debian History and Debian Cycle Shirish Agarwal spoke about the Debian philosophy and foundational documents such as the social contract and DFSG and how the release cycle works. Nothing new to an experienced user but informative to the newcomers in the audience and sparked some questions and discussion. Keysigning One of my main missions in attending was to help get as many isolated people as possible into the web of trust. Unfortunately the keysigning was not adequately publicized and few people were ready. I would have led them through the process of creating a new key there and then but with the lack of connectivity that idea had to be abandoned. I did manage to sign about 8-10 keys during other times. Future Directions for Debian-IN BOF I led this one. Lots of spirited discussion and I found feedback from new users in particular to be very helpful. Some take aways are: Lil' Debi Kumar Sukhani was a Debian GSoC student and his project which he demonstrated was to be able to install Debian on an Android phone. Why would you want to do this? Apart from the evergreen "Because I can", you can run server software such as sshd on your phone or even use it as an ARM development board. Unfortunately my phone uses Blackberry 10 OS which can run android apps (emulated under QNX) but wouldn't be able to use this. When I get a real Android phone I will try it out. Debian on ARM Siji Sunny gave this talk which was geared more towards hardware types which I am not but one thing I learned was thee difference between all the different ARM subarchitectures. I knew Siji first from a previous incarnation when he worked at CDAC with the late and much lamented Prof. R.K. Joshi. We had a long conversation about those days. Prof. Joshi/CDAC had developed an Indic rendering system called Indix which alas became the Betamax to Pango's VHS but he was also very involved in other Indic computing issues such as working with the Unicode Consortium and the preseration of Sanskrit manuscripts which is also an interest of mine. One good thing that cameout of Indix was some rather nice fonts. I had thought they were still buried in the dungeons of CDAC but apparently they were freed at one point. That's one more thing for me to look into. Evening/Next morning< My cousin met me and we had a leisurely dinner together. It was quite late by the time I got back to the hotel. FOSSEE had kindly lent me one of their tablets (which incidently are powerful enough to run LibreOffice comfortably.) so I thought I might be able to quickly redo my presentation before bedtime. Well, wouldn't you know it the wifi was not fixed. As I should have guessed but all the progress I'd had made me giddily optimistic. There was an option of trying to find an Internet cafe in a commercial area 15-20 minutes walk away. If this had been Gujarat I would have tried it but although I can more or less understand Hindi I can barely put together two sentences and Marathi I don't know at all. So I gave up that idea. I redid the slides from memory as best I could and went to sleep. In the morning I checked out and ferried myself and my suitcase via rickshaw back to the IIT campus. This time I got the driver to take me all the way in to the conference venue. Prof. Maudgalya kindly offered to let me keep the tablet to develop stuff on. I respectfully had to decline because although I love to collect bits of tech the fact it is it would have just gathered dust and ought to go to someone who can make a real contribution with it. I transferred my files to a USB key and borrowed a loaner laptop for my talk. Debian Packaging Workshop While waiting to do my talk I sat in on a workshop Praveen ran taking participants through the whole process of creating a Debian package (a ruby gem was the example.) He's done this before so it was a good presentation and well attended but the lack of connectivity did put a damper on things. Ask Me Anything It turned out the schedule had to be shuffled a bit so my talk was moved later from the announced time. A few people had already showed up so I took some random questions about Debian from them instead. GNOME Shell Accessibility With Orca Krishnakant Mane is remarkable. Although he is blind, he is a developer and a major contributor to Open Source projects. He talked about the Accessibility features of GNOME and compared them (favorably I might add) with proprietary screen readers. Not a subject that's directly useful to me but I found it interesting nonetheless. Rust: The memory safe language Manish Goregaokar talked about one of the new fad programming languages that have gotten a lot of buzz lately. This one is backed by Mozilla and it's interesting enough but I'll stick with C++ and Perl until one of the new ones "wins." Building a Mail Server With Debian Finally I got to give my talk and, yup, the video out on my borrowed laptop was incompatible with the projector. A slight delay to transfer everything to another laptop and I was able to begin. I talked about setting up BIND, postfix, and of course dovecot along with spamassassin, clamav etc. It turned out I had more than enough material and I went atleast 30 minutes over time and even then I had to rush at the end. People said they liked it so I'm happy. The End I gave the concluding remarks. Various people were thanked (including myself) mementos were given and pictures were taken. Despite a few mishaps I enjoyed myself and I am glad I attended. The level of enthusiasm was very high and lessons were learned so the next Debian-IN event should be even better. My departing flight wasn't due to leave until 1:20AM so I killed a few hours with my family before the flight. Once again I was stopping in Zurich, this time for most of a day. The last of my blunders was not to take my coat out of my suitcase and the temperature outside was 29F so I had to spend that whole time enjoing the (not so) many charms of Zurich airport. Atleast the second flight took me to Newark instead of JFK so I was able to get home a little earlier on Monday evening, exhausted but happy I made the trip.

22 January 2015

Erich Schubert: Year 2014 in Review as Seen by a Trend Detection System

We ran our trend detection tool Signi-Trend (published at KDD 2014) on news articles collected for the year 2014. We removed the category of financial news, which is overrepresented in the data set. Below are the (described) results, from the top 50 trends (I will push the raw result to appspot if possible due to file limits).
I have highlighted the top 10 trends in bold, but otherwise ordered them chronologically.
Updated: due to an error in a regexp, I had filtered out too many stories. The new results use more articles.

January
2014-01-29: Obama's state of the union address
February
2014-02-07: Sochi Olympics gay rights protests
2014-02-08: Sochi Olympics first results
2014-02-19: Violence in Ukraine and Maidan in Kiev
2014-02-20: Wall street reaction to Facebook buying WhatsApp
2014-02-22: Yanukovich leaves Kiev
2014-02-28: Crimea crisis begins
March
2014-03-01: Crimea crisis escalates futher
2014-03-02: NATO meeting on Crimea crisis
2014-03-04: Obama presents U.S. fiscal budget 2015 plan
2014-03-08: Malaysia Airlines MH-370 missing in South China Sea
2014-03-08: MH-370: many Chinese on board of missing airplane
2014-03-15: Crimean status referencum (upcoming)
2014-03-18: Crimea now considered part of Russia by Putin
2014-03-21: Russian stocks fall after U.S. sanctions.
April
2014-04-02: Chile quake and tsunami warning
2014-04-09: False positive? experience + views
2014-04-13: Pro-russian rebels in Ukraine's Sloviansk
2014-04-17: Russia-Ukraine crisis continues
2014-04-22: French deficit reduction plan pressure
2014-04-28: Soccer World Cup coverage: team lineups
May
2014-05-14: MERS reports in Florida, U.S.
2014-05-23: Russia feels sanctions impact
2014-05-25: EU elections
June
2014-06-06: World cup coverage
2014-06-13: Islamic state Camp Speicher massacre in Iraq
2014-06-14: Soccer world cup: Spain surprisingly destoyed by Netherlands
July
2014-07-05: Soccer world cup quarter finals
2014-07-17: Malaysian Airlines MH-17 shot down over Ukraine
2014-07-18: Russian blamed for 298 dead in airline downing
2014-07-19: Independent crash site investigation demanded
2014-07-20: Israel shelling Gaza causes 40+ casualties in a day
August
2014-08-07: Russia bans food imports from EU and U.S.
2014-08-08: Obama orders targeted air strikes in Iraq
2014-08-20: IS murders journalist James Foley, air strikes continue
2014-08-30: EU increases sanctions against Russia
September
2014-09-05: NATO summit with respect to IS and Ukraine conflict
2014-09-11: Scottish referendum upcoming - poll results are close
2014-09-23: U.N. on legality of U.S. air strikes in Syria against IS
2014-09-26: Star manager Bill Gross leaves Allianz/PIMCO for Janus
October
2014-10-22: Ottawa parliament shooting
2014-10-26: EU banking review
November
2014-11-05: U.S. Senate and governor elections
2014-11-12: Foreign exchange manipulation investigation results
2014-11-17: Japan recession
December
2014-12-11: CIA prisoner and U.S. torture centers revieled
2014-12-15: Sydney cafe hostage siege
2014-12-17: U.S. and Cuba relations improve unexpectedly
2014-12-18: Putin criticizes NATO, U.S., Kiev
2014-12-28: AirAsia flight QZ-8501 missing

As you can guess, we are really happy with this result - just like the result for 2013 it mentiones (almost) all the key events.
There probably is one "false positive" there: 2014-04-09 has a lot of articles talking about "experience" and "views", but not all refer to the same topic (we did not do topic modeling yet).
There are also some events missing that we would have liked to appear; many of these barely did not make it into the top 50, but do appear in the top 100, such as the Sony cyberattack (#51) and the Fergusson riots on November 11 (#66).
You can also explore the results online in a snapshot.

Diego Escalante Urrelo: Link Pack #04

Writing Your Way to Happiness (nytimes.com)
Researches believe that the way we think about, and remember, our story can be so powerful that it can actually influence our happiness and success. It s a nice little article summarizing actual research. The main study referred put fresh university students to test: a group received tools to rewrite their memory and story of their academic performance, another group didn t. The first group improved their grades and had only 1 student drop school within a year, the other group had 4 drop outs and no specific improvement. I ve been thinking about this as I recently rewrote my About page and also started writing down some past Travel journals. Looking back and rewriting your own story is incredibly empowering, it s a fantastic rush of confidence and self-assertion. Memory is always betraying us, and remembering our success is not particularly high on the list of things to keep.
The concept is based on the idea that we all have a personal narrative that shapes our view of the world and ourselves. But sometimes our inner voice doesn t get it completely right. Some researchers believe that by writing and then editing our own stories, we can change our perceptions of ourselves and identify obstacles that stand in the way of better health. It may sound like self-help nonsense, but research suggests the effects are real. Students who had been prompted to change their personal stories improved their grade-point averages and were less likely to drop out over the next year than the students who received no information. In the control group, which had received no advice about grades, 20 percent of the students had dropped out within a year. But in the intervention group, only 1 student or just 5 percent dropped out.
Old Masters at the Top of Their Game (nytimes.com)
Fantastic read on how these artists defy the conventions of old meaning useless. Masters at their art, they haven t quit nor have laid to rest and cash their reputation. They keep making, they stay alive (physically and metaphorically) through art. No rush to get to their age, but still a really interesting letter from the future . Full of cheat codes, read this now.
Now I am 79. I ve written many hundreds of essays, 10 times that number of misbegotten drafts both early and late, and I begin to understand that failure is its own reward. It is in the effort to close the distance between the work imagined and the work achieved wherein it is to be found that the ceaseless labor is the freedom of play, that what s at stake isn t a reflection in the mirror of fame but the escape from the prison of the self. T. H. White, the British naturalist turned novelist to write The Once and Future King, calls upon the druid Merlyn to teach the lesson to the young prince Arthur: You may grow old and trembling in your anatomies, you may lie awake at night listening to the disorder of your veins, you may miss your only love, you may see the world about you devastated by evil lunatics, or know your honour trampled in the sewers of baser minds. There is only one thing for it then to learn. Learn why the world wags and what wags it. That is the only thing which the mind can never exhaust, never alienate, never be tortured by, never fear or distrust, and never dream of regretting.
A Life with a View (ribbonfarm.com)
A somewhat tricky read, but with a nice payback. Take your time, and savor it slowly. It s a very interesting look into how we keep wanting new stuff, and how we shield from ourselves by looking for the place with no yearns , the place where we won t want anything anymore doesn t exist. Chains very well into the reads I shared a few days ago on practical contentment.
The arrival fallacy is about seeking a life from which one can look with a complacent equanimity upon the rest of reality, without yearning. It is an ideal of a life that is defined primarily by blindness to itself. You yearn while you see your life as others see it, until you arrive at a situation where you can disappear into the broader background, and see comfortably without being seen discomfittingly, especially by yourself. Once you re there, the yearning stops, so the theory goes. Of course it is a laughably bad theory.
How To Escape From A Moving Car (mrporter.com)
By Adam Kirley, stunt double for Daniel Craig in the crazy crane scene of Casino Royale (where 007 jumps from monkey nuts high to donkey bonkers high, a badger bum crazy distance). Really funny, and one of those things I always find myself thinking Almost as much as what to do in case of a Post Office Showdown (xkcd.com)
Everyone s first instinct is to put their hands or legs down first. That s the worst thing you can do: you will break something. The pointy parts of your body hurt elbows, knees, hips, ankles. Put your fists under your chin, and bring your elbows together. Keep your chin tucked in to your chest to protect your head. The best point of impact is the back of the shoulder and your back. If you dive out directly onto your shoulder you ll break it.
What the World Looks Like with Social Anxiety (collegehumor.com)
Funny vignettes about how the world looks like when you are socially anxious. I can only really identify with the last one:
cfd04d22a6dfa4fb858dee8d3d5592afShea Strauss.
Helsinki Bus Station Theory (fotocommunity.com)
Don t get off the bus. Art comes to those who wait and persevere. At first, you replicate the same route others have done, but only if you stay long enough in such path you begin to find your own path. Although perhaps a little more classic in conception, this is an interesting text advising artists to don t give up just because they don t compare well to the masters of their current art or genre. Only those who persevere will catch up and diverge from the masters. You could say that diverging early is also a way to find your path, but there s still a case to be made for learning from those who came before. Whether you want to imitate them, or rebel against them, you still need to know them. My take: it doesn t hurt to pick up some biographies or works from past masters and see what made them masters. Create your master genealogy, kinda like in Steal Like an Artist (which I recently read but haven t got around to write about yet).
Georges Braque has said that out of limited means, new forms emerge. I say, we find out what we will do by knowing what we will not do. And so, if your heart is set on 8 10 platinum landscapes in misty southern terrains, work your way through those who inspire you, ride their bus route and damn those who would say you are merely repeating what has been done before. Wait for the months and years to pass and soon your differences will begin to appear with clarity and intelligence, when your originality will become visible, even the works from those very first years of trepidation when everything you did seemed so done before.
At 90, She s Designing Tech For Aging Boomers (npr.org)
The inspiring tale of a 90 year old woman who joined IDEO to contribute a unique point of view to the design process. You can never stop learning, life never ceases to be interesting. It s short, and not incredibly shocking, but that this has happened somewhere as referenced and revered as IDEO says a lot.
And for the bulging demographic of baby boomers growing old, Beskind has this advice: Embrace change and design for it.

Previously on Link Pack

21 January 2015

Enrico Zini: miniscreen

Playing with python, terminfo and command output I am experimenting with showing progress on the terminal for a subcommand that is being run, showing what is happening without scrolling away the output of the main program, and I came out with this little toy. It shows the last X lines of a subcommand output, then gets rid of everything after the command has ended. Usability-wise, it feels like a tease to me: it looks like I'm being shown all sorts of information then they are taken away from me before I managed to make sense of them. However, I find it cute enough to share:
#!/usr/bin/env python3
#coding: utf-8
# Copyright 2015 Enrico Zini <enrico@enricozini.org>.  Licensed under the terms
# of the GNU General Public License, version 2 or any later version.
import argparse
import fcntl
import select
import curses
import contextlib
import subprocess
import os
import sys
import collections
import shlex
import shutil
import logging
def stream_output(proc):
    """
    Take a subprocess.Popen object and generate its output, line by line,
    annotated with "stdout" or "stderr". At process termination it generates
    one last element: ("result", return_code) with the return code of the
    process.
    """
    fds = [proc.stdout, proc.stderr]
    bufs = [b"", b""]
    types = ["stdout", "stderr"]
    # Set both pipes as non-blocking
    for fd in fds:
        fcntl.fcntl(fd, fcntl.F_SETFL, os.O_NONBLOCK)
    # Multiplex stdout and stderr with different prefixes
    while len(fds) > 0:
        s = select.select(fds, (), ())
        for fd in s[0]:
            idx = fds.index(fd)
            buf = fd.read()
            if len(buf) == 0:
                fds.pop(idx)
                if len(bufs[idx]) != 0:
                    yield types[idx], bufs.pop(idx)
                types.pop(idx)
            else:
                bufs[idx] += buf
                lines = bufs[idx].split(b"\n")
                bufs[idx] = lines.pop()
                for l in lines:
                    yield types[idx], l
    res = proc.wait()
    yield "result", res
@contextlib.contextmanager
def miniscreen(has_fancyterm, name, maxlines=3, silent=False):
    """
    Show the output of a process scrolling in a portion of the screen.

    has_fancyterm: true if the terminal supports fancy features; if false, just
    write lines to standard output

    name: name of the process being run, to use as a header

    maxlines: maximum height of the miniscreen

    silent: do nothing whatsoever, used to disable this without needing to
            change the code structure

    Usage:
        with miniscreen(True, "my process", 5) as print_line:
            for i in range(10):
                print_line(("stdout", "stderr")[i % 2], "Line # ".format(i))
    """
    if not silent and has_fancyterm:
        # Discover all the terminal control sequences that we need
        output_normal = str(curses.tigetstr("sgr0"), "ascii")
        output_up = str(curses.tigetstr("cuu1"), "ascii")
        output_clreol = str(curses.tigetstr("el"), "ascii")
        cols, lines = shutil.get_terminal_size()
        output_width = cols
        fg_color = (curses.tigetstr("setaf") or
                    curses.tigetstr("setf") or "")
        sys.stdout.write(str(curses.tparm(fg_color, 6), "ascii"))
        output_lines = collections.deque(maxlen=maxlines)
        def print_lines():
            """
            Print the lines in our buffer, then move back to the beginning
            """
            sys.stdout.write("  progress:".format(name))
            sys.stdout.write(output_clreol)
            for msg in output_lines:
                sys.stdout.write("\n")
                sys.stdout.write(msg)
                sys.stdout.write(output_clreol)
            sys.stdout.write(output_up * len(output_lines))
            sys.stdout.write("\r")
        try:
            print_lines()
            def _progress_line(type, line):
                """
                Print a new line to the miniscreen
                """
                # Add the new line to our output buffer
                msg = "   ".format("." if type == "stdout" else "!", line)
                if len(msg) > output_width - 4:
                    msg = msg[:output_width - 4] + "..."
                output_lines.append(msg)
                # Update the miniscreen
                print_lines()
            yield _progress_line
            # Clear the miniscreen by filling our ring buffer with empty lines
            # then printing them out
            for i in range(maxlines):
                output_lines.append("")
            print_lines()
        finally:
            sys.stdout.write(output_normal)
    elif not silent:
        def _progress_line(type, line):
            print(" :  ".format(type, line))
        yield _progress_line
    else:
        def _progress_line(type, line):
            pass
        yield _progress_line
def run_command_fancy(name, cmd, env=None, logfd=None, fancy=True, debug=False):
    quoted_cmd = " ".join(shlex.quote(x) for x in cmd)
    log.info("%s running command %s", name, quoted_cmd)
    if logfd: print("runcmd:", quoted_cmd, file=logfd)
    # Run the script itself on an empty environment, so that what was
    # documented is exactly what was run
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
    with miniscreen(fancy, name, silent=debug) as progress:
        stderr = []
        for type, val in stream_output(proc):
            if type == "stdout":
                val = val.decode("utf-8")
                if logfd: print("stdout:", val, file=logfd)
                log.debug("%s stdout: %s", name, val)
                progress(type, val)
            elif type == "stderr":
                val = val.decode("utf-8")
                if logfd: print("stderr:", val, file=logfd)
                stderr.append(val)
                log.debug("%s stderr: %s", name, val)
                progress(type, val)
            elif type == "result":
                if logfd: print("retval:", val, file=logfd)
                log.debug("%s retval: %d", name, val)
                retval = val
    if retval != 0:
        lastlines = min(len(stderr), 5)
        log.error("%s exited with code %s", name, retval)
        log.error("Last %d lines of standard error:", lastlines)
        for line in stderr[-lastlines:]:
            log.error("%s: %s", name, line)
    return retval
parser = argparse.ArgumentParser(description="run a command showing only a portion of its output")
parser.add_argument("--logfile", action="store", help="specify a file where the full execution log will be written")
parser.add_argument("--debug", action="store_true", help="debugging output on the terminal")
parser.add_argument("--verbose", action="store_true", help="verbose output on the terminal")
parser.add_argument("command", nargs="*", help="command to run")
args = parser.parse_args()
if args.debug:
    loglevel = logging.DEBUG
elif args.verbose:
    loglevel = logging.INFO
else:
    loglevel = logging.WARN
logging.basicConfig(level=loglevel, stream=sys.stderr)
log = logging.getLogger()
fancy = False
if not args.debug and sys.stdout.isatty():
    curses.setupterm()
    if curses.tigetnum("colors") > 0:
        fancy = True
if args.logfile:
    logfd = open("output.log", "wt")
else:
    logfd = None
retval = run_command_fancy("miniscreen example", args.command, logfd=logfd)
sys.exit(retval)

Jo Shields: mono-project.com Linux packages, January 2015 edition

The latest version of Mono has released (actually, it happened a week ago, but it took me a while to get all sorts of exciting new features bug-checked and shipshape). Stable packages This release covers Mono 3.12, and MonoDevelop 5.7. These are built for all the same targets as last time, with a few caveats (MonoDevelop does not include F# or ASP.NET MVC 4 support). ARM packages will be added in a few weeks time, when I get the new ARM build farm working at Xamarin s Boston office. Ahead-of-time support This probably seems silly since upstream Mono has included it for years, but Mono on Debian has never shipped with AOT d mscorlib.dll or mcs.exe, for awkward package-management reasons. Mono 3.12 fixes this, and will AOT these assemblies optimized for your computer on installation. If you can suggest any other assemblies to add to the list, we now support a simple manifest structure so any assembly can be arbitrarily AOT d on installation. Goodbye Mozroots! I am very pleased to announce that as of this release, Mono users on Linux no longer need to run mozroots to get SSL working. A new command, cert-sync , has been added to this release, which synchronizes the Mono SSL certificate store against your OS certificate store and this tool has been integrated into the packaging system for all mono-project.com packages, so it is automatically used. Just make sure the ca-certificates-mono package is installed on Debian/Ubuntu (it s always bundled on RPM-based) to take advantage! It should be installed on fresh installs by default. If you want to invoke the tool manually (e.g. you installed via make install, not packages) use
cert-sync /path/to/ca-bundle.crt
On Debian systems, that s
cert-sync /etc/ssl/certs/ca-certificates.crt
and on Red Hat derivatives it s
cert-sync /etc/pki/tls/certs/ca-bundle.crt
Your distribution might use a different path, if it s not derived from one of those. Windows installer back from the dead Thanks to help from Alex Koeplinger, I ve brought the Windows installer back from the dead. The last release on the website was for 3.2.3 (it s actually not this version at all it s complicated ), so now the Windows installer has parity with the Linux and OSX versions. The Windows installer (should!) bundles everything the Mac version does F#, PCL facades, IronWhatever, etc, along with Boehm and SGen builds of the Mono runtime done with Visual Studio 2013. An EXPERIMENTAL OH MY GOD DON T USE THIS IN PRODUCTION 64-bit installer is in the works, when I have the time to try and make a 64-build of Gtk#.

18 January 2015

Jonathan Wiltshire: Alcester BSP, day three

We have had a rather more successful weekend then I feared, as you can see from our log on the wiki page. Steve reproduced and wrote patches for several installer/bootloader bugs, and Neil and I spent significant time in a maze of twist zope packages (we have managed to provide more diagnostics on the bug, even if we couldn t resolve it). Ben and Adam have ploughed through a mixture of bugs and maintenance work. I wrongly assumed we would only be able to touch a handful of bugs, since they are now mostly quite difficult, so it was rather pleasant to recap our progress this evening and see that it s not all bad after all.
Alcester BSP, day three is a post from: jwiltshire.org.uk Flattr

EvolvisForge blog: Debian/m68k hacking weekend cleanup

OK, time to clean up tarent so people can work again tomorrow. Not much to clean though (the participants were nice and cleaned up after themselves ), so it s mostly putting stuff back to where it belongs. Oh, and drinking more of the cool Belgian beer Geert (Linux upstream) brought We were productive, reporting and fixing kernel bugs, fixing hardware, swapping and partitioning discs, upgrading software, getting buildds (mostly Amiga) back to work, trying X11 (kdrive) on a bare metal Atari Falcon (and finding a window manager that works with it), etc. I hope someone else writes a report; for now we have a photo and a screenshot (made with trusty xwd). Watch the debian-68k mailing list archives for things to come. I think that, issues with electric cars aside, everyone liked the food places too ;-)

16 January 2015

Rapha&#235;l Hertzog: Freexian s fifth report about Debian Long Term Support

Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In December 46 work hours have been equally split among 4 paid contributors (note that Thorsten and Rapha l have actually spent more hours because they took over some hours that Holger did not do over the former months). Their reports are available: Evolution of the situation Compared to last month, the number of paid work hours has almost not increased (we are at 48 hours per month). We still have a couple of new sponsors in the pipe but with the new year they did not complete the process yet. Hopefully next month will see a noticeable increase. As usual, we are looking for more sponsors to reach our our minimal goal of funding the equivalent of a half-time position. Those of you who are struggling to spend money in the last quarter due to budget overrun, now is a good time to see if you want to include Debian LTS support in your 2015 budget! In terms of security updates waiting to be handled, the situation looks similar to last month: the dla-needed.txt file lists 30 packages awaiting an update (3 more than last month), the list of open vulnerabilities in Squeeze shows about 56 affected packages in total. We do not manage to clear the backlog but it s not getting significantly worse either. Thanks to our sponsors

No comment Liked this article? Click here. My blog is Flattr-enabled.

15 January 2015

Lunar: 80%

Unfortunately I could not go on stage at the 31st Chaos Communication Congress to present reproducible builds in Debian alongside Mike Perry from the Tor Project and Seth Schoen from the Electronic Frontier Foundation. I've tried to make it up for it, though and we have made amazing progress. Wiki reorganization What was a massive and frightening wiki page now looks really more welcoming: Screenshot of ReproducibleBuilds on Debian wiki Depending on what one is looking for, it should be much easier to find. There's now a high-level status overview given on the landing page, maintainers can learn how to make their packages reproducible, enthusiasts can more easily find what can help the project, and we have even started writing some history. .buildinfo for all packages New year's eve saw me hacking Perl to write dpkg-genbuildinfo. Similar to dpkg-genchanges, it's run by dpkg-buildpackage to produce .buildinfo control files. This is where the build environment, and hash of source and binary packages are recorded. This script, integrated with dpkg, replace the previous debhelper interim solution written by Niko Tyni. We used to fix mtimes in control.tar and data.tar using a specific addition to debhelper named dh_fixmtimes. To better support the ALWAYS_EXCLUDE environment variable and for pragramtic reasons, we moved the process in dh_builddeb. Both changes were quickly pushed to our continuous integration platform. Before, only packages using dh would create a .buildinfo and thus eventually be considered reproducible. With these modifications, many more packages had their chance and this shows: Growing amount of packages considered reproducible Yes, with our experimental toolchain we are now at more than eighty percent! That's more than 17200 source packages! srebuild Another big item on the todo-list was crossed over by Johannes Schauer. srebuild is a wrapper around sbuild:
Given a .buildinfo file, it first finds a timestamp of Debian Sid from snapshot.debian.org which contains the requested packages in their exact versions. It then runs sbuild with the right architecture as given by the .buildinfo file and the right base system to upgrade from, as given by the version of the base-files package version in the .buildinfo file. Using two hooks it will install the right package versions and verify that the installed packages are in the right version before the build starts.
Understanding problems Over 1700 packages have now been reviewed to understand why build results could not be reproduced on our experimental platform. The variations between the two builds are currently limited to time and file ordering, but this still has uncovered many problems. There are still toolchain fixes to be made (more than 180 packages for the PHP registry) which can make many packages reproducible at once, but others like C pre-processor macros will require many individual changes. debbindiff, the main tool used to understand differences, has gained support for .udeb, TrueType and OpenType fonts, PNG and PDF files. It's less likely to crash on problems with encoding or external tool. But most importantly for large package, it has been made a lot faster, thanks to Reiner Herrmann and Helmut Grohne. Helmut has also been able to spot cross-compilation issues by using debbindiff! Targeting our efforts It gives warm fuzzy feelings to hit the 80% mark, but it would be a bit irrelevant if this would not concern packages that matter. Thankfully, Holger worked on producing statistics for more specific package sets. Mattia Rizzolo has also done great work to improve the scripts generating the various pages visible on reproducible.debian.net. All essential and build-esential packages, except gcc and bash, are considered reproducible or have patches ready. After some lengthy builds, I also managed to come up with a patch to make linux build reproducibly. Miscellaneous After my initial attempt to modify r-base to remove a timestamp in R packages, Dirk Eddelbuettel discussed the issue with upstream and came up with a better patch. The latter has already been merged upstream! Dirk's solution is to allow timestamps to be set using an external environment variable. This is also how I modified FontForge to make it possible to reproduce fonts. Identifiers generated by xsltproc have also been an issue. After reviewing my initial patch, Andrew Awyer came up with a much nicer solution. Its potential performance implications need to be evaluated before submission, though. Chris West has been working on packages built with Maven amongst other things. PDF generated by GhostScript, another painful source of troubles, is being worked on by Peter De Wachter. Holger got X.509 certificates signed by the CA cartel for jenkins.debian.net and reproducible.debian.net. No more scary security messages now. Let's hope next year we will be able to get certificates through Let's Encrypt! Let's make a difference together As you can imagine with all that happened in the past weeks, the #debian-reproducible IRC channel has been a cool place to hang out. It's very energizing to get together and share contributions, exchange tips and discuss hardest points. Mandatory quote:
* h01ger is very happy to see again and again how this is a nice
         learning circle...! i've learned a whole lot here too... in
         just 3 months... and its going on...!
Reproducible builds are not going to change anything for most of our users. They simply don't care how they get software on their computer. But they care to get the right software without having to worry about it. That's our responsibility, as developers. Enabling users to trust their software is important and a major contribution, we as Debian, can make to the wider free software movement. Once Jessie is released, we should make a collective effort to make reproducible builds an highlight of our next release.

14 January 2015

Ritesh Raj Sarraf: apt-offline 1.6

I am pleased to announce the release of apt-offline - 1.6 This release is mostly a bug fix release, which every user should upgrade to. It also fixes a major bug in the way we limited the validation of GPG integrity, for the APT repository lists (Thank you Paul Wise). Also, In the last release, we migrated from custom magic library to the python shipped ctype python-magic library. That allowed some bugs to creep, and hopefully now, all those bugs should be fixed. A big thanks to Roland Summers for his bug reports and continuous feedback. What is apt-offline ?
Description-en: offline APT package manager
 apt-offline is an Offline APT Package Manager.
 .
 apt-offline can fully update and upgrade an APT based distribution without
 connecting to the network, all of it transparent to APT.
 .
 apt-offline can be used to generate a signature on a machine (with no network).
 This signature contains all download information required for the APT database
 system. This signature file can be used on another machine connected to the
 internet (which need not be a Debian box and can even be running windows) to
 download the updates.
 The downloaded data will contain all updates in a format understood by APT and
 this data can be used by apt-offline to update the non-networked machine.
 .
 apt-offline can also fetch bug reports and make them available offline.
Debian changelog for the 1.6 release.
apt-offline (1.6) experimental; urgency=medium
  * [2a4a7f1] Don't abuse exception handlers.
    Thanks to R-Sommer
  * [afc51b3] MIME type for a deb package.
    Thanks to R-Sommer
  * [ec2d539] Also include debian-archive-keyring.
    Thanks to Hans-Christoph Steiner (Closes: #748082)
  * [dc602ac] Update MIME type for .gpg
  * [c4f9b71] Cycle through possible apt keyrings.
    Thanks to Paul Wise (Closes: #747163)
  * [de0fe4d] Clarify manpage for install
  * [b5e1075] Update manpage with some doc about argparse positional
    values to arguments
  * [c22d64d] Port is data type integer.
    Thanks to Roland Sommer
  * [67edebe] autodetect release name
  * [5803141] Disable python-apt support
 -- Ritesh Raj Sarraf <rrs@debian.org>  Wed, 14 Jan 2015 15:34:45 +0530
[1] https://alioth.debian.org/projects/apt-offline/

Categories:

Keywords:

Next.

Previous.