Search Results: "tar"

20 June 2011

NeuroDebian: Package swarm

Package swarm We are working on integrating neuroscience research software into the Debian operating system for almost six years now. Occasionally looking at the repository web server logs, we knew that a growing number of people were using the packages from http://neuro.debian.net for their Debian and Ubuntu machines. However, we never really looked deeper. For our upcoming booth at HBM2011 we need something like a presentation or slides that could run in a loop on a big screen to attract people. It would need to be visually appealing, but also informative. As slides are rather boring, we looked into creating an animation a visualization of our repository download statistics. There are many boring ways to visualize this type of data, but there is also at least one that is cool enough to yield an appealing, almost mystical, animation: code_swarm. This great software can visualize somewhat social processes based on commit statistics of source code repositories. It extracts information on who is working on which files of a project at any given point in time, and visualizes similarities in these patterns. People that work on the same files in temporal proximity cluster together, and files that are modified by the same people cluster together. From all this information code_swarm generates beautiful animations. Download statistics are not that different from commit statistics. Instead of Who is modifying what? it is simply: Who is downloading what? So we gathered all Apache logs of binary package downloads from our repository over the last two years. We converted the data so that the corresponding source package, became the author of a commit/download, and encoded the associated IPv4 address (the last 8 bit truncated) together with the target distribution as the modified file . And code_swarm did the rest (try watching in 480p or above): <iframe class="youtube-player" frameborder="0" height="375" src="http://www.youtube.com/embed/dp42rrkrNBI?hd=1" title="YouTube video player" type="text/html" width="640"></iframe>Few notes on the video: The animation shows some spikes in the download activity. Most of the big ones are related to new releases of FSL. FSL is undoubtedly our most popular package. First and foremost, because it is one of the standard tools for brain-imaging research, but also, because upstream completely relies on NeuroDebian for deploying FSL on the Debian platform. It looks like the majority of our downloads comes from countless Ubuntu machines maybe laptops and desktops with dynamic IPs. However, there are also fairly/large deployments of Debian 6.0 (white) and Ubuntu 10.04 (light blue) with static addresses probably labs at research institutions. Watch for the large stars towards the end. You might have noticed the yellow dots. These are downloads from the data suite of the NeuroDebian which we use to improve space-efficiency of our repository. This suite has packages (sometimes the size of 1GB in compressed form) that can be installed on any Debian or Ubuntu release and contain only data of various types and formats. Of course many applications depend on these packages, hence you can see some dots constantly changing color between yellow and one of the release colors. PS: OpenShot became an amazing video editor!

Mohammed Sameer: Introducing vcamera, a virtual V4L2 camera.

This is a kernel module I cooked in a couple of days. The idea is to expose a v4l device that gets its data from user space.

I had 2 use cases in mind:
1) Educational purpose for myself (I'm really a kernel noob).
2) Streaming movies over skype, google talk, ... etc.

The idea could be good or completely rubbish but hey ? Learning can only be done with stupid ideas!

The code is highly unstable. It shouldn't oops the kernel but I'm not responsible. I've been developing and testing it inside qemu.

Clone it from the git repository via:

git clone git@gitorious.org:vcamera/vcamera.git

Here are a few missing bits off the top of my head:

* I'm not following the kernel coding style yet ;-)
* I'm sure my locking, unlocking and concurrency handling is flawed.
* The code is a bit fragile.
* It'd be nice to implement mmap support for the character device. This should eliminate data copies.
* Perhaps expose the character device all the time and generate "fake" frames when streaming starts ? Problem now is one has to be very fast in feeding data to the module otherwise select() on the v4l device will timeout.
* Many more...

If someone finds this idea useful, please drop me a line.
Comments, use cases, ideas and tips are really welcomed!

If I see a lot of interest, I might try to push it to the kernel tree one day ;-)

Update: Seems vloopback already exists and renders my code useless. I might still do something with it as my idea seems a bit simpler but whatever.

Update 2: There's also v4l2loopback and its fork. read more

Joachim Breitner: gtk-vector-screenshot code published

After my post about the vector screenshots two days ago stirred quite some interest, I was hard-pressed to push the code out in proper form. So here you are: You can get release tarballs which can be installed with the usual configure && make && sudo make install. You can also get the code via git from the gitorious project page. Freshmeat and ohloh registration will follow soon. I don t think though that I need more infrastructure, e.g. a mailing list would be overkill, unless the project becomes much larger than I expect at the moment. After all, I do prefer small projects.
Over the version that you saw in the screencast two days ago (which almost broke my server, as I included the video via the <video> tag, and every reader of my blog entry automatically started downloading the 2MB file), I have added support for SVG, PostScript and (hear hear) PNG, made the button much more visible and improved minor usability things. Since Saturday, the work was covered in ycombinator s hacker news and on golem.de. If the interest stays high and I get positive feedback from users of the above code, I ll also upload it to Debian. I would really like to have this as a stand-alone program that can take screenshots of any running gtk-3 application, but I m not sure if this is possible in a clean fashion. Can I somehow hook into a running glib application and call object methods there, maybe similar to how gdb would do it? Alternatively, I could install the module system wide, so that it is loaded into every application, and then somehow triggered (e.g. using a signalling mechanism based on X window s property). Ideas and patches welcome!

Craig Small: psmisc 22.14 Released

The upstream and debian packages for psmisc version 22.14 were released tonight. This version has a lot of bugfixes. I ve also taken some time to clean up some of the minor compile warnings I used to get which were mainly variables set but not used. Two bugs that I ll concentrate on getting fixed for the next version are all about the command line. The first is that the fuser -s flag changes the return code for items found. It should be consistent so it doesnt matter if fuser displays values or not, 0 means found something and 1 means didn t find it. I can see why in the code it might be different, because the code follows different paths with the s option, but not exactly where the problem is. The second bug is the -M flag for fuser only works if it is before the mount points so -M -m /foo works but -m /foo -M does not. Unlike the -s problem, I can see where it is failing. Currently fuser scans the options one by one. If it finds a -m option, it checks at that point if the -M option is set. The solution is to store all potential targets and only after exhausting the entire command line do we add the potential targets to our real targets. It means scanning the command line, or a subset of it, twice but the performance penalty is tiny. Finally, I released there was a rather neat Makefile command make distcheck . This target makes a tar.g archive, extracts it, builds then cleans. It is a very good program for trapping things like a file exists in your local directory but is not included in the tar.gz (lists.h was missing until I ran this). For some make distcheck is something they re known for years, but if you have an autotools package you develop, have a look at it. With all the talk about testing and test driven releases, I was going to put some test cases in. The problem is there doesn t seem to be an easy way to do testing. I might have to just hack up some scripts but surely there is a better way. That would trap, for example that fuser -s returns 1 but fuser returns 0. I looked at check, but it doesn t work for single-file programs like psmisc is mainly made of.

Petter Reinholdtsen: Free Software vs. proprietary softare...

Reading the thingiverse blog, I came across two highlights of interesting parts of the Autodesk and Microsoft Kinect End User License Agreements (EULAs), which illustrates quite well why I stay away from software with EULAs. Whenever I take the time to read their content, the terms are simply unacceptable.

Gerfried Fuchs: Games Team IRC Meeting #4

Another month, another Games Team IRC Meeting happening. This time it was decided to have it again on Sunday, the time was set to 10 am UTC. To find out the time in your localtime, issue date -d '2011-06-26 10:00 UTC' in your shell. The agenda can be seen as always in the wiki. If the time or agenda doesn't fit your ideas, feel free to join our mailinglist to be informed about the discussion of agenda and time for the next meeting and raise your voice at that time. Please notice that the agenda isn't final yet, you can still drop your ideas for that. Enjoy, and join if you care about improving games packaging in Debian and influence future development!

/debian permanent link Comments: 0 Flattr this

Rapha&#235;l Hertzog: apt-get, aptitude, pick the right Debian package manager for you

This is a frequently asked question: What package manager shall I use? . And my answer is the one that suits your needs . In my case, I even use different package managers depending on what I m trying to do. APT vs dpkg, which one is the package manager? In the Debian world, we re usually thinking of APT-based software when we re referring to a package manager . But in truth, the real package manager is dpkg. It s the low-level tool that takes a .deb file and extracts its content on the disk, or that takes the name of a package to remove the associated files, etc. APT is better known because it s the part of the packaging infrastructure that matters to the user. APT makes collection of software available to the user and does the dirty work of downloading all the required packages and installing them by calling dpkg in the correct order to respect the dependencies. But APT is not a simple program, it s a library and several different APT frontends have been developed on top of that library. The most widely known is apt-get since it s the oldest one, and it s provided by APT itself. Graphical APT front-ends update-manager is a simple frontend useful to install security updates and other trivial daily upgrades (if you re using testing or sid). It s the one that you get when you click in the desktop notification that tells you that updates are available. In cases, where the upgrade is too complicated for update-manager, it will suggest to run synaptic which is full featured package manager. You can browse the list on installed/available packages in numerous ways, you can mark packages for installation/upgrade/removal/purge and then run in one go all the recorded actions. software-center aims to be an easy to use application installer, it will hide most of the packaging details and will only present installed/available applications (as defined by a .desktop file). It s very user friendly and has been developed by Ubuntu. Of the graphical front-ends, I use mainly synaptic and only when I m reviewing what I have installed to trim the system down. Console-based GUI APT front-ends In this category, I ll cite only aptitude. Run without parameter, it will start a powerful console-based GUI. Much like synaptic, you can have multiple views of the installed/available packages and mark packages for installation/upgrade/removal/purge before executing everything at once. Command-line based package managers and APT front-ends This is where the well known apt-get fits, but there are several other alternatives: aptitude, cupt, wajig. Wajig and cupt are special cases as they don t use libapt: the former wraps several tools including apt-get, and the latter is a (partial) APT reimplementation (versions 1.x were in Perl, 2.x are now is C++). You re welcome to try them out and find out which one you prefer, but I have never felt the need to use something else than apt-get and aptitude. apt-get or aptitude? First I want to make it clear that you can use both and mix them without problems. It used to be annoying when apt-get did not track which packages were automatically installed while aptitude did, but now that both packages share this list, there s no reason to avoid switching back and forth. I would recommend apt-get for the big upgrades (i.e. dist-upgrade from one stable to the next) because it will always find quickly a relatively good solution while aptitude can find several convoluted solutions (or none) and it s difficult to decide which one should be used. On the opposite for regular upgrades in unstable (or testing), I would recommend aptitude safe-upgrade . It does a better job than apt-get at keeping on hold packages which are temporarily broken due to some not yet finished changes while still installing new packages when required. With aptitude it s also possible to tweak dynamically the suggested operations while apt-get doesn t allow this. And aptitude s command line is probably more consistent: with apt-get you have to switch between apt-get and apt-cache depending on the operation that you want to do, aptitude on the other hand does everything by itself. Take some time to read their respective documentation and to try them.
Click here to subscribe to my free newsletter and get my monthly analysis on what s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

11 comments Liked this article? Click here. My blog is Flattr-enabled.

Nathan Handler: Google Summer of Code Report #2

A lot of work has taken place since my last report. First, since my original mentor, Matt Zimmerman, is going to be moving, he helped arrange for Stefano Zacchiroli to fill in for him temporarily. I have had a call with Stefano, and I am looking forward to getting to work with him more over the next couple of weeks. Shortly after my first report, we realized that having the dashboard only display bugs severely limited its usefulness. In our original design, we had it pulling a list of bugs with specific usertags from the BTS. This forced all information to be added to bugs, which we viewed as a good thing since it made the information available to the Debian Maintainers without requiring them to familiarize themselves with DEX. The issue was that when looking back at the ancient-patches project or forward to the big merges project, a large portion of the work will be triaging and determining whether or not the changes are even applicable in Debian. If they aren t, there is no reason for a bug to be filed on the BTS. As a result, I spent a fair amount of time re-writing most of the dashboard. It still supports finding bugs on the BTS that have been tagged with a specific DEX usertag, but it is now task-based rather than bug-based. A task can be a bug number, a package name, a patch name, or anything. It is merely something that needs to be done for a project. Each task will have a name, an assignee, a status, a note, and a bug associated with it. There are two ways to define a list of tasks for a project. The first is to tag bugs on the BTS with a user of debian-derivatives@lists.debian.org and a tag with a format of dex-<distribution>- <project> (i.e. dex-ubuntu-ancient-patches). The script will convert the bugs to tasks and automatically set all of the fields. The second method is to specify the list of tasks in a plain text file on the server the scripts are being run on. Eventually, this list will be able to be specified via a web interface. Currently, you are only able to specify the task name in the file. This means that all of the other fields for a task created in this manner are initially blank. I am still deciding on the best way to specify additional information about each task in the file. Once I decide on a method, it should not be too difficult to modify the script to parse and store that additional information. For the ancient patches DEX project, everyone involved had to update a status file stored in a VCS on alioth. This was rather annoying, so we decided early on that we want to be prevent this from being necessary for future projects. This means that the dashboard needs a way for users to modify the information on it. For tasks that are based on bugs, this is rather easy; you simply use the BTS email interface to adjust the bug. However, for other tasks, we needed the ability to modify them on the dashboard. I have managed to get this partially implemented. Currently, all cells on the table are text input fields (with css being used to hide the boxes). This means that you can click on any cell and edit its contents. Hitting the submit button will execute a script to save these changes. This is the tricky part. The dashboard is a collection of html files generated by a python script. The script is meant to be run by cron every n minutes. When you update something via a web interface, you usually expect to see the changes instantly. This means that we can t wait for the python script to run again and re-generate the html files. As a result, I currently have it setup so that when you submit the form, it will go and modify the html files to reflect the changes. The next time cron runs the python script, the html files will be re-created, and the changes made by the user will be lost. Once I come up with a method for storing additional information about a task, it shouldn t be too hard to make changes made by a user via the web interface permanent. I also did some work on a script to generate graphs. There are two types of graphs I hope to be able to display on the dashboard. One graph will help show which individuals have done the most work on a project. I have not started working on this graph yet. The second type of graph is meant to show the number of open tasks versus time. This will allow people to quickly see the progress of a project, estimate how long the project will take to complete, and notice if things start to slow down. I have some initial code for this graph using matplotlib. However, due to the many changes that were made to the dashboard, I had to hold off a bit on this graph, as it relies on certain information provided by the dashboard. I hope to sort out some of the last few issues with this graph, such as getting the axes to display properly, and then merge it with the dashboard sometime in the next few weeks. Allison Randal approached me a while ago to see if my Summer of Code project might be in a state where it could be used to assist with a Python 2.7 DEX project that she was working on. For a while, I did not think that it would be ready. However, last week, I helped her get the related bugs usertagged appropriately, and they now show up in the DEX dashboard. While the dashboard is still not complete, it does allow her and everyone else working on the project to clearly see the tasks (bugs) that they need to focus on and the current status of each of them. I plan to try and incorporate any feedback I receive from people working on this project into my designs. All of my code is available in a git repository on alioth. Keep in mind that it should not be considered stable at this point. Further status updates will be posted on my blog, and I will try to keep the wiki page updated. I am still trying work out where we will host the actual dashboard. In the meantime, you can try out my testing version of it. During the next few weeks, I will be focusing my efforts on coming up with a method for storing additional task information. That will allow me to support permanently storing changes made by users via the web interface. I also hope to finish the first graph script and merge it with the dashboard. Finally, I hope to find a permanent home for the dashboard, as having it on my personal server is not a viable long-term solution. Finally, if anyone is interested in getting involved with DEX, feel free to join the debian-derivatives mailing list, join #debian-derivatives on OFTC, or contact me via email or on IRC (nhandler).</project></distribution>

Rudy Godoy: Compute Clusters Integration for Debian Development and Building Report 2

Hello, this is the 2nd report regarding my project. I'll offer a brief
summary of how are we doing and then the details, so you don't have to
read everything at all.
I have to start saying that over the last two weeks I made a lot of
progress on the project. The most important part: Debian ARM image
integration with Eucalyptus is now cleared. The work to be done for the
next two weeks is pretty much fully coding to achieve what we've committed:
supporting Debian ARM images for Eucalyptus.
What was done so far
--------------------
- I've have identified and mapped what needs to be modified over the
  eucalyptus source for supporting ARM. I'm puting the details on the
  next section. I think this is the most crucial part that is now
  sucessfully achieved.
- I started a Project Log[1] on the wiki, so anyone can check and
  eventyally use the information I've already gathered. The log is being
  updated constantly. You can also check what I've done and what I'm
  working on.
- We got feeback[2] from Eucalyptus developers and packaging
  team regarding developers and packaging team regarding the ultimate
  source for debian/ packaging bits. This was quite important because
  in the past I've managed to build Eucalyptus from source using the debian/
  included in the upstream tarball. However the people in charge
  told us that they were using a different one, which indeed was
  different and resulted in different output than I expected. I'm
  still working on fixing the bits to get a clean build.
- I've joined the pkg-eucalyptus team and I've been given commit rights
  to the repo[2].
1- http://wiki.debian.org/SummerOfCode2011/BuildWithEucalyptus/ProjectLog
2- http://lists.alioth.debian.org/pipermail/pkg-eucalyptus-maintainers/2011-June/000289.html
What I'll be working over the next two weeks
--------------------------------------------
- Heavily code on the different bits that I've identified and need to be
  changed. They are:
  C code:
  - node/handlers_kvm. c,h  - handles nc creation. nc's are node
  clusters for Eucalyptus, the code is in charge of node creation. It
  prepares the "machine" that is able to run a compatible image.
  - util/data. c,h  - handles instance creation. The code over there
  implements the functions node/handlers_* use, like instance creation
  and libvirt's API parameter passing. Since libvirt's parameters for
  ARM vary from x86's ones, we need to support that.
  Perl code:
  - tools/gen_kvm_libvirt_xml - wrapper tool for creating KVM/QEMU
  libvirt's XML Domain definition. This needs to be heavily
  changed/updated - redesigned, to support ARM architecture.
- Depending on the previous task progress: Modify the Debian packaging
  bits for Eucalyptus and push them to the pkg-eucalyptus's team
  repository. This is not crucial for my project's success but I'd like
  to work with Debian official builds. I've yet to commit the
  Eucalyptus team version to the debian/ dir. I expect to have fixed
  the bits to get a successful and clean build, then commit.
Other plans
-----------
- I expect a slow down on the work pace over the second week due end of
  semester exams on the week from July 3 to July 9. After that week I'm
  on vacations and will be fully dedicated to the project.
- Code changes to Eucalyptus source will be pushed upstream. I would
  not like to keep a separate repo for this. I'll talk about this with
  Steffen. Changes to Debian bits will be pushed to the appropiate repos.
Other bits
----------
When the project started I noted it was too vague for some people and
it's results/output were unclear. This concerned me. However, while the
project started and we got lot of feedback things got much clear and I'd
like to take the opportunity to explain it a bit.
Currently, developers targeting ARM devices are struggling[1] with having
a development environment. There are missing bits that sometimes are not
in place and result on hard time for them. Also, due mobile devices
growth and vendors using Linux-based[2] operating systems it's a nice
opportunity for Debian to keep it's well known fame of: supporting every
arch over the sun. How this project fits, well by offering the ability
for developers to run ARM-based Debian images over x86/x86_64 CPU's
would impact the way they could be more productive and for Debian to be
more adopted as development environment.
I can say that I'm very enthusiast of this opportunity and I'm
certain that the outcome will result in more adoption. Even the Ubuntu
people is now pushing more and more efforts for ARM official support,
due the facts I've stated.
1- http://lists.debian.org/debian-arm/2011/06/msg00020.html
2- http://events.linuxfoundation.org/events/android-builders-summit/back

19 June 2011

Clint Adams: I still have not seen that documentary

Receiving a full rejection for DebConf travel sponsorship reminds me of a time I was asked to nominate members for a corporate board of directors. At the time I replied that I would not be of much help because I value things which are irrelevant to such positions. Sure, I can identify worthless, grandstanding blowhards, but of the people who are not, I am unqualified to guess which ones make good board members and which do not. The person who had asked me was somewhat confused by my answer, because we come from different perspectives on how the world works and possibly how the world should work.

Alastair McKinstry: So are we headed for a permanent El Ni o <80><93>?

Maps and Coastlines in Debian #tags enso,climate,paleooceanography #postdate 2011-06-19 18:00 #ignore yes Thanks to Brian Lawrence for a pointer to an interesting review by George Philander: Where are you from? Why are you here? An African perspective on global warming. in the Annual Review of Earth and Planetary Sciences. Scroxton et al. at Oxford, in the latest edition of Paleooceanography published < href="http://www.agu.org/pubs/crossref/2011/2010PA002097.shtm"> Persistent El Ni o Southern Oscillation variation during the Pliocene Epoch Scroxton et al. study fossil foraminera, plankton remains from the mid-Pliacezian warm period (3.26 - 3.03 million years ago). By studying isotope ratios in the carbonate skeletons of these plankton, they can reconstruct the surface water temperatures in the Eastern Pacific, where the El Nino is today. They discover conditions similar to today, with El Nino-like events.

Leo 'costela' Antunes: Importing an Outlook PST into IMAP

Well, every once in a while we re forced to do something that isn t particularly interesting or pleasant. Last week it happened again: I had to import a few pretty big PSTs (most 2Gig, one 10Gig, with about 100.000 emails) into our dovecot IMAP. Doing this with Outlook itself was out of the question: it took way too long, even on the local network (many hours for a 1G file) and was prone to hanging and crashes, which were obviously a pain to debug and start over.
Thunderbird was unfortunately not much better, since at least in our tests it didn t import the read status of the emails (they were all marked as unread) and also wasn t particularly good at handling folders with strange names, containing . , / or some more obscure characters. We had used it before for smaller files, where manually dealing with the problems was acceptable, but this time it required something a bit more elaborate, if we were to keep our sanity. Enter libpst. It includes the handy readpst utility which dumps all emails in usable formats in a directory tree, one directory per folder. Unfortunately the Debian version is somewhat outdated and doesn t support the newer Outlook formats, so I did some packaging and even a little bit of patching. It seems Thunderbird also uses this library, which would explain why it didn t handle the Read-Status (haven t confirmed this though; just read it somewhere). The last step was this not-so-little script, which uses the dumped directories from readpst and imports them in IMAP. It would have probably been a bit more elegant to use libpst directly, but I unfortunately didn t have the time to mess around with that. I did have to mess around a lot with encodings though, ergo the unholy chaos with unicode()s and str.encode()s thrown around like rice at a wedding (I could never really wrap my head around charset problems; the subject boggles my mind to this very day).
#!/usr/bin/env python2.7
#-*- encoding: utf-8 -*-
 
import os, sys, time
import re
 
from argparse import ArgumentParser
from getpass import getpass
 
import imaplib
from email.header import decode_header
from mailbox import mbox
 
from twisted.mail import imap4 # for their imap4-utf7 implementation
 
parser = ArgumentParser(description="""Recursively import mbox files in a directory 
                                        structure to an IMAP server.\n
                                        The expected structure is that generated by 
                                        'readpst -r'.""")
parser.add_argument('-s', dest='imap_server', default='localhost', help='IMAP server to import emails to')
parser.add_argument('-u', dest='imap_user', required=True, help='user for logging in to IMAP')
parser.add_argument('-p', dest='imap_passwd', help="will be prompted for if not provided")
parser.add_argument('-c', dest='charset', default='utf8', help='charset in which the folders are stored (for versions older than 2003)')
parser.add_argument('-f', dest='force', action='store_true', help='import mail even if we think it might be a duplicate')
parser.add_argument('-m', dest='mappings', help='a JSON file with mappings between folder names and mailbox names (no slashes or dots)')
parser.add_argument('folder', nargs='+', help="the base folders to import")
 
args = parser.parse_args()
 
if not args.imap_passwd:
    args.imap_passwd = getpass()
 
 
if args.mappings:
    import json
    folderToMailbox = json.load(open(args.mappings,'r'))
else:
    folderToMailbox =   
 
 
 
def mailboxFromPath(path):
    paths = []
    for p in path.split(os.path.sep):
        p = folderToMailbox.get(p, p)
 
        # only other invalid char besides '/', which can't be created by readpst anyway
        p = p.replace('.','')
 
        paths.append(p)
 
    return '.'.join(paths)
 
def imapFlagsFromMbox(flags):
    # libpst only sets R and O
    f = []
    if 'R' in flags or 'O' in flags:
        f.append(r'\Seen')
    if 'D' in flags:
        f.append(r'\Deleted')
    if 'A' in flags:
        f.append(r'\Answered')
    if 'F' in flags:
        f.append(r'\Flagged')
    return '('+' '.join(f)+')'
 
def utf7encode(s):
    return imap4.encoder(s)[0]
 
def headerToUnicode(s):
    h = decode_header(s)[0]
    try:
        if h[1]: # charset != None
            try:
                return unicode(*h)
            except LookupError:
                return unicode(h[0],'utf8','replace')
        else:
            return unicode(h[0], 'utf8')
    except UnicodeDecodeError:
        try:
            return unicode(h[0], 'cp1252') # the usual culprits for malformed headers
        except UnicodeDecodeError:
            pass
 
        try:
            return unicode(h[0], 'latin1') # the usual culprits for malformed headers
        except UnicodeDecodeError:
            pass
 
        return unicode(h[0], 'ascii', 'ignore') # give up...
 
def main():
    imap = imaplib.IMAP4_SSL(args.imap_server)
    imap.login(args.imap_user, args.imap_passwd)
 
    imap.select()
    for base in args.folder:
        print "importing folder "+base
        for root, dirs, files in os.walk(base):
            if 'mbox' in files:
                folder = unicode(os.path.relpath(root, base), args.charset)
                mailbox = mailboxFromPath(folder)
                print u'importing mbox in  0  to  1 '.format(folder, mailbox)
                mailbox_encoded = utf7encode(mailbox)
                m = mbox(os.path.join(root, 'mbox'), create=False)
                for msg in m:
                    if imap.select(mailbox_encoded)[0] != 'OK':
                        print "creating mailbox "+mailbox
                        r = imap.create(mailbox_encoded)
                        if r[0] != 'OK':
                            sys.stderr.write("Could not create mailbox: "+str(r))
                            continue
                        imap.subscribe(mailbox_encoded)
                        imap.select(mailbox_encoded)
 
 
                    # skip possibly duplicated msgs
                    query = 'FROM " 0 " SUBJECT " 1 "'.format(
                                utf7encode(headerToUnicode(msg['from']).replace('"','')),
                                utf7encode(headerToUnicode(msg['subject']).replace('"',r'\"'))
                            )
                    if msg.has_key('date'):
                        query += ' HEADER DATE " 0 "'.format(utf7encode(msg['date']))
                    if msg.has_key('message-id') and msg['message-id']:
                        query += ' HEADER MESSAGE-ID " 0 "'.format(utf7encode(msg['message-id']))
 
                    r = imap.search(None, '( 0 )'.format(query))
                    if r[1][0] and not args.force:
                        print "skipping "+mailbox+": '"+headerToUnicode(msg['subject'])[:20]+"' (mid: "+str(msg['message-id'])+")"
                        continue
 
                    r = imap.append(mailbox_encoded, '', imaplib.Time2Internaldate(time.time()), str(msg))
                    if r[0] != 'OK':
                        sys.stderr.write("failed to import  0  ( 1 ):  2 ".format(msg['message-id'], msg['date'], r[1]))
                        continue
                    num = re.sub(r'.*APPENDUID \d+ (\d+).*', r'\1', r[1][0])
                    r = imap.uid('STORE', str(num), "FLAGS", imapFlagsFromMbox(msg.get_flags()))
 
                    if r[0] != 'OK':
                        sys.stderr.write("failed to set flags for msg  0  in  1 ".format(num, mailbox))
 
    imap.logout()
 
if __name__ == '__main__':
    main()
Another useful automation is using the mappings file, which looks something like
 
"Deleted items": "Trash",
"Sent items": "Sent"
 
to automatically import in the right destinations, in case you have a standardized structure.

Alastair McKinstry: So are we headed for a permanent El Ni nino?

Maps and Coastlines in Debian #tags enso,climate,paleooceanography #postdate 2011-06-19 18:00 #ignore yes Thanks to Brian Lawrence for a pointer to an interesting review by George Philander: Where are you from? Why are you here? An African perspective on global warming. in the Annual Review of Earth and Planetary Sciences. Scroxton et al. at Oxford PALEOCEANOGRAPHY http://www.agu.org/pubs/crossref/2011/2010PA002097.shtml Persistent El Ni o Southern Oscillation variation during the Pliocene Epoch mid-Piacenzian Warm Period (mPWP) (3.26 3.03 Ma)

18 June 2011

Vincent Bernat: The sorry state of Flash with 64-bit Debian

Even with the availability <video> tag in HTML5, most websites still require the use of Adobe Flash plugin to access some content. If you run a 64-bit (Intel architecture) version of Debian, you can still get this plugin through the flashplugin-nonfree package. Unfortunately, Adobe never commited to a decent support for the 64-bit version of its plugin. The first 64-bit version was provided as late as the fall of 2008. The support was discontinued. Then, we recently got a new preview release but it is still lagging behind the 32-bit version. The 64-bit plugin has two major drawbacks: Even if you fix the second point, you have no way to fix the first one. What are the alternatives?

Lightspark Lightspark is a modern, free, open-source Flash player implementation featuring JIT compilation (with the help of LLVM) and hardware acceleration using OpenGL. It only supports Actionscript 3 (introduced in Flash 9) but it now fallbacks to Gnash if an older version is required. The main problem of Lightspark is that it is optimized for modern hardware and uses OpenGL textures to display video frames. This allows to achieve video compositing with the help of the graphic card. Unfortunately, if you use a video card with poor OpenGL performance, like a NVidia card with Nouveau driver, Lightspark is slow as hell. It works well with an Intel card. Currently, Lightspark is only available in experimental for Debian. Install browser-plugin-lightspark.

Gnash Gnash is a more mature project but does not support Actionscript 3. Youtube works well with Gnash, as well as most other applications unless they use too recent features. If your use of Flash is limited, it is a nice alternative. Unfortunately, more and more content are delivered with an application requiring Actionscript 3 support. On Debian, you can get started by installing browser-plugin-gnash.

32-bit Flash plugin on Debian Ubuntu provides Flash as a 32-bit plugin with the help of nspluginwrapper which is a special plugin that will enable the use of a 32-bit plugin inside a 64-bit browser. This seems a sensible choice since Adobe only really supports the 32-bit version. You can also use nspluginwrapper with Debian. You need to download 32-bit version of Flash, uncompress it and put libflashplayer.so in ~/.mozilla/plugins.
# aptitude purge flashplugin-nonfree
# aptitude install nspluginwrapper lib32asound2-plugins
$ cd ~/.mozilla/plugins
$ nspluginwrapper -i $PWD/libflashplayer.so
Beware that you need to watch for updates yourself. Mozilla provides a website checking if plugins are up-to-date. It works with other browsers than Firefox. This is a pity to still need this kind of hack in 2011 but it seems that this is currently the best solution. UPDATED: In a comment, Paul Rufous tells debian-multimedia.org repository provides flashplayer-mozilla package which is the 32-bit plugin with nspluginwrapper.

HTML5 Thanks to the <video> tag available in HTML5, most videos should be available without the need of a special plugin. Unfortunately, HTML5 does not define a list of video codecs and container formats that should be supported by any browser. Therefore, some browsers support Ogg Theora, some others H264 and some of them also support WebM, the new royalty-free format from Google. You can find a good summary of the problem from Mark Pilgrim. With Chromium as found in Debian, you have Ogg Theora, WebM and H264. Youtube has an opt-in program to enable HTML5 video but most other video content are just unavailable without Flash. Web without Flash is not a reality yet.

Grab video with helper tools UPDATED: Josh Triplett pointed in a comment the existence of several command line tools that could be used to download Flash video from popular websites. Those tools include:
  • youtube-dl
  • get-flash-videos
  • clive
Those tools can be useful even if you have installed a Flash plugin since they allow the use of the video player of your choice (better performance, working fullscreen mode). UPDATED: Another solution provided in a comment: Video DownloadHelper extension allows to extract videos from the current web page. In another comment, this is the extension FlashVideoReplacer which is featured. It replaces the Flash container with a standalone player.

John Goerzen: Music Powerful and Fun

Sometimes there s nothing quite so heartwarming as hearing a 4-year-old that doesn t think anybody is listening. When Jacob is all alone in his room, sometimes he will sing. Maybe it s a song he knows, or fragments of a song he knows. Or maybe it s something he just made up on the fly. It might be about trains, cats, letters, or who knows what. Maybe it will crack him up and he ll keep singing and then laugh. Or maybe it will stop in 30 seconds and he ll go on to something else. Brief and out of tune though it may be, it is still one of the most beautiful sounds I know and I love to hear it. Jacob loves to have other people sing, too. Sometimes I will make up songs while we drive: We are driving, we are driving, we are driving down the road; we are driving down the old country road and we re making some dust, yes a lot of dust Or Jacob likes me to sing the songs in the Winnie the Pooh book he has (the original, pre-Disney version, which has the words for the songs but not a tune, so I make up a little bit different one each time). Very occasionally he will sing with me, but if he does, he doesn t want anybody to comment about it. I sing with the Kansas Mennonite Men s Chorus. We re a group of about 300 singers (all men, though not all Kansan and not all Mennonite) that sing for charity. We ve raised over $600,000 for charity so far. I still remember the first time I went to a KMMC practice 3 years ago. It was my first year singing, and the first practice of the season. We started singing the first song, and wow what a powerful moment. Even at its imperfect first practice state, hearing 300 people sing quietly is a powerful sound and when they get to the top of their lungs, it s indescribable. No CD can ever quite do that justice. Jacob and Oliver come to KMMC concerts sometimes, but this year they took a special interest. Oliver pointed to, hm, perhaps 300 men and said dad for each one. (He was quite far away and probably couldn t pick me out specifically.) Jacob, for the first time, sat quietly engrossed in the concert until we got to Dry Bones (a song involving surprise and lots of homemade instruments) which made him laugh out loud. Then he would hide under his seat from the applause. Last week, KMMC went on tour. This was my first choir tour, and Terah came along while the boys spent time with grandparents in Kansas. We sang in Bloomington, IL, and then in Goshen, IN at the fairly new Sauder Concert Hall. That hall is a few years old, and has been reviewed as once of the best concert halls in the world acoustically. It seats 1100 people and was perfect for this. The 85 or so singers from KMMC that went on tour joined the Indiana and Ohio Men s Choruses to make a combined group of about 200 singers. And what an experience that was. I learned later this was the first event to completely pack Sauder Concert Hall. We ended the concert singing the Mennonite Anthem a souped-up version of the doxology. (You can see a video of a different group singing it to get an idea of what it was). This is often joked about as being part of Mennonite DNA. You can gather a random group of Mennonites and ask to hear 606 (its number in, well, an old hymnbook that we don t use anymore) and you can probably get a fairly well-sounding rendition, from memory, a cappella, complete with 4-part harmony, at the drop of the hat. Mennonite youth have been known to sing it in train stations, airports, soccer stadiums, and with kazoos. At Goshen, we had 200 men on stage, plus a brass band, plus 1100 people in the audience facing us. They were, of course, invited to sing along. (That s one of the rules of 606: you can t exclude anybody because even if you tried, they d still sing, because how can you resist?) So I was up there on stage, singing a song I love, and hearing the 1100 people in the audience sing it to me. I could glance out over the front rows and saw the smiles on so many faces as they sang, and what a moving moment that was. The excitement and thrill of it all hit me so much that I could barely finish the song. The numbers aren t all in yet, but I m guessing that we raised over $15,000 for charity last weekend (the choir members paid their own expenses in advance, so 100% of the money raised goes to international relief efforts.) A 4-year-old singing alone in his room is really heartwarming. I have to say that 1300 people singing a song that touches each one of them is a pretty close second.

17 June 2011

John Goerzen: APRS: World s Best Social Mapping and Wide-Area Ad-Hoc Wireless Mesh Network

That was quite a headline, and I m going to try to back it up below. APRS is the Automatic Packet Reporting System. It s a system for exchanging brief packets of information. It is most frequently used for mapping applications, but it really does a lot more than that. It has its biggest home in the amateur radio world, but isn t limited to that, either. The most common way to use APRS is to have some device hooked up to a GPS transmit packets with the GPS information in them. These packets can then be plotted on a map in real-time or with history. That in itself isn t particularly newsworthy these days. An interesting thing about APRS is that it s not just positioning. Let s say that there was a search-and-rescue operation. A person could draw a rectangle on the map indicating the search area, and within about 3 seconds everyone else s map also shows that rectangle. People have even been known to play chess by sharing and moving objects on APRS! The next piece that makes this interesting is that APRS is an ad-hoc mesh network. In its traditional implementation, VHF amateur radio, a radio emits a packet with a geolocation in it (a beacon ) and any other radio within direct range of that can receive it. Radios can display basic information (such as distance to the other radio, heading, etc.) or hooked up to a laptop or mapping device for a better display. So if everyone is within a few miles, APRS works without any pre-existing infrastructure at all. This makes it wonderful for use in disaster areas, and was put to heavy use in Joplin after the tornado there. But what about radios that are too far away? Any APRS station could also be a digipeater. When a packet is transmitted, it has a maximum hop count. A digipeater hears the packet, decrements the maximum hop count, and re-transmits it. With this mechanism, packets can travel hundreds of miles. It creates a highly resilient network, one that can route around trouble without even having to have an explicit backup route. I could bring in a digipeater in my car it can be small enough to hold in my hand and instantly improve APRS reception in an area. One interesting aspect is that packets can be digipeated more than the maximum hop count. For instance, if a packet leaves my radio and is picked up by a digipeater to my west and one to my east, it can keep on traveling in both directions. This is part of what leads to resiliency. APRS also functions over the Internet. There is a large network of interconnected Internet servers that exchange all global APRS traffic amongst themselves. Gateways between the radio (RF) and Internet (APRS-IS) services exist, and are called iGates. They are not generally required, but make useful websites like aprs.fi and email gateways possible. As long as an iGate is within a reasonable number of hops from you, you re effectively linked. And again, if one iGate drops off, another iGate is probably monitoring your traffic too and you never notice. It s an ad-hoc mesh network that is actually reliable how about that? On the PC side, there are many programs for using APRS. The most common one for Windows is called UI-View, but I don t use Windows so I can t comment. On Linux, there are programs (such as aprx) for running your digipeater, but the best-known program is Xastir. Xastir lets you download map files to your local disk, and can interface with the APRS-IS Internet service, radios, weather stations, or simply other arbitrary machines to exchange information. Xastir is a very nice program and is well worth the install, despite its somewhat dated-looking interface. APRS clients, such as APRSDroid, exist for Android and iOS platforms as well. So let s say you re doing something like helping handle food/water stations for a long bike ride. Even if you don t have anybody with an amateur radio license, you can use APRS to great effect. At your headquarters, you can run Xastir and turn on its server mode . This puts everyone on a map. Then you can have everyone turn on APRS on their phone, and have it report to your custom server instead of APRS-IS. Now you have instant visibility into your entire team s location and status. If you have transport people driving supplies between locations, that s especially helpful. In an amateur radio scenario, you would instead have people with radios at each location, and one laptop hooked up to a radio at HQ. This provides an added bonus of not relying on third-party infrastructure such as cellphone towers. APRS also has a messaging system, similar in concept to text messaging. It works the same as other things. If I want to send a message to Jane, my radio simply emits a packet that lists the message and Jane as a recipient. It s digipeated up to its maximum hop count. If Jane is within RF range of one of those digipeaters, she gets the message and her radio ACKs it. Otherwise, it s delivered into the APRS-IS network probably several times, which isn t a problem and the APRS-IS network delivers it to the iGate closest to her, and from there it gets digipeated the rest of the way to her. Here s an example of something created with APRS. While I was on a bus choir tour last weekend, I had a radio with me that was beaconing all the while. Now it was a small handheld radio inside a large metal bus, so it didn t always have a digipeater in range. But still, you can go see a detailed map with the trail and even see exactly what path each packet took before it hit the Internet. If you want to try out Xastir, please grab at least version 2.0 the version in squeeze has some bugs.

Timo Jyrinki: Betawikiss tuhannen euron porkkana

--- cut ---
Vuoden 2007 lopussa Siebrand asetti lokalisointitavoitteet MediaWikille.Tavoitteet olivat todella kunnianhimoisia. N ytt kin silt , ett n ihin ei p st . Emme kuitenkaan aio luovuttaa ilman taistelua. Viel on vajaa viikko aikaa saavuttaa nuo tavoitteet. Sin voit auttaa saavuttamaan tavoitteet suomen kielen osalta.

Yhteisty ss Stichting Open Progressin kanssa pystymme tarjoamaan sinulle kannustusta. Tarjoamme 1000 euroa jaettuna kaikkien k nt jien kesken, jotka ovat tehneet yli 500 uutta k nn st MediaWikiin ja sen laajennuksiin vuoden loppuun menness .

My s muut Betawikin projektit kaipaavat apua. Kiinnostaisiko sinua k nt vaikkapa FreeCol-peli tai Mantis-virheenseurantaj rjestelm . Yst v llinen ja yhteisty kykyinen ymp rist saattaa sinut vauhtiin sek auttaa sinua kehittym n suomentajana parantamalla k nn ksi yhdess .

Betawiki l ytyy osoitteesta http://translatewiki.net.
--- cut ---
(l hde)

Katso my s http://lokalisointi.org/.

Timo Jyrinki: Kernel Mode-Setting (KMS) on Neo FreeRunner + Debian

Edit Aug 18th: video available (ogv, also on youtube)

Forget AMD, Intel, NVIDIA! Thomas White's incredible work with Neo FreeRunner's puny graphics dec...accelerator, Smedia Glamo 3362 is starting to bear fruits. Not listening to such comments as the chip will never be used anywhere else and why not spend your time doing something more relevant , he has chosen to actually do what he likes and sees as an interesting challenge. And that's often the spirit of free software, so I don't really agree with the naysayers.

The camera shot on the right, running KMS-enabled X.org driver for the glamo chip on my Debian installation (visible software matchbox-window-manager, fbpanel, zhone), is a bit optimistic looking since Zhone happens to draw correctly. A lot of the drawing is not yet synced correctly, which shows as all text and images in eg. GTK applications being garbled. But as little as two days ago one couldn't yet much launch applications without X crashing, so the newest commits by Thomas were a big step forwards. I'm using Debian, and he's not, so I try to find time to help in debugging even though I really can't much help with the driver code.

The driver is not just one piece of code, but consists of a kernel drm driver (direct rendering manager) using GEM, libdrm support for the kernel driver and finally the X.org driver supporting these other components and offering buzz-words like DRI2. There is also a beginning of a Mesa 3D driver, though it is so far just a skeleton driver since the 2D/KMS/EXA/DRM parts are what should be done first before dwelling into the OpenGL realm.

Openmoko the company is basically nowadays just producing new Neo FreeRunners to resellers, but the community of the so far only Free phone is thriving. The final (Openmoko produced, from where gta02-core continues) version of the phone, with the famous buzzing problem fixed, appeared on store shelves some time ago, offering better out-of-the-box phone functionality in addition to all the mini computer features. It was also offered to Debconf visitors on discount.

It's not only cool to have kernel mode-setting, though it is indeed very cool as well. The reason for much sorrow in the whole OM project has been the graphics chip, and some of the problems with the chip are only finally solvable with the kernel doing the mode-setting. So it's both a very modern thing to use KMS, but there are also clear potential benefits of having accurate control over this very sensitive piece of silicon. The current non-KMS X.org driver for example busy loops in order to try to feed the chip at a correct pace, causing CPU usage every time there is any drawing going on.

This is somehow reminding me about getting the best out of 8-bit computers and other hardware with some specific limitations. The glamo chip is very limited in some ways, but also capable in some other ways like theoretically offering MPEG-4 decoding, OpenGL ES 3D support. It's a mixed bag of things, and I can very well imagine it's indeed an interesting challenge to work on it, if you just have the skills (I don't) and nerves (proper debugging tools help).

It's so nerdy to drool over lines in the X.org log, but I just can help it:
X.Org X Server 1.6.3
Release Date: 2009-7-31
X Protocol Version 11, Revision 0
Build Operating System: Linux 2.6.28 armv5tel Debian
Current Operating System: Linux neo 2.6.29-GTA02_mydrm #2 PREEMPT Mon Aug 17 18:24:39 EEST 2009 armv4tl
...
II) Using KMS!
...
(WW) Glamo(0): EXA hardware acceleration initialising
(II) EXA(0): Driver allocated offscreen pixmaps
(II) EXA(0): Driver registered support for the following operations:
(II) Solid
(II) Copy
(II) Glamo(0): Initialized EXA acceleration
...
(II) Glamo(0): RandR 1.2 enabled, ignore the following RandR disabled message.
...
(II) Glamo(0): [glamo-dri] Name of DRM device is '/dev/dri/card0'
(WW) Glamo(0): [DRI2] Version 1 API (broken front buffer rendering)
(II) Glamo(0): [DRI2] Setup complete
...
(II) AIGLX: enabled GLX_MESA_copy_sub_buffer
(II) AIGLX: Loaded and initialized /usr/lib/dri/glamo_dri.so
(II) GLX: Initialized DRI2 GL provider for screen 0
(II) Glamo(0): Adding framebuffer....!
(II) Glamo(0): 8 480 640 16 16 960
(II) Glamo(0): rootPixmap = 0x1d0a00
(II) Glamo(0): Done

Nathan Handler: Google Summer of Code Update #4

Since Matt is going to be busy the next few weeks moving and traveling, he arranged for Stefano Zacchiroli to fill in for him temporarily as my mentor. I must say, I am quite lucky to have the chance to have both Matt and Stefano as mentors this summer. Yesterday, I had my first call with Stefano. I started by filling him in on the current status of the project. We discussed how the dashboard is going to be based on tasks rather than bug . He reminded me that we want to make sure all useful information makes its way back to the BTS (ideally in an automated or) in a fast and easy way. Most of the information that will be stored in the dashboard probably won t be that useful to the package maintainers. However, there will be a column for DEX members to add notes about tasks. This column could potentially contain some useful information. That is one reason that the dashboard will parse this column for bug numbers and then display extended information about the bug. Hopefully, this will encourage DEX members to send information to the BTS when relevant. Having the bug numbers will also allow us to automate sending information from the DEX dashboard to the BTS if we choose to. I also spent some time this week working on a script to generate some graphs. These graphs will allow us to track the number of remaining tasks in a project vs. time. Stefano mentioned possibly borrowing some code from the Ubuntu burndown charts, but right now, I think the Ubuntu charts might be a bit more complicated than what we need. We can always decide to borrow some features/code down the road if necessary. It is also just about time for me to prepare my second formal report for the Summer of Code. I will use that to showcase the current status of the code and make it available for people to test and review.

16 June 2011

Wouter Verhelst: Attaching in mutt

All mail clients suck. Some just suck less. But they all have their own peculiarities. mutt, for instance, is a full-featured text-mode mail client. As such, it's pretty popular amongst many Linux enthousiasts, including me. A common error made amongst mutt users is to talk about attaching a file in a mail, but then forgetting to actually do so afterwards. This is because mutt starts an external editor to allow you to edit the message in question, which since it's not mutt itself would seem to make it impossible to attach a file while you're editing a mail, requiring one to remember to actually perform the attach step. One solution to this problem could be to use the little script that Steve wrote, which will complain if you mention something about an attachment in a mail without actually attaching anything. While I'm sure it works to some extent, this kind of script has the downside that it's never going to be 100% correct in all cases (you might mistype the word 'attach', or you might use another word that you didn't think about when you wrote the script, or you might be replying to someone who sent you an attachment). However, a better way could be to realize that mutt does, actually, support attaching files while you're editing a mail text, by way of the 'Attach:' pseudo-header. It requires two steps:
  1. Make sure that you have set 'edit_headers' to 'yes' in your configuration. This will cause mutt to fire your editor on a file that also contains the (relevant) headers for your mail, rather than just the body.
  2. When you want to attach a file, rather than trying to remember to do so after exiting your editor, just add an 'Attach:' header to the headers section of your mail, with as argument a filename optionally followed by a description. When mutt finds that header, it'll attach the named file, and all is well.
Yes, of course, that does mean you'll have to change your habits, and actually remember to add the damn pseudo-header rather than be like me and still forget to attach files. But hey, can't help you with that.

Next.

Previous.