Yupp, it s this time of the month again! Rhonda has already spammed about it, but I think repeating won t hurt: the next (4th) meeting of the Debian/Ubuntu Games Team will be held in #debian-games on June 26th at 10:00 UTC.
Another month, another Games Team IRC Meeting happening. This time it was decided to have it again on Sunday, the time was set to 10 am UTC. To find out the time in your localtime, issue date -d '2011-06-26 10:00 UTC'
This is a frequently asked question: What package manager shall I use? . And my answer is the one that suits your needs . In my case, I even use different package managers depending on what I m trying to do.
APT vs dpkg, which one is the package manager?
In the Debian world, we re usually thinking of APT-based software when we re referring to a package manager . But in truth, the real package manager is dpkg. It s the low-level tool that takes a .deb file and extracts its content on the disk, or that takes the name of a package to remove the associated files, etc.
APT is better known because it s the part of the packaging infrastructure that matters to the user. APT makes collection of software available to the user and does the dirty work of downloading all the required packages and installing them by calling dpkg in the correct order to respect the dependencies.
But APT is not a simple program, it s a library and several different APT frontends have been developed on top of that library. The most widely known is apt-get since it s the oldest one, and it s provided by APT itself.
Graphical APT front-ends
update-manager is a simple frontend useful to install security updates and other trivial daily upgrades (if you re using testing or sid). It s the one that you get when you click in the desktop notification that tells you that updates are available. In cases, where the upgrade is too complicated for update-manager, it will suggest to run synaptic which is full featured package manager. You can browse the list on installed/available packages in numerous ways, you can mark packages for installation/upgrade/removal/purge and then run in one go all the recorded actions.
software-center aims to be an easy to use application installer, it will hide most of the packaging details and will only present installed/available applications (as defined by a .desktop file). It s very user friendly and has been developed by Ubuntu.
Of the graphical front-ends, I use mainly synaptic and only when I m reviewing what I have installed to trim the system down.
Console-based GUI APT front-ends
In this category, I ll cite only aptitude. Run without parameter, it will start a powerful console-based GUI. Much like synaptic, you can have multiple views of the installed/available packages and mark packages for installation/upgrade/removal/purge before executing everything at once.
Command-line based package managers and APT front-ends
This is where the well known apt-get fits, but there are several other alternatives: aptitude, cupt, wajig. Wajig and cupt are special cases as they don t use libapt: the former wraps several tools including apt-get, and the latter is a (partial) APT reimplementation (versions 1.x were in Perl, 2.x are now is C++).
You re welcome to try them out and find out which one you prefer, but I have never felt the need to use something else than apt-get and aptitude.
apt-get or aptitude?
First I want to make it clear that you can use both and mix them without problems. It used to be annoying when apt-get did not track which packages were automatically installed while aptitude did, but now that both packages share this list, there s no reason to avoid switching back and forth.
I would recommend apt-get for the big upgrades (i.e. dist-upgrade from one stable to the next) because it will always find quickly a relatively good solution while aptitude can find several convoluted solutions (or none) and it s difficult to decide which one should be used.
On the opposite for regular upgrades in unstable (or testing), I would recommend aptitude safe-upgrade . It does a better job than apt-get at keeping on hold packages which are temporarily broken due to some not yet finished changes while still installing new packages when required. With aptitude it s also possible to tweak dynamically the suggested operations while apt-get doesn t allow this. And aptitude s command line is probably more consistent: with apt-get you have to switch between apt-get and apt-cache depending on the operation that you want to do, aptitude on the other hand does everything by itself.
Take some time to read their respective documentation and to try them.
Click here to subscribe to my free newsletter and get my monthly analysis on what s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.
11 comments Liked this article? Click here. My blog is Flattr-enabled.
A lot of work has taken place since my last report. First, since my original mentor, Matt Zimmerman, is going to be moving, he helped arrange for Stefano Zacchiroli to fill in for him temporarily. I have had a call with Stefano, and I am looking forward to getting to work with him more over the next couple of weeks.
Shortly after my first report, we realized that having the dashboard only display bugs severely limited its usefulness. In our original design, we had it pulling a list of bugs with specific usertags from the BTS. This forced all information to be added to bugs, which we viewed as a good thing since it made the information available to the Debian Maintainers without requiring them to familiarize themselves with DEX. The issue was that when looking back at the ancient-patches project or forward to the big merges project, a large portion of the work will be triaging and determining whether or not the changes are even applicable in Debian. If they aren t, there is no reason for a bug to be filed on the BTS. As a result, I spent a fair amount of time re-writing most of the dashboard. It still supports finding bugs on the BTS that have been tagged with a specific DEX usertag, but it is now task-based rather than bug-based.
A task can be a bug number, a package name, a patch name, or anything. It is merely something that needs to be done for a project. Each task will have a name, an assignee, a status, a note, and a bug associated with it. There are two ways to define a list of tasks for a project. The first is to tag bugs on the BTS with a user of debian-derivatives@lists.debian.org and a tag with a format of dex-<distribution>-
<project> (i.e. dex-ubuntu-ancient-patches). The script will convert the bugs to tasks and automatically set all of the fields. The second method is to specify the list of tasks in a plain text file on the server the scripts are being run on. Eventually, this list will be able to be specified via a web interface. Currently, you are only able to specify the task name in the file. This means that all of the other fields for a task created in this manner are initially blank. I am still deciding on the best way to specify additional information about each task in the file. Once I decide on a method, it should not be too difficult to modify the script to parse and store that additional information.
For the ancient patches DEX project, everyone involved had to update a status file stored in a VCS on alioth. This was rather annoying, so we decided early on that we want to be prevent this from being necessary for future projects. This means that the dashboard needs a way for users to modify the information on it. For tasks that are based on bugs, this is rather easy; you simply use the BTS email interface to adjust the bug. However, for other tasks, we needed the ability to modify them on the dashboard. I have managed to get this partially implemented. Currently, all cells on the table are text input fields (with css being used to hide the boxes). This means that you can click on any cell and edit its contents. Hitting the submit button will execute a script to save these changes. This is the tricky part. The dashboard is a collection of html files generated by a python script. The script is meant to be run by cron every n minutes. When you update something via a web interface, you usually expect to see the changes instantly. This means that we can t wait for the python script to run again and re-generate the html files. As a result, I currently have it setup so that when you submit the form, it will go and modify the html files to reflect the changes. The next time cron runs the python script, the html files will be re-created, and the changes made by the user will be lost. Once I come up with a method for storing additional information about a task, it shouldn t be too hard to make changes made by a user via the web interface permanent.
I also did some work on a script to generate graphs. There are two types of graphs I hope to be able to display on the dashboard. One graph will help show which individuals have done the most work on a project. I have not started working on this graph yet. The second type of graph is meant to show the number of open tasks versus time. This will allow people to quickly see the progress of a project, estimate how long the project will take to complete, and notice if things start to slow down. I have some initial code for this graph using matplotlib. However, due to the many changes that were made to the dashboard, I had to hold off a bit on this graph, as it relies on certain information provided by the dashboard. I hope to sort out some of the last few issues with this graph, such as getting the axes to display properly, and then merge it with the dashboard sometime in the next few weeks.
Allison Randal approached me a while ago to see if my Summer of Code project might be in a state where it could be used to assist with a Python 2.7 DEX project that she was working on. For a while, I did not think that it would be ready. However, last week, I helped her get the related bugs usertagged appropriately, and they now show up in the DEX dashboard. While the dashboard is still not complete, it does allow her and everyone else working on the project to clearly see the tasks (bugs) that they need to focus on and the current status of each of them. I plan to try and incorporate any feedback I receive from people working on this project into my designs.
All of my code is available in a git repository on alioth. Keep in mind that it should not be considered stable at this point. Further status updates will be posted on my blog, and I will try to keep the wiki page updated. I am still trying work out where we will host the actual dashboard. In the meantime, you can try out my testing version of it.
During the next few weeks, I will be focusing my efforts on coming up with a method for storing additional task information. That will allow me to support permanently storing changes made by users via the web interface. I also hope to finish the first graph script and merge it with the dashboard. Finally, I hope to find a permanent home for the dashboard, as having it on my personal server is not a viable long-term solution.
Finally, if anyone is interested in getting involved with DEX, feel free to join the debian-derivatives mailing list, join #debian-derivatives on OFTC, or contact me via email or on IRC (nhandler).</project></distribution>
Hello, this is the 2nd report regarding my project. I'll offer a brief summary of how are we doing and then the details, so you don't have to read everything at all. I have to start saying that over the last two weeks I made a lot of progress on the project. The most important part: Debian ARM image integration with Eucalyptus is now cleared. The work to be done for the next two weeks is pretty much fully coding to achieve what we've committed: supporting Debian ARM images for Eucalyptus. What was done so far -------------------- - I've have identified and mapped what needs to be modified over the eucalyptus source for supporting ARM. I'm puting the details on the next section. I think this is the most crucial part that is now sucessfully achieved. - I started a Project Log[1] on the wiki, so anyone can check and eventyally use the information I've already gathered. The log is being updated constantly. You can also check what I've done and what I'm working on. - We got feeback[2] from Eucalyptus developers and packaging team regarding developers and packaging team regarding the ultimate source for debian/ packaging bits. This was quite important because in the past I've managed to build Eucalyptus from source using the debian/ included in the upstream tarball. However the people in charge told us that they were using a different one, which indeed was different and resulted in different output than I expected. I'm still working on fixing the bits to get a clean build. - I've joined the pkg-eucalyptus team and I've been given commit rights to the repo[2]. 1- http://wiki.debian.org/SummerOfCode2011/BuildWithEucalyptus/ProjectLog 2- http://lists.alioth.debian.org/pipermail/pkg-eucalyptus-maintainers/2011-June/000289.html What I'll be working over the next two weeks -------------------------------------------- - Heavily code on the different bits that I've identified and need to be changed. They are: C code: - node/handlers_kvm. c,h - handles nc creation. nc's are node clusters for Eucalyptus, the code is in charge of node creation. It prepares the "machine" that is able to run a compatible image. - util/data. c,h - handles instance creation. The code over there implements the functions node/handlers_* use, like instance creation and libvirt's API parameter passing. Since libvirt's parameters for ARM vary from x86's ones, we need to support that. Perl code: - tools/gen_kvm_libvirt_xml - wrapper tool for creating KVM/QEMU libvirt's XML Domain definition. This needs to be heavily changed/updated - redesigned, to support ARM architecture. - Depending on the previous task progress: Modify the Debian packaging bits for Eucalyptus and push them to the pkg-eucalyptus's team repository. This is not crucial for my project's success but I'd like to work with Debian official builds. I've yet to commit the Eucalyptus team version to the debian/ dir. I expect to have fixed the bits to get a successful and clean build, then commit. Other plans ----------- - I expect a slow down on the work pace over the second week due end of semester exams on the week from July 3 to July 9. After that week I'm on vacations and will be fully dedicated to the project. - Code changes to Eucalyptus source will be pushed upstream. I would not like to keep a separate repo for this. I'll talk about this with Steffen. Changes to Debian bits will be pushed to the appropiate repos. Other bits ---------- When the project started I noted it was too vague for some people and it's results/output were unclear. This concerned me. However, while the project started and we got lot of feedback things got much clear and I'd like to take the opportunity to explain it a bit. Currently, developers targeting ARM devices are struggling[1] with having a development environment. There are missing bits that sometimes are not in place and result on hard time for them. Also, due mobile devices growth and vendors using Linux-based[2] operating systems it's a nice opportunity for Debian to keep it's well known fame of: supporting every arch over the sun. How this project fits, well by offering the ability for developers to run ARM-based Debian images over x86/x86_64 CPU's would impact the way they could be more productive and for Debian to be more adopted as development environment. I can say that I'm very enthusiast of this opportunity and I'm certain that the outcome will result in more adoption. Even the Ubuntu people is now pushing more and more efforts for ARM official support, due the facts I've stated. 1- http://lists.debian.org/debian-arm/2011/06/msg00020.html 2- http://events.linuxfoundation.org/events/android-builders-summit/back
Receiving a full rejection for DebConf travel sponsorship
reminds me of a time I was asked to nominate members for
a corporate board of directors.
At the time I replied that I would not be of much help
because I value things which are irrelevant to such
positions. Sure, I can identify worthless, grandstanding
blowhards, but of the people who are not, I am unqualified
to guess which ones make good board members and which do
not.
The person who had asked me was somewhat confused by my
answer, because we come from different perspectives on
how the world works and possibly how the world should
work.
Today we had our long-awaited Travel Sponsorship team meeting.
It took a little more than 90 minutes but had us check the ratings
we did in the past days, discuss various of the people and aspects of
our decisions, and then finally we decided on the cutoff levels for
this year.
Actually an easy step (in theory) cutoff levels, it is always hard.
On the one side there are the beancounters and the team working each
year to find money for the conference. Doing whatever it takes to have
our sponsors happy. Who, of course, can not give us unlimited amounts
of money.
On the other hand there are all the (possible) attendees. Many of which
can not afford the travel if we do not help them.
Getting both sides happy is near to impossible. We had about 100 people
asking for nearly 60 000 Euros. On the other hand we had 35 000 Euro to
give out. Doesn t match, and getting another 25 000 Euro just so is not
very likely to happen. Which lead to a Sorry, no money this year for
29 people.
We currently have 12 people still waiting for money. They got a maybe,
if someone cancels or we are lucky to get another sponsor mail, and we
hope we can help them, but we are not sure. We are waiting for another
7800 Euros here.
But the nice and shiny side of this all is: We have 56 people who we
agreed to sponsor travel for, in total 35 000 Euros.
I just posted a call for testing to the not yet well-known debian-stable-announce mailing list. Please test the packages in squeeze-proposed-updates on some stable machines if possible, so that we don't screw up your production machines with bad updates in a week. The point release is scheduled for June 25th, i.e. next Saturday. Don't forget to copy the debian-release mailing list when you encounter regressions. Thanks for your efforts.
Well, every once in a while we re forced to do something that isn t particularly interesting or pleasant. Last week it happened again: I had to import a few pretty big PSTs (most 2Gig, one 10Gig, with about 100.000 emails) into our dovecot IMAP.
Doing this with Outlook itself was out of the question: it took way too long, even on the local network (many hours for a 1G file) and was prone to hanging and crashes, which were obviously a pain to debug and start over.#!/usr/bin/env python2.7 #-*- encoding: utf-8 -*- import os, sys, time import re from argparse import ArgumentParser from getpass import getpass import imaplib from email.header import decode_header from mailbox import mbox from twisted.mail import imap4 # for their imap4-utf7 implementation parser = ArgumentParser(description="""Recursively import mbox files in a directory structure to an IMAP server.\n The expected structure is that generated by 'readpst -r'.""") parser.add_argument('-s', dest='imap_server', default='localhost', help='IMAP server to import emails to') parser.add_argument('-u', dest='imap_user', required=True, help='user for logging in to IMAP') parser.add_argument('-p', dest='imap_passwd', help="will be prompted for if not provided") parser.add_argument('-c', dest='charset', default='utf8', help='charset in which the folders are stored (for versions older than 2003)') parser.add_argument('-f', dest='force', action='store_true', help='import mail even if we think it might be a duplicate') parser.add_argument('-m', dest='mappings', help='a JSON file with mappings between folder names and mailbox names (no slashes or dots)') parser.add_argument('folder', nargs='+', help="the base folders to import") args = parser.parse_args() if not args.imap_passwd: args.imap_passwd = getpass() if args.mappings: import json folderToMailbox = json.load(open(args.mappings,'r')) else: folderToMailbox = def mailboxFromPath(path): paths = [] for p in path.split(os.path.sep): p = folderToMailbox.get(p, p) # only other invalid char besides '/', which can't be created by readpst anyway p = p.replace('.','') paths.append(p) return '.'.join(paths) def imapFlagsFromMbox(flags): # libpst only sets R and O f = [] if 'R' in flags or 'O' in flags: f.append(r'\Seen') if 'D' in flags: f.append(r'\Deleted') if 'A' in flags: f.append(r'\Answered') if 'F' in flags: f.append(r'\Flagged') return '('+' '.join(f)+')' def utf7encode(s): return imap4.encoder(s)[0] def headerToUnicode(s): h = decode_header(s)[0] try: if h[1]: # charset != None try: return unicode(*h) except LookupError: return unicode(h[0],'utf8','replace') else: return unicode(h[0], 'utf8') except UnicodeDecodeError: try: return unicode(h[0], 'cp1252') # the usual culprits for malformed headers except UnicodeDecodeError: pass try: return unicode(h[0], 'latin1') # the usual culprits for malformed headers except UnicodeDecodeError: pass return unicode(h[0], 'ascii', 'ignore') # give up... def main(): imap = imaplib.IMAP4_SSL(args.imap_server) imap.login(args.imap_user, args.imap_passwd) imap.select() for base in args.folder: print "importing folder "+base for root, dirs, files in os.walk(base): if 'mbox' in files: folder = unicode(os.path.relpath(root, base), args.charset) mailbox = mailboxFromPath(folder) print u'importing mbox in 0 to 1 '.format(folder, mailbox) mailbox_encoded = utf7encode(mailbox) m = mbox(os.path.join(root, 'mbox'), create=False) for msg in m: if imap.select(mailbox_encoded)[0] != 'OK': print "creating mailbox "+mailbox r = imap.create(mailbox_encoded) if r[0] != 'OK': sys.stderr.write("Could not create mailbox: "+str(r)) continue imap.subscribe(mailbox_encoded) imap.select(mailbox_encoded) # skip possibly duplicated msgs query = 'FROM " 0 " SUBJECT " 1 "'.format( utf7encode(headerToUnicode(msg['from']).replace('"','')), utf7encode(headerToUnicode(msg['subject']).replace('"',r'\"')) ) if msg.has_key('date'): query += ' HEADER DATE " 0 "'.format(utf7encode(msg['date'])) if msg.has_key('message-id') and msg['message-id']: query += ' HEADER MESSAGE-ID " 0 "'.format(utf7encode(msg['message-id'])) r = imap.search(None, '( 0 )'.format(query)) if r[1][0] and not args.force: print "skipping "+mailbox+": '"+headerToUnicode(msg['subject'])[:20]+"' (mid: "+str(msg['message-id'])+")" continue r = imap.append(mailbox_encoded, '', imaplib.Time2Internaldate(time.time()), str(msg)) if r[0] != 'OK': sys.stderr.write("failed to import 0 ( 1 ): 2 ".format(msg['message-id'], msg['date'], r[1])) continue num = re.sub(r'.*APPENDUID \d+ (\d+).*', r'\1', r[1][0]) r = imap.uid('STORE', str(num), "FLAGS", imapFlagsFromMbox(msg.get_flags())) if r[0] != 'OK': sys.stderr.write("failed to set flags for msg 0 in 1 ".format(num, mailbox)) imap.logout() if __name__ == '__main__': main()
"Deleted items": "Trash", "Sent items": "Sent"
Even with the availability <video> tag in HTML5, most websites still
require the use of Adobe Flash plugin to access some
content. If you run a 64-bit (Intel architecture) version of Debian,
you can still get this plugin through the flashplugin-nonfree
package. Unfortunately, Adobe never commited to a decent support for
the 64-bit version of its plugin. The first 64-bit version was
provided as late as the fall of 2008. The support was
discontinued. Then, we recently got a new preview release but it is
still lagging behind the 32-bit version.
The 64-bit plugin has two major drawbacks:
memcpy() in
place of memmove() and only happens with recent GNU libc (where
memcpy() has been optimized and is no longer equivalent to
memmove()). Adobe has no plan to fix this. There are several way
documented in the bug report to circumvent this bug.browser-plugin-lightspark.
browser-plugin-gnash.
nspluginwrapper which is a special plugin that will enable the use
of a 32-bit plugin inside a 64-bit browser. This seems a sensible
choice since Adobe only really supports the 32-bit version. You can
also use nspluginwrapper with Debian. You need to
download 32-bit version of Flash, uncompress it and put
libflashplayer.so in ~/.mozilla/plugins.
# aptitude purge flashplugin-nonfree # aptitude install nspluginwrapper lib32asound2-plugins $ cd ~/.mozilla/plugins $ nspluginwrapper -i $PWD/libflashplayer.so
flashplayer-mozilla package which is the 32-bit plugin
with nspluginwrapper.
<video> tag available in HTML5, most videos should be
available without the need of a special plugin. Unfortunately, HTML5
does not define a list of video codecs and container formats that
should be supported by any browser. Therefore, some browsers support
Ogg Theora, some others H264 and some of them also support WebM, the
new royalty-free format from Google. You can find a
good summary of the problem from Mark Pilgrim.
With Chromium as found in Debian, you have Ogg Theora, WebM and
H264. Youtube has an
opt-in program to enable HTML5 video but most other
video content are just unavailable without Flash. Web without Flash is
not a reality yet.
youtube-dlget-flash-videosclive
Note: This is an idea I had, not something I'm working on. If it interests
you, please work on it. I may come back to this some day, but not
anytime soon, but I thought I'd share it anyway.
When downloading updates for a Debian package, often much of the
data doesn't actually change, or at least not much. Because of
this, the debdelta service is a wonderful idea (and Debian
should adopt it officially).
However, a different way of achieving that might be like this:
Half a year ago, I experimented with gtk+ and cairo to extract a PDF screenshot out of a progam, i.e. a screenshot of the window that is not a table of colored dots as a regular PNG file would be, but rather a vector graphic that scales perfectly and has selectable text. The main use that I see is documentation: A scalable screenshots looks much better in print, and even in an online-PDF-file, having the full-text-search also find the text in the images is useful.
My experiments back then were successful in a proof-of-concept way, but they were very ugly hacks. With the new version 3 of gtk+, though, things become much easier, as only one cairo context is used, instead of each widget creating its own context to draw on.
By now, the first gtk-3 applications have reached Debian unstable, so I thought it would be time to re-do the experiment. And the result is very pleasing: The pdf screenshot program comes as a gtk module that you can load by setting an environment variable before launching a gtk-3 program. The application will run as usual, but there is an additional PDF screenshot button. If you click it, you get a Save File dialog (with a nice, live-updated preview of the screenshot, as an additional gimmick) that lets you save a PDF file. Its as easy as that. You can check out the resulting PDF file of gucharmap and a screencast of me creating that file, including copy n pasting from the resulting document.
Half a year ago, I experimented with gtk+ and cairo to extract a PDF screenshot out of a progam, i.e. a screenshot of the window that is not a table of colored dots as a regular PNG file would be, but rather a vector graphic that scales perfectly and has selectable text. The main use that I see is documentation: A scalable screenshots looks much better in print, and even in an online-PDF-file, having the full-text-search also find the text in the images is useful.
My experiments back then were successful in a proof-of-concept way, but they were very ugly hacks. With the new version 3 of gtk+, though, things become much easier, as only one cairo context is used, instead of each widget creating its own context to draw on.
By now, the first gtk-3 applications have reached Debian unstable, so I thought it would be time to re-do the experiment. And the result is very pleasing: The pdf screenshot program comes as a gtk module that you can load by setting an environment variable before launching a gtk-3 program. The application will run as usual, but there is an additional PDF screenshot button. If you click it, you get a Save File dialog (with a nice, live-updated preview of the screenshot, as an additional gimmick) that lets you save a PDF file. Its as easy as that. You can check out the resulting PDF file of gucharmap and a screencast of me creating that file, including copy n pasting from the resulting document.
Next.
