Search Results: "spa"

14 April 2013

Vasudev Kamath: Handling C++ symbols that vary across architectures

Disclaimer I'm no symbol expert and dealing with symbol files for first time from the time I started packaging. What I did here is depending on some suggestions I got in #debian-mentors. If you think what I did was wrong please enlighten me :-). Problem Recently 2 of my library packages pugixml and ctpp2 got accepted into Debian archive and when buildd tried to build them on remaining architectures other than one for which I uploaded (amd64) builds failed. This was expected as symbols file which I generated was for amd64. As usual I got 2 serious bug reports #704718 and #705135. Solution First I was not sure how to handle this. I read article on symbols files handling by rra [1] and tried to use pkgkde-symbolshelper tool only to quickly figure out that I need to use pkgkde_symbolshelper addon for dh sequencer. But this was not possible for me as I was using CDBS for packaging. I did a quick chat on #debian-mentors and some one suggested me to tag symbols which vary across architecture with (c++) tag. First I was not sure but after reading dpkg-gensymbols man page understood that I need to replace the entire mangled symbols lines with de-mangled version tagged with (c++) in the beginning. But this was hectic job searching for each deleted symbols and replacing it. So I thought of writing a script to do the job and after struggling for 3 days (yeah I was bit dumb that I didn't read manual date: "2013-04-14 18:39:20 +0530" pugixml and ctpp2 package using it, which is now waiting for Jonas for uploading it. Here is the script
#!/bin/bash
set +x
if [ $# -lt 3 ]; then
    echo "Usage: $0 failed_buildlogs_directory symbols_file package_version"
    exit 2
fi
    
BUILD_LOG_DIRECTORY=$1
SYMBOLS_FILE=$2
PACKAGE_VERSION=$3
VERSION_TO_REPLACE=" $PACKAGE_VERSION\""
for LOGFILE in $(ls $BUILD_LOG_DIRECTORY/*.build); do
    for i in $(grep '^-\s_Z' $LOGFILE   perl -pe 's/-//g;'); do
        if [ $i = $PACKAGE_VERSION ];then
            continue
        fi
        demangled_version="\""$(echo $i" "$PACKAGE_VERSION   c++filt)"\""
        tagged_version="(c++)"$ demangled_version%$VERSION_TO_REPLACE "\" "$PACKAGE_VERSION
        escaped_tagged_version=$(echo $tagged_version   sed 's/\&/\\\&/')
        sed -i "s#$i $PACKAGE_VERSION#$escaped_tagged_version#" $SYMBOLS_FILE
   done
done
So basically to make this work we need all buildlogs to be downloaded from buildd's again this was easy thanks to rra for developing pkgkde-getbuildlogs :-). Once you have build logs directory run the above script as follows
cppsymbol_replace.sh path_to_buildlogs path_to_symbol_file upstream_version
After replacing symbols I tried to build package on i386 chroot and build passed successfully but lintian told me that there are symbols which have Debian version appended to it and this might lead to date: "2013-04-14 18:42:42 +0530" back to mentors :-). It is this time I really understood concept of mangled names generated by compiler and why it vary across the architecture ;-). This time some one by nick pochu suggested me to pass option -v with package version to dpkg-gensymbols to make it generate symbols with package version and not Debian version. The following probably needs to be done if package uses dh sequencer but I'm not sure as I've not tested it. If wrong please correct me.
override_dh_makeshlibs:
    dh_makeshlibs -- -v$PACKAGEVERION #package version needs to be either extracted using parsechangelog or manually fed
If you are using CDBS this is pretty simple. Just add following to rules
DEB_DH_MAKESHLIBS_ARGS_$(pkgname) += -- -v$(DEB_UPSTREAM_VERSION)
I noticed that when I provide (c++) tagged de-mangled name dpkg-gensymbols simply replaces it with proper mangled name but the deletion doesn't trigger an error in dpkg-gensymbols. This script me allowed to replace 128 symbols which were very tricky and long with de-mangled and tagged version in ctpp2, so I hope it should work across different packages without any problem. Only silly mistake I did was occurrence of & symbol in function making sed go mad which I took one full day to debug :facepalm:. So that's it folks, if you see something wrong in what I did please let me know through the comments. [1] http://www.eyrie.org/~eagle/journal/2012-01/008.html

13 April 2013

Andrew Shadura: A bit more on TrueCrypt

Few days ago I was discussing my last blog post with a colleague of mine, Luka Tvrd , and he s mentioned that it actually is possible to harmlessly decrypt the TrueCrypted filesystem in online mode, resize it, and encrypt it back again. Well, I decided to try, because I have had some problems with that setup. Apart from init script reordering I had to make (not covered in the post, as I did it later and haven t found time yet to write about that), I ran into a trouble while trying to enable swapping. As the only partition I had was formatted into NTFS, placing a swap file there isn t the best idea ever. I actually tried doing so, but after two deadlocks I had in few hours I had to rethink that. Indeed, as NTFS is implemented in the user space using ntfs-3g, as soon as this process gets swapped out, the system is left helpless. The swap is on the file system driver of which is in the swap, which is on the file system driver of which is well, you get the idea. Basically, I got rid of ext3-on-the-loop, and now I have real ext3 in a LUKS container, plus a separate swap partition. Now the other problem is: I still have an encrypted NTFS filesystem which I occasionally need to access, but I don t want to type the pass phrase twice Something to think about.

12 April 2013

Craig Small: Pie Charts in TurboGears

You might of looked at Ralph Bean s tutorial on graphs and thought, that s nice but I d like something different. The great thing about ToscaWidgets using the jqPlot library is that pretty much anything you can do in jqPlot, you can do in ToscaWidgets and by extension in TurboGears. You want different graph types? jqPlot has heaps! My little application needed a Pie Chart to display the overall status of attributes. There are 6 states an attribute can be in: Up, Alert, Down, Testing, Admin Down and Unknown. The Pie Chart would show them all with some sort of colour representing the status. For this example I ve used hard-coded data but you would normally extract it from the database using a TurboGears model and possibly some bucket sorting code. I ve divided my code into widget specific, which is found in myapp/widgets/attribute.py and the controller code found unsurprisingly at myapp/controllers/attribute.py Also note that some versions of ToscaWidgets have a bug which means any jqPlot widget won t work, version 2.0.4 has the fix for issue 80 that explains briefly the bug. The widget code looks like this:
from tw2.jqplugins.jqplot import JQPlotWidget
from tw2.jqplugins.jqplot.base import pieRenderer_js
import tw2.core as twc
 
class AttributeStatusPie(JQPlotWidget):
  """
  Pie Chart of the Attributes' Status """
  id = 'attribute-status-pie'
  resources = JQPlotWidget.resources + [
  pieRenderer_js,
  ]
 
  options =  
  'seriesColors': [ "#468847", "#F89406", "#B94A48", "#999999", "#3887AD", "#222222"], 
  'seriesDefaults' :  
  'renderer': twc.js_symbol('$.jqplot.PieRenderer'),
   ,
  'legend':  
  'show': True, 
  'location': 'e',
   ,
   
Some important things to note are: Next the controller needs to be defined:
from myapp.widgets.attribute import AttributeStatusPie
 
@expose('myapp.templates.widget')
  def statuspie(self):
  data = [[
  ['Up', 20], ['Alert', 7], ['Down', 12], ['Admin Down', 3], ['Testing', 1], ['Unknown', 4],
  ]]
  pie = AttributeStatusPie(data=data)
  return dict(w=pie)
And that is about it, we now have a controller path attributes/statuspie which shows us the pie chart.
My template is basically a bare template with a $ w.display n in it to just show the widget for testing.
Pie Chart in Turbogears

Pie Chart in Turbogears

11 April 2013

Russ Allbery: Review: Familiar

Review: Familiar, by J. Robert Lennon
Publisher: Greywolf
Copyright: 2012
ISBN: 1-55597-535-6
Format: Hardcover
Pages: 205
This is the first book of an experiment. I'm fairly well-read in science fiction and fantasy and increasingly well-read in non-fiction of interest (although there's always far more of that than I'll get to in a lifetime), but woefully unfamiliar with what's called "mainstream" literature. Under the principal that things people are excited about are probably exciting, I've wanted to read more and understand the appeal. Powell's, which I like to support anyway, has a very nice (albeit somewhat expensive) book club called Indiespensable, which sends its subscribers very nice editions of new works that Powell's thinks are interesting, with a special focus on independent publishers. So I signed up and hope to stick with it for at least a year. (The trick will be fitting these books in amongst my regular reading.) Familiar is the first feature selection I received. Elisa Brown is a mother with a dead son and a living one, a failing marriage, an affair, and a life that is, in short, falling apart. Then, while driving back from her annual pilgrimage from her son's grave, the world seems to twist and change. She finds herself dressed for business, wearing a nametag and apparently coming back from a work-related convention, driving a car that's entirely unfamiliar to her. When she gets home, everything else has changed too: her marriage seems to be on firmer ground, but based on rules she doesn't understand. She has a different job, different clothes, a different body in some subtle ways. And both of her sons are alive. I'm going to have to have a long argument with myself about where to (meaninglessly) categorize this on my review site, since in construction it is an alternate reality story and therefore a standard SF trope. Any SF reader is going to immediately assume Elisa has somehow been transported into an alternate reality with a key divergence point from her own. But that's not Lennon's focus. He stays ambiguous on the question of whether this is really happening or whether Elisa had some sort of nervous breakdown, and while some amount of investigation of the situation does take place, it's the sort of investigation that an average person with no access to special resources or scientific knowledge and a completely unbelievable story would be able to do: Internet conspiracy chatrooms and some rather dodgy characters. The focus is instead on Elisa's reaction to the situation, her choices about how to treat this new life, and on how she processes her complex emotions about her family and herself. I had profoundly mixed feelings about this book when I finished it, and revisiting it to review it, I still do. The writing is excellent: spare, evocative, and enjoyable to read. Lennon has a knack for subjective description of emotion and physical experience. The reader feels Elisa's deep discomfort with her changed body and her changed car, her swings between closed-off emotions and sudden emotional connection with a specific situation, and her struggle with the baffling question of how to come to terms with a whole new life. The part of the book from about the middle to nearly the end is excellent. Video games make an appearance and are handled surprisingly well. And when Elisa starts being blunt with people, I found myself both liking her and caring about what happens to her. On the other hand, Familiar also has some serious problems, and one of the biggest is the reaction I feared I'd have to mainstream literature: until Elisa started opening up and taking action, I found it extremely difficult to care about anyone in this book. They're all so profoundly petty, so closed off and engrossed in what seem like depressing and utterly boring lives. I'm sure that some of this is intentional and is there to lay the groundwork for Elisa's own self-discovery, but even towards the end of that self-discovery, everything here is so relentlessly middle-class suburbia that I felt stifled just reading about it. I think it's telling that no one in this book ever seems to have any substantial problem with money, or even with work. Elisa walks into a job that she's never done before and within a few weeks is doing it so well that she can take large amounts of time to wander around for plot purposes. This is a book about highly privileged people being miserable in a bubble. While those people certainly do exist, and I can believe that they act like this, I'm not sure how much I want to read about them. Thankfully, the plot does lead Elisa to poke some holes in that bubble, if never get out of it entirely. This is also another one of those stories in which every character has massive communication problems. Now, this deserves some caveats: Elisa's communication problems with her husband are part of the problem that starts the book and are clearly intentional, as are her communication difficulties with her children. And she's not really close enough to anyone to confide in them. But even with those caveats, no one in this book really talks to anyone else. It's amazing that anyone forms any connections at all, given how many walls and barriers they have around themselves. As someone with a bit of a thing for communication, this drove me nuts to read about, particularly in the first half of the book. But the worst problem is that Lennon completely blows the ending. And by that I don't just mean that I disliked the ending. I mean the ending is so unbelievable and so contrary to the entire rest of the book, at least the way I was reading and understanding it, that I think Familiar is a much better novel if you just remove the final scene entirely. It was such a bizarre and unnecessary twist that I found it infuriating. I don't want to spoil an ending, even a bad ending, so I'll only say this: it felt to me like Lennon just wasn't comfortable with his setting and plot driver and couldn't leave it alone. I think an experienced SF author wouldn't have made this mistake. There were two obvious possible conclusions to draw from the setting, plus a few interesting combinations, and I think someone comfortable with this sort of alternate reality story would have taken one of those options, any of which would have been a reasonable dismount for the plot. Alternately, they could have left it entirely ambiguous to the end and explored why the explanation may not actually matter. But Lennon seemed to me to have a tin ear for plausibility and for the normal flow of this sort of story and seems to have taken it as license for arbitrary events, thus completely violating the internal consistency and emergent rules that he'd spent the rest of the book building. I've mostly talked about my reactions to the characters and the writing and have not said much about the plot. That's somewhat intentional, since figuring out where the story will go is one of the best parts of this book. It's surprisingly tense and well-crafted for not having that much inherent dramatic tension. The excellent writing kept me reading through the first part, when I hated everyone in the story, and then Elisa started taking responsibility for her own life and actions and I started really enjoying the book while being constantly surprised. I think it's the sort of story that's best to take without too much foreknowledge of where it's going. I'm going to call this first experiment a qualfied success. Familiar was certainly interesting to read, and quite different from what I normally read despite the SF premise. If it weren't for the ending, I'd be recommending it to other people. Rating: 6 out of 10

Bdale Garbee: Batteries and Pyro Circuits

Keith and I have discovered a change in the behavior of the protection circuits integrated in the LiPo batteries we sell for use with Altus Metrum products that poses a risk for our customers. This post is meant to document what we now know, communicate changes we're planning as a result, and explain what we think flyers of our existing electronics and batteries may want to do to maximize their chances of successful flights. Background Choosing batteries and designing pyro circuits for high power rocketry avionics involves a variety of trade-offs. Reliability is the highest concern, both because nobody wants to lose an airframe due to a failed pyro event, and also because airframes recovering anomalously have safety implications. But we also care about other factors including cost and weight, and usually want to minimize the complexity of both the electronics and the overall installation. The objective of a pyro circuit is to dump enough energy rapidly into an electric match to cause it to catch fire. We need batteries both to power the electronics that decide when to fire the charge, and to provide the energy that actually ignites the match. The two most common battery types seen in the rocketry hobby are alkaline cells, often the nominal 9V rectangular variety, and rechargeable batteries based on Lithium Polymer (LiPo) chemistry. LiPo cells are 3.7 volts per cell nominal voltage, are very light, and have a high energy density. LiPo capacity is measured in units of current times time, so an 850mAh cell should be able to deliver 850 milliamps for an hour. The battery industry also uses something called a "C rate" to describe how fast the battery can be usefully discharged, wich is a multiplier relative to the battery capacity. So a battery with 850mAh capacity and a "2C" rating can deliver current at a sustained rate of 1700mA until discharged, while the same capacity at a "5C" rating can deliver 4250mA. By comparison, most 9V alkaline batteries are actually composed of 6 individual 1.5V cells enclosed in a wrapper. It's hard to get hard numbers for capacity and discharge rate, since in an alkaline cell the two are not independent, and the discharge rate is related to the volume of each individual cell. The data sheet for an Energizer 522 shows just over 600mAh at a 25mA discharge rate, dropping to about 300mAh at a 500mA discharge rate. Importantly for use in pyro circuits, LiPo cells have a very low source impedance, which means they can source immense amounts of current. It's not unusual for cells in the 1000mAh range to have ratings in excess of 30C! Because this rapid discharge ability can pose a risk of fire, it's common for LiPo cells to come with a "protection board" integrated into the battery assembly that is designed to limit the current to some rate such as 2C continuous duty. In large airframes, or projects that involve staging, air-starts, or other complex pyro event sequences, the most reliable approach will always be to use separate batteries for the control electronics and the source of pyro firing power. In the limit, having separate pyro batteries for each pyro charge with the control electronics only providing the switching to connect the batteries to the charges could even make sense. But for most airframes, this is overkill, and the increases in mass, volume, and wiring complexity just don't make sense. The challenge, then, is how to design electronics that will robustly initiate pyro events without negatively affecting the rest of the electronics when operating from a single shared battery. Altus Metrum Pyro Circuits The very first prototypes of TeleMetrum were designed to use a single-cell LiPo battery, and had an on-board 100mA charging circuit. Because we needed 5 volts to power the accelerometer, we had a small switching regulator that up-converted the LiPo voltage, and we used some of that regulator's output to charge a 1000uF capacitor. The pyro circuit used Fairchild FDN335N N-channel MOSFET switches in a low-side switching configuration to dump the energy stored in the capacitor through an attached ematch. Those FETs had an on resistance of under a tenth of an ohm in our operating conditions. The circuit worked very reliably, but the 1000uF electrolytic cap was huge and we struggled with the mechanics of such a large part hanging off the board... It turns out that 3.7 volts is way more than enough to fire a typical low-current e-match or equivalents like the Quest Q2G2 igniters. In fact, bench testing with a good digital oscilloscope showed that a typical e-match with resistance of 1.3-1.8 ohms will fire in approximately 13 microseconds when hit with the nominal 3.7 volts from a LiPo. So, starting with our v0.2 boards, we switched to using the LiPo battery voltage directly to fire the pyro charges, eliminating the clunky electrolytic capacitor entirely. We also switched to the Fairchild FDS9926A dual N-channel MOSFET whose specs are better in all regards for our application. The on resistance is down around 40 milli-ohms in our circuit, such that the current ratings are much higher (FET current limits are primarily driven by how much heat is generated due to current flowing through the channel's on resistance). Because using the LiPo voltage directly means we're effectively temporarily putting a very low resistance across the battery during the pyro events, the input voltage to the voltage regulator gets pulled down. To ensure the processor could "ride through" these events, we added a 100uF surface mount bulk capacitor on the 3.3 volt regulated voltage rail, which bench testing demonstrated was more than sufficient to maintain processor operation through pyro events. And that is essentially the same pyro circuit on all the boards, both TeleMetrum and TeleMini, that we have shipped to date. What's Changed The LiPo batteries we source and sell with our electronics come with a protection board and cable terminated in a 2-pin, 2mm "JST" connector. The specs on the protection board have always been "2C continuous", but we observed the ability to source much higher currents for short periods such as the 13 micro-seconds or so required to fire an e-match. Thus these protection circuits seemed just fine .. we could fire e-matches with a burst of current but were protected against short circuits in the wiring or our boards by the 2C continuous limit. Unfortunately, the most recent batch of batteries we sourced seem to have a much "twitchier" protection circuit. We can draw more than 2C for short bursts, but not as much as with prior boards, and not for as long an interval. With a 1 ohm power resistor on the pyro terminals of one of our boards, we get about 9 milliseconds of power before the protection circuit cuts in and shuts the battery down. The power stays down until all load is removed, which at least means the board is turned off and back on again, and in some cases could even mean the battery is unplugged and re-plugged since we draw trace current to keep the GPS memory alive even when the power is turned off, and at least some of the new batteries see that as enough to keep the power turned off after an over-current event. For many e-matches, this isn't an issue, since 9 milliseconds is way longer than the 13 microseconds needed to fire the charge. Unfortunately, we've discovered that many of the e-matches bought and used in the rocketry hobby are actually made for use in the fireworks industry, where it is desireable to retain continuity after firing so that series connections of e-matches all can fire even if some fire faster than others! This means that while the resistance goes up some after firing, sometimes the drain on the battery remains sufficient to cause the protection circuit to kick in even after the pyro charge has fired. What We're Doing About It If we remove the protection circuit from the LiPo (or jumper around it), all existing Altus Metrum products will operate successfully with pyro charges thave have an effective resistance of as low as 1 ohm. That's lower than any e-match or Q2G2 we've ever seen, so in effect what this means is that if you have an existing Altus Metrum flight computer, and you remove the LiPo protection circuit, you're good to go. This does not really make things any more "dangerous", since our battery chargers are all current limited and our discharge patterns will never cause heating of the battery. Frankly, in a rocket, the most likely way to cause a problem with a LiPo is by smashing or puncturing the actual battery during bench work or during a crash... and those cause the same problems with or without a protection board present. In the future, we will ship batteries that have either a much higher C rating on the protection circuit, or have no protection circuit at all. The number of problems reported by actual customers that we think should be attributed to LiPo protection circuit boards is very low, and we suspect most of our customers who are happily flying their boards can continue to do so. Ground testing where you fire pyro charges (or at least e-matches) using RF to issue the commands (not USB, since the LiPo charger is running any time USB is connected!) will confirm whether there's a problem. If the board resets (does startup beeps) after a pyro event, or shuts down completely (no LED activity), then you have a problem. If the matches light and the board keeps running, you're good to go. However, any Altus Metrum customer with LiPo batteries sourced from us or our distributors who is worried about this problem (even if you haven't seen problems in ground testing or previous flights), and who doesn't want to try soldering on the battery circuit board yourself, is welcome to contact us about removing the protection circuit for you. We won't charge anything other than shipping. To take advantage of this offer, just send email to fixmybattery@altusmetrum.org telling us how many of which capacity batteries you have that you'd like updated, and we'll respond with an RMA number and shipping details. Going Even Further As previously indicated, with the LiPo protection circuits removed, all of our current products will work reliably with at least 1 ohm across the pyro terminals. That should cover all real-world flying conditions just fine, but we're not satisfied yet. We've designed a new pyro control circuit that transfers the maximum possible energy to the load regardless of battery voltage without ever allowing the voltage to the processor to droop at all. We're testing this new circuit in various prototypes now, and if it pans out it will probably show up first in MegaMetrum and then trickle down to TeleMetrum and TeleMini as those products are updated later this year. The new pyro circuit tolerates 0 ohms (dead shorts) on the pyro terminals for as long as the battery can provide current, which is as good as it gets. We think the circuit is clever enough that we'll probably write more about it once we're finished validating it.

8 April 2013

Daniel Pocock: Google Summer of Code, tips for students

I've had quite a few queries about the project ideas for Google Summer of Code on the Lumicall and Debian project pages and I'm publishing some notes here to help people get the best chance of participating, while also helping people's expectations. Firstly, Google is responsible for administering the program and payments, they have the last word on which projects and students participate and free software developers like myself have no direct control over this. In other words, while I would encourage students to make contributions to free software projects to familiarize themselves with the way we work, I can not promise that this will automatically lead to selection. If you are a student, do go ahead and feel free to comment on or make contributions to projects, this helps mentors to see your capabilities very quickly, but don't do any more than you would have done if the GSoC program didn't exist. In any case, building up a portfolio of work that is visible on sites like github does help your employment prospects in the long run. If a development manager is looking at 100 CVs for a job vacancy, and one of them has a strong portfolio on github or Debian, there is a good chance that one person will get an interview - although it doesn't guarantee they get the job. Where to start? If you are a student and you are already heavily involved in some free software project then you probably don't need to read the rest of this. The reality is that GSoC is meant to help bring more people into the free software community and this means we need to help people get their foot in the door. That is what I am addressing here, with examples from my own possible projects. When I went through university in the 90s, there was a very strong UNIX culture in our computer science department and the wider engineering faculty. There were also winds of change, for example, a new dean who wanted to change everything to Windows and a central IT management agenda to abolish campus computers and force students to carry and maintain their own laptops. Around the world, every campus has a different approach. I was fortunate to have UNIX topics like the vi text editor, shell programming and even LaTeX taught as a formal part of the course, students coming after me may have missed those opportunities. I understand that some students have not encountered these concepts at all today. That said, most mentors have a fairly wide tolerance for these things: it is not expected that students know these things, rather, it is an opportunity for you to start learning them whether it is part of your course or not. The key concept here is to be a self-starter: it is not what you know, but your enthusiasm to go and try the things you don't know. That is the very thing you can start doing now, even before the program commences. Doing this will definitely help you identity the areas of free software and Debian that you are most likely to get excited about, and that will lead to a successful project. Some general ideas to get started
  • If your campus hasn't taught UNIX in a formal sense, please familiarise yourself with the basics. While some companies do have more jobs working with other platforms, and some campuses have sadly tried to adapt their courses to prepare students for that limited subset of companies, all the best jobs I've had involve UNIX or Linux. I've never worked for a hedge fund that didn't use Linux. Hollywood uses Linux, NASA does too, but learning Linux is not rocket science. Basic things to understand include the vi text editor, shell scripts and using a command line tool for version control, preferably git. As you will be the system administrator of your own desktop or laptop, it's essential to understand some basic concepts like volume management too.
  • If you need to run some Windows software (some courses do require that), use VirtualBox or a similar solution to run both Windows and Linux on your computer or laptop. The VirtualBox documentation is very good and will help you start quickly. My own laptop runs Debian 7 (wheezy) as the main operating system, and I have Windows and Solaris in VirtualBox for some business applications.
  • My own course included a project that involved downloading the open source GNU make program, writing a patch and testing it. If you've never done something like that, please try. Instead of downloading a project tarball, these days you will typically clone the project's git repository. Even if you only make some trivial change that you will never share, it is good to prove that you can compile and successfully execute something like this.
Some specific ideas for my own projects
  • For Lumicall or Android development, just work your way through the build instructions that are published for all contributors. Trying the f-droid build instructions is also a good way to start.
  • For any of the Debian projects, please make sure you have installed Debian 7 (wheezy) on a desktop, laptop or virtual machine. Just follow the installation guide
  • Once you have installed Debian, try installing whichever one of my packages interests you. Try to configure and use it. Prove that the package works in a normal installation. Look at the log files. Run the test cases.
  • Next, try obtaining the source code and building the package. Debian makes it very easy. Here is an example for reSIProcate, the same works for any other package. As root:
    # apt-get update
    # apt-get build-dep resiprocate
    
    will obtain all the necessary build tools. Then, as a normal user:
    $ mkdir ~/src/resiprocate
    $ cd ~/src/resiprocate
    $ apt-get source resiprocate
    $ cd resiprocate
    $ dpkg-buildpackage -rfakeroot -j12
    
    In a few minutes, you should find the packages (*.deb files) are created.
  • Now, you could try making some small change to the package and compiling it again. Verify that your change works.
If you've successfully got to this point, then there is every chance you will be able to successfully complete a GSoC project. If you struggled along the way, that is OK too, the important thing to do is to engage with the community and learn how to ask for help. The Debian GSoC wiki provides some excellent ideas about engaging with the project using IRC chat and mailing lists. It is also very worthwhile trying to find a local group or a free software event that Debian is attending near you and spending a few hours with people who can look at any problems you have and give you quick solutions.

4 April 2013

Thomas Goirand: Git packaging workflow

Seeing what has been posted recently in planet.d.o, I would like to share as well my thoughts and work-flow, and tell that I do agree with Joey Hess on many of his arguments. Especially when he tells that Debian fetishises upstream tarballs. We re in 2013, at the age of Internet, and more and more upstream authors are using Git, and more and more they don t care about releasing tarballs. I ve seen some upstream authors who simply stopped doing so completely, as a Git tag is really enough. I also fully agree than disk space and network speed isn t much of a problem these days. When there are tags available, I use the following debian/gbp.conf:
[DEFAULT]
upstream-branch = master
debian-branch = debian
upstream-tag = %(version)s
compression = xz
[git-buildpackage]
export-dir = ../build-area/
On many of my packages, I now just use Git tags from upstream if they are available. To make it more easy, I now nearly always use the following piece of code in my debian/rules files:
DEBVERS         ?= $(shell dpkg-parsechangelog   sed -n -e 's/^Version: //p')
VERSION         ?= $(shell echo '$(DEBVERS)'   sed -e 's/^[[:digit:]]*://' -e 's/[-].*//')
DEBFLAVOR       ?= $(shell dpkg-parsechangelog   grep -E ^Distribution:   cut -d" " -f2)
DEBPKGNAME      ?= $(shell dpkg-parsechangelog   grep -E ^Source:   cut -d" " -f2)
DEBIAN_BRANCH   ?= $(shell cat debian/gbp.conf   grep debian-branch   cut -d'=' -f2   awk ' print $1 ')
GIT_TAG         ?= $(shell echo '$(VERSION)'   sed -e 's/~/_/')
get-upstream-sources:
        git remote add upstream git://git.example.org/proj/foo.git   true
        git fetch upstream
        if ! git checkout master ; then \
                echo "No upstream branch: checking out" ; \
                git checkout -b master upstream/master ; \
        fi
        git checkout $(DEBIAN_BRANCH)
make-orig-file:
        if [ ! -f ../$(DEBPKGNAME)_$(VERSION).orig.tar.xz ] ; then \
                git archive --prefix=$(DEBPKGNAME)-$(GIT_TAG)/ $(GIT_TAG)   xz >../$(DEBPKGNAME)_$(VERSION).orig.tar.xz ; \
        fi
        [ ! -e ../build-area ] && mkdir ../build-area   true
        [ ! -e ../build-area/$(DEBPKGNAME)_$(VERSION).orig.tar.xz ] && cp ../$(DEBPKGNAME)_$(VERSION).orig.tar.xz ../build-area   true
Packaging a new upstream VERSION now means that I only have to edit the debian/changelog, do ./debian/rules get-upstream-source so that I get new commits and tags, then git merge -X theirs VERSION to import the changes, then finally invoke ./debian/rules make-orig-file to create the orig.tar.xz. My debian branch is now ready for git-buildpackage. Note that the sed with the GIT_TAG thing is there because unfortunately, Git doesn t support the ~ char in tags, and that most of the time, upstream do not use _ in version numbers. Let s say upstream is releasing version 1.2.3rc1, then I simply do git tag 1.2.3_rc1 1.2.3rc1 so that I have a new tag which points to the same commit as 1.2.3rc1, but that can be used for the Debian 1.2.3~rc1-1 release and the make-orig-file. All this might looks overkill at first, but in fact it is really convenient and efficient. Also, even though there is a master branch above, it isn t needed to build the package. Git is smarter than this, so even if you haven t checked out upstream master branch from the upstream remote, make-orig-file and git-buildpackage will simply continue to work. Which is cool, because this means you can store a single branch on Alioth (which is what I do).

Michal Čihař: phpMyAdmin translations status

phpMyAdmin 4.0-rc1 is out and it's really time to work on translations if you want them to be ready for final release.. So let's look at which translations are at 100% right now (new ones are bold): Almost complete: As you can see, there is still lot of languages missing, this might be your opportunity to contribute to phpMyAdmin. Also you are welcome to translate phpMyAdmin 4.0 using translation server. If your language is already fully translated and you want to help as well, you can translate our documentation as well.

Filed under: English Phpmyadmin 0 comments Flattr this!

3 April 2013

Luke Faraone: Teaching free/open source to high school students

A few weeks ago I taught a class on Open Source: Contributing to free culture (catalog entry) for Spark, a one-day program put on by the student-run MIT Educational Studies Program. I was fortunate to have two helpful co-teachers, Tyler Hallada and Jacob Hurwitz, who assisted with the lesson plan and the in class lecture.

We ended up teaching 3 sessions of the 1hr 50min class that Saturday, with about 10 students in each session.

I was pretty impressed by the quality of the students; a number of them had used GNU/Linux before, but even those who hadn't were able to gain something from the experience. The class was broken up into three segments:

  1. Lecture on a brief history of open source and the free software movement
  2. Small research project on an open source project
  3. Lab where students could work through OpenHatch's training missions
The point was to mix up what could otherwise be a very boring lecture.

I think we might have missed the mark on the last bit, as I get the feeling that we didn't end up giving the students good actionables. While the quality of OpenHatch is high and the organization's campus outreach programs are amazing, skills practice only goes so far without clear direction to apply said skills. I'll be following up with the class participants to see how they're progressing on their own open source contributor journey, and will post updates if I have any.

While not an OpenHatch event, if this sort of thing interests you, OpenHatch runs a series of events like this one and has a mailing list for discussing planning and sharing best practices. Subscribe and say hi!

The presentation is enclosed below, and of course is licensed under CreativeCommons Attribution-ShareAlike 3.0. [PDF]

<iframe allowfullscreen="true" frameborder="0" height="389" mozallowfullscreen="true" src="https://docs.google.com/presentation/d/14R1o_5rOfjCw19mFtxZj29HQOhlJWK0F53MNj5tL8iU/embed?start=false&amp;loop=false&amp;delayms=3000" webkitallowfullscreen="true" width="480"></iframe>

Julien Danjou: Hy, Lisp in Python

I've meant to look at Hy since Paul Tagliamonte started to talk to me about it, but never took a chance until now. Yesterday, Paul indicated it was a good time for me to start looking at it, so I spent a few hours playing. But what's Hy? Python is very nice: it has a great community and a wide range of useful libraries. But let's face it, it misses a great language. Hy is an implementation of a Lisp on top of Python. Technically, Hy is built directly with a custom made parser (for now) which then translates expressions using the Python AST module to generate code, which is then run by Python. Therefore, it shares the same properties as Python, and is a Lisp-1 (i.e. with a single namespace for symbols and functions). If you're interested to listen Paul talking about Hy during last PyCon US, I recommend watching his lightning talk. As the name implies, it's only a few minutes long. <iframe allowfullscreen="allowfullscreen" class="shadow" frameborder="0" height="374" src="http://www.youtube.com/embed/1vui-LupKJI?wmode=transparent&amp;autohide=1&amp;egm=0&amp;hd=1&amp;iv_load_policy=3&amp;modestbranding=1&amp;rel=0&amp;showinfo=0&amp;showsearch=0#t=16m14s" width="500"></iframe> Does it work? I've been cloning the code and played around a bit with Hy. And to my greatest surprise and pleasure, it works quite well. You can imagine writing Python from there easily. Part of the syntax smells like Clojure's, which looks like a good thing since they're playing in the same area. You can try a Hy REPL in your Web browser if you want. Here's what some code look like:
(import requests)

(setv req (requests.get "http://hy.pault.ag"))
(if (= req.status_code 200)
(for (kv (.iteritems req.headers))
(print kv))
(throw (Exception "Wrong status code")))

This code would ouput:
('date', 'Wed, 03 Apr 2013 12:09:23 GMT')
('connection', 'keep-alive')
('content-encoding', 'gzip')
('transfer-encoding', 'chunked')
('content-type', 'text/html; charset=utf-8')
('server', 'nginx/1.2.6')
As you can see, it's really simple to write Lispy code that really uses Python idioms. There's obviously still a lots of missing features in Hy. The language if far from complete and many parts are moving, but it's really promising, and Paul's doing a great job implementing every idea. I actually started to hack a bit on Hy, and will try to continue to do so, since I'm really eager to learn a bit more about both Lisp and Python internals in the process. I've already send a few patches on small bugs I've encountered, and proposed a few ideas. It's really exciting to be able to influence early a language design that I'll love to use! Being a recent fan of Common Lisp, I tend to grab the good stuff from it to add them into Hy.

Joey Hess: upstream git repositories

Daniel Pocock posted The multiple repository conundrum in Linux packaging. While a generally good and useful post, which upstream developers will find helpful to understand how Debian packages their software, it contains this statement:
If it is the first download, the maintainer creates a new git repository. If it has been packaged before, he clones the repository. The important point here is that this is not the upstream repository, it is an independent repository for Debian packaging.
The only thing important about that point is that it highlights an unnecessary disconnect between the Debian developer and upstream development. One which upstream will surely find annoying and should certainly not be bothered with. There is absolutely no technical reason to not use the upstream git repository as the basis for the git repository used in Debian packaging. I would never package software maintained in a git repository upstream and not do so. The details are as follows: So using the upstream git repository is valuable. And there is absolutely no technical value, and plenty of potential friction in maintaining a history-disconnected git repository for Debian packaging.

Daniel Pocock: The multiple repository conundrum in Linux packaging

I'm involved with a number of free software projects as both a developer and as the maintainer of packages for various distributions such as Debian (which also feeds packages to Ubuntu) and OpenCSW. I regularly come across the following situations:
  • Developers of great software who would like to see it packaged, distributed and promoted conveniently through platforms like Debian
  • Users of Linux distributions who are keen to use free software if it is presented in a convenient and accessible manner they are familiar with.
Sadly, despite everybody having the best intentions, there is sometimes a chasm separating these two groups of people. Upstream developers are often busy developing new features and don't have time to work on the intricacies of packaging. I hope that by sharing a few of my own experiences I can help more developers get their software packaged more easily. Fortunately, a number of great tools like git-buildpackage have emerged for streamlining the packaging process, but this has also created more confusion for developers who have their own git repositories and don't quite understand how the Debian git repository and patching process relates to their own repository. The autotools world Here, I focus on autotools based software, because this type of software has it's own peculiar issues when packaging. In particular, these issues appear when using a version control system to track upstream releases. Some of the concepts can be applied to the study of regular Makefile or cmake projects as well. Here is a diagram giving an overview: Let's work through each of the steps in the diagram: The upstream release
  1. The developer/release manager updates the version number in
    configure.ac
    (sometimes called configure.in) and tags the code. (Usually this tag is on a dedicated release branch.)
  2. The developer checks out a copy of the code from the tag into a fresh working directory
  3. The developer runs the autoreconf/automake tools, usually from a bootstrap script. These tools create a number of new files that don't exist in the project repository. Finally, the developer runs
    make dist
    , which puts all the files,
    including the generated files, into a distribution tarball. It is worth emphasizing this point: the tarball is not just an archive of the files from the repository/tag, it also contains a number of
    files generated by autotools.
  4. The developer uploads the tarball to a web site such as the Sourceforge download page. Usually a release announcement is made now containing checksums for the tarball.
At this point, the upstream developer's work is done and packaging teams from various projects such as Debian will take over. Sometimes, the upstream developer is also building the packages and continues
onto the next steps himself: Packaging
  1. The package is downloaded by the package maintainer
  2. If it is the first download, the maintainer creates a new git repository. If it has been packaged before, he clones the repository. The important point here is that this is not the upstream repository, it is an independent repository for Debian packaging. The maintainer uses the git-import-orig tool to import the upstream tarball into the packaging repository. The git-import-orig tool captures an exact snapshot of the upstream release tarball contents in a branch called upstream. One point where the Debian repository differs fundamentally from the upstream repository is that all files from the tarball will be tracked in the Debian git repository, even those automatically generated files that were created by autotools and don't exist in the upstream repository.
  3. The maintainer creates or updates the various artifacts for packaging. These files are kept on the master branch, and the tarball contents from the upstream branch are merged into master to create packages.
  4. When the maintainer feels the code is ready, he will check out a clean copy of the repository to build the package from.
  5. The maintainer executes a tool such as git-buildpackage or regular dpkg-buildpackage, which creates the *.deb files.
  6. The files are checked with a tool like lintian and some manual testing/installation. If all is OK, a tag is made in the packaging repository, with a suffix appended to the upstream version number to indicate which iteration of the package it applies to.
AttachmentSize
release-packaging-workflow.dia3.97 KB

Paul Tagliamonte: Here s my talk from PyCon 2013 (I tried to queue it up to...

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="299" src="http://www.youtube.com/embed/1vui-LupKJI?wmode=transparent&amp;autohide=1&amp;egm=0&amp;hd=1&amp;iv_load_policy=3&amp;modestbranding=1&amp;rel=0&amp;showinfo=0&amp;showsearch=0#t=16m14s" width="400"></iframe>

Here s my talk from PyCon 2013 (I tried to queue it up to where I start (@ 16:14)

A bit more giggley then I intended, but apparently it s endearing.

If anyone s actually interested in Hy, check out #hy on irc.freenode.net, star the code @ git.io/hy - it s actually pretty feature complete. If you want to play with it, check out the magic repl I ve been pondering taking it a bit more serious, so there is even a bit of documentation thanks to Chris Webber (hey, thanks!). Rock on!

2 April 2013

Hideki Yamane: re-compress debian.tar. gz.bz2

You can use bzip2 to compress debian.tar source file in debian/source/options. But usually xz is more efficient than bzip2 (in my experience, and I love xz :) So, how about re-compressing those files with xz?

This is the result.

2.1G tar
503M current
414M xz_6e
410M xz_9e

non-compressed file is 2.1GB and we've compressed to 0.5GB. If we would use xz, it becomes 0.4GB.

debian.tar.bz2 files are 6.3M, if we would use gz, it becomes 40M, it means that package maintainers' choice is right.

6.3M bz2
40M gz
5.8M xz

Wouter Verhelst: Switching away from mutt

After having used mutt for about a decade, I decided it's time I move away from it, for various reasons that I won't go into here. I've been evaluating thunderbird^Wicedove now for a few weeks, but I can already see I won't keep using it forever, because I find too many issues with it:
  1. I can't tell the thing that I never ever ever want to write HTML-formatted emails. While there is an option "Compose messages in HTML format" in the "Account settings" dialog, that only seems to impact the behaviour of the client upon "new mail", not its reply behaviour, forcing me to go to Options->Format and switch it off.
  2. To make matters worse, the HTML editor and the plain-text editor are not the same dialog. That means it is not possible to switch on HTML formatting if it wasn't on originally (not a problem for me, but I can imagine it being a problem for others), but also that the fugly quoting style (vertical line next to the text) of the HTML editor is not removed when you switch from HTML to plain-text. Further on, it has issues when you want to remove white space, where it will remove entire quoting levels rather than just an empty line. This is horrible, horrible behaviour.
  3. While there is support for multiple email addresses connected to the same IMAP account, it does appear that there is no support for automatically choosing the right email address based on user-defined rules, such as "which mail folder are we currently in". Having to remember that I need to select the right mail address is tiresome and error-prone; e.g., when I'm in a Debian-related mail folder, I want my mail client to use my Debian mail address automatically. It appears the only way in which icedove can automatically select sender addresses is by checking the recipient list in the To: and Cc: headers for any of its configured mail addresses, and using that to decide what to put in the from: header.
Any one of the above would eventually be enough for me to grow tired of icedove and switch to something else; but the three combined mean that I'll be looking for something else fairly soon. Suggestions are, of course, welcome.

1 April 2013

Benjamin Mako Hill: Mystery Hunt 2013

A few months late, perhaps, but I wanted to mention that my team (Codex) competed, once again, in the MIT Mystery Hunt. The prize for winning is the responsibility of writing the hunt next year. After being on the 2012 writing team I have mixed feelings about the fact that we did not win again. Although I did not walk away with another coin, I did manage to make an appearance in a multimedia story that the MIT Tech shot about the hunt which provides nice introductions to those who are not familiar with it. <iframe allowfullscreen="" frameborder="0" height="315" src="http://www.youtube.com/embed/R-tmBfZ8CwQ" width="560"></iframe> It was fun to reflect a little bit on why I find the hunt so fun. I said:
In my day job, I work on a lot of problems that maybe don t have answers. It s really awesome to work in a space where if you put in 1-5 hours of mental energy, you will have a solution. Someone has test-solved for it, you know that it s solvable; you know that there is an answer and that there was designed to be an answer. There s something really satisfying about being able to do that.
If you re looking for a team to hunt on next year, feel free get in contact codex is generally quite open to new members.

Russ Allbery: Review: Echo

Review: Echo, by Jack McDevitt
Series: Alex Benedict #5
Publisher: Ace
Copyright: November 2010
Printing: November 2011
ISBN: 1-937007-00-6
Format: Mass market
Pages: 367
This is the fifth book in the ongoing Alex Benedict series, but reading the previous volumes is generally optional. While there are references to earlier events (and indeed I resorted to Wikipedia at one point to refresh my memory of the plot of the the first book in the series), they aren't critical to the plot. They mostly are background for the reactions of other characters to Benedict and his firm. Echo opens with the now-familiar prologue, giving us a fleeting and limited glimpse into the mystery that Benedict and Chase Kolpath will be pursuing for the rest of the book. This time, it features the explorer Somerest Tuttle, who has spent most of his life looking for intelligent alien life. (In the far future setting of these books, humanity has spread through a large region of the galaxy but has only encountered one other alien race.) In the twilight of his career and life, a friend who works as a tour guide for astronomical tours comes to him with a story about something she saw. What, we don't know, and won't for most of the book. But, whatever it is, Tuttle never goes public with it, which is directly contrary to what everyone would expect him to do with any news about alien life. Benedict and Kolpath get involved some years later, after Tuttle's death, when a stone slab with unknown markings that was part of his estate shows up on the future equivalent of Craigslist, free to the first person who will haul it away. Benedict finds that interesting enough to go pick up, but then is really hooked when someone else goes to considerable lengths to try to make the slab disappear before anyone can get a good look at it. This, slowly, leads into poking around Tuttle's life and connections, the star tour industry, and the possible origins of the slab. If you've read any of the previous books in this series, Echo is more of the same, and I mean that in a good way. I think McDevitt is at his best when writing this sort of cross between scientific puzzle, mystery, and non-military adventure, full of people who care deeply about knowing and understanding things and don't like to be thwarted in their efforts. There is McDevitt's trademark halting and red-herring-ridden investigation that succeeds in being engrossing and compelling, plenty of opportunities to guess what's really going on (a bit easier this book than in some others), and a bit of action that I thought was better-handled than the somewhat strained action scenes of The Devil's Eye. The book moves right along, despite multiple reversals in the progress of the investigation, and the final revelation does indeed justify the strange behavior of the various people Benedict encounters. As with McDevitt in general and this series in particular, there is one caveat: the astronomical details are often interesting, but the world building in general is not terribly believable. Despite supposedly taking place millennia into the future, the society is a straight transplant (projection would imply more change than exists) of middle-class American culture, with bonus air cars, space ships, and slightly better computers. Similarly, alien ecologies suffer from severe Star Trek problems: all the planets feel like copies of Earth with a few minor tweaks. This just goes with the territory, and if you're up to the fifth book of the series, you've probably gotten used to it, but the unrealistic lack of divergence is actually used as a plot point here. That strained my suspension of disbelief a bit. That said, this sort of extremely familiar social context does give the books an oddly calm, comfortable feeling. It's tempting to describe McDevitt as cozy science fiction. There's a bit of danger, a bit of suspense, but one is fairly certain that everything will work out in the end and the reader will leave the book knowing the solutions to most of the mysteries raised without having to work very hard at understanding the society or the puzzle. If that's what you're in the mood for (and I was when I read Echo), McDevitt delivers reliably. If you like the Alex Benedict series in general, recommended, as this is an excellent example of the type. Rating: 8 out of 10

30 March 2013

Joey Hess: Goodreads vs LibraryThing vs Free software

Four years ago I started using Goodreads to maintain the list of books I've read (which had lived in a flat text file for a decade+ before that). Now it's been aquired by Amazon. I doubt it will survive in its current form for more than 2 years. Anyway, while Goodreads has been a quite good way to find what my friends are reading, I've been increasingly annoyed by the quality of its recommendations, and its paucity of other features I need. It really doesn't seem to help me keep up with new and interesting fiction at all, unless my friends happen to read it. So I looked at LibraryThing. Actually, I seem to have looked at it several times before, since it had accounts named "joey", "joeyh", and "joeyhess" that were all mine. Which is what happens to me on sites that lack Openid or Browserid. Digging a little deeper this time, I am finding its recommendations much better than Goodreads' -- although it seems to sometimes recommend books I've already read. And it has some nice features like tracking series, so you can easily tell when you've read all the books in a series or not. The analytics overall seem quite impressive. The UI is cluttered and it seems to take 5 clicks to add and rate a single book. It supports half stars. Overall I get the feeling this was designed for a set of needs that doesn't quite match mine. For example, it seems it doesn't have a single database entry per book; instead each time I add a book, it seems to pull in data from primary sources (library of congress, Amazon cough) and treat this as a separate (but related) entry somehow. Weird. Perhaps this makes sense to say, librarians. I'm willing to adjust how I think about things if there's an underlying reason that can be grasped. There's a quite interesting thread on LibraryThing where the founder says:
Don't say we should open-source the code. That would be a nightmare! And I have limited confidence in APIs. LibraryThing has the book geeks, but not so much the computers geeks.
I assume that the nightmare is that there would be dozens of clones of the site, all balkanized, with no data transfer, no federation between them. Except, that's the current situation, as every Goodreads user who is now trying to use LibraryThing is discovering. Before I ever started using Goodreads, I made sure it met my minimum criteria for putting my data into a proprietary silo: That I could get the data back out. I can, and have. LibraryThing can import it. But the import process loses data! And it's majorly clunky. If I want to continue using Goodreads due to its better UI, and get the data into LibraryThing, for its better analytics, I have to do periodic dumps and loads of CSV files with manual fixups. This is why we have standards. This is why we're building federated social networks like status.net and the upcoming pump.io that can pass structured data between nodes transparently. It doesn't have to be a nightmare. It doesn't have to rely on proprietary APIs. We have the computer geeks. Thing is, sites like GoodReads and LibraryThing need domain-specific knowledge, and communities to curate data, and stuff like that. Things that work well in a smallish company. (LibraryThing even has a business model that makes sense, yearly payments to store more books in it.) With free software, it's much more appealing to sink the time we have into the most general-purpose solution we can. Why build a LibraryThing when we could build something that tracks not only books but movies and music? Why build that when we could build a generic federated network for structured social data? And that's great, as infrastructure, but if that infrastructure is only used to build a succession of proprietary data silos, what was the point? So, could some computer & book geeks please build a free software alternative to these things, focused on books, that federates using any of the fine APIs we have available? Bear in mind that there is already a nice start at a comprehensive collection of book data in the Open Library. I'd happily contribute to a crowd funded project doing this.

29 March 2013

Gunnar Wolf: Looking for a (small) place to host a Free Software-related meeting, course or similar in Mexico City?

Hey, Mexican hackers! If anybody is interested in holding a small Free Software-related meeting (say, with up to 10-15 people) in the South of Mexico City, please tell me We have adapted a nice room at our house where we want to invite people to come and do activities Courses, meetings, whatever. It is not very big (~5 5 meters), but it has all of the needed amenities (some chairs, a projector, coffee-related amenities, and is very conveniently located). We are not charging for hosting your activities (but will of course want to schedule it beforehand with you). So, if you have something to teach, or some project to hack on, and want a nice place to do it in, please drop us a line/call. (hmh, yes, this is one of the posts that should probably be in Spanish But this blog has a long-standing policy for English content ;-)

Olivier Berger: Managing Python code with UTF-8 (french chars) in org-mode + babel + minted for LaTeX export

The goal of this article is to illustrate how to manage Python code which includes comments in UTF-8 characters inside a latin-1 source org-mode for LaTeX export. Note that I ve pasted in wordpress the HTML generated by org-mode, so I hope it isn t broken too much. My typical use case is a french lecture on Python where the text is written in french, as well as some of the code comments and examples We ll use org-mode s babel module to include and manage the Python
examples. The goal is to write the source of the Python programs
directly in the same org source as the class book s text, and to extract them into a subdir (with the tangle feature), so that they can be shipped to the students to experiment with. The minted LaTeX environment is used, for babel, to make the Python syntax highlighting.

1.1 Issues
  • The source org-mode is in latin-1 so that it compiles with pdflatex
  • The examples source-code will be in UTF-8 so that Python 2.7 executes them well on a modern Linux desktop
  • Minted relies on pygmentize wich doesn t seem to handle UTF-8 so well

1.2 Solution
We ll show how to patch minted s use of pygmentize to take advantage
of its conversion capacity to convert the Python sources from UTF-8 to
latin-1 at document rendering time (LaTeX compilation). It s a hack, but works. The current document s source shows how this works

1.2.1 Customization
The org document should contain the following headers :
#+LANGUAGE:  fr
#+LaTeX_HEADER: \usepackage[latin1] inputenc 
#+LaTeX_HEADER: \usepackage[french] babel 
#+LaTeX_HEADER: \usepackage color \usepackage minted 
and the footers :
# Local Variables:
# coding: latin-1
# org-src-preserve-indentation: true
# tab-width: 4
# End:
Also note that pdflatex should use the -shell-escape option necessary for minted Now, the document should also include the following patch for minted :
#+LATEX_HEADER: \makeatletter
#+LATEX_HEADER: \renewcommand\minted@pygmentize[2][\jobname.pyg] 
#+LATEX_HEADER:   \def\minted@cmd pygmentize -l #2 -f latex -F tokenmerge
#+LATEX_HEADER:     \minted@opt gobble  \minted@opt texcl  \minted@opt mathescape 
#+LATEX_HEADER:     \minted@opt startinline  \minted@opt funcnamehighlighting 
#+LATEX_HEADER:     \minted@opt linenos  -P "verboptions=\minted@opt extra "
#+LATEX_HEADER:     -O encoding=UTF-8,outencoding=iso-8859-1 -o \jobname.out.pyg #1 
#+LATEX_HEADER:   \immediate\write18 \minted@cmd 
#+LATEX_HEADER:   % For debugging, uncomment:
#+LATEX_HEADER:   %\immediate\typeout \minted@cmd 
#+LATEX_HEADER:   \ifthenelse \equal \minted@opt@bgcolor 
#+LATEX_HEADER:     
#+LATEX_HEADER:     \begin minted@colorbg \minted@opt@bgcolor 
#+LATEX_HEADER:   \input \jobname.out.pyg 
#+LATEX_HEADER:   \ifthenelse \equal \minted@opt@bgcolor 
#+LATEX_HEADER:     
#+LATEX_HEADER:     \end minted@colorbg 
#+LATEX_HEADER:   \DeleteFile \jobname.out.pyg 
#+LATEX_HEADER: \makeatother
The important change is the addition of the -O encoding=UTF-8,outencoding=iso-8859-1 option of pygmentize, that will convert the UTF-8 source code to latin-1. Hopefully most of the characters will convert fine, for a document written in french.

2 Examples

2.1 Basic Python
This is a generic verbatim example, which doesn t use babel / minted.
$ python
Python 2.7.3 (default, Jan  2 2013, 16:53:07) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print "Hello, world!"
Hello, world!
>>>

2.2 Python source code for french speakers
Here, we first include the source code, so that it can be exported to
an examples/ subdir, with the following (note that the #+ BEGIN_src / #+ END_src should be changed to get rid of the space
character between + and BEGIN / END, and that the :shebang and following text should be on the BEGIN_SRC line I seem to have not quoted
these properly in this document):
#+name: helloworld
#+ BEGIN_src python :tangle examples/helloworld.py :noweb yes \\
:shebang #!/usr/bin/python :padline no :exports none
# -*- coding: utf-8 -*-
# Ceci est un exemple d'affichage d'une cha ne accentu e
name = raw_input("Quel est v tre nom ? ")
print "J'esp re que  a va bien aujourd'hui", name
#+ END_src
Editing it with C-c will open an UTF-8 buffer, so hopefully, this will be consistent with the utf-8 coding system declared on the second line. The regeneration of the examples/helloworld.py file is made with C-c C-v t (babel tangling ). Note that we don t add the #!/usr/bin/python first line in the source, as we want it to be added by the tangling process (which will also make the script executable). Once this is done, the script may be run :
$ python examples/helloworld.py 
Quel est v tre nom ? Fran ois
J'esp re que  a va bien aujourd'hui, cher Fran ois
Now, to add the colorized rendering in the lecture book, we do (here again, remove the spaces):
# +BEGIN_latex
\inputminted python examples/helloworld.py 
# +END_latex
And here it is, rendered by minted:
# -*- coding: utf-8 -*-
# Ceci est un exemple d'affichage d'une cha ne accentu e
name = raw_input("Quel est v tre nom ? ")
print "J'esp re que  a va bien aujourd'hui, cher", name
Here are links to : the source, the PDF result, and the generated Python program.

Next.

Previous.