Olivier Berger: Managing Python code with UTF-8 (french chars) in org-mode + babel + minted for LaTeX export
The goal of this article is to illustrate how to manage Python code which includes comments in UTF-8 characters inside a latin-1 source org-mode for LaTeX export.
Note that I ve pasted in wordpress the HTML generated by org-mode, so I hope it isn t broken too much.
My typical use case is a french lecture on Python where the text is written in french, as well as some of the code comments and examples
We ll use org-mode s babel module to include and manage the Pythonexamples. The goal is to write the source of the Python programs
directly in the same org source as the class book s text, and to extract them into a subdir (with the tangle feature), so that they can be shipped to the students to experiment with. The
minted LaTeX environment is used, for babel, to make the Python syntax highlighting.
1.1 Issues
- The source org-mode is in latin-1 so that it compiles with pdflatex
- The examples source-code will be in UTF-8 so that Python 2.7 executes them well on a modern Linux desktop
- Minted relies on pygmentize wich doesn t seem to handle UTF-8 so well
- The source org-mode is in latin-1 so that it compiles with pdflatex
- The examples source-code will be in UTF-8 so that Python 2.7 executes them well on a modern Linux desktop
- Minted relies on pygmentize wich doesn t seem to handle UTF-8 so well
1.2 Solution
We ll show how to patch minted s use of pygmentize to take advantage
of its conversion capacity to convert the Python sources from UTF-8 to
latin-1 at document rendering time (LaTeX compilation).
It s a hack, but works.
The current document s source shows how this works
1.2.1 Customization
The org document should contain the following headers :
#+LANGUAGE: fr
#+LaTeX_HEADER: \usepackage[latin1] inputenc
#+LaTeX_HEADER: \usepackage[french] babel
#+LaTeX_HEADER: \usepackage color \usepackage minted
and the footers :
# Local Variables:
# coding: latin-1
# org-src-preserve-indentation: true
# tab-width: 4
# End:
Also note that pdflatex should use the -shell-escape option necessary for minted
Now, the document should also include the following patch for minted :
#+LATEX_HEADER: \makeatletter
#+LATEX_HEADER: \renewcommand\minted@pygmentize[2][\jobname.pyg]
#+LATEX_HEADER: \def\minted@cmd pygmentize -l #2 -f latex -F tokenmerge
#+LATEX_HEADER: \minted@opt gobble \minted@opt texcl \minted@opt mathescape
#+LATEX_HEADER: \minted@opt startinline \minted@opt funcnamehighlighting
#+LATEX_HEADER: \minted@opt linenos -P "verboptions=\minted@opt extra "
#+LATEX_HEADER: -O encoding=UTF-8,outencoding=iso-8859-1 -o \jobname.out.pyg #1
#+LATEX_HEADER: \immediate\write18 \minted@cmd
#+LATEX_HEADER: % For debugging, uncomment:
#+LATEX_HEADER: %\immediate\typeout \minted@cmd
#+LATEX_HEADER: \ifthenelse \equal \minted@opt@bgcolor
#+LATEX_HEADER:
#+LATEX_HEADER: \begin minted@colorbg \minted@opt@bgcolor
#+LATEX_HEADER: \input \jobname.out.pyg
#+LATEX_HEADER: \ifthenelse \equal \minted@opt@bgcolor
#+LATEX_HEADER:
#+LATEX_HEADER: \end minted@colorbg
#+LATEX_HEADER: \DeleteFile \jobname.out.pyg
#+LATEX_HEADER: \makeatother
The important change is the addition of the -O encoding=UTF-8,outencoding=iso-8859-1 option of pygmentize, that will convert the UTF-8 source code to latin-1.
Hopefully most of the characters will convert fine, for a document written in french.
of its conversion capacity to convert the Python sources from UTF-8 to
latin-1 at document rendering time (LaTeX compilation). It s a hack, but works. The current document s source shows how this works
1.2.1 Customization
The org document should contain the following headers :
#+LANGUAGE: fr
#+LaTeX_HEADER: \usepackage[latin1] inputenc
#+LaTeX_HEADER: \usepackage[french] babel
#+LaTeX_HEADER: \usepackage color \usepackage minted
and the footers :
# Local Variables:
# coding: latin-1
# org-src-preserve-indentation: true
# tab-width: 4
# End:
Also note that pdflatex should use the -shell-escape option necessary for minted
Now, the document should also include the following patch for minted :
#+LATEX_HEADER: \makeatletter
#+LATEX_HEADER: \renewcommand\minted@pygmentize[2][\jobname.pyg]
#+LATEX_HEADER: \def\minted@cmd pygmentize -l #2 -f latex -F tokenmerge
#+LATEX_HEADER: \minted@opt gobble \minted@opt texcl \minted@opt mathescape
#+LATEX_HEADER: \minted@opt startinline \minted@opt funcnamehighlighting
#+LATEX_HEADER: \minted@opt linenos -P "verboptions=\minted@opt extra "
#+LATEX_HEADER: -O encoding=UTF-8,outencoding=iso-8859-1 -o \jobname.out.pyg #1
#+LATEX_HEADER: \immediate\write18 \minted@cmd
#+LATEX_HEADER: % For debugging, uncomment:
#+LATEX_HEADER: %\immediate\typeout \minted@cmd
#+LATEX_HEADER: \ifthenelse \equal \minted@opt@bgcolor
#+LATEX_HEADER:
#+LATEX_HEADER: \begin minted@colorbg \minted@opt@bgcolor
#+LATEX_HEADER: \input \jobname.out.pyg
#+LATEX_HEADER: \ifthenelse \equal \minted@opt@bgcolor
#+LATEX_HEADER:
#+LATEX_HEADER: \end minted@colorbg
#+LATEX_HEADER: \DeleteFile \jobname.out.pyg
#+LATEX_HEADER: \makeatother
The important change is the addition of the -O encoding=UTF-8,outencoding=iso-8859-1 option of pygmentize, that will convert the UTF-8 source code to latin-1.
Hopefully most of the characters will convert fine, for a document written in french.
#+LANGUAGE: fr #+LaTeX_HEADER: \usepackage[latin1] inputenc #+LaTeX_HEADER: \usepackage[french] babel #+LaTeX_HEADER: \usepackage color \usepackage mintedand the footers :
# Local Variables: # coding: latin-1 # org-src-preserve-indentation: true # tab-width: 4 # End:Also note that pdflatex should use the
-shell-escape option necessary for minted
Now, the document should also include the following patch for minted :
#+LATEX_HEADER: \makeatletter #+LATEX_HEADER: \renewcommand\minted@pygmentize[2][\jobname.pyg] #+LATEX_HEADER: \def\minted@cmd pygmentize -l #2 -f latex -F tokenmerge #+LATEX_HEADER: \minted@opt gobble \minted@opt texcl \minted@opt mathescape #+LATEX_HEADER: \minted@opt startinline \minted@opt funcnamehighlighting #+LATEX_HEADER: \minted@opt linenos -P "verboptions=\minted@opt extra " #+LATEX_HEADER: -O encoding=UTF-8,outencoding=iso-8859-1 -o \jobname.out.pyg #1 #+LATEX_HEADER: \immediate\write18 \minted@cmd #+LATEX_HEADER: % For debugging, uncomment: #+LATEX_HEADER: %\immediate\typeout \minted@cmd #+LATEX_HEADER: \ifthenelse \equal \minted@opt@bgcolor #+LATEX_HEADER: #+LATEX_HEADER: \begin minted@colorbg \minted@opt@bgcolor #+LATEX_HEADER: \input \jobname.out.pyg #+LATEX_HEADER: \ifthenelse \equal \minted@opt@bgcolor #+LATEX_HEADER: #+LATEX_HEADER: \end minted@colorbg #+LATEX_HEADER: \DeleteFile \jobname.out.pyg #+LATEX_HEADER: \makeatotherThe important change is the addition of the
-O encoding=UTF-8,outencoding=iso-8859-1 option of pygmentize, that will convert the UTF-8 source code to latin-1.
Hopefully most of the characters will convert fine, for a document written in french.
2 Examples
2.1 Basic Python
This is a generic verbatim example, which doesn t use babel / minted.
$ python
Python 2.7.3 (default, Jan 2 2013, 16:53:07)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print "Hello, world!"
Hello, world!
>>>
2.2 Python source code for french speakers
Here, we first include the source code, so that it can be exported to
an examples/ subdir, with the following (note that the #+ BEGIN_src / #+ END_src should be changed to get rid of the space
character between + and BEGIN / END, and that the :shebang and following text should be on the BEGIN_SRC line I seem to have not quoted
these properly in this document):
#+name: helloworld
#+ BEGIN_src python :tangle examples/helloworld.py :noweb yes \\
:shebang #!/usr/bin/python :padline no :exports none
# -*- coding: utf-8 -*-
# Ceci est un exemple d'affichage d'une cha ne accentu e
name = raw_input("Quel est v tre nom ? ")
print "J'esp re que a va bien aujourd'hui", name
#+ END_src
Editing it with C-c will open an UTF-8 buffer, so hopefully, this will be consistent with the utf-8 coding system declared on the second line.
The regeneration of the examples/helloworld.py file is made with C-c C-v t (babel tangling ).
Note that we don t add the #!/usr/bin/python first line in the source, as we want it to be added by the tangling process (which will also make the script executable).
Once this is done, the script may be run :
$ python examples/helloworld.py
Quel est v tre nom ? Fran ois
J'esp re que a va bien aujourd'hui, cher Fran ois
Now, to add the colorized rendering in the lecture book, we do (here again, remove the spaces):
# +BEGIN_latex
\inputminted python examples/helloworld.py
# +END_latex
And here it is, rendered by minted:
# -*- coding: utf-8 -*-
# Ceci est un exemple d'affichage d'une cha ne accentu e
name = raw_input("Quel est v tre nom ? ")
print "J'esp re que a va bien aujourd'hui, cher", name
Here are links to : the source, the PDF result, and the generated Python program.
2.1 Basic Python
This is a generic verbatim example, which doesn t use babel / minted.
$ python
Python 2.7.3 (default, Jan 2 2013, 16:53:07)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print "Hello, world!"
Hello, world!
>>>
$ python Python 2.7.3 (default, Jan 2 2013, 16:53:07) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> print "Hello, world!" Hello, world! >>>
2.2 Python source code for french speakers
Here, we first include the source code, so that it can be exported to
an examples/ subdir, with the following (note that the #+ BEGIN_src / #+ END_src should be changed to get rid of the space
character between + and BEGIN / END, and that the :shebang and following text should be on the BEGIN_SRC line I seem to have not quoted
these properly in this document):
#+name: helloworld
#+ BEGIN_src python :tangle examples/helloworld.py :noweb yes \\
:shebang #!/usr/bin/python :padline no :exports none
# -*- coding: utf-8 -*-
# Ceci est un exemple d'affichage d'une cha ne accentu e
name = raw_input("Quel est v tre nom ? ")
print "J'esp re que a va bien aujourd'hui", name
#+ END_src
Editing it with C-c will open an UTF-8 buffer, so hopefully, this will be consistent with the utf-8 coding system declared on the second line.
The regeneration of the examples/helloworld.py file is made with C-c C-v t (babel tangling ).
Note that we don t add the #!/usr/bin/python first line in the source, as we want it to be added by the tangling process (which will also make the script executable).
Once this is done, the script may be run :
$ python examples/helloworld.py
Quel est v tre nom ? Fran ois
J'esp re que a va bien aujourd'hui, cher Fran ois
Now, to add the colorized rendering in the lecture book, we do (here again, remove the spaces):
# +BEGIN_latex
\inputminted python examples/helloworld.py
# +END_latex
And here it is, rendered by minted:
# -*- coding: utf-8 -*-
# Ceci est un exemple d'affichage d'une cha ne accentu e
name = raw_input("Quel est v tre nom ? ")
print "J'esp re que a va bien aujourd'hui, cher", name
Here are links to : the source, the PDF result, and the generated Python program.
an
examples/ subdir, with the following (note that the #+ BEGIN_src / #+ END_src should be changed to get rid of the spacecharacter between
+ and BEGIN / END, and that the :shebang and following text should be on the BEGIN_SRC line I seem to have not quotedthese properly in this document):
#+name: helloworld
#+ BEGIN_src python :tangle examples/helloworld.py :noweb yes \\
:shebang #!/usr/bin/python :padline no :exports none
# -*- coding: utf-8 -*-
# Ceci est un exemple d'affichage d'une cha ne accentu e
name = raw_input("Quel est v tre nom ? ")
print "J'esp re que a va bien aujourd'hui", name
#+ END_src
Editing it with C-c will open an UTF-8 buffer, so hopefully, this will be consistent with the utf-8 coding system declared on the second line.
The regeneration of the examples/helloworld.py file is made with C-c C-v t (babel tangling ).
Note that we don t add the #!/usr/bin/python first line in the source, as we want it to be added by the tangling process (which will also make the script executable).
Once this is done, the script may be run :
$ python examples/helloworld.py Quel est v tre nom ? Fran ois J'esp re que a va bien aujourd'hui, cher Fran oisNow, to add the colorized rendering in the lecture book, we do (here again, remove the spaces):
# +BEGIN_latex \inputminted python examples/helloworld.py # +END_latexAnd here it is, rendered by minted:
# -*- coding: utf-8 -*- # Ceci est un exemple d'affichage d'une cha ne accentu e name = raw_input("Quel est v tre nom ? ") print "J'esp re que a va bien aujourd'hui, cher", nameHere are links to : the source, the PDF result, and the generated Python program.
A month ago, Mark Donfried from the 
I gave a presentation at 
We were lazy and wrote a simple PostgreSQL monitoring check in shell instead of
using some proper language. The code looked about this:
So 
This Clevo laptop is a new machine and like a lot of new machines, not all of its hardware has drivers in the current stable release of debian.
Happily, there is a driver for its rtl8723ae wireless networking device in the later 3.8 Linux kernel versions. So it s just a case of installing the package called kernel-package and following the instructions in it, to make a new linux-image package with the latest drivers in it.
One small thing which tripped me up is that you usually need to write make-kpkg rootcmd fakeroot initrd kernel-image now. I forgot the initrd option at first.
I just pushed out a new 
Given how Debian and Ubuntu nowadays enable IPv6 support by default, the routing table includes some IPv6 entries by standard. However, this has the unpleasant side-effect that lib C tries to connect to public IPv6 addresses, even in a case when the only IPv6 entries are for loopback and local link. Apparently, libc incorrectly assumes that the mere presence of any IPv6 means that there is a usable default IPv6 route. This results in, among other things, APT failing at fetching packages from the Debian repository if the DNS rotation returned the IPv6 address for the server first. I was thus wondering whether libc6 could be made to reject public IPv6 addresses, in cases when the host has no IPv6 gateway in its routing table? Or would there be a smarter way to prevent applications from failing at contacting IPv6 addresses whenever there is no IPv6 connection to the outside world?
People often ask me why I invest in low-tech, old-school investments like gold and silver bullion.
Surely for somebody working with technology every day, I would have some advantage in the world of investment?
The events of Cyprus this week should provide a clear answer: we've all been told that there is nothing safer than cash in the bank, protected by a Government deposit insurance scheme.
Yet it appears that in a pattern reminiscent of the 
The 2008 British film 
Back to the gold and silver. A recent report in 
It is customary during 
Dear users and supporters of the backports service!
The Backports Team is pleased to announce the next important step on getting backports more integrated. People who are reading 
After one week of campaign on -vote@, many subjects have been mentioned already. I m trying here to list the concrete, actionable ideas I found interesting (does not necessarily mean that I agree with all of them) and that may be worth further discussion at a less busy time. There s obviously some amount of subjectivity in such a list, and I m also slightly biased ;) . Feel free to point to missing ideas or references (when an idea appeared in several emails, I ve generally tried to use the first reference).
On the campaign itself, and having general discussions inside Debian: