Search Results: "zed"

30 March 2013

Joey Hess: Goodreads vs LibraryThing vs Free software

Four years ago I started using Goodreads to maintain the list of books I've read (which had lived in a flat text file for a decade+ before that). Now it's been aquired by Amazon. I doubt it will survive in its current form for more than 2 years. Anyway, while Goodreads has been a quite good way to find what my friends are reading, I've been increasingly annoyed by the quality of its recommendations, and its paucity of other features I need. It really doesn't seem to help me keep up with new and interesting fiction at all, unless my friends happen to read it. So I looked at LibraryThing. Actually, I seem to have looked at it several times before, since it had accounts named "joey", "joeyh", and "joeyhess" that were all mine. Which is what happens to me on sites that lack Openid or Browserid. Digging a little deeper this time, I am finding its recommendations much better than Goodreads' -- although it seems to sometimes recommend books I've already read. And it has some nice features like tracking series, so you can easily tell when you've read all the books in a series or not. The analytics overall seem quite impressive. The UI is cluttered and it seems to take 5 clicks to add and rate a single book. It supports half stars. Overall I get the feeling this was designed for a set of needs that doesn't quite match mine. For example, it seems it doesn't have a single database entry per book; instead each time I add a book, it seems to pull in data from primary sources (library of congress, Amazon cough) and treat this as a separate (but related) entry somehow. Weird. Perhaps this makes sense to say, librarians. I'm willing to adjust how I think about things if there's an underlying reason that can be grasped. There's a quite interesting thread on LibraryThing where the founder says:
Don't say we should open-source the code. That would be a nightmare! And I have limited confidence in APIs. LibraryThing has the book geeks, but not so much the computers geeks.
I assume that the nightmare is that there would be dozens of clones of the site, all balkanized, with no data transfer, no federation between them. Except, that's the current situation, as every Goodreads user who is now trying to use LibraryThing is discovering. Before I ever started using Goodreads, I made sure it met my minimum criteria for putting my data into a proprietary silo: That I could get the data back out. I can, and have. LibraryThing can import it. But the import process loses data! And it's majorly clunky. If I want to continue using Goodreads due to its better UI, and get the data into LibraryThing, for its better analytics, I have to do periodic dumps and loads of CSV files with manual fixups. This is why we have standards. This is why we're building federated social networks like status.net and the upcoming pump.io that can pass structured data between nodes transparently. It doesn't have to be a nightmare. It doesn't have to rely on proprietary APIs. We have the computer geeks. Thing is, sites like GoodReads and LibraryThing need domain-specific knowledge, and communities to curate data, and stuff like that. Things that work well in a smallish company. (LibraryThing even has a business model that makes sense, yearly payments to store more books in it.) With free software, it's much more appealing to sink the time we have into the most general-purpose solution we can. Why build a LibraryThing when we could build something that tracks not only books but movies and music? Why build that when we could build a generic federated network for structured social data? And that's great, as infrastructure, but if that infrastructure is only used to build a succession of proprietary data silos, what was the point? So, could some computer & book geeks please build a free software alternative to these things, focused on books, that federates using any of the fine APIs we have available? Bear in mind that there is already a nice start at a comprehensive collection of book data in the Open Library. I'd happily contribute to a crowd funded project doing this.

29 March 2013

Olivier Berger: Managing Python code with UTF-8 (french chars) in org-mode + babel + minted for LaTeX export

The goal of this article is to illustrate how to manage Python code which includes comments in UTF-8 characters inside a latin-1 source org-mode for LaTeX export. Note that I ve pasted in wordpress the HTML generated by org-mode, so I hope it isn t broken too much. My typical use case is a french lecture on Python where the text is written in french, as well as some of the code comments and examples We ll use org-mode s babel module to include and manage the Python
examples. The goal is to write the source of the Python programs
directly in the same org source as the class book s text, and to extract them into a subdir (with the tangle feature), so that they can be shipped to the students to experiment with. The minted LaTeX environment is used, for babel, to make the Python syntax highlighting.

1.1 Issues
  • The source org-mode is in latin-1 so that it compiles with pdflatex
  • The examples source-code will be in UTF-8 so that Python 2.7 executes them well on a modern Linux desktop
  • Minted relies on pygmentize wich doesn t seem to handle UTF-8 so well

1.2 Solution
We ll show how to patch minted s use of pygmentize to take advantage
of its conversion capacity to convert the Python sources from UTF-8 to
latin-1 at document rendering time (LaTeX compilation). It s a hack, but works. The current document s source shows how this works

1.2.1 Customization
The org document should contain the following headers :
#+LANGUAGE:  fr
#+LaTeX_HEADER: \usepackage[latin1] inputenc 
#+LaTeX_HEADER: \usepackage[french] babel 
#+LaTeX_HEADER: \usepackage color \usepackage minted 
and the footers :
# Local Variables:
# coding: latin-1
# org-src-preserve-indentation: true
# tab-width: 4
# End:
Also note that pdflatex should use the -shell-escape option necessary for minted Now, the document should also include the following patch for minted :
#+LATEX_HEADER: \makeatletter
#+LATEX_HEADER: \renewcommand\minted@pygmentize[2][\jobname.pyg] 
#+LATEX_HEADER:   \def\minted@cmd pygmentize -l #2 -f latex -F tokenmerge
#+LATEX_HEADER:     \minted@opt gobble  \minted@opt texcl  \minted@opt mathescape 
#+LATEX_HEADER:     \minted@opt startinline  \minted@opt funcnamehighlighting 
#+LATEX_HEADER:     \minted@opt linenos  -P "verboptions=\minted@opt extra "
#+LATEX_HEADER:     -O encoding=UTF-8,outencoding=iso-8859-1 -o \jobname.out.pyg #1 
#+LATEX_HEADER:   \immediate\write18 \minted@cmd 
#+LATEX_HEADER:   % For debugging, uncomment:
#+LATEX_HEADER:   %\immediate\typeout \minted@cmd 
#+LATEX_HEADER:   \ifthenelse \equal \minted@opt@bgcolor 
#+LATEX_HEADER:     
#+LATEX_HEADER:     \begin minted@colorbg \minted@opt@bgcolor 
#+LATEX_HEADER:   \input \jobname.out.pyg 
#+LATEX_HEADER:   \ifthenelse \equal \minted@opt@bgcolor 
#+LATEX_HEADER:     
#+LATEX_HEADER:     \end minted@colorbg 
#+LATEX_HEADER:   \DeleteFile \jobname.out.pyg 
#+LATEX_HEADER: \makeatother
The important change is the addition of the -O encoding=UTF-8,outencoding=iso-8859-1 option of pygmentize, that will convert the UTF-8 source code to latin-1. Hopefully most of the characters will convert fine, for a document written in french.

2 Examples

2.1 Basic Python
This is a generic verbatim example, which doesn t use babel / minted.
$ python
Python 2.7.3 (default, Jan  2 2013, 16:53:07) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print "Hello, world!"
Hello, world!
>>>

2.2 Python source code for french speakers
Here, we first include the source code, so that it can be exported to
an examples/ subdir, with the following (note that the #+ BEGIN_src / #+ END_src should be changed to get rid of the space
character between + and BEGIN / END, and that the :shebang and following text should be on the BEGIN_SRC line I seem to have not quoted
these properly in this document):
#+name: helloworld
#+ BEGIN_src python :tangle examples/helloworld.py :noweb yes \\
:shebang #!/usr/bin/python :padline no :exports none
# -*- coding: utf-8 -*-
# Ceci est un exemple d'affichage d'une cha ne accentu e
name = raw_input("Quel est v tre nom ? ")
print "J'esp re que  a va bien aujourd'hui", name
#+ END_src
Editing it with C-c will open an UTF-8 buffer, so hopefully, this will be consistent with the utf-8 coding system declared on the second line. The regeneration of the examples/helloworld.py file is made with C-c C-v t (babel tangling ). Note that we don t add the #!/usr/bin/python first line in the source, as we want it to be added by the tangling process (which will also make the script executable). Once this is done, the script may be run :
$ python examples/helloworld.py 
Quel est v tre nom ? Fran ois
J'esp re que  a va bien aujourd'hui, cher Fran ois
Now, to add the colorized rendering in the lecture book, we do (here again, remove the spaces):
# +BEGIN_latex
\inputminted python examples/helloworld.py 
# +END_latex
And here it is, rendered by minted:
# -*- coding: utf-8 -*-
# Ceci est un exemple d'affichage d'une cha ne accentu e
name = raw_input("Quel est v tre nom ? ")
print "J'esp re que  a va bien aujourd'hui, cher", name
Here are links to : the source, the PDF result, and the generated Python program.

24 March 2013

Lars Wirzenius: Two new t-shirt designs

I've made two new designs for Trunk Tees, my Cafepress store. Thank you to Richard Braakman for suggesting the .* one. Here's all the older designs as well:

22 March 2013

Matthew Palmer: Managing your Databases

The Standalone Sysadmin asks, How do you manage database server instances? :
Do you have one (or a few) centralized database servers, either standalone or clustered, or do you spread the load like we are currently?
His argument for centralisation is one of easing the management burden of configuration and backups, whereas the distributed approach eliminates a central point of failure and performance degradation. I go for distributed, all the way. For a start, we run so many databases for so many customers that there s no way on earth we could stand up a small number of database servers and handle all the load (hell, we ve got single customers who consume a cluster of machines with 384GB of RAM and all the SSDs you can eat). Security and permissions is a whole other kettle of fish; the contortions we d have to do to allow customers the level of management they need with a centralised database system would be immense. Then there s the need of some customers for MySQL, some for PgSQL, different performance tuning for different workloads nope, centralised DBs don t work for us. Given this, we ve bitten the bullet and solved pretty much all of the management problems. Installation and configuration is all handled via Puppet, and backups are trivial the same system that installs the DB server itself also drops a hook script that the backup agent uses to know that it has to dump a database server. Monitoring that this backup is taking place successfully is also automatically provisioned, so we know that we re not missing anything. Ultimately, this same approach applies to practically anything that you re tossing up between centralised and distributed. At scale, you can never rely on centralisation, so you may as well bite the bullet and learn how to do it distributed pretty much from the start. That saves some serious system shock when you discover what your hardware vendor wants for the next step up in big iron hardware

20 March 2013

Jan Wagner: Chemnitzer Linux-Tage 2013!

Also this year the Debian project was present at Chemnitzer Linuxtage, this time right next the debianforum.de booth. The booth folks arriving on friday organized a flashmob at Expitas after booth setup. Unfortunatly our second planned flashmob at the mensa was boycotted by much more students, so we ended up in the Turm-Brauhaus, which is a great location with good drinks but the service was very harshly. On the next two days at the booth we chatted and discussed with visitors and other exhibitors a wide variety of questions, including 'When will be (the next Debian version) released?' and 'Are there installation disks available?'. The answers was as always 'When we are ready and we will have reached the quality-level we defined', 'No we don't have installation medias, as they are always outdated. Do you have an USB-dongle with you?'. Merchandising was requested by visitors as always, but we just had some leftovers of fosdem, brought by Axel. The demonstration was as usual a small box running Babelbox and xpenguins which worked out the last years too. This year there were three lectures held by Debian related people, about Debian GIS, Aptitude - known but even unknown and SSH and unreliable network connections. The organisation team did a really great job. The social event at saturday night was very exciting and we left it early in the morning. The whole event was indeed fun and a pleasure to find new friends and meet old ones of the Free Software community. Many thanks to Florian Baumann, Jan Dittberner, Andreas Tille, Christian Hoffmann, Axel Beckert, Markus Rekkenbeil, Daniel Schier, Jonas Genannt, Jan H rsch and kurio for taking care and running the booth, which worked out this year extreme smoothly from my point of view. Likewise as the last years a special thanks to TMT GmbH & Co. KG, which kindly donated additional boothtickets, the equipment, its transportation and accommodation for almost half of the booth staff.

Michal Čihař: Server troubles

This night you might have noticed that some services like hosted Weblate or phpMyAdmin wiki being quite flaky. It was caused by DDOS attack which pretty much busted firewall and thus made unable to open new connections. In the morning when I've realized this, I introduced another breakage by too strict filtering on firewall :-). However now everything should be back to normal.

Filed under: English Phpmyadmin Weblate 0 comments Flattr this!

15 March 2013

John Goerzen: It s Warm Enough For Mud

Today the outdoor temperatures got to nearly 75F/24C. The boys had this idea to go play with their streams . Here s one of the cleaner moments: I had set up an old pipe and a hose, with the water coming slowly out of the pipe onto a little mound of dirt. They can then use their fingers to make channels in the dirt for the water to flow through. They ve found that sticks can become bridges, making a hole in the earth makes a pond, and, oh yes, it s quite muddy and a lot of fun. Oliver at one point realized that he could splash the mud all around quite nicely. And he wanted to make sure I took photos of his hands. They could have played out there for hours, I m sure. When it was time to clean up, they enjoyed seeing the mud come off their sandals, arms, hands, and feet. It was a perfect use for the first really warm day of spring.

14 March 2013

Iustin Pop: Types as control flow constructs

A bug I've recently seen in production code gave me the idea for this blog post. Probably smarter people already wrote better things on this topic, so this is mostly for myself, to better summarise my own thoughts. Corrections are welcome, please leave a comment! Let's say we have a somewhat standard API in Python or C++: Signalling failures to initialise the object can be done in two ways: either by raising an exception, or by returning a null/None result. There are advantages and disadvantages to both: The None model can create latent bugs, for example in the following code:
ok = True
for arg in input_list:
  t = init_t(arg)
  if t is None:
    ok = False
    continue
  t.process()
return ok
The presence of the continue statement there is critical. Removing it, or moving it after some other statements which work with t will result in a bug. So using value-based returns forces us to introduce (manually) control points, without having the possibility to validate the model by the compiler (e.g. in C++). So it would seem that this kind of latent bugs pushes us to use the exception model, with its drawbacks. Let's look at how this interface would be implemented in (IMHO) idiomatic Haskell (where the a and b types represent the input and output types):
initT :: a -> Maybe T
processT :: T -> b
my_fn =
  
 case initT arg of
   Nothing -> -- handle failure
   Just v -> processT v
Yes, this can be written better, but it's beside the main point. The main point is that by introducing a wrapper type around our main type (T), we are forced via the type system to handle the failure case. We can't simply pass the result of initT to a function which accepts T, because it won't type check. And, no matter what we do with the result value, there are no exceptions involved here, so we only have to think about types/values, and not control flow changes. In effect, types become automatically-validated control-flow instructions. Or so it looks to me . So using types properly, we can avoid the exception-vs-return-value debate, and have all advantages without the disadvantages of either. If that is the case, why isn't this technique used more in other languages? At least in statically typed languages, it would be possible to implement it (I believe), via a C++ template, for example. In Python, you can't actually apply it, as there's no static way of enforcing the correct decapsulation. I was very saddened to see that Google's Go language, which is quite recent, has many examples where initialisation functions return a tuple err, value = , separating the actual value from the error, making it not safer than Python. It might be that polymorphic types are not as easy to work with, or it might be the lack of pattern matching. In any case, I don't think this was the last time I've seen a null pointer dereference (or the equivalent AttributeError: 'NoneType' object has no attribute ). Sadly You can even go further in Haskell and introduce more control flow structure via wrapper types. Please bear another contrived example: an HTML form that gets some input data from the user, validates it, saves it to the database, and echoes it back to the user. Without types, you would have to perform these steps manually, and ensure they are kept in the correct order when doing future modifications. With types, you only have to design the types correctly and export only smart contructors (but not the plain ones):
module Foo ( ValidatedValue
           , validateValue
           , RecordId
           , CommittedValue
           , commitValue
           , buildFeedbackForm
           ) where
data ValidatedValue a = ValidatedValue a
validateValue :: a -> Maybe (ValidatedValue a)
data RecordId =  
data CommittedValue a = CommittedValue a RecordId
commitValue :: ValidatedValue a -> ComittedValue a
buildFeedbackForm :: CommittedValue a -> HTMLDoc
From these types, it follows more or less that the only correct workflow is:
  1. get a value from the user
  2. validate it
  3. commit it, getting a transaction/record ID
  4. send the response to the user
In other words:
handleFeedbackForm = buildFeedbackForm . commitValue . validateValue
There are still issues here, e.g. the type a is completely hidden behind the wrapper types, and we can't recover some basic properties (even if we use newtype, unless we use GeneralizedNewtypeDeriving). But it does offer a way to improve control flow safety. And that is my 0.02 currency unit for today.

Richard Hartmann: Gitify your life

sage@guug This January, I held an updated version of my "Gitify your life" talk at sage@guug, which is the Munich chapter of the German Unix User Group. Once again, it was quite successful and turnout was about three times that of normal attendance. Afterwards, I was asked if I would be willing to hold the same talk at the German Unix User Group's Spring Meeting in 2014 as the programme for 2013 has been finalized for quite some time, already. Obviously, I said I would be willing to do that and will submit the talk once registration opens. Open Source Days By the way, I am writing this post from a hotel room in Copenhagen. Tomorrow, there will be a world first... "Gitify your life" will not only be held as a talk, it will be expanded into a full three hour workshop (with the option of extending it if there's interest as the room will be free after my workshop). Having never held a workshop of this scope, I am really looking forward to this experience. Besides enabling me to get people up and running with their personal data, this will allow me to see where new users are having issues with vcsh. I keep hearing of people not getting their heads around the concept easily, but no matter how many promises of updates to my docs I get while helping them, no one really follows through with that promise once they know how to use vcsh. Maybe I will find someone willing to follow through and/or simply gather enough material for docs 2.0. Your place? As an aside, and I will probably try to put a real announcement together as well, if you know of a conference, user group or other place where a lightning talk, a full talk, or a workshop about one or all of these topics would fit, please let me know. I will gladly submit proposals to or accept invitations from most places as long as I can spread this message. Especially if the venue happens to be reasonably close to Southern Germany, covers travel cost, or if they can guarantee video recording of the talk. Joey Hess has speculated if the talk isn't really called "git naked" as I seem to keep on giving that particular talk at venues which don't have video recordings. Please help me make Joey happy... ...with a video recording, I mean.

13 March 2013

Raphaël Hertzog: Kali Linux 1.0, a new Debian derivative

Today, during Blackhat Europe, Offensive Security announced the availability of Kali Linux 1.0, which aims to be the most advanced, robust, and stable penetration testing distribution to date. It is the successor of Backtrack Linux. kali Kali s choice of Debian Kali s release is a significant event in the security auditing and penetration testing field, and I m proud to see that Debian was retained as the best distribution to create this new product. Here s what Mati Aharoni of Offensive Security told me:
Debian provides a reliable base to build a new distribution and yet can easily be customized to add bleeding edge features, thanks to the unstable and experimental distributions.
Kali s development policies Even though Kali was prepared in secret, from now on Kali s development happens in the open in public git repositories. There are repositories for all the packages that have been created (or forked) as well as for the ISO images creation script. Debian packages are maintained with git-buildpackage, pristine-tar and the associated helper tools, making it easy to integrate the latest changes of Debian. Kali packaged several hundreds tools that relate to their field and they intend to contribute those which are DFSG-free back to Debian. Kali s technical infrastructure In the last year, I have been working within the Kali team to setup large parts of their infrastructure as a proper Debian derivative. Kali s main ISO images are built with live-build. All the bugfixes that I contributed to Debian Live were the direct result of my work for Kali. The git repositories are managed with gitolite. The package repositories are built with reprepro. The build daemons use rebuildd and sbuild. The (push) mirrors are synchronized with the same tools than Debian (based on rsync), but there s also a central server which redirects to a mirror close to you (and which is used by default everywhere). This one runs mirrorbrain (and not Rapha l Geissert s redirector). The ARM build daemons (armel/armhf) run on machines powered by Calxeda s Highbank (4 cores, 4 GB RAM) that work pretty well. Even better, Offensive Security is willing to dedicate one node of this cluster for Debian s own usage. The future This first release is not an end. It s only the start of a journey. Not all applications have been packaged yet and there s lot of work left to integrate everything in Debian. I m really looking forward to continue my collaboration with the Kali team as this has been one of the most interesting project I ever had as a Debian consultant. And also one of the few where I could really contribute something back to Debian.

9 comments Liked this article? Click here. My blog is Flattr-enabled.

Lars Wirzenius: ttystatus 0.22 (terminal status and progress reporting for Python)

I've just released version 0.22 of ttystatus, my Python library for showing progress reporting and status updates on terminals, for (Unix) command line programs. Output is automatically adapted to the width of the terminal: truncated if it does not fit, and re-sized if the terminal size changes. Available on code.liw.fi only, due to the Debian freeze. Version 0.22, released 2013-03-12

Dirk Eddelbuettel: Rcpp master class in New York last weekend

On Saturday I had the opportunity to teach another one-day master class on Rcpp. The class had been organized by Jared Lander, and organized very well I might add. The weekend started with a slight disappointment. I had taken Friday off, and hoped to reach NY by early afternoon to join JJ there, and to spend the afternoon with the RStan team. However, the tail end of last week's snowstorm made it such that we both got to Columbia's stats department closer to 6pm rather than 1pm, and half the team had left. Dang. Very frustrating travel experience. We salvaged the evening by gabbing over a cold beverage or two, before sharing some sacred New York pizza with Wes McKinney and Jared. The class itself on Saturday went quite well. With JJ on deck, we were able to have every participant log into an EC2-hosted instance of RStudio Server, which worked very well for usage examples of Rcpp. It has been almost a year since I last taught the class, and many exciting things--such as Rcpp attributes, added by JJ himself--have appeared, which made it extra fun. Participants were rather kind with praise. Either they really liked it, or they really are hard-nosed New Yorkers who manage to lie to my face without me noticing. We ended the day with some hard-earned cold beverages, followed by some dinner at Sylvia's (as tweeted by Jared) followed by more drinks. Ended up a little past my usual bedtime, but I managed to get out and enjoy a lovely 6.5 miles run across Central Park the next morning before leaving town. All in all, a very nice weekend, the travel horror of Friday notwithstanding. And who know, maybe we'll just do it again another time...

12 March 2013

Richard Hartmann: Gitify your life

These Open Source Days were kind of a mixed experience. The social part Things started off quite nicely; the speaker dinner was very pleasant and the people at my table interesting to talk with. The way back to the hotel and breakfast on Saturday were even more interesting; talking with Amelia Andersdotter, a member of the European Parliament for the Swedish Pirate Party, about anti-trust measures, privacy protection, lobbyists, and creating the right incentives for companies to actually protect user data was fun and engaging. My own claim to fame is that I told her that while Microsoft is willing to sign UEFI payloads for x86, they refuse to do so for ARM which she didn't know. From my understanding, their interpretation is that any anti-trust rulings apply to PCs only, and not general computing devices like tablets. It will be interesting to see how that plays out in the long run, especially once ARM processors start appearing in laptops and desktops as main CPUs where Microsoft clearly still has a near monopoly. It was fascinating to discuss all those issues with someone who's actively involved with legislation, mainly because her style of talking was markedly different from what I am used to. One point which she stressed repeatedly is that the most important skill within the context of political work in the EU is to be able to track down, read, and then reference all the content that's being produced in Brussels and Strasbourg. Maybe that tidbit is useful to people who want to try themselves at political activism. She nicely, and maybe unknowingly, emphasized that point by always referring to specific papers, directives, and studies after making any point. Statement of opinion, reference to supporting document. Statement of opinion, reference to supporting document. Statement of opinion, reference to supporting document. Rinse, repeat. It's no secret that I'm continuously disappointed by the political process in general and politicians in particular. Yet, I have to say that this encounter had a lasting and positive impression on me. Amelia has an incredible amount of energy and genuinely cares about the issues which too many other politicians ignore completely. It's good to know someone like this is on our side. If you're Swedish, please inform yourself about her and consider voting in her favour. Other than that, I met several other people, new and old. This year's OSD felt a lot more relaxed and there were more informal and spontaneous conversations going on in hallways and at tables. Even though I failed to have any intense discussion of Mercurial vs git this year, all those conversation were still really interesting. The "workshop" I already knew that the room assignment for my workshop on Saturday afternoon would be a last-minute thing. What I didn't expect was that the final confirmation by Copenhagen Business School would be given on Friday evening. Mixed with a very cumbersome process to update the slides of the CBS-owned and -operated information beamer in the lobby, this resulted in the official schedule showing my room as "Workshop" instead of "SP 214". The official schedule repeatedly stated that people had to pre-register for the workshop which was a left-over from last year where they actually had to plan the amount of hardware kits needed; the registration itself was was very well hidden behind a rather generic link you saw exactly once after purchasing your ticket. Add to this that my room was literally in the farthest corner of the building and on the top floor and things start to turn sour real quick. All other rooms were either close to the main entrance or on the ground floor. Helpers at the front desk had a lot of people asking where my room was and gave out directions, yet... ...the grand total of six people who arrived were all telling me how they couldn't find the room and almost gave up on finding it. As more people asked for directions than turned up at the workshop, this means that even with directions, the room was really hard to find. Myself, it took me two attempts to find the room after which I was led there by one of the helpers who wasn't completely sure we were going in the right direction, himself. Still, I managed to get everyone who had a laptop and access to a UNIX-based system set up with working mr and vcsh repositories they can work from as well as a demo git-annex repository to play with. Sadly, not everyone had a laptop, or access to UNIX, or the tools installed, or even an idea where to install them from, even though I had noted these "requirements" in the description. Next time I do this workshop, I will need to make sure to include information on what distributions have the relevant packages and how to find and install the software on Max OS X in the workshop description. Feedback from the very few people who actually managed to find me was very good and one of the attendants asked if I would be willing to hold the workshop again at another conference in Denmark. I am still awaiting details on this, but I would obviously enjoy having the chance to reach more people. At least I fared better than the guys who did the workshop on "Binary exploitation 101" in yet another room. Two guys held a workshop with a total attendance of... one... I know the organizers of OSD were really unhappy with how this turned out and I believe them when they say that something like this will never happen again. It's the second year that this team is running the conference, running conferences is an incredible amount of work, and things were smoother this year than last year. Iterate and improve, etc. The talk Saturday evening, Alex offered me the chance to hold a reparation talk on Sunday morning to compensate for the botched workshop. Said talk was scheduled for the first slot and not really announced (but the schedule on both www and the beamer was changed over night). Additionally, this was merely an updated and extended version of the talk which I already held at OSD 2012 so at least part of the visitors knew most of what I had to say, already. To be honest I didn't expect too much, but more than twenty people turned up which is a very decent turnout, all things considered. Same as with the workshop, feedback was extremely positive and I was told afterwards by Bryan that several people mentioned my talk as one of the highlights, or even the highlight, of OSD when he was talking to them. To make things even better this talk has, finally, been videotaped; I will update this blog with a link as soon as the video is online.

10 March 2013

Axel Beckert: Rendering Markdown, Asciidoc and Friends automatically while Editing

Partially because of Markdown being Github s markup format of choice, I enjoy writing documents in simple markup formats more and more. There s though one common annoyance with these formats compared to writing plain HTML The Annoyance They need to be rendered (i.e. more or less compiled) before you can view your outpourings rendered, e.g. in the web browser. So the workflow usually is:
  1. Saving the current file in your favourite editor
  2. Switch to terminal with commandline
  3. Cursor up, Enter
  4. Switch to your favourite web browser
  5. Hit the reload button
Using a Specialized Editor with Live Preview One choice would be to use a specific editor with live rendering. The one I know in Debian (from Wheezy on) is ReText (Debian package retext). It supports Markdown and reStructuredText. But as with most simple GUI editors, I miss there many of the advanced editing commands possible with Emacs. Using Emacs Markdown Mode Then there is the Markdown Mode for Emacs (part of Debian s emacs-goodies-el package), where you can get a preview by pressing C-c C-c p. But for some reason this takes several seconds, opens a new buffer and window with the rendered HTML code and then starts (hardcoded) Firefox (which is not my preferred web browser). And if you do that a second time without closing Firefox first, it won t just reload the file but will open a new tab. You might think that just hitting reload should suffice. But no, the new tab has a different file name, so reload doesn t help. Additionally it may not use my preferred Markdown implementation. Meh. Well, I probably could fix all those issues with Markdown Mode, it s only Emacs Lisp. Heck, the called command is even configurable. But fixing at least four issues to fix one workflow annoyance? Maybe some other time, but not as long there are other nice choices Using inotifywait to Render on Write So everytime you save the currently edited file, you immediately want to rerender the same HTML file from it. This can be easily automated by using Linux inotify kernel subsystem which notices changes to the filesystem, and reports those to applications which ask for it. One such tool is inotifywait which can either output all or just specific events, or just exit if the first requested event occurs. With the latter it s easy to write a while loop on the commandline which regenerates a file after every write access. I use either Pandoc or Asciidoc for that since both generate full HTML pages including header and footer, but you can use that also with Markdown to render just the HTML body. Most browsers render it correctly anyway:
while inotifywait -q -e modify index.md; do pandoc -s -f markdown -t html -o index.html index.md; done
while inotifywait -q -e modify index.txt; do asciidoc index.txt; done
while inotifywait -q -e modify index.md; do markdown index.md > index.html; done
This solution is even editor- and build-system-agnostic (But not operating-system-agnostic.) inotifywait is part of inotify-tools, a useful set of commandline tools to interface with inotify. They re packaged in Debian as inotify-tools, too. Using mdpress for Markdown plus Impress.js based Slides The ruby-written mdpress is a special case of the previous case. It s a commandline tool to convert Markdown into Impress.js based slide shows and it has an option named --automatic which causes it to keep running and automatically update the presentation as soon as changes are made to the Markdown file. mdpress is not yet in Debian, but there s an ITP for it and Impress.js itself recently entered Debian as libjs-impress. Nevertheless, two dependencies (highlight.js, ITP ed, ruby-launchy, ITP ed) are still missing in Debian.

7 March 2013

Pau Garcia i Quiles: Mark s divisive leadership

Mark Shuttleworth recently critized Jonathan Riddell for proposing Xubuntu and others join the Kubuntu community. I thought I could make a few amendments to Mark s writing:

Jonathan Mark says that Canonical Kubuntu is not taking care of the Ubuntu community.

Consider for a minute, Jonathan Mark, the difference between our actions.

Canonical Kubuntu, as one stakeholder in the Ubuntu community, is spending a large amount of energy to evaluate how its actions might impact on all the other stakeholders, and offering to do chunks of work in support of those other stakeholder needs.

You, as one stakeholder in the Ubuntu community, are inviting people to contribute less to the broader project [all the X and Wayland -based desktops], and more to one stakeholder [Unity and Mir].

Hmm. Just because you may not get what you want is no basis for divisive leadership.

Yes, you should figure out what s important to Kubuntu Ubuntu Unity and Mir, and yes, you should motivate folks to help you achieve those goals. But it s simply wrong to suggest that Canonical Kubuntu isn t hugely accommodating to the needs of others, or that it s not possible to contribute or participate in the parts of Ubuntu which Canonical Kubuntu has a particularly strong interest in. Witness the fantastic work being done on both the system and the apps to bring Ubuntu Plasma to the phone and tablet. That may not be your cup of tea, but it s tremendously motivating and exciting and energetic. See Mark? I only needed to do a little search and replace on your words and suddenly, meaning is completely reversed! Canonical started looking only after its own a couple of years ago and totally dumped the community. Many people have noticed this and written about this in the past two years. How dare you say Jonathan or anyone from Kubuntu is proposing contributing less to the broader community? The broader community uses X and/or Wayland. Canonical recently came with Mir, a replacement for X and Wayland, out of thin air. Incompatible with X and Wayland. No mention of it at all to anyone from X or Wayland. No mention of it at FOSDEM one month ago, even though I, as the organizer of the Cross Desktop DevRoom, had been stalking your guy for months because we wanted diversity (and we got it: Gnome, KDE, Razor, XFCE, Enlightenment, etc, we even invided OpenBox, FVWM, CDE and others!). I even wrote a mail to you personally warning you Unity was going to lose its opportunity to be on the stand at FOSDEM. You never answered, of course. Don t you think Mir, a whole new replacement for X and Wayland, which has been in development for 8 months, deserved a mention at the largest open source event in Europe? Come on, man. It is perfectly fine to say yes, Canonical is not so interested in the community. It s our way or the highway . But do not pretend it s anything else or someone else is a bad guy. In fact, is there any bad guy in this story at all!? I think there is not, it s just people with different visions and chosen paths to achieve them. Maybe Mir and Unity are great ideas, much better than X and Wayland. But that s not what we are talking about. We are talking about community, and Canonical has been steadily destroying it for a long time already. If you cannot or do not want to see that, you ve got a huge problem going on.

6 March 2013

Keith Packard: composite-swap

Composite and Swap Getting it Right Where the author tries to make sure DRI3000 is going to do what we want now and in the future DRI3000 The basic DRI3000 plan seems pretty straightforward:
  1. Have applications allocate buffers full of new window contents, attach pixmap IDs to those buffers and pass them to the X server to get them onto the screen.
  2. Provide a mechanism to let applications know when those pixmaps are idle so that they can reuse them instead of creating new ones for every frame.
  3. Finally, allow the actual presentation of the contents to be scheduled for a suitable time in the future, generally synchronized with the monitor. Let the client know when this has happened in case they want to synchronize themselves to vblank.
The DRI3 extension provides a way to associate pixmap IDs and buffers, and given the MIT-SHM prototype I ve already implemented, I think we can safely mark this part as demonstrably implementable. That leaves us with a smaller problem, that of taking pixmap contents and presenting them on the screen at a suitable time and telling applications about the progress of that activity. In the absence of compositing, I m pretty sure the initial Swap extension design would do this job just fine, and should resolve some of the known DRI2 limitations related to buffer management. And, I think that goal is sufficient motivation to go and implement that. However, I wanted to write up some further ideas to see if the DRI3000 plan can be made to do precisely what we want in a composited world. The Composited Goal To make sure we re all on the same page, here s what I expect from the Swap extension in a composited world:
  1. Application calls Swap with new window pixmap
  2. Compositor hears about the new pixmap and uses that to construct a new screen pixmap
  3. Compositor calls Swap with new screen pixmap
  4. Vertical retrace happens, executing the pending swap operation
  5. Compositor hears about the swap completion for the screen
  6. Application hears about the swap completion for its window
In particular, applications should not hear that their swap operations are complete until the contents appear on the screen. This allows for applications to throttle themselves to the screen rate, either doing double or triple buffering as they choose. I didn t add steps here indicating buffers going idle or being allocated, because I think that should all happen behind the scenes from the application s perspective. Many applications won t care about the swap completion notification either, but some will and so that needs to be visible. Redirected Swaps? Owen Taylor suggested that one way of getting the compositor involved would be to have it somehow redirect Swap operations, much like we do with window management operations today. I think that idea may be a good direction to try:
  1. Application calls Swap with new window pixmap
  2. Swap is redirected to compositor, passing along the new window pixmap
  3. Compositor constructs a new screen pixmap using the new window pixmap
  4. Compositor calls Swap on the screen and the window, passing the new screen pixmap and the new window pixmap. When the screen update occurs, the screen and the window both receive swap completion events.
This has the added benefit that the X server knows when the compositor is expecting window pixmaps to change like this the compositor has to explicitly request Swap redirection. Window Pixmap Names and GEM Buffer Handles One issue that swapping window pixmaps around like this brings up is how to manage existing names for the window pixmap. Right now, applications expect that window pixmaps will only change when the window is resized. If the Swap extension is going to actually replace the window pixmap when running with a suitable compositor, then we need to figure out what the old names will reference. Are there non-compositor applications using NameWindowPixmap that matter to us? How about non-compositor applications using TextureFromPixmap to get a GEM handle for a window pixmap? For now, I m very tempted to just break stuff and see who complains, but knowing what we re breaking might be nice beforehand. Idling Pixmaps When an application is done drawing to a window pixmap and has passed it off to the X server for presentation, we d like for that pixmap to be automatically marked as discardable as soon as possible. This way, when memory is tight, the kernel can come steal those pages for something critical. Of course, applications may not want to let the server mark the pixmap as idle after being used, so a flag to the Swap call would be needed. Ideally, the pixmap would become idle immediately after the pixmap contents have been extracted. In the absence of a compositor, that would probably be when the Swap operation completes. With a compositor running, we d need explicit instruction from the compositor telling us that the window pixmap was now idle :
 
    SwapIdle
    drawable: Drawable
    pixmap: Pixmap
       
 
Furthermore, the application needs to know that the pixmap is in fact idle. I think that we ll need a synchronous X request that marks a buffer as no longer idle and have that return whether the buffer was discarded while idle. It doesn t seem sufficient to use events here as the application will need to completely reconstruct the pixmap contents in this case. This reply could also contain information about precisely what contents the pixmap does contain.
 
    SwapReuse
    drawable: Drawable
    pixmap: Pixmap
       
    valid: BOOL
    swap-hi: CARD32
    swap-lo: CARD32
 
Pixmap Lifetimes and Triple Buffered Applications If we redirect the Swap operation and send the original application window pixmap ID to the compositor, what happens when the application frees that pixmap before the compositor gets around to using the contents? Surely the Compositor must handle such cases, and not just crash. However, I m fine with requiring that the application not free the pixmap until told by the compositor.

4 March 2013

Lucas Nussbaum: Debian is (still) changing

(Looking for those graphs online, I realized that I never properly published them, besides that old post) I ve been playing with snapshot.d.o, which is a fantastic resource if you want to look at Debian from an historical perspective (well, since 2005 at least). Team maintenance comaint
We now have more team-maintained packages than packages maintained by someone alone. Interestingly, the small, ad-hoc group of developers model does not really take off. Maintenance using a VCS vcs A large majority of our packages are maintained in a VCS repository, with Git being the clear winner now. Possible goal for Jessie: standardize on a Git workflow, since every team tends to design its own? Packaging helpers helpers Again, we have a clear winner here, with dh. It s interesting to note that, while dh was designed as a CDBS killer, it kind-of fails in that role. Possible goal for Jessie: deprecate at least pure-debhelper packaging? Patch systems and packaging formats formats-patches Again, clear winner with 3.0 (quilt). The (dirty) scripts that generate those graphs are available in Git (but you need to connect to stabile to execute them, and it s rather time consuming hours/days).

2 March 2013

Bastian Blank: LDAP, Insignificant Space and Postfix

For some, LDAP, just like X.500, is black magic. I won't argue against. Sometimes it really shows surprising behavior. It always makes sense, if you think about what LDAP is built for. One surprising behavior is the handling of the "Insignificant Space"
LDAP supports syntax and comparator methods. The syntax specifies how the entry should look. Usually this is some form of text, but numbers or stuff like telephone numbers are supported as well. The comparators specifies how the values are compared. Most of the text comparators are defined to apply the handling of insignificant spaces. Insignificant space handling normalizes the use of spaces. First all leading and trailing spaces are removed. All the internal spaces are normalized to at most two spaces. At the end all strings starts and ends with one space to allow proper sub-string matches. The resulting strings are used for comparisons. This behavior makes sense most of the time. If the user want to find something in the directory, he usually don't cares about spaces, but about content. But I found one occurrence where this produces some grieve. Postfix supports LDAP since some time. And lets say, it does not care about spaces in its queries. This is no problem as e-mail addresses do not contain spaces. Or do they?
Yes, e-mail addresses can contain spaces. This is not widely known, but still allowed. This addresses are quoted and the command looks like RCPT TO:<" test"@example.com>. The local part is quoted and contains a space at the beginning. And this is where the problem starts. Postfix sanitizes the address. It uses a simplified internal representation of the address for all lookups. So the address gets test@example.com. This form is used in all table lookups. The LDAP table uses the internal form of the address. This address is copied verbatim into the query. This query may look this way (mail= test@example.com). It is sent to the server this way. The LDAP server applies the insignificant space modifications. The query is now interpreted and the comparator specified modifications are applied. The query gets effectively changed to (mail=test@example.com). And this is where the fun starts. Postfix accepts undeliverable mails. Depending on the setup, such LDAP queries may be used to check for valid addresses. Because of the space handling, the sender can add spaces to the beginning of the address and it will still be considered valid. In later steps this addresses are not longer valid. Addresses starting with spaces are considered invalid in some locations of Postfix. What surprised me a bit is that virtual alias handling did not map them. The unmodified addressed showed up on the LMTP backend server. That's why they showed up my radar. I would say Postfix is wrong in this case. The LDAP server applies the spec correctly and defines spaces in e-mail addresses as insignificant. Postfix on the other side considers them significant. The easiest fix would be to not allow any spaces in the LDAP table.

1 March 2013

Joerg Jaspert: tmux - like screen, just nicer. Replacing clusterssh too.

By now I am a long time happy user of tmux instead of screen. I started using it somewhere in 2011 and by now I only have one usage of screen left over (direct ttyS0 access for one serial). As usual I have a custom config for it, and if you are interested there is a link for you: tmux.conf. But the more interesting part and why I blog is that I also wrote a little helper around tmux. Ingeniously called tm. And a colleague said I should let other people use it too and place it somewhere accessible. So, err, here it is. Or better, the usage output of it, the actual script has a link somewhere below.
tmux helper by Joerg Jaspert <joerg@ganneff.de>
Call as: tm CMD [host]...
CMD is one of
 ls          List running sessions
 s           Open ssh session to host
 ms          Open multi ssh sessions to hosts, synchronizing input
 $anything   Either plain tmux session with name of $anything or
             session according to TMDIR file
TMDIR file:
Each file in $TMDIR defines a tmux session.
Filename is the commandline option $anything
Content is defined as:
  First line: Session name
  Second line: extra tmux options
  Any following line: A hostname to open a shell with in the normal
  ssh syntax. (ie [user@]hostname)
Environment variables recognized by this script:
TMPDIR - Where tmux stores its session information
         DEFAULT: If unset: /tmp
TMSORT - Should ms sort the hostnames, so it always opens the same
         session, no matter in which order hostnames are presented
         DEFAULT: true
TMOPTS - Extra options to give to the tmux call
         Note that this ONLY affects the final tmux call to attach
         to the session, not to the earlier ones creating it
         DEFAULT: -2
TMDIR  - Where are session information files stored
         DEFAULT: $ HOME /.tmux.d
TMWIN  - Where does your tmux starts numbering its windows?
         This script tries to find the information in your config,
         but as it only checks $HOME/.tmux.conf it might fail.
         So if your window numbers start at anything different to 0,
         like mine do at 1, then you can set TMWIN to 1

It is mainly useful for using tmux as a ssh-multiplexer and replacement for clusterssh though its happy to do any other tmux session too. As the help text may already have told you, you can use it to list existing tmux sessions (oh, wow!). Honestly, that one is there because tm ls is shorter than tmux ls More interestingly, tm s HOSTNAME opens a tmux session with the first window being a ssh shell to HOSTNAME. Should you later type tm s HOSTNAME again, it will attach to the existing session. But the real kicker for me (and colleagues) actually is the ms subcommand of it. ms being short for multi-session, it is my way of no-longer using clusterssh. Basically you type tm ms HOST1 user@HOST2 and it opens a session with one tmux window consisting of two panes. Showing you host1 and host2 at the same time. And the input to both is synchronized, that is - all your input is send to both of them. You aren t limited to just two hosts, do as many as you wish. Actually, I have no idea if tmux has an upper limit, but I did already have more than 100 hosts open at once. The major limitation there is not tmux - but simply the available screenspace, as all hosts are shown in ONE window! (The one advantage of clusterssh, it uses one xterm per host) This is quite helpful in all kinds of administration work involving clusters - imaging installing a package on 5 hosts of a cluster, you usually want it on all of them - or any kind of situations where one wants to run one command on many systems. As I m already said, I m lazy. And so there is a way to easily define multisessions that one uses regulary, by creating files inside the directory behind $TMDIR, usually $ HOME /.tmux.d. They are simple text files and let you easily define sessions you can then (re)open with a simple tm whateveritsname. And if you want the whole script by now: tm

Ritesh Raj Sarraf: I am so indebted to the community

As someone who learned computers on his own, I always acknowledged the value that the Free Software movement has brought. The accessibility of these topics, which are only supposed to be part of text books and schools, is available for anyone and everyone who has the resource and passion to do it. But this past week, 2 things made me pretty impressed with the maturity and quality of work that we do.

rrs@zan:/media$ sudo lvextend -r -v -L 100G /dev/BackupDisk/DATA

Finding volume group BackupDisk
Executing: fsadm --verbose check /dev/BackupDisk/DATA
fsadm: "ext4" filesystem found on "/dev/mapper/BackupDisk-DATA"
fsadm: Skipping filesystem check for device "/dev/mapper/BackupDisk-DATA" as the filesystem is mounted on /media/DATA
fsadm failed: 3
Archiving volume group "BackupDisk" metadata (seqno 7).
Extending logical volume DATA to 100.00 GiB
Found volume group "BackupDisk"
Found volume group "BackupDisk"
Loading BackupDisk-DATA table (254:2)
Suspending BackupDisk-DATA (254:2) with device flush
Found volume group "BackupDisk"
Resuming BackupDisk-DATA (254:2)
Creating volume group backup "/etc/lvm/backup/BackupDisk" (seqno 8).
Logical volume DATA successfully resized
Executing: fsadm --verbose resize /dev/BackupDisk/DATA 104857600K
fsadm: "ext4" filesystem found on "/dev/mapper/BackupDisk-DATA"
fsadm: Device "/dev/mapper/BackupDisk-DATA" size is 107374182400 bytes
fsadm: Parsing tune2fs -l "/dev/mapper/BackupDisk-DATA"
fsadm: Resizing filesystem on device "/dev/mapper/BackupDisk-DATA" to 107374182400 bytes (13107200 -> 26214400 blocks of 4096 bytes)
fsadm: Executing resize2fs /dev/mapper/BackupDisk-DATA 26214400
resize2fs 1.42.5 (29-Jul-2012)
Filesystem at /dev/mapper/BackupDisk-DATA is mounted on /media/DATA; on-line resizing required
old_desc_blocks = 4, new_desc_blocks = 7
The filesystem on /dev/mapper/BackupDisk-DATA is now 26214400 blocks long.
I didn't have much hope that this extend operation would succeed. But it did. When I initiated this operation, in the background, I had the backups being synced parallely (which actually made me resize my volume. :-)
The other item which made me happy yesterday was Audacity. Once upon a time, when I needed to split a music file to cut a ringtone out of it, I'd go looking for software that could do it. Then I would hope that one of those software vendors have a fully working version and not a demo with just 5 seconds clipping. Cowon was one media player I can recollect I've used to split audio files.
But in this case, I had a different requirement. I needed to increase the dB of my ringtone file so that it sounded really a ringtone (Example: Bourne Ultimatum OST). Audacity, not only did it do the job, it did it for me in just 3-5 minutes. And all with just button clicks. For a n00b with no experience in that domain, I was really impressed.

Next.

Previous.