Search Results: "xam"

16 March 2013

Steve Kemp: Handling bookmarks?

I've a collection of about 500 bookmarks which I've barely touched for a few years. I started organizing them late the other night, because I'd been off work sick for two days and that was about the most I felt up for doing with a computer. The intention was to "tidy" them, and then setup some way of syncing them across browsers/computers. In the end I didn't like any of the syncing plugins I could find - xmarks, etc - so I decided to take a step backwards. I'd exported my bookmarks to HTML page, via firefox, before I started, and then later in a fit of pique I deleted the whole damn lot of them. So now a few years worth of bookmarks are stored in a single HTML file. But wait, we can use revision control can't we? We can host that file on github/similar. We can rely upon merges to deal with conflicts - simple if we just add lines to the end, or delete lines. Maybe that's the best way to store bookmarks? I updated the bookmark file to read: 
<ul>
<li tags="debian, personal"><a href="http://www.debian-administration.org/">Debian Admin</a></li>
..
</ul>
Adding "tags" to the LI-container and then some simple jQuery code gave me the ability to search/filter the bookmarks and auto-populate tags. A small example placed online here: The obvious comment is that this makes adding new bookmarks a bit harder, but we'll see.. The javascript works in the browsers I tested, and for those that have none the bookmarks will just be a simple unordered list which should be universal. I expect the javascript could be improved by a real developer.

15 March 2013

Richard Hartmann: Release Critical Bug report for Week 11

Current prediction remains at six weeks. This is a tad meh. The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the Squeeze release cycle?
Week Squeeze Wheezy Diff
43 284 (213+71) 468 (332+136) +184 (+119/+65)
44 261 (201+60) 408 (265+143) +147 (+64/+83)
45 261 (205+56) 425 (291+134) +164 (+86/+78)
46 271 (200+71) 401 (258+143) +130 (+58/+72)
47 283 (209+74) 366 (221+145) +83 (+12/+71)
48 256 (177+79) 378 (230+148) +122 (+53/+69)
49 256 (180+76) 360 (216+155) +104 (+36/+79)
50 204 (148+56) 339 (195+144) +135 (+47/+90)
51 178 (124+54) 323 (190+133) +145 (+66/+79)
52 115 (78+37) 289 (190+99) +174 (+112/+62)
1 93 (60+33) 287 (171+116) +194 (+111/+83)
2 82 (46+36) 271 (162+109) +189 (+116/+73)
3 25 (15+10) 249 (165+84) +224 (+150/+74)
4 14 (8+6) 244 (176+68) +230 (+168/+62)
5 2 (0+2) 224 (132+92) +222 (+132/+90)
6 release! 212 (129+83) +212 (+129/+83)
7 release+1 194 (128+66) +194 (+128/+66)
8 release+2 206 (144+62) +206 (+144/+62)
9 release+3 174 (105+69) +174 (+105/+69)
10 release+4 120 (72+48) +120 (+72/+48)
11 release+5 115 (74+41) +115 (+74/+41)
12 release+6
13 release+7
14 release+8
15 release+9
16 release+10
17 release+11
18 release+12
Graphical overview of bug stats thanks to azhag:

14 March 2013

Iustin Pop: Types as control flow constructs

A bug I've recently seen in production code gave me the idea for this blog post. Probably smarter people already wrote better things on this topic, so this is mostly for myself, to better summarise my own thoughts. Corrections are welcome, please leave a comment! Let's say we have a somewhat standard API in Python or C++: Signalling failures to initialise the object can be done in two ways: either by raising an exception, or by returning a null/None result. There are advantages and disadvantages to both: The None model can create latent bugs, for example in the following code: 
ok = True
for arg in input_list:
  t = init_t(arg)
  if t is None:
    ok = False
    continue
  t.process()
return ok
The presence of the continue statement there is critical. Removing it, or moving it after some other statements which work with t will result in a bug. So using value-based returns forces us to introduce (manually) control points, without having the possibility to validate the model by the compiler (e.g. in C++). So it would seem that this kind of latent bugs pushes us to use the exception model, with its drawbacks. Let's look at how this interface would be implemented in (IMHO) idiomatic Haskell (where the a and b types represent the input and output types): 
initT :: a -> Maybe T
processT :: T -> b
my_fn =
  
 case initT arg of
   Nothing -> -- handle failure
   Just v -> processT v
Yes, this can be written better, but it's beside the main point. The main point is that by introducing a wrapper type around our main type (T), we are forced via the type system to handle the failure case. We can't simply pass the result of initT to a function which accepts T, because it won't type check. And, no matter what we do with the result value, there are no exceptions involved here, so we only have to think about types/values, and not control flow changes. In effect, types become automatically-validated control-flow instructions. Or so it looks to me . So using types properly, we can avoid the exception-vs-return-value debate, and have all advantages without the disadvantages of either. If that is the case, why isn't this technique used more in other languages? At least in statically typed languages, it would be possible to implement it (I believe), via a C++ template, for example. In Python, you can't actually apply it, as there's no static way of enforcing the correct decapsulation. I was very saddened to see that Google's Go language, which is quite recent, has many examples where initialisation functions return a tuple err, value = , separating the actual value from the error, making it not safer than Python. It might be that polymorphic types are not as easy to work with, or it might be the lack of pattern matching. In any case, I don't think this was the last time I've seen a null pointer dereference (or the equivalent AttributeError: 'NoneType' object has no attribute ). Sadly You can even go further in Haskell and introduce more control flow structure via wrapper types. Please bear another contrived example: an HTML form that gets some input data from the user, validates it, saves it to the database, and echoes it back to the user. Without types, you would have to perform these steps manually, and ensure they are kept in the correct order when doing future modifications. With types, you only have to design the types correctly and export only smart contructors (but not the plain ones): 
module Foo ( ValidatedValue
           , validateValue
           , RecordId
           , CommittedValue
           , commitValue
           , buildFeedbackForm
           ) where
data ValidatedValue a = ValidatedValue a
validateValue :: a -> Maybe (ValidatedValue a)
data RecordId =  
data CommittedValue a = CommittedValue a RecordId
commitValue :: ValidatedValue a -> ComittedValue a
buildFeedbackForm :: CommittedValue a -> HTMLDoc
From these types, it follows more or less that the only correct workflow is:
  1. get a value from the user
  2. validate it
  3. commit it, getting a transaction/record ID
  4. send the response to the user
In other words: 
handleFeedbackForm = buildFeedbackForm . commitValue . validateValue
There are still issues here, e.g. the type a is completely hidden behind the wrapper types, and we can't recover some basic properties (even if we use newtype, unless we use GeneralizedNewtypeDeriving). But it does offer a way to improve control flow safety. And that is my 0.02 currency unit for today.

Matthias Klumpp: Tanglu

Hi everyone! Today I make an announcement I thought I would never ever make. But things changed. Discussion about this has a long history, starting as a non-serious suggestion at DesktopSummit 2011, continued with people on IRC, but it was decided back then that it wouldn t be worth the effort. This has changed too, and a small team has formed to work on it. We hereby announce Tanglu, a new Debian-based-Linux distribution. A new logo?Why do we need another one? Let me explain the concepts of that distro: Tanglu will be based on Debian Testing and follow the Debian development closely. It will have a 6-months release-cycle and it s target audience are Linux desktop users. We will make installing and setting up the distro as easy as possible. Tanglu will be usable for both developers of upstream software and the average Linux user and Linux newbie. This is possible because in our opinion developers and users don t have different needs for a desktop system. Both kinds of users like a polished desktop which just works . We will, hwever, not apply any kind of fancy modification on upstream software, we will basically just distribute what upstream created, so users can get an almost pure GNOME and KDE experience. Tanglu is designed to be able to solve the issue that Debian is frozen for a long time and Debian Developers can t make new upstream versions available for testing easily. During a Debian freeze, DDs can upload their software to the current Tanglu development version and later start the new Debian cycle with already tested packages from Tanglu. The delta between Tanglu and Debian should be kept as minimal as possible. However, Tanglu is not meant as experimental distribution for Debian, so please upload experimental stuff to Experimental. Only packages good enough for a release should go into Tanglu. Ideally, Tanglu and Debian should be working well together in mixed environments, where you for example have Debian servers and multiple Tanglu desktops with the new software, targeted at desktop user. Since the differences between Tanglu and Debian should not be very high, administering both systems should be very easy (if you know Debian). Tanglu will be an open project, driven by community. At the beginning of each cycle, people can make suggestions for release goals they want to implement (similar to Fedora, but without FESCo). These proposals are discussed in public and are rejected if there are major technical concerns. If consensus about a certain proposal is lacking, a vote is done for it. The proposal can be accepted with absolute majority. If this does not happen, the proposal is postponed for the next release, where people can vote for it again. If nobody wants that function, it is rejected. In general, decisions made by Debian are superior and have to be followed. We don t think we know every package and every software better than the original upstream. That s why it makes much sense to rely on feedback from others and to have a community-based and peer-reviewed distribution, instead of secretly developing stuff and dumping it on the community. Tanglu will have a highly predictable set of features, defined at the beginning of a cycle, so you will know what you can expect from the next release as soon as possible and plan for it. Tanglu will make it easy to deploy applications for it. It will contain a software-center, similar to what Ubuntu has. We will also try to establish a solution for a Linux-AppCenter , a place for Linux applications, which will be open not only for Tanglu, but can be implemented in any other distribution too. Possible income will flow back into development of the platform. Now, let s answer the FQA (Future Questions Asked): Why don t you contribute to Debian directly and create yet another distribution? First of all, we contribute to Debian ;-) And for me, I can say that I will contribute to Debian even more. The point is that Debian can not cover all possible use-cases, and with Tanglu we want to make a distro which solves this. You might ask why we have to create a new distro for that, instead of creating improvements inside Debian? Creating a new distro allows us to do stuff we can never do in Debian. For example, we will include proprietary firmware in that distro, we will make installations of proprietary stuff possible easily (but don t ship with it by default) and we will have a time-based release cycle. These are already things which are a no-go for Debian, and that s fine. We don t want Debian to support these cases, as it is already a great distribution. We want to offer a distro as close to Debian as possible, but with a few modifications for use-cases which are not covered by Debian itself. Of course we will participate in DEX. If Debian Developers contribute to Tanglu, freezes will take even longer! This is an often-heard concern, it comes up on every mailinglist discussion about continuing development while freeze. I would disagree here, packaging new upstream stuff is not slowing down testing and improving of packages in Testing. Also, Tanglu is an offer for Debian developers to participate (we will sync privileges for their packages) we don t expect anyone to work on it, but as we think DDs know their packages best, we will make it possible for them to participate without extra barriers. We hope that Tanglu can add value to Debian and that Debian cycles can start with better-tested packages. You said you are a small team you cannot develop a whole distribution with it! Let s put that to the test! ;-) All people working on this are well aware of the issue that the project can not survive without much community-involvement on the long run. But we see a chance that many people are interested in it and that there is a high demand for it. At the beginning, we will just start with a small set of packages. We will also sync many packages from Ubuntu, to reduce workload. For example, it is planned to use the Ubuntu-Kernel and KDE packaging. By doing this, we keep the workload at the beginning low. We also reduce duplicate work with that. We even have some possible sponsors for the new distribution. But nothing is set in stone yet, so just wait for it to happen. :) Why not participate in Arch, OpenSUSE $other_distro? These are not Debian ;-) . I know, it sounds odd, but if you like the Debian way of doing things, you want to use a Debian-based distribution. There is nothing wrong with OpenSUSE. And Debian has issues too. But we want to be close to Debian and use it s tools and way of doing things. I hate you!!! You are doing it wrong!! The project is useless!
Well, that s fine. But there is no reason for hating us. If you dislike our idea, there are basically two options: First, you hate us but the project is successful. In that case, you have been wrong with hate, as there are definitely people who liked the project and contributed to it. Second, you hate us and we fail. In this case, there is no reason for hate, as the project will just vanish and you don t have to worry about it. So hating it would ve been just a big waste of energy. Also keep in mind that forking is a way to keep development healthy and to adapt software to new use-cases which it didn t target before. And we are not introducing incompatibilities here (like e.g. writing our own display server could). Instead, we want to stay close to Debian and reuse as much code as possible. Which desktop will you use? Everyone can add a new desktop to Tanglu, as long as the desktop-environment is present in Debian. Long term, we will have to offer Linux-newbies a default flavour, probably by setting a default download on the website. But as long as there is a community for a given desktop-environment, the desktop is considered as supported. At the beginning, we will focus on KDE, as many people have experience with it. But adding vanilla GNOME is planned too. Can you say something about the software used in Tanglu? Yes, but this is still in flow, so I can t promise final decisions here. On the server, side, we will try to use the Dak for repository management, as soon as we have enough server capacity. We will also use the standard Debian repository GUI and basically reuse most of the software there, to diverge less from Debian. The distribution itself will use a Linux Kernel from Ubuntu and systemd as the primary init system, as well as the login manager logind. It will be based on current Debian Testing with some fresh packages from Unstable and Experimental. We might also use the Ubuntu driver packages and KDE packaging. We expect to have a very rough start with the first release, but there will be enough time to polish Tanglu. Nice idea! How can I help? Well, you can help with basically anything at time from writing manuals, over designing logos and pages to administering a webserver and create packages. We are at an early stage of development at the moment, but we wanted to go public with it as soon as possible, to include the community and receive feedback so we can make that distro community-centric from the beginning. Most of the infrastructure is currently in progress too. So, if you want to get started with Tanglu, subscribe to our mailinglist tanglu-devel and write a mail to it, intruducing yourself. We can then include you in the loop. Generally, if you want to get access to our machines, a trusted GPG-signature will help a lot. If you want to talk to us, join #tanglu-devel or Freenode! Most discussions are currently happening there. And that s it! Tanglu will be awesome!

Some other projects of mine will develop a bit slower because I am now involved in Tanglu. But nothing will stop, and there is some pretty cool stuff coming for both GNOME and KDE (and I still have to implement DEP-11 for Debian). :)

Joerg Jaspert: zsh config - and prompt

As most people that are using that funny grey on black thing with the blinking white box AKA a commandline, I do have a heavily configured shell. zsh in my case. Some while ago my old configuration for it started to annoy me and so I set out to redo it. With the basic requirement to be flexible and able to adjust to various zsh versions (back as far as Lenny). I ve set out and looked how others do it and took the (I hope) best bits and put them together in a way I like it. Maybe others want to pick from me, so here I describe a little, maybe someone is interested First - the config is completly inside .zsh. The only thing directly in my $HOME is a symlink from $HOME/.zshenv to $HOME/.zsh/zshenv.home. All in one place, nicer this way. The zshenv just sets ZDOTDIR, so zsh then uses the .zsh itself. Next I am using modules . Well, actually everything is splitted out in small(er) files, so its way easier to maintain. And to overwrite stuff wherever I need to. So my .zshrc just loads those module files - and then goes and checks if it finds the same file inside a series of subdirectories. Those subdirectories are based on the hostname, the kernel name, the username, the domain name and the name of the distribution of the system. And a number of combinations of those. That system allows me, for example, to have a set of aliases defined only on Debian systems (like those to do with apt-get & co). Or define some extra variables if I login to the host franck, domain debian.org. Or whatever more. I m also on the way of doing something similar for all my dotfiles. While I was working at that I finally got fed up with the fact that my prompt, which is based on Phil! s wasn t doing what I wanted it on various of the hosts I access. Thankfully someone pointed me at a change to the grml zsh config: Frank Terbeck rewrote their prompt to make use of the zstyle system. Now that got me going, I like the idea of having the whole prompt easily configurable and changeable. My prompt always has shown slight differences depending on where the shell runs, which meant I had to carefully craft PS1 for every such change, and when one changes something one had to check all the places to adjust them too. No longer. I took Franks work and extended it for me. I wanted to still have a design matching my old prompt (minus colors, I changed color themes anyways), but with full flexibility the zstyle system can offer here. Now I am at a point where As an example, on the ftp-master host my prompt, while otherwise being the same as everywhere else, contains one extra item - displaying the current status of the archive (dinstall). If you want to try it, the setup file is available, but note that I don t give any warranty. :) Works for me . I load it in my Prompts module, while having the prompt setup file in my function path. The 4 commented lines, my testcase, are a nice example of a custom token, though a real life one, the setup for the ftp-master prompt with the one extra item can be found over here. Comments? Bugfixes? Enhancements? Update: Got asked for an example how to use my prompt to fake another. So here is what you need to do to use my prompt setup to imitate the prompt clint, as delivered with zsh: 
autoload promptinit && promptinit
zstyle ':prompt:ganneff' vcs_info true
# Alternatively you can set whatever you like it to be.
zstyle ':prompt:ganneff' set_vcs_info_defaults true
zstyle ':prompt:ganneff' colors true
## no right prompt
zstyle ':prompt:ganneff:right:setup' use-rprompt false
## color of brackets depending on variable
zstyle ':prompt:ganneff:*:items:openbracket' pre '$ $ SSH_CLIENT+"$ PR_YELLOW " :-"$ PR_RED " '
zstyle ':prompt:ganneff:*:items:openbracket' post '$ PR_NO_COLOR '
zstyle ':prompt:ganneff:*:items:closebracket' pre '$ $ SSH_CLIENT+"$ PR_YELLOW " :-"$ PR_RED " '
zstyle ':prompt:ganneff:*:items:closebracket' post '$ PR_NO_COLOR '
zstyle ':prompt:ganneff:*:items:openanglebracket' pre '$ $ SSH_CLIENT+"$ PR_YELLOW " :-"$ PR_RED " '
zstyle ':prompt:ganneff:*:items:openanglebracket' post '$ PR_NO_COLOR '
zstyle ':prompt:ganneff:*:items:closeanglebracket' pre '$ $ SSH_CLIENT+"$ PR_YELLOW " :-"$ PR_RED " '
zstyle ':prompt:ganneff:*:items:closeanglebracket' post '$ PR_NO_COLOR '
## extra date format, and its color
zstyle ':prompt:ganneff:*:items:date' token '%D %a %y/%m/%d %R %Z '
zstyle ':prompt:ganneff:*:items:date' pre '$ PR_CYAN '
## pts. %l instead of %y, also color again
zstyle ':prompt:ganneff:*:items:pts' token '%l'
zstyle ':prompt:ganneff:*:items:pts' pre '$ PR_GREEN '
zstyle ':prompt:ganneff:*:items:pts' post '$ PR_NO_COLOR '
## host ends with :
zstyle ':prompt:ganneff:*:items:host' token '%m:'
## and a different style for the rc level
zstyle ':prompt:ganneff:*:items:rc' token '%(?..[%?%1v] )'
zstyle ':prompt:ganneff:*:items:rc' pre ''
## change color for other parts that differ to my default
zstyle ':prompt:ganneff:*:items:user' pre '$ PR_GREEN '
zstyle ':prompt:ganneff:*:items:host' pre '$ PR_GREEN '
zstyle ':prompt:ganneff:*:items:at' pre '$ PR_GREEN '
zstyle ':prompt:ganneff:*:items:path' pre '$ PR_GREEN '
zstyle ':prompt:ganneff:*:items:history' pre ''
## Show the shell level in a different way
zstyle ':prompt:ganneff:*:items:shell-level' pre ''
zstyle ':prompt:ganneff:*:items:shell-level' token 'zsh%(2L./$SHLVL.) '
## And history is just a bold number. Lazy here, not using pre and post
zstyle ':prompt:ganneff:*:items:history' token '%B%h%b '
## And the last part in prompt, a # or % depending on privileges
## Using standard zsh tokens for bold
zstyle ':prompt:ganneff:*:items:privileges' pre '%B'
zstyle ':prompt:ganneff:*:items:privileges' post '%b'
## Now there are two parts not defined by default, so lets create them
## Both are simple existing variables, no need for extra precmd functions
## system info
zstyle ':prompt:ganneff:extra:ostype' pre '$ PR_CYAN '
zstyle ':prompt:ganneff:extra:ostype' post '$ PR_NO_COLOR '
zstyle ':prompt:ganneff:extra:ostype' token "$ MACHTYPE /$ OSTYPE /$(uname -r)"
## zsh version
zstyle ':prompt:ganneff:extra:zshvers' pre '$ PR_CYAN '
zstyle ':prompt:ganneff:extra:zshvers' post '$ PR_NO_COLOR '
zstyle ':prompt:ganneff:extra:zshvers' token "$ ZSH_VERSION "
zstyle ':prompt:ganneff:left:full:setup' items \
    openbracket date closebracket openbracket pts closebracket openbracket ostype closebracket \
    openbracket zshvers closebracket newline openanglebracket user at host \
    path closeanglebracket newline \
    shell-level space history rc space vcs privileges space
prompt ganneff
And if you now tell me Uh, thats so much more than just saying /prompt clint/ , then you are right. But then: This way is flexible. As I wrote above, my prompt on the ftpmaster host is different. Actually, my prompt on all machines inside the domain debian.org differs, by having 
zstyle ':prompt:ganneff:*:items:host' pre '$ PR_YELLOW '
in the Prompts definition for domain:debian.org. And thats enough to show me im on a .debian.org machine by a yellow hostname, instead of my default red. On the ftpmaster host I have the additional few lines: # Want one more piece in my prompt here, dinstall status zstyle :prompt:ganneff:left:full:setup items \ ulcorner line openparentheses user at host pts closeparentheses line history \ line dinstall line shell-level line flexline openparentheses path closeparentheses line urcorner newline \ llcorner line rc openparentheses time closeparentheses line vcs line change-root pipe space 
zstyle ':prompt:ganneff:extra:dinstall' pre '$ PR_CYAN '
zstyle ':prompt:ganneff:extra:dinstall' post '$ PR_NO_COLOR '
zstyle ':prompt:ganneff:extra:dinstall' token '$DINSTALL'
zstyle ':prompt:ganneff:extra:dinstall' precmd jj_update_dinstall
zmodload zsh/mapfile
jj_update_dinstall ()  
    DINSTALL="$ $ (z)$ (f)mapfile[/srv/ftp.debian.org/web/dinstall.status] [2] [3,99] "
And woo, thats simple. I think. (There ought to be a way to just easily add the item at a defined place in the items zstyle, but meh, too lazy to look).

Russ Allbery: Review: Higher Order Perl

Review: Higher Order Perl, by Mark Jason Dominus
Publisher: Morgan Kaufmann
Copyright: 2005
ISBN: 1-55860-701-3
Format: Trade paperback
Pages: 563
Disclaimer: The author and I both used to be (technically still are, but the newsgroup is dead) moderators of comp.lang.perl.moderated, and I knew him electronically from answering questions on the Perl Usenet newsgroups many years ago. Perl is, in general construction, a typical imperative language in the style of C, with object-oriented capabilities bolted (somewhat awkwardly) on. Most Perl code looks remarkably like C code with more powerful built-in data types and no memory management. A minority of Perl programs embrace object-oriented techniques and remind one of C++ or Java (or Python). But Perl also has powerful capabilities borrowed from functional programming languages and the Lisp tradition, even though many Perl programmers rarely use them and aren't familiar with them. Higher Order Perl focuses on those capabilities and how to use them effectively, starting with callbacks and uses of function pointers and moving into recursion and analysis of recursive functions, iterators, currying, and parsers. It concludes with a fully-worked example of constructing a declarative programming system using the techniques developed earlier in the book. Higher Order Perl is a programming book for an intermediate to advanced Perl programmer, already a rare topic. This is not a look at how to apply Perl to another area of programming, or a cookbook of techniques. It's an attempt to help a Perl programmer think about and use the language differently, and it follows through on that. It's refreshing and rare to read a programming technique book that's targeted at the practicing expert. (I think the size of the audience is often too small for publishers to target it.) Even the most experienced Perl programmer is probably going to learn something fundamental from this book, not just interesting trivia around the edges. On the negative side, though, I found the book a bit too focused on computer science and mathematics problems, particularly in the choice of examples and sample scripts. (And I say this as someone who has a master's degree in computer science with a software theory focus.) This is a bit hard to avoid for topics like recursion, where problems like computing Fibonacci numbers are classic, but throughout the book I struggled to focus past feelings of "but why would I ever do that"? The extended discussion of the Newton-Raphson method is the most memorable; I'm not sure that's a problem many Perl programmers would have and need a higher-order technique for. There's also a lot of discussion of recursion analysis and transformations between recursive and iterative expressions of problems, which is ground I remember well from my degree but which I've rarely had any practical use for in day-to-day programming. This is not a uniform problem, though, just a tendency. There are some great examples that I think are more in the mainstream of Perl problems, including a reinvention of File::Find that shows how to add more flexibility, a web spider, and a great discussion on how to construct a conventional queryable database on top of a flat file (a topic that's near and dear to my heart). And then there's the chapter on infinite streams. Dominus presents a method of using closures to create a version of an interator: a sort of infinite linked list that can keep generating additional elements. He presents this in several contexts, but one of them is log parsing, and that turned out to be exactly the solution that I needed for a problem I was working on while reading this book. I've written about this elsewhere, but this was a wonderful idea that helped me think about both Perl and a major application area in a completely new way, and I wrote an application using that knowledge that would have taken me much longer using different techniques and would have been much less fun. So, for me, this chapter was more than worth the entire book, and blankets the rest of it in a delighted feeling. Other people may or may not have that experience. I think it will depend on whether one of these techniques hits home for you the way that one did for me. This is a book with some idiosyncracies, and some sections that may drag. Having lots of fully-worked examples is a major plus, but some of those are so comprehensive that one can get a bit lost in the details. That particularly hit me with the last couple of chapters on parsing and on the example declarative programming application. Quite a bit of that text involved reinventing a recursive descent parser (another very computer science example), when I'm not quite sure why one wouldn't use one of the existing parser generators on CPAN for practical purposes, and inventing a lot of new syntax to try to make parser Perl code more readable. But one certainly can't complain that Dominus omits necessary details, and there is some appeal in watching an experienced programmer work through a problem from analysis to implementation. But, despite the idiosyncracies, I recommend this book to any experienced Perl programmer who wants to expand their view of the language. The techniques here closures, higher-order functions, iterators and streams, and formal parsers are powerful and underused in the Perl community. You may want to pick and choose which sections you pay close attention to, but I think everyone will find something of interest here. In the weeks since I read this book, my opinion of it has only grown. And I can't tell you just how much I loved the infinite stream concept. Rating: 8 out of 10

13 March 2013

Lars Wirzenius: cliapp 1.20130313 released (Python command line utility framework)

I've just released, to code.liw.fi, version 1.20130313 of cliapp, my Python framework for Unix-like command line programs. It contains the typical stuff such programs need to do, such as parsing the command line for options, and iterating over input files. Version 1.20130313 Bug fixes:

Dirk Eddelbuettel: Rcpp master class in New York last weekend

On Saturday I had the opportunity to teach another one-day master class on Rcpp. The class had been organized by Jared Lander, and organized very well I might add. The weekend started with a slight disappointment. I had taken Friday off, and hoped to reach NY by early afternoon to join JJ there, and to spend the afternoon with the RStan team. However, the tail end of last week's snowstorm made it such that we both got to Columbia's stats department closer to 6pm rather than 1pm, and half the team had left. Dang. Very frustrating travel experience. We salvaged the evening by gabbing over a cold beverage or two, before sharing some sacred New York pizza with Wes McKinney and Jared. The class itself on Saturday went quite well. With JJ on deck, we were able to have every participant log into an EC2-hosted instance of RStudio Server, which worked very well for usage examples of Rcpp. It has been almost a year since I last taught the class, and many exciting things--such as Rcpp attributes, added by JJ himself--have appeared, which made it extra fun. Participants were rather kind with praise. Either they really liked it, or they really are hard-nosed New Yorkers who manage to lie to my face without me noticing. We ended the day with some hard-earned cold beverages, followed by some dinner at Sylvia's (as tweeted by Jared) followed by more drinks. Ended up a little past my usual bedtime, but I managed to get out and enjoy a lovely 6.5 miles run across Central Park the next morning before leaving town. All in all, a very nice weekend, the travel horror of Friday notwithstanding. And who know, maybe we'll just do it again another time...

11 March 2013

Daniel Pocock: Firewalling IPv6

With my new ISP providing native IPv6 transit, enabling IPv6 in Debian is pretty trivial, just update /etc/networking/interfaces: 
iface eth0 inet6 static
	pre-up modprobe ipv6
	address 2001:67c:1388::2
	netmask 64
	gateway 2001:67c:1388::1
if you have a Linux router, enable packet forwarding: 
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
and add AAAA records to DNS: 
danielpocock.com.        IN      AAAA    2001:67c:1388:1000::9
Yes, that really is enough to get up and running. Apache was already configured to listen on IPv6 and just needed a restart. However, it is not quite secure: the configuration above routes all traffic on all ports. If somebody has the time to port scan my 64 billion trillion possible IPv6 addresses (every IPv6 customer is allocated a /48 as standard), they might end up connecting to some service that should not be public. ip6tables The ip6tables tool for IPv6 does just about everything that iptables does for IPv4. It is tempting to just rewrite an existing firewall script, adding an extra rule for the IPv6 version of every host. For example, my web server is 195.8.117.9, so I copy the 9 off the end and assign the IPv6 address 2001:67c:1388:1000::9 to the same server and add the ip6tables firewall rule: 
iptables --append FORWARD -p tcp -d 195.8.117.9 --dport http -j ACCEPT
ip6tables --append FORWARD -p tcp -d 2001:67c:1388:1000::9 --dport http -j ACCEPT
Notice that my IPv6 addressing simply involves re-using the same last byte as the IPv4 equivalent? This is a very easy way to start. However, it is not quite so simple. IPv6 requires a couple of extra rules, in the simplest form: 
# Allow traffic for link-local addresses:
ip6tables -A INPUT -s fe80::/10 -j ACCEPT
# Normally, link-local packets should NOT be forwarded and don't need an
# entry in the FORWARD rule.
# However, when bridging in Linux (e.g. in Xen or OpenWRT), the FORWARD rule is
# needed:
ip6tables -A FORWARD -s fe80::/10 -j ACCEPT
# Allow all ICMPv6 traffic (optional for IPv4, mandatory for IPv6):
ip6tables -A INPUT -p icmpv6 -j ACCEPT
ip6tables -A FORWARD -p icmpv6 -j ACCEPT
This works, but it turns out that it is less than ideal. Beyond that, the firewall can be significantly enhanced by rate-limiting the ICMPv6 packets and various other mechanisms as demonstrated with a practical netfilter example in Appendix B of RFC 4890 The question that arises, how to enable wide use of IPv6 without every sysadmin having to familiarise themselves with all these details of filtering ICMPv6 packets? Debian firewall packages My next step will be to look over the existing firewall packages in Debian to see which of them may answer that question. It would be really useful to hear feedback from anybody else who has already undertaken such an evaluation. If none of them are suitable, and if permitted by the licensing terms of RFC 4890, I may well wrap their sample code into a Debian package that can be used as a foundation for people building their own custom firewall scripts.

9 March 2013

Lars Wirzenius: Can the world be saved by coding?

I love programming. I was born to code. I want to save the world by writing code. I feel very strongly about software freedom. As I get older, and my eyes open to see more of the evil in the world, I become more concerned about freedom in general. Software freedom is, at least traditionally, about permission use, study, modify, and re-distribute software. The Free Software Foundation expresses this as the four essential freedoms. Debian expanded on that, and produced the Debian Free Software Guidelines. This is no longer enough. It is not enough to have all the freedom when you using your own computer, to have the source code to every bit of code that runs on your hardware. We live in the era of the Internet. Much of what we use computers for involves communication over the Internet, and the Internet is being actively used to curtail the freedom of people. For example, governments and corporations do large-scale surveillance of everyone, by eavesdropping on private communications, gathering enormous databases of personal data, and by analysing everything they can in order to find patterns and make conclusions both at the statistical level and about individuals. This ruins privacy. Without privacy, democracy cannot survive. Another example: in the name of various strawmen, such as terrorism, copyright violations, drugs, and child pornography, governments and corporations are collaborating in limiting private people's communications. I'm a Finnish citizen living in the UK. Both countries are among the closest ones to an ideal democractic nation state. Both countries arbitrarily block access to websites based on lists provided by private organisations, assuming that those organisations produce accurate lists of sites that contain copyright violations. As a result, both countries blocked, for example, a site where musical artists promoted their own works, bypassing the large media corporations that fund the list-making organisation. These issues transcend software freedom, though they interact with it. Software freedom is a necessary requirement for freedom, in a world where almost everything is done with, or controlled by, the use of computers. Software freedom is not enough to prevent censorship or surveillance: even if all the software in the world were free, including the national firewalls of China, Finland, and the UK, this would not prevent those countries from censoring and surveilling their citizens. The firewall would run free software, but that does not give the citizens the freedom to disable, or modify, the firewall systems. Fixing these issues is not a coding task. It is a job for politics, and it is going to take a long time, I fear. In the mean time, is there something a hacker can do to improve things? My main hobby project is Obnam, my backup program. Does that help people to protect their freedom? I think it does, in a small way: Obnam supports online, encrypted backups (and de-duplication even for encryption is used), which means they can make backups to servers anywhere on the Internet without having to fear their data gets read and analysed by hostile entities such as their own government, other governments, large corporations, or criminals. Obviously this does not help them if their government prohibits the use of encryption, or requires key escrow, or mandates backdoors to all encryption methods. But it's a step in the right direction. I don't claim Obnam will save the world, but a million small such steps by a thousand individual hackers, even without any particular organisation or guidance, will make a big impact. What step can you take?

8 March 2013

Richard Hartmann: Release Critical Bug report for Week 09

I extended UDD's rcblog historical bug overview table until week 18, which is the last week of April. Let's hope we won't need that long and I will be able to delete half of those. The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the Squeeze release cycle?
Week Squeeze Wheezy Diff
43 284 (213+71) 468 (332+136) +184 (+119/+65)
44 261 (201+60) 408 (265+143) +147 (+64/+83)
45 261 (205+56) 425 (291+134) +164 (+86/+78)
46 271 (200+71) 401 (258+143) +130 (+58/+72)
47 283 (209+74) 366 (221+145) +83 (+12/+71)
48 256 (177+79) 378 (230+148) +122 (+53/+69)
49 256 (180+76) 360 (216+155) +104 (+36/+79)
50 204 (148+56) 339 (195+144) +135 (+47/+90)
51 178 (124+54) 323 (190+133) +145 (+66/+79)
52 115 (78+37) 289 (190+99) +174 (+112/+62)
1 93 (60+33) 287 (171+116) +194 (+111/+83)
2 82 (46+36) 271 (162+109) +189 (+116/+73)
3 25 (15+10) 249 (165+84) +224 (+150/+74)
4 14 (8+6) 244 (176+68) +230 (+168/+62)
5 2 (0+2) 224 (132+92) +222 (+132/+90)
6 release! 212 (129+83) +212 (+129/+83)
7 release+1 194 (128+66) +194 (+128/+66)
8 release+2 206 (144+62) +206 (+144/+62)
9 release+3 174 (105+69) +174 (+105/+62)
Graphical overview of bug stats thanks to azhag:

Richard Hartmann: Release Critical Bug report for Week 10

By using highly advanced and secret analytical tools (i.e. stupid correlation), I am guessing that Wheezy will be released in about six weeks. Week 16... mark my words. The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the Squeeze release cycle?
Week Squeeze Wheezy Diff
43 284 (213+71) 468 (332+136) +184 (+119/+65)
44 261 (201+60) 408 (265+143) +147 (+64/+83)
45 261 (205+56) 425 (291+134) +164 (+86/+78)
46 271 (200+71) 401 (258+143) +130 (+58/+72)
47 283 (209+74) 366 (221+145) +83 (+12/+71)
48 256 (177+79) 378 (230+148) +122 (+53/+69)
49 256 (180+76) 360 (216+155) +104 (+36/+79)
50 204 (148+56) 339 (195+144) +135 (+47/+90)
51 178 (124+54) 323 (190+133) +145 (+66/+79)
52 115 (78+37) 289 (190+99) +174 (+112/+62)
1 93 (60+33) 287 (171+116) +194 (+111/+83)
2 82 (46+36) 271 (162+109) +189 (+116/+73)
3 25 (15+10) 249 (165+84) +224 (+150/+74)
4 14 (8+6) 244 (176+68) +230 (+168/+62)
5 2 (0+2) 224 (132+92) +222 (+132/+90)
6 release! 212 (129+83) +212 (+129/+83)
7 release+1 194 (128+66) +194 (+128/+66)
8 release+2 206 (144+62) +206 (+144/+62)
9 release+3 174 (105+69) +174 (+105/+69)
10 release+4 120 (72+48) +120 (+72/+48)
11 release+5
12 release+6
13 release+7
14 release+8
15 release+9
16 release+10
17 release+11
18 release+12
Graphical overview of bug stats thanks to azhag:

7 March 2013

Vincent Sanders: Man cannot discover new oceans unless he has the courage to lose sight of the shore

Continuing with my whirlwind introduction to NetSurf Development now is the time to start examining the code, how its arranged and how to interact with the existing developers.

The way the NetSurf source is structured is around the idea of the frontends each being a native browser. While this implies that there are nine separate browsers that happen to share a common code base the separation is not quite that well defined.

Each frontend provides the OS entry point (the main() function in a c program) and calls out to standard browser initialisation entry function netsurf_init() and then starts running the browsers main dispatch loop with netsurf_main_loop() when that exits the frontend cleans up with netsurf_exit().

The frontends provide a large selection of functions which are called from the core code. These routines run from running the event scheduler through to rendering graphics and text.
Finding your way aroundThe browsers directory layout is fairly shallow consisting of some Makefiles, the nine frontend directories and eleven others.

The Makefiles are GNU make and represent a pretty straightforward linear build system. We do not use recursive make or autotools. There are plans to use the core buildsystem that all the other components use.

The frontend directories contain the code for the frontend and the makefile fragments to build them which are included by the top level Makefile.

In addition there is:
desktop
This contains the non frontend specific code that actually behaves like a browser. For example desktop/netsurf.c contains the three primary functions we outlined in the introduction. You will also find much of the function and data structures interfaces the frontend must provide. It is unlikely someone new to the project will need to change anything in here (there be dragons) but is an important set of routines.
utils
Here you will find the utility and compatibility interfaces, things like url handling, logging, user messages, base64 handling. These are utility interfaces that do not justify splitting out their functionality to a separate library but are useful everywhere. Changing an interface in here would likely result in a major refactor.
For example one quirk is the logging macro was created before varadic C preprocessor macros were universal so it must be called as LOG(("hello %s world", world_type)) e.g. with double brackets. Fixing this and perhaps improving the logging functionality would be "nice" but the changes would be massive and potentially conflict with ongoing work.
content
This contains all the core code to handle contents i.e. html, css, javascript, images. The handling includes retrieval of the resources from URI , correct caching of the received objects and managing the objects state. It should be explicitly stated that the content handlers are separate and use this core functionality. The actual content handlers to deal with object contents such as the routines to decode image files or render html are elsewhere.
image
This is where the content handlers for the various image types are kept. The majority of these image types (jpeg, png, webp, gif, bmp) use a standard library to perform the actual decode (libjpeg, libpng etc.). One special feature used by most image handlers is that of a decoded image cache which is distinct and separate from the content cache.
The decoded image cache manages decoding of the source images (the jpegs and pngs) into frontend specific "render" bitmaps. For example the gtk frontend keeps the decoded images as a cairo surface ready for immediate plotting.
The cache uses a demand based (when the browser actually displays the image) just in time strategy which has been carefully balanced, with real world input, to reduce the overhead of unnecessary image decoding and processing against memory usage for the render bitmaps.
css
The css content handlers provide for the processing of css source text and use the NetSurf libcss library to process into a bytecode suitable for applying style selection at render time.
javascript
The javascript handlers (strictly speaking this should be named the "script" directory as all types of script are handled here) provide basic functionality to bind a javascript engine to the rest of the browser, principally the Document Object Model (DOM) accessed with libdom. The only engine that currently has bindings is Spidermonkey from the Mozilla foundation.
render
This is the heart of the browser containing the content handler for html (and plain text). This handler deals with:
  • Acquiring the html.
  • Running the base parser as data arrives which generates the DOM and hence DOM events from which additional resource (stylesheets etc.) fetches are started.
  • Deal with script loading
  • Constructing the box model used for layout
  • Performing the Layout and rendering of the document.
Because this module has so many jobs to do it has inevitably become very complex and involved, it is also the principle area of core development . Currently NetSurf lacks a dynamic renderer so changes made by scripts post document load event are not visible. This also has the side effect that the render is only started after the DOM has finished construction and all the resources on a page have completed their fetch which can lead to undesirable display latency.
Docs
Documentation about building and using NetSurf. If anyone wants a place to start improving NetSurf, this is it, it is very incomplete. It must be noted this is not where dynamically generated documentation is found. For the current Doxygen output the best place to look is the most recent build on the Continuous Integration system.
resources
These are runtime resources which are common to all frontends. To be strictly correct they may be the sources which get converted into runtime resources e.g. The FatMessages file which is teh message text for all frontends in all languages, this gets processed at build time into separate files ready.
!NetSurf
This is another resources directory and technically the resources for the riscos frontend. The naming and reliance on this directory are historical. To allow the RISC OS frontend to be run directly from the source directory and an inability of RISC OS to process symbolic links most common runtime resources end up in here and linked to from elsewhere.
test
These are some basic canned test programs and files, principally to test elements of the utils and perform specific exercise of various javascript components.
Getting started
Once a developer has a checked out working build environment and can run the executable for their chosen frontend (and maybe done some web browsing :-) it is time to look at contributing.

If a developer does not have a feature or bug in mind when they begin the best way to get started is often to go bug hunting. The NetSurf bugtacker has lots to choose from unfortunately. Do remember to talk to us (IRC is the best bet if you are bug hunting) about what you are up to but do not be impatient. Some of those bugs are dirty great Shelob types and are not being fixed because even the core developers are stumped!

When first getting going I cannot recommend reading the code enough, this seems to be a skill that many inexperienced open source developers have yet to acquire, especially if they are from a predominately proprietary development background. One wonderful feature of open source software is you get to see all of it, all the elegant nice code and all the "what the hell were they thinking" code too.

One important point is to use your tools well the source is in git, if you learn how to use git well you will gain a skill that is readily portable, not just for NetSurf. And not just revision control tools, learn to use your debugger well and tools like valgrind. Those skills will replay the time spent learning them themselves many times over.

When using git one thing to remember is commit early and often, it does not matter if you have lots of junk commits and dead ends, once you have something viable you can use 
git rebase --interactive
and rewrite it all into a single sensible set of commits. Also please do remember to develop on a branch, or if you did not 
git pull --rebase
is your friend to avoid unnecessary merges.
Playing nicely with othersThe NetSurf community is a small band of unpaid volunteers. On average we manage to collectively put in, perhaps, ten hours a week with the occasional developer weekend where we often manage over twenty hours each.

The result is that developer time is exceptionally valuable, add in a mature codebase with its fair share of technical debt and you get a group who, when they get to work on NetSurf, are incredibly busy. To be fair we are just like every other small open source project in that respect.

What this means to a new contributor is that they should try and do their homework before asking the obvious questions. The documentation is there for a reason and in spite of its obvious shortcomings please read it!

When asking questions it should be noted that currently the majority of active contributors are in the Europe so if you visit the IRC channel or post questions to lists the time difference is something to keep in mind.

I carefully said contributor above and not developer, users trying the CI builds and reporting results are welcome...as long as they report useful bugs to the bug tracker. Simon Tatham has produced an excellent resource on this subject.

Also we are always happy to receive translations to new languages (diff against the FatMessages file would be outstandingly useful but anything is welcome), artwork, documentation. Just recall what I mentioned about busy developers. Surest way to get us to see something is the development mailing list, you will probably get a reply, though I will not promise how fast!

Some of the more common mistakes when interacting with the community are:
In conclusion
Hopefully this has been of some use although I had hoped to cover more and provide deeper insights and advice on the codebase but there is only so much generalisation to be done before it is just easier for the developer to go read the code for themselves.

I look forward to lots of new contributions :-) though I fear this may all end up as more of a crib sheet for next time we do GSOC, time will tell.

Vincent Sanders: The way to get started is to quit talking and begin doing

When Walt Disney said that he almost certainly did not have software developers in mind. However it is still good advice, especially if you have no experience with a piece of software you want to change.

Others have written extensively on the topic of software as more than engineering and the creative aspects, comparing it to a craft, which is a description I am personally most comfortable with. As with any craft though you have to understand the material you have to work with and an existing codebase is often a huge amount of material.

While developing NetSurf we get a lot of people who talk a lot about what we "ought" or "should" do to improve the browser but few who actually get off their collective posteriors and contribute. In fact according to the ohloh analysis of the revision control system there are roughly six of us who contribute significantly on a regular basis and of those only four who have interests beyond a specific frontend.

It has been mentioned by a couple of new people who have recently compiled the browser from source that it is somewhat challenging to get started. To address this criticism I intend to give a whirlwind introduction to getting started with the NetSurf codebase, perhaps we can even get some more contributors!

This first post will cover the mechanics of acquiring and building the source and the next will look at working with the codebase and the Netsurf community.

This is my personal blog, the other developers might disagree with my approach, which is why this is in my blog and not on the NetSurf website. That being said comments are enabled and I am sure they will correct anything I get wrong.
ResourcesNetSurf has a selection of resources which are useful to a new developer:
Build environmentThe first thing a new developer has to consider is their build environment. NetSurf supports nine frontends on several Operating Systems (OS) but is limited on the build environment that can be used.

The developer will require a Unix like system but let's be honest, we have not tried with anything other than Linux distributions in some time or MAC OS X for the cocoa frontend because its a special snowflake.

Traditionally at this point in this kind of introduction it would be traditional to provide the command line for various packaging systems to install the build environment and external libraries. We do have documentation that does this but no one reads it, or at least it feels like that. Instead we have chosen to provide a shell fragment that encodes all the bootstrap knowledge in one place, its kept in the revision control system so it can be updated.

To use: download it, read it (hey running random shell code is always a bad idea), source it into your environment and run ns-apt-get-install on a Debain based system or ns-yum-install on Fedora. The rest of this posting will assume the functionality of this script is available, if you want to do it the hard way please refer to the script for the relevant commands and locations.

For Example:
$ wget http://git.netsurf-browser.org/netsurf.git/plain/Docs/env.sh
$ less env.sh
$ source env.sh
$ ns-apt-get-install

Historically NetSurf built on more platforms natively but the effort to keep these build environments working was extensive and no one was prepared to do the necessary maintenance work. This is strictly a build setup decision and does not impact the supported platforms.

Since the last release NetSurf has moved to the git version control system from SVN. This has greatly improved our development process and allows for proper branching and merging we previously struggled to implement.

In addition to the core requirements external libraries NetSurf depends on will need to be installed. Native frontends where the compiled output is run on the same system it was built on are pretty straightforward in that the native package management system can be used to install the libraries for that system.

For cross building to the less common frontends we provide a toolchain repository which will build the entire cross toolchain and library set (we call this the SDK) direct from source. This is what the CI system uses to generate its output so is well tested.
External LibrariesNetSurf depends upon several external development libraries for image handling, network fetching etc. The libraries for the GTK frontend are installed by default if using the development script previously mentioned.

Generally a minimum of libcurl, libjpeg and libpng are necessary along with whatever libraries are required for the toolkit.
Project Source and Internal Libraries
One important feature of NetSurf is that a lot of functionality is split out into libraries. These are internal libraries and although technically separate projects, releases bundle them all together and for development we assume they will all be built together.

The development script provides the ns-clone function which clones all the project sources directly from their various git repositories. Once cloned the ns-make script can be used to build and install all the internal libraries into a local target ready for building the browser.

For Example:

$ source env.sh
$ ns-clone
$ ns-make-libs install
Frontend selectionAs I have mentioned NetSurf supports several windowing environments (toolkits if you like) however on some OS there is only one toolkit so the two get conflated together.

NetSurf currently has nine frontends to consider:
Building and running NetSurfFor a developer new to the project I recommend that the gtk version be built natively which is what I describe here.

Once the internal libraries have been installed, building NetSurf itself is as simple as running make.

For Example:
$ source env.sh
$ ns-make -C $ TARGET_WORKSPACE /$ NS_BROWSER  TARGET=gtk

Though generally most developers would change into the netsurf source directory and run make there. The target (frontend) selection defaults to gtk on Linux systems so that can also be omitted Once the browser is built it can be run from the source tree to test.

For Example:
$ source env.sh
$ cd $ TARGET_WORKSPACE /$ NS_BROWSER 
$ ns-make
$ ./nsgtk

The build can be configured by editing a Makefile.config file. An example Makefile.config.example can be copied into place and the configuration settings overridden as required. The default values can be found in Makefile.defaults which should not be edited directly.

Logging is enabled with the command line switch -v and user options can be specified on the command line, options on the command line will override those sourced from a users personal configuration, generally found in ~/.netsurf/Choices although this can be compile time configured.

6 March 2013

Raphael Geissert: A bashism a week: returning

Inspired by Thorsten Glaser's comment about where you can break from, this "bashism a week" is about a behaviour not implemented by bash.

return is a special built-in utility, and it should only be used on functions and scripts executed by the dot utility. That's what the POSIX:2001 specification requires.

If you return from any other scope, for example by accidentally calling it from a script that was not sourced but executed directly, the bash shell won't forgive you: it does not abort the execution of commands. This can lead to undesired behaviour.

A wide variety of shell interpreters silently handle such calls to return as if exit had been called.

An easy way to avoid such undesired behaviours is to follow the best practice of setting the e option, i.e. 
set -e
. With that option set at the moment of calling return outside of the allowed scopes, bash will abort the execution, as desired.

The POSIX specification does not guarantee the above behaviour either as the result in such cases is "unspecified", however.

5 March 2013

Martin Pitt: Automatically generating documentation from GIR files

Many libraries build a GObject introspection repository (*.gir) these days which allows the library to be used from many scripting (Python, JavaScript, Perl, etc.) and other (e. g. Vala) languages without the need for manually writing bindings for each of those. One issue that I hear surprisingly often is there is zero documentation for those bindings . Tools for building documentation out of a .gir have existed for a long time already, just far too many people seem to not know about them. For example, to build Yelp XML documentation out of the libnotify bindings for Python: 
  $ g-ir-doc-tool --language=Python -o /tmp/notify-doc /usr/share/gir-1.0/Notify-0.7.gir
Then you can call yelp /tmp/notify-doc to browse the documentation. You can of course also use the standard Mallard tools to convert them to HTML for sticking them on a website: 
  $ cd /tmp/notify-doc
  $ yelp-build html .
Admittedly they are far from pretty, and there are still lots of refinements that should be done for the documentation itself (like adding language specific examples) and also for the generated result (prettification, dynamic search, and what not), but it s certainly far from nothign , and a good start. If you are interested in working on this, please show up in #introspection or discuss it on bugzilla, desktop-devel-list@, or the library specific lists/bug trackers.

Lisandro Dami&aacute;n Nicanor P&eacute;rez Meyer: My Debian freeze experience (so far)

This is the first freeze in which I'm involved with upload rights. And it turned out to be a quite interesting ride so far, so I thought it would be nice to write about it.

As some of you may know, I'm a part of the Qt/KDE team. Before the freeze I was mostly involved in leaf packages, with some patch here or there, nothing fancy. And then the freeze came...

Bugs in Qt

...and bugs appeared in Qt. But they didn't get solved, even if the patches were there. Due to personal reasons, the manpower in Qt/KDE land decreased below normal levels (which were already low).

I took the time to review them, apply them in a local branch, build and test the fixes. I did a Qt upload before, but it was a team-consented one. This time there was not much reaction in our IRC channel as it used to be, so I was doubting if going ahead or not. I asked Ana, my great friend and former sponsor, for an opinion on the subject, and she gave me a really important advice: the patches were looking good and there is one really big true: if something get's broken, it can be fixed with a later upload.

You might be asking yourself why I was that afraid of doing the upload. Well, when one maintains such a medular package for many users one has to be careful And I also got used that those "big ones" like Qt where normally handled by hand skilled people. Do not take me wrong here, it's not that those people where keeping them for themselves, it's knowing that one does not has the same skills nor experience as them.

But again, no one was able to upload and I had the chance and will to do another upload if needed, so off it went. That was Qt 4:4.8.2-2.

Then new experiences followed: asking a buildd maintainer for a giveback, asking the Release Team for an unblock (more on this later), etc. While sponsoring me, Ana gave me another excellent advice which I always keep in my mind:

You can't know **everything** about Debian.

And that also includes a not so technical skill: communicating with other teams. But finally we got this new version of Qt in testing. Cool :-)

Of course, new bugs appeared, and my lack of skills (and sometimes, time) where replaced by team work: Pino looking at patches and Sune contacting upstream. The eleven uploads that followed are a nice example of team work, even if I was the one who signed and did the uploads. Whoever uses Qt must know that these wonderful people (including those who are not so active nowadays like Modestas or Fathi) have done lots to bring the better to their users.Thank you guys!

Be careful, they might bite you back!

Coming back to the non-technical skills, sometimes you have to communicate with other teams in Debian. And each team is (naturally) a separate world: possibly different people, different goals, etc. Of course, we share the goal to make Debian the best experience we can, but we do not necessarily agree on the paths to achieve so.

During the freeze, there is a team that gets lots of pressure, and not by chance: the Release Team. They handle a very important task, which is to ride the freeze to get to a release. OK, that's what everyone knows. Now, one thing is knowing that and another is really understanding what does that means.

Of course I was in the first group. From the outside, communicating with the RT was a kind of "special art", and not an easy one. I have even been advised to not ask for more than one or two unblocks per weekend, as they might "bite me back". So I put on my flamesuit on and... launched reportbug release.debian.org.

Now I'm really happy to say that my experience was far from what I described above. And yes, I had the chance to even disagree on some stuff. But remember: non-technical skills, a.k.a. social skills. Once I started to know what was going on inside the RT (joining #debian-release was a big help for that) I learnt some nice tips to approach them. Please allow me to list some of them:

  • Remember: you are the maintainer of the package, they are like gatekeepers that are there to help us coordinate to do a release. But they don't maintain the code, you do that. So try to be verbose when needed, explain the changes and don't forget a nice diff. They need to understand what is going on: they can't read your mind.
  • They are human beings too: not everyday might be their best day (the same goes for you too!). And they are under the pressure of a release. Be patient, that finally pays off.
  • Does your changes seem not so clear? try to improve them.
  • The package has a lot of changes but you really feel they are needed? Try to explain that as good as you can.
  • Try to put yourself in their position: do we really want this? If in doubt, there is a nice way to know what they think: a pre-approval bug.
I want to make a stop in this last point. A pre-approval bug it's an unblock bug in which you edit the subject to add "pre-approval" in it. Easy, isn't it? It gives you the opportunity to know what the RT thinks before doing the upload. In other words: it gives you the chance to communicate and do things in the best possible way for all the parts involved.

I've have also seen pre-approval bugs that were really not needed. But to learn where the threshold of what can be directly uploaded and what deserves a pre-approval bug is you need to know the guidelines the RT gives you. Do you still have doubts? fire a pre-approval bug and try to be clear.

Of course, this are all fruits of my experience with the RT during this time. If the RT thinks different from what I'm writing here, please stand up: we are hear to listen to you and learn :-)

As a side note, I think I should file a wishlist bug to include the pre-approval bug option in reportbug. Yes, I'm lazy :-)

Summing up

Overall this was a very nice and positive experience. We are not done yet. Are we really done at some point? Let's hope not, because this is where the fun comes from :-)

Russell Coker: Kogan Mobile

Kogan Mobile is the newest virtual telco in Australia [1]. They resell Telstra 3G (not NextG or LTE) and while their coverage isn t as good as the full Telstra service it s more than adequate for my needs as they provide 3G coverage to 97% of the population and 2G+3G coverage to 98.5%. Their coverage is probably a lot better than Three who had the worst record of network coverage in Australia yet managed to always provide coverage where I wanted it I was a happy Three customer for more than 6 years. Kogan s main selling point is that they offer unlimited calls to Australian mobile phones and land-lines and unlimited SMS for a pre-paid fee of only $300 per annum ($25 per month on average). My parents have been getting unpleasantly large phone bills which have considerably more than $25 of calls to mobile phones every month so an obvious solution for them is to sign up for a Kogan mobile phone and use it for all such calls. There are other ways my parents could save money on calls (such as VOIP) but a mobile phone is easiest and offers other benefits such as running Android apps (when compared to using a non-smart phone). 6G of Data! Kogan also offers 6G of data per month, the down-side to this is that they bill in 1MB increments per session . I was worried that this might be per TCP connection or something else silly but I decided to sign my parents up for it as they aren t going to use a lot of data (they claim that they don t want to use the Internet on their phone but I know better). I ve done some tests on the SIM I got for my parents. For testing purposes I installed the Kogan SIM in my wife s new Nexus 4 and had it provide Wifi net access to my phone while we were playing Ingress. So far after 2 days which involved a reasonable amount of Ingress (I reached level 7) as well as all the usual stuff that happens in the background for two phones (checking email, news, weather, etc) Kogan considers that 177MB have been used out of the 6144MB for the month, which means that even with what is an unusual amount of traffic for us the account in question still isn t going to use half the quota for the month. Now that 10 days have elapsed with less intense usage Kogan considers that a total of 1373MB have been used.
days received sent Kogan
2 127.24 41.54 177
10 959.39 299.6 1373
Kogan also don t seem to mention whether they bill for transmitted data. I used the 3G Watchdog app to measure the amount data transferred, the above table has the amounts of data that 3G Watchdog considers were sent and received along with the amount that is listed by the Kogan Android app. My past experience with 3G Watchdog and Virgin Mobile is that it s usually quite accurate but has been over-reporting the data transfers recently (I think that Virgin is only billing me for downloads while 3G Watchdog counts uploads). So the relatively small difference between the 3G Watchdog report and what Kogan thinks I ve done means that either rounding the session up to the nearest meg doesn t make any significant difference (which would imply that a session can be a long time) or that Kogan isn t counting uploaded data and the session rounding up only adds about 40% to the total recorded transfer. My current plan with Virgin Mobile gives me 1.5G per month of quota, so as long as Kogan s rounding doesn t increase the recorded data transfer by a factor of 4 I will still be able to transfer more data with Kogan while paying less. One disadvantage of using Kogan is that I might have to tweak programs like my email program to poll less frequently to avoid excessive session charges (a program polling every 5 minutes would use up the 6G quota in 21 days if each poll counted as a session) although current tests indicate that this won t be necessary. But the up-side is that there are no extra fees with Kogan, they merely restrict data access for my use and that of most people I know it s better to have data access cut off than to receive a large bill. The Kogan Android App Kogan has an Android app that will give the status of your account and allow you to change the plan etc. This is quite nice but one major disadvantage is that it s also a sales app for the Kogan online store. This is bad for the user as some aspects of what I consider the core functionality are limited (for example there s no way to force a poll of the data usage count or determine how current the data is). But there s an obvious advantage to Kogan in providing a way to sell their goods that is going to be used by every customer of Kogan Mobile. The sales part of the app isn t very functional IMHO, it doesn t seem to have basic functionality such as sorting a list of items by price. Benefits of Kogan 6G of data is a lot! $300 per annum is quite cheap, anyone who makes any serious use of phones will be paying more than that in Australia. Lack of extra fees means that there is little need to restrict net access. I can risk getting cut off near the end of the month but I can t risk the potential for hundreds of dollars in excess fees. The Kogan app shows me the data used so I will probably uninstall 3G Watchdog, having one less program running is a good thing. You get a free SIM (value $5) when you buy a phone from Kogan. Disadvantages of Kogan They are new to the Telco business and admit that their customer service is lacking due to unexpected demand. If you order a SIM now they state that it will arrive in April. Apparently they are deliberately delaying orders because they can t cope with demand. The included call quota doesn t include international calls. While unlimited free calls in Australia is great if you make many international calls then this could end up costing you more. Other mobile telcos such as Lebara offer good deals for International calls, it could be an option to use a Lebara SIM with an old non-smart phone while using Kogan for your smart phone. I am concerned about the lack of detail about how data is accounted. If the definition of a session changes then 6G could turn out not to be enough. As Kogan is reselling a Telstra service it is possible that Telstra could change the deal without Kogan being able to stop them. Conclusion I will move my phone and my wife s phone to Kogan ASAP. My general idea is to sign us up for Kogan about 2 weeks apart, so if one phone runs out of the 6G data quota then the other phone can be used as a Wifi access point for 2 weeks. If the phones don t both have their quota end at the same time then there is less chance of both phones running out during a high traffic month.
Related posts:
  1. Australian Mobile Phone Costs I previously wrote about the changes to bundling prices by...
  2. Choosing an Australian Mobile Telco for use with Android Since playing with the IBM Seer augmented reality software [1]...
  3. Dual SIM Phones vs Amaysim vs Contract for Mobile Phones Currently Dick Smith is offering two dual-SIM mobile phones for...

2 March 2013

Bastian Blank: LDAP, Insignificant Space and Postfix

For some, LDAP, just like X.500, is black magic. I won't argue against. Sometimes it really shows surprising behavior. It always makes sense, if you think about what LDAP is built for. One surprising behavior is the handling of the "Insignificant Space"
LDAP supports syntax and comparator methods. The syntax specifies how the entry should look. Usually this is some form of text, but numbers or stuff like telephone numbers are supported as well. The comparators specifies how the values are compared. Most of the text comparators are defined to apply the handling of insignificant spaces. Insignificant space handling normalizes the use of spaces. First all leading and trailing spaces are removed. All the internal spaces are normalized to at most two spaces. At the end all strings starts and ends with one space to allow proper sub-string matches. The resulting strings are used for comparisons. This behavior makes sense most of the time. If the user want to find something in the directory, he usually don't cares about spaces, but about content. But I found one occurrence where this produces some grieve. Postfix supports LDAP since some time. And lets say, it does not care about spaces in its queries. This is no problem as e-mail addresses do not contain spaces. Or do they?
Yes, e-mail addresses can contain spaces. This is not widely known, but still allowed. This addresses are quoted and the command looks like RCPT TO:<" test"@example.com>. The local part is quoted and contains a space at the beginning. And this is where the problem starts. Postfix sanitizes the address. It uses a simplified internal representation of the address for all lookups. So the address gets test@example.com. This form is used in all table lookups. The LDAP table uses the internal form of the address. This address is copied verbatim into the query. This query may look this way (mail= test@example.com). It is sent to the server this way. The LDAP server applies the insignificant space modifications. The query is now interpreted and the comparator specified modifications are applied. The query gets effectively changed to (mail=test@example.com). And this is where the fun starts. Postfix accepts undeliverable mails. Depending on the setup, such LDAP queries may be used to check for valid addresses. Because of the space handling, the sender can add spaces to the beginning of the address and it will still be considered valid. In later steps this addresses are not longer valid. Addresses starting with spaces are considered invalid in some locations of Postfix. What surprised me a bit is that virtual alias handling did not map them. The unmodified addressed showed up on the LMTP backend server. That's why they showed up my radar. I would say Postfix is wrong in this case. The LDAP server applies the spec correctly and defines spaces in e-mail addresses as insignificant. Postfix on the other side considers them significant. The easiest fix would be to not allow any spaces in the LDAP table.

1 March 2013

Daniel Pocock: Monitoring JBoss, Tomcat and other application servers with JMXetric

JMXetric 1.0 was finally released in February, let's have a quick look at how conveniently it can be integrated with an application server. A more thorough analysis of gmetric4j and JMXetric can be found in the sections I contributed to the new O'Reilly book Monitoring with Ganglia General comments about JVM instrumentation and the JVM boot classpath Monitoring agents (there are a number of them out there) are loaded at an early stage by the JVM and it needs to have access to any of their dependencies through the boot classpath. JMXetric explicitly declares it's dependencies in MANIFEST.MF:
Boot-Class-Path: oncrpc-1.0.7.jar gmetric4j-1.0.2.jar
 As no absolute path is specified, the JVM will look for those JARs in the same directory where the jmxetric.jar is located. Furthermore, the JVM will expect the exact filenames to match. If other filenames are used (e.g. for a newer version of gmetric), then the JVM command line needs to specify the filenames and locations, for example:
-Xbootclasspath/a:/opt/ganglia/jmxetric/gmetric4j-1.0.3.jar
 Starting JBoss 7 with JMXetric To start with, unpack a vanilla JBoss distribution:
$ mkdir ~/opt
$ cd ~/opt
$ tar xzf ~/Downloads/jboss-as-7.1.0.Final.tar.gz
 Now get the JMXetric code and put it, together with dependencies, in the same directory:
$ mkdir -p ~/opt/ganglia/jmxetric
$ cd ~/opt/ganglia/jmxetric
$ wget http://search.maven.org/remotecontent?filepath=info/ganglia/jmxetric/jmxetric/1.0.2/jmxetric-1.0.2.jar
$ wget http://search.maven.org/remotecontent?filepath=info/ganglia/gmetric4j/gmetric4j/1.0.2/gmetric4j-1.0.2.jar
$ wget http://search.maven.org/remotecontent?filepath=org/acplt/oncrpc/1.0.7/oncrpc-1.0.7.jar
 Set up a config file for JMXetric:
$ mkdir -p ~/opt/ganglia/jmxetric/etc
$ cd ~/opt/ganglia/jmxetric/etc
$ wget https://raw.github.com/ganglia/jmxetric/master/etc/jmxetric.xml
$ cp jmxetric.xml jmxetric-jboss.xml
$ vi jmxetric-jboss.xml
 and insert the following code: 
                <!-- report web server request count -->
		<mbean name="jboss.as:subsystem=web,connector=http" pname="http">
			<attribute name="requestCount" type="int32"/>
		</mbean>
Finally, tell JBoss about the agent:
$ cd ~/opt/jboss-as-7.1.0.Final
$ vi bin/standalone.conf
 and apply the following changes to standalone.conf: 
--- bin/standalone.conf
+++ ../jboss-as-7.1.0.Final.for-jmxetric/bin/standalone.conf
@@ -4,6 +4,7 @@
 ##                                                                          ##
 ##############################################################################
 
+JBOSS_MODULES_SYSTEM_PKGS="org.jboss.logmanager"
 #
 # This file is optional; it may be removed if not needed.
 #
@@ -48,6 +49,8 @@
 #
 if [ "x$JAVA_OPTS" = "x" ]; then
    JAVA_OPTS="-Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
+   JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:$JBOSS_HOME/modules/org/jboss/logmanager/main/jboss-logmanager-1.2.2.GA.jar -Xbootclasspath/p:$JBOSS_HOME/modules/org/jboss/logmanager/log4j/main/jboss-logmanager-log4j-1.0.0.GA.jar -Xbootclasspath/p:$JBOSS_HOME/modules/org/apache/log4j/main/log4j-1.2.16.jar"
+   JAVA_OPTS="$JAVA_OPTS -javaagent:$HOME/opt/ganglia/jmxetric/jmxetric-1.0.2.jar=host=jboss-host.example.org,port=8649,wireformat31x=true,config=$HOME/opt/ganglia/jmxetric/etc/jmxetric-jboss.xml"
    JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"
    JAVA_OPTS="$JAVA_OPTS -Djboss.server.default.config=standalone.xml"
 else
Note that we are adding some of the core JBoss logging code to the boot classpath - that is because JMXetric depends on the logger, and like many monitoring agents, it is impacted by JBoss bug AS-3681. Adding these extra classpath settings is the workaround. Now go ahead and start JBoss, you should see a log message informing you that JMXetric is operational: 
$ cd ~/opt/jboss-as-7.1.0.Final
$ chmod a+x bin/*.sh
$ bin/standalone.sh
=========================================================================
  JBoss Bootstrap Environment
  JBOSS_HOME: ...
.
.
.
=========================================================================
JMXetricAgent instrumented JVM, see https://github.com/ganglia/jmxetric
12:59:34,395 INFO  [info.ganglia.gmetric4j.GMonitor] Setting up 1 samplers
12:59:34,536 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
12:59:34,617 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
12:59:34,653 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
.
.
.
I hope this example is useful for you. Please feel free to share recipes you develop for collecting useful JVM metrics, and also any custom charts/reports through collaboration with the Ganglia community mailing list, the Ganglia wiki or your own blog.

Next.

Previous.