As preannounced here the GNOME Infrastructure switched to a new Account Management System which is reachable at https://account.gnome.org. All the details will follow. Introduction It s been a while since someone actually touched the underlaying authentication infrastructure that powers the GNOME machines. The very first setup was originally configured by Jonathan Blandford (jrb) who configured an OpenLDAP istance with several customized schemas. (pServer fields in the old CVS days, pubAuthorizedKeys and GNOME modules related fields in recent times) While OpenLDAP-server was living on the GNOME machine called clipboard (aka ldap.gnome.org) the clients were configured to synchronize users, groups, passwords through the nslcd daemon. After several years Jeff Schroeder joined the Sysadmin Team and during one cold evening (date is Tue, February 1st 2011) spent some time configuring SSSD to replace the nslcd daemon which was missing one of the most important SSSD features: caching. What surely convinced Jeff to adopt SSSD (a very new but promising sofware at that time as the first release happened right before 2010 s Christmas) and as the commit log also states ( New sssd module for ldap information caching ) was SSSD s caching feature. It was enough for a certain user to log in once and the /var/lib/sss/db directory was populated with its login information preventing the LDAP daemon in charge of picking up login details (from the LDAP server) to query the LDAP server itself every single time a request was made against it. This feature has definitely helped in many occasions especially when the LDAP server was down for a particular reason and sysadmins needed to access a specific machine or service: without SSSD this wasn t ever going to work and sysadmins were probably going to be locked out from the machines they were used to manage. (except if you still had /etc/passwd , /etc/group and /etc/shadow entries as fallback) Things were working just fine except for a few downsides that appeared later on:

1. the web interface (view) on our LDAP user database was managed by Mango, an outdated tool which many wanted to rewrite in Django that slowly became a huge dinosaur nobody ever wanted to look into again
2. the Foundation membership information were managed through a MySQL database, so two databases, two sets of users unrelated to each other
3. users were not able to modify their own account information on their own but even a single e-mail change required them to mail the GNOME Accounts Team which was then going to authenticate their request and finally update the account.

Today s infrastructure changes are here to finally say the issues outlined at (1, 2, 3) are now fixed. What has changed? The GNOME Infrastructure is now powered by Red Hat s FreeIPA which bundles several FOSS softwares into one big bundle all surrounded by an easy and intuitive web UI that will help users update their account information on their own without the need of the Accounts Team or any other administrative entity. Users will also find two custom fields on their Overview page, these being Foundation Member since and Last Renewed on date . As you may have understood already we finally managed to migrate the Foundation membership database into LDAP itself to store the information we want once and for all. As a side note it might be possible that some users that were Foundation members in the past won t find any detail stored on the Foundation fields outlined above. That is actually expected as we were able to migrate all the current and old Foundation members that had an LDAP account registered at the time of the migration. If that s your case and you still would like the information to be stored on the new setup please get in contact with the Membership Committee at stating so. Where can I get my first login credentials? Let s make a little distinction between users that previously had access to Mango (usually maintainers) and users that didn t. If you were used to access Mango before you should be able to login on the new Account Management System by entering your GNOME username and the password you were used to use for loggin in into Mango. (after loggin in the very first time you will be prompted to update your password, please choose a strong password as this account will be unique across all the GNOME Infrastructure) If you never had access to Mango, you lost your password or the first time you read the word Mango on this post you thought why is he talking about a fruit now? you should be able to reset it by using the following command:

ssh -l yourgnomeuserid account.gnome.org

The command will start an SSH connection between you and account.gnome.org, once authenticated (with the SSH key you previously had registered on our Infrastructure) you will trigger a command that will directly send your brand new password on the e-mail registered for your account. From my tests seems GMail sees the e-mail as a phishing attempt probably because the body contains the word password twice. That said if the e-mail won t appear on your INBOX, please double-check your Spam folder. Now that Mango is gone how can I request a new account? With Mango we used to have a form that automatically e-mailed the maintainer of the selected GNOME module which was then going to approve / reject the request. From there and in the case of a positive vote from the maintainer the Accounts Team was going to create the account itself. With the recent introduction of a commit robot directly on l10n.gnome.org the number of account requests reduced its numbers. In addition to that users will now be able to perform pretty much all the needed maintenance on their accounts themselves. That said and while we will probably work on building a form in the future we feel that requesting accounts can definitely be achieved directly by mailing the Accounts Team itself which will mail the maintainer of the respective module and create the account. As just said the number of account creations has become very low and the queue is currently clear. The documentation has been updated to reflect these changes at: https://wiki.gnome.org/AccountsTeam
https://wiki.gnome.org/AccountsTeam/NewAccounts The migration of all the user data and ACLs has been massive and I ve been spending a lot of time reviewing the existing HBAC rules trying to spot possible errors or misconfigurations. If you happen to not being able to access a certain service as you were used to in the past, please get in contact with the Sysadmin Team. All the possible ways to contact us are available at https://wiki.gnome.org/Sysadmin/Contact. What is missing still? Now that the Foundation membership information has been moved to LDAP I ll be looking at porting some of the existing membership scripts to it. What I managed to port already are welcome e-mails for new or existing members. (renewals) Next step will be generating a membership page from LDAP (to populate http://www.gnome.org/foundation/membership) and all the your-membership-is-going-to-lapse e-mails that were being sent till today. Other news /home/users mount on master.gnome.org You will notice that loggin in into master.gnome.org will result in your home directory being empty, don t worry, you did not lose any of your files but master.gnome.org is now currently hosting your home directories itself. As you may have been aware of adding files to the public_html directory on master resulted in them appearing on your people.gnome.org/~userid space. That was unfortunately expected as both master and webapps2 (the machine serving people.gnome.org s webspaces) were mounting the same GlusterFS share. We wanted to prevent that behaviour to happen as we wanted to know who has access to what resource and where. From today master s home directories will be there just as a temporary spot for your tarballs, just scp and use ftpadmin against them, that should be all you need from master. If you are interested in receiving or keeping using your people.gnome.org s webspace please mail stating so. Other news a shiny and new error 500 page has been deployed Thanks to Magdalen Berns (magpie) a new error 500 web page has been deployed on all the Apache istances we host. The page contains an iframe of status.gnome.org and will appear every single time the web server behind the service you are trying to reach will be unreachable for maintenance or other purposes. While I hope you won t see the page that often you can still enjoy it at https://static.gnome.org/error-500/500.html. Make sure to whitelist status.gnome.org on your browser as it currently loads it without https. (as the service is currently hosted on OpenShift which provides us with a *.rhcloud.com wildcard certificate, which differs from the CN the browser would expect it to be)

As preannounced here the GNOME Infrastructure switched to a new Account Management System which is reachable at https://account.gnome.org. All the details will follow. Introduction It s been a while since someone actually touched the underlying authentication infrastructure that powers the GNOME machines. The very first setup was originally configured by Jonathan Blandford (jrb) who configured an OpenLDAP istance with several customized schemas. (pServer fields in the old CVS days, pubAuthorizedKeys and GNOME modules related fields in recent times) While OpenLDAP-server was living on the GNOME machine called clipboard (aka ldap.gnome.org) the clients were configured to synchronize users, groups, passwords through the nslcd daemon. After several years Jeff Schroeder joined the Sysadmin Team and during one cold evening (date is Tue, February 1st 2011) spent some time configuring SSSD to replace the nslcd daemon which was missing one of the most important SSSD features: caching. What surely convinced Jeff to adopt SSSD (a very new but promising sofware at that time as the first release happened right before 2010 s Christmas) and as the commit log also states ( New sssd module for ldap information caching ) was SSSD s caching feature. It was enough for a certain user to log in once and the /var/lib/sss/db directory was populated with its login information preventing the LDAP daemon in charge of picking up login details (from the LDAP server) to query the LDAP server itself every single time a request was made against it. This feature has definitely helped in many occasions especially when the LDAP server was down for a particular reason and sysadmins needed to access a specific machine or service: without SSSD this wasn t ever going to work and sysadmins were probably going to be locked out from the machines they were used to manage. (except if you still had /etc/passwd , /etc/group and /etc/shadow entries as fallback) Things were working just fine except for a few downsides that appeared later on:

1. the web interface (view) on our LDAP user database was managed by Mango, an outdated tool which many wanted to rewrite in Django that slowly became a huge dinosaur nobody ever wanted to look into again
2. the Foundation membership information were managed through a MySQL database, so two databases, two sets of users unrelated to each other
3. users were not able to modify their own account information on their own but even a single e-mail change required them to mail the GNOME Accounts Team which was then going to authenticate their request and finally update the account.

Today s infrastructure changes are here to finally say the issues outlined at (1, 2, 3) are now fixed. What has changed? The GNOME Infrastructure is now powered by Red Hat s FreeIPA which bundles several FOSS softwares into one big bundle all surrounded by an easy and intuitive web UI that will help users update their account information on their own without the need of the Accounts Team or any other administrative entity. Users will also find two custom fields on their Overview page, these being Foundation Member since and Last Renewed on date . As you may have understood already we finally managed to migrate the Foundation membership database into LDAP itself to store the information we want once and for all. As a side note it might be possible that some users that were Foundation members in the past won t find any detail stored on the Foundation fields outlined above. That is actually expected as we were able to migrate all the current and old Foundation members that had an LDAP account registered at the time of the migration. If that s your case and you still would like the information to be stored on the new setup please get in contact with the Membership Committee at stating so. Where can I get my first login credentials? Let s make a little distinction between users that previously had access to Mango (usually maintainers) and users that didn t. If you were used to access Mango before you should be able to login on the new Account Management System by entering your GNOME username and the password you were used to use for loggin in into Mango. (after loggin in the very first time you will be prompted to update your password, please choose a strong password as this account will be unique across all the GNOME Infrastructure) If you never had access to Mango, you lost your password or the first time you read the word Mango on this post you thought why is he talking about a fruit now? you should be able to reset it by using the following command:

ssh -l yourgnomeuserid account.gnome.org

The command will start an SSH connection between you and account.gnome.org, once authenticated (with the SSH key you previously had registered on our Infrastructure) you will trigger a command that will directly send your brand new password on the e-mail registered for your account. From my tests seems GMail sees the e-mail as a phishing attempt probably because the body contains the word password twice. That said if the e-mail won t appear on your INBOX, please double-check your Spam folder. Now that Mango is gone how can I request a new account? With Mango we used to have a form that automatically e-mailed the maintainer of the selected GNOME module which was then going to approve / reject the request. From there and in the case of a positive vote from the maintainer the Accounts Team was going to create the account itself. With the recent introduction of a commit robot directly on l10n.gnome.org the number of account requests reduced its numbers. In addition to that users will now be able to perform pretty much all the needed maintenance on their accounts themselves. That said and while we will probably work on building a form in the future we feel that requesting accounts can definitely be achieved directly by mailing the Accounts Team itself which will mail the maintainer of the respective module and create the account. As just said the number of account creations has become very low and the queue is currently clear. The documentation has been updated to reflect these changes at: https://wiki.gnome.org/AccountsTeam
https://wiki.gnome.org/AccountsTeam/NewAccounts I was used to have access to a specific service but I don t anymore, what should I do? The migration of all the user data and ACLs has been massive and I ve been spending a lot of time reviewing the existing HBAC rules trying to spot possible errors or misconfigurations. If you happen to not being able to access a certain service as you were used to in the past, please get in contact with the Sysadmin Team. All the possible ways to contact us are available at https://wiki.gnome.org/Sysadmin/Contact. What is missing still? Now that the Foundation membership information has been moved to LDAP I ll be looking at porting some of the existing membership scripts to it. What I managed to port already are welcome e-mails for new or existing members. (renewals) Next step will be generating a membership page from LDAP (to populate http://www.gnome.org/foundation/membership) and all the your-membership-is-going-to-lapse e-mails that were being sent till today. Other news /home/users mount on master.gnome.org You will notice that loggin in into master.gnome.org will result in your home directory being empty, don t worry, you did not lose any of your files but master.gnome.org is now currently hosting your home directories itself. As you may have been aware of adding files to the public_html directory on master resulted in them appearing on your people.gnome.org/~userid space. That was unfortunately expected as both master and webapps2 (the machine serving people.gnome.org s webspaces) were mounting the same GlusterFS share. We wanted to prevent that behaviour to happen as we wanted to know who has access to what resource and where. From today master s home directories will be there just as a temporary spot for your tarballs, just scp and use ftpadmin against them, that should be all you need from master. If you are interested in receiving or keeping using your people.gnome.org s webspace please mail <accounts AT gnome DOT org> stating so. Other news a shiny and new error 500 page has been deployed Thanks to Magdalen Berns (magpie) a new error 500 web page has been deployed on all the Apache istances we host. The page contains an iframe of status.gnome.org and will appear every single time the web server behind the service you are trying to reach will be unreachable for maintenance or other purposes. While I hope you won t see the page that often you can still enjoy it at https://static.gnome.org/error-500/500.html. Make sure to whitelist status.gnome.org on your browser as it currently loads it without https. (as the service is currently hosted on OpenShift which provides us with a *.rhcloud.com wildcard certificate, which differs from the CN the browser would expect it to be) Updates UPDATE on status.gnome.org s SSL certificate: the certificate has been provisioned and it should result in the 500 s page to be displayed correctly with no warnings from your browser. UPDATE from Adam Young on Kerberos ports being closed on many DC s firewalls:

The next version of upstream MIT Kerberos will have support for fetching a ticket via ports 443 and marshalling the request over HTTPS. We ll need to run a proxy on the server side, but we should be able to make it work: Read up here
http://adam.younglogic.com/2014/06/kerberos-firewalls

The upcoming version of Lintian will feature a new set of API that attempts to promote safer code. It is hardly a ground-breaking discovery , just a much needed feature. The primary reason for this API is that writing safe and correct code is simply too complicated that people get it wrong (including yours truly on occasion). The second reason is that I feel it is a waste having to repeat myself when reviewing patches for Lintian. Fortunately, the kind of issues this kind of mistake creates are usually minor information leaks, often with no chance of exploiting it remotely without the owner reviewing the output first[0]. Part of the complexity of writing correct code originates from the fact that Lintian must assume Debian packages to be hostile until otherwise proven[1]. Consider a simplified case where we want to read a file (e.g. the copyright file):

package Lintian::cpy_check;
use strict; use warnings; use autodie;
sub run  
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  # BAD: This is an example of doing it wrong
  open(my $fd, '<', $info->unpacked($filename));
  ...;
  close($fd);
  return;
 

This has two trivial vulnerabilities[2].

1. Any part of the path (usr,usr/share, ) can be asymlink to somewhere else like /
   1. Problem: Access to potentially any file on the system with the credentials of the user running Lintian. But even then, Lintian generally never write to those files and the user has to (usually manually) disclose the report before any information leak can be completed.
2. The target path can point to a non-file.
   1. Problem: Minor inconvenience by DoS of Lintian. Examples include a named pipe, where Lintian will get stuck until a signal kills it.

Of course, we can do this right[3]:

package Lintian::cpy_check;
use strict; use warnings; use autodie;
use Lintian::Util qw(is_ancestor_of);
sub run  
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  my $root = $info->unpacked
  my $path = $info->unpacked($filename);
  if ( -f $path and is_ancestor_of($root, $path))  
    open(my $fd, '<', $path);
    ...;
    close($fd);
   
  return;
 

Where is_ancestor_of is the only available utility to assist you currently. It hides away some 10-12 lines of code to resolve the two paths and correctly asserting that $path is (an ancestor of) $root. Prior to Lintian 2.5.12, you would have to do that ancestor check by hand in each and every check[4]. In the new version, the correct code would look something like this:

package Lintian::cpy_check;
use strict; use warnings; use autodie;
sub run  
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  my $path = $info->index_resolved_path($filename);
  if ($path and $path->is_open_ok)  
    my $fd = $path->open;
    ...;
    close($fd);
   
  return;
 

Now, you may wonder how that promotes safer code. At first glance, the checking code is not a lot simpler than the previous correct example. However, the new code has the advantage of being safer even if you forget the checks. The reasons are:

1. The return value is entirely based on the file index of the package (think: tar vtf data.tar.gz). At no point does it use the file system to resolve the path. Whether your malicious package trigger an undef warning based on the return value of index_resolved_index leaks nothing about the host machine.
   1. However, it does take safe symlinks into account and resolves them for you. If you ask for foo/bar and foo is a symlink to baz and baz/bar exists in the package, you will get baz/bar . If baz/bar happens to be a symlink, then it is resolved as well.
   2. Bonus: You are much more likely to trigger the undef warning during regular testing, since it also happens if the file is simply missing.
2. If you attempt to call $path->open without calling $path->is_open_ok first, Lintian can now validate the call for you and stop it on unsafe actions.

It also has the advantage of centralising the code for asserting safe access, so bugs in it only needs to be fixed in one place. Of course, it is still possible to write unsafe code. But at least, the new API is safer by default and (hopefully) more convenient to use. [0] Lintian.debian.org being the primary exception here. [1] This is in contrast to e.g. piuparts, which very much trusts its input packages by handing the package root access (albeit chroot ed, but still). [2] And also a bug. Not all binary packages have a copyright instead some while have a symlink to another package. [3] The code is hand-typed into the blog without prior testing (not even compile testing it). The code may be subject to typos, brown-paper-bag bugs etc. which are all disclaimed (of course). [4] Fun fact, our documented example for doing it correctly prior to implementing is_ancestor_of was in fact not correct. It used the root path in a regex (without quoting the root path) fortunately, it just broke lintian when your TMPDIR / LINTIAN_LAB contained certain regex meta-characters (which is pretty rare).

Review: Turn the Ship Around!, by L. David Marquet

Publisher:	Portfolio
Copyright:	2012
ISBN:	1-101-62369-1
Format:	Kindle
Pages:	272

Turn the Ship Around! (yes, complete with the irritating exclamation point in the title) is marketed to the business and management non-fiction market, which is clogged with books claiming to provide simple techniques to be a great manager or fix an organization. If you're like me, this is a huge turn-off. The presentation of the books is usually just shy of the click-bait pablum of self-help books. Many of the books are written by famous managers best known for doing horrible things to their staff (*cough* Jack Welch). It's hard to get away from the feeling that this entire class of books is an ocean of bromides covering a small core of outright evil. This book is not like that, and Marquet is not one of those managers. It can seem that way at times: it is presented in the format that caters to short attention span, with summaries of primary points at the end of every short chapter and occasionally annoying questions sprinkled throughout. I'm capable of generalizing information to my own life without being prompted by study questions, thanks. But that's just form. The core of this book is a surprisingly compelling story of Marquet's attempt to introduce a novel management approach into one of the most conservative and top-down of organizations: a US Navy nuclear submarine. I read this book as an individual employee, and someone who has no desire to ever be a manager. But I recently changed jobs and significantly disrupted my life because of a sequence of really horrible management decisions, so I have strong opinions about, at least, the type of management that's bad for employees. A colleague at my former employer recommended this book to me while talking about the management errors that were going on around us. It did such a good job of reinforcing my personal biases that I feel like I should mention that as a disclaimer. When one agrees with a book this thoroughly, one may not have sufficient distance from it to see the places where its arguments are flawed. At the start of the book, Marquet is assigned to take over as captain of a nuclear submarine that's struggling. It had a below-par performance rating, poor morale, and the worst re-enlistment rate in the fleet, and was not advancing officers and crew to higher ranks at anywhere near the level of other submarines. Marquet brought to this assignment some long-standing discomfort with the normal top-down decision-making processes in the Navy, and decided to try something entirely different: a program of radical empowerment, bottom-up decision-making, and pushing responsibility as far down the chain of command as possible. The result (as you might expect given that you can read a book about it) was one of the best-performing submarines in the fleet, with retention and promotion rates well above average. There's a lot in here about delegated decision-making and individual empowerment, but Turn the Ship Around! isn't only about that. Those are old (if often ignored) rules of thumb about how to manage properly. I think the most valuable part of this book is where Marquet talks frankly about his own thought patterns, his own mistakes, and the places where he had to change his behavior and attitude in order to make his strategy successful. It's one thing to say that individuals should be empowered; it's quite another to stop empowering them (which is still a top-down strategy) and start allowing them to be responsible. To extend trust and relinquish control, even though you're the one who will ultimately be held responsible for the performance of the people reporting to you. One of the problems with books like this is that they focus on how easy the techniques presented in the book are. Marquet does a more honest job in showing how difficult they are. His approach was not complex, but it was emotionally quite difficult, even though he was already biased in favor of it. The control, hierarchy, and authority parts of the book are the most memorable, but Marquet also talks about, and shows through specific examples from his command, some accompanying principles that are equally important. If everyone in an organization can make decisions, everyone has to understand the basis for making those decisions and understand the shared goals, which requires considerable communication and open discussion (particularly compared to a Navy ideal of an expert and taciturn captain). It requires giving people the space to be wrong, and requires empowering people to correct each other without blame. (There's a nice bit in here about the power of deliberate action, and while Marquet's presentation is more directly applicable to the sorts of physical actions taken in a submarine, I was immediately reminded of code review.) Marquet also has some interesting things to say about the power of, for lack of a better term, esprit de corps, how to create it, and the surprising power of acting like you have it until you actually develop it. As mentioned, this book is very closely in alignment with my own biases, so I'm not exactly an impartial reviewer. But I found it fascinating the degree to which the management situation I left was the exact opposite of the techniques presented in this book in nearly every respect. I found it quite inspiring during my transition period, and there are bits of it that I want to read again to keep some of the techniques and approaches fresh in my mind. There is a fair bit of self-help-style packaging and layout here, some of which I found irritating. If, like me, you don't like that style of book, you'll have to wade through a bit of it. I would have much preferred a more traditional narrative story from which I could draw my own conclusions. But it's more of a narrative than most books of this sort, and Marquet is humble enough to show his own thought processes, tensions, and mistakes, which adds a great deal to the presentation. I'm not sure how directly helpful this would be for a manager, since I've never been in that role, but it gave me a lot to think about when analyzing successful and unsuccessful work environments. Rating: 8 out of 10

So Monty and Sean have recently blogged about about the structures (1, 2) they think may work better for OpenStack. I like the thrust of their thinking but had some mumblings of my own to add. Firstly, I very much like the focus on social structure and needs what our users and deployers need from us. That seems entirely right. And I very much like the getting away from TC picking winners and losers. That was never an enjoyable thing when I was on the TC, and I don t think it has made OpenStack better. However, the thing that picking winners and losers did was that it allowed users to pick an API and depend on it. Because it was the X API for OpenStack . If we don t pick winners, then there is no way to say that something is the X API for OpenStack , and that means that there is no forcing function for consistency between different deployer clouds. And so this appears to be why Ring 0 is needed: we think our users want consistency in being able to deploy their application to Rackspace or HP Helion. They want vendor neutrality, and by giving up winners-and-losers we give up vendor neutrality for our users. Thats the only explanation I can come up with for needing a Ring 0 because its still winners and losers (e.g. picking an arbitrary project) keystone, grandfathering it in, if you will. If we really want to get out of the role of selecting projects, I think we need to avoid this. And we need to avoid it without losing vendor neutrality (or we need to give up the idea of vendor neutrality). One might say that we must pick winners for the very core just by its, but I don t think thats true. If the core is small, many people will still want vendor neutrality higher up the stack. If the core is large, then we ll have a larger % of APIs covered and stable granting vendor neutrality. So a core with fixed APIs will be under constant pressure to expand: not just from developers of projects, but from users that want API X to be fixed and guaranteed available and working a particular way at [most] OpenStack clouds. Ring 0 also fulfils a quality aspect we can check that it all works together well in a realistic timeframe with our existing tooling. We are essentially proposing to pick functionality that we guarantee to users; and an API for that which they have everywhere, and the matching implementation we ve tested. To pull from Monty s post: What does a basic end user need to get a compute resource that works and seems like a computer? (end user facet) What does Nova need to count on existing so that it can provide that. He then goes on to list a bunch of things, but most of them are not needed for that: We need Nova (its the only compute API in the project today). We don t need keystone (Nova can run in noauth mode and deployers could just have e.g. Apache auth on top). We don t need Neutron (Nova can do that itself). We don t need cinder (use local volumes). We need Glance. We don t need Designate. We don t need a tonne of stuff that Nova has in it (e.g. quotas) end users kicking off a simple machine have -very- basic needs. Consider the things that used to be in Nova: Deploying containers. Neutron. Cinder. Glance. Ironic. We ve been slowly decomposing Nova (yay!!!) and if we keep doing so we can imagine getting to a point where there truly is a tightly focused code base that just does one thing well. I worry that we won t get there unless we can ensure there is no pressure to be inside Nova to win . So there s a choice between a relatively large set of APIs that make the guaranteed available APIs be comprehensive, or a small set that that will give users what they need just at the beginning but might not be broadly available and we ll be depending on some unspecified process for the deployers to agree and consolidate around what ones they make available consistently. In sort one of the big reasons we were picking winners and losers in the TC was to consolidate effort around a single API not implementation (keystone is already on its second implementation). All the angst about defcore and compatibility testing is going to be multiplied when there is lots of ecosystem choice around APIs above Ring 0, and the only reason that won t be a problem for Ring 0 is that we ll still be picking winners. How might we do this? One way would be to keep picking winners at the API definition level but not the implementation level, and make the competition be able to replace something entirely if they implement the existing API [and win hearts and minds of deployers]. That would open the door to everything being flexible and its happened before with Keystone. Another way would be to not even have a Ring 0. Instead have a project/program that is aimed at delivering the reference API feature-set built out of a single, flat Big Tent and allow that project/program to make localised decisions about what components to use (or not). Testing that all those things work together is not much different than the current approach, but we d have separated out as a single cohesive entity the building of a product (Ring 0 is clearly a product) from the projects that might go into it. Projects that have unstable APIs would clearly be rejected by this team; projects with stable APIs would be considered etc. This team wouldn t be the TC : they too would be subject to the TC s rulings. We could even run multiple such teams as hinted at by Dean Troyer one of the email thread posts. Running with that I d then be suggesting

• IaaS product: selects components from the tent to make OpenStack/IaaS
• PaaS product: selects components from the tent to make OpenStack/PaaS
• CaaS product (containers)
• SaaS product (storage)
• NaaS product (networking but things like NFV, not the basic Neutron we love today). Things where the thing you get is useful in its own right, not just as plumbing for a VM.

So OpenStack/NaaS would have an API or set of APIs, and they d be responsible for considering maturity, feature set, and so on, but wouldn t own Neutron, or Neutron incubator or any other component they would be a *cross project* team, focused at the product layer, rather than the component layer, which nearly all of our folk end up locked into today. Lastly Sean has also pointed out that we have large N N^2 communication issues I think I m proposing to drive the scope of any one project down to a minimum, which gives us more N, but shrinks the size within any project, so folk don t burn out as easily, *and* so that it is easier to predict the impact of changes clear contracts and APIs help a huge amount there.

I m not used to talking about my day job but here s an exception. Over the past few years I worked in two startups (3 years each). It was nice to spend time in different areas: one job was mostly about research and development in a Linux cluster environment; the other one was about maintaining a highly-customized, Linux-based operating system, managing a small support team, and performing technological surveillance in IT security. In the meanwhile I ve reached a milestone: 10 years with Debian. I had been wondering for a few months whether I could try my luck going freelance, becoming a Debian consultant. I finally decided to go ahead and started in August! The idea is to lend a hand for various Debian-related things like systems administration, development/debugging, packaging/repository maintenance, or Debian Installer support, be it one-shot or on a regular basis. I didn t think about trainings/workshops at first but sharing knowledge is something I ve always liked, even if I didn t become a teacher. For those interested, details can be found on my website: https://mraw.org/. Of course this doesn t mean I m going to put an end to my volunteer activities within Debian, especially as a Debian Installer release manager. Quite the contrary in fact! See the August and September debian-boot@ archives, which have been busy months. :)

the perl 5.20 transition is over, debconf14 is over, so I should have more time for RC bugs? yes & no: I fixed some, but only in "our" (as in: pkg-perl) packages:

• #711418 src:libanyevent-dbi-perl: "libanyevent-dbi-perl: FTBFS: Failed test 'Using an unknown function results in error'"
  add patch for newer SQLite (pkg-perl)
• #756566 libxml-dt-perl: "libxml-dt-perl: Insecure use of temporary files (CVE-2014-5260)"
  upload new upstream release (pkg-perl)
• #759838 src:padre: "padre: FTBFS: Failed test 'no warnings'"
  fix 2/3 of the test failures in git, last one fixed by dod (pkg-perl)
• #759942 src:cpanminus: "cpanminus: FTBFS: Can't write to cpanm home '/sbuild-nonexistent/.cpanm': You should fix it with chown/chmod first."
  set HOME once more in debian/rules (pkg-perl)
• #759964 src:libhtml-formhandler-model-dbic-perl: "libhtml-formhandler-model-dbic-perl: FTBFS: dh_auto_test: make -j1 test returned exit code 2"
  upload new upstream release (pkg-perl)
• #761312 libdbix-class-resultset-recursiveupdate-perl: "libdbix-class-resultset-recursiveupdate-perl: missing dependency on liblist-moreutils-perl"
  add missing dependency (pkg-perl)
• #761313 libgeo-google-mapobject-perl: "libgeo-google-mapobject-perl: missing dependency on libjson-perl"
  add missing dependency (pkg-perl)
• #761315 libfile-read-perl: "libfile-read-perl: missing dependency on libfile-slurp-perl"
  add missing dependency (pkg-perl)
• #761319 libpod-wordlist-hanekomu-perl: "libpod-wordlist-hanekomu-perl: missing dependency on libtest-spelling-perl"
  add missing dependency (pkg-perl)
• #761526 src:liblocale-maketext-gettext-perl: "liblocale-maketext-gettext-perl: FTBFS: Tests failures"
  skip test which needs internet and a writable $HOME (pkg-perl)
• #761558 src:libposix-strptime-perl: "libposix-strptime-perl: FTBFS: Tests failures"
  skip test which needs internet and a writable $HOME (pkg-perl)

When we setup Freexian s offer to bring together funding from multiple companies in order to sponsor the work of multiple developers on Debian LTS, one of the rules that I imposed is that all paid contributors must provide a public monthly report of their paid work. While the LTS project officially started in June, the first month where contributors were actually paid has been July. Freexian sponsored Thorsten Alteholz and Holger Levsen for 10.5 hours each in July and for 16.5 hours each in August. Here are their reports:

• Thorsten Alteholz: July / August
• Holger Levsen: July / August

It s worth noting that Freexian sponsored Holger s work to fix the security tracker to support squeeze-lts. It s my belief that using the money of our sponsors to make it easier for everybody to contribute to Debian LTS is money well spent. As evidenced by the progress bar on Freexian s offer page, we have not yet reached our minimal goal of funding the equivalent of a half-time position. And it shows in the results, the dla-needed.txt still shows around 30 open issues. This is slightly better than the state two months ago but we can improve a lot on the average time to push out a security update To have an idea of the relative importance of the contributions of the paid developers, I counted the number of uploads made by Thorsten and Holger since July: of 40 updates, they took care of 19 of them, so about the half. I also looked at the other contributors: Rapha l Geissert stands out with 9 updates (I believe that he is contracted by lectricit de France for doing this) and most of the other contributors look like regular Debian maintainers taking care of their own packages (Paul Gevers with cacti, Christoph Berg with postgresql, Peter Palfrader with tor, Didier Raboud with cups, Kurt Roeckx with openssl, Balint Reczey with wireshark) except Matt Palmer and Luciano Bello who (likely) are benevolent members of the LTS team. There are multiple things to learn here:

1. Paid contributors already handle almost 70% of the updates. Counting only on volunteers would not have worked.
2. Quite a few companies that promised help (and got mentioned in the press release) have not delivered the promised help yet (neither through Freexian nor directly).

Last but not least, this project wouldn t exist without the support of multiple companies and organizations. Many thanks to them:

• Gold sponsors:
  • The Positive Internet
• Silver sponsors:
  • AD&D David Ayers IntarS Austria
  • Blablacar
  • Domeneshop AS
  • Evolix
  • Universit Lille 3
• Bronze sponsors:
  • Freeside Internet Service
  • MyTux
  • Offensive Security
  • Seznam.cz, a.s.

Hopefully this list will expand over time! Any help to reach out to new companies and organizations is more than welcome.

One comment Liked this article? Click here. My blog is Flattr-enabled.

Debian LTS - feedback about the feedback from my LTS talk at DebConf14 So, I'm more or less back from dc14 and today, five days later, I think I might have mostly overcome jetlag. Probably... So, at DebConf14 I gave a talk about LTS and while I'm sorry that I was that tired, I'm more or less happy how the talk went. Thankfully at least I was calm and relaxed... There are a couple of things I learned from the talk: a.) LTS has been really really perceived well b.) it fits a demand c.) people already take it for granted (eg plan for Wheezy LTS) d.) people expect the same non-intrusive changes as currently done for security updates. To explain the last point: when I explained the - so far - rather theoretical problem that ''squeeze-lts'' has no gatekeeper mechanisms whatsoever (eg no ''proposed-updates'', no NEW queue..) the reaction in the audience was basically "something like this should exist, else how can we deploy this in large scale / on important setup?!". Also currently there is no, well-documented, easily to be found policy for what kind of updates are acceptable. I said that we basically follow the same rules as there are for debian-security updates, but this should really be documented properly. This doesn't seem very hard to fix, just like many things it "just" needs someone to do the work. IOW: we explain how to use LTS, we explain how to contribute to LTS (through uploads or financially) but we lack a simple explaination what LTS is and what kind of updates to expect. It's kinda self evident, but only kinda. So since giving the talk I changed one thing in my personal usage of LTS: I don't use my personal LTS repo anymore, where I made sure only good packages got in. This is for two reasons: a.) I had too add new packages too often and b.) if it really is a problem that LTS has no gatekeeping mechanism (which I'm not sure anymore it is, after all, the updates are prepared by reasonable people with a common goal...) then I want to suffer this first hand, so I can build solutions which benefit everyone, not just me. That personal LTS repo only helped me. On the technical side I prepared five DLAs, for lzo2, libwpd, squid3, lua5.1 and bind9. Not much to see here, they all were very smooth. I still enjoyed the challenge of digging in unknown sourcecode, as described in my previous post. Then more interestingly, and with the help of Raphael Geissert and Salvatore Bonaccorso I fixed the security-tracker to also know about oldstable, after waiting for more than 8 weeks to someone else doing it. I'm very glad that this is done now, as without it was really tedious to check which issues were applying to oldstable. Oh, and another afterthought from giving the talk: currently at least parts of the security-tracker codebase assume that there won't ever be support for oldoldstable, but once jessie has been released this won't be true anymore. Then we will support stable, oldstable and oldoldstable. And oldstable will be wheezy, not squeeze. We have something like 6 months to fix this, hopefully we won't have much more time... Oh, and surely there are other places than just the security-tracker which will need to be taught about this.

Debian LTS - feedback about the feedback from my LTS talk at DebConf14 So, I'm more or less back from dc14 and today, five days later, I think I might have mostly overcome jetlag. Probably... So, at DebConf14 I gave a talk about LTS and while I'm sorry that I was that tired, I'm more or less happy how the talk went. Thankfully at least I was calm and relaxed. There are a couple of things I learned from the talk: a.) LTS has been really really perceived well b.) it fits a demand c.) people already take it for granted (eg plan for Wheezy LTS) d.) people expect the same non-intrusive changes as currently done for security updates. To explain the last point: when I explained the - so far - rather theoretical problem that ''squeeze-lts'' has no gatekeeper mechanisms whatsoever (eg no ''proposed-updates'', no NEW queue..) the reaction in the audience was basically "something like this should exist, else how can we deploy this in large scale / on important setup?!". Also currently there is no, well-documented, easily to be found policy for what kind of updates are acceptable. I said that we basically follow the same rules as there are for debian-security updates, but this should really be documented properly. This doesn't seem very hard to fix, just like many things it "just" needs someone to do the work. IOW: we explain how to use LTS, we explain how to contribute to LTS (through uploads or financially) but we lack a simple explaination what LTS is and what kind of updates to expect. It's kinda self evident, but only kinda. So since giving the talk I changed one thing in my personal usage of LTS: I don't use my personal LTS repo anymore, where I made sure only good packages got in. This is for two reasons: a.) I had too add new packages too often and b.) if it really is a problem that LTS has no gatekeeping mechanism (which I'm not sure anymore it is, after all, the updates are prepared by reasonable people with a common goal...) then I want to suffer this first hand, so I can build solutions which benefit everyone, not just me. That personal LTS repo only helped me. On the technical side I prepared five DLAs, for lzo2, libwpd, squid3, lua5.1 and bind9. Not much to see here, they all were very smooth. I still enjoyed the challenge of digging in unknown sourcecode, as described in my previous post. Then more interestingly, and with the help of Raphael Geissert and Salvatore Bonaccorso I fixed the security-tracker to also know about oldstable, after waiting for more than 8 weeks to someone else doing it. I'm very glad that this is done now, as without it was really tedious to check which issues were applying to oldstable. Oh, and another afterthought from giving the talk: currently at least parts of the security-tracker codebase assume that there won't ever be support for oldoldstable, but once jessie has been released this won't be true anymore. Then we will support stable, oldstable and oldoldstable. And oldstable will be wheezy, not squeeze. We have something like 6 months to fix this, hopefully we won't have much more time... Oh, and surely there are other places than just the security-tracker which will need to be taught about this.

Another on-my-the-journey-back blog post; this time from the Frankfurt Airport Train Station my flight was delayed (if I knew that I could have watched the remaining Lightning Talks), and so was my train, but despite 5min of running through the Airport just not enough. And now that the free 30 Minutes of Railway Station Internet are used up, I have nothing else to do but blog... Last week I was attending ICFP 2014 in Gothenburg, followed by the Haskell Symposium and the Haskell Implementors Workshop. The justification to attend was the paper on Safe Coercions (joint work with Richard Eisenberg, Simon Peyton Jones and Stephanie Weirich), although Richard got to hold the talk, and did so quite well. So I got to leisurely attend the talks, while fighting the jet-lag that I brought from Portland. There were as expected quite a few interesting talks. Among them the first keynote, Kathleen Fisher on the need for formal methods in cars and toy-quadcopters and unmanned battle helicopters, which made me conclude that my Isabelle skills might eventually become relevant in practical applications. And did you know that if someone gains access to your car s electronics, they can make the seat belt pull you back hard? Stefanie Weirich s keynote (and the subsequent related talks by Jan Stolarek and Richard Eisenberg) on what a dependently typed Haskell would look like and what we could use it for was mouth-watering. I am a bit worried that Haskell will be become a bit obscure for newcomers and people that simply don t want to think about types too much, on the other hand it seems that Haskell as we know it will always stay there, just as a subset of the language. Similarly interesting were refinement types for Haskell (talks by Niki Vazou and by Eric Seidel), in the form of LiquidTypes, something that I have not paid attention to yet. It seems to be a good way for more high assurance in Haskell code. Finally, the Haskell Implementors Workshop had a truckload of exciting developments in and around Haskell: More on GHCJS, Partial type signatures, interactive type-driven development like we know it from Agda, the new Haskell module system and amazing user-defined error messages the latter unfortunately only in Helium, at least for now. But it s not the case that I only sat and listened. During the Haskell Implementors Workshop I held a talk Contributing to GHC with a live demo of me fixing a (tiny) bug in GHC, with the aim of getting more people to hack on GHC (slides, video). The main message here is that it is not that big of deal. And despite me not actually saying much interesting in the talk, I got good feedback afterwards. So if it now actually motivates someone to contribute to GHC, I m even more happier. And then there is of course the Hallway Track. I discussed the issues with fusing a left fold (unfortunately, without a great solution). In order to tackle this problem more systematically, John Wiegley and I created the beginning of a List Fusion Lab , i.e. a bunch of list benchmark and the possibility to compare various implementations (e.g. with different RULES) and various compilers. With that we can hopefully better assess the effect of a change to the list functions. PS: The next train is now also delayed, so I ll likely miss my tram and arrive home even later... PPS: I really have to update my 10 year old picture on my homepage (or redesign it completely). Quite a few people knew my name, but expected someone with shoulder-long hair... PPPS: Haskell is really becoming mainstream: I just talked to a randomly chosen person (the boy sitting next to me in the train), and he is a Haskell enthusiast, building a structured editor for Haskell together with his brother. And all that as a 12th-grader...

Part 1 - Debconf 2014 This year I went to my first Debconf, which took place in Portland, OR during the last week of August 2014. All in all I have to rate my experience as very enlightening and in the end quite fun. First of all, it was a little daunting to go to a conference in 1 - A city I d never been to before; 2 - A conference with 300+ people, only 3 of which I knew and even then I only knew them virtually. Not to mention I was in the presence of some extremely brilliant and known contirbutors in the Debian community which was somewhat intimidating. Just to give you an idea, Linus Torvalds showed up for a Q&A session last Friday morning! Jealous? Actually I missed that too. It was kind of a last minute thing, booked for coincidentally the exact time I d be flying out of Portland. I found out about it much too late. But luckily for me and maybe you, the session was filmed and can be seen here. Isn t that a treat? Point made, there are lots of really talented people there, both techies and non-techies. It s easy to feel you re out of league, at least I did. But I d highly encourage you to ignore such feelings if you re ever in the same situation. Debian has been built on for a long time now, but although a lot has been done, a lot still needs to be done. The Debian community is very welcoming of new contributors and users, regardless of the level of expertise. So far I haven t been snubbed by anyone. To the contrary, all my interactions with the Debian community members has been extremely positive. So go ahead and attend the meetings and presentations, even if you think it s not your area of expertise. Debconf was organized (or at least this one was) as a series of talks, meet ups and ad hoc sessions, some of which occured simultaneously. The sessions were all about different components of the Debian universe, from presenting new features to overviews of accomplishments to discussing issues and how to fix them. A schedule with the location and description of each session was posted on the Debconf wiki. Sometimes none of the sessions at a certain time was on a topic I knew very much about. But I d sit in anyways. There s no rule to attending the sessions, no minimum qualifications required. You ll likely learn something new and you just might find out there is something you can do to contribute. There are also hackathons that are quite the thing or so I heard. Or you could walk about and meet new people, do some networking. I have to say networking was the highlight of the Debconf for me. Remember I said I knew about 3 people who were at the conference? Well, I had actually just corresponded with those people. I didn t really know them. So on my first day I spent quite some time shyly peeking at people s name tags, trying to recognize someone I had met over email or IRC. But with 300 or so people at the conference, I was unsuccessful. So I finally gave up on that strategy and walked up to a random person, stuck out my hand and said, Hi. My name is Juliana. This is my first Debconf. What s your name and what do you do for Debian? This may not be according to protocol, but it worked for me. I got to meet lots of people that way, met some Debian contributos from my home country (Brazil), some from my current city (NYC), and yet others that had similar interests as I do who I might work with in the near future. For example, I love Machine Learning, I m currently beginning my graduate studies on that track. Several Debian contributors offered to introduce me to a well known Machine Learning researcher and Debian contributor who is in NYC. Others had tried out JSCommunicator and had lots of suggestions for new features and fixes, or wanted to know more about the project and WebRTC in general. Also, not everyone there is a super experienced Debian contributor or user. There are a lot of newbies like me. I got to do a quick 20-min presentation and demo of the work I had done on JSCommunicator during GSoC 2014. Oh my goodness that was nerve-wracking, but not half as painful as I expected. My mentor (Daniel Pocock) wisely suggested that when confronted with a question I didn t know how to answer, to redirect the question to to the audience. Chances are, there is someone there that knows the answer. If not, it will at least spark a good discussion. When meeting new people at Debian, a question almost everyone asked is How did you start working with/for Debian? . So I thought it would be a good topic to post about. Part 2 - How I Became a Debian Contributor Sometime in late October of 2013 (I think) I received an email from one of my professors at UNIRIO forwarding a description of the Outreach Program for Women. OPW is a program organized by the GNOME which endeavors to get more women involved in FOSS. OPW is similar to Google Summer of Code; you work remotely from home, guided by an assigned mentor. Debian was one of the 8 participating organizations that year. There was a list of project proposals which I perused, a few of them caught my eye and these projects were all Debian. I d already been a fan of FOSS before. I had used the Ubuntu and Debian OS, I d migrated to GIMP from Photoshop and to Open Office from Microsoft Office, for example. I d strongly advocated the use of some of my prefered open source apps and programs to my friends and family. But I hadn t ever contributed to a FOSS project. There s no time like the present, so I reached out the the mentor responsible for one of the projects I was interested in, Daniel Pocock. Daniel guided me through making a small contribution to a FOSS project, which serves as a token demonstration of my abilities and is part of the application process. I added a small feature to JMXetric and suggested a fix for an issue in the xTuple project. Actually, I had forgotten about this. Recently I made another contribution to xTuple, it s funny to see things come full circle. I also had to write a profile-ish description of my experience and how I intended on contributing during OPW on the Debian wiki, if you d like you can check it out here. I wouldn t follow my example to a T, because in the end I didn t make the OPW selection. Actually, I take that back. The fact I wasn t chosen for OPW that year doesn t mean I was incompetent or incapable of making a valuable contribution. OPW and GSoC do not have unlimited resources; they can t include everyone they d like to. They receive thousands of proposals from very talented engineers and not everyone can participate at a given moment. But even though I wasn t selected, like I said, I could still pitch in. It s good to keep in mind that people usually aren t paid to contribute to FOSS. It s usually volunteer based, which I think is one of the beauties of the FOSS community and in my opinion one of the causes of it s success and great quality. People contribute because they want to, not because they have to. I will say I was a little disappointed at not being chosen. But after being reassured that this rejection wasn t due to any lack on my part, I decided to continue contributing to the Debian project I d applied to. I was begining the final semester of my undergraduate studies which included writing a thesis. To be able to focus on my thesis and graduate on time, I d stopped working temporarily and was studying full time. But I didn t want to lose practice and contributing to a FOSS project is a great way to stay in coding shape while doing something useful. So continue contributing I did. It paid off. I gained experience, added value to a FOSS project and I think my previous contributions added weight to the application I later made for GSoC 2014. I passed this time. To be honest, I really wasn t counting on it. Actually, I was certain I wouldn t pass for some reason - insecure much? But with GSoC I wasn t too anxious about it as I was with the OPW application because by then, I was already hooked . I d learned about all the benefits of becoming a FOSS contributor and I wasn t stopping anytime soon. I had every intention of still working on my FOSS project with or without GSoC. GSoC 2014 ended a week ago (August 18th 2014). There s a list of things I still want to do with JSCommunicator and you can be sure I ll keep working on them. P.S. This is not to say that programs like OPW and GSoC aren t amazing programs. Try it out if you can, it s really a great experience.

Today we have a little diversion to talk about the National Health Service. The NHS is the publicly funded healthcare system in the UK.

Actually there are four such services in the UK, only one of which has this name:

• The national health service (England)
• Health and Social Care in Northern Ireland.
• NHS Scotland.
• NHS Wales.

In theory this doesn't matter, if you're in the UK and you break your leg you get carried to a hospital and you get treated. There are differences in policies because different rules apply, but the basic stuff "free health care" applies to all locations. (Differences? In Scotland you get eye-tests for free, in England you pay.)

My wife works as an accident & emergency doctor, and has recently changed jobs. Hearing her talk about her work is fascinating. The hospitals she's worked in (Dundee, Perth, Kirkcaldy, Edinburgh, Livingstone) are interesting places. During the week things are usually reasonably quiet, and during the weekend things get significantly more busy. (This might mean there are 20 doctors to hand, versus three at quieter times.) Weekends are busy largely because people fall down hills, get drunk and fight, and are at home rather than at work - where 90% of accidents occur. Of course even a "quiet" week can be busy, because folk will have heart-attacks round the clock, and somebody somewhere will always be playing with a power tool, a ladder, or both! So what was the point of this post? Well she's recently transferred to working for a childrens hospital (still in A&E) and the patiences are so very different. I expected the injuries/patients she'd see to differ. Few 10 year olds will arrive drunk (though it does happen), and few adults fall out of trees, or eat washing machine detergent, but talking to her about her day when she returns home is fascinating how many things are completely different from how I expected. Adults come to hospital mostly because they're sick, injured, or drunk. Children come to hospital mostly because their parents are paranoid. A child has a rash? Doctors are closed? Lets go to the emergency ward! A child has fallen out of a tree and has a bruise, a lump, or complains of pain? Doctors are closed? Lets go to the emergency ward! I've not kept statistics, though I wish I could, but it seems that she can go 3-5 days between seeing an actually injured or chronicly-sick child. It's the first-time-parents who bring kids in when they don't need to. Understandable, completely understandable, but at the same time I'm sure it is more than a little frustrating for all involved. Finally one thing I've learned, which seems completely stupid, is the NHS-Scotland approach to recruitment. You apply for a role, such as "A&E doctor" and after an interview, etc, you get told "You've been accepted - you will now work in Glasgow". In short you apply for a post, and then get told where it will be based afterward. There's no ability to say "I'd like to be a Doctor in city X - where I live", you apply, and get told where it is post-acceptance. If it is 100+ miles away you either choose to commute, or decline and go through the process again. This has lead to Kirsi working in hospitals with a radius of about 100km from the city we live in, and has meant she's had to turn down several posts. And that is all I have to say about the NHS for the moment, except for the implicit pity for people who have to pay (inflated and life-changing) prices for things in other countries.

Recently I was doing some work on the alioth infrastructure like fixing things or cleaning up things. One of the more visible things I done was the switch from gitweb to cgit. cgit is a lot of faster and looks better than gitweb. The list of repositories is generated every hour. The move also has the nice effect that user repositories are available via the cgit index again. I don t plan to disable the old gitweb, but I created a bunch of redirect rules that - hopefully - redirect most use cases of gitweb to the equivalent cgit url. If I broke something, please tell me, if I missed a common use case, please tell me. You can usually reach me on #alioth@oftc or via mail (formorer@d.o) People also asked me to upload my cgit package to Debian, the package is now waiting in NEW. Thanks to Nicolas Dandrimont (olasd) we also have a patch included that generates proper HTTP returncodes if repos doesn t exist.

Modern web applications are much more complicated than the simple Perl CGI scripts or PHP pages of the past. They usually start with a framework and include lots of external components both on the front-end and on the back-end. Here's an example from the Node.js back-end of a real application:

$ npm list   wc -l
256

What if one of these 256 external components has a security vulnerability? How would you know and what would you do if of your direct dependencies had a hard-coded dependency on the vulnerable version? It's a real problem and of course one way to avoid this is to write everything yourself. But that's neither realistic nor desirable. However, it's not a new problem. It was solved years ago by Linux distributions for C and C++ applications. For some reason though, this learning has not propagated to the web where the standard approach seems to be to "statically link everything". What if we could build on the work done by Debian maintainers and the security team?

Case study - the Libravatar project As a way of discussing a different approach to the problem of dependency management in web applications, let me describe the decisions made by the Libravatar project.

Description Libravatar is a federated and free software alternative to the Gravatar profile photo hosting site. From a developer point of view, it's a fairly simple stack:

• Apache
• Postgres
• Django
• Python
• Gearman

The service is split between the master node, where you create an account and upload your avatar, and a few mirrors, which serve the photos to third-party sites. Like with Gravatar, sites wanting to display images don't have to worry about a complicated protocol. In a nutshell, all that a site needs to do is hash the user's email and add that hash to a base URL. Where the federation kicks in is that every email domain is able to specify a different base URL via an SRV record in DNS. For example, francois@debian.org hashes to 7cc352a2907216992f0f16d2af50b070 and so the full URL is:

http://cdn.libravatar.org/avatar/7cc352a2907216992f0f16d2af50b070

whereas francois@fmarier.org hashes to 0110e86fdb31486c22dd381326d99de9 and the full URL is:

http://fmarier.org/avatar/0110e86fdb31486c22dd381326d99de9

due to the presence of an SRV record on fmarier.org.

Ground rules The main rules that the project follows is to:

1. only use Python libraries that are in Debian
2. use the versions present in the latest stable release (including backports)

Deployment using packages In addition to these rules around dependencies, we decided to treat the application as if it were going to be uploaded to Debian:

• It includes an "upstream" Makefile which minifies CSS and JavaScript, gzips them, and compiles PO files (i.e. a "build" step).
• The Makefile includes a test target which runs the unit tests and some lint checks (pylint, pyflakes and pep8).
• Debian packages are produced to encode the dependencies in the standard way as well as to run various setup commands in maintainer scripts and install cron jobs.
• The project runs its own package repository using reprepro to easily distribute these custom packages.
• In order to update the repository and the packages installed on servers that we control, we use fabric, which is basically a fancy way to run commands over ssh.
• Mirrors can simply add our repository to their apt sources.list and upgrade Libravatar packages at the same time as their system packages.

Results Overall, this approach has been quite successful and Libravatar has been a very low-maintenance service to run. The ground rules have however limited our choice of libraries. For example, to talk to our queuing system, we had to use the raw Python bindings to the C Gearman library instead of being able to use a nice pythonic library which wasn't in Debian squeeze at the time. There is of course always the possibility of packaging a missing library for Debian and maintaining a backport of it until the next Debian release. This wouldn't be a lot of work considering the fact that responsible bundling of a library would normally force you to follow its releases closely and keep any dependencies up to date, so you may as well share the result of that effort. But in the end, it turns out that there is a lot of Python stuff already in Debian and we haven't had to package anything new yet. Another thing that was somewhat scary, due to the number of packages that were going to get bumped to a new major version, was the upgrade from squeeze to wheezy. It turned out however that it was surprisingly easy to upgrade to wheezy's version of Django, Apache and Postgres. It may be a problem next time, but all that means is that you have to set a day aside every 2 years to bring everything up to date.

Problems The main problem we ran into is that we optimized for sysadmins and unfortunately made it harder for new developers to setup their environment. That's not very good from the point of view of welcoming new contributors as there is quite a bit of friction in preparing and testing your first patch. That's why we're looking at encoding our setup instructions into a Vagrant script so that new contributors can get started quickly. Another problem we faced is that because we use the Debian version of jQuery and minify our own JavaScript files in the build step of the Makefile, we were affected by the removal from that package of the minified version of jQuery. In our setup, there is no way to minify JavaScript files that are provided by other packages and so the only way to fix this would be to fork the package in our repository or (preferably) to work with the Debian maintainer and get it fixed globally in Debian. One thing worth noting is that while the Django project is very good at issuing backwards-compatible fixes for security issues, sometimes there is no way around disabling broken features. In practice, this means that we cannot run unattended-upgrades on our main server in case something breaks. Instead, we make use of apticron to automatically receive email reminders for any outstanding package updates. On that topic, it can occasionally take a while for security updates to be released in Debian, but this usually falls into one of two cases:

1. You either notice because you're already tracking releases pretty well and therefore could help Debian with backporting of fixes and/or testing;
2. or you don't notice because it has slipped through the cracks or there simply are too many potential things to keep track of, in which case the fact that it eventually gets fixed without your intervention is a huge improvement.

Finally, relying too much on Debian packaging does prevent Fedora users (a project that also makes use of Libravatar) from easily contributing mirrors. Though if we had a concrete offer, we would certainly look into creating the appropriate RPMs.

Is it realistic? It turns out that I'm not the only one who thought about this approach, which has been named "debops". The same day that my talk was announced on the DebConf website, someone emailed me saying that he had instituted the exact same rules at his company, which operates a large Django-based web application in the US and Russia. It was pretty impressive to read about a real business coming to the same conclusions and using the same approach (i.e. system libraries, deployment packages) as Libravatar. Regardless of this though, I think there is a class of applications that are particularly well-suited for the approach we've just described. If a web application is not your full-time job and you want to minimize the amount of work required to keep it running, then it's a good investment to restrict your options and leverage the work of the Debian community to simplify your maintenance burden. The second criterion I would look at is framework maturity. Given the 2-3 year release cycle of stable distributions, this approach is more likely to work with a mature framework like Django. After all, you probably wouldn't compile Apache from source, but until recently building Node.js from source was the preferred option as it was changing so quickly. While it goes against conventional wisdom, relying on system libraries is a sustainable approach you should at least consider in your next project. After all, there is a real cost in bundling and keeping up with external dependencies. This blog post is based on a talk I gave at DebConf 14: slides, video.

Yesterday I carried out the upgrade of a Debian host from Squeeze to Wheezy for a friend. I like doing odd-jobs like this as they're generally painless, and when there are problems it is a fun learning experience. I accidentally forgot to check on the status of the MySQL server on that particular host, which was a little embarassing, but later put together a reasonably thorough serverspec recipe to describe how the machine should be setup, which will avoid that problem in the future - Introduction/tutorial here. The more I use serverspec the more I like it. My own personal servers have good rules now:

shelob ~/Repos/git.steve.org.uk/server/testing $ make
..
Finished in 1 minute 6.53 seconds
362 examples, 0 failures

Slow, but comprehensive. In other news I've now migrated every single one of my personal mercurial repositories over to git. I didn't have a particular reason for doing that, but I've started using git more and more for collaboration with others and using two systems felt like an annoyance. That means I no longer have to host two different kinds of repositories, and I can use the excellent gitbucket software on my git repository host. Needless to say I wrote a policy for this host too:

#
#  The host should be wheezy.
#
describe command("lsb_release -d") do
  its(:stdout)   should match /wheezy/  
end
#
# Our gitbucket instance should be running, under runit.
#
describe supervise('gitbucket') do
  its(:status)   should eq 'run'  
end
#
# nginx will proxy to our back-end
#
describe service('nginx') do
  it   should be_enabled    
  it   should be_running    
end
describe port(80) do
  it   should be_listening  
end
#
#  Host should resolve
#
describe host("git.steve.org.uk" ) do
  it   should be_resolvable.by('dns')  
end

Simple stuff, but being able to trigger all these kind of tests, on all my hosts, with one command, is very reassuring.

I love to see there is a lot of crypto discussions going on at DebConf. Maybe I'm skewed by my role as keyring-maint, but I have been involved in more than one discussion every day on what do/should signatures mean, on best key handling practices, on some ideas to make key maintenance better, on how the OpenPGPv4 format lays out a key and its components on disk, all that. I enjoy some of those discussions pose questions that leave me thinking, as I am quite far from having all answers. Discussions should be had face to face, but some start online and deserve to be answered online (and also pose opportunity to become documentation). Simon Josefsson blogs about The case for short OpenPGP key validity periods. This will be an important issue to tackle, as we will soon require keys in the Debian keyring to have a set expiration date (surprise surprise!) and I agree with Simon, setting an expiration date far in the future means very little. There is a caveat with using, as he suggests, very short expiry periods: We have a human factor sitting in the middle. Keyring updates in Debian are done approximately once a month, and I do not see the period shortening. That means, only once a month we (currently Jonathan McDowell and myself, and we expect to add Daniel Kahn Gillmor soon) take the full changeset and compile a new keyring that replaces the active one in Debian. This means that if you have, as Simon suggests, a 100-day validity key, you have to remember to update it at least every 70 days, or you might be locked out during the days it takes us to process it. I set my expiration period to two years, although I might shorten it to only one. I expect to add checks+notifications before we enable this requirement project-wide (so that Debian servers will mail you when your key is close to expiry); I think that mail can be sent at approximately [expiry date - 90 days] to give you time both to you and to us to act. Probably the optimal expiration periods under such conditions would be between 180 and 365 days. But, yes, this is by far not yet a ruling, but a point in the discussion. We still have some days of DebConf, and I'll enjoy revising this point. And Simon, even if we correct some bits for these details, I'd like to have your permission to use this fine blog post as part of our documentation! (And on completely unrelated news: Congratulations to our dear and very much missed friend Bubulle for completely losing his sanity and running for 28 hours and a half straight! He briefly describes this adventure when it was about to start, and we all want him to tell us how it was. Mr. Running French Guy, you are amazing!)

---

---

Today is the first time I've taken an interstate train trip in something like 15 years. A few things about the trip were pleasantly surprising. Most of these will come as no surprise:

1. Less time wasted in security theater at the station prior to departure.
2. On-time departure
3. More comfortable seats than a plane or bus.
4. Quiet.
5. Permissive free wifi

Wifi was the biggest surprise. Not that it existed, since we're living in the future and wifi is expected everywhere. It's IPv4 only and stuck behind a NAT, which isn't a big surprise, but it is reasonably open. There isn't any port filtering of non-web TCP ports, and even non-TCP protocols are allowed out. Even my aiccu IPv6 tunnel worked fine from the train, although I did experience some weird behavior with it. I haven't used aiccu much in quite a while, since I have a native IPv6 connection at home, but it can be convenient while travelling. I'm still trying to figure out happened today, though. The first symptoms were that, although I could ping IPv6 hosts, I could not actually log in via IMAP or ssh. Tcpdump showed all the standard symptoms of a PMTU blackhole. Small packets flow fine, large ones are dropped. The interface MTU is set to 1280, which is the minimum MTU for IPv6 and any path on the internet is expected to handle packets of at least that size. Experimentation via ping6 reveals that the largest payload size I can successfully exchange with a peer is 820 bytes. Add 8 bytes for the ICMPv6 header for 828 bytes of payload, plus 40 bytes for the IPv6 header gives an 868 byte packet, which is well under what should be the MTU for this path. I've worked around this problem with an ip6tables rule to rewrite the MSS on outgoing SYN packets to 760 bytes, which should leave 40 for the IPv6 header and 20 for any extension headers:

sudo ip6tables -t mangle -A OUTPUT -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 760

It is working well and will allow me to publish this from the train, which I'd otherwise have been unable to do. But... weird.