Andrea Veri: Three years and counting
It s been a while since my last what s been happening behind the scenes e-mail so I m here to report on what has been happening within the GNOME Infrastructure, its future plans and my personal sensations about a challenge that started around three (3) years ago when Sriram Ramkrishna and Jeff Schroeder proposed my name as a possible candidate for coordinating the team that runs the systems behind the GNOME Project. All this followed by the official hiring achieved by Karen Sandler back in February 2013.
The GNOME Infrastructure has finally reached stability both in terms of reliability and uptime, we didn t have any service disruption this and the past year and services have been running smoothly as they were expected to in a project like the one we are managing.
As many of you know service disruptions and a total lack of maintenance were very common before I joined back in 2013, I m so glad the situation has dramatically changed and developers, users, passionates are now able to reach our websites, code repositories, build machines without experiencing slowness, downtimes orunreachability. Additionally all these groups of people now have a reference point they can contact in case they need help when coping with the infrastructure they daily use. The ticketing system allows users to get in touch with the members of the Sysadmin Team and receive support right away within a very short period of time (Also thanks to Pagerduty, service the Foundation is kindly sponsoring) Before moving ahead to the future plans I d like to provide you a summary of what has been done during these roughly three years so you can get an idea of why I define the changes that happened to the infrastructure a complete revamp:
- Recycled several ancient machines migrating services off of them while consolidating them by placing all their configuration on our central configuration management platform ran by Puppet. This includes a grand total of 7 machines that were replaced by new hardware and extended warranties the Foundation kindly sponsored.
- We strenghten our websites security by introducing SSL certificates everywhere and recently replacing them with SHA2 certificates.
- We introduced several services such as Owncloud, the Commits Bot, the Pastebin, the Etherpad, Jabber, the GNOME Github mirror.
- We restructured the way we backup our machines also thanks to the Fedora Project sponsoring the disk space on their backup facility. The way we were used to handle backups drastically changed from early years where a magnetic tape facility was in charge of all the burden of archiving our data to today where a NetApp is used together with rdiff-backup.
- We upgraded Bugzilla to the latest release, a huge thanks goes to Krzesimir Nowak who kindly helped us building the migration tools.
- We introduced the GNOME Apprentice program open-sourcing our internal Puppet repository and cleansing it (shallow clones FTW!) from any sensitive information which now lives on a different repository with restricted access.
- We retired Mango and our OpenLDAP instance in favor of FreeIPA which allows users to modify their account information on their own without waiting for the Accounts Team to process the change.
- We documented how our internal tools are customized to play together making it easy for future Sysadmin Team members to learn how the infrastructure works and supersede existing members in case they aren t able to keep up their position anymore.
- We started providing hosting to the GIMP and GTK projects which now completely rely on the GNOME Infrastructure. (DNS, email, websites and other services infrastructure hosting)
- We started providing hosting not only to the GIMP and GTK projects but to localized communities as well such as GNOME Hispano and GNOME Greece
- We configured proper monitoring for all the hosted services thanks to Nagios and Check-MK
- We migrated the IRC network to a newer ircd with proper IRC services (Nickserv, Chanserv) in place.
- We made sure each machine had a configured management (mgmt) and KVM interface for direct remote access to the bare metal machine itself, its hardware status and all the operations related to it. (hard reset, reboot, shutdown etc.)
- We upgraded MoinMoin to the latest release and made a substantial cleanup of old accounts, pages marked as spam and trashed pages.
- We deployed DNSSEC for several domains we manage including gnome.org, guadec.es, gnomehispano.es, guadec.org, gtk.org and gimp.org
- We introduced an account de-activation policy which comes into play when a contributor not committing to any of the hosted repositories at git.gnome.org since two years is caught by the script. The account in question is marked as inactive and the gnomecvs (from the old cvs days) and ftpadmin groups are removed.
- We planned mass reboots of all the machines roughly every month for properly applying security and kernel updates.
- We introduced Mirrorbrain (MB), the mirroring service serving GNOME and related modules tarballs and software all over the world. Before introducing MB GNOME had several mirrors located in all the main continents and at the same time a very low amount of users making good use of them. Many organizations and companies behind these mirrors decided to not host GNOME sources anymore as the statistics of usage were very poor and preferred providing the same service to projects that really had a demand for these resources. MB solved all this allowing a proper redirect to the closest mirror (through mod_geoip) and making sure the sources checksum matched across all the mirrors and against the original tarball uploaded by a GNOME maintainer and hosted at master.gnome.org.
$ cd /git/GNOME/puppet && git shortlog -ns 3520 Andrea Veri 506 Olav Vitters 338 Owen W. Taylor 239 Patrick Uiterwijk 112 Jeff Schroeder 71 Christer Edwards 4 Daniel Mustieles 4 Matanya Moses 3 Tobias Mueller 2 John Carr 2 Ray Wang 1 Daniel Mustieles Garc a 1 Peter Baumgartenand from the Request Tracker database (52388 being my assigned ID):
mysql> select count(*) from Tickets where LastUpdatedBy = '52388'; +----------+ count(*) +----------+ 3613 +----------+ 1 row in set (0.01 sec) mysql> select count(*) from Tickets where LastUpdatedBy = '52388' and Status = 'Resolved'; +----------+ count(*) +----------+ 1596 +----------+ 1 row in set (0.03 sec)It s been a long run which made me proud, for the things I learnt, for the tasks I ve been able to accomplish, for the great support the GNOME community gave me all the time and most of all for the same fact of being part of the team responsible of the systems hosting the GNOME Project. Thank you GNOME community for your continued and never ending backing, we daily work to improve how the services we host are delivered to you and the support we receive back is fundamental for our passion and enthusiasm to remain high!
A new package! A couple of weeks ago folks at Google released 
.
I think it s important that organizations as 
Here is my monthly update covering a large part of what I have been doing in the free software world (
Yesterday, 
Eric Raymond, author of 
For the past fifteen years, I have been 
This post is intended to document some elements of workflow that I ve setup to manage videos produced for a MOOC, where different colleagues work collaboratively on a set of video sequences, in a remote way.
We are a team of several schools working on the same course, and we have an incremental process, so we need some collaboration over a quite long period of many remote authors, over a set of video sequences.
We re probably going to review some of the videos and make changes, so we need to monitor changes, and submit versions to colleagues on remote sites so they can criticize and get later edits. We may have more that one site doing video production. Thus we need to share videos along the flow of production, editing and revision of the course contents, in a way that is manageable by power users (we re all computer scientists, used to SVN or Git).
I ve decided to start an experiment with Git and 
The drawback is that we need manual intervention for starting the rsync, and also that we must make sure that the rsync target is ready to get a backup.
The target of the rsync on Linux is a git-annex clone using the default indirect mode, which handles the files as symlinks to the actual copies managed by git-annex inside the .git/ directory. But that ain t suitable to be compared to the origin of the rsync mirror which are plain files on the Windows computer.
We must then do a git-annex edit on the whole target of the rsync mirror before the rsync, so that the files are there as regular video files. This is costly, in terms of storage, and also copying time (our repo contains around 50 Gb, and the Linux host is a rather tiny laptop).
After the rsync, all the files need to be compared to the SHA256 known to git-annex so that only modified files are taken into account in the commit. We perform a git-annex add on the whole files (for new files having appeared at rsync time), and then a git-annex sync . That takes a lot of time, since all SHA256 computations are quite long for such a set of big files (the video rushes and edited videos are in HD).
So the process needs to be the following, on the target Linux host:
I would have preferred to have a working git-annex on windows, but that is a much more manageable process for me for now, and until we have more videos to handle in our repo that our Linux laptop can hold, we re quite safe.
Next steps will probably involve gardening the contents of the repo on the Linux host so we re only keeping copies of current files, and older copies are only kept on the 2 servers, in case of later need.
I hope this can be useful to others, and I d welcome suggestions on how to improve our process.
Get involved
After 
Finally, bzed.de is back online and I m planning to start blogging again! Part
of the reason why I became inactive was the usage of 
I don't really remember where or how I stumbled upon this four women so I'm sorry that I can't give credit where credit is due, and I even do believe that I started writing a blog entry about them already somewhere. Anyway, I want to present you today