Search Results: "jan"

7 October 2014

Joachim Breitner: ghc-heap-view for GHC 7.8

Since the last release of ghc-heap-view, which was compatible with GHC-7.6, I got 8 requests for a GHC-7.8 compatible version. I started working on it in January, but got stuck and then kept putting it off. Today, I got the ninths request, and I did not want to wait for the tenth, so I finally finished the work and you can use the new ghc-heap-view-0.5.2 with GHC-7.8. I used this chance to migrate its source repository from Darcs to git (mirrored on GitHub), so maybe this means that when 7.10 comes out, the requests to update it come with working patches :-). I also added a small test script so that travis can check it: ghc-heap-view I did not test it very thoroughly yet. In particular, I did not test whether ghc-vis works as expected. I still think that the low-level interface that ghc-heap-view creates using custom Cmm code should move into GHC itself, so that it does not break that easily, but I still did not get around to propose a patch for that.

Andrea Veri: The GNOME Infrastructure is now powered by FreeIPA!

As preannounced here the GNOME Infrastructure switched to a new Account Management System which is reachable at https://account.gnome.org. All the details will follow. Introduction It s been a while since someone actually touched the underlaying authentication infrastructure that powers the GNOME machines. The very first setup was originally configured by Jonathan Blandford (jrb) who configured an OpenLDAP istance with several customized schemas. (pServer fields in the old CVS days, pubAuthorizedKeys and GNOME modules related fields in recent times) While OpenLDAP-server was living on the GNOME machine called clipboard (aka ldap.gnome.org) the clients were configured to synchronize users, groups, passwords through the nslcd daemon. After several years Jeff Schroeder joined the Sysadmin Team and during one cold evening (date is Tue, February 1st 2011) spent some time configuring SSSD to replace the nslcd daemon which was missing one of the most important SSSD features: caching. What surely convinced Jeff to adopt SSSD (a very new but promising sofware at that time as the first release happened right before 2010 s Christmas) and as the commit log also states ( New sssd module for ldap information caching ) was SSSD s caching feature. It was enough for a certain user to log in once and the /var/lib/sss/db directory was populated with its login information preventing the LDAP daemon in charge of picking up login details (from the LDAP server) to query the LDAP server itself every single time a request was made against it. This feature has definitely helped in many occasions especially when the LDAP server was down for a particular reason and sysadmins needed to access a specific machine or service: without SSSD this wasn t ever going to work and sysadmins were probably going to be locked out from the machines they were used to manage. (except if you still had /etc/passwd , /etc/group and /etc/shadow entries as fallback) Things were working just fine except for a few downsides that appeared later on:
  1. the web interface (view) on our LDAP user database was managed by Mango, an outdated tool which many wanted to rewrite in Django that slowly became a huge dinosaur nobody ever wanted to look into again
  2. the Foundation membership information were managed through a MySQL database, so two databases, two sets of users unrelated to each other
  3. users were not able to modify their own account information on their own but even a single e-mail change required them to mail the GNOME Accounts Team which was then going to authenticate their request and finally update the account.
Today s infrastructure changes are here to finally say the issues outlined at (1, 2, 3) are now fixed. What has changed? The GNOME Infrastructure is now powered by Red Hat s FreeIPA which bundles several FOSS softwares into one big bundle all surrounded by an easy and intuitive web UI that will help users update their account information on their own without the need of the Accounts Team or any other administrative entity. Users will also find two custom fields on their Overview page, these being Foundation Member since and Last Renewed on date . As you may have understood already we finally managed to migrate the Foundation membership database into LDAP itself to store the information we want once and for all. As a side note it might be possible that some users that were Foundation members in the past won t find any detail stored on the Foundation fields outlined above. That is actually expected as we were able to migrate all the current and old Foundation members that had an LDAP account registered at the time of the migration. If that s your case and you still would like the information to be stored on the new setup please get in contact with the Membership Committee at stating so. Where can I get my first login credentials? Let s make a little distinction between users that previously had access to Mango (usually maintainers) and users that didn t. If you were used to access Mango before you should be able to login on the new Account Management System by entering your GNOME username and the password you were used to use for loggin in into Mango. (after loggin in the very first time you will be prompted to update your password, please choose a strong password as this account will be unique across all the GNOME Infrastructure) If you never had access to Mango, you lost your password or the first time you read the word Mango on this post you thought why is he talking about a fruit now? you should be able to reset it by using the following command:
ssh -l yourgnomeuserid account.gnome.org
The command will start an SSH connection between you and account.gnome.org, once authenticated (with the SSH key you previously had registered on our Infrastructure) you will trigger a command that will directly send your brand new password on the e-mail registered for your account. From my tests seems GMail sees the e-mail as a phishing attempt probably because the body contains the word password twice. That said if the e-mail won t appear on your INBOX, please double-check your Spam folder. Now that Mango is gone how can I request a new account? With Mango we used to have a form that automatically e-mailed the maintainer of the selected GNOME module which was then going to approve / reject the request. From there and in the case of a positive vote from the maintainer the Accounts Team was going to create the account itself. With the recent introduction of a commit robot directly on l10n.gnome.org the number of account requests reduced its numbers. In addition to that users will now be able to perform pretty much all the needed maintenance on their accounts themselves. That said and while we will probably work on building a form in the future we feel that requesting accounts can definitely be achieved directly by mailing the Accounts Team itself which will mail the maintainer of the respective module and create the account. As just said the number of account creations has become very low and the queue is currently clear. The documentation has been updated to reflect these changes at: https://wiki.gnome.org/AccountsTeam
https://wiki.gnome.org/AccountsTeam/NewAccounts The migration of all the user data and ACLs has been massive and I ve been spending a lot of time reviewing the existing HBAC rules trying to spot possible errors or misconfigurations. If you happen to not being able to access a certain service as you were used to in the past, please get in contact with the Sysadmin Team. All the possible ways to contact us are available at https://wiki.gnome.org/Sysadmin/Contact. What is missing still? Now that the Foundation membership information has been moved to LDAP I ll be looking at porting some of the existing membership scripts to it. What I managed to port already are welcome e-mails for new or existing members. (renewals) Next step will be generating a membership page from LDAP (to populate http://www.gnome.org/foundation/membership) and all the your-membership-is-going-to-lapse e-mails that were being sent till today. Other news /home/users mount on master.gnome.org You will notice that loggin in into master.gnome.org will result in your home directory being empty, don t worry, you did not lose any of your files but master.gnome.org is now currently hosting your home directories itself. As you may have been aware of adding files to the public_html directory on master resulted in them appearing on your people.gnome.org/~userid space. That was unfortunately expected as both master and webapps2 (the machine serving people.gnome.org s webspaces) were mounting the same GlusterFS share. We wanted to prevent that behaviour to happen as we wanted to know who has access to what resource and where. From today master s home directories will be there just as a temporary spot for your tarballs, just scp and use ftpadmin against them, that should be all you need from master. If you are interested in receiving or keeping using your people.gnome.org s webspace please mail stating so. Other news a shiny and new error 500 page has been deployed Thanks to Magdalen Berns (magpie) a new error 500 web page has been deployed on all the Apache istances we host. The page contains an iframe of status.gnome.org and will appear every single time the web server behind the service you are trying to reach will be unreachable for maintenance or other purposes. While I hope you won t see the page that often you can still enjoy it at https://static.gnome.org/error-500/500.html. Make sure to whitelist status.gnome.org on your browser as it currently loads it without https. (as the service is currently hosted on OpenShift which provides us with a *.rhcloud.com wildcard certificate, which differs from the CN the browser would expect it to be)

Andrea Veri: The GNOME Infrastructure is now powered by FreeIPA!

As preannounced here the GNOME Infrastructure switched to a new Account Management System which is reachable at https://account.gnome.org. All the details will follow. Introduction It s been a while since someone actually touched the underlying authentication infrastructure that powers the GNOME machines. The very first setup was originally configured by Jonathan Blandford (jrb) who configured an OpenLDAP istance with several customized schemas. (pServer fields in the old CVS days, pubAuthorizedKeys and GNOME modules related fields in recent times) While OpenLDAP-server was living on the GNOME machine called clipboard (aka ldap.gnome.org) the clients were configured to synchronize users, groups, passwords through the nslcd daemon. After several years Jeff Schroeder joined the Sysadmin Team and during one cold evening (date is Tue, February 1st 2011) spent some time configuring SSSD to replace the nslcd daemon which was missing one of the most important SSSD features: caching. What surely convinced Jeff to adopt SSSD (a very new but promising sofware at that time as the first release happened right before 2010 s Christmas) and as the commit log also states ( New sssd module for ldap information caching ) was SSSD s caching feature. It was enough for a certain user to log in once and the /var/lib/sss/db directory was populated with its login information preventing the LDAP daemon in charge of picking up login details (from the LDAP server) to query the LDAP server itself every single time a request was made against it. This feature has definitely helped in many occasions especially when the LDAP server was down for a particular reason and sysadmins needed to access a specific machine or service: without SSSD this wasn t ever going to work and sysadmins were probably going to be locked out from the machines they were used to manage. (except if you still had /etc/passwd , /etc/group and /etc/shadow entries as fallback) Things were working just fine except for a few downsides that appeared later on:
  1. the web interface (view) on our LDAP user database was managed by Mango, an outdated tool which many wanted to rewrite in Django that slowly became a huge dinosaur nobody ever wanted to look into again
  2. the Foundation membership information were managed through a MySQL database, so two databases, two sets of users unrelated to each other
  3. users were not able to modify their own account information on their own but even a single e-mail change required them to mail the GNOME Accounts Team which was then going to authenticate their request and finally update the account.
Today s infrastructure changes are here to finally say the issues outlined at (1, 2, 3) are now fixed. What has changed? The GNOME Infrastructure is now powered by Red Hat s FreeIPA which bundles several FOSS softwares into one big bundle all surrounded by an easy and intuitive web UI that will help users update their account information on their own without the need of the Accounts Team or any other administrative entity. Users will also find two custom fields on their Overview page, these being Foundation Member since and Last Renewed on date . As you may have understood already we finally managed to migrate the Foundation membership database into LDAP itself to store the information we want once and for all. As a side note it might be possible that some users that were Foundation members in the past won t find any detail stored on the Foundation fields outlined above. That is actually expected as we were able to migrate all the current and old Foundation members that had an LDAP account registered at the time of the migration. If that s your case and you still would like the information to be stored on the new setup please get in contact with the Membership Committee at stating so. Where can I get my first login credentials? Let s make a little distinction between users that previously had access to Mango (usually maintainers) and users that didn t. If you were used to access Mango before you should be able to login on the new Account Management System by entering your GNOME username and the password you were used to use for loggin in into Mango. (after loggin in the very first time you will be prompted to update your password, please choose a strong password as this account will be unique across all the GNOME Infrastructure) If you never had access to Mango, you lost your password or the first time you read the word Mango on this post you thought why is he talking about a fruit now? you should be able to reset it by using the following command:
ssh -l yourgnomeuserid account.gnome.org
The command will start an SSH connection between you and account.gnome.org, once authenticated (with the SSH key you previously had registered on our Infrastructure) you will trigger a command that will directly send your brand new password on the e-mail registered for your account. From my tests seems GMail sees the e-mail as a phishing attempt probably because the body contains the word password twice. That said if the e-mail won t appear on your INBOX, please double-check your Spam folder. Now that Mango is gone how can I request a new account? With Mango we used to have a form that automatically e-mailed the maintainer of the selected GNOME module which was then going to approve / reject the request. From there and in the case of a positive vote from the maintainer the Accounts Team was going to create the account itself. With the recent introduction of a commit robot directly on l10n.gnome.org the number of account requests reduced its numbers. In addition to that users will now be able to perform pretty much all the needed maintenance on their accounts themselves. That said and while we will probably work on building a form in the future we feel that requesting accounts can definitely be achieved directly by mailing the Accounts Team itself which will mail the maintainer of the respective module and create the account. As just said the number of account creations has become very low and the queue is currently clear. The documentation has been updated to reflect these changes at: https://wiki.gnome.org/AccountsTeam
https://wiki.gnome.org/AccountsTeam/NewAccounts I was used to have access to a specific service but I don t anymore, what should I do? The migration of all the user data and ACLs has been massive and I ve been spending a lot of time reviewing the existing HBAC rules trying to spot possible errors or misconfigurations. If you happen to not being able to access a certain service as you were used to in the past, please get in contact with the Sysadmin Team. All the possible ways to contact us are available at https://wiki.gnome.org/Sysadmin/Contact. What is missing still? Now that the Foundation membership information has been moved to LDAP I ll be looking at porting some of the existing membership scripts to it. What I managed to port already are welcome e-mails for new or existing members. (renewals) Next step will be generating a membership page from LDAP (to populate http://www.gnome.org/foundation/membership) and all the your-membership-is-going-to-lapse e-mails that were being sent till today. Other news /home/users mount on master.gnome.org You will notice that loggin in into master.gnome.org will result in your home directory being empty, don t worry, you did not lose any of your files but master.gnome.org is now currently hosting your home directories itself. As you may have been aware of adding files to the public_html directory on master resulted in them appearing on your people.gnome.org/~userid space. That was unfortunately expected as both master and webapps2 (the machine serving people.gnome.org s webspaces) were mounting the same GlusterFS share. We wanted to prevent that behaviour to happen as we wanted to know who has access to what resource and where. From today master s home directories will be there just as a temporary spot for your tarballs, just scp and use ftpadmin against them, that should be all you need from master. If you are interested in receiving or keeping using your people.gnome.org s webspace please mail <accounts AT gnome DOT org> stating so. Other news a shiny and new error 500 page has been deployed Thanks to Magdalen Berns (magpie) a new error 500 web page has been deployed on all the Apache istances we host. The page contains an iframe of status.gnome.org and will appear every single time the web server behind the service you are trying to reach will be unreachable for maintenance or other purposes. While I hope you won t see the page that often you can still enjoy it at https://static.gnome.org/error-500/500.html. Make sure to whitelist status.gnome.org on your browser as it currently loads it without https. (as the service is currently hosted on OpenShift which provides us with a *.rhcloud.com wildcard certificate, which differs from the CN the browser would expect it to be) Updates UPDATE on status.gnome.org s SSL certificate: the certificate has been provisioned and it should result in the 500 s page to be displayed correctly with no warnings from your browser. UPDATE from Adam Young on Kerberos ports being closed on many DC s firewalls:
The next version of upstream MIT Kerberos will have support for fetching a ticket via ports 443 and marshalling the request over HTTPS. We ll need to run a proxy on the server side, but we should be able to make it work: Read up here
http://adam.younglogic.com/2014/06/kerberos-firewalls

5 October 2014

Thomas Goirand: OpenStack packaging activity: September 2014

I decided I d post this monthly. It may be a bit boring, sorry, but I think it s a nice thing to have this public. The log starts on the 6th, because on the 4th I was back from Debconf (after a day in San Francisco, plus 20 hours of traveling and 15 hours of time gap). It is to be noted that every time something is uploaded in Debian for Icehouse (in Sid), or for Juno (in Experimental), there s also a corresponding backport produced for Wheezy. Saturday 6th & Sunday 7th:
packaged libjs-twitter-bootstrap-wizard (in new queue)
Uploaded python-pint after reviewing the debian/copyright
Worked on updating python-eventlet in Experimental, and adding Python3 support. It seems Python3 support isn t ready yet, so I will probably remove that feature from the package update.
Tried to apply the Django 1.7 patches for python-django-bootstrap-form. They didn t work, but Raphael came back on Monday morning with new versions
of the patches, which should be good this time.
Helped the DSA (Debian System Administrators) with the Debian OpenStack cloud. It s looking good and working now (note: I helped them during Debconf 14).
Started a page about adding more tasksel tasks: https://wiki.debian.org/tasksel/MoreTasks. It s looking like Joey Hess is adding new tasks by default in Tasksel, with OpenStack compute node and OpenStack proxy node . It will be nice to have them in the default Debian installer! :)
Packaged and uploaded python-dib-utils, now in NEW queue. Monday 8th:
Uploaded fixed python-django-bootstrap-form with patch for Django 1.7.
Packaged and uploaded python-pysaml2.
Finilized and uploaded python-jingo which is needed for python-django-compressor unit tests
Finalized and uploaded python-coffin which is needed for python-django-compressor unit tests
Worked on running the unit tests for python-django-compressor, as I needed to know if it could work with Django 1.7. It was hard to find the correct way to run the unit tests, but finally, they all passed. I will add the unit tests once coffin and jingo will be accepted in Sid.
Applied patches in the Debian BTS for python-django-openstack-auth and Django 1.7. Uploaded the fixed package.
Fixed python-django-pyscss compat with Django 1.7, uploaded the result.
Updated keystone to Juno b3.
Built Wheezy backports of some JS libs needed for Horizon in Juno, which I already uploaded to Sid last summer:
o libjs-twitter-bootstrap-datepicker
o libjs-jquery.quicksearch
o libjs-spin.js
Upstreamed the Django 1.7 patch for python-django-openstack-auth: https://review.openstack.org/119972 Tuesday 9:
Updated and uploaded Swift 2.1.0. Added swift-object-expirer package to it, together with init script. Wednesday 10:
Basically, cleaned the Debian BTS of almost all issues today :P
Added it.po update to nova (Closes: #758305).
Backported libvirt 1.2.7 to Wheezy, to be able to close this bug: https://bugs.debian.org/757548 (eg: changed dependency from libvirt-bin to libvirt-daemon-system)
Uploaded the fixed nova package using libvirt-daemon-system
Upgraded python-trollius to 1.0.1
Fixed tuskar-ui to work with Django 1.7. Disabled pep8 tests during build. Added build-conflicts: python-unittest2.
Fixed python-django-compressor for Django 1.7, and now running unit tests with it, after python-coffin and python-jingo got approved in Sid by FTP masters.
Fixed python-xstatic wrong upstream URLs.
Added it.po debconf translation to Designate.
Added de.po debconf translation to Tuskar.
Fixed copyright holders in python-xstatic-rickshaw
Added python-passlib as dependency for python-cinder. Remaining 3 issues in the BTS: ceilometer FTBFS, Horizon unit test with Django 1.7, Designate fail to install. All of the 3 are harder to fix, and I may try to do so later this week. Thursday 11:
Fixed python-xstatic-angular and python-xstatic-angular-mock to deal with the new libjs-angularjs version (closes 2 Debian RC bugs: uninstallable).
Fixed ceilometer FTBFS (Closes rc bug) Friday 12:
Fixed wrong copyright file for libjs-twitter-bootstrap-wizard after the FTP masters told me, and reuploaded to Sid.
Reuploaded wrong upload of ceilometer (wrong hash for orig.tar.xz)
Packaged and uploaded python-xstatic-bootstrap-scss
Packaged and uploaded python-xstatic-font-awesome
Packaged and uploaded ntpstat Monday 15:
packaged and uploaded python-xstatic-jquery.bootstrap.wizard
Fixed python-xstatic-angular-cookies to use new libjs-angularjs version (fixed version dependencies)
Fixed Ceilometer FTBFS (Closes: #759967)
Backported all python-xtatic packages to Wheezy, including all dependencies. This includes backporting of a bunch of packages from nodejs which were needed as build-dependencies (around 70 packages ). Filed about 5 or 6 release critical bugs as some nodejs packages were not buildable as-is.
Fixed some too restrictive python-xstatic-angular* dependencies on the libjs-angularjs (the libjs-angularjs increased version). Tuesday 16:
Uploaded updates to Experimental:
o python-eventlet 0.15.2 (this one took a long time as it needed maintenance)
o oslo-config
o python-oslo.i18n
Uploaded to Sid:
o python-diskimage-builder 0.1.30-1
o python-django-pyscss 1.0.2-1
Fixed horizon libapache-mode-wsgi to be a dependency of openstack-dashboard-apache and not just openstack-dashboard (in both Icehouse & Juno).
Removed the last failing Django 1.7 unit test from Horizon. It doesn t seem relevant anyway.
Backported python-netaddr 0.7.12 to Wheezy (needed by oslo-config).
Started working on oslo.rootwrap, though it failed to build in Wheezy with a unit test failure. Wednesday 17:
To experimental:
o Uploaded oslo.rootwrap 1.3.0.0~a1. It needed a build-depends on iproute2 because of a new test.
o Uploaded python-oslo.utils 0.3.0
o Uploaded python-oslo.vmware 0.6.0, fixed sphinx-build conf.py and filed a bug about it: https://bugs.launchpad.net/oslo.vmware/+bug/1370370 plus emailed the commiter of the issue (which appeared 2 weeks ago).
o Uploaded python-pycadf 0.6.0
o Uploaded python-pyghmi 0.6.17
o Uploaded python-oslotest 1.1.0.0~a2, including patch for Wheezy, which I also submited upstream: https://review.openstack.org/122171/
o Uploaded glanceclient 0.14.0, added a patch to not use the embedded version of urllib3 in requests: https://review.openstack.org/122184
To Sid:
o Uploaded python-zake_0.1.6-1 Thesday 18:
Backported zeromq3-4.0.4+dfs, pyzmq-14.3.1, pyasn1-0.1.7, python-pyasn1-modules-0.0.5
Uploaded keystoneclient 0.10.1, fixed the saml2 unit tests which were broken using testtools >= 0.9.39. Filed bug, and warned code author: https://bugs.launchpad.net/python-keystoneclient/+bug/1371085
Uploade swiftclient 2.3.0 to experimental.
Uploaded ironicclient 0.2.1 to experimental.
Uploaded saharaclient, filed bug with saharaclient expecting an up and running keystone server: https://bugs.launchpad.net/python-saharaclient/+bug/1371177 Friday 19:
Uploaded keystone Juno b3, filed but about unit tests downloading with git, while no network access should be performed during package build (forbidden by
Debian policy)
Uploaded python-oslo.db 1.0.0 which I forgot in the dependency list, and which was needed for Neutron.
Uploaded nova 2014.2~b3-1 (added a new nova-serialproxy service daemon to the nova-consoleproxy) Saturday 20:
Uploaded Neutron Juno b3.
Uploaded python-retrying 1.2.3 (was missing from depends upload)
Uploaded Glance Juno b3.
Uploaded Cinder Juno b3.
Fixed python-xstatic-angular-mock which had a .pth packaged, as well as the data folder (uploaded debian release -3).
Fixed missing depends and build-conflicts in python-xstatic-jquery. Sunday 21:
Dropped python-pil & python-django-discover-runner from runtime Depends: of python-django-pyscss, as it s only needed for tests. It also created a conflicts, because python-django-discover-runner depends on python-unittest2 and horizon build-conflicts with it.
Forward-ported the Django 1.7 patches for Horizon. Opened new patch: https://review.openstack.org/122992 (since the old fix has gone away after a refactor of the unit test).
Uploaded Horizon Juno b3.
Applied https://review.openstack.org/#/c/122768/ to the keystone package, so that it doesn t do git clone of the keystoneclient during build.
Uploaded oslo.messaging 1.4.0.0 (which really is 1.4.0) to experimental
Uploaded oslo.messaging 1.4.0.0+really+1.3.1-1 to fix the issue in Sid/Jessie after the wrong upload (due to Zul wrong tagging of Keystone in the 2014.1.2 point release). Monday 22:
Uploaded ironic 2014.2~b3-1 to experimental
Uploaded heat 2014.2~b3-1 (with some fixes for sphinx doc build)
Uploaded ceilometer 2014.2~b3-1 to experimental
Uploaded openstack-doc-tools 0.19-1 to experimental
Uploaded openstack-trove 2014.2~b3-1 to experimental Tuesday 23:
Uploaded python-neutronclient with fixed version number for cliff and six. This missing requirement for cliff version produced an error in Trove, which I don t want to happen again.
Added fix for unit tests in Trove: https://review.openstack.org/#/c/123450/1,publish
Uploaded oslo.messaging 1.4.1 in Experimental, fixing the version conflicts with the one in Sid/Jessie. Thanks to Doug Hellman for doing the tagging. I will need to upload new versions of the following packages with the >= 1.4.1 depends:
> ceilometer
> ironic
> keystone
> neutron
> nova
> oslo-config
> oslo.rootwrap
> oslo.i18n
> python-pycadf
See http://lists.openstack.org/pipermail/openstack-dev/2014-September/046795.html for more explanation about the mess I m repairing
Uploaded designate Juno b3. Wednesday 24:
Uploaded oslosphinx 2.2.0.0
Uploaded update to django-openstack-auth (new last minute requirement for Horizon).
Uploaded final oslo-config package version 1.4.0.0 (really is 1.4.0)
Packaged and uploaded Sahara. This needs some tests by someone else as I don t even know how it works. Thuesday 25:
Uploaded python-keystonemiddleware 1.0.0-3, fixing CVE-2014-7144] TLS cert verification option not honoured in paste configs. https://bugs.debian.org/762748
Packaged and uploaded python-yaql, sent pull request for fixing print statements into Python3 compatible print function calls: https://github.com/ativelkov/yaql/pull/15
Packaged and uploaded python-muranoclient.
Started the packaging of Murano (not finished yet).
Uploaded python-keystoneclient 0.10.1-2 with the CVE-2014-7144 fix to Sid, with urgency=high. Uploaded 0.11.1-1 to Experimental.
Uploaded python-keystonemiddleware fix for CVE-2014-7144.
Uploaded openstack-trove 2014.2~b3-3 with last unit test fix from https://review.openstack.org/#/c/123450/ Friday 26:
Uploaded a fix for murano-agent, which makes it run as root.
Finished the packaging of Murano
Started packaging murano-dashboard, sent this patch to fix the wrong usage of the /usr/bin/coverage command: https://review.openstack.org/124444
Fixed wrong BASE_DIR in python-xstatic-angular and python-xstatic-angular-mock Saturday 27:
uploaded python-xstatic-boostrap-scss which I forgot to upload :(
uploaded python-pyscss 1.2.1 Sunday 28:
After a long investigation, I found out that the issue when installing the openstack-dasboard package was due to a wrong patch I did for Python 3.2 in Wheezy in python-pyscss. Corrected the patch from version 1.2.1-1, and uploaded version 1.2.1-2, the dashboard now installs correctly. \o/
Did a new version of an Horizon patch at https://review.openstack.org/122992/ to address Django 1.7 compat. Monday 29:
Uploaded new version of python-pyscss fixing the last issue with Python 3 (there was a release critical bug on it).
Uploaded fixup for python-django-openstack-auth fail to build in the Sid version, which was broken since the last upload of keystoneclient (which makes some of its API now as private).
Uploaded python-glance-store 0.1.8, including Ubuntu patch to fix unit tests.
Reviewed the packaging of python-strict-rfc3339 (see https://bugs.debian.org761152).
Uploaded Sheepdog with fix in the init script to start after corosync (Closes: #759216).
Uploaded pt_BR.po Brazilian Portuguese debconf templates translation for nova Icehouse in Sid (only commited it in Git for Juno).
Same for Glance. Tuesday 30:
Added Python3 support in python-django-appconf, uploaded to Sid
Upgraded to python-django-pyscss 1.0.3, and fixed broken unit tests with this new release under Django 1.7. Created pull request: https://github.com/fusionbox/django-pyscss/pull/22
Fixed designate requirements.txt in Sid (Icehouse) to allow SQLA 0.9.x. Uploaded resulting package to Sid.
Uploaded new Debian fix for python-tooz: kills memcached only if the package scripts started it (plus cleans .testrepository on clean).
Uploaded initial release of murano
Uploaded python-retrying with patch from Ubuntu to remove embedded copy of six.py code.
Uploaded python-oslo.i18n 1.0.0 to experimental (same as before, just bump of version #)
Uploaded python-oslo.utils 1.0.0 to experimental (same as before, just bump of version #)
Uploaded Keystone Juno RC1
Uploaded Glance Juno RC1

2 October 2014

Rapha&#235;l Hertzog: My Free Software Activities in September 2014

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (26.6 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Django 1.7 Since Django 1.7 got released early September, I updated the package in experimental and continued to push for its inclusion in unstable. I sent a few more patches to multiple reverse build dependencies who had asked for help (python-django-bootstrap-form, horizon, lava-server) and then sent the package to unstable. At that time, I bumped the severity of all bug filed against packages that were no longer building with Django 1.7. Later in the month, I made sure that the package migrated to testing, it only required a temporary removal of mumble-django (see #763087). Quite a few packages got updated since then (remaining bugs here). Debian Long Term Support I have worked towards keeping Debian Squeeze secure, see the dedicated article: My Debian LTS report for September 2014. Distro Tracker The pace of development on tracker.debian.org slowed down a bit this month, with only 30 new commits in the repository, closing 6 bugs. Some of the changes are noteworthy though: the news now contain true links on bugs, CVE and plain URLs (example here). I have also fixed a serious issue with the way users were identified when they used their Alioth account credentials to login via sso.debian.org. On the development side, we re now able to generate the test suite code coverage which is quite helpful to identify parts of the code that are clearly missing some tests (see bin/gen-coverage.sh in the repository). Misc packaging Publican. I have been behind packaging new upstream versions of Publican and with the freeze approaching, I decided to take care of it. Unfortunately, it wasn t as easy as I had hoped and found numerous issues that I have filed upstream (invalid public identifier, PDF build fails with noNumberLines function available, build of the manual requires the network). Most of those have been fixed upstream in the mean time but the last issue seems to be a problem in the way we manage our Docbook XML catalogs in Debian. I have thus filed #763598 (docbook-xml: xmllint fails to identify local copy of docbook entities file) which is still waiting an answer from the maintainer. Package sponsorship. I have sponsored new uploads of dolibarr (RC bug fix), tcpdf (RC bug fix), tryton-server (security update) and django-ratelimit. GNOME 3.14. With the arrival of GNOME 3.14 in unstable, I took care of updating gnome-shell-timer and also filed some tickets for extensions that I use: https://github.com/projecthamster/shell-extension/issues/79 and https://github.com/olebowle/gnome-shell-timer/issues/25 git-buildpackage. I filed multiple bugs on git-buildpackage for little issues that have been irking me since I started using this tool: #761160 (gbp pq export/switch should be smarter), #761161 (gbp pq import+export should preserve patch filenames), #761641 (gbp import-orig should be less fragile and more idempotent). Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

1 October 2014

Thorsten Alteholz: My Debian Activities in September 2014

FTP assistant Starting an article with self laudation might be bad style, but this month I was busy as a bee and could accept 312 packages, 75 packages more than last month. 34 times I contacted the maintainer to ask a question and 51 times I had to reject a package. These numbers remain constant. The number of packages in NEW dropped to about 180. If you want your package included in Jessie, please double-check it and upload an improved version. Squeeze LTS This was my third month that I did some work for the Squeeze LTS initiative, started by Raphael Hertzog at Freexian. All in all I got assigned a workload of 11h for September and I spent these hours to upload new versions of I further tried to upload a new version of python-django. Unfortunately I could not figure out why some of the internal tests of the package failed. So I fowarded the package to Raphael, who could resolve all issues. The Squeeze version of PHP5 contains 140 patches. According to quilt 47 of them are identified to be already in 5.3.29 and 48 patches need to be revised. Some of them are really big, rather old and not really supported in the new 5.3.n version.
As nobody will talk about Squeeze LTS in a few months, I better better avoid the hassle of preparing a point release and concentrate only on security patches further on. Other packages This month I uploaded a new version of net-dns-fingerprint, which closes an RC bug. Unfortunately the package does not work with all DNS servers anymore. Patches or hints what happened are very welcome :-) . Support If you would like to support my Debian work you could either be part of the Freexian initiative (see above) or consider to send some bitcoins to 1JHnNpbgzxkoNexeXsTUGS6qUp5P88vHej. Contact me at donation@alteholz.eu if you prefer another way to donate. Every kind of support is most appreciated.

30 September 2014

Rapha&#235;l Hertzog: My Debian LTS report for September

Thanks to the sponsorship of multiple companies, I have been paid to work 11 hours on Debian LTS this month. CVE triagingI started by doing lots of triage in the security tracker (if you want to help, instructions are here) because I noticed that the dla-needed.txt list (which contains the list of packages that must be taken care of via an LTS security update) was missing quite a few packages that had open vulnerabilities in oldstable. In the end, I pushed 23 commits to the security tracker. I won t list the details each time but for once, it s interesting to let you know the kind of things that this work entailed: And when I say I reviewed it s a simplification for this kind of process: CVE triaging is often almost half the work in the general process: once you know that you are affected and that you have a patch, the process to release an update is relatively straightforward (sometimes there s still work to do to backport the patch). Once I was over that first pass of triaging, I had already spent more than the 11 hours paid but I still took care of preparing the security update for python-django. Thorsten Alteholz had started the work but got stuck in the process of backporting the patches. Since I m co-maintainer of the package, I took over and finished the work to release it as DLA-65-1.

No comment Liked this article? Click here. My blog is Flattr-enabled.

25 September 2014

Jan Wagner: Redis HA with Redis Sentinel and VIP

For an actual project we decided to use Redis for some reasons. As there is availability a critical part, we discovered that Redis Sentinel can monitor Redis and handle an automatic master failover to a available slave. Setting up the Redis replication was straight forward and even setting up Sentinel. Please keep in mind, if you configure Redis to require an authentication password, you even need to provide that for the replication process (masterauth) and for the Sentinel connection (auth-pass). The more interesting part is, how to migrate over the clients to the new master in case of a failover process. While Redis Sentinel could also be used as configuration provider, we decided not to use this feature, as the application needs to request the actual master node from Redis Sentinel much often, which will maybe a performance impact.
The first idea was to use some kind of VRRP, implemented into keepalived or something like this. The problem with such a solution is, you need to notify the VRRP process when a redis failover is in progress.
Well, Redis Sentinel has a configuration option called 'sentinel client-reconfig-script':
# When the master changed because of a failover a script can be called in
# order to perform application-specific tasks to notify the clients that the
# configuration has changed and the master is at a different address.
# 
# The following arguments are passed to the script:
#
# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
#
# <state> is currently always "failover"
# <role> is either "leader" or "observer"
# 
# The arguments from-ip, from-port, to-ip, to-port are used to communicate
# the old address of the master and the new address of the elected slave
# (now a master).
#
# This script should be resistant to multiple invocations.
This looks pretty good and as there is provided a <role>, I thought it would be a good idea to just call a script which evaluates this value and based on it's return, it adds the VIP to the local network interface, when we get 'leader' and removes it when we get 'observer'. It turned out that this was not working as <role> didn't returned 'leader' when the local redis instance got master and 'observer' when got slave in any case. This was pretty annoying and I was short before giving up.
Fortunately I stumpled upon a (maybe) chinese post about Redis Sentinal, where was done the same like I did. On the second look I recognized that the decision was made on $ 6 which is <to-ip>, nothing more then the new IP of the Redis master instance. So I rewrote my tiny shell script and after some other pitfalls this strategy worked out well. Some notes about convergence. Actually it takes round about 6-7 seconds to have the VIP migrated over to the new node after Redis Sentinel notifies a broken master. This is not the best performance, but as we expect this happen not so often, we need to design the application using our Redis setup to handle this (hopefully) rare scenario.

Petter Reinholdtsen: Suddenly I am the new upstream of the lsdvd command line tool

I use the lsdvd tool to handle my fairly large DVD collection. It is a nice command line tool to get details about a DVD, like title, tracks, track length, etc, in XML, Perl or human readable format. But lsdvd have not seen any new development since 2006 and had a few irritating bugs affecting its use with some DVDs. Upstream seemed to be dead, and in January I sent a small probe asking for a version control repository for the project, without any reply. But I use it regularly and would like to get an updated version into Debian. So two weeks ago I tried harder to get in touch with the project admin, and after getting a reply from him explaining that he was no longer interested in the project, I asked if I could take over. And yesterday, I became project admin. I've been in touch with a Gentoo developer and the Debian maintainer interested in joining forces to maintain the upstream project, and I hope we can get a new release out fairly quickly, collecting the patches spread around on the internet into on place. I've added the relevant Debian patches to the freshly created git repository, and expect the Gentoo patches to make it too. If you got a DVD collection and care about command line tools, check out the git source and join the project mailing list. :)

22 September 2014

Gunnar Wolf: One month later: How is the set of Debian keyrings faring?

OK, it's almost one month since we (the keyring-maintainers) gave our talk at DebConf14; how are we faring regarding key transitions since then? You can compare the numbers (the graphs, really) to those in our DC14 presentation. Since the presentation, we have had two keyring pushes: First of all, the Non-uploading keyring is all fine: As it was quite recently created, and as it is much smaller than our other keyrings, it has no weak (1024 bit) keys. It briefly had one in 2010-2011, but it's long been replaced. Second, the Maintainers keyring: In late July we had 222 maintainers (170 with >=2048 bit keys, 52 with weak keys). By the end of August we had 221: 172 and 49 respectively, and by September 18 we had 221: 175 and 46. As for the Uploading developers, in late July we had 1002 uploading developers (481 with >=2048 bit keys, 521 with weak keys). By the end of August we had 1002: 512 and 490 respectively, and by September 18 we had 999: 531 and 468. Please note that these numbers do not say directly that six DMs or that 50 uploading DDs moved to stronger keys, as you'd have to factor in new people being added, keys migrating between different keyrings (mostly DM DD), and people retiring from the project; you can get the detailed information looking at the public copy of our Git repository, particularly of its changelog. And where does that put us? Of course, I'm very happy to see that the lines in our largest keyring have already crossed. We now have more people with >=2048 bit keys. And there was a lot of work to do this processing done! But that still means... That in order not to lock a large proportion of Debian Developers and Maintainers out of the project, we have a real lot of work to do. We would like to keep the replacement slope high (because, remember, in January 1st we will remove all small keys from the keyring). And yes, we are willing to do the work. But we need you to push us for it: We need you to get a new key created, to gather enough (two!) DD signatures in it, and to request a key replacement via RT. So, by all means: Do keep us busy!
AttachmentSize
Debian Developers (uploading)266.66 KB
Debian Developers (non-uploading)204.17 KB
Debian Maintainers296.73 KB

19 September 2014

Dariusz Dwornikowski: getting real "done date" of a bug from Debian BTS

As I wrote in my last post currently, SOAP interface, nor Ultimate Debian Database do not provide a date when a given bug was closed (done date). It is quite hard to calculate statistics on a bug tracker when you do not know when a bug was closed (!!). Done date of bug can be found in its log. The log itself can be downloaded by SOAP method get_bug_log but the processing of it is quite complicated. The same comes to web scrapping of a BTS's web interface. Fortunatelly the web interface gives a possibility to download a log in an mbox format. Below is a script that extracts the done date of a bug from its log in mbox format. It uses requests to download the mbox and caches the result in ~/.cache/rfs_bugs, which you need to create. It performs different checks:
  1. Check existence of a header e.g. Received: (at 657783-done) by bugs.debian.org; 29 Jan 2012 13:27:42 +0000
  2. Check for header CC: NUMBER-close done
  3. Check for header TO: NUMBER-close done
  4. Check for Close: NUMBER in body.
The code is below:
import requests
from datetime import datetime
import mailbox
import re
import os
import tempfile
def get_done_date(bug_num):
    CACHE_DIR = os.path.expanduser("~") + "/.cache/rfs_bugs/"
    def get_from_cache():
        if os.path.exists(" ".format(CACHE_DIR, bug_num)):
            with open(" ".format(CACHE_DIR, bug_num)) as f:
                return datetime.strptime(f.readlines()[0].rstrip(), "%Y-%m-%d").date()
        else:
            return None
    done_date = get_from_cache()
    if done_date is not None:
        return done_date
    else:
        r = requests.get("https://bugs.debian.org/cgi-bin/bugreport.cgi?mbox=yes;bug= ;mboxstatus=yes".format(self._num))
        d = try_header(r.text)
        if d is None:
            d = try_cc(r.text)
        if d is None:
            d = try_body(r.text)
        if d is not None:
            with open(" ".format(CACHE_DIR, bug_num), "w") as f:
                f.write(" ".format(d.date()))
        else:
            return None
        return d.date()
    def try_body(text):
        reg = "\(at\s.+\)\s+by\sbugs\.debian\.org;\s(\d 1,2 \s\w\w\w\s\d\d\d\d)"
        handle, name = tempfile.mkstemp()
        with open(name, "w") as f:
            f.write(text.encode('latin-1'))
        mbox = mailbox.mbox(name)
        for i in mbox.items():
            if i[1].is_multipart():
                for m in i[1].get_payload():
                    if "close" in str(m) or "done" in str(m):
                        try:
                            result = re.search(reg, i[1]['Received'])
                            return datetime.strptime(result.group(1), "%d %b %Y")
                        except:
                            return None
            else:
                if "close" in i[1].get_payload() or "done" in i[1].get_payload():
                    try:
                        result = re.search(reg, i[1]['Received'])
                        return datetime.strptime(result.group(1), "%d %b %Y")
                    except:
                        return None
        return None
    def try_header(text):
        reg = "Received:\s\(at\s\d\d\d\d\d\d-(close done)\)\s+by.+"
        try:
            result = re.search(reg, r.text)
            line = result.group(0)
            reg2 = "\d 1,2 \s\w\w\w\s\d\d\d\d"
            result = re.search(reg2, line)
            d = datetime.strptime(result.group(0), "%d %b %Y")
            return d
        except:
            return None
    def try_cc(text):
        reg = "\(at\s.+\)\s+by\sbugs\.debian\.org;\s(\d 1,2 \s\w\w\w\s\d\d\d\d)"
        handle, name = tempfile.mkstemp()
        with open(name, "w") as f:
            f.write(text.encode('latin-1'))
        mbox = mailbox.mbox(name)
        for i in mbox.items():
            if ('CC' in i[1] and "done" in i[1]['CC']) or ('To' in i[1] and "done" in i[1]['To']):
                try:
                    result = re.search(reg, i[1]['Received'])
                    return datetime.strptime(result.group(1), "%d %b %Y")
                except:
                    return None
if __name__ == "__main__":
    print get_done_date(752210)
PS: I hope that the script will be not needed in the near future, as Don Armstrong plans a new BTS database, a Debconf14 video is here.

6 September 2014

Joachim Breitner: ICFP 2014

Another on-my-the-journey-back blog post; this time from the Frankfurt Airport Train Station my flight was delayed (if I knew that I could have watched the remaining Lightning Talks), and so was my train, but despite 5min of running through the Airport just not enough. And now that the free 30 Minutes of Railway Station Internet are used up, I have nothing else to do but blog... Last week I was attending ICFP 2014 in Gothenburg, followed by the Haskell Symposium and the Haskell Implementors Workshop. The justification to attend was the paper on Safe Coercions (joint work with Richard Eisenberg, Simon Peyton Jones and Stephanie Weirich), although Richard got to hold the talk, and did so quite well. So I got to leisurely attend the talks, while fighting the jet-lag that I brought from Portland. There were as expected quite a few interesting talks. Among them the first keynote, Kathleen Fisher on the need for formal methods in cars and toy-quadcopters and unmanned battle helicopters, which made me conclude that my Isabelle skills might eventually become relevant in practical applications. And did you know that if someone gains access to your car s electronics, they can make the seat belt pull you back hard? Stefanie Weirich s keynote (and the subsequent related talks by Jan Stolarek and Richard Eisenberg) on what a dependently typed Haskell would look like and what we could use it for was mouth-watering. I am a bit worried that Haskell will be become a bit obscure for newcomers and people that simply don t want to think about types too much, on the other hand it seems that Haskell as we know it will always stay there, just as a subset of the language. Similarly interesting were refinement types for Haskell (talks by Niki Vazou and by Eric Seidel), in the form of LiquidTypes, something that I have not paid attention to yet. It seems to be a good way for more high assurance in Haskell code. Finally, the Haskell Implementors Workshop had a truckload of exciting developments in and around Haskell: More on GHCJS, Partial type signatures, interactive type-driven development like we know it from Agda, the new Haskell module system and amazing user-defined error messages the latter unfortunately only in Helium, at least for now. But it s not the case that I only sat and listened. During the Haskell Implementors Workshop I held a talk Contributing to GHC with a live demo of me fixing a (tiny) bug in GHC, with the aim of getting more people to hack on GHC (slides, video). The main message here is that it is not that big of deal. And despite me not actually saying much interesting in the talk, I got good feedback afterwards. So if it now actually motivates someone to contribute to GHC, I m even more happier. And then there is of course the Hallway Track. I discussed the issues with fusing a left fold (unfortunately, without a great solution). In order to tackle this problem more systematically, John Wiegley and I created the beginning of a List Fusion Lab , i.e. a bunch of list benchmark and the possibility to compare various implementations (e.g. with different RULES) and various compilers. With that we can hopefully better assess the effect of a change to the list functions. PS: The next train is now also delayed, so I ll likely miss my tram and arrive home even later... PPS: I really have to update my 10 year old picture on my homepage (or redesign it completely). Quite a few people knew my name, but expected someone with shoulder-long hair... PPPS: Haskell is really becoming mainstream: I just talked to a randomly chosen person (the boy sitting next to me in the train), and he is a Haskell enthusiast, building a structured editor for Haskell together with his brother. And all that as a 12th-grader...

Jonathan McDowell: Breaking up with America

Back in January I changed jobs. This took me longer to decide to do than it should have. My US visa (an L-1B) was tied to the old job, and not transferable, so leaving the old job also meant leaving the US. That was hard to do; I'd had a mostly fun 3 and a half years in the SF Bay Area. The new job had an office in Belfast, and HQ in the Bay Area. I went to work in Belfast, and got sent out to the US to meet coworkers and generally get up to speed. During that visit the company applied for an H-1B visa for me. This would have let me return to the US in October 2014 and start working in the US office; up until that point I'd have continued to work from Belfast. Unfortunately there were 172,500 applications for 85,000 available visas and mine was not selected for processing. I'm disappointed by this. I've enjoyed my time in the US. I had a green card application in process, but after nearly 2 years it still hadn't completed the initial hurdle of the labor certification stage (a combination of a number of factors, human, organizational and governmental). However the effort of returning to live here seems too great for the benefits gained. I can work for a US company with a non-US office and return on an L-1B after a year. And once again have to leave should I grow out of the job, or the job change in some way that doesn't suit me, or the company hit problems and have to lay me off. Or I can try again for an H-1B next year, aiming for an October 2015 return, and hope that this time my application gets selected for processing. Neither really appeals. Both involve putting things on hold in the hope longer terms pans out as I hope. And to be honest I'm bored of that. I've loved living in America, but I ended up spending at least 6 months longer in the job I left in January than I'd have done if I'd been freely able to change employer without having to change continent. So it seems the time has come to accept that America and I must part ways, sad as that is. Which is why I'm currently sitting in SFO waiting for a flight back to Belfast and for the first time in 5 years not having any idea when I might be back in the US.

2 September 2014

Rapha&#235;l Hertzog: My Free Software Activities in August 2014

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (65.55 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Distro Tracker Even though I was officially in vacation during 3 of the 4 weeks of August, I spent many nights working on Distro Tracker. I m pleased to have managed to bring back Python 3 compatibility over all the (tested) code base. The full test suite now passes with Python 3.4 and Django 1.6 (or 1.7). From now on, I ll run tox on all code submitted to make sure that we won t regress on this point. tox also runs flake8 for me so that I can easily detect when the submitted code doesn t respect the PEP8 coding style. It also catches other interesting mistakes (like unused variable or too complex functions). Getting the code to pass flake8 was also a major effort, it resulted in a huge commit (89 files changed, 1763 insertions, 1176 deletions). Thanks to the extensive test suite, all those refactoring only resulted in two regressions that I fixed rather quickly. Some statistics: 51 commits over the last month, 41 by me, 3 by Andrew Starr-Bochicchio, 3 by Christophe Siraut, 3 by Joseph Herlant and 1 by Simon Kainz. Thanks to all of them! Their contributions ported some features that were already available on the old PTS. The new PTS is now warning of upcoming auto-removals, is displaying problems with uptream URLs, includes a short package description in the page title, and provides a link to screenshots (if they exist on screenshots.debian.net). We still have plenty of bugs to handle, so you can help too: check out https://tracker.debian.org/docs/contributing.html. I always leave easy bugs for others to handle, so grab one and get started! I ll review your patch with pleasure. :-) Tryton After my last batch of contributions to Tryton s French Chart of Accounts (#4108, #4109, #4110, #4111) C dric Krier granted me commit rights to the account_fr mercurial module. Debconf 14 I wasn t able to attend this year but thanks to awesome work of the video team, I watched some videos (and I still have a bunch that I want to see). Some of them were put online the day after they had been recorded. Really amazing work! Django 1.7 After the initial bug reports, I got some feedback of maintainers who feared that it would be difficult to get their packages working with Django 1.7. I helped them as best as I can by providing some patches (for horizon, for django-restricted-resource, for django-testscenarios). Since I expected many maintainers to be not very pro-active, I rebuilt all packages with Django 1.7 to detect at least those that would fail to build. I tagged as confirmed all the corresponding bug reports. Looking at https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=python-django@packages.debian.org;tag=django17, one can see that some progress has been made with 25 packages fixed. Still there are at least 25 others that are still problematic in sid and 35 that have not been investigated at all (except for the automatic rebuild that passed). Again your help is more than welcome! It s easy to install python-django 1.7 from experimental and they try to use/rebuild the packages from the above list. Dpkg translation With the freeze approaching, I wanted to ensure that dpkg was fully translated in French. I thus pinged debian-l10n-french@lists.debian.org and merged some translations that were done by volunteers. Unfortunately it looks like nobody really stepped up to maintain it in the long run so I did myself the required update when dpkg 1.17.12 got uploaded. Is there anyone willing to manage dpkg s French translation? With the latest changes in 1.17.13, we have again a few untranslated strings:
$ for i in $(find . -name fr.po); do echo $i; msgfmt -c -o /dev/null --statistics $i; done
./po/fr.po
1083 translated messages, 4 fuzzy translations, 1 untranslated message.
./dselect/po/fr.po
268 translated messages, 3 fuzzy translations.
./scripts/po/fr.po
545 translated messages.
./man/po/fr.po
2277 translated messages, 8 fuzzy translations, 3 untranslated messages.
Misc stuff I made an xsane QA upload (it s currently orphaned) to drop the (build-)dependency on liblcms1 and avoid getting it removed from Debian testing (see #745524). For the record, how-can-i-help warned me of this after one dist-upgrade. With the Django 1.7 work and the need to open up an experimental branch, I decided to switch python-django s packaging to git even though the current team policy is to use subversion. This triggered (once more) the discussion about a possible switch to git and I was pleased to see more enthusiasm this time around. Barry Warsaw tested a few workflows, shared his feeling and pushed toward a live discussion of the switch during Debconf. It looks like it might happen for good this time. I contributed my share in the discussions on the mailing list. Thanks See you next month for a new summary of my activities.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

31 August 2014

Thorsten Alteholz: My Debian activities in August 2014

FTP assistant By pure chance I was able to accept 237 packages, the same number as last month. 33 times I contacted the maintainer to ask a question about a package and 55 times I had to reject a package. The reject number increased a bit as I also worked on packages that already got a note but had not been fully processed. In contrast I only filed three serious bugs this month. Currently there are about 200 packages still waiting in the NEW queue As the freeze for Jessie comes closer every day, I wonder whether all of them can be processed in time. So I don t mind if every maintainer checks the package again and maybe uploads an improved version that can be processed faster. Squeeze LTS This was my second month that I did some work for the Squeeze LTS initiative, started by Raphael Hertzog at Freexian All in all I got assigned a workload of 16.5h for August. I spent these hours to upload new versions of As last month I prepared these uploads on the basis of the corresponding DSAs for Wheezy. For these packages backporting the Wheezy patches to Squeeze was rather easy. I also had a look at python-django and eglibc. Although the python-django patches apply now, the package fails some tests and these issues need some further investigation. In case of eglibc, my small pbuilder didn t have enough resources and trying to build the package resulted in a full disk after more than three hours of work. For PHP5 Ond ej Sur (the real maintainer) suggested to use point releases of upstream instead of applying only patches. I am curious about how much effort is needed for this approach. Stay tuned, next month you will be told more details! Anyway, this is still a lot of fun and I hope I can finish python-django, eglibc and php5 in September. Other packages This month my meep packages plus mpb have been part of a small hdf5 transition. All five packages needed a small patch and a new upload. As the patch was already provided by Gilles Filippini, this was done rather quickly. Support If you would like to support my Debian work you could either be part of the Freexian initiative (see above) or consider to send some bitcoins to 1JHnNpbgzxkoNexeXsTUGS6qUp5P88vHej. Contact me at donation@alteholz.eu if you prefer another way to donate. Every kind of support is most appreciated.

30 August 2014

Francois Marier: Outsourcing your webapp maintenance to Debian

Modern web applications are much more complicated than the simple Perl CGI scripts or PHP pages of the past. They usually start with a framework and include lots of external components both on the front-end and on the back-end. Here's an example from the Node.js back-end of a real application:
$ npm list   wc -l
256
What if one of these 256 external components has a security vulnerability? How would you know and what would you do if of your direct dependencies had a hard-coded dependency on the vulnerable version? It's a real problem and of course one way to avoid this is to write everything yourself. But that's neither realistic nor desirable. However, it's not a new problem. It was solved years ago by Linux distributions for C and C++ applications. For some reason though, this learning has not propagated to the web where the standard approach seems to be to "statically link everything". What if we could build on the work done by Debian maintainers and the security team?

Case study - the Libravatar project As a way of discussing a different approach to the problem of dependency management in web applications, let me describe the decisions made by the Libravatar project.

Description Libravatar is a federated and free software alternative to the Gravatar profile photo hosting site. From a developer point of view, it's a fairly simple stack: The service is split between the master node, where you create an account and upload your avatar, and a few mirrors, which serve the photos to third-party sites. Like with Gravatar, sites wanting to display images don't have to worry about a complicated protocol. In a nutshell, all that a site needs to do is hash the user's email and add that hash to a base URL. Where the federation kicks in is that every email domain is able to specify a different base URL via an SRV record in DNS. For example, francois@debian.org hashes to 7cc352a2907216992f0f16d2af50b070 and so the full URL is:
http://cdn.libravatar.org/avatar/7cc352a2907216992f0f16d2af50b070
whereas francois@fmarier.org hashes to 0110e86fdb31486c22dd381326d99de9 and the full URL is:
http://fmarier.org/avatar/0110e86fdb31486c22dd381326d99de9
due to the presence of an SRV record on fmarier.org.

Ground rules The main rules that the project follows is to:
  1. only use Python libraries that are in Debian
  2. use the versions present in the latest stable release (including backports)

Deployment using packages In addition to these rules around dependencies, we decided to treat the application as if it were going to be uploaded to Debian:
  • It includes an "upstream" Makefile which minifies CSS and JavaScript, gzips them, and compiles PO files (i.e. a "build" step).
  • The Makefile includes a test target which runs the unit tests and some lint checks (pylint, pyflakes and pep8).
  • Debian packages are produced to encode the dependencies in the standard way as well as to run various setup commands in maintainer scripts and install cron jobs.
  • The project runs its own package repository using reprepro to easily distribute these custom packages.
  • In order to update the repository and the packages installed on servers that we control, we use fabric, which is basically a fancy way to run commands over ssh.
  • Mirrors can simply add our repository to their apt sources.list and upgrade Libravatar packages at the same time as their system packages.

Results Overall, this approach has been quite successful and Libravatar has been a very low-maintenance service to run. The ground rules have however limited our choice of libraries. For example, to talk to our queuing system, we had to use the raw Python bindings to the C Gearman library instead of being able to use a nice pythonic library which wasn't in Debian squeeze at the time. There is of course always the possibility of packaging a missing library for Debian and maintaining a backport of it until the next Debian release. This wouldn't be a lot of work considering the fact that responsible bundling of a library would normally force you to follow its releases closely and keep any dependencies up to date, so you may as well share the result of that effort. But in the end, it turns out that there is a lot of Python stuff already in Debian and we haven't had to package anything new yet. Another thing that was somewhat scary, due to the number of packages that were going to get bumped to a new major version, was the upgrade from squeeze to wheezy. It turned out however that it was surprisingly easy to upgrade to wheezy's version of Django, Apache and Postgres. It may be a problem next time, but all that means is that you have to set a day aside every 2 years to bring everything up to date.

Problems The main problem we ran into is that we optimized for sysadmins and unfortunately made it harder for new developers to setup their environment. That's not very good from the point of view of welcoming new contributors as there is quite a bit of friction in preparing and testing your first patch. That's why we're looking at encoding our setup instructions into a Vagrant script so that new contributors can get started quickly. Another problem we faced is that because we use the Debian version of jQuery and minify our own JavaScript files in the build step of the Makefile, we were affected by the removal from that package of the minified version of jQuery. In our setup, there is no way to minify JavaScript files that are provided by other packages and so the only way to fix this would be to fork the package in our repository or (preferably) to work with the Debian maintainer and get it fixed globally in Debian. One thing worth noting is that while the Django project is very good at issuing backwards-compatible fixes for security issues, sometimes there is no way around disabling broken features. In practice, this means that we cannot run unattended-upgrades on our main server in case something breaks. Instead, we make use of apticron to automatically receive email reminders for any outstanding package updates. On that topic, it can occasionally take a while for security updates to be released in Debian, but this usually falls into one of two cases:
  1. You either notice because you're already tracking releases pretty well and therefore could help Debian with backporting of fixes and/or testing;
  2. or you don't notice because it has slipped through the cracks or there simply are too many potential things to keep track of, in which case the fact that it eventually gets fixed without your intervention is a huge improvement.
Finally, relying too much on Debian packaging does prevent Fedora users (a project that also makes use of Libravatar) from easily contributing mirrors. Though if we had a concrete offer, we would certainly look into creating the appropriate RPMs.

Is it realistic? It turns out that I'm not the only one who thought about this approach, which has been named "debops". The same day that my talk was announced on the DebConf website, someone emailed me saying that he had instituted the exact same rules at his company, which operates a large Django-based web application in the US and Russia. It was pretty impressive to read about a real business coming to the same conclusions and using the same approach (i.e. system libraries, deployment packages) as Libravatar. Regardless of this though, I think there is a class of applications that are particularly well-suited for the approach we've just described. If a web application is not your full-time job and you want to minimize the amount of work required to keep it running, then it's a good investment to restrict your options and leverage the work of the Debian community to simplify your maintenance burden. The second criterion I would look at is framework maturity. Given the 2-3 year release cycle of stable distributions, this approach is more likely to work with a mature framework like Django. After all, you probably wouldn't compile Apache from source, but until recently building Node.js from source was the preferred option as it was changing so quickly. While it goes against conventional wisdom, relying on system libraries is a sustainable approach you should at least consider in your next project. After all, there is a real cost in bundling and keeping up with external dependencies. This blog post is based on a talk I gave at DebConf 14: slides, video.

29 August 2014

Antti-Juhani Kaijanaho: Licentiate Thesis is now publicly available

My recently accepted Licentiate Thesis, which I posted about a couple of days ago, is now available in JyX. Here is the abstract again for reference: Kaijanaho, Antti-Juhani
The extent of empirical evidence that could inform evidence-based design of programming languages. A systematic mapping study.
Jyv skyl : University of Jyv skyl , 2014, 243 p.
(Jyv skyl Licentiate Theses in Computing,
ISSN 1795-9713; 18)
ISBN 978-951-39-5790-2 (nid.)
ISBN 978-951-39-5791-9 (PDF)
Finnish summary Background: Programming language design is not usually informed by empirical studies. In other fields similar problems have inspired an evidence-based paradigm of practice. Central to it are secondary studies summarizing and consolidating the research literature. Aims: This systematic mapping study looks for empirical research that could inform evidence-based design of programming languages. Method: Manual and keyword-based searches were performed, as was a single round of snowballing. There were 2056 potentially relevant publications, of which 180 were selected for inclusion, because they reported empirical evidence on the efficacy of potential design decisions and were published on or before 2012. A thematic synthesis was created. Results: Included studies span four decades, but activity has been sparse until the last five years or so. The form of conditional statements and loops, as well as the choice between static and dynamic typing have all been studied empirically for efficacy in at least five studies each. Error proneness, programming comprehension, and human effort are the most common forms of efficacy studied. Experimenting with programmer participants is the most popular method. Conclusions: There clearly are language design decisions for which empirical evidence regarding efficacy exists; they may be of some use to language designers, and several of them may be ripe for systematic reviewing. There is concern that the lack of interest generated by studies in this topic area until the recent surge of activity may indicate serious issues in their research approach. Keywords: programming languages, programming language design, evidence-based paradigm, efficacy, research methods, systematic mapping study, thematic synthesis

26 August 2014

Simon Josefsson: The Case for Short OpenPGP Key Validity Periods

After I moved to a new OpenPGP key (see key transition statement) I have received comments about the short life length of my new key. When I created the key (see my GnuPG setup) I set it to expire after 100 days. Some people assumed that I would have to create a new key then, and therefore wondered what value there is to sign a key that will expire in two months. It doesn t work like that, and below I will explain how OpenPGP key expiration works; how to extend the expiration time of your key; and argue why having a relatively short validity period can be a good thing.
The OpenPGP message format has a sub-packet called the Key Expiration Time, quoting the RFC:
5.2.3.6. Key Expiration Time
   (4-octet time field)
   The validity period of the key.  This is the number of seconds after
   the key creation time that the key expires.  If this is not present
   or has a value of zero, the key never expires.  This is found only on
   a self-signature.
You can print the sub-packets in your OpenPGP key with gpg --list-packets. See below an output for my key, and notice the created 1403464490 (which is Unix time for 2014-06-22 21:14:50) and the subpkt 9 len 4 (key expires after 100d0h0m) which adds up to an expiration on 2014-09-26. Don t confuse the creation time of the key ( created 1403464321 ) with when the signature was created ( created 1403464490 ).
jas@latte:~$ gpg --export 54265e8c   gpg --list-packets  head -20
:public key packet:
	version 4, algo 1, created 1403464321, expires 0
	pkey[0]: [3744 bits]
	pkey[1]: [17 bits]
:user ID packet: "Simon Josefsson "
:signature packet: algo 1, keyid 0664A76954265E8C
	version 4, created 1403464490, md5len 0, sigclass 0x13
	digest algo 10, begin of digest be 8e
	hashed subpkt 27 len 1 (key flags: 03)
	hashed subpkt 9 len 4 (key expires after 100d0h0m)
	hashed subpkt 11 len 7 (pref-sym-algos: 9 8 7 13 12 11 10)
	hashed subpkt 21 len 4 (pref-hash-algos: 10 9 8 11)
	hashed subpkt 30 len 1 (features: 01)
	hashed subpkt 23 len 1 (key server preferences: 80)
	hashed subpkt 2 len 4 (sig created 2014-06-22)
	hashed subpkt 25 len 1 (primary user ID)
	subpkt 16 len 8 (issuer key ID 0664A76954265E8C)
	data: [3743 bits]
:signature packet: algo 1, keyid EDA21E94B565716F
	version 4, created 1403466403, md5len 0, sigclass 0x10
jas@latte:~$ 
So the key will simply stop being valid after that time? No. It is possible to update the key expiration time value, re-sign the key, and distribute the key to people you communicate with directly or indirectly to OpenPGP keyservers. Since that date is a couple of weeks away, now felt like the perfect opportunity to go through the exercise of taking out my offline master key and boot from a Debian LiveCD and extend its expiry time. See my earlier writeup for LiveCD and USB stick conventions.
user@debian:~$ export GNUPGHOME=/media/FA21-AE97/gnupghome
user@debian:~$ gpg --edit-key 54265e8c
gpg (GnuPG) 1.4.12; Copyright (C) 2012 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Secret key is available.
pub  3744R/54265E8C  created: 2014-06-22  expires: 2014-09-30  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048R/32F8119D  created: 2014-06-22  expires: 2014-09-30  usage: S   
sub  2048R/78ECD86B  created: 2014-06-22  expires: 2014-09-30  usage: E   
sub  2048R/36BA8F9B  created: 2014-06-22  expires: 2014-09-30  usage: A   
[ultimate] (1). Simon Josefsson 
[ultimate] (2)  Simon Josefsson 
gpg> expire
Changing expiration time for the primary key.
Please specify how long the key should be valid.
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 150
Key expires at Fri 23 Jan 2015 02:47:48 PM UTC
Is this correct? (y/N) y
You need a passphrase to unlock the secret key for
user: "Simon Josefsson "
3744-bit RSA key, ID 54265E8C, created 2014-06-22
pub  3744R/54265E8C  created: 2014-06-22  expires: 2015-01-23  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048R/32F8119D  created: 2014-06-22  expires: 2014-09-30  usage: S   
sub  2048R/78ECD86B  created: 2014-06-22  expires: 2014-09-30  usage: E   
sub  2048R/36BA8F9B  created: 2014-06-22  expires: 2014-09-30  usage: A   
[ultimate] (1). Simon Josefsson 
[ultimate] (2)  Simon Josefsson 
gpg> key 1
pub  3744R/54265E8C  created: 2014-06-22  expires: 2015-01-23  usage: SC  
                     trust: ultimate      validity: ultimate
sub* 2048R/32F8119D  created: 2014-06-22  expires: 2014-09-30  usage: S   
sub  2048R/78ECD86B  created: 2014-06-22  expires: 2014-09-30  usage: E   
sub  2048R/36BA8F9B  created: 2014-06-22  expires: 2014-09-30  usage: A   
[ultimate] (1). Simon Josefsson 
[ultimate] (2)  Simon Josefsson 
gpg> expire
Changing expiration time for a subkey.
Please specify how long the key should be valid.
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 150
Key expires at Fri 23 Jan 2015 02:48:05 PM UTC
Is this correct? (y/N) y
You need a passphrase to unlock the secret key for
user: "Simon Josefsson "
3744-bit RSA key, ID 54265E8C, created 2014-06-22
pub  3744R/54265E8C  created: 2014-06-22  expires: 2015-01-23  usage: SC  
                     trust: ultimate      validity: ultimate
sub* 2048R/32F8119D  created: 2014-06-22  expires: 2015-01-23  usage: S   
sub  2048R/78ECD86B  created: 2014-06-22  expires: 2014-09-30  usage: E   
sub  2048R/36BA8F9B  created: 2014-06-22  expires: 2014-09-30  usage: A   
[ultimate] (1). Simon Josefsson 
[ultimate] (2)  Simon Josefsson 
gpg> key 2
pub  3744R/54265E8C  created: 2014-06-22  expires: 2015-01-23  usage: SC  
                     trust: ultimate      validity: ultimate
sub* 2048R/32F8119D  created: 2014-06-22  expires: 2015-01-23  usage: S   
sub* 2048R/78ECD86B  created: 2014-06-22  expires: 2014-09-30  usage: E   
sub  2048R/36BA8F9B  created: 2014-06-22  expires: 2014-09-30  usage: A   
[ultimate] (1). Simon Josefsson 
[ultimate] (2)  Simon Josefsson 
gpg> key 1
pub  3744R/54265E8C  created: 2014-06-22  expires: 2015-01-23  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048R/32F8119D  created: 2014-06-22  expires: 2015-01-23  usage: S   
sub* 2048R/78ECD86B  created: 2014-06-22  expires: 2014-09-30  usage: E   
sub  2048R/36BA8F9B  created: 2014-06-22  expires: 2014-09-30  usage: A   
[ultimate] (1). Simon Josefsson 
[ultimate] (2)  Simon Josefsson 
gpg> expire
Changing expiration time for a subkey.
Please specify how long the key should be valid.
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 150
Key expires at Fri 23 Jan 2015 02:48:14 PM UTC
Is this correct? (y/N) y
You need a passphrase to unlock the secret key for
user: "Simon Josefsson "
3744-bit RSA key, ID 54265E8C, created 2014-06-22
pub  3744R/54265E8C  created: 2014-06-22  expires: 2015-01-23  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048R/32F8119D  created: 2014-06-22  expires: 2015-01-23  usage: S   
sub* 2048R/78ECD86B  created: 2014-06-22  expires: 2015-01-23  usage: E   
sub  2048R/36BA8F9B  created: 2014-06-22  expires: 2014-09-30  usage: A   
[ultimate] (1). Simon Josefsson 
[ultimate] (2)  Simon Josefsson 
gpg> key 3
pub  3744R/54265E8C  created: 2014-06-22  expires: 2015-01-23  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048R/32F8119D  created: 2014-06-22  expires: 2015-01-23  usage: S   
sub* 2048R/78ECD86B  created: 2014-06-22  expires: 2015-01-23  usage: E   
sub* 2048R/36BA8F9B  created: 2014-06-22  expires: 2014-09-30  usage: A   
[ultimate] (1). Simon Josefsson 
[ultimate] (2)  Simon Josefsson 
gpg> key 2
pub  3744R/54265E8C  created: 2014-06-22  expires: 2015-01-23  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048R/32F8119D  created: 2014-06-22  expires: 2015-01-23  usage: S   
sub  2048R/78ECD86B  created: 2014-06-22  expires: 2015-01-23  usage: E   
sub* 2048R/36BA8F9B  created: 2014-06-22  expires: 2014-09-30  usage: A   
[ultimate] (1). Simon Josefsson 
[ultimate] (2)  Simon Josefsson 
gpg> expire
Changing expiration time for a subkey.
Please specify how long the key should be valid.
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 150
Key expires at Fri 23 Jan 2015 02:48:23 PM UTC
Is this correct? (y/N) y
You need a passphrase to unlock the secret key for
user: "Simon Josefsson "
3744-bit RSA key, ID 54265E8C, created 2014-06-22
pub  3744R/54265E8C  created: 2014-06-22  expires: 2015-01-23  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048R/32F8119D  created: 2014-06-22  expires: 2015-01-23  usage: S   
sub  2048R/78ECD86B  created: 2014-06-22  expires: 2015-01-23  usage: E   
sub* 2048R/36BA8F9B  created: 2014-06-22  expires: 2015-01-23  usage: A   
[ultimate] (1). Simon Josefsson 
[ultimate] (2)  Simon Josefsson 
gpg> save
user@debian:~$ gpg -a --export 54265e8c > /media/KINGSTON/updated-key.txt
user@debian:~$ 
I remove the transport USB stick from the offline computer, and back on my laptop I can inspect the new updated key. Let s use the same command as before. The key creation time is the same ( created 1403464321 ), of course, but the signature packet has a new time ( created 1409064478 ) since it was signed now. Notice created 1409064478 and subpkt 9 len 4 (key expires after 214d19h35m) . The expiration time is computed based on when the key was generated, not when the signature packet was generated. You may want to double-check the pref-sym-algos, pref-hash-algos and other sub-packets so that you don t accidentally change anything else. (Btw, re-signing your key is also how you would modify those preferences over time.)
jas@latte:~$ cat /media/KINGSTON/updated-key.txt  gpg --list-packets   head -20
:public key packet:
	version 4, algo 1, created 1403464321, expires 0
	pkey[0]: [3744 bits]
	pkey[1]: [17 bits]
:user ID packet: "Simon Josefsson "
:signature packet: algo 1, keyid 0664A76954265E8C
	version 4, created 1409064478, md5len 0, sigclass 0x13
	digest algo 10, begin of digest 5c b2
	hashed subpkt 27 len 1 (key flags: 03)
	hashed subpkt 11 len 7 (pref-sym-algos: 9 8 7 13 12 11 10)
	hashed subpkt 21 len 4 (pref-hash-algos: 10 9 8 11)
	hashed subpkt 30 len 1 (features: 01)
	hashed subpkt 23 len 1 (key server preferences: 80)
	hashed subpkt 25 len 1 (primary user ID)
	hashed subpkt 2 len 4 (sig created 2014-08-26)
	hashed subpkt 9 len 4 (key expires after 214d19h35m)
	subpkt 16 len 8 (issuer key ID 0664A76954265E8C)
	data: [3744 bits]
:user ID packet: "Simon Josefsson "
:signature packet: algo 1, keyid 0664A76954265E8C
jas@latte:~$ 
Being happy with the new key, I import it and send it to key servers out there.
jas@latte:~$ gpg --import /media/KINGSTON/updated-key.txt 
gpg: key 54265E8C: "Simon Josefsson " 5 new signatures
gpg: Total number processed: 1
gpg:         new signatures: 5
jas@latte:~$ gpg --send-keys 54265e8c
gpg: sending key 54265E8C to hkp server keys.gnupg.net
jas@latte:~$ gpg --keyserver keyring.debian.org  --send-keys 54265e8c
gpg: sending key 54265E8C to hkp server keyring.debian.org
jas@latte:~$ 
Finally: why go through this hassle, rather than set the key to expire in 50 years? Some reasons for this are:
  1. I don t trust myselt to keep track of a private key (or revocation cert) for 50 years.
  2. I want people to notice my revocation certificate as quickly as possible.
  3. I want people to notice other changes to my key (e.g., cipher preferences) as quickly as possible.
Let s look into the first reason a bit more. What would happen if I lose both the master key and the revocation cert, for a key that s valid 50 years? I would start from scratch and create a new key that I upload to keyservers. Then there would be two keys out there that are valid and identify me, and both will have a set of signatures on it. None of them will be revoked. If I happen to lose the new key again, there will be three valid keys out there with signatures on it. You may argue that this shouldn t be a problem, and that nobody should use any other key than the latest one I want to be used, but that s a technical argument and at this point we have moved into usability, and that s a trickier area. Having users select which out of a couple of apparently all valid keys that exist for me is simply not going to work well. The second is more subtle, but considerably more important. If people retrieve my key from keyservers today, and it expires in 50 years, there will be no need to refresh it from key servers. If for some reason I have to publish my revocation certificate, there will be people that won t see it. If instead I set a short validity period, people will have to refresh my key once in a while, and will then either get an updated expiration time, or will get the revocation certificate. This amounts to a CRL/OCSP-like model. The third is similar to the second, but deserves to be mentioned on its own. Because the cipher preferences are expressed (and signed) in my key, and that ciphers come and go, I would expect that I will modify those during the life-time of my long-term key. If I have a long validity period of my key, people would not refresh it from key servers, and would encrypt messages to me with ciphers I may no longer want to be used. The downside of having a short validity period is that I have to do slightly more work to get out the offline master key once in a while (which I have to once in a while anyway because I m signing other peoples keys) and that others need to refresh my key from the key servers. Can anyone identify other disadvantages? Also, having to explain why I m using a short validity period used to be a downside, but with this writeup posted that won t be the case any more. :-) flattr this!

Daniel Pocock: GSoC talks at DebConf 14 today

This year I mentored two students doing work in support of Debian and free software (as well as those I mentored for Ganglia). Both of them are presenting details about their work at DebConf 14 today. While Juliana's work has been widely publicised already, mainly due to the fact it is accessible to every individual DD, Andrew's work is also quite significant and creates many possibilities to advance awareness of free software. The Java project that is not just about Java Andrew's project is about recursively building Java dependencies from third party repositories such as the Maven Central Repository. It matches up well with the wonderful new maven-debian-helper tool in Debian and will help us to fill out /usr/share/maven-repo on every Debian system. Firstly, this is not just about Java. On a practical level, some aspects of the project are useful for many other purposes. One of those is the aim of scanning a repository for non-free artifacts, making a Git mirror or clone containing a dfsg branch for generating repackaged upstream source and then testing to see if it still builds. Then there is the principle of software freedom. The Maven Central repository now requires that people publish a sources JAR and license metadata with each binary artifact they upload. They do not, however, demand that the sources JAR be complete or that the binary can be built by somebody else using the published sources. The license data must be specified but it does not appeared to be verified in the same way as packages inspected by Debian's legendary FTP masters. Thanks to the transitive dependency magic of Maven, it is quite possible that many Java applications that are officially promoted as free software can't trace the source code of every dependency or build plugin. Many organizations are starting to become more alarmed about the risk that they are dependent upon some rogue dependency. Maybe they will be hit with a lawsuit from a vendor stating that his plugin was only free for the first 3 months. Maybe some binary dependency JAR contains a nasty trojan for harvesting data about their corporate network. People familiar with the principles of software freedom are in the perfect position to address these concerns and Andrew's work helps us build a cleaner alternative. It obviously can't rebuild every JAR for the very reason that some of them are not really free - however, it does give the opportunity to build a heat-map of trouble spots and also create a fast track to packaging for those heirarchies of JARs that are truly free. Making WebRTC accessible to more people Juliana set out to update rtc.debian.org and this involved working on JSCommunicator, the HTML5/JavaScript softphone based on WebRTC. People attending the session today or participating remotely are advised to set up your RTC / VoIP password at db.debian.org well in advance so the server will allow you to log in and try it during the session. It can take 30 minutes or so for the passwords to be replicated to the SIP proxy and TURN server. Please also check my previous comments about what works and what doesn't and in particular, please be aware that Iceweasel / Firefox 24 on wheezy is not suitable unless you are on the same LAN as the person you are calling.

25 August 2014

Erich Schubert: Analyzing Twitter - beware of spam

This year I started to widen up my research; and one data source of interest was text because of the lack of structure in it, that makes it often challenging. One of the data sources that everybody seems to use is Twitter: it has a nice API, and few restrictions on using it (except on resharing data). By default, you can get a 1% random sample from all tweets, which is more than enough for many use cases.
We've had some exciting results which a colleague of mine will be presenting tomorrow (Tuesday, Research 22: Topic Modeling) at the KDD 2014 conference:
SigniTrend: Scalable Detection of Emerging Topics in Textual Streams by Hashed Significance Thresholds
Erich Schubert, Michael Weiler, Hans-Peter Kriegel
20th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
You can also explore some (static!) results online at signi-trend.appspot.com

In our experiments, the "news" data set was more interesting. But after some work, we were able to get reasonable results out of Twitter as well. As you can see from the online demo, most of these fall into pop culture: celebrity deaths, sports, hip-hop. Not much that would change our live; and even less that wasn't before captured by traditional media.
The focus of this post is on the preprocessing needed for getting good results from Twitter. Because it is much easier to get bad results!
The first thing you need to realize about Twitter is that due to the media attention/hype it gets, it is full of spam. I'm pretty sure the engineers at Twitter already try to reduce spam; block hosts and fraud apps. But a lot of the trending topics we discovered were nothing but spam.
Retweets - the "like" of Twitter - are an easy source to see what is popular, but are not very interesting if you want to analyze text. They just reiterate the exact same text (except for a "RT " prefix) than earlier tweets. We found results to be more interesting if we removed retweets. Our theory is that retweeting requires much less effort than writing a real tweet; and things that are trending "with effort" are more interesting than those that were just liked.
Teenie spam. If you ever searched for a teenie idol on Twitter, say this guy I hadn't heard of before, but who has 3.88 million followers on Twitter, and search for Tweets addressed to him, you will get millions over millions of results. Many of these tweets look like this: Now if you look at this tweet, there is this odd "x804" at the end. This is to defeat a simple spam filter by Twitter. Because this user did not tweet this just once: instead it is common amongst teenie to spam their idols with follow requests by the dozen. Probably using some JavaScript hack, or third party Twitter client. Occassionally, you see hundreds of such tweets, each sent within a few seconds of the previous one. If you get a 1% sample of these, you still get a few then...
Even worse (for data analysis) than teenie spammers are commercial spammers and wannabe "hackers" that exercise their "sk1llz" by spamming Twitter. To get a sample of such spam, just search for weight loss on Twitter. There is plenty of fresh spam there, usually consisting of some text pretending to be news, and an anonymized link (there is no need to use an URL shortener such as bit.ly on Twitter, since Twitter has its own URL shortener t.co; and you'll end up with double-shortened URLs). And the hacker spam is even worse (e.g. #alvianbencifa) as he seems to have trojaned hundreds of users, and his advertisement seems to be a nonexistant hash tag, which he tries to get into Twitters "trending topics".
And then there are the bots. Plenty of bots spam Twitter with their analysis of trending topics, reinforcing the trending topics. In my opinion, bots such as trending topics indonesia are useless. No wonder there are only 280 followers. And of the trending topics reported, most of them seem to be spam topics...

Bottom line: if you plan on analyzing Twitter data, spend considerable time on preprocessing to filter out spam of various kind. For example, we remove singletons and digits, then feed the data through a duplicate detector. We end up discarding 20%-25% of Tweets. But we still get some of the spam, such as that hackers spam.
All in all, real data is just messy. People posting there have an agenda that might be opposite to yours. And if someone (or some company) promises you wonders from "Big data" and "Twitter", you better have them demonstrate their use first, before buying their services. Don't trust visions of what could be possible, because the first rule of data analysis is: Garbage in, garbage out.

Next.

Previous.