Support appending arbitrary text in the Compose textarea to the subject via a context menu. (#4)
Avoid brittle parsing of the page and use the Javascript API instead. (diff)
Updated my Strava Enhancement Suite, a Chrome extension that improves and fixes annoyances in the web interface of the Strava cycling and running tracker:
Merge support from Jeff Schuler to support "Flyby" activity defaults. (#50)
Correctly fallback to the default value after introducing a new value. (diff)
Started work on packaging lute-tab, but upstream was unwilling to relicense to a DFSG-friendly license so the enterprise was aborted.
Updated the SSL certificate for try.diffoscope.org, a hosted version of the diffoscope in-depth and content-aware diff utility. Thanks to Bytemark for sponsoring the hardware.
Started on the slides for the Reproducible Builds talk at DebConf16.
A while back, I saw a Kickstarter
for one of the most well designed and pretty sleep trackers on the market. I
fell in love with it, and it has stuck with me since.
A few months ago, I finally got my hands on one and started to track my data.
Naturally, I now want to store this new data with the rest of the data I have
on myself in my own databases.
I went in search of an API, but I found that the Sense API hasn't been published
yet, and is being worked on by the team. Here's hoping it'll land soon!
After some subdomain guessing, I hit on api.hello.is.
So, naturally, I went to take a quick look at their Android app and network
traffic, lo and behold, there was a pretty nicely designed API.
This API is clearly an internal API, and as such, it's something that
should not be considered stable. However, I'm OK with a fragile API,
so I've published a quick and dirty API wrapper for the Sense API
to my GitHub..
I've published it because I've found it useful, but I can't promise the world,
(since I'm not a member of the Sense team at Hello!), so here are a few ground
rules of this wrapper:
I make no claims to the stability or completeness.
I have no documentation or assurances.
I will not provide the client secret and ID. You'll have to find them on
your own.
This may stop working without any notice, and there may even be really nasty
bugs that result in your alarm going off at 4 AM.
Send PRs! This is a side-project for me.
This module is currently Python 3 only. If someone really needs Python 2
support, I'm open to minimally invasive patches to the codebase using
six to support Python 2.7.
Working with the API:
First, let's go ahead and log in using python -m sense.
$ python -m sense
Sense OAuth Client ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sense OAuth Client Secret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sense email: paultag@gmail.com
Sense password:
Attempting to log into Sense's APISuccess!Attempting to query the Sense APIThe humidity is **just right**.The air quality is **just right**.The light level is **just right**.It's **pretty hot** in here.
The noise level is **just right**.
Success!
Now, let's see if we can pull up information on my Sense:
>>>fromsenseimportSense>>>sense=Sense()>>>sense.devices()'senses':[ 'id':'xxxxxxxxxxxxxxxx','firmware_version':'11a1','last_updated':1466991060000,'state':'NORMAL','wifi_info':'rssi':0,'ssid':'Pretty Fly for a WiFi (2.4 GhZ)','condition':'GOOD','last_updated':1462927722000 ,'color':'BLACK' ],'pills':[ 'id':'xxxxxxxxxxxxxxxx','firmware_version':'2','last_updated':1466990339000,'battery_level':87,'color':'BLUE','state':'NORMAL' ]
Neat! Pretty cool. Look, you can even see my WiFi AP! Let's try some more
and pull some trends out.
I plan to keep maintaining it as long as it's needed, so I welcome
co-maintainers, and I'd love to see what people build with it! So far, I'm
using it to dump my room data into InfluxDB, pulling information on my room
into Grafana. Hopefully more to come!
Happy hacking!
I've just released version 1.3.0 of Nageru,
my live software video mixer.
Things have been a bit quiet on the Nageru front recently, for two reasons:
First, I've been busy with moving (from Switzerland to Norway) and associated
job change (from Google to MySQL/Oracle). Things are going well, but these
kinds of changes tend to take, well, time and energy.
Second, the highlight of Nageru
1.3.0 is encoding of H.264 streams meant for end users (using x264), not just
the Quick Sync Video streams from earlier versions, which work more as a
near-lossless intermediate format meant for transcoding to something else
later. Like with most things video, hitting such features really hard (I've
been doing literally weeks of continuous stream testing) tends to expose
weaknesses in upstream software.
In particular, I wanted x264 speed control,
where the quality is tuned up and down live as the content dictates. This
is mainly because the content I want to stream this summer (demoscene
competitions) varies from the very simple
to downright ridiculously complex
(as you can see, YouTube just basically gives up and creates gray blocks).
If you have only one static quality setting, you will have the choice between
something that looks like crap for everything, and one that drops frames like
crazy (or, if your encoding software isn't all that, like e.g. using
ffmpeg(1) directly, just gets behind and all your clients' streams just stop)
when the tricky stuff comes. There was an unofficial patch for speed control,
but it was buggy, not suitable for today's hardware and not kept at all up
to date with modern x264 versions. So to get speed control, I had to work
that patch pretty heavily (including making it so that it could work in
Nageru directly instead of requiring a patched x264) and then it exposed a
bug in x264 proper that would cause corruption when changing between some
presets, and I couldn't release 1.3.0 before that fix had at least hit git.
Similarly, debugging this exposed an issue with how I did streaming with
ffmpeg and the MP4 mux (which you need to be able to stream H.264 directly to
HTML5 <video> without any funny and latency-inducing segmenting
business); to know where keyframes started, I needed to flush the mux before
each one, but this messes up interleaving, and if frames were ever dropped right in front of a keyframe
(which they would on the most difficult content, even at speed control's
fastest presets!), the duration field of the frame would be wrong, causing
the timestamps to be wrong and even having pts < dts in some cases.
(VLC has to deal with flushing in exactly the same way, and thus would have
exactly the same issue, although VLC generally doesn't transcode
variable-framerate content so well to begin with, so the heuristics would
be more likely to work. Incidentally, I wrote the VLC code for this flushing
back in the day, to be able to stream WebM for some Debconf.) I cannot take
credit for the ffmpeg/libav fixes (that was all done by Martin Storsj ),
but again, Nageru had to wait for the new API they introduce (that just
signals to the application when a keyframe is about to begin, removing the
need for flushing) to get into git mainline. Hopefully, both fixes will get
into releases soon-ish and from there one make their way into stretch.
Apart from that, there's a bunch of fixes as always. I'm still occasionally
(about once every two weeks of streaming or so) hitting what I believe is
a bug in NVIDIA's proprietary OpenGL drivers, but it's nearly impossible to
debug without some serious help from them, and they haven't been responding
to my inquiries. Every two weeks means that you could be hitting it in
a weekend's worth of streaming, so it would be nice to get it fixed, but
it also means it's really really hard to make a reproducible test case. :-)
But the fact that this is currently the worst stability bug (and that you
can work around it by using e.g. Intel's drivers) also shows that Nageru is
pretty stable these days.
Here comes my journey till the mid-term (June 21st) as a blog to share my experience.
I have previously worked on some social related projects such as smart guidance for blind and sensor based wireless controller . I have been selected as a speaker for FOSSASIA-16 (Asia s premier technological event)to talk on the project smart guidance for blind .FOSSASIA speakers. It was a great experience participating in the technological event in Singapore science centre.Got an opportunity to meet open source contributors from all over the world(even though it is an Asian event, participation was all over from the world). There were pre-meetups for FOSSASIA on the day before three-day event.I have attended the one organised by RedHat, Singapore.Discussed on many topics related to open source.
Three days of FOSSASIA event was a great experience.It was the second time as a speaker in an international conference.My talk was on the second day.Sharing is the best way to increase your knowledge. Talks and workshops were brainstorming.Learnt many new things and got the courage to contribute to the open source.Met Daniel Pocock in Debian exhibition table.Meeting awesome people can be the turning point of life.Had a discussion about the Debian projects and it motivated me for open source software.We have discussed about the Real Time Communication and was encouraged to apply for GSoC (Google Summer of Code). As per our discussion, prepared the project proposal on improving voice,video and chat communication with free software and submitted it for GSoC. I have been selected to contribute for Debian with stipend from Google.
This was my first application for GSoC and I have been selected to contribute for open source and free software. I would like to thank Google and Debian for giving this amazing experience.
Learning and coding have begun.Updated my laptop with Jessie, latest version of Debian.Get acquainted with the new platform.Got to learn many things about Real Time Communication.Learnt more about SIP, XMPP, peer-to-peer technology to work on my project.It s always better to be clear with theory before coding.When it comes to voice and video over IP, most people nowadays are quick to use Skype, Whatsapp, or Viber. My main goals of the project are helping people to avoid using proprietary communications tools like Skype, Viber and WhatsApp and simplifying the setup of free alternatives like Jitsi, Linphone, Ekiga, Tox (qtox), Mumble.Downloaded some of the already available open source VoIP to find the problems behind it and improve it further.Bootstrapping any business relevant network based on these free alternatives is still hard.
Would you like to list the senders, receivers and date of the messages in the inbox of your mail.Python has a library file IMAP which can be used to connect to an email account, examine every message in every folder and look at the To , From and CC headers of every email message in the folder.
Do you have phone numbers and other contact details in old emails? Would you like a quick way to data-mine your inbox to find them and help migrate them to your address book? Got the help from phonenumbers library for parsing, formatting, and validating international phone numbers.I would like to share how I imported this library file into my coding.Download the given library file and open the file in the terminal.Type
$ python setup.py install
to install the library file.Now you can call the functions by importing phonenumbers.
You can go through the code in my GitHub profile here.(Recently started committing my projects in GitHub)
Iain R. Learmonth joined my journey as a mentor.Helped in solving some issues in my coding through GitHub.
It was a wonderful journey till now.Will be working further to improve voice, video and chat communication with free software.Stay connected to know more about my further journey through GSoC.
ikiwiki-hosting is a spin-off from ikiwiki. I wrote it to manage
many ikiwiki instances for Branchable,
and made it free software out of principle.
While Branchable has not reached the point of providing much income, it's
still running after 6 years. Ikiwiki-hosting makes it pretty easy to
maintain it, and I host all of my websites there.
A couple of other people have also found ikiwiki-hosting useful, which is
not only nice, but led to some big improvements to it. Mostly though,
releasing the software behind the business as free software caused us to
avoid shortcuts and build things well.
Next: twenty years of free software -- part 5 pristine-tar
Contribute constructively - no one likes to be told "You've got a REALLY ugly baby there" or equivalent.
Think through what you post: check references and check that it reads clearly and is spelled correctly
Add value
When contributing bug reports:
Provide as full details of hardware and software as you have
Answer questions carefully: Ask questions the smart way: http://www.catb.org/esr/faqs/smart-questions.html
Be prepared to follow up queries / provide sufficient evidence to reproduce behaviour or provide pathological test cases
Provide a patch if possible: even if it's only pseudocode
When adding to / modifying FLOSS software:
Keep pristine sources that you have downloaded
Maintain patch series against pristine source
Talk to the originators of the software / current maintainers elsewhere
Follow upstream style if feasible / a consistent house style if not
Be generous in what you accept: be precise in what you put out
Don't produce licence conflicts - check and check again that your software can be distributed.
Don't apply inconsistent copyrights
When writing new FLOSS software / "freeing" prior commercial/closed code under a FLOSS licence
Make permissions explicit and publish under a well established FLOSS licence
Be generous to potential contributors and collaborators: render them every assistance so that they can help you better
Be generous in what you accept: be precise in what you put out
Don't produce licence conflicts - check and check again that your software can be distributed.
Don't apply inconsistent copyrights: software you write is your copyright at the outset until you assign it elsewhere
Contribute documentation / examples
Maintain a bugtracker and mailing lists for your software
If you are required to sign a contributor license agreement [CLA]
Ensure that you have the rights you purport to assign
Assign the minimum of rights necessary - if you can continue to allow full and free use of your code, do so
Meet any required code of conduct [CoC] stipulations in addition to the CLA
Always remember in all of this: just because you understand your code and your working practices doesn't mean that anyone else will. There is no automatic right to contribution nor any necessary assumption or precondition that collaborators will come forward. Just because you love your own code doesn't mean that it merits anyone else's interest or that anyone else should value it thereby "Just because it scratches your itch doesn't mean that it scratches anyone else's - or that it's actually any good / any use to anyone else"
My work colleagues know me well as a Free/Libre software zealot, constantly pointing out to them how people should behave, how FLOSS software trumps commercial software and how this is the only way forward. This for the last 20 odd years. It's a strain to argue this repeatedly: at various times, I have been asked to set out more clearly why I use FLOSS, what the advantages are, why and how to contribute to FLOSS software.
"We are creating a world that all may enter without privilege or prejudice accorded by race, economic power, military force, or station of birth. We are creating a world where anyone, anywhere may express his or her beliefs, no matter how singular, without fear of being coerced into silence or conformity. Your legal concepts of property, expression, identity, movement, and context do not apply to us. They are all based on matter, and there is no matter here ... In our world, whatever the human mind may create can be reproduced and distributed infinitely at no cost. The global conveyance of thought no longer requires your factories to accomplish." [John Perry Barlow - Declaration of the independence of cyberspace 1996 https://www.eff.org/cyberspace-independence]
That's some of it right there: I was seduced by a modem and the opportunities it gave. I've lived in this world since 1994, come to appreciate it and never really had the occasion to regret it.
I'm involved in the Debian community - which is very much a "do-ocracy" - and I've lived with Debian GNU Linux since 1995 and not had much cause to regret that either, though I do regret that force of circumstance has meant that I can't contribute as much as I'd like. Pretty much every machine I touch ends up running Debian, one way or the other, or should do if I had my way. Digging through my emails since then on the various mailing lists - some of them are deeply technical, though fewer these days: some are Debian political: most are trying to help people with problems / report successes or, occasionally thanks and social chit chat. Most people in the project have never met me - though that's not unusual in an organisation with a thousand developers spread worldwide - and so the occasional chance to talk to people in real life is invaluable.
The crucial thing is that there is common purpose and common intelligence - however crazy mailing list flame wars can get sometimes - and committed, caring people. Some of us may be crazy zealots, some picky and argumentative - Debian is what we have in common, pretty much.
It doesn't depend on physical ability. Espy (Joel Klecker) was one of our best and brightest until his death at age 21: almost nobody knew he was dying until after his death. My own physical limitations are pretty much irrelevant provided I can type.
It does depend on collaboration and the strange, dysfunctional family that is our community and the wider FLOSS community in which we share and in which some of us have multiple identities in working with different projects. This is going to end up too long for Planet Debian - I'll end this post here and then continue with some points on how to contribute and why employers should let their employers work on FLOSS.
In a prior post,
I gave an overview of the OpenStack CI system and how jobs were
started. In that I said
(It is a gross oversimplification, but for the purposes of OpenStack
CI, Jenkins is pretty much used as a glorified ssh/scp wrapper. Zuul
Version 3, under development, is working to remove the need for
Jenkins to be involved at all).
Well some recent security issues with Jenkins and other changes has
led to a roll-out of what is being called Zuul 2.5, which has indeed
removed Jenkins and makes extensive use of Ansible as the basis for
running CI tests in OpenStack. Since I already had the diagram, it
seems worth updating it for the new reality.
OpenStack CI Overview
While previous post was really focused on the image-building
components of the OpenStack CI system, overview is the same but more
focused on the launchers that run the tests.
The process starts when a developer uploads their code to
gerrit via the git-review tool. There is no further action
required on their behalf and the developer simply waits for
results of their jobs.
Gerrit provides a JSON-encoded "fire-hose" output of everything
happening to it. New reviews, votes, updates and more all get sent
out over this pipe. Zuul is the overall scheduler
that subscribes itself to this information and is responsible for
managing the CI jobs appropriate for each change.
Zuul has a configuration
that tells it what jobs to run for what projects. Zuul can do lots
of interesting things, but for the purposes of this discussion we
just consider that it puts the jobs it wants run into gearman
for a launcher to consume. gearman is a
job-server; as they explain it "[gearman] provides a generic
application framework to farm out work to other machines or
processes that are better suited to do the work". Zuul puts into
gearman basically a tuple (job-name,node-type) for each
job it wants run, specifying the unique job name to run and what
type of node it should be run on.
A group of Zuul launchers
are subscribed to gearman as workers. It is these Zuul
launchers that will consume the job requests from the queue and
actually get the tests running. However, a launcher needs two
things to be able to run a job a job definition (what to actually
do) and a worker node (somewhere to do it).
The first part what to do is provided by job-definitions
stored in external YAML files. The Zuul launcher knows how to
process these files (with some help from Jenkins Job Builder, which
despite the name is not outputting XML files for Jenkins to
consume, but is being used to help parse templates and macros
within the generically defined job definitions). Each Zuul
launcher gets these definitions pushed to it constantly by Puppet,
thus each launcher knows about all the jobs it can run
automatically. Of course Zuul also knows about these same job
definitions; this is the job-name part of the tuple we said it
put into gearman.
The second part somewhere to run the test takes some more
explaining. To the next point...
Several cloud companies donate capacity in their clouds for
OpenStack to run CI tests. Overall, this capacity is managed by a
customized management tool called nodepool
(you can see the details of this capacity at any given time by
checking the nodepool configuration).
Nodepool watches the gearman queue and sees what requests are
coming out of Zuul. It looks at node-type of jobs in the queue
(i.e. what platform the job has requested to run on) and decides
what types of nodes need to start and which cloud providers have
capacity to satisfy demand.
Nodepool will start fresh virtual machines (from images built daily
as described in the prior post), monitor their start-up and, when
they're ready, put a new "assignment job" back into gearman with
the details of the fresh node. One of the active Zuul launchers
will pick up this assignment job and register the new node to
itself.
At this point, the Zuul launcher has what it needs to actually get
jobs started. With an fresh node registered to it and waiting for
something to do, the Zuul launcher can advertise its ability to
consume one of the waiting jobs from the gearman queue. For
example, if a ubuntu-trusty node is provided to the Zuul
launcher, the launcher can now consume from gearman any job it
knows about that is intended to run on an ubuntu-trusty node
type. If you're looking at the launcher code
this is driven by the NodeWorker class you can see this being
created in response to an assignment via
LaunchServer.assignNode.
To actually run the job where the "job hits the metal" as it were
the Zuul launcher will dynamically construct an Ansible playbook to run. This
playbook is a concatenation of common setup and teardown operations
along with the actual test scripts the jobs wants to run. Using
Ansible to run the job means all the flexibility an orchestration
tool provides is now available to the launcher. For example, there
is a custom console streamer
library that allows us to live-stream the console output for the
job over a plain TCP connection, and there is the possibility to
use projects like ARA
for visualisation of CI runs. In the future, Ansible will allow
for better coordination when running multiple-node testing jobs
after all, this is what orchestration tools such as Ansible are
made for! While the Ansible run can be fairly heavyweight
(especially when you're talking about launching thousands of jobs
an hour), the system scales horizontally with more launchers able
to consume more work easily.
When checking your job results on logs.openstack.org you will
see a _zuul_ansible directory now which contains copies of the
inventory, playbooks and other related files that the launcher used
to do the test run.
Eventually, the test will finish. The Zuul launcher will put the
result back into gearman, which Zuul will consume (log copying
is interesting but a topic for another day). The testing node will
be released back to nodepool, which destroys it and starts all over
again nodes are not reused and also have no sensitive details on
them, as they are essentially publicly accessible. Zuul will wait
for the results of all jobs for the change and post the result back
to Gerrit; it either gives a positive vote or the dreaded negative
vote if required jobs failed (it also handles merges to git, but
that is also a topic for another day).
Work will continue within OpenStack Infrastructure to further enhance
Zuul; including better support for multi-node jobs and "in-project"
job definitions (similar to the https://travis-ci.org/ model);
for full details see the spec.
This year a significant number of students are working on RTC-related projects as part of Google Summer of Code, under the umbrella of the Debian Project. You may have already encountered some of them blogging on Planet or participating in mailing lists and IRC.
WebRTC plugins for popular CMS and web frameworks
There are already a range of pseudo-WebRTC plugins available for CMS and blogging platforms like WordPress, unfortunately, many of them are either not releasing all their source code, locking users into their own servers or requiring the users to download potentially untrustworthy browser plugins (also without any source code) to use them.
Mesut is making plugins for genuinely free WebRTC with open standards like SIP. He has recently created the WPCall plugin for WordPress, based on the highly successful DruCall plugin for WebRTC in Drupal.
Keerthana has started creating a similar plugin for MediaWiki.
What is great about these plugins is that they don't require any browser plugins and they work with any server-side SIP infrastructure that you choose. Whether you are routing calls into a call center or simply using them on a personal blog, they are quick and convenient to install. Hopefully they will be made available as packages, like the DruCall packages for Debian and Ubuntu, enabling even faster installation with all dependencies.
Would you like to try running these plugins yourself and provide feedback to the students? Would you like to help deploy them for online communities using Drupal, WordPress or MediaWiki to power their web sites? Please come and discuss them with us in the Free-RTC mailing list.
You can read more about how to run your own SIP proxy for WebRTC in the RTC Quick Start Guide.
Finding all the phone numbers and ham radio callsigns in old emails
Do you have phone numbers and other contact details such as ham radio callsigns in old emails? Would you like a quick way to data-mine your inbox to find them and help migrate them to your address book?
Jaminy is working on Python scripts to do just that. Her project takes some inspiration from the Telify plugin for Firefox, which detects phone numbers in web pages and converts them to hyperlinks for click-to-dial. The popular libphonenumber from Google, used to format numbers on Android phones, is being used to help normalize any numbers found. If you would like to test the code against your own mailbox and address book, please make contact in the #debian-data channel on IRC.
A truly peer-to-peer alternative to SIP, XMPP and WebRTC
The team at Savoir Faire Linux has been busy building the Ring softphone, a truly peer-to-peer solution based on the OpenDHT distribution hash table technology.
Several students (Simon, Olivier, Nicolas and Alok) are actively collaborating on this project, some of them have been fortunate enough to participate at SFL's offices in Montreal, Canada. These GSoC projects have also provided a great opportunity to raise Debian's profile in Montreal ahead of DebConf17 next year.
Linux Desktop Telepathy framework and reSIProcate
Another group of students, Mateus, Udit and Balram have been busy working on C++ projects involving the Telepathy framework and the reSIProcate SIP stack. Telepathy is the framework behind popular softphones such as GNOME Empathy that are installed by default on the GNU/Linux desktop.
I previously wrote about starting a new SIP-based connection manager for Telepathy based on reSIProcate. Using reSIProcate means more comprehensive support for all the features of SIP, better NAT traversal, IPv6 support, NAPTR support and TLS support. The combined impact of all these features is much greater connectivity and much greater convenience.
The students are extending that work, completing the buddy list functionality, improving error handling and looking at interaction with XMPP.
Streamlining provisioning of SIP accounts
Currently there is some manual effort for each user to take the SIP account settings from their Internet Telephony Service Provider (ITSP) and transpose these into the account settings required by their softphone.
Pranav has been working to close that gap, creating a JAR that can be embedded in Java softphones such as Jitsi, Lumicall and CSipSimple to automate as much of the provisioning process as possible. ITSPs are encouraged to test this client against their services and will be able to add details specific to their service through Github pull requests.
The project also hopes to provide streamlined provisioning mechanisms for privately operated SIP PBXes, such as the Asterisk and FreeSWITCH servers used in small businesses.
Improving SIP support in Apache Camel and the Jitsi softphone
Apache Camel's SIP component and the widely known Jitsi softphone both use the JAIN SIP library for Java.
Nik has been looking at issues faced by SIP users in both projects, adding support for the MESSAGE method in camel-sip and looking at why users sometimes see multiple password prompts for SIP accounts in Jitsi.
If you are trying either of these projects, you are very welcome to come and discuss them on the mailing lists, Camel users and Jitsi users.
GSoC students at DebConf16 and DebConf17 and other events
Many of us have been lucky to meet GSoC students attending DebConf, FOSDEM and other events in the past. From this year, Google now expects the students to complete GSoC before they become eligible for any travel assistance. Some of the students will still be at DebConf16 next month, assisted by the regular travel budget and the diversity funding initiative. Nik and Mesut were already able to travel to Vienna for the recent MiniDebConf / LinuxWochen.at
As mentioned earlier, several of the students and the mentors at Savoir Faire Linux are based in Montreal, Canada, the destination for DebConf17 next year and it is great to see the momentum already building for an event that promises to be very big.
Explore the world of Free Real-Time Communications (RTC)
If you are interesting in knowing more about the Free RTC topic, you may find the following resources helpful:
Try Lumicall and Ring apps for Android with your freinds
RTC mentoring team 2016
We have been very fortunate to build a large team of mentors around the RTC-themed projects for 2016. Many of them are first time GSoC mentors and/or new to the Debian community. Some have successfully completed GSoC as students in the past. Each of them brings unique experience and leadership in their domain.
Helping GSoC projects in 2016 and beyond
Not everybody wants to commit to being a dedicated mentor for a GSoC student. In fact, there are many ways to help without being a mentor and many benefits of doing so.
Simply looking out for potential applicants for future rounds of GSoC and referring them to the debian-outreach mailing list or an existing mentor helps ensure we can identify talented students early and design projects around their capabilities and interests.
Testing the projects on an ad-hoc basis, greeting the students at DebConf and reading over the student wikis to find out where they are and introduce them to other developers in their area are all possible ways to help the projects succeed and foster long term engagement.
Google gives Debian a USD $500 grant for each student who completes a project successfully this year. If all 2016 students pass, that is over $10,000 to support Debian's mission.
I came across Debian Backports recently after a kind soul suggested it in my Jessie Installation article. Backports provide the bridge between two stable releases of Debian. Say you are running Jessie stable in your system now. While it runs great, Jessie stable will be a bit outdated compared to the testing release. This is solved by Backports which recompiles and adjusts these testing packages for the stable environment. Backports can be included into the source file next to the stable sources unlike the testing sources which can misbehave and have a significant risk, Backports are much more compatible and lower risk.
However, Backports cannot be tested as extensively as Debian stable, and backports are provided on an as-is basis, with risk of incompatibilities with other components in Debian stable. Use with care! It is therefore recommended to only select single backported packages that fit your needs, and not use all available backports.
To add Backports to your system,
Open the sources.list file : sudo nano /etc/apt/sources.list
For jessie add the line : deb http://ftp.debian.org/debian jessie-backports main
For wheezy add the line : deb http://ftp.debian.org/debian wheezy-backports main
After any change be sure to run sudo apt-get update
To know more about Backports and to contribute consult the main page here and the wiki here. Happy hacking!
As some of the world knows full well by now, I've been noodling with Go
for a few years, working through its pros, its cons, and thinking a lot
about how humans use code to express thoughts and ideas. Go's got a lot of
neat use cases, suited to particular problems, and used in the right place,
you can see some clear massive wins.
I've started writing Debian tooling in Go, because it's a pretty natural fit.
Go's fairly tight, and overhead shouldn't be taken up by your operating system.
After a while, I wound up hitting the usual blockers, and started to build up
abstractions. They became pretty darn useful, so, this blog post is announcing
(a still incomplete, year old and perhaps API changing) Debian package for Go.
The Go importable name is pault.ag/go/debian. This contains a lot of utilities
for dealing with Debian packages, and will become an edited down "toolbelt"
for working with or on Debian packages.
Module Overview
Currently, the package contains 4 major sub packages. They're a changelog
parser, a control file parser, deb file format parser, dependency parser
and a version parser. Together, these are a set of powerful building blocks
which can be used together to create higher order systems with reliable
understandings of the world.
changelog
The first (and perhaps most incomplete and least tested) is a changelog file
parser.. This provides the
programmer with the ability to pull out the suite being targeted in the
changelog, when each upload was, and the version for each. For example, let's
look at how we can pull when all the uploads of Docker to sid took place:
funcmain()resp,err:=http.Get("http://metadata.ftp-master.debian.org/changelogs/main/d/docker.io/unstable_changelog")iferr!=nilpanic(err)allEntries,err:=changelog.Parse(resp.Body)iferr!=nilpanic(err)for_,entry:=rangeallEntriesfmt.Printf("Version %s was uploaded on %s\n",entry.Version,entry.When)
The output of which looks like:
Version 1.8.3~ds1-2 was uploaded on 2015-11-04 00:09:02 -0800 -0800
Version 1.8.3~ds1-1 was uploaded on 2015-10-29 19:40:51 -0700 -0700
Version 1.8.2~ds1-2 was uploaded on 2015-10-29 07:23:10 -0700 -0700
Version 1.8.2~ds1-1 was uploaded on 2015-10-28 14:21:00 -0700 -0700
Version 1.7.1~dfsg1-1 was uploaded on 2015-08-26 10:13:48 -0700 -0700
Version 1.6.2~dfsg1-2 was uploaded on 2015-07-01 07:45:19 -0600 -0600
Version 1.6.2~dfsg1-1 was uploaded on 2015-05-21 00:47:43 -0600 -0600
Version 1.6.1+dfsg1-2 was uploaded on 2015-05-10 13:02:54 -0400 EDT
Version 1.6.1+dfsg1-1 was uploaded on 2015-05-08 17:57:10 -0600 -0600
Version 1.6.0+dfsg1-1 was uploaded on 2015-05-05 15:10:49 -0600 -0600
Version 1.6.0+dfsg1-1~exp1 was uploaded on 2015-04-16 18:00:21 -0600 -0600
Version 1.6.0~rc7~dfsg1-1~exp1 was uploaded on 2015-04-15 19:35:46 -0600 -0600
Version 1.6.0~rc4~dfsg1-1 was uploaded on 2015-04-06 17:11:33 -0600 -0600
Version 1.5.0~dfsg1-1 was uploaded on 2015-03-10 22:58:49 -0600 -0600
Version 1.3.3~dfsg1-2 was uploaded on 2015-01-03 00:11:47 -0700 -0700
Version 1.3.3~dfsg1-1 was uploaded on 2014-12-18 21:54:12 -0700 -0700
Version 1.3.2~dfsg1-1 was uploaded on 2014-11-24 19:14:28 -0500 EST
Version 1.3.1~dfsg1-2 was uploaded on 2014-11-07 13:11:34 -0700 -0700
Version 1.3.1~dfsg1-1 was uploaded on 2014-11-03 08:26:29 -0700 -0700
Version 1.3.0~dfsg1-1 was uploaded on 2014-10-17 00:56:07 -0600 -0600
Version 1.2.0~dfsg1-2 was uploaded on 2014-10-09 00:08:11 +0000 +0000
Version 1.2.0~dfsg1-1 was uploaded on 2014-09-13 11:43:17 -0600 -0600
Version 1.0.0~dfsg1-1 was uploaded on 2014-06-13 21:04:53 -0400 EDT
Version 0.11.1~dfsg1-1 was uploaded on 2014-05-09 17:30:45 -0400 EDT
Version 0.9.1~dfsg1-2 was uploaded on 2014-04-08 23:19:08 -0400 EDT
Version 0.9.1~dfsg1-1 was uploaded on 2014-04-03 21:38:30 -0400 EDT
Version 0.9.0+dfsg1-1 was uploaded on 2014-03-11 22:24:31 -0400 EDT
Version 0.8.1+dfsg1-1 was uploaded on 2014-02-25 20:56:31 -0500 EST
Version 0.8.0+dfsg1-2 was uploaded on 2014-02-15 17:51:58 -0500 EST
Version 0.8.0+dfsg1-1 was uploaded on 2014-02-10 20:41:10 -0500 EST
Version 0.7.6+dfsg1-1 was uploaded on 2014-01-22 22:50:47 -0500 EST
Version 0.7.1+dfsg1-1 was uploaded on 2014-01-15 20:22:34 -0500 EST
Version 0.6.7+dfsg1-3 was uploaded on 2014-01-09 20:10:20 -0500 EST
Version 0.6.7+dfsg1-2 was uploaded on 2014-01-08 19:14:02 -0500 EST
Version 0.6.7+dfsg1-1 was uploaded on 2014-01-07 21:06:10 -0500 EST
control
Next is one of the most complex, and one of the oldest parts of go-debian,
which is the control file parser
(otherwise sometimes known as deb822). This module was inspired by the way
that the json module works in Go, allowing for files to be defined in code
with a struct. This tends to be a bit more declarative, but also winds up
putting logic into struct tags, which can be a nasty anti-pattern if used too
much.
The first primitive in this module is the concept of a Paragraph, a struct
containing two values, the order of keys seen, and a map of string to string.
All higher order functions dealing with control files will go through this
type, which is a helpful interchange format to be aware of. All parsing of
meaning from the Control file happens when the Paragraph is unpacked into
a struct using reflection.
The idea behind this strategy that you define your struct, and let the Control
parser handle unpacking the data from the IO into your container, letting you
maintain type safety, since you never have to read and cast, the conversion
will handle this, and return an Unmarshaling error in the event of failure.
Additionally, Structs that define an anonymous member of control.Paragraph
will have the raw Paragraph struct of the underlying file, allowing the
programmer to handle dynamic tags (such as X-Foo), or at least, letting
them survive the round-trip through go.
The default decoder
contains an argument, the ability to verify the input control file using an
OpenPGP keyring, which is exposed to the programmer through the
(*Decoder).Signer() function. If the passed argument is nil, it will not
check the input file signature (at all!), and if it has been passed, any
signed data must be found or an error will fall out of the NewDecoder call.
On the way out, the opposite happens, where the struct is introspected,
turned into a control.Paragraph, and then written out to the io.Writer.
Here's a quick (and VERY dirty) example showing the basics of reading and
writing Debian Control files with go-debian.
packagemainimport("fmt""io""net/http""strings""pault.ag/go/debian/control")typeAllowedPackagestructPackagestringFingerprintstringfunc(a*AllowedPackage)UnmarshalControl(instring)errorin=strings.TrimSpace(in)chunks:=strings.SplitN(in," ",2)iflen(chunks)!=2returnfmt.Errorf("Syntax sucks: '%s'",in)a.Package=chunks[0]a.Fingerprint=chunks[1][1:len(chunks[1])-1]returnniltypeDMUAstructFingerprintstringUidstringAllowedPackages[]AllowedPackage control:"Allow" delim:"," funcmain()resp,err:=http.Get("http://metadata.ftp-master.debian.org/dm.txt")iferr!=nilpanic(err)decoder,err:=control.NewDecoder(resp.Body,nil)iferr!=nilpanic(err)fordmua:=DMUAiferr:=decoder.Decode(&dmua);err!=niliferr==io.EOFbreakpanic(err)fmt.Printf("The DM %s is allowed to upload:\n",dmua.Uid)for_,allowedPackage:=rangedmua.AllowedPackagesfmt.Printf(" %s [granted by %s]\n",allowedPackage.Package,allowedPackage.Fingerprint)
Output (truncated!) looks a bit like:
...
The DM Allison Randal <allison@lohutok.net> is allowed to upload:
parrot [granted by A4F455C3414B10563FCC9244AFA51BD6CDE573CB]
...
The DM Benjamin Barenblat <bbaren@mit.edu> is allowed to upload:
boogie [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
dafny [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
transmission-remote-gtk [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
urweb [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
...
The DM <aelmahmoudy@sabily.org> is allowed to upload:
covered [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
dico [granted by 6ADD5093AC6D1072C9129000B1CCD97290267086]
drawtiming [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
fonts-hosny-amiri [granted by BD838A2BAAF9E3408BD9646833BE1A0A8C2ED8FF]
...
...
deb
Next up, we've got the deb module. This contains code to handle reading
Debian 2.0 .deb files. It contains a wrapper that will parse the control
member, and provide the data member through the
archive/tar interface.
Here's an example of how to read a .deb file, access some metadata, and
iterate over the tar archive, and print the filenames of each of the
entries.
dependency
The dependency package provides an interface to parse and compute
dependencies. This package is a bit odd in that, well, there's no other
library that does this. The issue is that there are actually two different
parsers that compute our Dependency lines, one in Perl (as part of dpkg-dev)
and another in C (in dpkg).
To date, this has resulted in me filing
threedifferentbugs.
I also found a broken package in the
archive,
which actually resulted in another bug being (totally accidentally)
already fixed.
I hope to continue to run the archive through my parser in hopes of finding
more bugs! This package is a bit complex, but it basically just returns what
amounts to be an AST
for our Dependency lines. I'm positive there are bugs, so file them!
funcmain()dep,err:=dependency.Parse("foo bar, baz, foobar [amd64] bazfoo [!sparc], fnord:armhf [gnu-linux-sparc]")iferr!=nilpanic(err)anySparc,err:=dependency.ParseArch("sparc")iferr!=nilpanic(err)for_,possi:=rangedep.GetPossibilities(*anySparc)fmt.Printf("%s (%s)\n",possi.Name,possi.Arch)
Gives the output:
foo (<nil>)
baz (<nil>)
fnord (armhf)
version
Right off the bat, I'd like to thank
Michael Stapelberg for letting me graft this
out of dcs and into the go-debian package.
This was nearly entirely his work (with a one or two line function I added
later), and was amazingly helpful to have. Thank you!
This module implements Debian version comparisons and parsing, allowing for
sorting in lists, checking to see if it's native or not, and letting the
programmer to implement smart(er!) logic based on upstream (or Debian)
version numbers.
This module is extremely easy to use and very straightforward, and not worth
writing an example for.
Final thoughts
This is more of a "Yeah, OK, this has been useful enough to me at this point
that I'm going to support this" rather than a "It's stable!" or even
"It's alive!" post. Hopefully folks can report bugs and help iterate on
this module until we have some really clean building blocks to build
solid higher level systems on top of. Being able to have multiple libraries
interoperate by relying on go-debian will be a massive ease.
I'm in need of more documentation, and to finalize some parts of the older
sub package APIs, but I'm hoping to be at a "1.0" real soon now.
Hello
While hacking on a script to update build dependencies on a Debian package, it occured to me that using Config::Model in a Perl program should be no more complicated than using cme from a shell script. That was an itch that I scratched immediately.
Fast forward a few days, Config::Model now features new cme() and modify() functions that have a behavior similar to cme modify command.
For instance, the following program is enough to update popcon s configuration file:
use strict; # let's not forget best practices ;-)
use warnings;
use Config::Model qw(cme); # cme function must be imported
cme('popcon')->modify("PARTICIPATE=yes");
The object returned by cme() is a Config;:Model::Instance. All its methods are available for a finer control. For instance:
my $instance = cme('popcon');
$instance->load("PARTICIPATE=yes");
$instance->apply_fixes;
$instance->say_changes;
$instance->save;
When run as root, the script above shows:
Changes applied to popcon configuration:
- PARTICIPATE: 'no' -> 'yes'
If need be, you can also retrieve the root node of the configuration tree to use Config;:Model::Node methods:
my $root_node = cme('popcon')->config_root;
say "is popcon active ? ",$root_node->fetch_element_value('PARTICIPATE');
In summary, using cme in a Perl program is now as easy as using cme from a shell script.
To provide feedback, comments, ideas, patches or to report problems, please follow the instructions from CONTRIBUTING page on github.
All the best Tagged: config-model, configuration, Perl
This week I have been gathered with 38 KDE people in Randa, Switzerland. Randa is a place in a valley in the middle of the Alps close to various peaks like Matterhorn. It has been a week of intense hacking, bugfixing, brainstorming and a bit of enjoying the nature.
R is for Reproducible builds
I spent the first couple of days trying to get the Qt Documentation generation tool to reproducible generate documentation. Some of the fixes were of the usual put data in an randomized datastructure, then iterate over it and create output , where the fix is similar well known: Sort the datastructure first. Others were a bit more severe bugs that lead to the documentation to shuffle around the obsolete bit, and the inheritance chains. Most of these fixes have been reviewed and submitted to the Qt 5.6 branch, one is still pending review, but that hopefully gets fixed soon. Then most of Qt (except things containing copies of (parts) of webkit and derivatives) should be reproducible.
R is for Roaming around in the mountains
Sleeping, hacking and dining in the same building sometimes leads to a enormous desire for fresh air. Luckily in the middle of the alps, it is readily available, and at least once a day many people went for a walk. To say hi to a sheep. Or to just go uphill until tired and then going back down. Or just finding a circle around. For this area, OpenStreetMap seems to have better maps than Google. We also went on a nice group trip to Zermatt and surroundings, sponsored by our friends in Edeltech.
R is for Releasing
One of the tasks I set myself for was to get my barcode generation library (prison. you know. being behind bars.) ready for release. A bit of api cleanup, including some future proofing, was done, and all users adapted. Hopefully it will be released as part of the next KDE Frameworks release.
R is for Reviewing code
When signing up for the sprint, one has to declare a couple of tasks to work on. One of the things I put myself up to was reviewing David Faure s code changes. First, he is very productive, and second, he often gets into creating patches in code areas where many other contributors are scared to look. So someone has to do it, and code never scared me.
R is for Running
I planned on going running along the river monday, wednesday and friday. Fortunately that happened, but due to Switzerland having a bit more ups and downs than flat Denmark, it didn t go that fast.
R is for Random bugfixing
When in the hacking mood surrounded by great developers, it is very easy to just fix minor bugs when you encounter them. There is likely someone around who knows the code in question. Or you are just in the mood to actually fix it, rather than living with a missing clock applet or a corner case crash.
R is for Rubber ducking
I am a brilliant person sized rubber duck. And I did get the opportunity to show off my skills a couple of times, as well as using some of the other people for that.
R is for Raising money
These sprints in Randa is only possible because of all the nice donations from people and companies around the world. The fundraiser is still running, and can be found at
Hi to all the souls on planet.debian.org . I hope to meet many of you in Debconf16 which is being held at UCT (University of Cape Town), Rhondenbosch, South Africa and am excited to be part of it. I pushed a blog post about my journey to debconf till date.
I express my sympathy and condolences to all the people who died in the cowardly shooting spree done by a madman in Orlando . I have been upset about this development but as what s done is done, it s best to just keep moving.
Closer to my own reality, I was shocked to discover during my whole visa experience that nowhere there is any knowledge about vaccinations that people should have when they are traveling internationally. I was kinda rudely awakened by this mail which prompted me to take Hep A shot and also start a thread on the mailing list. The more comprehensive info. I got was at the CDC site . It seems to be a go-to site to find about what is recommended. While I will not be taking any shots now as it s nearing to the date of travel, if I had known before, it would have been a valuable resource in itself. Definitely something to be bookmarked if you are going overseas and are worried or have health concerns.
On another note, About few days back, there was a discussion as recently, travel advisories have been issued about possible terror action in South Africa by various embassies due to the holy month of Ramadan.
And the recent attack just proves that it takes just one twisted personality with a perverse sense of justice or whatever s/he/ thinks as just to do what s/he did and guns and more security and not the answer.
My take on it some would describe as simplistic, there are things which you can control, there are things beyond your control. No security agency, no country can guarantee it. By being either home or away, you can t wish you will not get bitten. Got images of Final Destination invoked when I was sharing that, for as an Indian, do believe a bit on fate and karma. Also, me being single also plays a part, perhaps I would have been more cautious or have different motivations if I had a family so I do understand some of the concerns which have been raised by people in that thread. At the end of it, it really is a non-choice in my book.
If you don t take part due to fear,uncertainty of a possible attack, you have already given in to fear and uncertainty and I believe this goes against the very philosophy of what Debian stands for, being bold, taking chances and having trust in your fellow men. If we haven t allowed proprietary, commercial software to win over us, how can we allow less than 0.1% radically motivated people to scare us ? And the recent attack just proves that it takes just one twisted personality with a perverse sense of justice or whatever s/he/ thinks as just to do what s/he did. More guns and more security are certainly not the answer here.
A more troubling part is not terrorism but caste-ism and racism which have been also making news (not in a good way) in India. Now while I cannot claim to have any knowledge about Africans apart from 2-3 conversations which didn t go anywhere, two-three preconceived notions about them can easily be countered. As far as drugs are concerned, IF some Africans are doing drugs smuggling, it would be wrong to pin all of it onto them. India has been fighting drugs smuggling from the 70 s itself. From what we know and have learnt over the years from consuming media, India shares porous borders with almost all our neighbors. Of those, Pakistan is supposed to be the largest grower and supplier and then Nepal where young boys are used as Traffickers. The recent film Rocky Handsome and the recent upcoming controversial movie Udta Punjab are trying to explore these issues. As far as drinking in open is concerned, I have seen and been part of Punjabi parties where both men and women drink without abandon in farm parties. Russians, Germans and Israelis can also out-drink a person on their day/mood. As far as sexuality is concerned, we are the second most populous country in the world, so the less said the better .
I believe though that underneath this racism is money, greed, phobia, language barriers and just maybe some lifestyle choices as well. I have seen North-Eastern, Chinese, Buddhist people being colored with the same brush. The more interesting case is with the Buddhist as can be seen in Himachal Pradesh, where locals feel they are deprived as Europeans come and indulge Buddhists for their monasteries and their way of life while locals don t get much money from them. This is partly true, but also due to our own short-comings in dealing with westerners. I have never seen my own countrymen going out of their way to make westerners or any tourists for that matter feel welcome in a genuine way. More often, the behavior is between hostility, jealousy and a perverted sense of hero-worship due to the color of skin. As far as being racist and bigoted are concerned, it seems we are not alone, just day before came across this news from Malaysia which is interesting in the sense that how people from different communities frame their own history and people around them, forget that is false and misleading to what we in India know to be the truth, it probably is/would be interesting for somebody who does comparative history, myth and folklore analysis. I was actually planning to talk about some of the talks I am looking forward to hearing and seeing on Debconf but guess that will have to wait for few more days. Hope to publish one more before actually flying to Debconf. Till later. Filed under: Miscellenous Tagged: #Debconf16, #racism, #terrorism, #vaccinations
What happened in the Reproducible
Builds effort between June 5th and June 11th 2016:
Media coverage
Ed Maste gave a talk at BSDCan 2016 on
reproducible builds
(slides,
video).
GSoC and Outreachy updates
Weekly reports by our participants:
Scarlett Clark
worked on making some packages reproducible, focusing on KDE backend and
utility programs.
Ceridwen
published an initial design for the interface for reprotest, including a
discussion on different types of build variations and the difficulties of
specifying certain types of variations.
Valerie Young improved documentation for
building our tests website, began migrating Debian-specific pages into a new
namespace, and planned future work around its navigation.
Documentation update
- Ximin Luo proposed a modification
to our SOURCE_DATE_EPOCH spec explaining FORCE_SOURCE_DATE.
Some upstream build tools (e.g. TeX, see below) have expressed a desire to
control which cases of embedded timestamps should obey SOURCE_DATE_EPOCH.
They were not convinced by our arguments on why this is a bad idea, so we
agreed on an environment variable FORCE_SOURCE_DATE for them to implement
their desired behaviour - named generically, so that at least we can set it
centrally. For more details, see the text just linked. However, we strongly
urge most build tools not to use this, and instead obey SOURCE_DATE_EPOCHunconditionally in all cases.
Toolchain fixes
TeX Live 2016 released
with SOURCE_DATE_EPOCH support for all engines except LuaTeX and original TeX.
Continued discussion
(alternative archive)
with TeX upstream, about SOURCE_DATE_EPOCH corner cases, eventually
resulting in the FORCE_SOURCE_DATE proposal from above.
gcc-5/5.4.0-4 by Matthias Klose now avoids storing
-fdebug-prefix-map in DW_AT_producer, thanks to original patch by
Daniel Kahn Gillmor.
sphinx/1.4.3-1 by Dmitry Shachnev now drops Debian-specific patches
relating to SOURCE_DATE_EPOCH applied upstream, original patch by Alexis
Bienven e.
asciidoctor/1.5.4-2 by C dric Boutillier now supports
SOURCE_DATE_EPOCH, thanks to original patch by Alexis Bienven e.
dh-python/1.5.4-2 by Piotr O arowski now behaves better in some
cases, thanks to original patch by Chris Lamb.
Andrew Ayer uploaded strip-nondeterminism/0.018-2 fixing #826700, a packaging improvement for Multi-Arch to ease cross-build
situations.
2 days later Andrew released strip-nondeterminism/0.019; now
strip-nondeterminism is able to:
recursively normalize JAR files embedded within JAR files (#823917)
clamp the timestamp, the same way tar >=1.28-2.2 can (for now available only for gzip archives)
disorderfs development
Andrew Ayer released disorderfs/0.4.3, fixing a issue with umask handling (#826891)
tests.reproducible-builds.org
Valerie Young namespaced the Debian-specific pages to /debian/
namespace, with redirects to
for the previous URLs.
Holger Levsen improved the reliability of build jobs: the availability of
both build nodes (for a given build) is now being tested when a build job is
started, to better cope when one of the 25 build nodes go down for some reason.
Ximin Luo improved the index of identified
issues to
include the total popcon scores of each issue, which is now also used for
sorting that page.
Misc.
Steven Chamberlain submitted a patch to FreeBSD's
makefs
to allow reproducible builds of the kfreebsd installer.
Ed Maste committed a patch to FreeBSD's
binutils to
enable determinstic archives by default in GNU ar.
Helmut Grohne
experimentedwith
cross+native reproductions of dash with some success, using
rebootstrap.
This week's edition was written by Ximin Luo, Chris Lamb, Holger Levsen, Mattia
Rizzolo and reviewed by a bunch of Reproducible builds folks on IRC.
Yesterday I was fixing an issue with one of the servers behind kallithea-scm.org: the hook intended to propagage pushes from Our Own Kallithea to Bitbucket stopped working. Until yesterday, that server was using Debian s flavour of System V init and djb s d montools to keep things running. To make the hook asynchronous, I wrote a service to be managed to d montools, so that concurrency issued would be solved by it. However, I didn t implement any timeouts, so when last week wget froze while pulling Weblate s hook, there was nothing to interrupt it, so the hook stopped working since d montools thought it s already running and wouldn t re-trigger it. Killing wget helped, but I decided I need to do something with it to prevent the situation from happening in the future.
I ve been using systemd at work for the last year, so I am now confident I m happier with systemd than with d montools, so I decided to switch the server to systemd. Not surprisingly, I prepared unit files in about 5 minutes without having to look into the manuals again, while with d montools I had to check things every time I needed to change something. The tricky thing was the switch itself. It is a virtual server, presumably running in Xen, and I don t have access to the console, so if I bork break something, I need to summon Bradley Kuhn or someone from Conservancy, who s kindly donated the server to the project. In any case, I decided to attempt to upgrade without a reboot, so that I have more options to roll back my changes in the case things go wrong.
After studying the manpages of both systemd s init and sysvinit s init, I realised I can install systemd as /sbin/init and ask already running System V init to re-exec. However, systemd s init can t talk to System V init, so before installing systemd I made a backup on it. It s also important to stop all running services (except probably ssh) to make sure systemd doesn t start second instances of each. And then: /tmp/init u and we re running systemd! A couple of additional checks, and it s safe to reboot.
Only when I did all that I realised that in the case of systemd not working I d probably not be able to undo my changes if my connection interrupted. So, even though at the end it worked, probably it s not a good idea to perform such manipulations when you don t have an alternative way to connect to the server :)
Allison Lortie has provoked a lot of comment with her blog post on
a new proposal for how GTK is versioned.
Here's some more context from the discussion at the GTK hackfest that
prompted that proposal: there's actually quite a close analogy in
how new Debian versions are developed.
The problem we're trying to address here is the two sides of a trade-off:
Without new development, a library (or an OS) can't go anywhere new
New development sometimes breaks existing applications
Historically, GTK has aimed to keep compatible within a major version,
where major versions are rather far apart (GTK 1 in 1998, GTK 2 in
2002, GTK 3 in 2011, GTK 4 somewhere in the future). Meanwhile, fixing
bugs, improving performance and introducing new features sometimes
results in major changes behind the scenes. In an ideal world, these
behind-the-scenes changes would never break applications; however, the
world isn't ideal. (The Debian analogy here is that as much as we
aspire to having the upgrade from one stable release to the next not break
anything at all, I don't think we've ever achieved that in practice -
we still ask users to read the release notes, even though ideally
that wouldn't be necessary.)
In particular, the perceived cost of doing a proper ABI break (a fully
parallel-installable GTK 4) means there's a strong temptation to make
changes that don't actually remove or change C symbols, but are clearly
an ABI break, in the sense that an application that previously worked
and was considered correct no longer works. A prominent recent example is
the theming changes in GTK 3.20: the ABI in terms of functions available
didn't change, but what happens when you call those functions changed in
an incompatible way. This makes GTK hard to rely on for applications
outside the GNOME release cycle, which is a problem that needs to
be fixed (without stopping development from continuing).
The goal of
the plan we discussed today
is to decouple the latest branch of development, which moves fast and
sometimes breaks API, from the API-stable branches, which only get bug
fixes. This model should look quite familiar to Debian contributors,
because it's a lot like the way we release Debian and Ubuntu.
In Debian, at any given time we have a development branch
(testing/unstable) - currently "stretch", the future Debian 9. We
also have some stable branches, of which the most recent are Debian
8 "jessie" and Debian 7 "wheezy". Different users of Debian have
different trade-offs that lead them to choose one or the other of
these. Users who value stability and want to avoid unexpected changes,
even at a cost in terms of features and fixes for non-critical bugs,
choose to use a stable release, preferably the most recent; they only
need to change what they run on top of Debian for OS API changes (for
instance webapps, local scripts, or the way they interact with the GUI)
approximately every 2 years, or perhaps less often than that with the
Debian-LTS project supporting non-current stable releases. Meanwhile,
users who value the latest versions and are willing to work with a
"moving target" as a result choose to use testing/unstable.
The GTK analogy here is really quite close. In the new versioning model,
library users who value stability over new things would prefer to use
a stable-branch, ideally the latest; library users who want the latest
features, the latest bug-fixes and the latest new bugs would use the
branch that's the current focus of development. In practice we expect
that the latter would be mostly GNOME projects. There's been some
discussion at the hackfest about how often we'd have a new
stable-branch: the fastest rate that's been considered is a stable-branch
every 2 years, similar to Ubuntu LTS and Debian, but there's no consensus
yet on whether they will be that frequent in practice.
How many stable versions of GTK would end up shipped in Debian depends
on how rapidly projects move from "old-stable" to "new-stable" upstream,
how much those projects' Debian maintainers are willing to patch them
to move between branches, and how many versions the release team will
tolerate. Once we reach a steady state, I'd hope that we might have
1 or 2 stable-branched versions active at a time, packaged as separate
parallel-installable source packages (a lot like how we handle Qt).
GTK 2 might well stay around as an additional active version just
from historical inertia. The stable versions are intended to be
fully parallel-installable, just like the situation with GTK 1.2,
GTK 2 and GTK 3 or with the major versions of Qt.
For the "current development" version, I'd anticipate that we'd probably
only ship one source package, and do ABI transitions for one version
active at a time, a lot like how we deal with libgnome-desktop and the
evolution-data-server family of libraries. Those versions would have
parallel-installable runtime libraries but non-parallel-installable
development files, again similar to libgnome-desktop.
At the risk of stretching the Debian/Ubuntu analogy too far, the
intermediate "current development" GTK releases that would accompany
a GNOME release are like Ubuntu's non-LTS suites: they're more up to
date than the fully stable releases (Ubuntu LTS, which has a release
schedule similar to Debian stable), but less stable and not supported
for as long.
Hopefully this plan can meet both of its goals: minimize breakage for
applications, while not holding back the development of new APIs.
Elsa Bike Trophy 2016 my first bike race!
So today, after two months of intermittent training using Zwift and
some actual outside rides, I did my first bike race. Not of 2016,
not of 2000+, but like ever.
Which is strange, as I learned biking very young, and I did like to
bike. But as it turned out, even though I didn't like running as a
child, I did participate in a number of running events over the years,
but no biking ones.
The event
Elsa Bike Trophy is a
mountain bike event cross-country, not downhill or anything crazy; it
takes part in Estavayer-le-Lac, and has two courses - one 60Km with
1'791m altitude gain, and a smaller one of 30Km with 845m altitude
gain. I went, of course, for the latter. 845m is more than I ever did
in a single ride, so it was good enough for a first try. The web page
says that this smaller course est nerveux, technique et ne laisse
que peu de r pit . I choose to think that's a bit of an exaggeration,
and that it will be relatively easy (as I'm not too skilled
technically).
The atmosphere there was like for the running races, with the
exception of bike stuff being sold, and people on very noisy
rollers. I'm glad for my trainer which sounds many decibels quieter
The long race started at 12:00, and the shorter one at 12:20. While
waiting for the start I had to concerns in mind: whether I'm able to
do the whole course (endurance), and whether it will be too cold (the
weather kept moving towards rain). I had a small concern about the
state of the course, as it was not very nice weather recently, but a
small one.
And then, after one hour plus of waiting, go!
Racing, with a bit of "swimming"
At first thing went as expected. Starting on paved roads, moving
towards the small town exit, a couple of 14% climbs, then more flat
roads, then a nice and hard 18% short climb (I'll never again complain
about < 10%!), then entering the woods. It became quickly apparent
that the ground in the forest was in much worse state than I
feared. Much worse as in a few orders of magnitude.
In about 5 minutes after entering the tree cover, my reasonably clean,
reasonably light bike became a muddy, heavy monster. And the pace that
until then went quite OK became walking pace, as the first rider that
didn't manage to keep going up because the wheel turned out of the
track blocked the one behind him, which had to stop, and repeat until
we were one line (or two, depending on how wide the trail was) of
riders walking their bikes up. While on dry ground walking your bike
up is no problem, or hiking through mud with good hiking shoes is also
no problem, walking up with biking shoes is a pain. Your foot slides
and you waste half of your energy "swimming" in the mud.
Once the climb is over, you get on the bike, and of course the pedals
and cleats are full of heavy mud, so it takes a while until you can
actually clip in. Here the trail version of SPD was really useful, as
I could pedal reasonably well without being clipped-in, just had to be
careful and push too hard.
Then maybe you exit the trail and get on paved road, but the wheels
are so full of mud that you still are very slow (and accelerate very
slowly), until the shed enough of the mud to become somewhat more
"normal".
After a bit of this "up through mud, flat and shedding mud", I came
upon the first real downhill section. I would have been somewhat
confident in dry ground, but I got scared and got off my bike. Better
safe than sorry was the thing for now.
And after this is was a repetition of the above: climb, sometimes
(rarely) on the bike, most times pushing the bike, fast flat sections
through muddy terrain where any mistake of controlling the bike can
send the front wheel flying due to the mud being highly viscous, slow
flat sections through very liquid mud where it definitely felt like
swimming, or any dry sections.
My biggest fear, uphill/endurance, was unfounded. The most gains I
made were on the dry uphills, where I had enough stamina to
overtake. On flat ground I mostly kept order (i.e. neither being
overtaken nor overtaking), but on downhill sections, I lost lots of
time, and was overtaken a lot. Still, it was a good run.
And then, after about 20 kilometres out of the 30, I got tired enough
of getting off the bike, on the bike, and also tired mentally and not
being careful enough, that I stopped getting off the bike on
downhills. And the feeling was awesome! It was actually much much
easier to flow through the mud and rocks and roots on downhill, even
when it was difficult (for me) like 40cm drops (estimated), than doing
it on foot, where you slide without control and the bike can come
crashing down on you. It was a liberating feel, like finally having
overcome the mud. I was soo glad to have done a one-day training
course with
Swiss Alpine Adventure,
as it really helped. Thanks Dave!
Of course, people were still overtaking me, but I also overtook some
people (who were on foot; he he, I wasn't the only one it seems). And
being easier, I had some more energy so I was able to push a bit
harder on the flats and dry uphill sections.
And then the remaining distance started shrinking, and the last
downhill was over, I entered through familiar roads the small town, a
passer-by cries "one kilometre left", I push hard (I mean, hard as I
could after all the effort), and I reach the finish.
Oh, and my other concern, the rain? Yes it did rain somewhat, and I
was glad for it (I keep overheating); there was a single moment I felt
cold, when exiting a nice cosy forest into a field where the wind was
very strong headwind, of course.
Lessons learned
I did learn a lot in this first event.
indoor training sessions only help with endurance (but they do good
on this); they don't help with technique, and most importantly, they
don't teach how to handle the bike in inclement weather; biking to
work on paved road also doesn't help.
nevertheless, indoor training does help with endurance
mud guards ; before the race, I thought they'll help; during the
race, I cursed at the extra weight and their seemingly uselessness;
after the race, after I saw how other people looked, I realised that
indeed they helped a lot I was only dirty on my legs, mostly below
the knee, but not on my upper body. Unsure whether I will use the
again.
a drop seat is not needed if your seat is set in-between, but sure
damn would have been more easy with one
installing your GPS on your handle-bar with elastic bands in a
section of non-constant diameter is a very bad idea, as it lives in
an unstable equilibrium: any move towards the thinner section makes
the mount very loose, and you have to lose time fixing it.
Results
So, how did I do after all? As soon as I reached the finish and
recovered my items, among which the phone, I checked the datasport
page: I was rank 59/68 in my category. Damn, I hoped (and thought) I
would do better. Similar % in the overall ranking for this distance.
That aside, it was mighty fun. So much fun I'd do it again tomorrow! I
forgot the awesome atmosphere of such events, even in the back of the
rankings.
And then, after I reach drive home and open on my workstation the
datasport page, I get very confused: the overall number of
participants was different. And the I realised: not everybody finished
the race when I first checked (d'oh)! Final ranking: 59 out of 84 in
my category,
and 247/364 in the
overall 30km rankings. That
makes it 70% and 67% respectively, which matches somewhat with my
usual running results a few years back (but a bit worse). It is in any
case better than what I thought originally, yay!
Also, Strava activity for some more statistics (note that my Garmin
says it was not 800+ meters of altitude ):
I'd embed a Veloviewer nice 3D-map but I can't seem to get the embed
option, hmm
TODO: train more endurance, train more technique, train in more
various conditions!
As nearly anyone who's worked with me will attest to, I've long since
touted nedbat's talk
Pragmatic Unicode, or, How do I stop the pain?
as one of the most foundational talks, and required watching for all programmers.
The reason is because netbat hits on something bigger - something more
fundamental than how to handle Unicode -- it's how to handle data which is
relative.
For those who want the TL;DR, the argument is as follows:
Facts of Life:
Computers work with Bytes. Bytes go in, Bytes go out.
Relative Data
I've started to think more about why we do the things we do when we write
code, and one thing that continues to be a source of morbid schadenfreude
is watching code break by failing to handle Unicode right. It's hard! However,
watching what breaks lets you gain a bit of insight into how the author
thinks, and what assumptions they make.
When you send someone Unicode, there are a lot of assumptions that have to be
made. Your computer has to trust what you (yes, you!) entered into your web
browser, your web browser has to pass that on over the network (most of the
time without encoding information), to a server which reads that bytestream,
and makes a wild guess at what it should be. That server might save it to a
database, and interpolate it into an HTML template in a different encoding
(called Mojibake), resulting
in a bad time for everyone involved.
Everything's awful, and the fact our computers can continue to display
text to us is a goddamn miracle. Never forget that.
When it comes down to it, when I see a byte sitting on a page, I don't know
(and can't know!) if it's Windows-1252, UTF-8, Latin-1, or EBCDIC. What's
a poem to me is terminal garbage to you.
Over the years, hacks have evolved. We have
magic numbers,
and plain ole' hacks to just guess based on the content. Of course, like
all good computer programs, this has lead to its fair share of hilarious
bugs,
and there's nothing stopping files from (validly!) being multiple things at the
same time.
Like many things, it's all in the eye of the beholder.
Timezones
Just like Unicode, this is a word that can put your friendly neighborhood
programmer into a series of profanity laden tirades. Go find one in the wild,
and ask them about what they think about timezone handling bugs they've seen.
I'll wait. Go ahead.
Rants are funny things. They're fun to watch. Hilarious to give. Sometimes
just getting it all out can help. They can tell you a lot about the true
nature of problems.
It's funny to consider the isomorphic nature of Unicode rants and Timezone
rants.
I don't think this is an accident.
U n i c o d e timezone Sandwich
Ned's Unicode Sandwich applies -- As early as we can, in the lowest level
we can (reading from the database, filesystem, wherever!), all datetimes
must be timezone qualified with their correct timezone. Always. If you mean
UTC, say it's in UTC.
Treat any unqualified datetimes as "bytes". They're not to be trusted.
Never, never, never trust 'em. Don't
process any datetimes until you're sure they're in the right timezone.
This lets the delicious inside of your datetime sandwich handle timezones
with grace, and finally, as late as you can, turn it back into bytes
(if at all!). Treat locations as tzdb entries, and qualify datetime
objects into their absolute timezone (EST, EDT, PST, PDT)
It's not until you want to show the datetime to the user again should you
consider how to re-encode your datetime to bytes. You should think about
what flavor of bytes, what encoding -- what timezone -- should I be
encoding into?
TEST
Just like Unicode, testing that your code works with datetimes is important.
Every time I think about how to go about doing this, I think about that
one time that mjg59 couldn't book a flight
starting Tuesday from AKL, landing in HNL on Monday night, because
United couldn't book the last leg to SFO. Do you ever assume dates only go
forward as time goes on? Remember timezones.
Construct test data, make sure someone in New Zealand's
+13:45 can correctly talk with
their friends in
Baker Island's -12:00,
and that the events sort right.
Just because it's Noon on New Years Eve in England doesn't mean it's not
1 AM the next year in New Zealand. Places a few miles apart may go on Daylight
savings different days. Indian Standard Time is not even aligned on the hour
to GMT (+05:30)!
Test early, and test often. Memorize a few timezones, and challenge
your assumptions when writing code that has to do with time. Don't use
wall clocks to mean monotonic time. Remember there's a whole world out there,
and we only deal with part of it.
It's also worth remembering, as Andrew Pendleton
pointed out to me, that it's possible that a datetime isn't even unique for a
place, since you can never know if 2016-11-06 01:00:00 in America/New_York
(in the tzdb) is the first one, or second one. Storing EST or EDT along
with your datetime may help, though!
Pitfalls
Improper handling of timezones can lead to some interesting things, and failing
to be explicit (or at least, very rigid) in what you expect will lead to an
unholy class of bugs we've all come to hate. At best, you have confused
users doing math, at worst, someone misses a critical event, or our
security code fails.
I recently found what I regard to be a pretty bad
bug in apt (which David has prepared a
fix
for and is pending upload, yay! Thank you!), which boiled down to documentation
and code expecting datetimes in a timezone, but accepting any timezone, and
silently treating it as UTC.
The solution is to hard-fail, which is an interesting choice to me (as a vocal
fan of timezone aware code), but at the least it won't fail by
misunderstanding what the server is trying to communicate, and I do understand
and empathize with the situation the apt maintainers are in.
Final Thoughts
Overall, my main point is although most modern developers know how to deal
with Unicode pain, I think there is a more general lesson to learn -- namely,
you should always know what data you have, and always remember what it is.
Understand assumptions as early as you can, and always store them with the data.
Historically, the adt-run command line has allowed multiple tests; as a consequence, arguments like --binary or --override-control were position dependent, which confused users a lot (#795274, #785068, #795274, LP #1453509). On the other hand I don t know anyone or any CI system which actually makes use of the multiple tests on a single command line feature.
The command line also was a bit confusing in other ways, like the explicit --built-tree vs. --unbuilt-tree and the magic / vs. // suffixes, or option vs. positional arguments to specify tests.
The other long-standing confusion is the pervasive adt acronym, which is still from the very early times when autopkgtest was called autodebtest (this was changed one month after autodebtest s inception, in 2006!).
Thus in some recent night/weekend hack sessions I ve worked on a new command line interface and consistent naming. This is now available in autopkgtest 4.0 in Debian unstable and Ubuntu Yakkety. You can download and use the deb package on Debian jessie and Ubuntu 14.04 LTS as well. (I will provide official backports after the first bug fix release after this got some field testing.)
New autopkgtest command
The adt-run program is now superseded by autopkgtest:
It accepts only exactly one tested source package, and gives a proper error if none or more than one (often unintend) is given. Binaries to be tested, --override-control, etc. can now be specified in any order, making the arguments position independent. So you now can do things like:
autopkgtest *.dsc *.deb [...]
Before, *.deb only applied to the following test.
The explicit --source, --click-source etc. options are gone, the type of tested source/binary packages, including built vs. unbuilt tree, is detected automatically. Tests are now only specified with positional arguments, without the need (or possibility) to explicitly specify their type. The one exception is --installed-click com.example.myapp as possible names are the same as for apt source package names.
The virtualization server is now separated with a double instead of a tripe dash, as the former is standard Unix syntax.
It defaults to the current directory if that is a Debian source package. This makes the command line particularly simple for the common case of wanting to run tests in the package you are just changing:
autopkgtest -- schroot sid
Assuming the current directory is an unbuilt Debian package, this will build the package, and run the tests in ./debian/tests against the built binaries.
The virtualization server must be specified with its short name only, e. g. ssh instead of adt-virt-ssh . They also don t get installed into $PATH any more, as it s hardly useful to call them directly.
README.running-tests got updated to the new CLI, as usual you can also read the HTML online.
The old adt-run CLI is still available with unchanged behaviour, so it is safe to upgrade existing CI systems to that version.
Image build tools
All adt-build* tools got renamed to autopkgtest-build*, and got changed to build images prefixed with autopkgtest instead of adt . For example, adt-build-lxc ubuntu xenial now produces an autopkgtest-xenial container instead of adt-xenial.
In order to not break existing CI systems, the new autopkgtest package contains symlinks to the old adt-build* commands, and when being called through them, also produce images with the old adt- prefix.
Environment variables in tests
Finally there is a set of environment variables that are exported by autopkgtest for using in tests and image customization tools, which now got renamed from ADT_* to AUTOPKGTEST_*:
AUTOPKGTEST_APT_PROXY
AUTOPKGTEST_ARTIFACTS
AUTOPKGTEST_AUTOPILOT_MODULE
AUTOPKGTEST_NORMAL_USER
AUTOPKGTEST_REBOOT_MARK
AUTOPKGTEST_TMP
As these are being used in existing tests and tools, autopkgtest also exports/checks those under their old ADT_* name. So tests can be converted gradually over time (this might take several years).
Feedback
As usual, if you find a bug or have a suggestion how to improve the CLI, please file a bug in Debian or in Launchpad. The new CLI is recent enough that we still have some liberty to change it.
Happy testing!