Out of 46,500,001 electorate 17,410,742 voted to leave, which is a mere 37.4% or just over a third. [source]. On my books this is not a clear expression of the UK wishes.
The reaction that the results have caused are devastating. The Scottish First Minister has announced plans for 2nd Scottish Independence referendum [source]. Londoners are filing petitions calling for Independent London [source, source]. The Prime Minister announced his resignation [source]. Things are not stable.
I do not believe that super majority of the electorate are in favor of leaving the EU. I don't even believe that those who voted to leave have considered the break up of the UK as the inevitable outcome of the leave vote. There are numerous videos on the internet about that, impossible to quantify or reliably cite, but for example this [source]
So What Now?
P R O T E S T
I urge everyone to start protesting the outcome of the mistake that happened last Thursday. 4th of July is a good symbolic date to show your discontent with the UK governemnt and a tiny minority who are about to cause the country to fall apart with no other benefits. Please stand up and make yourself heard.
There are 64,100,000 people living in the UK according to the World Bank, maybe the government should fear and listen to the unheard third. The current "majority" parliament was only elected by 24% of electorate.
It is time for people to actually take control, we can fix our parliament, we can stop austerity, we can prevent the break up of the UK, and we can stay in the EU. Over to you.
ps. How to elect next PM?
Electing next PM will be done within the Conservative Party, and that's kind of a bummer, given that the desperate state the country currently is in. It is not that hard to predict that Boris Johnson is a front-runner. If you wish to elect a different PM, I urge you to splash out 25 quid and register to be a member of the Conservative Party just for one year =) this way you will get a chance to directly elect the new Leader of the Conservative Party and thus the new Prime Minister. You can backdoor the Conservative election here.
[...] a little bit about the reason I wrote pristine-tar in the
first place. There were two reasons:
1. I was once in a talk where someone mentioned that Ubuntu had/was
developing something that involved regenerating orig tarballs
from version control.
I asked the obvious question: How could that possibly be done
technically?
The (slightly hung over) presenter did not have a satesfactory
response, so my curiosity was piqued to find a way to do it.
(I later heard that Ubuntu has been using pristine-tar..)
2. Sometimes code can be subversive. It can change people's perspective
on a topic, nudging discourse in a different direction. It can even
point out absurdities in the way things are done. I may or may not
have accomplished the subversive part of my goals with pristine-tar.
Code can also escape its original intention. Many current uses of
pristine-tar fall into that category. So it seems likely that some
people will want it to continue to work even if it's met the two goals
above already.
For me, the best part of building pristine-tar was finding an
answer to the question "How could that possibly be done technically?"
It was also pretty cool to be able to use every tarball in Debian as the test
suite for pristine-tar.
I'm afraid I kind of left Debian in the lurch when I stopped maintaining
pristine-tar.
"Debian has probably hundreds, if not thousands of git repositories using
pristine-tar. We all rely now on an unmaintained, orphaned, and buggy piece
of software." -- Norbert Preining
So I was relieved when it finally got a new maintainer just recently.
Still, I don't expect I'll ever use pristine-tar again. It's the only
software I've built in the past ten years that I can say that about.
Next: twenty years of free software -- part 6 moreutils
Here comes my journey till the mid-term (June 21st) as a blog to share my experience.
I have previously worked on some social related projects such as smart guidance for blind and sensor based wireless controller . I have been selected as a speaker for FOSSASIA-16 (Asia s premier technological event)to talk on the project smart guidance for blind .FOSSASIA speakers. It was a great experience participating in the technological event in Singapore science centre.Got an opportunity to meet open source contributors from all over the world(even though it is an Asian event, participation was all over from the world). There were pre-meetups for FOSSASIA on the day before three-day event.I have attended the one organised by RedHat, Singapore.Discussed on many topics related to open source.
Three days of FOSSASIA event was a great experience.It was the second time as a speaker in an international conference.My talk was on the second day.Sharing is the best way to increase your knowledge. Talks and workshops were brainstorming.Learnt many new things and got the courage to contribute to the open source.Met Daniel Pocock in Debian exhibition table.Meeting awesome people can be the turning point of life.Had a discussion about the Debian projects and it motivated me for open source software.We have discussed about the Real Time Communication and was encouraged to apply for GSoC (Google Summer of Code). As per our discussion, prepared the project proposal on improving voice,video and chat communication with free software and submitted it for GSoC. I have been selected to contribute for Debian with stipend from Google.
This was my first application for GSoC and I have been selected to contribute for open source and free software. I would like to thank Google and Debian for giving this amazing experience.
Learning and coding have begun.Updated my laptop with Jessie, latest version of Debian.Get acquainted with the new platform.Got to learn many things about Real Time Communication.Learnt more about SIP, XMPP, peer-to-peer technology to work on my project.It s always better to be clear with theory before coding.When it comes to voice and video over IP, most people nowadays are quick to use Skype, Whatsapp, or Viber. My main goals of the project are helping people to avoid using proprietary communications tools like Skype, Viber and WhatsApp and simplifying the setup of free alternatives like Jitsi, Linphone, Ekiga, Tox (qtox), Mumble.Downloaded some of the already available open source VoIP to find the problems behind it and improve it further.Bootstrapping any business relevant network based on these free alternatives is still hard.
Would you like to list the senders, receivers and date of the messages in the inbox of your mail.Python has a library file IMAP which can be used to connect to an email account, examine every message in every folder and look at the To , From and CC headers of every email message in the folder.
Do you have phone numbers and other contact details in old emails? Would you like a quick way to data-mine your inbox to find them and help migrate them to your address book? Got the help from phonenumbers library for parsing, formatting, and validating international phone numbers.I would like to share how I imported this library file into my coding.Download the given library file and open the file in the terminal.Type
$ python setup.py install
to install the library file.Now you can call the functions by importing phonenumbers.
You can go through the code in my GitHub profile here.(Recently started committing my projects in GitHub)
Iain R. Learmonth joined my journey as a mentor.Helped in solving some issues in my coding through GitHub.
It was a wonderful journey till now.Will be working further to improve voice, video and chat communication with free software.Stay connected to know more about my further journey through GSoC.
Debian:
I am very late on post due to travel, Flu, jetlag sorry!
choqok:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825322
For this I was able to come up with a patch for kconfig_compiler to encode generated files to utf-8.
Review request is here:
https://git.reviewboard.kde.org/r/128102/
This has been approved and I will be pushing it as soon as I patch the qt5 frameworks version.
Both kde4libs and kf5 kconfig has been pushed upstream kde.
kxmlgui:
WIP this has been a steep learning curve, according to the notes it was an easy embedded kernel version, that was not the case! After grueling hours of
trying to sort out randomness in debug output I finally narrowed it down to cases where QStringLiteral was used and there were non letter characters eg. ( <")
These were causing debug symbols to generate with ( lambda() ) which caused unreproducible symbol/debug files. It is now a case of fixing all of these in the code to use QString::fromUtf8 seems to fix this. I am working on a mega patch for upstream and it should be ready early in the week.
This last week I spent a large portion making my through a mega patch for kxmlgui, when it was suggested to me to write a small qt app
to test QStringLiteral isolated and sure enough two build were byte for byte identical. So this means that QStringLiteral may not be the issue at all. With some
more assistance I am to expand my test app with several QStringLiterals of varying lengths, we have suspicion it is a padding issue, which complicates things.
I am still fighting with this one, will set aside to simmer for now, as I have no idea how to fix padding issues.
extra-cmake-modules:
I am testing a patch to fix umask issues for anyone that uses the kapptemplate generation macro. Thank you Simon for pointing me to this.
known affected:
plasma-framework
kdevplatform:
The kapptemplate generation users/groups and umask patch has been pushed upstream.
https://bugs.kde.org/show_bug.cgi?id=363615
KDE Randa!:
Despite managing to get a terrible Flu I accomplished more than I would have at home without awesome devs to help me out!
I have delegated the windows backend to Hannah and Kevin, if emerge is successful with Windows we will implement it on OSX as well.
Android docker image is up and running.
Several snappy packages done. Improved the snapcraft.yaml creation automation scripts started by Harald. Got help from
David ( he even made a patch! ) with some issues we were facing with kio.
KDE CI DSL adjustments for 5 new platforms
Port tools/* python scripts to python3
CI TODO:
Python automation scripts can no longer find projects except qt5 Need to get help from Ben as these are originally his.
Finish yaml CI files
Randa as usual was an amazing experience. Yes it is very hard work, but you have the beauty of the Swiss Alps at your fingertips! Not to mention all the
friendly faces and collaboration. A big thank you to all supporters and the Randa team!
Please help make KDE better by supporting the very important Randa Sprint:
https://www.kde.org/fundraisers/randameetings2016/
Have a great day.
"We will encourage you to develop the three great virtues of a programmer: laziness, impatience, and hubris." [Larry Wall, Programming Perl, O'Reilly Assoc. (and expanded at http://c2.com/cgi/wiki?LazinessImpatienceHubris) ]
Because "A mind is a terrible thing to waste" [The above copyright Young and Rubicam, advertisers, for UNC Fund, 1960s]
"Why I Must Write GNUI consider that the Golden Rule requires that if I like a program I must share it with other people who like it. Software sellers want to divide the users and conquer them, making each user agree not to share with others. I refuse to break solidarity with other users in this way. I cannot in good conscience sign a nondisclosure agreement or a software license agreement. ... " [rms, GNU Manifesto copyright 1985-2014 Free Software Foundation Inc. https://www.gnu.org/gnu/manifesto.html]
"La p dagogie, l information, la culture et le d bat d opinion sont le seul fait des utilisateurs, des webmestres ind pendants et des initiatives universitaires et associatives." Education, information, culture and debate can only come from users, independent webmasters, academic or associative organizations. [le minir zo http://www.uzine.net/article60.html]
We value:
Contributors and facilitators over editors and authors
Collaboration over indiviualised production
Here and now production over sometime soon production
Meaningful credit for all contributors over single author attribution
https://github.com/greyscalepress/manifestos - from whom much of the above quotations were abstracted - Manifestos for the Internet Age Grayscale Press ISBN-13:978-2-940561-02-5]
[Note] Github repository is marked with licence of CC-Zero but explicitly states that licences of the individual pieces of writing should be respected
So - collaboration matters. Not repeating needless make-work that someone else has already done matters. Giving due credit: sharing: doing and "do-ocracy" matters above all
Perversely, Acknowledging prior work and prior copyright correctly is the beginning and end of the law. Only by doing this conscientiously and sharing in giving due credit can any of us truly participate.
It seems clear to me at least that contributing openly and freely, allowing others to make use of your expertise, opinions, prior experience can anyone progress in good conscience.
Accordingly, I recommend to my work colleagues and those I advise that they only consider FLOSS licences, that they do not make use of code snippets or random, unlicensed code culled form Github and that they contribute
Having seen some posts about this elsewhere on the 'Net:
Your copyright remains your own unless you assign it
Establish what you are being paid for: are you being paid for :
Your specific area of FLOSS expertise (or)
Your time / hours in an area unrelated to your FLOSS expertise (or)
A job that has no impact or bearing on your FLOSS expertise (or)
Your time / hours only - and negotiate accordingly
Your employer may be willing to negotiate / grant you an opt-out clause to protect your FLOSS expertise / accept an additional non-exclusive licence to your FLOSS code / be prepared to sign an assignment e.g.
"You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program Gnomovision' (which makes passes at compilers) written by James Hacker.
signature of Ty Coon, 1 April 1989 Ty Coon, President of Vice"
If none of the above is feasible: don't contribute anything that crosses the streams and mingles commercial and FLOSS expertise, however much you're offered to do so.
Patents / copyrights
"In the 1980s I had not yet realized how confusing it was to speak of the issue of intellectual property . That term is obviously biased; more subtle is the fact that it lumps together various disparate laws which raise very different issues. Nowadays I urge people to reject the term intellectual property entirely, lest it lead others to suppose that those laws form one coherent issue. The way to be clear is to discuss patents, copyrights, and trademarks separately. See further explanation of how this term spreads confusion and bias." [http://www.gnu.org/gnu/manifesto.en.html - footnote 8.]
If you want to assert a patent - it's probably not FLOSS. Go away :)
If you want to assert a trademark of your own - it's probably not FLOSS. Go away :) [Trademarks may ordinarily be outside the scope of normal FLOSS legal considerations - but should be acknowledged wherever they occur both as a matter of law and as a matter of courtesy]
Copyright gives legal standing (locus standi in the terminology of English common law) to sue for infringement - that's the basis of licence enforcement actions.
Employees of governments and those doing government work
Still have the right to own authorship and copyrights and to negotiate accordingly
May need to establish more clearly what they're being paid for
May be able to advise, influence or direct policy towards FLOSS in their own respective national jurisdiction
Should, ideally, be primariily acknowledged as individuals, holding and maintaining an individual reputation and only secondarily as contractors/employees/others associated with government work.
Contribution to national / international standards, international agreements and shared working practices should be informed in the light of FLOSS work.
This is complex: some FLOSS contributors see a significant amount of this as immaterial to them in the same way that some indigenous populations do not acknowledge imposed colonial legal structures as valid - but both value systems can co-exist
Contribute constructively - no one likes to be told "You've got a REALLY ugly baby there" or equivalent.
Think through what you post: check references and check that it reads clearly and is spelled correctly
Add value
When contributing bug reports:
Provide as full details of hardware and software as you have
Answer questions carefully: Ask questions the smart way: http://www.catb.org/esr/faqs/smart-questions.html
Be prepared to follow up queries / provide sufficient evidence to reproduce behaviour or provide pathological test cases
Provide a patch if possible: even if it's only pseudocode
When adding to / modifying FLOSS software:
Keep pristine sources that you have downloaded
Maintain patch series against pristine source
Talk to the originators of the software / current maintainers elsewhere
Follow upstream style if feasible / a consistent house style if not
Be generous in what you accept: be precise in what you put out
Don't produce licence conflicts - check and check again that your software can be distributed.
Don't apply inconsistent copyrights
When writing new FLOSS software / "freeing" prior commercial/closed code under a FLOSS licence
Make permissions explicit and publish under a well established FLOSS licence
Be generous to potential contributors and collaborators: render them every assistance so that they can help you better
Be generous in what you accept: be precise in what you put out
Don't produce licence conflicts - check and check again that your software can be distributed.
Don't apply inconsistent copyrights: software you write is your copyright at the outset until you assign it elsewhere
Contribute documentation / examples
Maintain a bugtracker and mailing lists for your software
If you are required to sign a contributor license agreement [CLA]
Ensure that you have the rights you purport to assign
Assign the minimum of rights necessary - if you can continue to allow full and free use of your code, do so
Meet any required code of conduct [CoC] stipulations in addition to the CLA
Always remember in all of this: just because you understand your code and your working practices doesn't mean that anyone else will. There is no automatic right to contribution nor any necessary assumption or precondition that collaborators will come forward. Just because you love your own code doesn't mean that it merits anyone else's interest or that anyone else should value it thereby "Just because it scratches your itch doesn't mean that it scratches anyone else's - or that it's actually any good / any use to anyone else"
This is a brief report on my last week work with Debian Reproducible Builds.
In week 4, I mostly worked on designing an interfaces and tackling different issues related to argument completion feature of diffoscope and in week 5 I worked on adding hiding .buildinfo from .changes files.
Update for last week s activities
I researched different diffoscope outputs. In reproducible-builds testing framework only differences of .buildinfo files are given but I needed diffoscope outputs for .changes files. Hence, I had to build packages locally using our experimental tool chain. My goal was to generate different outputs and to see how I can hide .buildinfo files from .changes.
I updated argument completion patch as per suggestions given by Paul Wise (pabs). Patch has been reviewed by Mattia Rizzolo, Holger Levsen and merged by Reiner Herrmann (deki) into diffoscope master. This patch closes #826711. Thanks all for support.
Last week I researched more on .changes and .buildinfo files. After getting guidelines from Lunar I was able to understand the need of this feature. I am in the middle of implementation of this particular problem.
Goal for upcoming week:
Finish the implementation of hiding .buildinfo from .changes
Start thinking on interfaces and discuss about different use cases.
I am thankful to Shirish Agarwal for helping me through visa process. But, unfortunately I won t get visa till 5th July. So I don t think, I would make it to debconf this year. I will certainly attend Debconf 2017.
Good news for me is I have passed mid-term evaluations of Google Summer of Code 2016. I will continue my work to improve Debian.
Even, I have post GSoC plans ready for Debian project ;)
Have a nice day :)
I am submitting a comment to Wen Wen and Chris Forman's Viewpoint on the Communications of the ACM, titled Economic and business dimensions: Do patent commons and standards-setting organizations help navigate patent thickets?. I believe my comment is worth sharing a bit more openly, so here it goes. Nevertheless, please refer to the original article; it makes very interesting and valid points, and my comment should be taken as an extra note on a great text only!
I was very happy to see an article with this viewpoint published. This article, however, mentions some points I believe should be further stressed out as problematic and important. Namely, still at the introduction, after mentioning that patents are intended to provide incentives for innovation by granting to inventors temporary monopoly rights , the next paragraph continues, The presence of patent thickets may create challenges for ICT producers. When introducing a new product, a firm must identify patents its product may infringe upon.
The authors continue explaining the needed process But this simple statement should be enough to explain how the patent system is broken and needs repair.
A requisite for patenting an invention was originally the inventive and non-obvious characteristics. Anything worth being granted a patent should be inventive enough, it should be non-obvious to an expert in the field.
When we see huge bodies of awarded (and upheld) patents falling in the case the authors mention, it becomes clear that the patent applications were not thoroughly researched prior to their patent grant. Sadly, long gone are the days where the United States Patent and Trademarks Office employed minds such as Albert Einstein's; nowadays, the office is more a rubber-stamping bureaucracy where most patents are awarded, and this very important requisite is left open to litigation: If somebody is found in breach of a patent, they might choose to defend the issue that the patent was obvious to an expert. But, of course, that will probably cost more in legal fees than settling for an agreement with the patent holder.
The fact that in our line of work we must take care to search for patents before releasing any work speaks a lot about the process. Patents are too easily granted. They should be way stricter; the occurence of an independent developer mistakenly (and innocently!) breaching a patent should be most unlikely, as patents should only be awarded to truly non-obvious solutions.
I bought some awful WiFi lightbulbs a few months ago. The short version: they introduced terrible vulnerabilities on your network, they violated the GPL and they were also just bad at being lightbulbs. Since then I've bought some other Internet of Things devices, and since people seem to have a bizarre level of fascination with figuring out just what kind of fractal of poor design choices these things frequently embody, I thought I'd oblige.
Today we're going to be talking about the KanKun SP3, a plug that's been around for a while. The idea here is pretty simple - there's lots of devices that you'd like to be able to turn on and off in a programmatic way, and rather than rewiring them the simplest thing to do is just to insert a control device in between the wall and the device andn ow you can turn your foot bath on and off from your phone. Most vendors go further and also allow you to program timers and even provide some sort of remote tunneling protocol so you can turn off your lights from the comfort of somebody else's home.
The KanKun has all of these features and a bunch more, although when I say "features" I kind of mean the opposite. I plugged mine in and followed the install instructions. As is pretty typical, this took the form of the plug bringing up its own Wifi access point, the app on the phone connecting to it and sending configuration data, and the plug then using that data to join your network. Except it didn't work. I connected to the plug's network, gave it my SSID and password and waited. Nothing happened. No useful diagnostic data. Eventually I plugged my phone into my laptop and ran adb logcat, and the Android debug logs told me that the app was trying to modify a network that it hadn't created. Apparently this isn't permitted as of Android 6, but the app was handling this denial by just trying again. I deleted the network from the system settings, restarted the app, and this time the app created the network record and could modify it. It still didn't work, but that's because it let me give it a 5GHz network and it only has a 2.4GHz radio, so one reset later and I finally had it online.
The first thing I normally do to one of these things is run nmap with the -O argument, which gives you an indication of what OS it's running. I didn't really need to in this case, because if I just telnetted to port 22 I got a dropbear ssh banner. Googling turned up the root password ("p9z34c") and I was logged into a lightly hacked (and fairly obsolete) OpenWRT environment.
It turns out that here's a whole community of people playing with these plugs, and it's common for people to install CGI scripts on them so they can turn them on and off via an API. At first this sounds somewhat confusing, because if the phone app can control the plug then there clearly is some kind of API, right? Well ha yeah ok that's a great question and oh good lord do things start getting bad quickly at this point.
I'd grabbed the apk for the app and a copy of jadx, an incredibly useful piece of code that's surprisingly good at turning compiled Android apps into something resembling Java source. I dug through that for a while before figuring out that before packets were being sent, they were being handed off to some sort of encryption code. I couldn't find that in the app, but there was a native ARM library shipped with it. Running strings on that showed functions with names matching the calls in the Java code, so that made sense. There were also references to AES, which explained why when I ran tcpdump I only saw bizarre garbage packets.
But what was surprising was that most of these packets were substantially similar. There were a load that were identical other than a 16-byte chunk in the middle. That plus the fact that every payload length was a multiple of 16 bytes strongly indicated that AES was being used in ECB mode. In ECB mode each plaintext is split up into 16-byte chunks and encrypted with the same key. The same plaintext will always result in the same encrypted output. This implied that the packets were substantially similar and that the encryption key was static.
Some more digging showed that someone had figured out the encryption key last year, and that someone else had written some tools to control the plug without needing to modify it. The protocol is basically ascii and consists mostly of the MAC address of the target device, a password and a command. This is then encrypted and sent to the device's IP address. The device then sends a challenge packet containing a random number. The app has to decrypt this, obtain the random number, create a response, encrypt that and send it before the command takes effect. This avoids the most obvious weakness around using ECB - since the same plaintext always encrypts to the same ciphertext, you could just watch encrypted packets go past and replay them to get the same effect, even if you didn't have the encryption key. Using a random number in a challenge forces you to prove that you actually have the key.
At least, it would do if the numbers were actually random. It turns out that the plug is just calling rand(). Further, it turns out that it never calls srand(). This means that the plug will always generate the same sequence of challenges after a reboot, which means you can still carry out replay attacks if you can reboot the plug. Strong work.
But there was still the question of how the remote control works, since the code on github only worked locally. tcpdumping the traffic from the server and trying to decrypt it in the same way as local packets worked fine, and showed that the only difference was that the packet started "wan" rather than "lan". The server decrypts the packet, looks at the MAC address, re-encrypts it and sends it over the tunnel to the plug that registered with that address.
That's not really a great deal of authentication. The protocol permits a password, but the app doesn't insist on it - some quick playing suggests that about 90% of these devices still use the default password. And the devices are all based on the same wifi module, so the MAC addresses are all in the same range. The process of sending status check packets to the server with every MAC address wouldn't take that long and would tell you how many of these devices are out there. If they're using the default password, that's enough to have full control over them.
There's some other failings. The github repo mentioned earlier includes a script that allows arbitrary command execution - the wifi configuration information is passed to the system() command, so leaving a semicolon in the middle of it will result in your own commands being executed. Thankfully this doesn't seem to be true of the daemon that's listening for the remote control packets, which seems to restrict its use of system() to data entirely under its control. But even if you change the default root password, anyone on your local network can get root on the plug. So that's a thing. It also downloads firmware updates over http and doesn't appear to check signatures on them, so there's the potential for MITM attacks on the plug itself. The remote control server is on AWS unless your timezone is GMT+8, in which case it's in China. Sorry, Western Australia.
It's running Linux and includes Busybox and dnsmasq, so plenty of GPLed code. I emailed the manufacturer asking for a copy and got told that they wouldn't give it to me, which is unsurprising but still disappointing.
The use of AES is still somewhat confusing, given the relatively small amount of security it provides. One thing I've wondered is whether it's not actually intended to provide security at all. The remote servers need to accept connections from anywhere and funnel decent amounts of traffic around from phones to switches. If that weren't restricted in any way, competitors would be able to use existing servers rather than setting up their own. Using AES at least provides a minor obstacle that might encourage them to set up their own server.
Overall: the hardware seems fine, the software is shoddy and the security is terrible. If you have one of these, set a strong password. There's no rate-limiting on the server, so a weak password will be broken pretty quickly. It's also infringing my copyright, so I'd recommend against it on that point alone.
In a prior post,
I gave an overview of the OpenStack CI system and how jobs were
started. In that I said
(It is a gross oversimplification, but for the purposes of OpenStack
CI, Jenkins is pretty much used as a glorified ssh/scp wrapper. Zuul
Version 3, under development, is working to remove the need for
Jenkins to be involved at all).
Well some recent security issues with Jenkins and other changes has
led to a roll-out of what is being called Zuul 2.5, which has indeed
removed Jenkins and makes extensive use of Ansible as the basis for
running CI tests in OpenStack. Since I already had the diagram, it
seems worth updating it for the new reality.
OpenStack CI Overview
While previous post was really focused on the image-building
components of the OpenStack CI system, overview is the same but more
focused on the launchers that run the tests.
The process starts when a developer uploads their code to
gerrit via the git-review tool. There is no further action
required on their behalf and the developer simply waits for
results of their jobs.
Gerrit provides a JSON-encoded "fire-hose" output of everything
happening to it. New reviews, votes, updates and more all get sent
out over this pipe. Zuul is the overall scheduler
that subscribes itself to this information and is responsible for
managing the CI jobs appropriate for each change.
Zuul has a configuration
that tells it what jobs to run for what projects. Zuul can do lots
of interesting things, but for the purposes of this discussion we
just consider that it puts the jobs it wants run into gearman
for a launcher to consume. gearman is a
job-server; as they explain it "[gearman] provides a generic
application framework to farm out work to other machines or
processes that are better suited to do the work". Zuul puts into
gearman basically a tuple (job-name,node-type) for each
job it wants run, specifying the unique job name to run and what
type of node it should be run on.
A group of Zuul launchers
are subscribed to gearman as workers. It is these Zuul
launchers that will consume the job requests from the queue and
actually get the tests running. However, a launcher needs two
things to be able to run a job a job definition (what to actually
do) and a worker node (somewhere to do it).
The first part what to do is provided by job-definitions
stored in external YAML files. The Zuul launcher knows how to
process these files (with some help from Jenkins Job Builder, which
despite the name is not outputting XML files for Jenkins to
consume, but is being used to help parse templates and macros
within the generically defined job definitions). Each Zuul
launcher gets these definitions pushed to it constantly by Puppet,
thus each launcher knows about all the jobs it can run
automatically. Of course Zuul also knows about these same job
definitions; this is the job-name part of the tuple we said it
put into gearman.
The second part somewhere to run the test takes some more
explaining. To the next point...
Several cloud companies donate capacity in their clouds for
OpenStack to run CI tests. Overall, this capacity is managed by a
customized management tool called nodepool
(you can see the details of this capacity at any given time by
checking the nodepool configuration).
Nodepool watches the gearman queue and sees what requests are
coming out of Zuul. It looks at node-type of jobs in the queue
(i.e. what platform the job has requested to run on) and decides
what types of nodes need to start and which cloud providers have
capacity to satisfy demand.
Nodepool will start fresh virtual machines (from images built daily
as described in the prior post), monitor their start-up and, when
they're ready, put a new "assignment job" back into gearman with
the details of the fresh node. One of the active Zuul launchers
will pick up this assignment job and register the new node to
itself.
At this point, the Zuul launcher has what it needs to actually get
jobs started. With an fresh node registered to it and waiting for
something to do, the Zuul launcher can advertise its ability to
consume one of the waiting jobs from the gearman queue. For
example, if a ubuntu-trusty node is provided to the Zuul
launcher, the launcher can now consume from gearman any job it
knows about that is intended to run on an ubuntu-trusty node
type. If you're looking at the launcher code
this is driven by the NodeWorker class you can see this being
created in response to an assignment via
LaunchServer.assignNode.
To actually run the job where the "job hits the metal" as it were
the Zuul launcher will dynamically construct an Ansible playbook to run. This
playbook is a concatenation of common setup and teardown operations
along with the actual test scripts the jobs wants to run. Using
Ansible to run the job means all the flexibility an orchestration
tool provides is now available to the launcher. For example, there
is a custom console streamer
library that allows us to live-stream the console output for the
job over a plain TCP connection, and there is the possibility to
use projects like ARA
for visualisation of CI runs. In the future, Ansible will allow
for better coordination when running multiple-node testing jobs
after all, this is what orchestration tools such as Ansible are
made for! While the Ansible run can be fairly heavyweight
(especially when you're talking about launching thousands of jobs
an hour), the system scales horizontally with more launchers able
to consume more work easily.
When checking your job results on logs.openstack.org you will
see a _zuul_ansible directory now which contains copies of the
inventory, playbooks and other related files that the launcher used
to do the test run.
Eventually, the test will finish. The Zuul launcher will put the
result back into gearman, which Zuul will consume (log copying
is interesting but a topic for another day). The testing node will
be released back to nodepool, which destroys it and starts all over
again nodes are not reused and also have no sensitive details on
them, as they are essentially publicly accessible. Zuul will wait
for the results of all jobs for the change and post the result back
to Gerrit; it either gives a positive vote or the dreaded negative
vote if required jobs failed (it also handles merges to git, but
that is also a topic for another day).
Work will continue within OpenStack Infrastructure to further enhance
Zuul; including better support for multi-node jobs and "in-project"
job definitions (similar to the https://travis-ci.org/ model);
for full details see the spec.
Time passes. I had left several (too many?) pending things to be done un the quiet weeks between the end of the lective semestre and the beginning of muy Summer trip to Winter. But Saturday gets closer every moment... And our long trip to the South begins.
Among many other things, I wanted to avance with some Deb an stuff - both packaging and WRT keyring analysis. I want to contacto some people I left pending interactions with, but honestly, that will only come face to face un Capetown.
As to "real life", I hace too many pending issues at work to even begin with; I hope to get some time at South frica todo do some decent UNAM sysadmining. Also, I want to play the idea of using Git for my students' workflow (handing in projects and assignments, at least)... This can be interesting to talk with the Deb an colleagues about, actually.
As a Masters student, I'm making good advances, and will probably finish muy class work next semester, six months ahead of schedule, but muy thesis work so far has progressed way slower than what I'd like. I have at least a better defined topic and approach, so I'll start the writing phase soon.
And the personal life? Family? I am more complete and happy than ever before. My life su completely different from two years ago. Yes, that was obvious. But it's also the only thing I can come up with. Having twin babies (when will they make the transition from "babies" to "kids"? No idea... We will find out as it comes) is more than beautiful, more than great. Our life has changed in every possible aspect. And yes, I admire my loved Regina for all of the energy and love she puts on the babies... Life is asymetric, I am out for most of the day... Mommy is always there.
As I said, happier than ever before.
texlive-bin/2016.20160513.41080-3 has been uploaded to unstable,
featuring support for FORCE_SOURCE_DATE. See the last post
for details on it.
doxygen/1.4.4-1 has been uploaded to unstable, fixing #822197
(upstream bug), which caused
some generated html file to contain unreproducible memory addresses of Python
objects used at build time.
debhelper/9.20160618 has been uploaded to unstable, fixing #824490,
which instructs ant to not save the username of the build user in the
generated files. Original patch by Emmanuel Bourg.
epydoc/3.0.1+dfsg-13 and 3.0.1+dfsg-14 have been uploaded by Kenneth J.
Pronovici which fixes nondeterministic ordering issues in generated
documentation and removes memory addresses. Original patches (#825968
and #827416) by Sascha Steinbiss.
With this upload of texlive-bin we decided to stop keeping our patched fork of
as most of the patches for SOURCE_DATE_EPOCH support had been integrated
upstream already, and the last one (making FORCE_SOURCE_DATE default to 1)
had been refused. So, we are now going to let the archive be rebuilt against
unstable's texlive-bin and see how many packages will become unreproducible
with this change; once enough data will be collected we will ponder whether
FORCE_SOURCE_DATE should be exported by helper tools (such as debhelper) or
manually exported by every package that needs it.
(For those wondering: we still recommend to follow SOURCE_DATE_EPOCH always
and don't recommend other projects to implement FORCE_SOURCE_DATE )
With the drop of texlive-bin we now have only three modified packages in our
experimental repository.
Reproducible work in other projects
Ed Maste sent a patch to have
ELF Tool Chain's elfcopy support SOURCE_DATE_EPOCH.
Ed Maste changed
FreeBSD's ar(1) to have a reproducible output by default when invoked with -s.
Ed Maste merged changes
from NetBSD's makefs to FreeBSD's to add a command line option for setting the
timestamp.
Holger Levsen spent some time thinking and documenting how to store package notes
and issues in a multi-distribution friendly manner, so that we can share
these issues and notes between reproducible efforts across different projects.
Thoughts we had about this topic
during the Athens meeting last December are included in the updated README.
Uploads with reproducibility fixes that currently fail to build:
ruby2.3/2.3.1-3 by Christian Hofstaedtler, avoids unreproducible rbconfig.rb files by always using bash for building.
Patches submitted that have not made their way to the archive yet:
#827109 against asciijump by Reiner Herrmann: sort source files for deterministic linking order.
#827112 against boswars by Reiner Herrmann: sort source files for deterministic linking order.
#827114 against overgod by Reiner Herrmann: use C locale for sorting source files.
#827115 against netpbm by Alexis Bienven e: honour SOURCE_DATE_EPOCH while generating output.
#827124 against funguloids by Reiner Herrmann: use C locale for sorting files.
#827145 against scummvm by Reiner Herrmann: don't embed extra fields in zip archive and build with proper host architecture.
#827150 against netpanzer by Reiner Herrmann: sort source files for deterministic linking order.
#827172 against reaver by Alexis Bienven e: sort object files for deterministic linking order.
#827187 against latex2html by Alexis Bienven e: iterate deterministic over Perl hashes; honour SOURCE_DATE_EPOCH for output; strip username from output; sort index keys.
#827313 against cherrypy3 by Sascha Steinbiss: prevent memory addresses in output.
#827361 against matplotlib by Alexis Bienven e: honour SOURCE_DATE_EPOCH in output and sort keys while iterating over dict.
#827382 against dwarfutils by Reiner Herrmann: fix array size, which caused memory from outside a table to be embedded into output.
#827384 against skytools3 by Sascha Steinbiss: use stable sorting order and remove timestamps from documentation.
#827419 against ldaptor by Sascha Steinbiss: sort list of input files and prevent home directory from leaking into documentation.
#827546 against git-buildpackage by Sascha Steinbiss: replace timestamps in documentation with changelog date; prevent temporary paths in documentation.
#827572 against xprobe by Reiner Herrmann: sort list of object files in static library archives.
Package reviews
36 reviews have been added, 12 have been updated and 31 have been removed in this week.
17 FTBFS bugs have been reported by Chris Lamb, Santiago Vila and Dominic Hargreaves.
diffoscope development
Satyam worked on argument completion (#826711) for diffoscope.
strip-nondeterminism development
Mattia Rizzolo uploaded strip-nondeterminism 0.019-1~bpo8+1 to
jessie-backports.
reprotest development
Ceridwen filed an Intent To Package (ITP) bug for reprotest as #827293.
tests.reproducible-builds.org
Mattia Rizzolo uploaded pbuilder 0.225 to unstable, providing built-in
support for eatmydata. We're planning to use it in armhf and i386 builders
where we don't build in tmpfs, to increase the build speed some more.
Valery Young reworked the appearance of the package page, hopefully making them more
intuitive and usable. In the process she changed the script generating them
to use a real templating system, thus improving
maintenance for the future.
Holger adjusted the scheduler to reschedule packages in state 'depwait' after
two days instead of three.
Mattia added the bug title next to the bug numbers in the notes.
Misc.
This week's edition was written by Mattia Rizzolo, Reiner Herrmann, Ed Maste and Holger Levsen and reviewed by a bunch of Reproducible builds folks on IRC.
I'm forty years old. I've been developing free software for twenty years.
A decade ago, I wrote a series of posts about my first
ten years of free software,
looking back over projects I'd developed. These retrospectives seem even more
valuable in retrospect; there are things in the old posts that jog my memory,
and other details I've forgotten by now.
So, I'm doing it again. Over the next two weeks (with a week off in the middle
for summer vacation), I'll be writing one post each day about a free software
project I've developed in the past decade.
We begin with Ikiwiki. I started it 10 years ago, and still use it
to this day; it's the engine that builds this website, and nearly all
my other websites, as well as wikis and websites belonging to tons and
tons of other projects, like NetBSD, X.org, Freedesktop.org, FreedomBox and
many other users.
Indeed I'm often reading a website and find myself wondering "hey.. is this
using Ikiwiki?", and glance at the html and find that yes, it is. So,
Ikiwiki is a reasonably successful and widely used peice of software,
at least in its niche.
More important to me, it was a static site generator before we knew what
those were. It wasn't the first, but it broke plenty of new ground. I'm
particularly proud of the way it combines a wiki with blogging support
seamlessly, and the incremental updating of the static pages including
updating wikilinks and backlinks. Some of these and other features are
still pretty unique to Ikiwiki despite the glut of other static site
generators available now.
Ikiwiki is written in Perl, which was great for getting lots of other
contributions (including many of its 113 plugins), but has also held it
back some lately. There are less Perl programmers these days. And over the
past decade, concurrency has become very important, but Ikiwiki's
implementation is stubbornly single threaded, and multithreading such a
Perl program is a losing propoisition. I occasionally threaten to rewrite
it in Haskell, but I doubt I will.
Ikiwiki has several developers now, and I'm the least active of them. I
stepped back because I can't write Perl very well anymore, and am mostly
very happy with how Ikiwiki works, so only pop up now and then when
something annoys me.
Next: twenty years of free software -- part 2 etckeeper
I'm back from the GTK hackfest in Toronto, Canada and mostly recovered from
jetlag, so it's time to write up my notes on what we discussed there.
Despite the hackfest's title, I was mainly there to talk about non-GUI
parts of the stack, and technologies that fit more closely in what could
be seen as the freedesktop.org platform than they do in GNOME. In particular,
I'm interested in Flatpak as a way to deploy self-contained "apps" in a
freedesktop-based, sandboxed runtime environment layered over
the Universal Operating System and
its many derivatives,
with both binary and source compatibility with other GNU/Linux distributions.
I'm mainly only writing about discussions I was directly involved in:
lots of what sounded like good discussion about the actual graphics toolkit
went over my head completely :-)
More notes, mostly from Matthias
Clasen, are available on the GNOME wiki.
In no particular order:
Thinking with portals
We spent some time discussing Flatpak's portals, mostly on Tuesday.
These are the components that expose a subset of desktop functionality
as D-Bus services that can be used by contained applications: they are
part of the security boundary between a contained app and the rest of the
desktop session. Android's intents are a similar concept seen elsewhere.
While the portals are primarily designed for Flatpak, there's no real
reason why they couldn't be used by other app-containment solutions
such as Canonical's Snap.
One major topic of discussion was their overall design and layout. Most
portals will consist of a UX-independent part in Flatpak itself, together
with a UX-specific implementation of any user interaction the portal
needs. For example, the portal for file selection has a D-Bus service
in Flatpak, which interacts with some UX-specific service that will
pop up a standard UX-specific "Open" dialog for GNOME and probably
other GTK environments, that dialog is in (a branch of) GTK.
A design principle that was reiterated in this discussion is that
the UX-independent part should do as much as possible, with the
UX-specific part only carrying out the user interactions that need
to comply with a particular UX design (in the GTK case, GNOME's design).
This minimizes the amount of work that needs to be redone for other
desktop or embedded environments, while still ensuring that the other
environments can have their chosen UX design. In particular, it's
important that, as much as possible, the security- and performance-sensitive
work (such as data transport and authentication) is shared between all
environments.
The aim is for portals to get the user's permission to carry out actions,
while keeping it as implicit as possible, avoiding an "are you sure?" step
where feasible. For example, if an application asks to open a file,
the user's permission is implicitly given by them selecting the file
in the file-chooser dialog and pressing OK: if they do not want this
application to open a file at all, they can deny permission by cancelling.
Similarly, if an application asks to stream webcam data, the UX we expect
is for GNOME's Cheese app (or a similar non-GNOME app) to appear, open
the webcam to provide a preview window so they can see what they are about
to send, but not actually start sending the
stream to the requesting app until the user has pressed a "Start" button.
When defining the API "contracts" to be provided by applications in that
situation, we will need to be clear about whether the provider is expected
to obtain confirmation like this: in most cases I would anticipate that
it is.
One security trade-off here is that we have to have a small amount of
trust in the providing app. For example, continuing the example of Cheese
as a webcam provider, Cheese could (and perhaps should) be a contained
app itself, whether via something like Flatpak, an LSM like AppArmor
or both. If Cheese is compromised somehow, then whenever it is running, it
would be technically possible for it to open the webcam, stream video and
send it to a hostile third-party application. We concluded that this is an
acceptable trade-off: each application needs to be trusted with the privileges
that it needs to do its job, and we should not put up barriers that are easy
to circumvent or otherwise serve no purpose.
The main (only?) portal so far is the file chooser, in which the contained
application asks the wider system to show an "Open..." dialog, and if the user
selects a file, it is returned to the contained application through a FUSE
filesystem, the document portal. The reference implementation of the UX for
this is in GTK, and is basically a GtkFileChooserDialog. The intention is
that other environments such as KDE will substitute their own equivalent.
Other planned portals include:
image capture (scanner/camera)
opening a specified URI
this needs design feedback on how it should work for non-http(s)
sharing content, for example on social networks (like Android's Sharing menu)
proxying joystick/gamepad input (perhaps via Wayland or FUSE, or perhaps
by modifying libraries like SDL with a new input source)
network proxies (GProxyResolver) and availability (GNetworkMonitor)
contacts/address book, probably vCard-based
notifications, probably based on freedesktop.org Notifications
video streaming (perhaps using Pinot, analogous to PulseAudio but for video)
Environment variables
GNOME on Wayland currently has a problem with environment variables:
there are some traditional ways to set environment variables for X11
sessions or login shells using shell script fragments (/etc/X11/Xsession.d,
/etc/X11/xinit/xinitrc.d, /etc/profile.d), but these do not apply to
Wayland, or to noninteractive login environments like cron and
systemd --user. We are also keen to avoid requiring a Turing-complete shell
language during session startup, because it's difficult to reason about
and potentially rather inefficient.
Some uses of environment variables can be dismissed as unnecessary or even
unwanted, similar to the statement in Debian Policy 9.9: "A program must not
depend on environment variables to get reasonable defaults." However,
there are two common situations where environment variables can be necessary
for proper OS integration: search-paths like $PATH, $XDG_DATA_DIRS and
$PYTHONPATH (particularly necessary for things like Flatpak), and
optionally-loaded modules like $GTK_MODULES and $QT_ACCESSIBILITY
where a package influences the configuration of another package.
There is a stopgap solution in GNOME's gdm display manager,
/usr/share/gdm/env.d, but this is gdm-specific and
insufficiently expressive to provide the functionality needed by
Flatpak: "set XDG_DATA_DIRS to its specified default value if unset,
then add a couple of extra paths".
pam_env comes closer PAM is run at every transition from "no user logged
in" to "user can execute arbitrary code as themselves" but it doesn't
support .d fragments, which are required if we want distribution packages
to be able to extend search paths. pam_env also turns off per-user
configuration by default, citing security concerns.
I'll write more about this when I have a concrete proposal for how to solve it.
I think the best solution is probably a PAM module similar to pam_env but
supporting .d directories, either by modifying pam_env directly or
out-of-tree, combined with clarifying what the security concerns for
per-user configuration are and how they can be avoided.
Relocatable binary packages
On Windows and OS X, various GLib APIs automatically discover where the
application binary is located and use search paths relative to that;
for example, if C:\myprefix\bin\app.exe is running, GLib might put
C:\myprefix\share into the result of g_get_system_data_dirs(),
so that the application can ask to load app/data.xml from the data
directories and get C:\myprefix\share\app\data.xml. We would like
to be able to do the same on Linux, for example so that the apps in a
Flatpak or Snap package can be constructed from RPM or dpkg packages without
needing to be recompiled for a different --prefix, and so that other
third-party software packages like the games on Steam and gog.com can
easily locate their own resources.
Relatedly, there are currently no well-defined semantics for what happens
when a .desktop file or a D-Bus .service file has Exec=./bin/foo.
The meaning of Exec=foo is well-defined (it searches $PATH) and the
meaning of Exec=/opt/whatever/bin/foo is obvious. When this came up in
D-Bus previously, my assertion was that the meaning should be the same as
in .desktop files, whatever that is.
We agreed to propose that the meaning of a non-absolute path in a .desktop
or .service file should be interpreted relative to the directory
where the .desktop or .service file was found: for example, if
/opt/whatever/share/applications/foo.desktop says Exec=../../bin/foo,
then /opt/whatever/bin/foo would be the right thing to execute.
While preparing a mail to the freedesktop and D-Bus mailing lists proposing
this, I found that I had
proposed the same thingalmost 2 years ago...
this time I hope I can actually make it happen!
Flatpak and OSTree bug fixing
On the way to the hackfest, and while the discussion moved to topics that I
didn't have useful input on, I spent some time fixing up the Debian packaging
for Flatpak and its dependencies. In particular, I did my first upload
as a co-maintainer of bubblewrap,
uploaded ostree to unstable (with the known limitation that the grub, dracut
and systemd integration is missing for now
since I haven't been able to test it yet), got
most of the way through packaging Flatpak 0.6.5 (which I'll upload soon),
cherry-picked the right patches to make ostree compile on Debian 8 in an effort
to make backports trivial, and spent some time disentangling
a flatpak test failure
which was breaking the Debian package's installed-tests.
I'm still looking into
ostree test failures on little-endian MIPS,
which I was able to reproduce on a Debian porterbox just before the end of the
hackfest.
OSTree + Debian
I also had some useful conversations with developers from
Endless, who recently opened up a version of their
OSTree build scripts
for public access. Hopefully that information brings me a bit closer to
being able to publish a walkthrough for
how to deploy a simple Debian derivative using OSTree
(help with that is very welcome of course!).
GTK life-cycle and versioning
The life-cycle of GTK releases has already been
mentioned here and elsewhere, and there are some interesting responses
in the comments on my earlier blog post.
It's important to note that what we
discussed at the hackfest is only a proposal: a hackfest discussion between
a subset of the GTK maintainers and a small number of other GTK users
(I am in the latter category)
doesn't, and shouldn't, set policy for all of GTK or for all of GNOME. I
believe the intention is that the GTK maintainers will discuss the proposals
further at GUADEC, and make a decision after that.
As I said before, I hope that being more realistic about API and ABI
guarantees can avoid GTK going too far towards either of the possible
extremes: either becoming unable to advance because it's too constrained
by compatibility, or breaking applications because it isn't constrained
enough. The current situation, where it is meant to be compatible within
the GTK 3 branch but in practice applications still sometimes break,
doesn't seem ideal for anyone, and I hope we can do better in future.
Acknowledgements
Thanks to everyone involved, particularly:
Matthias Clasen, who organised the hackfest and took a lot of notes
Allison Lortie, who provided on-site cat-herding and led us to some
excellent restaurants
Red Hat Inc., who provided the venue (a conference room
in their Toronto office), snacks, a lot of coffee, and several participants
my employers Collabora Ltd., who sponsored my travel and accomodation
This year a significant number of students are working on RTC-related projects as part of Google Summer of Code, under the umbrella of the Debian Project. You may have already encountered some of them blogging on Planet or participating in mailing lists and IRC.
WebRTC plugins for popular CMS and web frameworks
There are already a range of pseudo-WebRTC plugins available for CMS and blogging platforms like WordPress, unfortunately, many of them are either not releasing all their source code, locking users into their own servers or requiring the users to download potentially untrustworthy browser plugins (also without any source code) to use them.
Mesut is making plugins for genuinely free WebRTC with open standards like SIP. He has recently created the WPCall plugin for WordPress, based on the highly successful DruCall plugin for WebRTC in Drupal.
Keerthana has started creating a similar plugin for MediaWiki.
What is great about these plugins is that they don't require any browser plugins and they work with any server-side SIP infrastructure that you choose. Whether you are routing calls into a call center or simply using them on a personal blog, they are quick and convenient to install. Hopefully they will be made available as packages, like the DruCall packages for Debian and Ubuntu, enabling even faster installation with all dependencies.
Would you like to try running these plugins yourself and provide feedback to the students? Would you like to help deploy them for online communities using Drupal, WordPress or MediaWiki to power their web sites? Please come and discuss them with us in the Free-RTC mailing list.
You can read more about how to run your own SIP proxy for WebRTC in the RTC Quick Start Guide.
Finding all the phone numbers and ham radio callsigns in old emails
Do you have phone numbers and other contact details such as ham radio callsigns in old emails? Would you like a quick way to data-mine your inbox to find them and help migrate them to your address book?
Jaminy is working on Python scripts to do just that. Her project takes some inspiration from the Telify plugin for Firefox, which detects phone numbers in web pages and converts them to hyperlinks for click-to-dial. The popular libphonenumber from Google, used to format numbers on Android phones, is being used to help normalize any numbers found. If you would like to test the code against your own mailbox and address book, please make contact in the #debian-data channel on IRC.
A truly peer-to-peer alternative to SIP, XMPP and WebRTC
The team at Savoir Faire Linux has been busy building the Ring softphone, a truly peer-to-peer solution based on the OpenDHT distribution hash table technology.
Several students (Simon, Olivier, Nicolas and Alok) are actively collaborating on this project, some of them have been fortunate enough to participate at SFL's offices in Montreal, Canada. These GSoC projects have also provided a great opportunity to raise Debian's profile in Montreal ahead of DebConf17 next year.
Linux Desktop Telepathy framework and reSIProcate
Another group of students, Mateus, Udit and Balram have been busy working on C++ projects involving the Telepathy framework and the reSIProcate SIP stack. Telepathy is the framework behind popular softphones such as GNOME Empathy that are installed by default on the GNU/Linux desktop.
I previously wrote about starting a new SIP-based connection manager for Telepathy based on reSIProcate. Using reSIProcate means more comprehensive support for all the features of SIP, better NAT traversal, IPv6 support, NAPTR support and TLS support. The combined impact of all these features is much greater connectivity and much greater convenience.
The students are extending that work, completing the buddy list functionality, improving error handling and looking at interaction with XMPP.
Streamlining provisioning of SIP accounts
Currently there is some manual effort for each user to take the SIP account settings from their Internet Telephony Service Provider (ITSP) and transpose these into the account settings required by their softphone.
Pranav has been working to close that gap, creating a JAR that can be embedded in Java softphones such as Jitsi, Lumicall and CSipSimple to automate as much of the provisioning process as possible. ITSPs are encouraged to test this client against their services and will be able to add details specific to their service through Github pull requests.
The project also hopes to provide streamlined provisioning mechanisms for privately operated SIP PBXes, such as the Asterisk and FreeSWITCH servers used in small businesses.
Improving SIP support in Apache Camel and the Jitsi softphone
Apache Camel's SIP component and the widely known Jitsi softphone both use the JAIN SIP library for Java.
Nik has been looking at issues faced by SIP users in both projects, adding support for the MESSAGE method in camel-sip and looking at why users sometimes see multiple password prompts for SIP accounts in Jitsi.
If you are trying either of these projects, you are very welcome to come and discuss them on the mailing lists, Camel users and Jitsi users.
GSoC students at DebConf16 and DebConf17 and other events
Many of us have been lucky to meet GSoC students attending DebConf, FOSDEM and other events in the past. From this year, Google now expects the students to complete GSoC before they become eligible for any travel assistance. Some of the students will still be at DebConf16 next month, assisted by the regular travel budget and the diversity funding initiative. Nik and Mesut were already able to travel to Vienna for the recent MiniDebConf / LinuxWochen.at
As mentioned earlier, several of the students and the mentors at Savoir Faire Linux are based in Montreal, Canada, the destination for DebConf17 next year and it is great to see the momentum already building for an event that promises to be very big.
Explore the world of Free Real-Time Communications (RTC)
If you are interesting in knowing more about the Free RTC topic, you may find the following resources helpful:
Try Lumicall and Ring apps for Android with your freinds
RTC mentoring team 2016
We have been very fortunate to build a large team of mentors around the RTC-themed projects for 2016. Many of them are first time GSoC mentors and/or new to the Debian community. Some have successfully completed GSoC as students in the past. Each of them brings unique experience and leadership in their domain.
Helping GSoC projects in 2016 and beyond
Not everybody wants to commit to being a dedicated mentor for a GSoC student. In fact, there are many ways to help without being a mentor and many benefits of doing so.
Simply looking out for potential applicants for future rounds of GSoC and referring them to the debian-outreach mailing list or an existing mentor helps ensure we can identify talented students early and design projects around their capabilities and interests.
Testing the projects on an ad-hoc basis, greeting the students at DebConf and reading over the student wikis to find out where they are and introduce them to other developers in their area are all possible ways to help the projects succeed and foster long term engagement.
Google gives Debian a USD $500 grant for each student who completes a project successfully this year. If all 2016 students pass, that is over $10,000 to support Debian's mission.
After last week's bike ride, I had to pack my bags, and get on Monday
morning on a plane to Seattle. The time was so short that I even left
the bike mounted on the car.
So 8:15, plane to Frankfurt, and then plane to Seattle. To my big
surprise, the Lufthansa "extended leg room" seats were overly
generous; I could actually extend my foot completely and put it on the
back of the seat in front of me. Very good value when travelling in
economy The only downside was that these were "standard" not
"premium" economy, so the seat had leg room but was very narrow. And
with normal sized adults on either side of me, it was somewhat
difficult. The leg space allowed me to work on my laptop without
fearing the person in front of me will recline their seat and break my
screen (almost happened once).
The funniest thing when travelling is that food is always tricky: even
familiar food can be not what you expect. Case in point, me at the
salad buffet, seeing slices of green vegetables, and asking myself:
Are those jalapeno slices, or bell pepper slices? Hmm, I'm sure
they're bell pepper , which resulted in my first vegetable salad with
jalapeno. Would definitely recommend if you like spicy things!
Otherwise the trip was as usual, but shorter and more densely packed
with meetings with a one day exception: had the opportunity to
experience for the first time Whirlyball, which was
more fun and more difficult than it first looked. Also spent an
afternoon on Whidbey Island, which fortunately was also
the nicest day, weather-wise, of the week (all phone pictures, not
colour corrected, straight-out-of-phone):
A few more pictures
here. I
keep being amazed by the nature in this area, definitely my preferred
place in US from the relatively few I visited.
Had a nice dinner as well at Cafe Langley,
which surprisingly had reasonably-authentic Mediterranean food; the
Baba Ganoush was excellent.
Other than that outing, nothing worth describing, except that I really
missed my Zwift or outdoor rides. The experience of using the
stationary bikes in the hotel does not compare, so I resorted more to
running on the treadmill (hmm, Zwift for running, hmm ); if the foot
pod calibration is to be trusted, I continued to slightly improve my
1K, 1mi, and 5K times. Not bad, I might want to join some running
races as well this summer, but I need to take it easy though and make
sure to not get injured again.
And finally, week over, flew back home, slept a bit mid-day (which
will ruin my jet lag recovery program), unloaded my bike from the car
(and checked it still works), and andand got on the trainer and
did a Zwift ride. Jet lagged, but managed to beat my Watopia sprint
record by a tiny bit, and complete a new workout ( The Gorby ), which
was interesting. I can stop any time I want, definitely (I just need
to take a trip away from home ).
I came across Debian Backports recently after a kind soul suggested it in my Jessie Installation article. Backports provide the bridge between two stable releases of Debian. Say you are running Jessie stable in your system now. While it runs great, Jessie stable will be a bit outdated compared to the testing release. This is solved by Backports which recompiles and adjusts these testing packages for the stable environment. Backports can be included into the source file next to the stable sources unlike the testing sources which can misbehave and have a significant risk, Backports are much more compatible and lower risk.
However, Backports cannot be tested as extensively as Debian stable, and backports are provided on an as-is basis, with risk of incompatibilities with other components in Debian stable. Use with care! It is therefore recommended to only select single backported packages that fit your needs, and not use all available backports.
To add Backports to your system,
Open the sources.list file : sudo nano /etc/apt/sources.list
For jessie add the line : deb http://ftp.debian.org/debian jessie-backports main
For wheezy add the line : deb http://ftp.debian.org/debian wheezy-backports main
After any change be sure to run sudo apt-get update
To know more about Backports and to contribute consult the main page here and the wiki here. Happy hacking!
As some of the world knows full well by now, I've been noodling with Go
for a few years, working through its pros, its cons, and thinking a lot
about how humans use code to express thoughts and ideas. Go's got a lot of
neat use cases, suited to particular problems, and used in the right place,
you can see some clear massive wins.
I've started writing Debian tooling in Go, because it's a pretty natural fit.
Go's fairly tight, and overhead shouldn't be taken up by your operating system.
After a while, I wound up hitting the usual blockers, and started to build up
abstractions. They became pretty darn useful, so, this blog post is announcing
(a still incomplete, year old and perhaps API changing) Debian package for Go.
The Go importable name is pault.ag/go/debian. This contains a lot of utilities
for dealing with Debian packages, and will become an edited down "toolbelt"
for working with or on Debian packages.
Module Overview
Currently, the package contains 4 major sub packages. They're a changelog
parser, a control file parser, deb file format parser, dependency parser
and a version parser. Together, these are a set of powerful building blocks
which can be used together to create higher order systems with reliable
understandings of the world.
changelog
The first (and perhaps most incomplete and least tested) is a changelog file
parser.. This provides the
programmer with the ability to pull out the suite being targeted in the
changelog, when each upload was, and the version for each. For example, let's
look at how we can pull when all the uploads of Docker to sid took place:
funcmain()resp,err:=http.Get("http://metadata.ftp-master.debian.org/changelogs/main/d/docker.io/unstable_changelog")iferr!=nilpanic(err)allEntries,err:=changelog.Parse(resp.Body)iferr!=nilpanic(err)for_,entry:=rangeallEntriesfmt.Printf("Version %s was uploaded on %s\n",entry.Version,entry.When)
The output of which looks like:
Version 1.8.3~ds1-2 was uploaded on 2015-11-04 00:09:02 -0800 -0800
Version 1.8.3~ds1-1 was uploaded on 2015-10-29 19:40:51 -0700 -0700
Version 1.8.2~ds1-2 was uploaded on 2015-10-29 07:23:10 -0700 -0700
Version 1.8.2~ds1-1 was uploaded on 2015-10-28 14:21:00 -0700 -0700
Version 1.7.1~dfsg1-1 was uploaded on 2015-08-26 10:13:48 -0700 -0700
Version 1.6.2~dfsg1-2 was uploaded on 2015-07-01 07:45:19 -0600 -0600
Version 1.6.2~dfsg1-1 was uploaded on 2015-05-21 00:47:43 -0600 -0600
Version 1.6.1+dfsg1-2 was uploaded on 2015-05-10 13:02:54 -0400 EDT
Version 1.6.1+dfsg1-1 was uploaded on 2015-05-08 17:57:10 -0600 -0600
Version 1.6.0+dfsg1-1 was uploaded on 2015-05-05 15:10:49 -0600 -0600
Version 1.6.0+dfsg1-1~exp1 was uploaded on 2015-04-16 18:00:21 -0600 -0600
Version 1.6.0~rc7~dfsg1-1~exp1 was uploaded on 2015-04-15 19:35:46 -0600 -0600
Version 1.6.0~rc4~dfsg1-1 was uploaded on 2015-04-06 17:11:33 -0600 -0600
Version 1.5.0~dfsg1-1 was uploaded on 2015-03-10 22:58:49 -0600 -0600
Version 1.3.3~dfsg1-2 was uploaded on 2015-01-03 00:11:47 -0700 -0700
Version 1.3.3~dfsg1-1 was uploaded on 2014-12-18 21:54:12 -0700 -0700
Version 1.3.2~dfsg1-1 was uploaded on 2014-11-24 19:14:28 -0500 EST
Version 1.3.1~dfsg1-2 was uploaded on 2014-11-07 13:11:34 -0700 -0700
Version 1.3.1~dfsg1-1 was uploaded on 2014-11-03 08:26:29 -0700 -0700
Version 1.3.0~dfsg1-1 was uploaded on 2014-10-17 00:56:07 -0600 -0600
Version 1.2.0~dfsg1-2 was uploaded on 2014-10-09 00:08:11 +0000 +0000
Version 1.2.0~dfsg1-1 was uploaded on 2014-09-13 11:43:17 -0600 -0600
Version 1.0.0~dfsg1-1 was uploaded on 2014-06-13 21:04:53 -0400 EDT
Version 0.11.1~dfsg1-1 was uploaded on 2014-05-09 17:30:45 -0400 EDT
Version 0.9.1~dfsg1-2 was uploaded on 2014-04-08 23:19:08 -0400 EDT
Version 0.9.1~dfsg1-1 was uploaded on 2014-04-03 21:38:30 -0400 EDT
Version 0.9.0+dfsg1-1 was uploaded on 2014-03-11 22:24:31 -0400 EDT
Version 0.8.1+dfsg1-1 was uploaded on 2014-02-25 20:56:31 -0500 EST
Version 0.8.0+dfsg1-2 was uploaded on 2014-02-15 17:51:58 -0500 EST
Version 0.8.0+dfsg1-1 was uploaded on 2014-02-10 20:41:10 -0500 EST
Version 0.7.6+dfsg1-1 was uploaded on 2014-01-22 22:50:47 -0500 EST
Version 0.7.1+dfsg1-1 was uploaded on 2014-01-15 20:22:34 -0500 EST
Version 0.6.7+dfsg1-3 was uploaded on 2014-01-09 20:10:20 -0500 EST
Version 0.6.7+dfsg1-2 was uploaded on 2014-01-08 19:14:02 -0500 EST
Version 0.6.7+dfsg1-1 was uploaded on 2014-01-07 21:06:10 -0500 EST
control
Next is one of the most complex, and one of the oldest parts of go-debian,
which is the control file parser
(otherwise sometimes known as deb822). This module was inspired by the way
that the json module works in Go, allowing for files to be defined in code
with a struct. This tends to be a bit more declarative, but also winds up
putting logic into struct tags, which can be a nasty anti-pattern if used too
much.
The first primitive in this module is the concept of a Paragraph, a struct
containing two values, the order of keys seen, and a map of string to string.
All higher order functions dealing with control files will go through this
type, which is a helpful interchange format to be aware of. All parsing of
meaning from the Control file happens when the Paragraph is unpacked into
a struct using reflection.
The idea behind this strategy that you define your struct, and let the Control
parser handle unpacking the data from the IO into your container, letting you
maintain type safety, since you never have to read and cast, the conversion
will handle this, and return an Unmarshaling error in the event of failure.
Additionally, Structs that define an anonymous member of control.Paragraph
will have the raw Paragraph struct of the underlying file, allowing the
programmer to handle dynamic tags (such as X-Foo), or at least, letting
them survive the round-trip through go.
The default decoder
contains an argument, the ability to verify the input control file using an
OpenPGP keyring, which is exposed to the programmer through the
(*Decoder).Signer() function. If the passed argument is nil, it will not
check the input file signature (at all!), and if it has been passed, any
signed data must be found or an error will fall out of the NewDecoder call.
On the way out, the opposite happens, where the struct is introspected,
turned into a control.Paragraph, and then written out to the io.Writer.
Here's a quick (and VERY dirty) example showing the basics of reading and
writing Debian Control files with go-debian.
packagemainimport("fmt""io""net/http""strings""pault.ag/go/debian/control")typeAllowedPackagestructPackagestringFingerprintstringfunc(a*AllowedPackage)UnmarshalControl(instring)errorin=strings.TrimSpace(in)chunks:=strings.SplitN(in," ",2)iflen(chunks)!=2returnfmt.Errorf("Syntax sucks: '%s'",in)a.Package=chunks[0]a.Fingerprint=chunks[1][1:len(chunks[1])-1]returnniltypeDMUAstructFingerprintstringUidstringAllowedPackages[]AllowedPackage control:"Allow" delim:"," funcmain()resp,err:=http.Get("http://metadata.ftp-master.debian.org/dm.txt")iferr!=nilpanic(err)decoder,err:=control.NewDecoder(resp.Body,nil)iferr!=nilpanic(err)fordmua:=DMUAiferr:=decoder.Decode(&dmua);err!=niliferr==io.EOFbreakpanic(err)fmt.Printf("The DM %s is allowed to upload:\n",dmua.Uid)for_,allowedPackage:=rangedmua.AllowedPackagesfmt.Printf(" %s [granted by %s]\n",allowedPackage.Package,allowedPackage.Fingerprint)
Output (truncated!) looks a bit like:
...
The DM Allison Randal <allison@lohutok.net> is allowed to upload:
parrot [granted by A4F455C3414B10563FCC9244AFA51BD6CDE573CB]
...
The DM Benjamin Barenblat <bbaren@mit.edu> is allowed to upload:
boogie [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
dafny [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
transmission-remote-gtk [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
urweb [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
...
The DM <aelmahmoudy@sabily.org> is allowed to upload:
covered [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
dico [granted by 6ADD5093AC6D1072C9129000B1CCD97290267086]
drawtiming [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
fonts-hosny-amiri [granted by BD838A2BAAF9E3408BD9646833BE1A0A8C2ED8FF]
...
...
deb
Next up, we've got the deb module. This contains code to handle reading
Debian 2.0 .deb files. It contains a wrapper that will parse the control
member, and provide the data member through the
archive/tar interface.
Here's an example of how to read a .deb file, access some metadata, and
iterate over the tar archive, and print the filenames of each of the
entries.
dependency
The dependency package provides an interface to parse and compute
dependencies. This package is a bit odd in that, well, there's no other
library that does this. The issue is that there are actually two different
parsers that compute our Dependency lines, one in Perl (as part of dpkg-dev)
and another in C (in dpkg).
To date, this has resulted in me filing
threedifferentbugs.
I also found a broken package in the
archive,
which actually resulted in another bug being (totally accidentally)
already fixed.
I hope to continue to run the archive through my parser in hopes of finding
more bugs! This package is a bit complex, but it basically just returns what
amounts to be an AST
for our Dependency lines. I'm positive there are bugs, so file them!
funcmain()dep,err:=dependency.Parse("foo bar, baz, foobar [amd64] bazfoo [!sparc], fnord:armhf [gnu-linux-sparc]")iferr!=nilpanic(err)anySparc,err:=dependency.ParseArch("sparc")iferr!=nilpanic(err)for_,possi:=rangedep.GetPossibilities(*anySparc)fmt.Printf("%s (%s)\n",possi.Name,possi.Arch)
Gives the output:
foo (<nil>)
baz (<nil>)
fnord (armhf)
version
Right off the bat, I'd like to thank
Michael Stapelberg for letting me graft this
out of dcs and into the go-debian package.
This was nearly entirely his work (with a one or two line function I added
later), and was amazingly helpful to have. Thank you!
This module implements Debian version comparisons and parsing, allowing for
sorting in lists, checking to see if it's native or not, and letting the
programmer to implement smart(er!) logic based on upstream (or Debian)
version numbers.
This module is extremely easy to use and very straightforward, and not worth
writing an example for.
Final thoughts
This is more of a "Yeah, OK, this has been useful enough to me at this point
that I'm going to support this" rather than a "It's stable!" or even
"It's alive!" post. Hopefully folks can report bugs and help iterate on
this module until we have some really clean building blocks to build
solid higher level systems on top of. Being able to have multiple libraries
interoperate by relying on go-debian will be a massive ease.
I'm in need of more documentation, and to finalize some parts of the older
sub package APIs, but I'm hoping to be at a "1.0" real soon now.
* find somebody in Europe who prints good stickers and doesn't require illustrator (or other proprietary software) to submit files for non-rectangular shapes * find out which Debian team I should contact to submit the files so that they can be used by everybody interested.
But neither will happen today, nor probably tomorrow, because lazy O:-)
Edit: now that I'm awake I realized I forgot to thank @Enrico Zini Zini and MadameZou for their help in combining my two proposals in a better design.