Search Results: "will"

12 May 2015

Michal Čihař: python-gammu 2.2

After recent porting python-gammu to Python 3, it was quite obvious to me that new release will have some problems. Fortunately they have proven to be rather cosmetic and no big bugs were found so far. Anyway it's time to push the minor fixes to the users, so here comes python-gammu 2.2. As you can see, the changes are pretty small, but given that I don't expect much development in the future, it's good to release them early.

Filed under: English Gammu python-gammu SUSE Wammu 0 comments

11 May 2015

Julien Danjou: My interview about software tests and Python

I've recently been contacted by Johannes Hubertz, who is writing a new book about Python in German called "Softwaretests mit Python" which will be published by Open Source Press, Munich this summer. His book will feature some interviews, and he was kind enough to let me write a bit about software testing. This is the interview that I gave for his book. Johannes translated to German and it will be included in Johannes' book, and I decided to publish it on my blog today. Following is the original version. How did you come to Python?
I don't recall exactly, but around ten years ago, I saw more and more people using it and decided to take a look. Back then, I was more used to Perl. I didn't really like Perl and was not getting a good grip on its object system. As soon as I found an idea to work on if I remember correctly that was rebuildd I started to code in Python, learning the language at the same time. I liked how Python worked, and how fast I was to able to develop and learn it, so I decided to keep using it for my next projects. I ended up diving into Python core for some reasons, even doing things like briefly hacking on projects like Cython at some point, and finally ended up working on OpenStack. OpenStack is a cloud computing platform entirely written in Python. So I've been writing Python every day since working on it. That's what pushed me to write The Hacker's Guide to Python in 2013 and then self-publish it a year later in 2014, a book where I talk about doing smart and efficient Python. It had a great success, has even been translated in Chinese and Korean, so I'm currently working on a second edition of the book. It has been an amazing adventure! Zen of Python: Which line is the most important for you and why?
I like the "There should be one and preferably only one obvious way to do it". The opposite is probably something that scared me in languages like Perl. But having one obvious way to do it is and something I tend to like in functional languages like Lisp, which are in my humble opinion, even better at that. For a python newbie, what are the most difficult subjects in Python?
I haven't been a newbie since a while, so it's hard for me to say. I don't think the language is hard to learn. There are some subtlety in the language itself when you deeply dive into the internals, but for beginners most of the concept are pretty straight-forward. If I had to pick, in the language basics, the most difficult thing would be around the generator objects (yield). Nowadays I think the most difficult subject for new comers is what version of Python to use, which libraries to rely on, and how to package and distribute projects. Though things get better, fortunately. When did you start using Test Driven Development and why?
I learned unit testing and TDD at school where teachers forced me to learn Java, and I hated it. The frameworks looked complicated, and I had the impression I was losing my time. Which I actually was, since I was writing disposable programs that's the only thing you do at school. Years later, when I started to write real and bigger programs (e.g. rebuildd), I quickly ended up fixing bugs I already fixed. That recalled me about unit tests and that it may be a good idea to start using them to stop fixing the same things over and over again. For a few years, I wrote less Python and more C code and Lua (for the awesome window manager), and I didn't use any testing. I probably lost hundreds of hours testing manually and fixing regressions that was a good lesson. Though I had good excuses at that time it is/was way harder to do testing in C/Lua than in Python. Since that period, I have never stopped writing "tests". When I started to hack on OpenStack, the project was adopting a "no test? no merge!" policy due to the high number of regressions it had during the first releases. I honestly don't think I could work on any project that does not have at least a minimal test coverage. It's impossible to hack efficiently on a code base that you're not able to test in just a simple command. It's also a real problem for new comers in the open source world. When there are no test, you can hack something and send a patch, and get a "you broke this" in response. Nowadays, this kind of response sounds unacceptable to me: if there is no test, then I didn't break anything! In the end, it's just too much frustration to work on non tested projects as I demonstrated in my study of whisper source code. What do you think to be the most often seen pitfalls of TDD and how to avoid them best?
The biggest problems are when and at what rate writing tests. On one hand, some people starts to write too precise tests way too soon. Doing that slows you down, especially when you are prototyping some idea or concept you just had. That does not mean that you should not do test at all, but you should probably start with a light coverage, until you are pretty sure that you're not going to rip every thing and start over. On the other hand, some people postpone writing tests for ever, and end up with no test all or a too thin layer of test. Which makes the project with a pretty low coverage. Basically, your test coverage should reflect the state of your project. If it's just starting, you should build a thin layer of test so you can hack it on it easily and remodel it if needed. The more your project grow, the more you should make it sold and lay more tests. Having too detailed tests is painful to make the project evolve at the start. Having not enough in a big project makes it painful to maintain it. Do you think, TDD fits and scales well for the big projects like OpenStack?
Not only I think it fits and scales well, but I also think it's just impossible to not use TDD in such big projects. When unit and functional tests coverage was weak in OpenStack at its beginning it was just impossible to fix a bug or write a new feature without breaking a lot of things without even noticing. We would release version N, and a ton of old bugs present in N-2 but fixed in N-1 were reopened. For big projects, with a lot of different use cases, configuration options, etc, you need belt and braces. You cannot throw code in a repository thinking it's going to work ever, and you can't afford to test everything manually at each commit. That's just insane.

Craig Small: procps using GitLab CI

procps-ciThe procps project for a few years has been hosted at Gitorious. With the announcement that Gitorious has been acquired by GitLab and that all repositories need to move there, procps moved along to GitLab. At first I thought it would just be a like for like thing, but then I noticed that GitLab has this GitLab CI feature and had to try it out. CI here stands for Continuous Integration and is a way of automatically testing your program builds using a bunch of test scripts. procps already has a set of tests, with some a level of coverage that has room for improvement, so it was a good candidate to use for CI. The way GitLab works is they have a central control point that is linked to the git repo and you create runners, which are the systems that actually compile the programs and run the tests. The runners then feed back their results and GitLab CI shows it all in pretty red or green. The first problem was building this runner. Most of the documentation assumes you are building testing for Ruby. procps is built using C with the test scripts in TCL, so there was going to be some need to make some adjustments. I chose to use docker containers so that there was at least some isolation between the runners and the base system. I soon found the docker container I used (ruby:2.6 as suggested by GitLab) didn t have autopoint which mean autogen.sh failed so I had no configure script so no joy there. Now my second problem was I had never used docker before and beyond that it was some sort of container thing so like virtual machines lite, I didn t know much about it. The docker docs are very good and soon I had built my own custom docker image that had all the programs I need to compile and test procps. It s basically a cut-down Debian image with a few things like gettext-bin and dejagnu added in. Docker is not a full system With that all behind me and a few oh I need that too (don t forget you need git) moments we had a working CI runner. This was a Good Thing. You then find that your assumptions for your test cases may not always be correct. This is especially noticeable when testing something like procps which needs to work of the proc filesystem. A good example is, what uses session ID 1? Generally its init or systemd, but in Docker this is what everything runs as. A test case which assumes things about SID=1 will fail, as it did. This probably won t be a problem for testing a lot of normal programs that don t need to dig a deep into the host system as procps does, but it is something to remember. The docker environment looks a lot like a real system from the inside, but there are differences, so the lesson here is to write better tests (or fix the ones that failed, like I did). The runner and polling Now, there needs to be communication between the CI website and the runner, so the runner knows there is something for it to do. The gitlab runner has this setup rather well, except the runner is rather aggressive about its polling, hitting the website every 3 seconds. For someone on some crummy Australian Internet with low speeds and download quotas, this can get expensive in network resources. As there is on average an update once a week or so these seems rather excessive. My fix is pretty manual, actually its totally manual. I stop the daemon, then if I notice there are pending jobs I start the daemon, let it do its thing, then shut it down again. This is certainly not an optimal solution but works for now. I will look into doing something more clever later, possibly with webhooks.

Zlatan Todori : Reboot to roots

I am working in software development field for more then 5 years. During that period I came across of other developers which vast majority can be described as terrible people. That is what sets probably difference between hacker mind and average software engineer. People tend to think if they have position of software developer they are god-like and should be treated like that. Hackers know who they are and tend to be Socrates-wise "I know that I know nothing". As student of mechanical engineering (majoring in mechatronics) I didn't have much of programming so naturally I had some personal difficulties to grasp software field but once I got in I stopped almost totally doing my student duties as it didn't cross path with my love towards hacking. And now after that many years in software field, in which I mostly worked as freelancer for other people and companies (where it wasn't unusual that I got hired personally by a guy who couldn't or didn't know how to finish his work, I hand the code which he would present as his own work, but I don't mind as I needed the money to get on) and my work ended as proprietary - I am just fed up with all this nonsense. Someone would think what is problem there - well for me it is, as being part of Debian and wider Free software ecosystem I know proprietary software is not fostering better society in all together so it made me a bit unhappy about it. The thing that actually is really annoying me are asshole so-called software engineers and the most - bad behavior towards users. Users should be treated as supporters of your software and if proprietary company treats their users as shit then it's obvious that they don't care for the users but rather for the money and only money. Proprietary isn't here the only bad thing, open source community has it share of terrible responses - which comes naturally if you take the sheer amount of people in open source. I for instance was called on Debian IRC a peasant and even I sheep in my early days - but Debian as all communities has its bad seeds and sometimes even non-Debianities trolls hop on our channels. I didn't let myself put down because I personally got the chance to meet many of them and they change the flow of my life at that point. I am unhappy at the current state of things worldwide and personally but I have now made a decision that I am shifting towards being technical customer support engineer or for me better named as Happiness Hero. Why, one would ask because as software engineer you would make more? Well, firstly I don't care that much about money (or to be even more precise, as capitalism has made a sick money-grabbing society, you can never pay me enough as I am more worth then entire capitalistic system so in the end I don't care that much how I am paid) and secondly happiness hero can earn more then software engineer which latest #talkpay hashtag on Twitter showed. I have made IMHO many good pieces of code that will never be contributed as mine, but I achieved no happiness and rather achieved few months ago a rather bad burnout. My shift towards happiness hero should look like this - I am pretty well prepared technically for that position, I have experience with helping users and in general with people and I tend to be pretty good at that (but that we would need to ask them) and with part that I enjoy helping people to debug their technical problem so they can spend more time on others things (read family, creativity, hobbies) is something I believe will put me well into position to achieve happiness so I can create for myself time to finish damn faculty (where TBH professors showed that they can be one of the worst enemies of their students - and I am highly doubting in this generic way of schooling for centuries) and if I get some decent paycheck with this capitalistic system I will have a chance to sponsor a poor family that has child talented in artistic field - or to say SHARE something material for cultural and emotional gain. Take that capitalism! And the best part, I will start to believe more in myself so I have already few offers for that kind of position but am waiting for few more and I am screening companies to find one that fits me best - shares values with mine and has a great interaction with their users. One of the companies that makes me want to screen more companies is Buffer. When you read this and this you will see their way of life and perks are really something every company should do - then you will understand why I want to give my self a chance which I believe I finally deserved. And, no, I didn't apply for Buffer as I don't use their service nor did have time to read the two books needed for applying (which I think is really a great thing to foster people into getting more knowledge). I can only wish them best luck in future. I am currently doing one small software development thing (freelance) for next two weeks and after that I am hoping to get enough good offer to start with happiness job and retreat from software development for some good amount of time. That doesn't mean I will cease my Free software activities - on contrary I think I will have more time and even more will to work on Free software development in my spare time. So if anyone or any company that works with open source community has a position for happiness hero, get in touch with me and we will have a good talk. End point - I want to grow and become who am I, and not who I should be according to others.

10 May 2015

Eduard Sanou: Reproducible builds on Debian for GSoC 2015

This is the first blog post of a series I will be writing about my experiences contributing to Debian for Google Summer of Code 2015.

A bit about myself I m a Spanish student doing a master s in Computer Science in Barcelona. I graduated on Electrical Engineering (we call it Telecommunications here). I ve always been interested in computing and programming and I have worked on several projects on my own using C, python and go. My main interests in the field are programming languages, security, cryptography, distributed and decentralized systems and embedded systems. I m an advocate of free software, I try to use it as much as I m able to in my devices and also try to convince my friends and family of its benefits. I have been using GNU/Linux for nearly ten years as my main operating system and I have tried several *BSD s recently. One of my latest personal projects is a gameboy emulator written in C (still work in progress) which already plays many games (without sound though) . You can find other minor projects in my github page (I try to publish all the code I write online, under free software licence) After so many years of using free software and benefiting from it, I thought it was about time to contribute back! That s why I gave GSoC a try and applied to work on the Reproducible Builds project for Debian :) And I got accepted!

Reproducible Builds The idea behind this project is that currently many packages aren t built in a reproducible manner; that is, they contain timestamps, building machine name, unique IDs, and results from other processes that happen differently between machines, like file ordering in compressed files. The project aims to patch all the Debian packages / the building scripts in order to generate the same binary (bit by bit) independently of the machine, timezone, etc where it is built. This way, a cryptographic hash of the built package can be distributed and many people can rebuild the package to verify that the binary in the repositories indeed corresponds to the right source code by means of comparing the hash.

Motivation One of the main advantages of the free software is that source code is available for peer review. This makes it easier for users to trust their software, as they can check the source to verify that the program is not doing anything bad. Even if the user doesn t do that, they can trust the wider community with that task. But many distributions serve packages in binary form, so how do we know that the binary comes from the publicly available source code? The current solution is that the developers who build the packages sign them cryptographically; but this lands all the trust to the developer and the machines used for building. I became interested in this topic with a very nice talk given at 31c3 by Mike Perry from Tor and Seth Schoen from the EFF. They focused on reproducible builds applied to the tor browser bundle, showing a small demo of how a building machine could be compromised to add hidden functionalities when compiling code (so that the developer could be signing a compromised package without their knowledge).

Benefits There are two main groups who benefit with reproducible builds:

For users The user can be more secure when installing packages in binary form since they don t need to trust a specific developer or building machine. Even if they don t rebuild the package by themselves to verify it, there would be others doing so, who will easily alert the community when the binary doesn t match the source code.

For developers The developer no longer has the responsibility of using his identity to sign the package for wide distribution, nor is that much responsible of the damage to users if their machine is compromised to alter the building process, since the community will easily detect it and alert them. This later point is specially useful with secure and privacy aware software. The reason is that there are many powerful organizations around the world with interest on having backdoors in widely used software, be it to spy on users or to target specific groups of people. Considering the amount of money these organizations have for such purposes, it s not hard to imagine that they could try to blackmail developers into adding such backdoors on the built packages. Or they could try to compromise the building machine. With reproducible builds the developer is safer, as such attack is no longer useful.

Reproducible Builds in Debian The project kicked-off at Debian at mid 2013 , leaded by Lunar and soon followed by many other developers (h01ger, deki, mapreri, ). Right now about 80% of the packages in the unstable branch of Debian can be built reproducibly. The project is very active, with many developers sending patches every week. A machine running Jenkins (which was set up at the end of 2012 for other purposes) is being used since late 2014 to continuously build packages in different settings to check if they are built reproducibly or not. In order to analyze why packages fail to build reproducibly, a tool called debbindiff has been developed, which is able to output in text or html form a smart diff of two builds. Another tool called strip-nondeterminism has been developed to remove non-determinism from files during the building process. For this GSoC I plan on helping improving these tools (mainly debbindiff), write many patches to achieve reproducibility in more packages and write documentation about it. Some of the packages fail to build reproducibly due to specifics of their building processes, whereas others fail due to the usage of toolchains that add non-determinism. I ll focus more on the later ones in order to improve the state more packages. akira will also be working on this project for this GSoC. Finally, I just want to add that I m looking forward to contribute to Debian, meet the community and learn more about the internals of this awesome distribution!

Petter Reinholdtsen: Norwegian citizens now required by law to give their fingerprint to the police

5 days ago, the Norwegian Parliament decided, unanimously, that all citizens of Norway, no matter if they are suspected of something criminal or not, are required to give fingerprints to the police (vote details from Holder de ord). The law make it sound like it will be optional, but in a few years there will be no option any more. The ID will be required to vote, to get a bank account, a bank card, to change address on the post office, to receive an electronic ID or to get a drivers license and many other tasks required to function in Norway. The banks plan to stop providing their own ID on the bank cards when this new national ID is introduced, and the national road authorities plan to change the drivers license to no longer be usable as identity cards. In effect, to function as a citizen in Norway a national ID card will be required, and to get it one need to provide the fingerprints to the police. In addition to handing the fingerprint to the police (which promised to not make a copy of the fingerprint image at that point in time, but say nothing about doing it later), a picture of the fingerprint will be stored on the RFID chip, along with a picture of the face and other information about the person. Some of the information will be encrypted, but the encryption will be the same system as currently used in the passports. The codes to decrypt will be available to a lot of government offices and their suppliers around the globe, but for those that do not know anyone in those circles it is good to know that the encryption is already broken. And they can be read from 70 meters away. This can be mitigated a bit by keeping it in a Faraday cage (metal box or metal wire container), but one will be required to take it out of there often enough to expose ones private and personal information to a lot of people that have no business getting access to that information. The new Norwegian national IDs are a vehicle for identity theft, and I feel sorry for us all having politicians accepting such invasion of privacy without any objections. So are the Norwegian passports, but it has been possible to function in Norway without those so far. That option is going away with the passing of the new law. In this, I envy the Germans, because for them it is optional how much biometric information is stored in their national ID. And if forced collection of fingerprints was not bad enough, the information collected in the national ID card register can be handed over to foreign intelligence services and police authorities, "when extradition is not considered disproportionate". Update 2015-05-12: For those unable to believe that the Parliament really could make such decision, I wrote a summary of the sources I have for concluding the way I do (Norwegian Only, as the sources are all in Norwegian).

9 May 2015

Daniel Silverstone: Kimchi Trial 1 - Part 1

I have spent today making my first ever batch of Kimchi. I have been documenting it in photos as I go, but thought I'd write up what I did so that if anyone else fancies having a go too, we can compare results. For a start, this recipe is nowhere near "traditional" because I don't have access to certain ingredients such as glutinous rice flour. I'm sure if I searched in many of the asian supermarkets around the city centre I could find it, but I'm lazy so I didn't even try. I am not writing this up as a traditional recipe because I'm kinda making it up as I go along, with hints from various sources including the great and wonderful Maangchi whose YouTube channel I follow. Observant readers or followers of Maangchi will recognise the recipe as being close to her Easy Kimchi recipe, however since I'm useless, it won't be exact. If this batch turns out okay then I'll write it up as a proper recipe for you to follow. I started off with three Chinese Leaf cabbages which seemed to be about 1.5kg or so once I'd stripped the less nice outer leaves, cored and chopped them. Chopped up cabbage I then soaked and drained the cabbage in cold water... Soaking cabbage ...before sprinkling a total of one third of a cup of salt over the cabbage and mixing it to distribute the salt. Salted Cabbage Then I returned to the cabbage every 30 minutes to re-mix it a total of three times. After the cabbage had been salted for perhaps 1h45m or so, I rinsed it out. Maangchi recommends washing the cabbage three times so that's what I did before setting it out to drain in a colander. Drained salted cabbage 1h45m later Maangchi then calls for the creation of a porridge made from sweet rice flour which it turns out is very glutinous. Since I lack the ability to get that flour easily I substituted cornflour which I hope will be okay and then continued as before. One cup of water, one third of a cup of cornflour was heated until it started to bubble and then one sixth of a cup of sugar was added. Stirred throughout, once it went translucent I turned the heat off and proceeded. Porridge thingy One half of a red onion, a good thumb (once peeled) of ginger, half a bulb of garlic and one third of a cup of fish sauce went into a mini-zizzer. I then diagonal-chopped about five spring onions, and one leek, before cutting a fair sized carrot into inch long pieces before halving and then thinly slicing it. Maangchi calls for julienned carrots but I am not that patient. Veggybits Into the cooled porridge I put two thirds of a cup of korean hot pepper flakes (I have the coarse, but a mix of coarse and fine would possibly be better), the zizzed onion/garlic/ginger/fish mix and the vegetables... All in the pan ...before mixing that thoroughly with a spatula. Mixed vegetables Next came the messy bit (I put latex gloves on, and thoroughly washed my gloved hands for this). Into my largest mixing bowl I put a handful of the drained cabbage into the bowl and a handful of the pepper mix. Thoroughly mixing this before adding another handful of cabbage and pepper mix I repeated until all the cabbage and hot pepper mixed vegetables are well combined. I really got your arms into it, squishing it around, separating the leek somewhat, etc. Bowl of kimchi As a final task, I scooped the kimchi into a clicklok type box, pressing it down firmly to try and remove all the air bubbles, before sealing it in for a jolly good fermenting effort. I will have to remove a little tonight for our dinner (beef strips marinated in onion, ginger and garlic, with kimchi on rice) but the rest will then sit to ferment for a bit. Expect a part-2 with the report from tonight's dinner and a part-3 with the results after fermentation. Box of kimchi
As an aside, I got my hot pepper flakes from Sous Chef who, it turns out, also stock glutinous rice flour -- I may have to get some more from them in the future. (#notsponsored)

John Goerzen: First impressions and review of OwnCloud

In my recent post (I give up on Google), a lot of people suggested using OwnCloud as a replacement for several Google services. I ve been playing around with it for a few days, and it is something of a mix of awesome and disappointing, in my opinion. Files OwnCloud started as a file-sync tool, somewhat akin to Google Drive and Dropbox. It has clients for every platform, and it is also a client for every platform: you can have subfolders of your OwnCloud installation stored on WebDav, *FTP*, Google Drive, Dropbox, you name it. It is a pretty nice integrator of other storage services, and provides the only way to use some of them on Linux (*cough* Google Drive *cough*) One particularly interesting feature is the live editing in the browser of ODT, DOCX, and TXT files. This is somewhat similar to Google Docs and the only such thing I ve seen in Open Source software. It writes changes directly back to the documents and, in my limited testing, seems to work well. A very nice feature! I ve tested the syncing only on Linux so far, but it looks solid. There are two surprising issues, however: there is no deduplication and no delta-uploads. Add 10 bytes to the end of a 1GB file, and you re-upload the 1GB file. Thankfully the OwnCloud GUI client is smart enough to use inotify to notice an mv, but my guess is and I haven t tested this, but apparently OwnCloud doesn t use hashes at all that the CLI client would require a reupload after any mv, because it doesn t run continuously. In some situations, Syncany may be a useful work-around for this, as it does chunk-based dedup and client-side encryption. However, you would lose a lot of the sharing features inside OwnCloud by doing this, and the integration with the OwnCloud apps for photos, videos, and music. The Android/mobile apps support all the usual auto-upload options. Calendar A lot of people report using OwnCloud as a calendar server, and it does indeed use CalDAV. With a program like DAVDroid or Mozilla Lightning, this makes, in theory, a full-functioning calendar syncing tool. There is, of course, also a web interface to the calendar. It, sadly, is limited. Or shall we say, VERY limited. Even something like sending an invite is missing and in fact, the GUI for sharing an event is baffling. You can share it with someone, they get no say in whether or not it shows up, and it shows up on their calendar on the web only (not on synced copies) and they have no way to remove it! Sharing calendars is similar; you can hide the display of any one of your calendars on the web interface, but not of any calendars shared with you. Baffling. Address Book I haven t tested this yet, but there s not much to test, I suspect. It can be shared with others, which I could see as a nice feature. Bookmarks An interesting bookmarks manager, though mysteriously not with Firefox sync support. There is Chrome sync support, and a separate Mozilla Sync support, but it doesn t provide cross-browser syncing, apparently. Music It is designed to present an interface to music that is stored in Files. It provides an Ampache-compatible API, so there are a lot of clients that can stream music. It has very few options, not even for transcoding, so I don t see how it would be useful for my FLAC collection. Pictures Sort of a gallery view of photos synced up with Files. Very basic. Has a sharing button to share a link to an entire folder, but no option to embed photos in blog posts at a lower resolution or shortcut to sharing individual photos. Notes, Tasks, etc. I haven t had the chance to look at this much. Some of them sync to various clients. The Notes are saved as HTML files that get synced down. Clients overall There is a very helpful page that lists all the sync clients for OwnCloud not just for files, but also for calendars, contacts, etc. The list is extensive! Other options The two other Open Source options mentioned on my blog post were Kolab and Sogo, and there is also Zimbra which also has a community edition. The Debian Groupware page lists a number of other groupware options as well. Citadel caught my eye (wow, it s still around!). Sogo has ActiveSync support, which might make phone integration a lot easier. It is not dead-simple to set up like OwnCloud is, though, so I haven t tried it out, but I will probably be looking at it and Citadel next.

8 May 2015

Daniel Kahn Gillmor: Cheers to audacity!

When paultag recently announced a project to try to move debian infrastructure to python3, my first thought was how large that undertaking would likely be. It seems like a classic engineering task, full of work and nit-picky details to get right, useful/necessary in the long-term, painful in the short-term, and if you manage to pull it off successfully, the best you can usually hope for is that no one will notice that it was done at all. I always find that kind of task a little off-putting and difficult to tackle, but I was happy to see someone driving the project, since it does need to get done. Debian is potentially also in a position to help the upstream python community, because we have a pretty good view of what things are being used, at least within our own ecosystem. I'm happy to say that i also missed one of the other great benefits of paultag's audacious proposal, which is how it has engaged people who already knew about debian but who aren't yet involved. Evidence of this engagement is already visible on the py3porters-devel mailing list. But if that wasn't enough, I ran into a friend recently who told me, "Hey, I found a way to contribute to debian finally!" and pointed me to the py3-porters project. People want to contribute to the project, and are looking for ways in. So cheers to the people who propose audacious projects and make them inviting to everyone, newcomers included. And cheers to the people who step up to potentially daunting work, stake out a task, roll up their sleeves, and pitch in. Even if the py3porters project doesn't move all of debian's python infrastructure to pyt3 as fast as paultag wants it to, i think it's already a win for the project as a whole. I am looking forward to seeing what comes out of it (and it's reminding me i need to port some of my own python work, too!) The next time you stumble over something big that needs doing in debian, even something that might seem impossible, please make it inviting, and dive in. The rest of the project will grow and improve from the attempt.Tags: py3-porters

Gunnar Wolf: Guests in the Classroom: Felipe Esquivel (@felipeer) on the applications on parallelism, focusing on 3D animation

I love having guests give my classes :) This time, we had Felipe Esquivel, a good friend who had been once before invited by me to the Faculty, about two years ago. And it was due time to invite him again! Yes, this is the same Felipe I recently blogged about To give my blog some credibility, you can refer to Felipe's entry in IMDb and, of course, to the Indiegogo campaign page for Natura. Felipe knows his way around the different aspects of animation. For this class (2015-04-15), he explained how traditional ray-tracing techniques work, and showed clear evidences on the promises and limits of parallelism Relating back to my subject and to academic rigor, he clearly shows the speed with which we face Amdahl's Law, which limits the efficiency of parallelization at a certain degree perprogram construct, counterpointed against Gustafson's law, where our problem will be able to be solved in better detail given more processing abilities (and will thus not hit Amdahl's hard ceiling). A nice and entertaining talk. But I know you are looking for the videos! Get them, either at my server or at archive.org.

7 May 2015

Ben Hutchings: Debian LTS work, April 2015

This was my fifth month working on Debian LTS. I was assigned 16 hours by Freexian's Debian LTS initiative. I worked on several packages but haven't uploaded updates yet. linux-2.6 There is a steady stream of security issues in all Linux kernel versions. As we don't want to prompt reboots too often, these won't necessarily result in an update every month; it depends on the severity of the issue(s). However, I triaged the current issues and committed the available fixes to version control. dpkg The PGP signature validation in dpkg-source has a number of bugs, including CVE-2015-0840 which allows the validation to be completely subverted. I backported fixes for these from the 1.16 (wheezy) branch to the 1.15 (squeeze) branch. dpkg has a test suite and the fixes all came with new regression tests, so this was easy to verify. As dpkg is a native package (i.e. there is little separation between the 'upstream' and Debian packaging), I preferred that the dpkg maintainers would turn this into an 'upstream' release that I could upload. Guillem Jover has now reviewed and (semi-)released my changes, so I will complete this update soon. tiff A large number of bugs have been found in the venerable tiff library and tools, mostly by 'fuzzing' them with afl. The bugs have been unhelpfully grouped into a smaller number of CVE IDs, although it's not clear that these groups were introduced at the same time and they certainly weren't fixed at the same time. Most of the upstream bug reports come with samples to reproduce the bug (crash or memory corruption detectable with valgrind), but many of these did not turn out to be reproducible (either upstream or with the version in 'squeeze'). Where they were reproducible, I've verified that the patches fix the issue. Unfortunately tiff does not have a test suite, so I made a call for testing on the debian-lts list. If I don't get any responses soon, I'll run my own basic tests with programs that use libtiff before uploading.

6 May 2015

Matthieu Caneill: Debsources got swag and continous integration

Debsources (http://sources.debian.net) is still under active development. We recently had a Gnome Outreachy intern, Jingjie Jiang, and we're about to work with 2 GSoC students, Cl ment Schreiner and Orestis Ioannou. I will present here the GitHub mirror we've set up, in order to allow external pull requests to be submitted, and to use the continous integration service provided by Travis-CI. GitHub and Travis-CI Debsources' source code is hosted on Debian's git servers, and from there is mirrored to GitHub. Every time a commit is pushed (to master or other branches) or a pull request is open, the test suite will be automatically run on Travis-CI, and the result (tests pass or don't) is displayed on GitHub. This allows us to quickly filter external contributions (when they are submitted on GitHub), and be sure everything works with our setup, before reviewing work. Travis-CI runs the tests on OpenVZ containers. The complete infrastructure was a bit challenging to setup, but as we now have a Docker recipe to quicly begin to hack on Debsources, most of the work could be done using the Dockerfile instructions. In average, a run on Travis-CI (which includes git cloning the code and test data, setup the server, and run the tests suite) takes 7 minutes, which is an ok amount of time to wait for before submitting a pull request, in my opinion. Bugs discovered in the process Setting up this continuous integration infrastructure made me discover a few bugs. Python magic does black magic Debsources runs fine on Debian (not surprisingly), but I got tricked by black magic when I tried to run it on Ubuntu (which is the OS run in Travis-CI's containers). We use the magic library to guess the type of files we're dealing with, for instance when we need to decide between rendering a file (for text files) or downloading it (for binary files). Here comes the tricky part: the Python bindings for libmagic are not the same in Debian and Pypi. Debsources uses Debian package python-magic, which is not in Ubuntu 12.04. Moreover, there's no Python egg for it on Pypi, which has however another package (called magic) which provides a different API. I solved this with a dirty hack, using the fact python-magic lies in a single file:
mkdir /tmp/python-magic && wget https://raw.githubusercontent.com/file/file/master/python/magic.py -O /tmp/python-magic/magic.py && export PYTHONPATH=/tmp/python-magic/:$PYTHONPATH
It simply downloads the library, saves it in a temporary folder and includes it in the Python path. Let's see for how long it works before everything breaks! Size of a directory One test in the suite was ensuring the information returned by ls -l on a directory and stored in the DB was the right information. Inode metadata was tested, such as name, permissions, type, or size. Interestingly enough, the size of a directory was tested, and expected to be 4096 bytes. The size of a directory actually depends on the filesystem in use, and on the number of files this directory contains. We often see 4096 because it's the size of a not-too-big directory on ext4. Travis-CI doesn't use ext4:
$ df -T
Filesystem            Type     1K-blocks      Used Available Use%
Mounted on
/vz/private/209140041 simfs    125829120 103460612  22368508  83% /
none                  devtmpfs   1572864         8   1572856   1% /dev
none                  tmpfs       314576        56    314520   1% /run
none                  tmpfs         5120         4      5116   1%
/run/lock
none                  tmpfs      1572864         0   1572864   0%
/run/shm
/dev/null             tmpfs       786432    171584    614848  22%
/var/ramfs
Simfs is a container filesystem for OpenVZ, on which directories have different sizes than on ext4:
$ ls -al /
total 0
drwxr-xr-x 23 root     root      480 Feb  4 18:08 .
drwxr-xr-x 23 root     root      480 Feb  4 18:08 ..
drwxr-xr-x  2 root     root     2480 Feb  4 18:20 bin
drwxr-xr-x  2 root     root       40 Apr 19  2012 boot
drwxr-xr-x  5 root     root      660 Apr 30 13:56 dev
drwxr-xr-x 99 root     root     3560 Apr 30 13:56 etc
-rw-r--r--  1 root     root        0 Feb  4 17:56 fastboot
drwxr-xr-x  3 root     root       80 Feb  4 17:57 home
[...]
Directory sizes are not even powers of 2. Hence I changed the test to not check directory sizes. Hopefully this will help to make Debsources work on more filesystems! An empty file is hiding Last but not least, because this bug is still open in the wild. A file, which appears to be empty, is not taken into account by Debsources' updater. This file is sources/non-free/m/make-doc-non-dfsg/4.0-2/.pc/applied-patches. It is present in the filesystem in the container, is not the only empty file over there, but still doesn't appear in the database, and make fail the test which counts files. The test has been commented out (booooooh), so that we still can use Travis-CI's platform for our GSoC students, before it's fixed. Conclusion Making Debsources run automatically on a different platform as the one we usually use permitted us to spot bugs, write dirty hacks, and expand the filesystems it's supposed to run on. Now, let's hope the continuous integration will help our GSoC students, and let's wish them good luck!

John Goerzen: I Give Up on Google: Free is Too Expensive

I am really tired of things Google has done lately. The most recent example being retiring Classic Maps. That s a problem, because the current Maps mysteriously doesn t show most of my saved ( starred ) places. Google has known about this since at least 2013. There are posts all over their forums about it going back to when what is now regular Google Maps was beta. Google employees even knew about it and did nothing. For someone that made heavy use of it, this was quite annoying. But there have been plenty of others: I even used to use Flickr, then moved to Picasa when Yahoo stopped investing in Flickr. Now I m back to Flickr, because Google stopped investing in Picasa. The takeaway is that you can t really rely on Google for anything. Counting on something being there for an upcoming trip and then having it be suddenly yanked away is a level of frustration that just makes the service not so useful. Never knowing when obvious things (7-day calendar view) will be removed means you just can t depend on it. So, are there good alternatives? Things I m thinking of include: Anybody else moving off Google?

5 May 2015

Raphaël Hertzog: My Free Software Activities in April 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 26.25 hours on Debian LTS. In that time I did the following: Now, still related to Debian LTS, but on unpaid hours I did quite a few other things: Other Debian work Feature request in update-alternatives. After a discussion with Josselin Mouette during the Mini-DebConf in Lyon, I filed #782493 to request the possibility to override at a system-wide level the default priority of alternatives recorded in update-alternatives. This would make it easier for derivatives to make different choices than Debian. Sponsored a dnsjava NMU. This NMU introcuded a new upstream version which is needed by jitsi. And I also notified the MIA team that the dnsjava maintainers have disappeared. python-crcmod bug fix and uploads to *-backports. A member of the Google Cloud team wanted this package (with its C extension) to be available to Wheezy users so I NMUed the package in unstable (to fix #782379) and prepared backports for wheezy-backports and jessie-backports (the latter only once the release team rejected a fix in jessie proper, see #782766). Old and new PTS updates for Jessies s release. I took care to update tracker.debian.org and packages.qa.debian.org to take into account Jessie s release (which, most notably, introduced the oldoldstable suite as the new name for Squeeze until its end of life). Received thanks with pleasure. This is not something that I did but I enjoyed reading so many spontaneous thanks in response to Guillem s terse and thankless notification of me stepping down from dpkg maintenance. I love the Debian community. Thank you. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Laura Arjona: Debian Publicity Team meeting today!

Today at 18:00 UTC (this evening for me) there will be a Debian Publicity Team IRC meeting (open meeting, everybody invited), and I m very happy because it will be the first meeting that I know of, since I joined the team (years!). Being part of the Publicity team There are many tasks handled by Publicity, and when I joined, I supposed that I was going to be part of a team with many members and well structured. And it was true but not as I imagined. Publicity is a great team, in the sense that it accepts contributions from many people, and the few core members do an amazing work: on one side, get things done; on the other side, integrate all those occasional contributions from the wider community. But there are fewer core contributors than what one would expect by the output of the team. I would say we are maximum 10 people (out of 353 voters, 1033 Debian Developers, and 1197 contributors in the Debian Community in 2015). And as far as I know, everybody is member of some other teams too (I m a translator, others in website team, sysadmins, packaging teams and now we have a member sharing Publicity membership with DPL-ship!). Organisation around the tasks Publicity regular tasks (announcements, the newsletter Debian Project News , posting in social networks and in bits.debian.org, and other ) are all well defined and documented, in order to allow anybody jump in and help, and this is great, because it ensures a way for contributions to arrive the wide audience from the very first day: you pick something, you follow the instructions, and you re done. I love this approach, because I tend to prefer to follow instructions than to create something, and my Debian time is made of small chunks at random days/times. But sometimes I feel that we all work alone , in something like a cold, robotic do-ocracy, and I also wonder how many people don t contribute or become regular contributors just because they don t understand the procedures, or they don t like them, or other reasons IRC meeting IRC is something that I use only for contributing to free software, not in other parts of my life. I like IRC, it s productive and fun, but I m not always there, and I don t save logs when I am idle, and I usually prefer email for communication. However, I try to be more present in the Debian IRC channels of the teams where I contribute, because I ve learned that it plays a big role in feeling at home in Debian . Currently you can find me in #debconf-team #debconf15-germany #debian-i18n #debian-l10n-spanish #debian-publicity #debian-women and #debian-www . I ve attended some meetings in IRC (MediaGoblin monthly meetings, and DebConf15 meetings) and I ve learned about MeetBot and more or less how to chair an IRC meeting. Today it will be my first time chairing, it s hard to emulate so great chairs as Marga or Chris Webber, bu I hope I do it decently, and we all have a nice time knowing each other and sharing ideas for the Publicity team. Want to attend? All the details (when, where, agenda ) in the wiki page of the meeting. See you in a few hours!
Filed under: My experiences and opinion Tagged: Communities, Contributing to libre software, Debian, English, Free Software, IRC, libre software, Project Management

4 May 2015

Mike Gabriel: Rebasing NXv3 against latest X.Org -- already ~2,700,000 lines of code removed

We have set sails for a great endeavour. We are cleaning up the nx-X11 / nxagent code tree [1] and rebase it against latest / recent X.Org. Until now, we have been able to drop ~2,700,000 lines of code [2] from the source tree that originally got released by NoMachine. The plan is... For more details, see the Readme.md[3] file on our Github project. What we have achieved so far... read more

Mike Gabriel: My FLOSS activities in April 2015

April 2015 has been my first month on the Debian LTS team (as a paid contractor working underneath the Freexian [1] umbrella). Working in the team of paid Debian LTS developers requires to write a monthly summary about sponsored work on Debian LTS. Thanks to Raphael Hertzog for having me on his company's team and providing the framework for providing paid work on Debian LTS. I will use this requirement for a monthly report as a starting point for documenting my FLOSS activities on a monthly basis via my blog (not only for Debian LTS, but also for other projects). Work on Debian LTS For the Debian LTS team I have been doing 8h of contracted work in April 2015 (and at the beginning of May 2015). The work focused on: Several more hours have been spent by myself (and also Raphael) for getting me started in the team. Thanks for your patience. Work on Debian jessie The Debian MATE Packaging team was able to provide several fixes last-minute before the Debian jessie release (mate-control-center[5], caja-extensions[6], mate-desktop [7]). Thanks to the release team for processing the last-minute unblock requests so smoothly. read more

Lunar: Reproducible builds: first week in Stretch cycle

Debian Jessie has been released on April 25th, 2015. This has opened the Stretch development cycle. Reactions to the idea of making Debian build reproducibly have been pretty enthusiastic. As the pace is now likely to be even faster, let's see if we can keep everyone up-to-date on the developments. Before the release of Jessie The story goes back a long way but a formal announcement to the project has only been sent in February 2015. Since then, too much work has happened to make a complete report, but to give some highlights: Lunar did a pretty improvised lightning talk during the Mini-DebConf in Lyon. This past week It seems changes were pilling behind the curtains given the amount of activity that happened in just one week. Toolchain fixes We also rebased the experimental version of debhelper twice to merge the latest set of changes. Lunar submitted a patch to add a -creation-date to genisoimage. Reiner Herrmann opened #783938 to request making -notimestamp the default behavior for javadoc. Juan Picca submitted a patch to add a --use-date flag to texi2html. Packages fixed The following packages became reproducible due to changes of their build dependencies: apport, batctl, cil, commons-math3, devscripts, disruptor, ehcache, ftphs, gtk2hs-buildtools, haskell-abstract-deque, haskell-abstract-par, haskell-acid-state, haskell-adjunctions, haskell-aeson, haskell-aeson-pretty, haskell-alut, haskell-ansi-terminal, haskell-async, haskell-attoparsec, haskell-augeas, haskell-auto-update, haskell-binary-conduit, haskell-hscurses, jsch, ledgersmb, libapache2-mod-auth-mellon, libarchive-tar-wrapper-perl, libbusiness-onlinepayment-payflowpro-perl, libcapture-tiny-perl, libchi-perl, libcommons-codec-java, libconfig-model-itself-perl, libconfig-model-tester-perl, libcpan-perl-releases-perl, libcrypt-unixcrypt-perl, libdatetime-timezone-perl, libdbd-firebird-perl, libdbix-class-resultset-recursiveupdate-perl, libdbix-profile-perl, libdevel-cover-perl, libdevel-ptkdb-perl, libfile-tail-perl, libfinance-quote-perl, libformat-human-bytes-perl, libgtk2-perl, libhibernate-validator-java, libimage-exiftool-perl, libjson-perl, liblinux-prctl-perl, liblog-any-perl, libmail-imapclient-perl, libmocked-perl, libmodule-build-xsutil-perl, libmodule-extractuse-perl, libmodule-signature-perl, libmoosex-simpleconfig-perl, libmoox-handlesvia-perl, libnet-frame-layer-ipv6-perl, libnet-openssh-perl, libnumber-format-perl, libobject-id-perl, libpackage-pkg-perl, libpdf-fdf-simple-perl, libpod-webserver-perl, libpoe-component-pubsub-perl, libregexp-grammars-perl, libreply-perl, libscalar-defer-perl, libsereal-encoder-perl, libspreadsheet-read-perl, libspring-java, libsql-abstract-more-perl, libsvn-class-perl, libtemplate-plugin-gravatar-perl, libterm-progressbar-perl, libterm-shellui-perl, libtest-dir-perl, libtest-log4perl-perl, libtext-context-eitherside-perl, libtime-warp-perl, libtree-simple-perl, libwww-shorten-simple-perl, libwx-perl-processstream-perl, libxml-filter-xslt-perl, libxml-writer-string-perl, libyaml-tiny-perl, mupen64plus-core, nmap, openssl, pkg-perl-tools, quodlibet, r-cran-rjags, r-cran-rjson, r-cran-sn, r-cran-statmod, ruby-nokogiri, sezpoz, skksearch, slurm-llnl, stellarium. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which did not make their way to the archive yet: Improvements to reproducible.debian.net Mattia Rizzolo has been working on compressing logs using gzip to save disk space. The web server would uncompress them on-the-fly for clients which does not accept gzip content. Mattia Rizzolo worked on a new page listing various breakage: missing or bad debbindiff output, missing build logs, unavailable build dependencies. Holger Levsen added a new execution environment to run debbindiff using dependencies from testing. This is required for packages built with GHC as the compiler only understands interfaces built by the same version. debbindiff development Version 17 has been uploaded to unstable. It now supports comparing ISO9660 images, dictzip files and should compare identical files much faster. Documentation update Various small updates and fixes to the pages about PDF produced by LaTeX, DVI produced by LaTeX, static libraries, Javadoc, PE binaries, and Epydoc. Package reviews Known issues have been tagged when known to be deterministic as some might unfortunately not show up on every single build. For example, two new issues have been identified by building with one timezone in April and one in May. RD and help2man add current month and year to the documentation they are producing. 1162 packages have been removed and 774 have been added in the past week. Most of them are the work of proper automated investigation done by Chris West. Summer of code Finally, we learned that both akira and Dhole were accepted for this Google Summer of Code. Let's welcome them! They have until May 25th before coding officialy begins. Now is the good time to help them feel more comfortable by sharing all these little bits of knowledge on how Debian works.

Ricardo Mones: Bye bye DebConf15

Yep, I had planned to go, but given the last mail from registration seems there's a overwhelming number of sponsorship requests, so I've dediced to withdraw my request. There's lots of people doing much more important things for Debian than me which deserve that help. Having to complete my MSc project does also help to take this decision, of course.

I guess the Debian MIA meeting will have to wait for next planetary alignment ;-) well, not really, any other member of the team can set up it, hint! hint!

See you in DebConf17 or a nearby local event!

Mike Hommey: Gnome shell Hello world

Gnome Shell, besides providing the main user interface for GNOME 3, is a Javascript shell with bindings to many native interfaces that allow e.g. Window manipulation, graphics rendering and animations, compositing, etc. It also allows developers to write extensions changing Gnome Shell s behavior. Less known is that it is possible to replace the entire Javascript code base that Gnome Shell uses. It can be useful to hack on Gnome Shell itself (no need to fiddle with system files, or, since 3.12, no need to rebuild libgnome-shell.so), but it can also be used to implement a completely new User Interface in Javascript. I m starting to experiment with the latter, because I want to try building a window manager that fits my needs, while keeping away the boring details of EWMH, xinerama, and other X11 things. And because it s fun. But baby steps, first: let s bootstrap a Hello world with Gnome shell. I tested this with Gnome Shell 3.14. Trying various older versions, I got different results for reasons I don t know. 3.4 doesn t display anything unless, paradoxically, global.stage.show() is removed, and 3.8 doesn t display anything no matter what. I guess the next step is to go through some Clutter tutorials and transpose them to Javascript. Update: On the other hand, a lot of the window managing is still done by mutter under the hood, which doesn t leave a lot of space for something really different.

Next.

Previous.