Search Results: "nate"

9 August 2015

Yves-Alexis Perez: WPS and Network Manager

So, everybody knows that WPS (Wi-Fi Protected Setup) is broken. But sometimes, you don't own the access point, and you'd just want the wireless to work. That happens for example when you're a guest in some place using an Orange Livebox and you don't have the WPA passphrase (usually because it's written somewhere you don't have access too, or because someone forgot to tell you). Liveboxes WPS is the press button thing: you press a button on the front for one second, then any device can connect in the next two minutes. That works fine with Android devices, for example, but it didn't work with my laptop and NetworkManager, which doesn't support WPS at all. Fortunately, the underlying piece of software (wpa_supplicant) does support WPS, and even the push button style. And you can nicely ask it to reveal the passphrase to you with the following trick.
  1. Disconnect NetworkManager from the network, disable the wireless link, stop it; just make sure wpa_supplicant is not running;
  2. Put a stub wpa_supplicant.conf file with only the following content: 
    update_config=1
  3. Start wpa_supplicant in the foreground with your stub config file: 
    wpa_supplicant -iwlan0 -c wpa_supplicant.conf
  4. Start wpa_cli
Inside wpa_cli:
  1. Scan the network: 
    scan
  2. Get the results: 
    scan_results
    and identity the bssid of the Livebox
  3. Press the WPS button on the Livebox
  4. Run 
    wps_pbc <bssid>
    ; some text should appear in the wpa_cli window, and it should eventually connect successfully (at that point you can even run a dhclient on wlan0)
  5. Run 
    save_config
The last command will update your stub configuration file, adding a new network block with the passphrase in the clear. You can then use that passphrase inside Network Manager if it's more convenient for you. There might be something easier, but at least it worked just fine for me during the holidays.

8 August 2015

Dirk Eddelbuettel: drat 0.1.0: Even more repository support

drat and double drat A new version 0.1.0 of the drat package arrived on CRAN today. Its name stands for drat R Archive Template, and it helps with easy-to-create and easy-to-use repositories for R packages, and is finding increasing by other projects. This version 0.1.0 builds on the previous releases and now adds complete support for binaries for both Windows and OS X. This builds on what had been added in the previous release 0.0.4. This and other new features are listed below: Also of note: The rOpenSci project now uses drat to distribute their code, and Carl Boettiger has written a nice blog post about it. Courtesy of CRANberries, there is a comparison to the previous release. More detailed information is on the drat page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

7 August 2015

Norbert Preining: Japanese-English dictionary for Kobo

Since ever the Kobo firmwares also allowed downloading of a bunch of dictionaries, most of which I don t need. As I am fluent in most languages I read and write, the only real dictionary I would like to see is a Japanese-English (I don t dare asking for a Japanese-German). Unfortunately, Kobo never shipped one. OTOH, starting with firmware 3.16.10 they ships two different English-Japanese dictionaries, one excellent Japanese-Japanese dictionary, but not one Japanese-English. So I took the liberty to write a script that allows everyone to enrich the shipped Japanese-Japanese dictionary with English definitions. Kobo Logo Till now I have used Tshering s excellent Japanese dictionary which was enriched with English definitions. It was based on the older Japanese-Japanese dictionary shipped with firmwares before 3.16.10. I didn t want to loose the more complete dictionary from the new firmware, so here is a script that updates the dictionary for you Prerequisites The script is neither fool-proof nor does everything by itself. Furthermore, it requires a set of programs. In details: Supported dictionaries At the moment the program can use the following two dictionaries as sources: Edict2 and Japanese3. Edict2 The Edict dictionary is a free dictionary which is the base of most other dictionaries. Created by the JMdict/EDICT Project it provides a very complete Japanese-English dictionary. To use the dict with the current program, one need to download the edict2.gz file and unpack it with gunzip edict2.gz. If you put it into the same directory as the Perl script, nothing else needs to be done, otherwise one can use the command line option --edict PATH-TO-EDICT to specify the location of the edict2 file. Japanese3 The Japanese3 is a dictionary application for iOS which provides another very complete Japanese-English dictionary. My feeling is that it is 90% and more based on Edict, so adding it will not buy you much, but still a bit (see below for how much!). If you have purchased this dictionary/application, and you manage to get access to your iOS device (via jailbreaking or some other tools), then you need to get the file Japanese3.db from the application folder, and then generate a file via sqlite3 as follows: 
$ sqlite3 Japanese3.db
.output japanese3-data
select Entry, Furigana, Summary from entries;
If you save the generated file japanese3-data into the same directory as the Perl script, nothing else needs to be done, otherwise one can use the command line option --japanese3 PATH-TO-J3-DATA to specify the location of the Japanese3 data file. Command line options The program supports the following command line options: Note that in case you pass in the option --unpacked, the files should be properly named (encodings are a problem!). Furthermore, note that the unpacked directory contains lots of .gif and .html files that are actually gzip-compressed files, but lacking the .gz extension. If you want to unpack the dictionary, by advised that the file names in the Zip directory are already encoded in UTF-8, but normal programs (unzip, 7z) assume that they are encoded in some other encoding. Thus, if your are using UTF-8 locale, it is necessary to set LC_CTYPE to C to make sure that the encoding is used, as in 
LC_CTYPE=C 7z x ...
Typical run In the following example we use both dict2 and Japanese3 dictionaries, and prefer Edict2: 
$ perl enhance-dictionary.pl --dicts edict2 --dicts japanese3
Using the following dictionaries as source for translations: edict2 japanese3
loading edict2 ... done
loading Japanese3 data ... done
unpacking original dictionary ... done
loading dict files ... done
searching for words and updating ... done
total words 805521, matches: 130563 (edict: 123388, japanese3: 7175)
creating output html ... done
creating update dictionary in dicthtml-jaxxdjs-201508072201.zip ... done
$
Installing the dictionary After having created the enhanced dictionary, one can install the generated file as KOBO/.kobo/dicts/dicthtml-jaxxdjs.zip (where KOBO is the mount point of the eReader). The dictionary should be picked up automatically. And the next lookup should give you something like the following:
kobo-glohd-enhanced-ja-dict There is only one caveat: Syncing with Kobo will re-download the original dictionary and overwrite the enhanced one. There is at least one solution here, which I am employing, see this post on MobileRead. Download and development place I am using the github repository kobo-ja-dict-enhance to develop the program. Please report bugs, and feature requests there. The file for the last released version can be downloaded from here: enhance-dictionary.pl Future plans I am planning to get rid of the 7z dependency and use Archive::Zip for all the unpacking and packing. This would also allow to do everything in memory and thus make it faster. There is a branch on github with ongoing work on this matter. I also think about including translations for Hiragana words by providing the translations of all possible Kanji readings. Conclusions Even if Kobo does not provide us with a decent Japanese-English dictionary, adding at least a huge amount of translations to the current dictionary is now easily possible. For serious students of Japanese who are reading or starting to read Japanese eBooks, this will be of great help. Enjoy, and don t forget to give feedback, suggestions, and improvements either here or better via the Github page.

Ben Hutchings: Debian LTS work, July 2015

This was my eighth month working on Debian LTS. I was assigned 14.75 hours of work by Freexian's Debian LTS initiative. linux-2.6 I didn't upload any new version of the kernel this month, but I did send all the recent security fixes to Willy Tarreau who maintains the 2.6.32.y branch at kernel.org. I also spent more time working on a fix for bug #770492 aka CVE-2015-1350, which is not yet fixed upstream. I now have a candidate patch for 2.6.32.y/squeeze, and automated tests covering many of the affected filesystems. Front desk The LTS 'front desk' role is now assigned on a rota, and I was my first turn in the third week of July. I investigated which new CVEs affected LTS-supported packages in squeeze, recorded this in the secure-testing repository, and mailed the package maintainers to give them a chance to handle the updates. groovy Groovy had a single issue (CVE-2015-3253) with a simple fix that I could apply to the version in squeeze. Unfortunately the previous version in squeeze had not been properly updated during the squeeze release cycle and could no longer be built from source. I eventually worked out what the build-dependencies should be, uploaded the fix and issued DLA-274-1. ruby1.9.1 Ruby 1.9.1 also had a single issue (CVE-2014-6438), though the fixes were more complicated and hard to find. (The original bug report for this is still not public in the upstream bug tracker.) I also had to find an earlier upstream change that they depended on. As I've mentioned before, Ruby has an extensive test suite so I could be quite confident in my backported changes. I uploaded and issued DLA-275-1. libidn The GNU library for Internationalized Domain Names, libidn, required applications to pass only valid UTF-8 strings as domain names. The Jabber instant messaging server turned out not to be validating untrusted domain names, leading to a security issue there (CVE-2015-2059). As there are likely to be other applications with similar bugs, this was resolved by adding UTF-8 validation to libidn. The fix for this involved importing additional functions from the GNU portability library, gnulib, and there my difficulties began. Confusingly, libidn has two separate sets of functions imported from gnulib, and due to interdependencies it turned out that I would have to update both of these wholesale rather than just importing the new functions that were wanted. This resulted in a 35,000 line patch. Following that I needed to autoreconf the package (and debug that process when it failed), ending up with another 26,000 line patch. Finally, it turned out that the new gnulib code needed gperf to build a header file for Solaris (when building for Linux? huh?). I ended up adding that with another patch instead. libidn has a decent test suite, so I could at least be confident in the result of my changes. I uploaded and issued DLA-277-1. Dear upstream developers, please use sane libraries instead of gnulib.

6 August 2015

Eduard Sanou: Reproducible builds on Debian for GSoC 2015, 1st update

This is the second blog post I m writing about my experiences contributing to Debian for Google Summer of Code 2015 (check my first post)

Status update

First month It s been two months and a few days since the GSoC started. During the first month I worked on fixing specific packages, mainly concerning issues with timestamps, which is a very common source of unreproducibility. In many cases, during the build process files are compressed into gzip or zip archives, which store the creation time of files in the metadata. This can lead to unreproducible results when there is timezone variation between builds (easily fixed setting the timezone to UTC before the compression happens). In some cases the compressed files are generated during the build, and thus add build times in the metadata of compressed files (in this case the creation date of the files needs to be normalized somehow). As explained in my application, I finished exams on the end of June, that s why I chose to work on small fixes first, so that I could make the most out of my time between studying and finishing university projects and reports. I m happy with my first month, as I have worked as originally planned. Actually, my estimation of the number of bugs I could submit every week was surpassed in reality!

Second month Once the university was over, I started dedicating myself fully to the project. This allowed me to start working on toolchain fixes, following my original plan on working with timestamp related issues. In particular I have been working a lot in implementing a proposal for deterministic timestamps that appeared in the reproducible builds project. The idea is to define an environment variable called SOURCE_DATE_EPOCH that contains a known timestamp in Unix epoch format. With this variable exported, tools that would embed the localtime in their generated or processed files, can use this externally supplied date. This would happen only if SOURCE_DATE_EPOCH is exported, so the behaviour of the tool wouldn t change if the variable is not set. The first package I patched to implement this behaviour was gcc. The reason behind this is that there are about 420 unreproducible packages due to using the __DATE__, __TIME__ and __TIMESTAMP__ C macros. My patch changes the behavior of the macros __DATE__ and __TIME__ if SOURCE_DATE_EPOCH is exported. I submitted this patch to the gcc-patches list. Even though there was some interesting discussions in the list, the patch has not been accepted yet. Seeing how the reproducible builds idea is gaining momentum and becoming widespread, I m positive that at some point the gcc people will be more receptive for such patch. The second work with SOURCE_DATE_EPOCH was in debhelper; I patched this building tool to export the variable with the latest debian/changelog entry timestamp. With this patch, all the tools that run under dh will be able to use it to embed deterministic timestamps. Unfortunately some parts of the build process of some packages don t happen under debhelper, so the variable needs to be exported in a different way. Having submitted the debhelper patch allowed many packages to become reproducible after the tools that embedded timestamps were patched to honour SOURCE_DATE_EPOCH. As of today, the toolchain packages I have patched to do that are: gcc, libxslt, gettext, ghostscript and qt4-x11 (qhelpgenerator). I have also continued working on fixing individual packages affected by timestamps, random orderings (such as the ones from listing hash keys) and locale depending orderings; I have tagged packages in our infrastructure to note what kind of issue makes them unreproducible; I have updated some parts of the Reproducible Builds Wiki.

Impressions about reproducible builds The work I did during the first month felt a bit tedious sometimes: it didn t require much creativity or thinking as most of the fixes where quite mechanical, following a recipe. After I became free from university duties, I started looking into less trivial issues, which require more deep investigation and feel more rewarding once they are fixed. I also worked on toolchain fixes, which need more work and need more care. Fixing toolchain packages feels particularly rewarding because they can cause many packages to become reproducible at once. There is a very active community in the reproducible builds project! It s great to see so many people contributing to this project spending so many hours and putting so much effort. I ve felt very welcome from the beginning and I have gotten kind replies and helpful answers to all the questions and doubts I ve had, both from my mentor and from the other people in the project. I want to add that I m still amazed by the awesome infrastructure set up for the reproducible builds project. The team is using a Jenkins machine to continuously build packages to check for reproducibility, with irc notifications for changes, and also with a really useful web interface to list all the packages organized by issues that allows exploring them individually with all the available information. Also not only the infrastructure is used to build Debian amd64 packages, but also FreeBSD, NetBSD, OpenWRT, coreboot and lately Debian armhf with the help of a few new arm nodes.

Impressions about working on a free software project This was my first time working on a community driven free software project and I ve learned so many things. Something I learned during the first days, which is even more present in such wide project like the reproducible builds, is that contributing is not just writing patches for the features you want; you also need to convince the maintainer of the package to accept the patch! I was a bit surprised at the beginning because even if this is a Debian project, that aims to make changes to the whole distribution, decisions are not absolute for the whole Debian project. After taking decisions within the reproducible builds teams on how to approach things, we need to convince the rest of the Debian developers (mainly maintainers) to follow them, as they are allowed to disagree. So it is required to work together for solutions that makes everyone comfortable, usually with discussions on mailing lists or irc. There is another fact that I wasn t expecting before getting involved in this project. The kind of teamwork I have done previously involves having a leader who decides how stuff is done, who takes decisions when needed and oversees the whole project. There seems to be a different philosophy in Debian. Instead of having a leader, everyone tries their best, trying to convince the others that their solution is good, often by showing an implementation of the solution and providing proof that it works, rather than trying to get the solution accepted before coding it. Also, solutions and ideas are valued for their quality rather than by the position of the person submitting them, and there is no hierarchy within the group: all comments and advices are taken equally, valuing their usefulness regardless of who gives them. Usually there are no votes when deciding things. Members try their best on their approaches, trying to convince others as best as they can. And even if someone disagrees they may end up accepting the solution if they don t manage to convince the original proposer of doing things differently. The idea is to spend more time working and coding than arguing and deciding on the way to do things. So far I ve seen this approach to be very efficient :) I ve been told by my mentor that for difficult cases there exist a Debian committee that helps mediate on disagreements, but that is only used as a last option, probably when the discussion gets heated up.

Personal experience Overall I m very happy to finally having set my foot in the free software community, where I m able to contribute to the kind of software I have been using for years. The sense of community in Debian is really big, and everyone is invited to participate. I think that Google is doing an awesome job with the Google Summer of Code, not only because it gives a lot to free software but because it helps students to join the free software world as contributors (which is something that may be difficult to get into when you don t know how to begin, as it happened to myself for some time). I plan to continue contributing to the free software world, and I d encourage anyone to find projects to get involved and contribute as well! Happy hacking to everyone!

Norbert Preining: Upgrading my eReader Kobo GloHD

After long time of dear service, I have decided to exchange my Kobo Glo to a new device, the Kob GloHD. The reasons are simple: higher resolution, better reading quality.
kobo-glohd-1 While the photos are not showing these details, the quality of the screen is definitely much better I have now read a few hours with the new device, both Japanese and English books, and the crispness of the letters has definitely improved. Very nice. Together with the new device came a new firmware, only released in Japan, 3.16.10. Of course it can be found on MobileRead. As with previous version, I will prepare a MegaPack containing nickel patches, ssh, custom dicts, etc etc, please wait a bit longer until KSM compatibility with the new firmware is perfect again. Talking of dictionaries: The new version brings a few more dicts, especially there are now two English-Japanese dictionaries, and a much more complete Japanese dictionary.
kobo-glohd-dictlist Unfortunately, as before, there is no Japanese-English dictionary. Till now I have used Tshering s excellent Japanese dictionary which was enriched with English definitions. But the new version brought a new dictionary that is much more complete:
kobo-glohd-dicsearch I have already a script ready that enriches this new big dictionary automatically with definitions from the Edict project, so everyone can now update his own dictionary easily. I will write a blog entry about this soon. That s it for the moment, back to reading. I love the new device, and will surely post many articles about it in the future!

5 August 2015

Don Armstrong: Introducing dqsub

I've been using qsub for a while now on the cluster here at the IGB at UofI. qsub is a command line program which is used to submit jobs to a scheduler to eventually be run on one (or more) nodes of a cluster. Unfortunately, qsub's interface is horrible. It requires that you write a shell script for every single little thing you run, and doesn't do simple things like providing defaults or running multiple jobs at once with slightly different arguments. I've dealt with this for a while using some rudimentary shell scripting, but I finally had enough. So instead, I wrote a wrapper around qsub called dqsub. What used to require a complicated invocation like: 
echo -e '#!/bin/bash\nmake foo'  \
 qsub -q default -S /bin/bash -d $(pwd) \
  -l mem=8G,nodes=1:ppn=4 -;
can now be run with 
dqsub --mem 8G --ppn 4 make foo;
Want to run some command in every single directory which starts with SRX? That's easy: 
ls -1 SRX* dqsub --mem 8G --ppn 4 --array chdir make bar;
Want instead to behave like xargs but do the same thing? 
ls -1 SRX* dqsub --mem 8G --ppn 4 --array xargs make bar -C;
Now, this wrapper isn't complete yet, but it's already more than enough to do what I require, and has saved me quite a bit of time already. You can steal dqsub for yourself Feel free to request specific features, too.

Dirk Eddelbuettel: RcppArmadillo 0.5.300.4.0

armadillo image A new Armadillo release 5.300.4 was prepared by Conrad the other day, and we prepared a new corresponding RcppArmadillo release 0.5.300.4.0 which is now on CRAN and in way into Debian. Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. This release brings a considerable amount of new code extending support for sparse matrices, support for Schur decompositions, and much more as shown in the extract from NEWS file:
Changes in RcppArmadillo version 0.5.300.4 (2015-08-03)
  • Upgraded to Armadillo 5.300.4 ("Plutocrazy Incorporated")
    • added generalised Schur decomposition: qz()
    • added .has_inf() and .has_nan()
    • expanded interp1() to handle out-of-domain locations
    • expanded sparse matrix class with .set_imag() and .set_real()
    • expanded imag(), real() and conj() to handle sparse matrices
    • expanded diagmat(), reshape() and resize() to handle sparse matrices
    • faster sparse sum()
    • faster row-wise sum(), mean(), min(), max()
    • updated physical constants to NIST 2014 CODATA values
    • fixes for handling sparse submatrix views
    • Armadillo can make use of GPUs by linking with NVIDIA NVBLAS (a GPU-accelerated implementation of BLAS), or by linking with AMD ACML (which can use GPUs via OpenCL)
  • Added importFrom statements for R functions not from base
  • Added explicit Rcpp::sourceCpp() reference as well
  • Updated one formatting detail in vignette to please TeXlive2015
Courtesy of CRANberries, there is also a diffstat report for the most recent CRAN release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

4 August 2015

Julien Danjou: Ceilometer, Gnocchi & Aodh: Liberty progress

It's been a while since I talked about Ceilometer and its companions, so I thought I'd go ahead and write a bit about what's going on this side of OpenStack. I'm not going to cover new features and fancy stuff today, but rather a shallow overview of the new project processes we initiated. Ceilometer growing Ceilometer has grown a lot since that time when we started it 3 years ago. It has evolved from a system designed to fetch and store measurements, to a more complex system, with agents, alarms, events, databases, APIs, etc. All those features were needed and asked for by users and operators, but let's be honest, some of them should never have ended up in the Ceilometer code repository, especially not all at the same time. The reality is we picked a pragmatic approach due to the rigidity of the OpenStack Technical Committee in regards to new projects to become OpenStack integrated and, therefore, blessed projects. Ceilometer was actually the first project to be incubated and then integrated. We had to go through the very first issues of that process. Fortunately, now that time has passed, and all those constraints have been relaxed. To me, the OpenStack Foundation is turning into something that looks like the Apache Foundation, and there's, therefore, no need to tie technical solutions to political issues. Indeed, the Big Tent now allows much more flexibility to all of that. Back a year ago, we were afraid to bring Gnocchi into Ceilometer. Was the Technical Committee going to review the project? Was the project going to be in the scope of Ceilometer for the Technical Committee? Now we don't have to ask ourselves those questions, now that we have that freedom, it empowers us to actually do what we think is good in term of technical design without worrying too much about political issues.
Ceilometer development activity
Acknowledging Gnocchi The first step in this new process was to continue working on Gnocchi (a timeserie database and resource indexer designed to overcome historical Ceilometer storage issue) and to decide that it was not the right call to merge it into Ceilometer as some REST API v3, but that it was better to keep it standalone. We managed to get traction to Gnocchi, getting a few contributors and users. We're even seeing talks proposed to the next Tokyo Summit where people leverage Gnocchi, such as "Service of predictive analytics on cost and performance in OpenStack", "Suveil" and "Cutting Edge NFV On OpenStack: Healing and Scaling Distributed Applications". We are also doing some progress on pushing Gnocchi outside of the OpenStack community, as it can be a self-sufficient timeserie and resource database that can be used without any OpenStack interaction. Branching Aodh Rather than continuing to grow Ceilometer, during the last summit we all decided that it was time to reorganize and split Ceilometer into the different components it is made of, leveraging a more service-oriented architecture. The alarm subsystem of Ceilometer being mostly untied to the rest of Ceilometer, we decided it was the first and perfect candidate to do that. I personally engaged into doing the work and created a new repository with only the alarm code from Ceilometer, named Aodh.
Aodh is an Irish word meaning fire. A word picked so it also had some relation to Heat, and because we have some Irish influence around the project .
This made sense for a lot of reason. First because Aodh can now work completely standalone, using either Ceilometer or Gnocchi as a backend or any new plugin you'd write. I love the idea that OpenStack projects can work standalone like Swift does for example without implying any other OpenStack component. I think it's a proof of good design. Secondly, because it allows us to resonate on a smaller chunk of software a reason really under-estimated today in OpenStack. I believe that the size of your software should match a certain ratio to the size of your team. Aodh is, therefore, a new project under the OpenStack Telemetry program (or what remains of OpenStack programs now), alongside Ceilometer and Gnocchi, forked from the original Ceilometer alarm feature. We'll deprecate the latter with the Liberty release, and we'll remove it in the Mitaka release. Lessons learned Actually, moving that code out of Ceilometer (in the case of Aodh), or not merging it in (in the case of Gnocchi) had a few side effects that I admit I think we probably under-estimated back then. Indeed, the code size of Gnocchi or Aodh ended up being much smaller than the entire Ceilometer project Gnocchi is 7 smaller and Aodh 5x smaller than Ceilometer and therefore much more easy to manipulate and to hack on. That allowed us to merge dozens of patches in a few weeks, cleaning-up and enhancing a lot of small things in the code. Those tasks are very much harder in Ceilometer, due to the bigger size of the code base and the small size of our team. By having our small team working on smaller chunks of changes even when it meant actually doing more reviews greatly improved our general velocity and the number of bugs fixed and features implemented. On the more sociological side, I think it gave the team the sensation of finally owning the project. Ceilometer was huge, and it was impossible for people to know every side of it. Now, it's getting possible for people inside a team to cover a much larger portion of those smaller project, which gives them a greater sense of ownership and caring. Which ends up being good for the project quality overall. That also means that we technically decided to have different core teams by project (Ceilometer, Gnocchi, and Aodh) as they all serve different purposes and can all be used standalone or with each others. Meaning we could have contributors completely ignoring other projects. All of that reminds me some discussion I heard about projects such as Glance, trying to fit new features in - some that are really orthogonal to the original purpose. It's now clear to me that having different small components interacting together that can be completely owned and taken care of by a (small) team of contributors is the way to go. People that can therefore trust each others and easily bring new people in, makes a project really incredibly more powerful. Having a project covering a too wide set of features make things more difficult if you don't have enough manpower. This is clearly an issue that big projects inside OpenStack are facing now, such as Neutron or Nova.

3 August 2015

Jamie McClelland: Docker: evil spawn or useful tool

There are plenty of criticisms of docker, the system for building a container-based virtual machine running just a single application. I've read many of them have have consistently been either in agreement or at least amused. The most relevant criticism is about the basic approach of building single-application virtual machines. To understand this criticism, let's remember - in traditional application deployment there are at least three distinct jobs: These three groups have lived in happy tension for years. Now we have containers. The problem with containers is that suddenly the system administrators are out entirely and the distribution developers' role may have been dramatically minimized or circumvented altogether. Meanwhile, the application developer is both liberated from any constraints imposed by either the system administrator or the distribution developers (hooray!) but is also sadled with the enormous responsibilities of these two roles - which may not be apparent to the application developer at first. I have practically no distribution developer experience and have considerable experience as a system administrator and an application developer. And, it took me months to sort through how to properly develop and deploy an application using docker in a way that I thought was responsible and secure. I started by learning about how docker wants you to deploy images - by downloading them from their shared regisry. As I mentioned, I have very little experience in the realm of distribution development, but I at least know enough about Debian to know that a lot of time and thought has gone into cryptographically verifying packages that I install, which apparently is not done at all with Docker images. Is that obvious to everyone using Docker?? Fortunately, you can work around this problem by creating your own base image which is trivially easy. So, now I build all images, from scratch, locally. That helps put Debian developers back into the mix. Next, I started looking at Docker Files that I could use to construct my images and discovered something else troubling. Take, for example, the official nginx Docker image. It is based on Debian Jessie (hooray - our distribution developers are in the mix!). However, it then proceeds to install nginx from the nginx repository, not the debian repository. Well, I guess if you are nginx you want to have full control, but still, the Debian developers version of nginx has been vetted to ensure it works with Debian Jessie, so you are really losing something here. So... my next break from Docker convention is to use all Debian packages in my Docker files. Once my images were built and my application was tested and running, I was done, right? Wrong. Well, I would have been done if I didn't care about upgrading my systems, backing up my data or running cron jobs. Remember those things? Those are things that distribution developers and system administrators have been perfecting for decades. And, they are not trivial. I then built a series of scripts to help alert me when a image I am using has updates (you can't just use cron-apt any more since cron isn't running in your container) and help me update the image and deploy it to all my applications (which involves restarting the application). Backing up data is a whole different can of worms - sometimes involving interacting with your container (if it's a database container, you have to launch a database dump) or simply copying files from the host, assuming you got the right Docker volume strategy (which took me days to fully understand). Lastly, I had to run a cron job from the host that then runs whatever commands are needed on each of my containers. This was complicated. In the end, was it worth it? Yes, I think so. However, not because it was simple, which seems to be the Docker mantra. I think it's worth it because I can run 100 instances of my application using significantly less resources than when I was using full virtualization and because I can more easily and flexibly adjust the resource allocation. However, check in with me in a year and I'll probably have a different opinion.

1 August 2015

Russ Allbery: Review: The Pyramid Waltz

Review: The Pyramid Waltz, by Barbara Ann Wright
Series: Katya and Starbride #1
Publisher: Bold Strokes
Copyright: September 2012
ISBN: 1-60282-792-3
Format: Kindle
Pages: 264
Princess Katya Nar Umbriel is publicly a bored, womanizing, and difficult daughter to the rulers of Farraday. It's all an act, though, with the full knowledge of her parents. As the second child, she's the leader of the Order of Vestra: the equivalent of the Secret Service, devoted to protecting the royal family and, by extension, the kingdom, particularly against magical attacks. Starbride is new to court and entirely out of place. From a northern neighboring country, and far more comfortable in practical clothing than the frilled court dresses that her mother wants her to wear, she has been sent to court to make contacts. Her people are getting the bad side of various trade contracts and desperately need some political maneuvering space of their own. Starbride's best hope for this is to study law in the palace library when she can manage to avoid the other courtiers. But then she and Katya stumble across each other, outside of the roles they're playing, and might have an opportunity for a deeper connection. One that neither of them want to entangle in their personal worries. This is the last of a set of books I picked up while looking for lesbian romance with fantasy or science fiction elements. On the romance front, it's one of the better entries in that set. Both Katya and Starbride are likeable, in large part due to their mutual exasperation with the trappings of the court. (Making the protagonists more serious, thoughtful, and intelligent than the surrounding characters is an old trick, but it works.) Wright has a good ear for banter, particularly the kind when two people of good will are carefully feeling each other out. And despite Katya's need to keep a deep secret from Starbride for some of the book, The Pyramid Waltz mostly avoids irritating communication failures as a plot driver. The fantasy portion and the plot drivers, alas, are weaker. The world building is not exactly bad, but it's just not that interesting. There are a couple of moderately good ideas, in the form of pyramid magic and secret (and dangerous) magical powers that run in the royal family, but they're not well-developed. Pyramid magic turns out to look much like any other generic fantasy magic system, with training scenes that could have come from a Valdemar or Wheel of Time novel (and without as much dramatic tension). And the royal family's secret, while better-developed and integral to the plot, still felt rather generic and one-sided. Maybe that's something Wright develops better in future novels in this series, but that was another problem: the ending of The Pyramid Waltz was rather weak. Partly, I think, this is because the cast is too large and not well-developed. I cared about Katya and Starbird, and to a lesser extent their servants and one of the Order members. (Wright has a moderately interesting bit of worldbuilding about how servants work in Starbride's culture, which I wish we'd seen more of.) But there are a bunch of other Order of Vesta members, Katya's family, and various other bits of history and hinted world views, none of which seemed to get much depth. The ending climax involved a lot of revelations and twists that primarily concerned characters I didn't care about. It lost something in the process. This book is clearly set up for a sequel. There is an ending, but it's not entirely satisfying. Unfortunately, despite liking Katya and Starbird a lot, the rest of the story wasn't compelling enough to make me want to buy it, particularly since the series apparently goes through another three books before reaching a real ending. I enjoyed parts of this book, particularly Katya and Starbird feeling each other out and discovering similarities in their outlook. Katya teasing Starbird, and Starbird teasing herself, over her mother's choice of her clothing was probably the best part. It's not bad for what it's trying to do, but I think it's a bit too generic and not satisfying enough to really recommend. Followed by For Want of a Fiend. Rating: 6 out of 10

31 July 2015

Rapha&#235;l Hertzog: My Free Software Activities in July 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 15 hours on Debian LTS. In that time I did the following: Kali Linux / Debian Stretch work kaliKali Linux wants to experiment something close to Debian Constantly Usable Testing: we have a kali-rolling release that is based on Debian Testing and we want to take a new snapshot every 4 months (in order to have 3 releases per year). More specifically we have a kali-dev repository which is exactly Debian Stretch + our own Kali packages (the kali package take precedence) updated 4 times a day, just like testing is. And we have a britney2 setup that generates kali-rolling out of kali-dev (without any requirement in terms of delay/RC bugs, it just ensures that dependencies are not broken), also 4 times a day. We have jenkins job that ensures that our metapackages are installable in kali-dev (and kali-rolling) and that we can build our ISO images. When things break, I have to fix them and I try to fix them on the Debian side first. So here are some examples of stuff I did in response to various failures: If you analyze this list, you will see that a large part of the issues we had come down to package getting removed from testing due to RC bugs. We should be able to anticipate those issues and monitor the packages that have an impact on Kali. We will probably add new jenkins job that installs all the metapackages and then run how-can-i-help -s testing-autorm --old I just submitted #794238 as a wishlist against how-can-i-help. At the same time, there are bugs that make it into testing and that I fix / work around on the Kali side. But those fixes / work around might be more useful if they were pushed to testing via testing-proposed-updates. I tried to see whether other derivatives had similar needs to see if derivatives could join their efforts at this level but it does not look like so for now. Last but not least, bugs reported on the Kali side also resulted in Debian improvements: Other Debian work Sponsorship. I sponsored multiple packages for Daniel Stender who is packaging prospector, a software that I requested earlier (through RFP bug). So I reviewed and uploaded python-requirements-detector, python-setoptconf, pylint-celery and pylint-common. During a review I also discovered a nice bug in dh-python (#793609a comment in the middle of a Build-Depends could break a package). I also sponsored an upload of notmuch-addrlookup (new package requested by a Freexian customer). Packaging. I uploaded python-django 1.7.9 in unstable and 1.8.3 in experimental to fix security issues. I uploaded a new upstream release of ditaa through a non-maintainer uploaded (again at the request of a Freexian customer). Distro Tracker. Beside the work to integrate detailed security status, I fixed the code to be compatible with Django 1.8 and modified the tox configuration to ensure that the test suite is regularly run against Django 1.8. I also merged multiple patches of Christophe Siraut (cf #784151 and #754413). Thanks See you next month for a new summary of my activities.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

28 July 2015

Lisandro Dami&aacute;n Nicanor P&eacute;rez Meyer: Plasma/KF5 : Testing situation

Dear Debian/KDE users,

We are aware that the current situation in testing is very unfortunate, with two main issues:

  1. systemsettings transitioned to testing before the corresponding KDE Control Modules. The result is that systemsettings displays an empty screen. This is tracked in the following bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=790703.
  2. plasmoids such as plasma-nm transitioned to testing before plasma-desktop 5. The result is that the plasmoid are no longer displayed in the system tray.

We are working on getting plasma-desktop to transition to testing as soon as possible (hopefully in 2 days time), which will resolve both those issues. We appreciate that the transition to KF5 is much rougher than we would have liked, and apologize to all those impacted.

On behalf of the Qt/KDE team,
Lisandro.

27 July 2015

Kees Cook: 3D printing Poe

I helped print this statue of Edgar Allan Poe, through We the Builders , who coordinate large-scale crowd-sourced 3D print jobs: Poe's Face You can see one of my parts here on top, with -Kees on the piece with the funky hair strand: Poe's Hair The MakerWare I run on Ubuntu works well. I wish they were correctly signing their repositories. Even if I use non-SSL to fetch their key, as their Ubuntu/Debian instructions recommend, it still doesn t match the packages: 
W: GPG error: http://downloads.makerbot.com trusty Release: The following signatures were invalid: BADSIG 3D019B838FB1487F MakerBot Industries dev team <dev@makerbot.com>
And it s not just my APT configuration: 
$ wget http://downloads.makerbot.com/makerware/ubuntu/dists/trusty/Release.gpg
$ wget http://downloads.makerbot.com/makerware/ubuntu/dists/trusty/Release
$ gpg --verify Release.gpg Release
gpg: Signature made Wed 11 Mar 2015 12:43:07 PM PDT using RSA key ID 8FB1487F
gpg: requesting key 8FB1487F from hkp server pgp.mit.edu
gpg: key 8FB1487F: public key "MakerBot Industries LLC (Software development team) <dev@makerbot.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
gpg: BAD signature from "MakerBot Industries LLC (Software development team) <dev@makerbot.com>"
$ grep ^Date Release
Date: Tue, 09 Jun 2015 19:41:02 UTC
Looks like they re updating their Release file without updating the signature file. (The signature is from March, but the Release file is from June. Oops!)

2015, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

Dirk Eddelbuettel: Evading the "Hadley tax": Faster Travis tests for R

Hadley is a popular figure, and rightly so as he successfully introduced many newcomers to the wonders offered by R. His approach strikes some of us old greybeards as wrong---I particularly take exception with some of his writing which frequently portrays a particular approach as both the best and only one. Real programming, I think, is often a little more nuanced and aware of tradeoffs which need to be balanced. As a book on another language once popularized: "There is more than one way to do things." But let us leave this discussion for another time. As the reach of the Hadleyverse keeps spreading, we sometimes find ourselves at the receiving end of a cost/benefit tradeoff. That is what this post is about, and it uses a very concrete case I encountered yesterday. As blogged earlier, the RcppZiggurat package was updated. I had not touched it in a year, but Brian Ripley had sent a brief and detailed note concerning something flagged by the Solaris compiler (correctly suggesting I replace fabs() with abs() on integer types). (Allow me to stray from the main story line here for a second to stress just how insane a work load he is carrying, essentially for all of us. R and the R community are so just so indebted to him for all his work---which makes the usual social media banter about him so unfortunate. But that too shall be left for another time.) Upon making the simple fix, and submitting to GitHub the usual Travis CI was triggered. And here is what I saw: first travis build in a year
All happy, all green. Previous build a year ago, most recent build yesterday, both passed. But hold on: test time went from 2:54 minutes to 7:47 minutes for an increase of almost five minutes! And I knew that I had not added any new dependencies, or altered any build options. What did happen was that among the dependencies of my package, one had decided to now also depend on ggplot2. Which leads to a chain of sixteen additional packages being loaded besides the four I depend upon---when it used to be just one. And that took five minutes as all those packages are installed from source, and some are big and take a long time to compile. There is however and easy alternative, and for that we have to praise Michael Rutter who looks after a number of things for R on Ubuntu. Among these are the R builds for Ubuntu but also the rrutter PPA as well as the c2d4u PPA. If you have not heard this alphabet soup before, a PPA is a package repository for Ubuntu where anyone (who wants to sign up) can upload (properly setup) source files which are then turned into Ubuntu binaries. With full dependency resolution and all other goodies we have come to expect from the Debian / Ubuntu universe. And Michael uses this facility with great skill and calm to provide us all with Ubuntu binaries for R itself (rebuilding what yours truly uploads into Debian), as well as a number of key packages available via the CRAN mirrors. Less know however is this "c2d4u" which stands for CRAN to Debian for Ubuntu. And this builds on something Charles Blundell once built under my mentorship in a Google Summer of Code. And Michael does a tremdous job covering well over a thousand CRAN source packages---and providing binaries for all. Which we can use for Travis! What all that means is that I could now replace the line 
 - ./travis-tool.sh install_r RcppGSL rbenchmark microbenchmark highlight
which implies source builds of the four listed packages and all their dependencies with the following line implying binary installations of already built packages: 
 - ./travis-tool.sh install_aptget libgsl0-dev r-cran-rcppgsl r-cran-rbenchmark r-cran-microbenchmark r-cran-highlight
In this particular case I also needed to build a binary package of my RcppGSL package as this one is not (yet) handled by Michael. I happen to have (re-)discovered the beauty of PPAs for Travis earlier this year and revitalized an older and largely dormant launchpad account I had for this PPA of mine. How to build a simple .deb package will also have to left for a future post to keep this more concise. This can be used with the existing r-travis setup---but one needs to use the older, initial variant in order to have the ability to install .deb packages. So in the .travis.yml of RcppZiggurat I just use 
before_install:
## PPA for Rcpp and some other packages
- sudo add-apt-repository -y ppa:edd/misc
## r-travis by Craig Citro et al
- curl -OL http://raw.github.com/craigcitro/r-travis/master/scripts/travis-tool.sh
- chmod 755 ./travis-tool.sh
- ./travis-tool.sh bootstrap
to add my own PPA and all is good. If you do not have a PPA, or do not want to create your own packages you can still benefit from the PPAs by Michael and "mix and match" by installing from binary what is available, and from source what is not. Here we were able to use an all-binary approach, so let's see the resulting performance: latest travis build
Now we are at 1:03 to 1:15 minutes---much better. So to conclude, while the every expanding universe of R packages is fantastic for us as users, it can be seen to be placing a burden on us as developers when installing and testing. Fortunately, the packaging infrastructure built on top of Debian / Ubuntu packages can help and dramatically reduce build (and hence test) times. Learning about PPAs can be a helpful complement to learning about Travis and continued integration. So maybe now I need a new reason to blame Hadley? Well, there is always snake case ...

Follow-up: The post got some pretty immediate feedback shortly after I posted it. Craig Citro pointed out (quite correctly) that I could use r_binary_install which would also install the Ubuntu binaries based on their R packages names. Having built R/CRAN packages for Debian for so long, I am simply more used to the r-cran-* notations, and I think I was also the one contributing install_aptget to r-travis ... Yihui Xie spoke up for the "new" Travis approach deploying containers, caching of packages and explicit whitelists. It was in that very (GH-based) discussion that I started to really lose faith in the new Travis approach as they want use to whitelist each and every package. With 6900 and counting at CRAN I fear this simply does not scale. But different approaches are certainly welcome. I posted my 1:03 to 1:15 minutes result. If the "New School" can do it faster, I'd be all ears.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

26 July 2015

Dirk Eddelbuettel: RcppZiggurat 0.1.3: Faster Random Normal Draws

ziggurats After a slight hiatus since the last release in early 2014, we are delighted to announce a new release of RcppZiggurat which is now on the CRAN network for R. The RcppZiggurat package updates the code for the Ziggurat generator which provides very fast draws from a Normal distribution. The package provides a simple C++ wrapper class for the generator improving on the very basic macros, and permits comparison among several existing Ziggurat implementations. This can be seen in the figure where Ziggurat from this package dominates accessing the implementations from the GSL, QuantLib and Gretl---all of which are still way faster than the default Normal generator in R (which is of course of higher code complexity). This release contains a few internal cleanups relative to the last release. It was triggered by a very helpful email from Brian Ripley who notices compiler warnings on the Solaris platform due to my incorrect use of on integer variables. The NEWS file entry below lists all changes.
Changes in version 0.1.3 (2015-07-25)
  • Use the SHR3 generator for the default implementation just like Leong et al do, making our default implementation identical to theirs (but 32- and 64-bit compatible)
  • Switched generators from float to double ensuring that results are identical on 32- and 64-bit platforms
  • Simplified builds with respect to GSL use via the RcppGSL package; added a seed setter for the GSL variant
  • Corrected use of fabs() to abs() on integer variables, with a grateful nod to Brian Ripley for the hint (based on CRAN checks on the beloved Slowlaris machines)
  • Accelerated Travis CI tests by relying exclusively on r-cran-* packages from the PPAs by Michael Rutter and myself
  • Updated DESCRIPTION and NAMESPACE according to current best practices, and R-devel CMD check --as-cran checks
Courtesy of CRANberries, there is also a diffstat report for the most recent release. More information is on the RcppZiggurat page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

25 July 2015

Dirk Eddelbuettel: Rcpp 0.12.0: Now with more Big Data!

big-data image A new release 0.12.0 of Rcpp arrived on the CRAN network for GNU R this morning, and I also pushed a Debian package upload. Rcpp has become the most popular way of enhancing GNU R with C++ code. As of today, 423 packages on CRAN depend on Rcpp for making analyses go faster and further. Note that this is 60 more packages since the last release in May! Also, BioConductor adds another 57 packages, and casual searches on GitHub suggests many more. And according to Andrie De Vries, Rcpp has now page rank of one on CRAN as well! And with this release, Rcpp also becomes ready for Big Data, or, as they call it in Texas, Data. Thanks to a lot of work and several pull requests by Qiang Kou, support for R_xlen_t has been added. That means we can now do stunts like 
R> library(Rcpp)
R> big <- 2^31-1
R> bigM <- rep(NA, big)
R> bigM2 <- c(bigM, bigM)
R> cppFunction("double getSz(LogicalVector x)   return x.length();  ")
R> getSz(bigM)
[1] 2147483647
R> getSz(bigM2)
[1] 4294967294
R>
where prior versions of Rcpp would just have said 
> getSz(bigM2)
Error in getSz(bigM2) :
  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:137
>
which is clearly not Texas-style. Another wellcome change, also thanks to Qiang Kou, adds encoding support for strings. A lot of other things got polished. We are still improving exception handling as we still get the odd curveballs in a corner cases. Matt Dziubinski corrected the var() computation to use the proper two-pass method and added better support for lambda functions in Sugar expression using sapply(), Qiang Kou added more pull requests mostly for string initialization, and Romain added a pull request which made data frame creation a little more robust, and JJ was his usual self in tirelessly looking after all aspects of Rcpp Attributes. As always, you can follow the development via the GitHub repo and particularly the Issue tickets and Pull Requests. And any discussions, questions, ... regarding Rcpp are always welcome at the rcpp-devel mailing list. Last but not least, we are also extremely pleased to annouce that Qiang Kou has joined us in the Rcpp-Core team. We are looking forward to a lot more awesome! See below for a detailed list of changes extracted from the NEWS file.
Changes in Rcpp version 0.12.0 (2015-07-24)
  • Changes in Rcpp API:
    • Rcpp_eval() no longer uses R_ToplevelExec when evaluating R expressions; this should resolve errors where calling handlers (e.g. through suppressMessages()) were not properly respected.
    • All internal length variables have been changed from R_len_t to R_xlen_t to support vectors longer than 2^31-1 elements (via pull request 303 by Qiang Kou).
    • The sugar function sapply now supports lambda functions (addressing issue 213 thanks to Matt Dziubinski)
    • The var sugar function now uses a more robust two-pass method, supports complex numbers, with new unit tests added (via pull request 320 by Matt Dziubinski)
    • String constructors now allow encodings (via pull request 310 by Qiang Kou)
    • String objects are preserving the underlying SEXP objects better, and are more careful about initializations (via pull requests 322 and 329 by Qiang Kou)
    • DataFrame constructors are now a little more careful (via pull request 301 by Romain Francois)
    • For R 3.2.0 or newer, Rf_installChar() is used instead of Rf_install(CHAR()) (via pull request 332).
  • Changes in Rcpp Attributes:
    • Use more robust method of ensuring unique paths for generated shared libraries.
    • The evalCpp function now also supports the plugins argument.
    • Correctly handle signature termination characters (' ' or ';') contained in quotes.
  • Changes in Rcpp Documentation:
    • The Rcpp-FAQ vignette was once again updated with respect to OS X issues and Fortran libraries needed for e.g. RcppArmadillo.
    • The included Rcpp.bib bibtex file (which is also used by other Rcpp* packages) was updated with respect to its CRAN references.
Thanks to CRANberries, you can also look at a diff to the previous release As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Michal &#268;iha&#345;: Migrating phpMyAdmin from SourceForge.net

Some time ago we've decided to move phpMyAdmin out of SourceForge.net services. This was mostly motivated by issues with sf.net bundling crapware with installers (though we were not affected), but also we've missed some features that we would like to have and were not possible there. The project relied on SourceForge.net with several services. The biggest ones being website and downloads hosting, issue tracking and mailing lists. We've chosen different approach for each of these. As first, we've moved away website and downloads. Thanks to generous offer of CDN77.com, everything went quite smoothly and we now have HTTPS secured website and downloads, see our announcement. Oh and on the way we've started to PGP sign the releases as well, so you can verify the download. Shortly after this SourceForge.net was hit by major problems with infrastructure. Unfortunately we were not yet completely ready with rest of the migration, but this has definitely pushed us to make progress faster. During the outage, we've opened up issue tracker on GitHub, to be able to receive bug reports from our users. On the background I've worked on the issue migration. The good news is that as of now almost all issues are migrated. There are few missing ones, but these will be hopefully handled in upcoming days as well. Last but not least, we had mailing lists on SourceForge.net. We've shortly discussed available options and decided to run own mail server with these. It will allow us greater flexibility while still using well know software in background. Initial attempts with Mailman 3 failed, so we got back to Mailman 2, which is stable and easy to configure. See also our news posts for official announcement. Thanks to SourceForge.net, it has been great home for us, but now we have better places to live.

Filed under: English phpMyAdmin 0 comments

24 July 2015

Vincent Sanders: NetSurf developers and the Order of the Phoenix

Once more the NetSurf developers gathered to battle the forces of darkness, or as they are more commonly known web specifications.

Michael Drake, Vincent Sanders, John-Mark Bell and Daniel Silverstone at the Codethink manchester officesThe fifth developer weekend was an opportunity for us to gather in a pleasant setting and work together in person. We were graciously hosted, once again, by Codethink in their Manchester offices.

Four developers managed to attend in person from around the UK: Michael Drake, John-Mark Bell, Daniel Silverstone and Vincent Sanders.

The main focus of the weekends activities was to address two areas that have become overwhelmingly important: JavaScript and Layout.

Although the browser obviously already has both these features they are somewhat incomplete and incapable of supporting the features of the modern web.
JavaScriptThe discussion started with JavaScript and its implementation. We had previously looked at the feasibility of changing our JavaScript engine from Spidermonkey to DukTape. We had decided this was a change we wanted to make when DukTape was mature enough to support the necessary features.

The main reasons for the change are that Spidermonkey is a poor fit to NetSurf as it is relatively large and does not provide a stable API guarantee. The lack of a stable API requires extensive engineering to update to new releases. Additionally support for compiling on minority platforms is very challenging meaning that most platforms are stuck using version 1.7 or 1.85 (current release is version 31 with 38 due).

We started the move to Duktape by creating a development branch, integrating the Duktape library and open coding a minimal implementation of the core classes as a proof of concept. This work was mostly undertaken by Daniel with input from myself and John-Mark. This resulted in a build that was substantially smaller than using Spidermonkey with all the existing functionality our tests cover.

The next phase of this work is to take the prototype implementation and turn it into something that can be reliably used and covers the entire JavaScript DOM interface. This is no small job as there are at least 200 classes and 1500 methods and properties to implement.
LayoutThe layout library design discussion was an extensive and very involved. The layout engine is a browsers most important component. It takes all the information processed by the CSS and DOM libraries, applies a vast number of involved rules and produces a list of operations that can be rendered.

This reimplementation of our rendering engine has been in planning for many years. The existing engine stems from the browsers earliest days more than a decade ago and has many shortcomings in architecture and implementation we hope to address.

The work has finally started on libnslayout with Michael taking the lead and defining the initial API and starting the laborious work of building the test harness, a feature the previous implementation lacked!
The second war beginsFor a war you need people and it is a little unfortunate that this was our lowest ever turnout for the event. This is true of the project overall with declining numbers of commits and interest outside our core group. If anyone is interested we are always happy to have new contributors and there are opportunities to contribute in many areas from image assets, through translations, to C programming.

We discussed some ways to encourage new developers and try and get committed developers especially for the minority platform frontends. The RISC OS frontend for example has needed a maintainer since the previous one stepped down. There was some initial response from its community, culminating in a total of two patches, when we announced the port was under threat of not being releasable in future. Unfortunately nothing further came from this and it appears our oldest frontend may soon become part of our history.

We also covered some issues from the bug tracker mostly to see if there were any patterns that we needed to address before the forthcoming 3.4 release.

There was discussion about recent improvements to the CI system which generate distribution packages from the development branch and how this could be extended to benefit more users. This also included authorisation to acquire storage and other miscellaneous items necessary to keep the project infrastructure running.

We managed over 20 hours of work in the two days and addressed our current major shortcomings. Now it just requires a great deal of programming to complete the projects started here.

23 July 2015

Antoine Beaupr : Is it safe to use open wireless access points?

I sometimes get questions when people use my wireless access point, which, for as long as I can remember, has been open to everyone; that is without any form of password protection or encryption. I arguably don't use the access point much myself, as I prefer the wired connection for the higher bandwidth, security and reliability it provides. Apart from convenience for myself and visitors, the main reason why I leave my wireless access open is that I believe in a free (both as in beer and freedom) internet, built with principles of solidarity rather than exploitation and profitability. In these days of ubiquitous surveillance, freedom often goes hand in hand with anonymity, which implies providing free internet access to everyone. I also believe that, as more and more services get perniciously transferred to the global internet, access to the network is becoming a basic human right. This is therefore my small contribution to the struggle, now also part of the R seau Libre project. So here were my friends question, in essence:
My credit card info was stolen when I used a wifi hotspot in an airport... Should I use open wifi networks? Is it safe to use my credit card for shopping online?
Here is a modified version of an answer I sent to a friend recently which I thought could be useful to the larger internet community. The short answer is "sorry about that", "it depends, you generally can, but be careful" and "your credit card company is supposed to protect you".

Sorry! First off, sorry to hear that our credit card was stolen in an airport! That has to be annoying... Did the credit card company reimburse you? Normally, the whole point of credit cards is that they protect you in case of theft like this and they are supposed to reimburse you if you credit card gets stolen or abused...

The complexity and unreliability of passwords Now of course, securing every bit of your internet infrastructure helps in protecting against such attacks. However: there is a trade-off! First off, it does makes it more complicated for people to join the network. You need to make up some silly password (which has its own security problems: passwords can be surprisingly easy to guess!) that you will post on the fridge or worst, forget all the time! And if it's on the fridge, anyone with a view to that darn fridge, be it one-time visitor or sneaky neighbor, can find the password and steal your internet access (although, granted, that won't allow them to directly spy on your internet connection). In any case, if you choose to use a password, you should use the tricks I wrote in the koumbit wiki to generate the password and avoid writing it on the fridge.

The false sense of security of wireless encryption Second, it can also give a false sense of security: just because a wifi access point appears "secure" (ie. that the communication between your computer and the wifi access point is encrypted) doesn't mean the whole connection is secure. In fact, one attack that can be done against access points is exactly to masquerade as an existing access point, with no security security at all. That way, instead of connecting to the real secure and trusted access point, you connect to an evil one which spies on our connection. Most computers will happily connect to such a hotspot even with degraded security without warning. It may be what happened at the airport, in fact. Of course this particular attack would be less likely to happen if you live in the middle of the woods than an airport, but it's some important distinction to keep in mind, because the same attack can be performed after the wireless access point, for example by your countryside internet access provider or someone attacking it. Your best protection for your banking details is to rely on good passwords (for your back account) but also, and more importantly, what we call end-to-end encryption. That is usually implemented using the "HTTPS" with a pad lock icon in your address bar. This ensures that the communication between your computer and the bank or credit card company is secure, that is: that no wifi access point or attacker between your computer and them can intercept your credit card number.

The flaws of internet security Now unfortunately, even the HTTPS protocol doesn't bring complete security. For example, one attack that can be done is similar to the previous one and that is to masquerade as a legitimate bank site, but either strip out the encryption or even fake the encryption. So you also need to look at the address of the website you are visiting. Attackers are often pretty clever and will use many tricks to hide the real address of the website in the address bar. To work around this, I always explicitly type my bank website address (https://accesd.desjardins.com/ in my case) directly myself instead of clicking on links, bookmarks or using a search engine to find my bank site. In the case of credit cards, it is much trickier because when you buy stuff online, you end up putting that credit card number on different sites which you do not necessarily trust. There's no good solution but complaining to your credit card company if you believe a website you used has stolen your credit card details. You can also use services like Paypal, Dwolla or Bitcoin that hide your credit card details from the seller, if they support the service. I usually try to avoid putting my credit card details on sites I do not trust, and limit myself to known parties (e.g. Via Rail, Air Canada, etc). Also, in general, I try to assume the network connection between me and the website I visit is compromised. This forced me to get familiar with online security and use of encryption. It is more accessible to me than trying to secure the infrastructure i am using, because i often do not control it at all (e.g. internet cafes...). Internet security is unfortunately a hard problem, and things are not getting easier as more things move online. The burden is on us programmers and system administrators to create systems that are more secure and intuitive for our users so, as I said earlier, sorry the internet sucks so much, we didn't think so many people would join the acid trip of the 70s. ;)

Next.

Previous.