Search Results: "beh"

14 December 2016

Antoine Beaupr : Django debates privacy concern

In recent years, privacy issues have become a growing concern among free-software projects and users. As more and more software tasks become web-based, surveillance and tracking of users is also on the rise. While some software may use advertising as a source of revenue, which has the side effect of monitoring users, the Django community recently got into an interesting debate surrounding a proposal to add user tracking actually developer tracking to the popular Python web framework.

Tracking for funding A novel aspect of this debate is that the initiative comes from concerns of the Django Software Foundation (DSF) about funding. The proposal suggests that "relying on the free labor of volunteers is ineffective, unfair, and risky" and states that "the future of Django depends on our ability to fund its development". In fact, the DSF recently hired an engineer to help oversee Django's development, which has been quite successful in helping the project make timely releases with fewer bugs. Various fundraising efforts have resulted in major new Django features, but it is difficult to attract sponsors without some hard data on the usage of Django. The proposed feature tries to count the number of "unique developers" and gather some metrics of their environments by using Google Analytics (GA) in Django. The actual proposal (DEP 8) is done as a pull request, which is part of Django Enhancement Proposal (DEP) process that is similar in spirit to the Python Enhancement Proposal (PEP) process. DEP 8 was brought forward by a longtime Django developer, Jacob Kaplan-Moss. The rationale is that "if we had clear data on the extent of Django's usage, it would be much easier to approach organizations for funding". The proposal is essentially about adding code in Django to send a certain set of metrics when "developer" commands are run. The system would be "opt-out", enabled by default unless turned off, although the developer would be warned the first time the phone-home system is used. The proposal notes that an opt-in system "severely undercounts" and is therefore not considered "substantially better than a community survey" that the DSF is already doing.

Information gathered The pieces of information reported are specifically designed to run only in a developer's environment and not in production. The metrics identified are, at the time of writing:
  • an event category (the developer commands: startproject, startapp, runserver)
  • the HTTP User-Agent string identifying the Django, Python, and OS versions
  • a user-specific unique identifier (a UUID generated on first run)
The proposal mentions the use of the GA aip flag which, according to GA documentation, makes "the IP address of the sender 'anonymized'". It is not quite clear how that is done at Google and, given that it is a proprietary platform, there is no way to verify that claim. The proposal says it means that "we can't see, and Google Analytics doesn't store, your actual IP". But that is not actually what Google does: GA stores IP addresses, the documentation just says they are anonymized, without explaining how. GA is presented as a trade-off, since "Google's track record indicates that they don't value privacy nearly as high" as the DSF does. The alternative, deploying its own analytics software, was presented as making sustainability problems worse. According to the proposal, Google "can't track Django users. [...] The only thing Google could do would be to lie about anonymizing IP addresses, and attempt to match users based on their IPs". The truth is that we don't actually know what Google means when it "anonymizes" data: Jannis Leidel, a Django team member, commented that "Google has previously been subjected to secret US court orders and was required to collaborate in mass surveillance conducted by US intelligence services" that limit even Google's capacity of ensuring its users' anonymity. Leidel also argued that the legal framework of the US may not apply elsewhere in the world: "for example the strict German (and by extension EU) privacy laws would exclude the automatic opt-in as a lawful option". Furthermore, the proposal claims that "if we discovered Google was lying about this, we'd obviously stop using them immediately", but it is unclear exactly how this could be implemented if the software was already deployed. There are also concerns that an implementation could block normal operation, especially in countries (like China) where Google itself may be blocked. Finally, some expressed concerns that the information could constitute a security problem, since it would unduly expose the version number of Django that is running.

In other projects Django is certainly not the first project to consider implementing analytics to get more information about its users. The proposal is largely inspired by a similar system implemented by the OS X Homebrew package manager, which has its own opt-out analytics. Other projects embed GA code directly in their web pages. This is apparently the option chosen by the Oscar Django-based ecommerce solution, but that was seen by the DSF as less useful since it would count Django administrators and wasn't seen as useful as counting developers. Wagtail, a Django-based content-management system, was incorrectly identified as using GA directly, as well. It actually uses referrer information to identify installed domains through the version updates checks, with opt-out. Wagtail didn't use GA because the project wanted only minimal data and it was worried about users' reactions. NPM, the JavaScript package manager, also considered similar tracking extensions. Laurie Voss, the co-founder of NPM, said it decided to completely avoid phoning home, because "users would absolutely hate it". But NPM users are constantly downloading packages to rebuild applications from scratch, so it has more complete usage metrics, which are aggregated and available via a public API. NPM users seem to find this is a "reasonable utility/privacy trade". Some NPM packages do phone home and have seen "very mixed" feedback from users, Voss said. Eric Holscher, co-founder of Read the Docs, said the project is considering using Sentry for centralized reporting, which is a different idea, but interesting considering Sentry is fully open source. So even though it is a commercial service (as opposed to the closed-source Google Analytics), it may be possible to verify any anonymity claims.

Debian's response Since Django is shipped with Debian, one concern was the reaction of the distribution to the change. Indeed, "major distros' positions would be very important for public reception" to the feature, another developer stated. One of the current maintainers of Django in Debian, Rapha l Hertzog, explicitly stated from the start that such a system would "likely be disabled by default in Debian". There were two short discussions on Debian mailing lists where the overall consensus seemed to be that any opt-out tracking code was undesirable in Debian, especially if it was aimed at Google servers. I have done some research to see what, exactly, was acceptable as a phone-home system in the Debian community. My research has revealed ten distinct bug reports against packages that would unexpectedly connect to the network, most of which were not directly about collecting statistics but more often about checking for new versions. In most cases I found, the feature was disabled. In the case of version checks, it seems right for Debian to disable the feature, because the package cannot upgrade itself: that task is delegated to the package manager. One of those issues was the infamous "OK Google" voice activation binary blog controversy that was previously reported here and has since then been fixed (although other issues remain in Chromium). I have also found out that there is no clearly defined policy in Debian regarding tracking software. What I have found, however, is that there seems to be a strong consensus in Debian that any tracking is unacceptable. This is, for example, an extract of a policy that was drafted (but never formally adopted) by Ian Jackson, a longtime Debian developer:
Software in Debian should not communicate over the network except: in order to, and as necessary to, perform their function[...]; or for other purposes with explicit permission from the user.
In other words, opt-in only, period. Jackson explained that "when we originally wrote the core of the policy documents, the DFSG [Debian Free Software Guidelines], the SC [Social Contract], and so on, no-one would have considered this behaviour acceptable", which explains why no explicit formal policy has been adopted yet in the Debian project. One of the concerns with opt-out systems (or even prompts that default to opt-in) was well explained back then by Debian developer Bas Wijnen:
It very much resembles having to click through a license for every package you install. One of the nice things about Debian is that the user doesn't need to worry about such things: Debian makes sure things are fine.
One could argue that Debian has its own tracking systems. For example, by default, Debian will "phone home" through the APT update system (though it only reports the packages requested). However, this is currently not automated by default, although there are plans to do so soon. Furthermore, Debian members do not consider APT as tracking, because it needs to connect to the network to accomplish its primary function. Since there are multiple distributed mirrors (which the user gets to choose when installing), the risk of surveillance and tracking is also greatly reduced. A better parallel could be drawn with Debian's popcon system, which actually tracks Debian installations, including package lists. But as Barry Warsaw pointed out in that discussion, "popcon is 'opt-in' and [...] the overwhelming majority in Debian is in favour of it in contrast to 'opt-out'". It should be noted that popcon, while opt-in, defaults to "yes" if users click through the install process. [Update: As pointed out in the comments, popcon actually defaults to "no" in Debian.] There are around 200,000 submissions at this time, which are tracked with machine-specific unique identifiers that are submitted daily. Ubuntu, which also uses the popcon software, gets around 2.8 million daily submissions, while Canonical estimates there are 40 million desktop users of Ubuntu. This would mean there is about an order of magnitude more installations than what is reported by popcon. Policy aside, Warsaw explained that "Debian has a reputation for taking privacy issues very serious and likes to keep it".

Next steps There are obviously disagreements within the Django project about how to handle this problem. It looks like the phone-home system may end up being implemented as a proxy system "which would allow us to strip IP addresses instead of relying on Google to anonymize them, or to anonymize them ourselves", another Django developer, Aymeric Augustin, said. Augustin also stated that the feature wouldn't "land before Django drops support for Python 2", which is currently estimated to be around 2020. It is unclear, then, how the proposal would resolve the funding issues, considering how long it would take to deploy the change and then collect the information so that it can be used to spur the funding efforts. It also seems the system may explicitly prompt the user, with an opt-out default, instead of just splashing a warning or privacy agreement without a prompt. As Shai Berger, another Django contributor, stated, "you do not get [those] kind of numbers in community surveys". Berger also made the argument that "we trust the community to give back without being forced to do so"; furthermore:
I don't believe the increase we might get in the number of reports by making it harder to opt-out, can be worth the ill-will generated for people who might feel the reporting was "sneaked" upon them, or even those who feel they were nagged into participation rather than choosing to participate.
Other options may also include gathering metrics in pip or PyPI, which was proposed by Donald Stufft. Leidel also proposed that the system could ask to opt-in only after a few times the commands are called. It is encouraging to see that a community can discuss such issues without heating up too much and shows great maturity for the Django project. Every free-software project may be confronted with funding and sustainability issues. Django seems to be trying to address this in a transparent way. The project is willing to engage with the whole spectrum of the community, from the top leaders to downstream distributors, including individual developers. This practice should serve as a model, if not of how to do funding or tracking, at least of how to discuss those issues productively. Everyone seems to agree the point is not to surveil users, but improve the software. As Lars Wirzenius, a Debian developer, commented: "it's a very sad situation if free software projects have to compromise on privacy to get funded". Hopefully, Django will be able to improve its funding without compromising its principles.
Note: this article first appeared in the Linux Weekly News.

Dirk Eddelbuettel: anytime 0.1.2: Another bugfix

Another update, now at release 0.1.2, of anytime arrived at CRAN earlier today. anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, ... format to either POSIXct or Date objects -- and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples, or just consider the following illustration:
R> library(anytime)
R> anytime("20161107 202122")   ## all digits
[1] "2016-11-07 20:21:22 CST"
R> utctime("2016Nov07 202122")  ## UTC parse example
[1] "2016-11-07 14:21:22 CST"
R> 
Release 0.1.2 addresses a somewhat bizarre Windows-only bug reported at GitHub in #33 and at StackOverflow. Formats of the %Y-%b-%d form, ie 2016-Dec-12 for today would fail on Windows as the contiguous string was apparently getting split by a routine looking for splits on spaces. Really strange. Anyway, I switched to using more helper functions from the Boost String Algorithms library, and things are behaving now. An extra shoutout once more to G bor Cs rdi and the R Consortium for the most awesome R-Builder. I was able to test and fix on Windows during the weekend with no access to an actual windows environment. The NEWS file summarises the release:

Changes in anytime version 0.1.2 (2016-12-13)
  • The (internal) string processing and splitting now uses Boost algorithm functions which avoids a (bizarre) bug on Windows.
  • Test coverage was increased.

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

13 December 2016

Shirish Agarwal: Eagle Encounters, pier Stellenbosch

Before starting, have to say hindsight as they say is always 20/20. I was moaning about my 6/7 hour trip few blog posts back but now came to know about the 17.5 hr. flights (17.5x800km/hr=14000 km.) which are happening around me. I would say I was whining about nothing seeing those flights. I can t even imagine how people would feel in those flights. Six hours were too much in the tin-can, thankfully though I was in the aisle seat. In 14 hours most people would probably give to Air rage . I just saw an excellent article on the subject. I also came to know that seat-selection and food on a long-haul flights are a luxury, hence that changes the equation quite a bit as well. So on these facts, it seems Qatar Airways treated me quite well as I was able to use both those options. Disclaimer My knowledge about birds/avian is almost non-existent, Hence feel free to correct me if I do go wrong anywhere. Coming back to earth literally , I will have to share a bit of South Africa as that is part and parcel of what I m going to share next. Also many of the pictures shared in this particular blog post belong to KK who has shared them with me with permission to share it with the rest of the world. When I was in South Africa, in the first couple of days as well as what little reading of South African History I had read before travelling, had known that the Europeans, specifically the Dutch ruled on South Africa for many years. What was shared to me in the first day or two that Afrikaans is mostly spoken by Europeans still living in South Africa, some spoken by the coloured people as well. This tied in with the literature I had already read. The Wikipedia page shares which language is spoken by whom and how the demographics play out if people are interested to know that. One of the words or part of the word for places we came to know is bosch as is used in many a places. Bosch means wood or forest. After this we came to know about many places which were known as somethingbosch which signified to us that area is or was a forest. On the second/third day Chirayu (pictured, extreme left) shared the idea of going to Eagle Encounters. Other people pictured in the picture are yours truly, some of the people from GSOC, KK is in the middle, the driver Leonard something who took us to Eagle Encounters on the right (pictured extreme right). Update I was informed that it was a joint plan between Chirayu and KK. They also had some other options planned which later got dropped by the wayside. The whole gang/group along with Leonard coming from eagle encounters It was supposed to be somewhat near, (Spier, Stellenbosch). While I was not able to able to see/figure out where Eagle Encounters is on Openstreetmap, somebody named Firefishy added Spier to OSM few years back. So thank you for that Firefishy so I can at least pin-point a closer place. I didn t see/know/try to figure out about the place as Chirayu said it s a zoo . I wasn t enthusiastic as much as I had been depressed by most zoos in India, while you do have national reserves/Parks in India where you see animals in their full glory. I have been lucky to been able to seen Tadoba and Ranthambore National parks and spend some quality time (about a week) to have some idea as to what can/happens in forests and people living in the buffer-zones but those stories are for a different day altogether. I have to say I do hope to be part of the Ranthambore experience again somewhere in the future, it really is a beautiful place for flora and fauna and fortunately or unfortunately this is the best time apart from spring, as you have the game of mist/fog and animals . North India this time of the year is something to be experienced. I wasn t much enthused as zoos in India are claustrophobic for animals and people both. There are small cages and you see and smell the shit/piss of the animals, generally not a good feeling. Chirayu shared with us also the possibility of being able to ride of Segways and range of bicycles which relieved me so that in case we didn t enjoy the zoo we would enjoy the Segway at least and have a good time (although it would have different expenses than the ones at Eagle Encounters). My whole education about what a zoo could be was turned around at Eagle Encounters as it seems to be somewhere between a zoo and what I know as national parks where animals roam free. We purchased the tickets and went in, the first event/happening was Eagle Encounters itself. One of the families at Eagle Encounter handling a snowy eagle Our introduction to the place started by two beautiful volunteer/trainers who were in charge of all the birds in the Eagle Encounters vicinity. The introduction started by every one of us who came for the Eagle Encounter show to wear a glove and to have/hold one of the pair of snowy owls to sit on the glove. That picture is of a family who was part of our show. Before my turn came, I was a little apprehensive/worried about holding a Owl -period. To my surprise, they were so soft and easy-going, I could hardly feel the weight on my hand. While the trainer/volunteers were constantly feeding them earthworm-bits (I didn t ask, just guessing) and we were all happy as they along with the visitors were constantly playing and interacting with the birds, sharing with us the life-cycle of the snowy Owl. It s only then I understood why in the Harry Potter Universe, the owl plays such an important part. They seem to be a nice, curious, easy-going, proud creatures which fits perfectly in the HP Universe. In hind-sight I should have videod the whole experience as the trainer/volunteer showed a battery of owls, eagles, vultures, Hawks (different birds of prey) what have you. I have to confess my knowledge of birds is and was non-existent. Vulture at the Eagle Encounters show Vulture, One of the larger birds we saw at the Eagle Encounters show. Some of the birds could be dangerous, especially in the wild. The other trainer showing off a Black Eagle at Eagle Encounters That was the other Volunteer-Trainer who was showing off the birds. I especially liked the t-shirt she was wearing. The shop at Eagle Encounters had whole lot of them, they were a bit expensive and just not my size Tidbit Just a few years ago, it was a shocker to me to know/realize that what commonly goes/known in the country as a parrot by most people is actually a Parakeet. As can be seen in the article linked, they are widely distributed in India. While I was young, I used to see the rose-ringed parakeets quite a bit around but nowadays due to probably pollution and other factors, they are noticeably less. They are popular as pets in India. I don t know what Pollito would think about that, don t think he would think good. Trainer showing off a Hawk at Eagle Encounters As I cannot differentiate between Hawk, Vulture, Eagle, etc. I would safely say a Bird of Prey as that was what he was holding. This photo was taken after the event was over where we all were curious to know about the volunteer/trainer, their day job and what it meant for them to be taking care of these birds. Update KK has shared with me what those specific birds are called, so in case the names or species are wrong, please take the truck with her and not me. While I don t remember the name of the trainer/volunteer, among other things it was shared that the volunteers/trainers aren t paid enough and they never have enough funds to take care of all the birds who come to them. Trainer showing Hawk and background chart Where the picture was shot (both this and earlier) was sort of open-office. If you look closely, you will see that there are names of the birds, for instance, people who loved LOTR would easily see Gandalf . that board lists how much food (probably in grams) did the bird eat in a day and week. While it was not shared, I m sure there would be a lot of paperwork, studies to get the birds as well as possible. From a computer science perspective, there seemed to be lot of potential for avian and big-data professionals to do lot of computer modelling and analysis and give more insight into the rehabilitation efforts so the process could be more fine-tuned, efficient and economic perhaps. Hawk on stand This is how we saw the majority of the birds. Most of them had a metal/plastic string which was tied to small artificial branches as the one above. I forgot to share a very important point. Eagle Encounters is not a zoo but a Rehabilitation Centre. While the cynic/skeptic part of me tried to not feel or see the before and after pictures of the birds bought to the rehabilitation centre, the caring part was moved to see most of the birds being treated with love and affection. From our conversations with the Volunteer-Trainer it emerged that every week they had to turn away lots of birds due to space constraints. It is only the most serious/life-threatening cases for which they could provide care in a sustainable way they would keep. Some of the birds who were in the cages were large, airy. I wouldn t say clean as what little I read before as well later is that birds shit enormously so cleaning cages is quite an effort. Most of the cages and near those artificial branches there were placards of people who were sponsoring a bird or two to look after them. From what was shared, many of the birds who came had been abused in many ways. Some of them had their bones crushed or/and other cruel ways. As I had shared that I had been wonderfully surprised by seeing birds come so close to me and most of my friends, I felt rage about those who had treated the birds in such evil, bad ways. What was shared with us that while they try to heal the birds as much as possible, it is always suspect how well the birds would survive on their own in nature, hence many of these birds would go to the sponsor or to some other place when they are well. The Secretary birds - cage- sponsors-adopted If you look at the picture closely, maybe look at the higher resolution photo in the gallery, you will see that both the birds have been adopted by two different couples. The birds as the name tag shows are called Secretaries . The Secretaries make a typical sound which is similar to the sound made by old typewriters. Just as woodpeckers make Morse Code noises when they are pecking with their beaks on trees, something similar to the sound of keys emitted by Old Remington typewriters when clicked on was done by the Secretaries. One of the birds in the cage, This is one of the birds in one of the few cages. If you see a higher-resolution picture of the earlier picture, the one which has Secretaries . Also as can be seen in the picture, there is wood-working happening and they are trying to expand the Rehabilitation Centre. All in all, an excursion which was supposed to be for just an hour, extended to something like 3 odd hours. KK shot more than a 1000 odd pictures while trying to teach/converse in Malyalam to some of the birds. She shot well over 1000 photos which would have filled something like 30 odd traditional photo albums. Jaminy (KK s partner-in-crime) used her selfie stick to desired effect, taking pictures with most of the birds as one does with celebrities. I had also taken some but most of them were over-exposed as was new to mobile photography at that time, still am but mostly it works. Lake with Barn Owls near Eagle Encounters That is the lake we discovered/saw after coming back from Eagle Encounters. We had good times. Lastly, a virtual prize distribution ceremony a. Chirayu and KK A platinum trophy for actually thinking and pitching the place in the first place. b. Shirish and Deven Bansod Metal cups for not taking more than 10 minutes to freshen up and be back after hearing the plan to go to Eagle Encounters. c. All the girls/women Spoons for actually making it to the day. All the girls took quite sometime to freshen up, otherwise it might have been possible to also experience the Segways, who knows. All-in-all an enjoyable day spent in being part of Eagle Encounters .
Filed under: Miscellenous Tagged: #Birds of Prey, #Debconf16, #Eagle Encounters, #Rehabilitation, #South African History, #Stellenbosch

11 December 2016

Colin Watson: The sad tale of CVE-2015-1336

Today I released man-db 2.7.6 (announcement, NEWS, git log), and uploaded it to Debian unstable. The major change in this release was a set of fixes for two security vulnerabilities, one of which affected all man-db installations since 2.3.12 (or 2.3.10-66 in Debian), and the other of which was specific to Debian and its derivatives. It s probably obvious from the dates here that this has not been my finest hour in terms of responding to security issues in a timely fashion, and I apologise for that. Some of this is just the usual life reasons, which I shan t bore you by reciting, but some of it has been that fixing this properly in man-db was genuinely rather complicated and delicate. Since I ve previously advocated man-db over some of its competitors on the basis of a better security posture, I think it behooves me to write up a longer description. I took over maintaining man-db over fifteen years ago in slightly unexpected circumstances (I got annoyed with its bug list and made a couple of non-maintainer uploads, and then the previous maintainer died, so I ended up taking over both in Debian and upstream). I was a fairly new developer at the time, and there weren t a lot of people I could ask questions of, but I did my best to recover as much of the history as I could and learn from it. One thing that became clear very quickly, both from my own inspection and from the bug list, was that most of the code had been written in a rather more innocent time. It was absolutely riddled with dangerous uses of the shell, poor temporary file handling, buffer overruns, and various common-or-garden deficiencies of that kind. I spent several years reworking large swathes of the codebase to be more robust against those kinds of bugs by design, and for example libpipeline came out of that effort. The most subtle and risky set of problems came from the fact that the man and mandb programs were installed set-user-id to the man user. Part of this was so that man could maintain preformatted cat pages , and part of it was so that users could run mandb if the system databases were out of date (this is now much less useful since most package managers, including dpkg, support some kind of trigger mechanism that can run mandb whenever new system-level manual pages are installed). One of the first things I did was to make this optional, and this has been a disabled-by-default debconf option in Debian for a long time now. But it s still a supported option and is enabled by default upstream, and when running setuid man and mandb need to take care to drop privileges when dealing with user-controlled data and to write files with the appropriate ownership and permissions. My predecessor had problems related to this such as Debian #26002, and one of the ways they dealt with them was to make /var/cache/man/ set-group-id root, in order that files written to that directory would have consistent group ownership. This always struck me as rather strange and I meant to do something about it at some point, but until the first vulnerability report above I regarded it as mainly a curiosity, since nothing in there was group-writeable anyway. As a result, with the more immediate aim of making the system behave consistently and dealing with bug reports, various bits of code had accreted that assumed that /var/cache/man/ would be man:root 2755, and not all of it was immediately obvious. This interacted with the second vulnerability report in two ways. Firstly, at some level it caused it because I was dealing with the day-to-day problems rather than thinking at a higher level: a series of bugs led me down the path of whacking problems over the head with a recursive chown of /var/cache/man/ from cron, rather than working out why things got that way in the first place. Secondly, once I d done that, I couldn t remove the chown without a much more extensive excursion into all the code that dealt with cache files, for fear of reintroducing those bugs. So although the fix for the second vulnerability is very simple in itself, I couldn t get there without dealing with the first vulnerability. In some ways, of course, cat pages are a bit of an anachronism. Most modern systems can format pages quickly enough that it s not much of an issue. However, I m loath to drop the feature entirely: I m generally wary of assuming that just because I have a fast system that everyone does. So, instead, I did what I should have done years ago: make man and mandb set-group-id man as well as set-user-id man, at which point we can simply make all the cache files and directories be owned by man:man and drop the setgid bit on cache directories. This should be simpler and less prone to difficult-to-understand problems. I expect that my next substantial upstream release will switch to --disable-setuid by default to reduce exposure, though, and distributions can start thinking about whether they want to follow that (Fedora already does, for example). If this becomes widely disabled without complaints then that would be good evidence that it s reasonable to drop the feature entirely. I m not in a rush, but if you do need cat pages then now is a good time to write to me and tell me why. This is the fiddliest set of vulnerabilities I ve dealt with in man-db for quite some time, so I hope that if there are more then I can get back to my previous quick response time.

9 December 2016

John Goerzen: Giant Concrete Arrows, Old Maps, and Fascinated Kids

Let me set a scene for you. Two children, ages 7 and 10, are jostling for position. There s a little pushing and shoving to get the best view. This is pretty typical for siblings this age. But what, you may wonder, are they trying to see? A TV? Video game? No. Jacob and Oliver were in a library, trying to see a 98-year-old map of the property owners in Township 23, range 1 East, Harvey County, Kansas. And they were super excited about it, somewhat to the astonishment of the research librarian, who I am sure is more used to children jostling for position over the DVDs in the youth section than poring over maps in the non-circulating historical archives! All this started with giant concrete arrows in the middle of nowhere. Nearly a century ago, the US government installed a series of arrows on the ground in Kansas. These were part of a primitive air navigation system that led to the first transcontinental airmail service. Every so often, people stumble upon these abandoned arrows and there is a big discussion online. Even Snopes has had to verify their authenticity (verdict: true). Entire websites exist to tracking and locating the remnants of these arrows. And as one of the early air mail routes went through Kansas, every so often people find these arrows around here. I got the idea that it would be fun to replicate a journey along the old routes. Maybe I d spot a few old arrows and such. So I started collecting old maps: a Contract Airmail Route #34 (CAM 34) map from 1927, aviation sectionals from 1933 and 1946, etc. I noticed an odd thing on these maps: the Newton, KS airport was on the other side of the city from its present location, sometimes even several miles outside the city. What was going on? 1927 Airway Map
(1927 Airway Map) 1946 Wichita Sectional
(1946 Wichita sectional) So one foggy morning, I explained my puzzlement to the boys. I highlighted all the mysteries: were these maps correct? Were there really two Newton airports at one time? How many airports were there, and where were they? Why did they move? What was the story behind them? And I offered them the chance to be history detectives with me. And oh my goodness, were they ever excited! We had some information from a very helpful person at the Harvey County Historical Museum (thanks Kris!) So we suspected one airport at least was established in 1927. We also had a description of its location, though given in terms of township maps. So the boys and I made the short drive over to the museum. We reviewed their property maps, though they were all a little older than the time period we needed. We looked through books and at pictures. Oliver pored over a railroad map of Newton from a century ago, fascinated. Jacob was excited to discover on one map that there used to be a train track down the middle of Main Street! I was interested that the present Newton Airport was once known as Wirt Field, rather to my surprise. I somehow suspect most 2nd and 4th graders spend a lot less excited time on their research floor! Then on to the Newton Public Library to see if they d have anything more and that s when the map that produced all the excitement came out. It, by itself, didn t answer the question, but by piecing together a number of pieces of information newspaper stories, information from the museum, and the maps we were able to come up with a pretty good explanation, much to their excitement. Apparently, a man named Tangeman owned a golf course (the golf links according to the paper), and around 1927 the city of Newton purchased it, because of all the planes that were landing there. They turned it into a real airport. Later, they bought land east of the city and moved the airport there. However, during World War II, the Navy took over that location, so they built a third airport a few miles west of the city but moved back to the current east location after the Navy returned that field to them. Of course, a project like this just opens up all sorts of extra questions: why isn t it called Wirt Field anymore? What s the story of Frank Wirt? What led the Navy to take over Newton s airport? Why did planes start landing on the golf course? Where precisely was the west airport located? How long was it there? (I found an aerial photo from 1956 that looks like it may have a plane in that general area, but it seems later than I d have expected) So now I have the boys interested in going to the courthouse with me to research the property records out there. Jacob is continually astounded that we are discovering things that aren t in Wikipedia, and also excited that he could be the one to add them. To be continued, apparently!

8 December 2016

Stig Sandbeck Mathisen: MIME types and applications

On a Linux system with desktop-file-utils installed, the default application for opening a file with a file manager, from a web browser, or using xdg-open on the command line is not static. The last installed or upgraded application becomes the default. For example: After installing gimp, that application will be used to open any of the many types of files it supports. This lasts until another application which can open those mime types is installed or upgraded. If I later install or upgrade mupdf , that application will be used for PDF, until, etcetera. There are several bug reports filed for this confusing behaviour: Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=525077 Ubuntu: https://bugs.launchpad.net/ubuntu/+source/gimp/+bug/574342 Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=727422

Components

/usr/bin/update-desktop-database is a command in the package desktop-file-utils This command is run in the package postinst script, and triggers on writes to /usr/share/applications where .desktop files are written.

/usr/share/applications This directory contains a list of applications (files ending with .desktop). These desktop files include mime types they are able to work with. The mupdf.desktop example shows it is able to work with (among other) application/pdf
[Desktop Entry]
Encoding=UTF-8
Name=MuPDF
GenericName=PDF file viewer
Comment=PDF file viewer
Exec=mupdf %f
TryExec=mupdf
Icon=mupdf
Terminal=false
Type=Application
MimeType=application/pdf;application/x-pdf;
Categories=Viewer;Graphics;
NoDisplay=true
[Desktop Action View]
Exec=mupdf %f
The gimp.desktop application entry shows it is more capable:
[Desktop Entry]
Version=1.0
Type=Application
Name=GNU Image Manipulation Program
# [...]
MimeType=image/bmp;image/g3fax;image/gif;image/x-fits;image/x-pcx;image/x-portable-anymap;image/x-portable-bitmap;image/x-portable-graymap;image/x-portable-pixmap;image/x-psd;image/x-sgi;image/x-tga;image/x-xbitmap;image/x-xwindowdump;image/x-xcf;image/x-compressed-xcf;image/x-gimp-gbr;image/x-gimp-pat;image/x-gimp-gih;image/tiff;image/jpeg;image/x-psp;application/postscript;image/png;image/x-icon;image/x-xpixmap;image/svg+xml;application/pdf;image/x-wmf;image/x-xcursor;
However, I m quite sure I do not want gimp to be the default viewer for all those file types.

/usr/share/applications/mimeinfo.cache This is a list of MIME types, with a list of applications able to open them. The first entry in the list is the default application. You may also have one of these in ~/.local/share/applications for applications installed in the user s home directory. Examples: With gimp.desktop first, xdg-open test.pdf will use gimp
[MIME Cache]
# [...]
application/pdf=gimp.desktop;mupdf.desktop;evince.desktop;libreoffice-draw.desktop;
After uninstalling and reinstalling mupdf, mupdf.desktop is first in the list, and xdg-open test.pdf will use mupdf
[MIME Cache]
# [...]
application/pdf=mupdf.desktop;gimp.desktop;evince.desktop;libreoffice-draw.desktop;
The order of .desktop files in mimeinfo.cache is the reverse of the order they are added to that directory. The last installed utility is first in that list.

Application Trace This was fun to dig into. I ve just gotten some training which included a a better look at auditd. Auditd is a nice hammer, and this problem was a good nail. I ran the command under autrace , and then looked for the order of reads from each run. When mupdf is installed last, mupdf.desktop is read last, and placed first in the list of applications:
root@laptop:~# autrace /usr/bin/update-desktop-database
Waiting to execute: /usr/bin/update-desktop-database
Cleaning up...
Trace complete. You can locate the records with 'ausearch -i -p 13507'
root@laptop:~# ausearch -p 13507   aureport --file   egrep 'gimp mupdf'
389. 12/09/2016 17:35:37 /usr/share/applications/gimp.desktop 4 yes /usr/bin/update-desktop-database 1000 8002
390. 12/09/2016 17:35:37 /usr/share/applications/gimp.desktop 2 yes /usr/bin/update-desktop-database 1000 8003
391. 12/09/2016 17:35:37 /usr/share/applications/mupdf.desktop 4 yes /usr/bin/update-desktop-database 1000 8010
392. 12/09/2016 17:35:37 /usr/share/applications/mupdf.desktop 2 yes /usr/bin/update-desktop-database 1000 8011
root@laptop:~# grep application/pdf /usr/share/applications/mimeinfo.cache
application/pdf=mupdf.desktop;gimp.desktop;evince.desktop;libreoffice-draw.desktop;
Reinstalling gimp puts that first in the entry for application/pdf
root@laptop:~# apt install --reinstall gimp
[...]
Preparing to unpack .../gimp_2.8.18-1_amd64.deb ...
Unpacking gimp (2.8.18-1) over (2.8.18-1) ...
Processing triggers for mime-support (3.60) ...
Processing triggers for desktop-file-utils (0.23-1) ...
Setting up gimp (2.8.18-1) ...
Processing triggers for gnome-menus (3.13.3-8) ...
[...]
root@laptop:~# autrace /usr/bin/update-desktop-database
Waiting to execute: /usr/bin/update-desktop-database
Cleaning up...
Trace complete. You can locate the records with 'ausearch -i -p 15043'
root@laptop:~# ausearch -p 15043   aureport --file   egrep 'gimp mupdf'
389. 12/09/2016 17:39:53 /usr/share/applications/mupdf.desktop 4 yes /usr/bin/update-desktop-database 1000 9550
390. 12/09/2016 17:39:53 /usr/share/applications/mupdf.desktop 2 yes /usr/bin/update-desktop-database 1000 9551
391. 12/09/2016 17:39:53 /usr/share/applications/gimp.desktop 4 yes /usr/bin/update-desktop-database 1000 9556
392. 12/09/2016 17:39:53 /usr/share/applications/gimp.desktop 2 yes /usr/bin/update-desktop-database 1000 9557
root@laptop:~# grep application/pdf /usr/share/applications/mimeinfo.cache
application/pdf=gimp.desktop;mupdf.desktop;evince.desktop;libreoffice-draw.desktop;

Configuration The solution to this is to add configuration so it will use something else than the default. The xdg-mime command is your tool. The various desktop environments often do this for you. However, if you have a lightweight work environment, you may need to do this yourself for the MIME types you care about.
ssm@laptop ~ :) % xdg-mime query default application/pdf
gimp.desktop
ssm@laptop ~ :) % xdg-mime default mupdf.desktop application/pdf
ssm@laptop ~ :) % xdg-mime query default application/pdf
mupdf.desktop
This updates ~/.local/share/applications/mimeapps.list , and you should now have set your preferred PDF reader.

Dirk Eddelbuettel: RcppAPT 0.0.3

A new version of RcppAPT -- our interface from R to the C++ library behind the awesome apt, apt-get, apt-cache, ... commands and their cache powering Debian, Ubuntu and the like -- is now on CRAN. We changed the package to require C++11 compilation as newer Debian systems with g++-6 and the current libapt-pkg-dev library cannot build under the C++98 standard which CRAN imposes (and let's not get into why ...). Once set to C++11 we have no issues. We also added more examples to the manual pages, and turned on code coverage. A bit more information about the package is available here as well as as the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

7 December 2016

Jonas Meurer: On CVE-2016-4484, a (securiy)? bug in the cryptsetup initramfs integration

On CVE-2016-4484, a (security)? bug in the cryptsetup initramfs integration On November 4, I was made aware of a security vulnerability in the integration of cryptsetup into initramfs. The vulnerability was discovered by security researchers Hector Marco and Ismael Ripoll of CyberSecurity UPV Research Group and got CVE-2016-4484 assigned. In this post I'll try to reflect a bit on

What CVE-2016-4484 is all about Basically, the vulnerability is about two separate but related issues:

1. Initramfs rescue shell considered harmful The main topic that Hector Marco and Ismael Ripoll address in their publication is that Debian exits into a rescue shell in case of failure during initramfs, and that this can be triggered by entering a wrong password ~93 times in a row. Indeed the Debian initramfs implementation as provided by initramfs-tools exits into a rescue shell (usually a busybox shell) after a defined amount of failed attempts to make the root filesystem available. The loop in question is in local_device_setup() at the local initramfs script In general, this behaviour is considered as a feature: if the root device hasn't shown up after 30 rounds, the rescue shell is spawned to provide the local user/admin a way to debug and fix things herself. Hector Marco and Ismael Ripoll argue that in special environments, e.g. on public computers with password protected BIOS/UEFI and bootloader, this opens an attack vector and needs to be regarded as a security vulnerability:
It is common to assume that once the attacker has physical access to the computer, the game is over. The attackers can do whatever they want. And although this was true 30 years ago, today it is not. There are many "levels" of physical access. [...] In order to protect the computer in these scenarios: the BIOS/UEFI has one or two passwords to protect the booting or the configuration menu; the GRUB also has the possibility to use multiple passwords to protect unauthorized operations. And in the case of an encrypted system, the initrd shall block the maximum number of password trials and prevent the access to the computer in that case.
While Hector and Ismael have a valid point in that the rescue shell might open an additional attack vector in special setups, this is not true for the vast majority of Debian systems out there: in most cases a local attacker can alter the boot order, replace or add boot devices, modify boot options in the (GNU GRUB) bootloader menu or modify/replace arbitrary hardware parts. The required scenario to make the initramfs rescue shell an additional attack vector is indeed very special: locked down hardware, password protected BIOS and bootloader but still local keyboard (or serial console) access are required at least. Hector and Ismael argue that the default should be changed for enhanced security:
[...] But then Linux is used in more hostile environments, this helpful (but naive) recovery services shall not be the default option.
For the reasons explained about, I tend to disagree to Hectors and Ismaels opinion here. And after discussing this topic with several people I find my opinion reconfirmed: the Debian Security Team disputes the security impact of the issue and others agree. But leaving the disputable opinion on a sane default aside, I don't think that the cryptsetup package is the right place to change the default, if at all. If you want added security by a locked down initramfs (i.e. no rescue shell spawned), then at least the bootloader (GNU GRUB) needs to be locked down by default as well. To make it clear: if one wants to lock down the boot process, bootloader and initramfs should be locked down together. And the right place to do this would be the configurable behaviour of grub-mkconfig. Here, one can set a password for GRUB and the boot parameter 'panic=1' which disables the spawning of a rescue shell in initramfs. But as mentioned, I don't agree that this would be sane defaults. The vast majority of Debian systems out there don't have any security added by locked down bootloader and initramfs and the benefit of a rescue shell for debugging purposes clearly outrivals the minor security impact in my opinion. For the few setups which require the added security of a locked down bootloader and initramfs, we already have the relevant options documented in the Securing Debian Manual: After discussing the topic with initramfs-tools maintainers today, Guilhem and me (the cryptsetup maintainers) finally decided to not change any defaults and just add a 'sleep 60' after the maximum allowed attempts were reached. 2. tries=n option ignored, local brute-force slightly cheaper Apart from the issue of a rescue shell being spawned, Hector and Ismael also discovered a programming bug in the cryptsetup initramfs integration. This bug in the cryptroot initramfs local-top script allowed endless retries of passphrase input, ignoring the tries=n option of crypttab (and the default of 3). As a result, theoretically unlimited attempts to unlock encrypted disks were possible when processed during initramfs stage. The attack vector here was that local brute-force attacks are a bit cheaper. Instead of having to reboot after max tries were reached, one could go on trying passwords. Even though efficient brute-force attacks are mitigated by the PBKDF2 implementation in cryptsetup, this clearly is a real bug. The reason for the bug was twofold:
  • First, the condition in setup_mapping() responsible for making the function fail when the maximum amount of allowed attempts is reached, was never met:
    setup_mapping()
     
      [...]
      # Try to get a satisfactory password $crypttries times
      count=0                              
    while [ $crypttries -le 0 ] [ $count -lt $crypttries ]; do export CRYPTTAB_TRIED="$count" count=$(( $count + 1 )) [...] done if [ $crypttries -gt 0 ] && [ $count -gt $crypttries ]; then message "cryptsetup: maximum number of tries exceeded for $crypttarget" return 1 fi [...]
    As one can see, the while loop stops when $count -lt $crypttries. Thus the second condition $count -gt $crypttries is never met. This can easily be fixed by decreasing $count by one in case of a successful unlock attempt along with changing the second condition to $count -ge $crypttries:
    setup_mapping()
     
      [...]
      while [ $crypttries -le 0 ]   [ $count -lt $crypttries ]; do
          [...]
          # decrease $count by 1, apparently last try was successful.
          count=$(( $count - 1 ))
          [...]
      done
      if [ $crypttries -gt 0 ] && [ $count -ge $crypttries ]; then
          [...]
      fi
      [...]
     
    
    Christian Lamparter already spotted this bug back in October 2011 and provided a (incomplete) patch, but back then I even managed to merge the patch in an improper way, making it even more useless: The patch by Christian forgot to decrease $count by one in case of a successful unlock attempt, resulting in warnings about maximum tries exceeded even for successful attemps in some circumstances. But instead of adding the decrease myself and keeping the (almost correct) condition $count -eq $crypttries for detection of exceeded maximum tries, I changed back the condition to the wrong original $count -gt $crypttries that again was never met. Apparently I didn't test the fix properly back then. I definitely should do better in future!
  • Second, back in December 2013, I added a cryptroot initramfs local-block script as suggested by Goswin von Brederlow in order to fix bug #678692. The purpose of the cryptroot initramfs local-block script is to invoke the cryptroot initramfs local-top script again and again in a loop. This is required to support complex block device stacks. In fact, the numberless options of stacked block devices are one of the biggest and most inglorious reasons that the cryptsetup initramfs integration scripts became so complex over the years. After all we need to support setups like rootfs on top of LVM with two separate encrypted PVs or rootfs on top of LVM on top of dm-crypt on top of MD raid. The problem with the local-block script is that exiting the setup_mapping() function merely triggers a new invocation of the very same function. The guys who discovered the bug suggested a simple and good solution to this bug: When maximum attempts are detected (by second condition from above), the script sleeps for 60 seconds. This mitigates the brute-force attack options for local attackers - even rebooting after max attempts should be faster.

About disclosure, wording and clickbaiting I'm happy that Hector and Ismael brought up the topic and made their argument about the security impacts of an initramfs rescue shell, even though I have to admit that I was rather astonished about the fact that they got a CVE assigned. Nevertheless I'm very happy that they informed the Security Teams of Debian and Ubuntu prior to publishing their findings, which put me in the loop in turn. Also Hector and Ismael were open and responsive when it came to discussing their proposed fixes. But unfortunately the way they advertised their finding was not very helpful. They announced a speech about this topic at the DeepSec 2016 in Vienna with the headline Abusing LUKS to Hack the System. Honestly, this headline is missleading - if not wrong - in several ways:
  • First, the whole issue is not about LUKS, neither is it about cryptsetup itself. It's about Debians integration of cryptsetup into the initramfs, which is a compeletely different story.
  • Second, the term hack the system suggests that an exploit to break into the system is revealed. This is not true. The device encryption is not endangered at all.
  • Third - as shown above - very special prerequisites need to be met in order to make the mere existance of a LUKS encrypted device the relevant fact to be able to spawn a rescue shell during initramfs.
Unfortunately, the way this issue was published lead to even worse articles in the tech news press. Topics like Major security hole found in Cryptsetup script for LUKS disk encryption or Linux Flaw allows Root Shell During Boot-Up for LUKS Disk-Encrypted Systems suggest that a major security vulnerabilty was revealed and that it compromised the protection that cryptsetup respective LUKS offer. If these articles/news did anything at all, then it was causing damage to the cryptsetup project, which is not affected by the whole issue at all. After the cat was out of the bag, Marco and Ismael aggreed that the way the news picked up the issue was suboptimal, but I cannot fight the feeling that the over-exaggeration was partly intended and that clickbaiting is taking place here. That's a bit sad.

6 December 2016

Mirco Bauer: Secure USB boot with Debian

Foreword The moment you leave your laptop, say in a hotel room, you can no longer trust your system as it could have been modified while you were away. Think you are safe because you have a crypted disk? Well, if the boot partition is on the laptop itself, it can be manipulated and you will not notice because the boot partition can't be encrypted. The BIOS needs to access the MBR and boot loader and that loads the Linux kernel, all unencrypted. There has been some reports lately that the Linux cryptsetup is insecure because you can spawn a root shell by hitting the enter key for 70 seconds. This is not the real threat to your system, really. If someone has physical access to your hardware, he can get a root shell in less than a second by passing init=/bin/bash as parameter to the Linux kernel in the boot loader regardless if cryptsetup is used or not! The attacker can also use other ways like booting a live system from CD/USB etc. The real insecurity here is the unencrypted boot partition and not some script that gets executed from it. So how to prevent this physical access attack vector? Just keep reading this guide. This guide explains how to install Debian securely on your laptop with using an external USB boot disk. The disk inside the laptop should not contain your /boot partition since that is an easy target for manipulation. An attacker could for example change the boot scripts inside the initrd image to capture your passphrase of your crypted volume. With an USB boot partition, you can unplug the USB stick after the operating system has booted. Best practice here is to have the USB stick together with your bunch of keys. That way you will disconnect your USB stick early after the boot as finished so you can put it back into your pocket.

Secure Hardware Assumptions We have to assume here that the hardware you are using to download and verify the install media is safe to use. Same applies with the hardware where you are doing the fresh Debian install. Say the hardware does not contain any malware in the form of code in EFI or other manipulation attempts that influence the behavior of the operating system we are going to install.

Download Debian Install ISO Feel free to use any Debian mirror and install flavor. For this guide I am using the download mirror in Germany and the DVD install flavor.
wget http://ftp.de.debian.org/debian-cd/current/amd64/iso-dvd/debian-8.6.0-amd64-DVD-1.iso

Verify hashsum of ISO file To know if the ISO file was downloaded without modification we have to check the hashsum of the file. The hashsum file can be found in the same directory as the ISO file on the download mirror. With hashsums if a single bit differs in the file, the resulting SHA512 sum will be completely different. Obtain the hashsum file using:
wget http://ftp.de.debian.org/debian-cd/current/amd64/iso-dvd/SHA512SUMS
Calculate a local hashsum from the downloaded ISO file:
sha512sum debian-8.6.0-amd64-DVD-1.iso
Now you need to compare the hashsum with that is in the SHA512SUMS file. Since the SHA512SUMS file contains the hashsums of all files that are in the same directory you need to find the right one first. grep can do this for you:
grep debian-8.6.0-amd64-DVD-1.iso SHA512SUMS
Both commands executed after each other should show following output:
$ sha512sum debian-8.6.0-amd64-DVD-1.iso
c3883edfc95e3b09152d46ce29a032eed1de71531549aee86bb98dab1528088a16f0b4d628aee8ac6cc420364e208d3d5e19d0dea3576f53b904c18e8f604d8c  debian-8.6.0-amd64-DVD-1.iso
$ grep debian-8.6.0-amd64-DVD-1.iso SHA512SUMS
c3883edfc95e3b09152d46ce29a032eed1de71531549aee86bb98dab1528088a16f0b4d628aee8ac6cc420364e208d3d5e19d0dea3576f53b904c18e8f604d8c  debian-8.6.0-amd64-DVD-1.iso
As you can see the hashsum found in the SHA512SUMS file matches with the locally generated hashsum using the sha512sum command. At this point we are not finished yet. These 2 matching hashsums just means whatever was on the download server matches what we have received and stored locally on your disk. The ISO file and SHA512SUM file could still be a modified version! And this is where GPG signatures chime in, covered in the next section.

Download GPG Signature File GPG signature files usually have the .sign file name extension but could also be named .asc. Download the signature file using wget:
wget http://ftp.de.debian.org/debian-cd/current/amd64/iso-dvd/SHA512SUMS.sign

Obtain GPG Key of Signer Letting gpg verify the signature will fail at this point as we don't have the public key of the signer:
$ gpg --verify SHA512SUMS.sign
gpg: assuming signed data in 'SHA512SUMS'
gpg: Signature made Mon 19 Sep 2016 12:23:47 AM HKT
gpg:                using RSA key DA87E80D6294BE9B
gpg: Can't check signature: No public key
Downloading a key is trivial with gpg, but more importantly we need to verify that this key (DA87E80D6294BE9B) is trustworthy, as it could also be a key of the infamous man-in-the-middle. Here you can find the GPG fingerprints of the official signing keys used by Debian. The ending of the "Key fingerprint" line should match the key id we found in the signature file from above.
gpg:                using RSA key DA87E80D6294BE9B
Key fingerprint = DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B
DA87E80D6294BE9B matches Key fingerprint = DF9B 9C49 EAA9 2984 3258 9D76 DA87 E80D 6294 BE9B To download and import this key run: $ gpg --keyserver keyring.debian.org --recv-keys DA87E80D6294BE9B

Verify GPG Signature of Hashsum File Ok, we are almost there. Now we can run the command which checks if the signature of the hashsum file we have, was not modified by anyone and matches what Debian has generated and signed.
gpg: assuming signed data in 'SHA512SUMS'
gpg: Signature made Mon 19 Sep 2016 12:23:47 AM HKT
gpg:                using RSA key DA87E80D6294BE9B
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: Good signature from "Debian CD signing key <debian-cd@lists.debian.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B
The important line in this output is the "Good signature from ..." one. It still shows a warning since we never certified (signed) that Debian key. This can be ignored at this point though.

Write ISO Image to Install Media With a verified pristine ISO file we can finally start the install by writing it to an USB stick or blank DVD. So use your favorite tool to write the ISO to your install media and boot from it. I have used dd and a USB stick attached as /dev/sdb.
dd if=debian-8.6.0-amd64-DVD-1.iso of=/dev/sdb bs=1M oflag=sync

Install Debian on Crypted Volume with USB boot partition I am not explaining each step of the Debian install here. The Debian handbook is a good resource for covering each install step. Follow the steps until the installers wants to partition your disk. There you need to select the "Guided, use entire disk and set up encrypted LVM" option. After that select the built-in disk of your laptop, which usually is sda but double check this before you go ahead, as it will overwrite the data! The 137 GB disk in this case is the built-in disk and the 8 GB is the USB stick. It makes no difference at this point if you select "All files in one partition" or "Separate /home partition". The USB boot partition can be selected a later step. Confirm that you want to overwrite your built-in disk shown as sda. It will take a while as it will write random data to the disk to ensure there is no unencrypted data left on the disk from previous installations for example. Now you need to enter your passphrase that will be used to protect the private key of the crypt volume. Choose something long enough like a sentence and don't forget the passphrase else you can no longer access your data! Don't save the passphrase on any computer, smartphone or password manager. If you want to make a backup of your passphrase then use a ball pen and paper and store the paper backup in a secure location. The installer will show you a summary of the partitioning as shown above but we need to make the change for the USB boot disk. At the moment it wants to put /boot on sda which is the built-in disk, while our USB stick is sdb. Select /boot and hit enter, after that select "Delete this partition". After /boot was deleted we can create /boot on the USB stick shown as sdb. Select sdb and hit enter. It will ask if you want to create an empty partition table. Confirm that question with yes. The partition summary shows sdb with no partitions on it. Select FREE SPACE and select "Create a new partition". Confirm the suggested partition size. Confirm the partition type to be "Primary". It is time to tell the installer to use this new partition on the USB stick (sdb1) as /boot partition. Select "Mount point: /home" and in the next dialog select "/boot - static files of the boot loader" as shown below: Confirm the made changes by selecting "Done setting up the partition". The final partitioning should look now like the following screenshot: If the partition summary looks good, go ahead with the installation by selecting "Finish partitioning and write changes to disk". When the installer asks if it should force EFI, then select no, as EFI is not going to protect you. Finish the installation as usual, select your preferred desktop environment etc.

GRUB Boot Loader Confirm the dialog that wants to install GRUB to the master boot record. Here it is important to install it to the USB stick and not your built-in SATA/SSD disk! So select sdb (the USB stick) in the next dialog.

First Boot from USB Once everything is installed, you can boot from your USB stick. As simple test you can unplug your USB stick and the boot should fail with "no operating system found" or similar error message from the BIOS. If it doesn't boot even though the USB stick is connected, then most likely your BIOS is not configured to boot from USB media. Also a blank screen and nothing happening is usually meaning the BIOS can't find a boot device. You need to change the boot setting in your BIOS. As the steps are very different for each BIOS, I can't provide a detailed step-by-step list here. Usually you can enter the BIOS using F1, F2 or F12 after powering on your computer. In the BIOS there is a menu to configure the boot order. In that list it should show USB disk/storage as the first position. After you have made the changes save and exit the BIOS. Now it will boot from your USB stick first and GRUB will show up and proceeds with the boot process till it will ask for your passphrase to unlock the crypt volume.

Unmount /boot partition after Boot If you boot your laptop from the USB stick, we want to remove the stick after it has finished booting. This will prevent an attacker to make modifications to your USB stick. To avoid data loss, we should not simply unplug the USB stick but unmount /boot first and then unplug the stick. Good news is that we can automate this unmounting and you just need to unplug the stick after the laptop has finished booting to your login screen. Just add this line to your /etc/rc.local file:
umount /boot
After boot you can once verify that it automatically unmounts /boot for you by running:
mount   grep /boot
If that command produces no output, then /boot is not mounted and you can safely unplug the USB stick.

Final Words From time to time you need to upgrade your Linux kernel of course which is on the /boot partition. This can still be done the regular way using apt-get upgrade, except that you need to mount /boot before that and unmount it again after the kernel upgrade. Enjoy your secured laptop. Now you can leave it in a hotel room without the possibility of someone trying you obtain your passphrase by putting a key logger in your boot partition. All the attacker will see is a fully encrypted harddisk. If he tries to mess with your crypted disk, you will notice as the decryption will fail. Disclaimer: there are still other attack vectors possible, but they are much harder to do. Your hardware or BIOS can still be modified. But not by holding down the enter key for 70 seconds or by booting a live system.

3 December 2016

Vincent Bernat: Build-time dependency patching for Android

This post shows how to patch an external dependency for an Android project at build-time with Gradle. This leverages the Transform API and Javassist, a Java bytecode manipulation tool.
buildscript  
    dependencies  
        classpath 'com.android.tools.build:gradle:2.2.+'
        classpath 'com.android.tools.build:transform-api:1.5.+'
        classpath 'org.javassist:javassist:3.21.+'
        classpath 'commons-io:commons-io:2.4'
     
 
Disclaimer: I am not a seasoned Android programmer, so take this with a grain of salt.

Context This section adds some context to the example. Feel free to skip it. Dashkiosk is an application to manage dashboards on many displays. It provides an Android application you can install on one of those cheap Android sticks. Under the table, the application is an embedded webview backed by the Crosswalk Project web runtime which brings an up-to-date web engine, even for older versions of Android1. Recently, a security vulnerability has been spotted in how invalid certificates were handled. When a certificate cannot be verified, the webview defers the decision to the host application by calling the onReceivedSslError() method:
Notify the host application that an SSL error occurred while loading a resource. The host application must call either callback.onReceiveValue(true) or callback.onReceiveValue(false). Note that the decision may be retained for use in response to future SSL errors. The default behavior is to pop up a dialog.
The default behavior is specific to Crosswalk webview: the Android builtin one just cancels the load. Unfortunately, the fix applied by Crosswalk is different and, as a side effect, the onReceivedSslError() method is not invoked anymore2. Dashkiosk comes with an option to ignore TLS errors3. The mentioned security fix breaks this feature. The following example will demonstrate how to patch Crosswalk to recover the previous behavior4.

Simple method replacement Let s replace the shouldDenyRequest() method from the org.xwalk.core.internal.SslUtil class with this version:
// In SslUtil class
public static boolean shouldDenyRequest(int error)  
    return false;
 

Transform registration Gradle Transform API enables the manipulation of compiled class files before they are converted to DEX files. To declare a transform and register it, include the following code in your build.gradle:
import com.android.build.api.transform.Context
import com.android.build.api.transform.QualifiedContent
import com.android.build.api.transform.Transform
import com.android.build.api.transform.TransformException
import com.android.build.api.transform.TransformInput
import com.android.build.api.transform.TransformOutputProvider
import org.gradle.api.logging.Logger
class PatchXWalkTransform extends Transform  
    Logger logger = null;
    public PatchXWalkTransform(Logger logger)  
        this.logger = logger
     
    @Override
    String getName()  
        return "PatchXWalk"
     
    @Override
    Set<QualifiedContent.ContentType> getInputTypes()  
        return Collections.singleton(QualifiedContent.DefaultContentType.CLASSES)
     
    @Override
    Set<QualifiedContent.Scope> getScopes()  
        return Collections.singleton(QualifiedContent.Scope.EXTERNAL_LIBRARIES)
     
    @Override
    boolean isIncremental()  
        return true
     
    @Override
    void transform(Context context,
                   Collection<TransformInput> inputs,
                   Collection<TransformInput> referencedInputs,
                   TransformOutputProvider outputProvider,
                   boolean isIncremental) throws IOException, TransformException, InterruptedException  
        // We should do something here
     
 
// Register the transform
android.registerTransform(new PatchXWalkTransform(logger))
The getInputTypes() method should return the set of types of data consumed by the transform. In our case, we want to transform classes. Another possibility is to transform resources. The getScopes() method should return a set of scopes for the transform. In our case, we are only interested by the external libraries. It s also possible to transform our own classes. The isIncremental() method returns true because we support incremental builds. The transform() method is expected to take all the provided inputs and copy them (with or without modifications) to the location supplied by the output provider. We didn t implement this method yet. This causes the removal of all external dependencies from the application.

Noop transform To keep all external dependencies unmodified, we must copy them:
@Override
void transform(Context context,
               Collection<TransformInput> inputs,
               Collection<TransformInput> referencedInputs,
               TransformOutputProvider outputProvider,
               boolean isIncremental) throws IOException, TransformException, InterruptedException  
    inputs.each  
        it.jarInputs.each  
            def jarName = it.name
            def src = it.getFile()
            def dest = outputProvider.getContentLocation(jarName, 
                                                         it.contentTypes, it.scopes,
                                                         Format.JAR);
            def status = it.getStatus()
            if (status == Status.REMOVED)   //  
                logger.info("Remove $ src ")
                FileUtils.delete(dest)
              else if (!isIncremental   status != Status.NOTCHANGED)   //  
                logger.info("Copy $ src ")
                FileUtils.copyFile(src, dest)
             
         
     
 
We also need two additional imports:
import com.android.build.api.transform.Status
import org.apache.commons.io.FileUtils
Since we are handling external dependencies, we only have to manage JAR files. Therefore, we only iterate on jarInputs and not on directoryInputs. There are two cases when handling incremental build: either the file has been removed ( ) or it has been modified ( ). In all other cases, we can safely assume the file is already correctly copied.

JAR patching When the external dependency is the Crosswalk JAR file, we also need to modify it. Here is the first part of the code (replacing ):
if ("$ src " ==~ ".*/org.xwalk/xwalk_core.*/classes.jar")  
    def pool = new ClassPool()
    pool.insertClassPath("$ src ")
    def ctc = pool.get('org.xwalk.core.internal.SslUtil') //  
    def ctm = ctc.getDeclaredMethod('shouldDenyRequest')
    ctc.removeMethod(ctm) //  
    ctc.addMethod(CtNewMethod.make("""
public static boolean shouldDenyRequest(int error)  
    return false;
 
""", ctc)) //  
    def sslUtilBytecode = ctc.toBytecode() //  
    // Write back the JAR file
    //  
  else  
    logger.info("Copy $ src ")
    FileUtils.copyFile(src, dest)
 
We also need the following additional imports to use Javassist:
import javassist.ClassPath
import javassist.ClassPool
import javassist.CtNewMethod
Once we have located the JAR file we want to modify, we add it to our classpath and retrieve the class we are interested in ( ). We locate the appropriate method and delete it ( ). Then, we add our custom method using the same name ( ). The whole operation is done in memory. We retrieve the bytecode of the modified class in . The remaining step is to rebuild the JAR file:
def input = new JarFile(src)
def output = new JarOutputStream(new FileOutputStream(dest))
//  
input.entries().each  
    if (!it.getName().equals("org/xwalk/core/internal/SslUtil.class"))  
        def s = input.getInputStream(it)
        output.putNextEntry(new JarEntry(it.getName()))
        IOUtils.copy(s, output)
        s.close()
     
 
//  
output.putNextEntry(new JarEntry("org/xwalk/core/internal/SslUtil.class"))
output.write(sslUtilBytecode)
output.close()
We need the following additional imports:
import java.util.jar.JarEntry
import java.util.jar.JarFile
import java.util.jar.JarOutputStream
import org.apache.commons.io.IOUtils
There are two steps. In , all classes are copied to the new JAR, except the SslUtil class. In , the modified bytecode for SslUtil is added to the JAR. That s all! You can view the complete example on GitHub.

More complex method replacement In the above example, the new method doesn t use any external dependency. Let s suppose we also want to replace the sslErrorFromNetErrorCode() method from the same class with the following one:
import org.chromium.net.NetError;
import android.net.http.SslCertificate;
import android.net.http.SslError;
// In SslUtil class
public static SslError sslErrorFromNetErrorCode(int error,
                                                SslCertificate cert,
                                                String url)  
    switch(error)  
        case NetError.ERR_CERT_COMMON_NAME_INVALID:
            return new SslError(SslError.SSL_IDMISMATCH, cert, url);
        case NetError.ERR_CERT_DATE_INVALID:
            return new SslError(SslError.SSL_DATE_INVALID, cert, url);
        case NetError.ERR_CERT_AUTHORITY_INVALID:
            return new SslError(SslError.SSL_UNTRUSTED, cert, url);
        default:
            break;
     
    return new SslError(SslError.SSL_INVALID, cert, url);
 
The major difference with the previous example is that we need to import some additional classes.

Android SDK import The classes from the Android SDK are not part of the external dependencies. They need to be imported separately. The full path of the JAR file is:
androidJar = "$ android.getSdkDirectory().getAbsolutePath() /platforms/" +
             "$ android.getCompileSdkVersion() /android.jar"
We need to load it before adding the new method into SslUtil class:
def pool = new ClassPool()
pool.insertClassPath(androidJar)
pool.insertClassPath("$ src ")
def ctc = pool.get('org.xwalk.core.internal.SslUtil')
def ctm = ctc.getDeclaredMethod('sslErrorFromNetErrorCode')
ctc.removeMethod(ctm)
pool.importPackage('android.net.http.SslCertificate');
pool.importPackage('android.net.http.SslError');
//  

External dependency import We must also import org.chromium.net.NetError and therefore, we need to put the appropriate JAR in our classpath. The easiest way is to iterate through all the external dependencies and add them to the classpath.
def pool = new ClassPool()
pool.insertClassPath(androidJar)
inputs.each  
    it.jarInputs.each  
        def jarName = it.name
        def src = it.getFile()
        def status = it.getStatus()
        if (status != Status.REMOVED)  
            pool.insertClassPath("$ src ")
         
     
 
def ctc = pool.get('org.xwalk.core.internal.SslUtil')
def ctm = ctc.getDeclaredMethod('sslErrorFromNetErrorCode')
ctc.removeMethod(ctm)
pool.importPackage('android.net.http.SslCertificate');
pool.importPackage('android.net.http.SslError');
pool.importPackage('org.chromium.net.NetError');
ctc.addMethod(CtNewMethod.make(" "))
// Then, rebuild the JAR...
Happy hacking!

  1. Before Android 4.4, the webview was severely outdated. Starting from Android 5, the webview is shipped as a separate component with updates. Embedding Crosswalk is still convenient as you know exactly which version you can rely on.
  2. I hope to have this fixed in later versions.
  3. This may seem harmful and you are right. However, if you have an internal CA, it is currently not possible to provide its own trust store to a webview. Moreover, the system trust store is not used either. You also may want to use TLS for authentication only with client certificates, a feature supported by Dashkiosk.
  4. Crosswalk being an opensource project, an alternative would have been to patch Crosswalk source code and recompile it. However, Crosswalk embeds Chromium and recompiling the whole stuff consumes a lot of resources.

30 November 2016

Chris Lamb: Free software activities in November 2016

Here is my monthly update covering what I have been doing in the free software world (previous month):
Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users. The motivation behind the Reproducible Builds effort is to permit verification that no flaws have been introduced either maliciously or accidentally during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

This month:

My work in the Reproducible Builds project was also covered in our weekly reports. (#80, #81, #82 #83.

Toolchain issues I submitted the following patches to fix reproducibility-related toolchain issues with Debian:

strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.


jenkins.debian.net

jenkins.debian.net runs our comprehensive testing framework.

  • buildinfo.debian.net has moved to SSL. (ac3b9e7)
  • Submit signing keys to keyservers after generation. (bdee6ff)
  • Various cosmetic changes, including
    • Prefer if X not in Y over if not X in Y. (bc23884)
    • No need for a dictionary; let's just use a set. (bf3fb6c)
    • Avoid DRY violation by using a for loop. (4125ec5)

I also submitted 9 patches to fix specific reproducibility issues in apktool, cairo-5c, lava-dispatcher, lava-server, node-rimraf, perlbrew, qsynth, tunnelx & zp.

Debian

Debian LTS This month I have been paid to work 11 hours on Debian Long Term Support (LTS). In that time I did the following:
  • "Frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 697-1 for bsdiff fixing an arbitrary write vulnerability.
  • Issued DLA 705-1 for python-imaging correcting a number of memory overflow issues.
  • Issued DLA 713-1 for sniffit where a buffer overflow allowed a specially-crafted configuration file to provide a root shell.
  • Issued DLA 723-1 for libsoap-lite-perl preventing a Billion Laughs XML expansion attack.
  • Issued DLA 724-1 for mcabber fixing a roster push attack.

Uploads
  • redis:
    • 3.2.5-2 Tighten permissions of /var/ lib,log /redis. (#842987)
    • 3.2.5-3 & 3.2.5-4 Improve autopkgtest tests and install upstream's MANIFESTO and README.md documentation.
  • gunicorn (19.6.0-9) Adding autopkgtest tests.
  • libfiu:
    • 0.94-1 Add autopkgtest tests.
    • 0.95-1, 0.95-2 & 0.95-3 New upstream release and improve autopkgtest coverage.
  • python-django (1.10.3-1) New upstream release.
  • aptfs (0.8-3, 0.8-4 & 0.8-5) Adding and subsequently improving the autopkgtext tests.


I performed the following QA uploads:


Finally, I also made the following non-maintainer uploads:
  • libident (0.22-3.1) Move from obsolete Source-Version substvar to binary:Version. (#833195)
  • libpcl1 (1.6-1.1) Move from obsolete Source-Version substvar to binary:Version. (#833196)
  • pygopherd (2.0.18.4+nmu1) Move from obsolete Source-Version substvar to $ source:Version . (#833202)


RC bugs


I also filed 59 FTBFS bugs against arc-gui-clients, asyncpg, blhc, civicrm, d-feet, dpdk, fbpanel, freeciv, freeplane, gant, golang-github-googleapis-gax-go, golang-github-googleapis-proto-client-go, haskell-cabal-install, haskell-fail, haskell-monadcatchio-transformers, hg-git, htsjdk, hyperscan, jasperreports, json-simple, keystone, koji, libapache-mod-musicindex, libcoap, libdr-tarantool-perl, libmath-bigint-gmp-perl, libpng1.6, link-grammar, lua-sql, mediatomb, mitmproxy, ncrack, net-tools, node-dateformat, node-fuzzaldrin-plus, node-nopt, open-infrastructure-system-images, open-infrastructure-system-images, photofloat, ppp, ptlib, python-mpop, python-mysqldb, python-passlib, python-protobix, python-ttystatus, redland, ros-message-generation, ruby-ethon, ruby-nokogiri, salt-formula-ceilometer, spykeviewer, sssd, suil, torus-trooper, trash-cli, twisted-web2, uftp & wide-dhcpv6.

FTP Team

As a Debian FTP assistant I ACCEPTed 70 packages: bbqsql, coz-profiler, cross-toolchain-base, cross-toolchain-base-ports, dgit-test-dummy, django-anymail, django-hstore, django-html-sanitizer, django-impersonate, django-wkhtmltopdf, gcc-6-cross, gcc-defaults, gnome-shell-extension-dashtodock, golang-defaults, golang-github-btcsuite-fastsha256, golang-github-dnephin-cobra, golang-github-docker-go-events, golang-github-gogits-cron, golang-github-opencontainers-image-spec, haskell-debian, kpmcore, libdancer-logger-syslog-perl, libmoox-buildargs-perl, libmoox-role-cloneset-perl, libreoffice, linux-firmware-raspi3, linux-latest, node-babel-runtime, node-big.js, node-buffer-shims, node-charm, node-cliui, node-core-js, node-cpr, node-difflet, node-doctrine, node-duplexer2, node-emojis-list, node-eslint-plugin-flowtype, node-everything.js, node-execa, node-grunt-contrib-coffee, node-grunt-contrib-concat, node-jquery-textcomplete, node-js-tokens, node-json5, node-jsonfile, node-marked-man, node-os-locale, node-sparkles, node-tap-parser, node-time-stamp, node-wrap-ansi, ooniprobe, policycoreutils, pybind11, pygresql, pysynphot, python-axolotl, python-drizzle, python-geoip2, python-mockupdb, python-pyforge, python-sentinels, python-waiting, pythonmagick, r-cran-isocodes, ruby-unicode-display-width, suricata & voctomix-outcasts. I additionally filed 4 RC bugs against packages that had incomplete debian/copyright files against node-cliui, node-core-js, node-cpr & node-grunt-contrib-concat.

29 November 2016

Shirish Agarwal: The Iziko South African Museum

This would be a bit long on my stay in Cape Town, South Africa after Debconf16. Before I start, let me share the gallery works, you can see some photos that I have been able to upload to my gallery . It seems we are using gallery 2 while upstream had made gallery 3 and then it sort of died. I actually asked in softwarerecs stackexchange site if somebody knows of a drop-in replacement for gallery and was told/shared about Pwigo . I am sure the admin knows about it. There would be costs to probably migrate from gallery to Pwigo with the only benefit that it would be something which would perhaps be more maintainable. The issues I face with the current gallery system are few things a. There is no way to know how much your progress your upload has taken.
b. After it has submit, it gives a fake error message saying some error has occurred. This has happened on every occasion/attempt. Now I don t know whether it is because I have slow upload speeds or something else altogether. I had shared the error page last time in the blog post hence not sharing again. Although, all the pictures which would be shared in this blog post would be from the same gallery Another thing I would like to share is a small beginner article I wrote about why I like Debian. Another interesting/tit-bit of news I came to know few days back that both Singapore and Qatar have given 96 hours visa free stopovers for Indians for select destinations. Now to start with the story/experience due to some unknown miracle/angel looking upon me I got the chance to go to Debconf16, South Africa. I m sure there was lot of backend discussions but in the end I was given the opportunity to be part of Debcamp and Debconf. While I hope to recount my Debcamp and Debconf experience in another or two blog posts, this would be exclusively the Post-Debconf Experiences I had. As such opportunities to visit another country are rare, I wanted to make the most of it. Before starting from Pune, I had talked with Amey about Visas, about Debconf as he had just been to Debconf15 the year before and various things related to travel. He was instrumental in me having a bit more knowledge about how to approach things. I was also lucky to have both Graham and Bernelle who also suggested, advised and made it possible to have a pleasant stay both during Debcamp and Debconf. The only quibble is I didn t know heaters were being made available to us without any cost. Moving on, a day or two before Debconf was about to conclude, I asked Bernelle s help even though she was battling a burn-out I believe as I was totally clueless about Cape Town. She accepted my request and asked me to look at hostels near Longmarket Street. I had two conditions a. It should not be very far from the airport
b. It should be near to all or most cultural experiences the city has to offer. We looked at hostelworld and from the options listed, it looked like Homebasecapetown looked to be a perfect fit. It was one of the cheaper options and they also had breakfast included in the pricing. I booked through hostelworld for a mixed dorm for 2 days as I was unsure how it would be (the first night effect I have shared about previously) . When I reached there, I found it to be as good as the pictures shared were, the dorm was clean (most important), people were friendly (also important) as well as toilets and shower were also clean while the water was hot, so all in all it was a win-win situation for me. Posters I saw at homebasecapetown While I m not much of an adrenaline-junkie it was nice to know the activities that could be done/taken. Brochures and Condoms just left of main hall. This was again interesting. While apologies for the poor shaky quality of the picture, I believe it is easy to figure out. There were Brochures of the city attractions as well as condoms that people could discreetly use if need be. I had seen such condoms in few toilets during and around Debconf and it felt good that the public were aware and prioritizing safety for their guests and students instead of having fake holier than thou attitudes that many places have. For instance, you wouldn t find something like this in toilets of most colleges in India or anywhere else for that matter. There are few vending machines in what are termed as red light areas or where prostitution is known/infamous to happen and even then most times it is empty. I have 2-3 social workers as friends and they are a source of news on such things. While I went to few places and each had an attraction to it, the one which had my literally eyes out of socket was the Iziko South African Museum . I have been lucky to been quite a few museums in India, the best rated science museum in India in my limited experience has been the Visvesvaraya Industrial & Technological Museum, Bengaluru India . A beer from me if a European can get it right. Don t worry if you mispronounce it, I mispronounce it couple of times till I get it right . Looking up the word Iziko the meaning of the word seems to be the hearth and if you look at the range of collections in the museum, you would think it fits. I was lucky to find couple of friends, one of whom was living at homebase and we decided to go to the museum together. Making friends on the road So Eduardo, my friend on the left and his friend, we went to the museum. While viewing the museum, there were no adjectives to describe it other than Wow and Endless . See fossils of fish-whale-shark ? OR Giant fish-whale-dolphin-shark some million years ago. and Reminder of JAWS ;) While I have more than a few pictures, the point is easily made. It seems almost inconceivable that creatures of such masses actually were on earth. While I played with the model of the jaws of a whale/shark in reality if something like that happened, I would have been fighting for my life. The only thing I missed or could have been better if they had some interactive installations to showcase the now universally accepted Charles Darwin s On the Origin of Species I had never seen anything like this. Sadly, there was nobody around to help us figure out things as I had read that most species of fish don t leave a skeleton behind so how were these models made? It just boggles the mind. Apart from the Science Museum I was also introduced to the bloody history that South Africa had. I saw The 1913 native land act which was not honored . I had been under the impression that India had got a raw deal when it was under British rule but looking at South African history I don t know. While we got our freedom in 1947 they got rid of apartheid about 20 years+ . I talked to lot of young African males and there was lot of naked hostility for the Europeans even today. It was a bit depressing but could relate to their point of view as similar sentiments were echoed by our forefathers. I read in the newspapers and it seemed to be a pretty mixed picture. I can t comment as only South Africans can figure out the way forward. For me, it was enough to know and see that we both had similar political histories as nations. It seemed the racial divide and anger was much more highly pronounced towards Europeans and divisive then the caste divisions here between Indians. I also shared with them my limited knowledge and understanding of the Indian history (as history is re-written all the time) and it was clear to them that we had common/similar pasts. As a result, what was surprising (actually not) is that many South Africans have no knowledge of Indian history. as well otherwise the political differences that South Africa and India has in the current scenario wouldn t have been. In the end, the trip proved to be fun, stimulating, educative, thought-provoking as questions about self-identity , national identity, our place in the Universe kinda questions which should be asked all the time. Thank you Bremmer and the team for letting me experience Cape Town, South Africa, I would have been poorer if I hadn t had the experience.
Filed under: Miscellenous Tagged: #Debconf16, #Dinosaur Fishes, #gallery, #Identity, #Iziko South African Museum, #Nation-state Identity, #pwigo

21 November 2016

Steve Kemp: Detecting fraudulent signups?

I run a couple of different sites that allow users to sign-up and use various services. In each of these sites I have some minimal rules in place to detect bad signups, but these are a little ad hoc, because the nature of "badness" varies on a per-site basis. I've worked in a couple of places where there are in-house tests of bad signups, and these usually boil down to some naive, and overly-broad, rules: Some places penalise users based upon location too: At the moment I've got a simple HTTP-server which receives a JSON post of a new users' details, and returns "200 OK" or "403 Forbidden" based on some very very simple critereon. This is modeled on the spam detection service for blog-comments server I use - something that is itself becoming less useful over time. (Perhaps time to kill that? A decision for another day.) Unfortunately this whole approach is very reactive, as it takes human eyeballs to detect new classes of problems. Code can't guess in advance that it should block usernames which could collide with official ones, for example allowing a username of "admin", "help", or "support". I'm certain that these systems have been written a thousand times, as I've seen at least five such systems, and they're all very similar. The biggest flaw in all these systems is that they try to classify users in advance of them doing anything. We're trying to say "Block users who will use stolen credit cards", or "Block users who'll submit spam", by correlating that behaviour with other things. In an ideal world you'd judge users only by the actions they take, not how they signed up. And yet .. it is better than nothing. For the moment I'm continuing to try to make the best of things, at least by centralising the rules for myself I cut down on duplicate code. I'll pretend I'm being cool, modern, and sexy, and call this a micro-service! (Ignore the lack of containers for the moment!)

16 November 2016

Shirish Agarwal: The long tail in a common s man journey to debconf16 2

This is an extension of part 1 which I shared few days ago. This would be a longish one so please bear. First of all somebody emailed me this link so in the future a layover at Doha Airport will be a bit expensive from before, approx INR 700/- added to the ticket costs Moving on, Let me share an experience I shared one of the last few days I had while I was in Cape Town
Singer singing some great oldies from 60's , 70's till 90's.

Singer singing some great oldies from 60 s , 70 s till 90 s.

I had booked a place near Long Street, Cape Town using Bernelle s help. What I had not known at that time that near Long Street there are free walking tours every couple of hours. I took part in all the tours and those were nice experiences. Where they start the walk, there was the gentleman pictured above. I was amazed by this gentleman s rich voice. He strummed lot of classics from the 60 s, 70 s till the 90 s . I had two coffees and thought I was at a premium rock concert. It was a bitter-sweet experience for me because I could see that he has such prodigious talent and still he had to struggle to survive to make ends meet. I did my 2 bit but wish I could have done something more. Side note Before I forget there is one trick of feh which I use to view images without it getting very high-resolution (especially on my low-end systems) [shirish@debian] - [/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C006%5D/Card/DCIM/Camera] - [4621]
[$] feh -g 1350x1000 .
This actually makes it far far easier to traverse through the 1000 odd photos of the trip that I have in my personal archive without doing any sort of conversion methodology. Btw, it took me time but finally was able to create an album at gallery.debconf.org . Haven t been able to upload photos as came across an error which I have shared at https://lists.debconf.org/lurker/message/20161113.215659.fce58823.en.html Moving on, here s the funny story/experience I wanted to share could have been arrested ;) What happened was this. This is from the Doha Airport. I had seen big buggies (ones similar to golf carts) which was ferrying people from end of the concourse to the other. I had been walking the whole day and even with the horizontal escalators and everything, it takes a toll. I was half-tired, half-sleepy and saw a buggy stationed. From behind it looked like the buggies I had seen. As there was no place to park my behind there, I entered into the buggy and sat there. Around 15-20 minutes later a Doha cop in another buggy came to me and asked me if something had happened ? I had no clue what he was talking about. He told/shared/asked me in friendly tone whether I had committed a crime or wanted to report a crime. When I replied in negative to both, he asked then why I was sitting there. I replied it was for stretching my legs and it was the buggy which was being used to transport people from A. to B. He gently told me I had entered into the wrong one and it was actually a cop buggy. I couldn t believe it. He did go his own way as he saw I was dead-tired. After 10-15 minutes, half-believingly I came out of the buggy and to my shock the gentleman was right. There was nothing to do but solder on to find a spot in this big airport. I shared this with few friends and family and managed to elicit few laughs hence sharing. The somewhat sad one was I had met a couple with a baby. Now as shared before, Most Airports including the Doha Airport is Air-Conditioned/Climate-Controlled and is probably in mid-20 s so it was more than cold for me. The couples with the baby were from Asian sub-continent. From their clothes and the way they were, they were not very well off. I do remember them sharing that they had a death in the family and hence were going. I didn t know at that point in time that there was something called bereavement fares and if they were able to take opportunity of those tickets. But this is besides the point . The issue was that their baby had been running a high-fever and the A/C was making matters worse. I had seen a pharmacy but no clinic in the airport. It was much later I came across http://dohahamadairport.com/airport-guide/facilities-services/medical-emergencies but as can be seen on the web-page it doesn t tell whether the services are chargeable or not. I assume it would be paid, although in some of the developed/industrialized countries it is rumoured not to be for simple ailments such as the baby was going through. Have no idea if that s true or not. I also don t know how it equates with travel insurance as well as most travel insurance is also supposed to help you in situations like these. I was concerned as it was a baby and babies as all know are very very fragile. If anybody has an idea or had similar experience would like to know specifically related to International Airport environment as it has transit issues unlike in domestic airports where I don t think it would be a bit more easy. Now coming to my own inadequacies/lack of foresight which I had mentioned I will share, I had asked/queried and got to lead a Debian-installation workshop on the Open Debian Day. I had done a few earlier and had installed it a few times on my system and for my friends, relatives and some clients. The only bad experiences I had were to do with UEFI but even those in the jessie releases had got resolved quite a bit, so was pretty confident. The day before the Installfest was to happen, Mensah Nyarko Yaa Dufie (one full name) of Ghana approached me to install Debian on her system. I had some older version of the Debian DVD either 8.1 or 8.3 and had known that 8.5 had been released just a few days back. Had seen pretty fast internet (as far as downloading Debian DVD) is concerned hence asked her to wait a bit while I downloaded the newest image. I sha256summed it to make sure that the image was bit-to-bit perfect. Now I hadn t bought a pen drive/disk from India as I was under the impression that in such conferences, pen drives should not be an issue. I had asked Bernelle privately before via e-mail as well and she had assured me that some pen-drives would be available. She gave me a handful of HP pen drives. The pen drives as we came know during our usage were somewhat flaky. It would pop out/lose connection even with the slightest nudge to the lappy. Somehow I was able to transfer the image to the usb disk. As people say hindsight is 50:50 maybe it was not such a smart move on my part to download the big DVD image and maybe I should have got the netinstall iso . Be careful, the link I have just shared is of the old version, if you have good web link and want to try the newest stable netinstall head to cdimage.debian.org . Apart from that goof-up I dunno (still) of anyway to know if a copy from an .iso image to usb was successful or not and did it do correctly I did the following command sudo dd if=/path_to/debian-dvd.iso of=/dev/usb-mount-point which is usually /dev/sdb on all of my systems . Her system was a brand new HP (don t remember the model details) which she had bought just a few weeks/months before debconf. We tried a few times but it failed at installing the boot-loader stage. I asked Ritesh Raj Saraff (a friend and DD) and while he had some ideas, none of them worked. Ritesh later pointed out Steve McIntyre and shared he is part of the Debian-Installer team. At that point in time, I had no clue who Steve McIntyre was otherwise I probably would not have approached him. He quickly acquiesced to my request and shared that he would be there for the workshop. With load of my mind little bit, I apologized to mensah and asked her to be at the workshop the following day. I had no clue what was wrong at this point in time, whether it was the iso image in the usb disk or a UEFI issue. This also wasn t good for my confidence but as somebody from the Debian-Installer team was there, I was somewhat relaxed. Next day, some more people came for the Installfest. While I had made 2-3 copies, clearly it was not enough as more people came. I was in a frenzy and asked Deven Bansod, Keerthana Krishnan, Prabaharan Jaminy (the whole GSOC and Outreachy attendees) to volunteer to help out in making more iso images on usb disks. I introduced mensah to Steve McIntyre and we tried 2-3 times to get debian installed on the system but it didn t move from the same place. Ritesh shared that dd had a memory leak and hence cat was a better way to do it. So we did

$ cat debian.iso > /dev/sdb
and soon other machines had debian sporting on their desktops. But mensa s lappy wouldn t get move from the boot-loader stage. Suddenly Steve had the bright idea (light bulb moment) that maybe the .iso is corrupted/usb disk is bad or something is incomplete. We started on another usb disk. Now this is where I have a query While I don t want to compare, in Ubuntu there was an image self-checking mechanism where probably behind the scenes (backend) the checksums published in a file are compared with checksums generated by apps. which are on the .iso image. While it does extend your time, the end result is you know if there is some issue on the decompressed image on the usb disk. AFAIK we don t have anything similar. The only two things I know is the wiki page and of course the various checksums of the image as shared at http://cdimage.debian.org/debian-cd/8.6.0/amd64/iso-cd/ or http://cdimage.debian.org/debian-cd/8.6.0/amd64/iso-dvd/ If anybody knows of any movement or a bug in the BTS which I can follow for the above issue please let me know. This time Steve was able to install it without any issues. I asked him whether he had to make some specific FAT/Ex-FAT/NTFS partitions as some new UEFI-based lappies need one or more but he replied in the negative. While mensa did get her debian install, the GUI didn t come while command-prompt was available. Then Steve added backports to the sources.list, got the new kernel, new Intel/Nvidia drivers (think it was one of those hybrid models IIRC) and she was able to boot into GNOME-Debian. I didn t saw any bug-reports about checksumming state of the applications before installation but did couple of reports about badblocks support and memory checking and from action on both bug-reports it is also need of the hour (although the earlier one has been marked as won t fix :(). In this whole thing, I liked/appreciated the way Steve handled things, I intuitively understood/knew that he wasn t just part of the Debian-installer team but someone better. I can t explain it but it was there. A little investigation in the evening and it turned out that he had been Debian Project Leader for two consecutive years (2008 and 2009) . In hindsight it probably was a good thing I didn t know that before otherwise I probably wouldn t have interacted with him and it would have been my loss. To have been the DPL and still being so humble while technically being so proficient, I was amazed and didn t know what to make of it. Here i.e. in India, if somebody wins even the mohalla elections (neighbourhood elections) the person carries a big chip on her/is shoulder not just till he is on the seat but even beyond, and here was an example of a previous DPL asking time from one of the developers in a video if it s possible in the next couple of days. Lastly,last week have able to report 2 bugs upstream. The first one is of youtube-dl . It s somewhat complicated hence will not go there atm. The second and more surprising one was from nano our esteemed text-editor- Hopefully the bug will be fixed once a new version comes.
Filed under: Miscellenous Tagged: #buggy, #cop, #Debconf16, #doha airport, #Installfest, #nano, #singer, #youtube-dl, travel

15 November 2016

Antoine Beaupr : The Turris Omnia router: help for the IoT mess?

The Turris Omnia router is not the first FLOSS router out there, but it could well be one of the first open hardware routers to be available. As the crowdfunding campaign is coming to a close, it is worth reflecting on the place of the project in the ecosystem. Beyond that, I got my hardware recently, so I was able to give it a try.

A short introduction to the Omnia project The Turris Omnia Router The Omnia router is a followup project on CZ.NIC's original research project, the Turris. The goal of the project was to identify hostile traffic on end-user networks and develop global responses to those attacks across every monitored device. The Omnia is an extension of the original project: more features were added and data collection is now opt-in. Whereas the original Turris was simply a home router, the new Omnia router includes:
  • 1.6GHz ARM CPU
  • 1-2GB RAM
  • 8GB flash storage
  • 6 Gbit Ethernet ports
  • SFP fiber port
  • 2 Mini-PCI express ports
  • mSATA port
  • 3 MIMO 802.11ac and 2 MIMO 802.11bgn radios and antennas
  • SIM card support for backup connectivity
Some models sold had a larger case to accommodate extra hard drives, turning the Omnia router into a NAS device that could actually serve as a multi-purpose home server. Indeed, it is one of the objectives of the project to make "more than just a router". The NAS model is not currently on sale anymore, but there are plans to bring it back along with LTE modem options and new accessories "to expand Omnia towards home automation". Omnia runs a fork of the OpenWRT distribution called TurrisOS that has been customized to support automated live updates, a simpler web interface, and other extra features. The fork also has patches to the Linux kernel, which is based on Linux 4.4.13 (according to uname -a). It is unclear why those patches are necessary since the ARMv7 Armada 385 CPU has been supported in Linux since at least 4.2-rc1, but it is common for OpenWRT ports to ship patches to the kernel, either to backport missing functionality or perform some optimization. There has been some pressure from backers to petition Turris to "speedup the process of upstreaming Omnia support to OpenWrt". It could be that the team is too busy with delivering the devices already ordered to complete that process at this point. The software is available on the CZ-NIC GitHub repository and the actual Linux patches can be found here and here. CZ.NIC also operates a private GitLab instance where more software is available. There is technically no reason why you wouldn't be able to run your own distribution on the Omnia router: OpenWRT development snapshots should be able to run on the Omnia hardware and some people have installed Debian on Omnia. It may require some customization (e.g. the kernel) to make sure the Omnia hardware is correctly supported. Most people seem to prefer to run TurrisOS because of the extra features. The hardware itself is also free and open for the most part. There is a binary blob needed for the 5GHz wireless card, which seems to be the only proprietary component on the board. The schematics of the device are available through the Omnia wiki, but oddly not in the GitHub repository like the rest of the software.

Hands on I received my own router last week, which is about six months late from the original April 2016 delivery date; it allowed me to do some hands-on testing of the device. The first thing I noticed was a known problem with the antenna connectors: I had to open up the case to screw the fittings tight, otherwise the antennas wouldn't screw in correctly. Once that was done, I simply had to go through the usual process of setting up the router, which consisted of connecting the Omnia to my laptop with an Ethernet cable, connecting the Omnia to an uplink (I hooked it into my existing network), and go through a web wizard. I was pleasantly surprised with the interface: it was smooth and easy to use, but at the same time imposed good security practices on the user. Install wizard performing automatic updates For example, the wizard, once connected to the network, goes through a full system upgrade and will, by default, automatically upgrade itself (including reboots) when new updates become available. Users have to opt-in to the automatic updates, and can chose to automate only the downloading and installation of the updates without having the device reboot on its own. Reboots are also performed during user-specified time frames (by default, Omnia applies kernel updates during the night). I also liked the "skip" button that allowed me to completely bypass the wizard and configure the device myself, through the regular OpenWRT systems (like LuCI or SSH) if I needed to. The Omnia router about to rollback to latest snapshot Notwithstanding the antenna connectors themselves, the hardware is nice. I ordered the black metal case, and I must admit I love the many LED lights in the front. It is especially useful to have color changes in the reset procedure: no more guessing what state the device is in or if I pressed the reset button long enough. The LEDs can also be dimmed to reduce the glare that our electronic devices produce. All this comes at a price, however: at \$250 USD, it is a much higher price tag than common home routers, which typically go for around \$50. Furthermore, it may be difficult to actually get the device, because no orders are being accepted on the Indiegogo site after October 31. The Turris team doesn't actually want to deal with retail sales and has now delegated retail sales to other stores, which are currently limited to European deliveries.

A nice device to help fight off the IoT apocalypse It seems there isn't a week that goes by these days without a record-breaking distributed denial-of-service (DDoS) attack. Those attacks are more and more caused by home routers, webcams, and "Internet of Things" (IoT) devices. In that context, the Omnia sets a high bar for how devices should be built but also how they should be operated. Omnia routers are automatically upgraded on a nightly basis and, by default, do not provide telnet or SSH ports to run arbitrary code. There is the password-less wizard that starts up on install, but it forces the user to chose a password in order to complete the configuration. Both the hardware and software of the Omnia are free and open. The automatic update's EULA explicitly states that the software provided by CZ.NIC "will be released under a free software licence" (and it has been, as mentioned earlier). This makes the machine much easier to audit by someone looking for possible flaws, say for example a customs official looking to approve the import in the eventual case where IoT devices end up being regulated. But it also makes the device itself more secure. One of the problems with these kinds of devices is "bit rot": they have known vulnerabilities that are not fixed in a timely manner, if at all. While it would be trivial for an attacker to disable the Omnia's auto-update mechanisms, the point is not to counterattack, but to prevent attacks on known vulnerabilities. The CZ.NIC folks take it a step further and encourage users to actively participate in a monitoring effort to document such attacks. For example, the Omnia can run a honeypot to lure attackers into divulging their presence. The Omnia also runs an elaborate data collection program, where routers report malicious activity to a central server that collects information about traffic flows, blocked packets, bandwidth usage, and activity from a predefined list of malicious addresses. The exact data collected is specified in another EULA that is currently only available to users logged in at the Turris web site. That data can then be turned into tweaked firewall rules to protect the overall network, which the Turris project calls a distributed adaptive firewall. Users need to explicitly opt-in to the monitoring system by registering on a portal using their email address. Turris devices also feature the Majordomo software (not to be confused with the venerable mailing list software) that can also monitor devices in your home and identify hostile traffic, potentially leading users to take responsibility over the actions of their own devices. This, in turn, could lead users to trickle complaints back up to the manufacturers that could change their behavior. It turns out that some companies do care about their reputations and will issue recalls if their devices have significant enough issues. It remains to be seen how effective the latter approach will be, however. In the meantime, the Omnia seems to be an excellent all-around server and router for even the most demanding home or small-office environments that is a great example for future competitors.
Note: this article first appeared in the Linux Weekly News.

13 November 2016

Andrew Cater: MiniDebconf ARM Cambridge 13/11/16 - Day 4 post 4 - lightning talks

Ian Jackson - dgit in two slides with attitude :)

Mike Crowe - Code Club - volunteering to teach programming to school children in after school clubs. Scratch: version 1 is packaged for Debian and Raspberry Pi CSS HTML Python - all materials provided. Projects for Microbits as well. No slides.

Dimitri Ledkov - SPI board - explaining about SPI. SPI - US non-profit, also registered in Europe. Collects donations, can register trademarks, can hold assets on your project's behalf. Volunteer project: controlled by member orgs.

Guus Sliepen - ifupdown - dual protocol, bonding, VPNs, wireless,
- too many slides :)






Andrew Cater: Debian MiniConf, ARM Cambridge 13/11/16 - Day 4 post 1

Just got here, partway into Guus Sliepen's talk about issues with sourceless packages and the DFSG. Lots of problems with, for example, music for games and imagery and also areas where packages autobuild / use generated blobs which are not supplied or are not able to be regenerated.

Checked in by front desk - thanks again to Lisa, to Jo McIntyre and Lucy Wayland who have been very patient waiting for us, very helpful as ever, chasing us at the end of each day to reconcile badges and people and doing all the useful stuff that no one sees.

ARM security folk are still here, obviously, and the building work has stopped for the weekend so we can get in and out of the building more readily. Thanks once again to the ARM staffers who are also Debian-ites and to the ARM management who have to authorise all of this and allow us to use their facilities, buildings and guest WiFi. It's testament to a whole lot of hard work behind the scenes that this all seems seamless and (generally) works so well

12 November 2016

Andrew Cater: Debian Miniconf ARM Cambridge Day 3 12/11/16 - post 2

Steve Capper on Java.

Room quiet - lots of folk listening and typing simultaneously.

A couple of folk behind me are writing for later in the day.

Thanks to all for their help (and what is it with me and typing dates this week) :)

11 November 2016

Chris Lamb: Awarded Core Infrastructure Initiative grant for Reproducible Builds

I'm delighted to announce that I have been awarded a grant from the Core Infrastructure Initiative (CII) to fund my previously-voluntary work on Reproducible Builds. Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users. The motivation behind the Reproducible Builds effort is to permit verification that no flaws have been introduced either maliciously or accidentally during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. I'd like to sincerely thank the CII, not only for their material support but also for their recognition of my existing contributions. I am looking forward to working with my co-grantees towards fulfilling our shared goal. You can read the CII's press release here.

9 November 2016

Russ Allbery: Some thoughts on the US elections

I apparently am not going to get anything done today until I write this. Some thoughts, in no particular order. I don't have any profound conclusions. I'm honestly pretty upset. And pretty scared. But we have to talk to each other. And we have to listen to each other. And we have to persaude each other. And we have to be willing to be persuaded. And please go tell someone this morning that you have their back.

Next.

Previous.