Search Results: "nate"

12 July 2016

Dirk Eddelbuettel: RProtoBuf 0.4.4, and new JSS paper

A new release 0.4.4 of RProtoBuf is now on CRAN, and corresponds to the source archive for the Journal of Statistical Software paper about RProtoBuf as JSS vol71 issue 02. The paper is also included as a pre-print in the updated package. RProtoBuf provides R bindings for the Google Protocol Buffers ("Protobuf") data encoding library used and released by Google, and deployed as a language and operating-system agnostic protocol by numerous projects. This release brings small cleanups as well as build-system updates for the updated R 3.3.0 infrastructure based on g++ 4.9.*.
Changes in RProtoBuf version 0.4.4 (2016-07-10)
  • New vignette based on our brand-new JSS publication (v71 i02)
  • Some documentation enhancements were made, as well as other minor cleanups to file modes and operations
  • Unit-test vignette no longer writes to /tmp per CRAN request
  • The new Windows toolchain (based on g++ 4.9.*) is supported
CRANberries also provides a diff to the previous release. The RProtoBuf page has an older package vignette, a 'quick' overview vignette, a unit test summary vignette, and the pre-print for the JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

11 July 2016

Jonathan Dowland: iPod

iPod iPod with rockbox
opened up open iPod with iFlash
It's been four years since I last wrote about music players. In the meantime of my Sansa Fuzes broke, and the third does not have great rockbox support. I've also been using a Sansa Clip+ (a leaving present from my last job, thanks again!) and a Sansa Clip Zip. Unfortunately Sandisk's newer Sansa devices (Sport, Jam - the only ones still in production) are not supported by Rockbox. The Clips have been very reliable and sturdy players, but I have missed the larger display of the Fuze. Since I've been exploring HD audio I've also been interested in something with an A/D converter that can handle it. I also still wish to carry my entire music library around with me, which limits my options. I decided to try an iPod. The older iPods had a Wolfson-manufacturered ADC which had a good reputation and supported (in headline terms at least) 24/48. The iPod colour (aka "4th gen") and subsequent models have a large colour display. The click-wheel interface is also very nice. Apple have now discontinued the "classic" iPod and their second hand value has greatly increased, but I managed to get an older 5th generation model ("video", with a base capacity of 30G) whilst trading in some unwanted DVDs. The case was scratched to heck but a replacement was readily and cheaply available from auction sites. Rockbox support in these iPods is pretty good, and you can mod the hardware to support CF or SD cards with kits such as the iFlash kits by Tarkan Akdam, which I picked up, along with a new 128G SD card. Unfortunately I have found writing to the iPod to be very poor with Rockbox, but it's fine for playback, and booting the iPod in OF or DFU mode is very easy and works reliably. Whilst Rockbox on the iPod works pretty well, installing it is far harder than on the Sandisk Sansa devices. The difficulty in my case is because rockbox requires a PC-formatted iPod to install, and I had a Mac-formatted one. I couldn't find a way to convert the iPod to PC format using a Mac. I tried doing so on a PC but for some reason the PC wasn't playing ball so I gave up after a few hours. In the end I assembled the filesystem by hand using dd(1) and dumps of partition tables from other people's iPods, via a Linux machine. This was enough to convince iTunes on Mac to restore it's hidden partition and boot software without reverting back to a Mac disklabel format.

Dirk Eddelbuettel: Rcpp now used by over 700 CRAN packages

600 Rcpp packages Earlier this morning, Rcpp reached another milestone: 701 packages on CRAN now depend on it (as measured by Depends, Imports and LinkingTo declarations). The graph is on the left depicts the growth of Rcpp usage over time. Rcpp cleared 300 packages in November 2014. It passed 400 packages in June of last year (when I only tweeted about it), 500 packages in late October and 600 packages exactly four months ago in March. The chart extends to the very beginning via manually compiled data from CRANberries and checked with crandb. Then next part uses manually saved entries, and the final and largest part of the data set was generated semi-automatically via a short script appending updates to a small file-based backend. A list of packages using Rcpp is kept on this page. Also displayed in the graph is the relative proportion of CRAN packages using Rcpp. The four per-cent hurdle was cleared just before useR! 2014 where I showed a similar graph (as two distinct graphs) in my invited talk. We passed five percent in December of 2014, six percent last July, seven percent just before Christmas and now criss-crosses 8 eight percent, or a little less than one in twelve R packages. 700 user packages is a really large and humbling number. This places quite some responsibility on us in the Rcpp team as we continue to try our best try to keep Rcpp as performant and reliable as it has been. So with that a very big Thank You! to all users and contributors of Rcpp for help, suggestions, bug reports, documentation or, of course, code.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

10 July 2016

Paul Tagliamonte: SNIff

A while back, I found myself in need of two webservers that would terminate TLS (with different rules). I wanted to run some custom code I d written (which uses TLS peer authentication), and also nginx on port 443. The best way I figured out how to do this was to write a tool to sit on port 443, and parse TLS Client Hello packets, and dispatch to the correct backend depending on the SNI name. SNI, or Server Name Indication allows the client to announce (yes over cleartext!) what server it s looking for, similar to the HTTP Host header. Sometimes, like in the case above, the Host header won t work, since you ve already done a TLS handshake by the time you figure out who they re looking for. I also spun the Client Hello parser out into its own importable package, just in case someone else finds themselves in this same boat. The code s up on github.com/paultag/sniff!

Michal Čihař: Weblate 2.7

Slightly later than on monthly schedule but Weblate 2.7 is out today. This release brings improvements to the API and is first to officially support wlc a command line client for Weblate. Full list of changes for 2.7: If you are upgrading from older version, please follow our upgrading instructions. You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Aptoide, FreedomBox, Weblate itself and many other projects. Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure. Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Filed under: Debian English SUSE Weblate 0 comments

Craig Small: procps 3.3.12

The procps developers are happy to announce that version 3.3.12 of procps was released today. This version has a mixture of bug fixes and enhancements. This unfortunately means another API bump but we are hoping this will be fixed with the new library API coming soon. procps is developed on gitlab and the new version of procps can be found at https://gitlab.com/procps-ng/procps/tree/newlib procps 3.3.12 can be found at https://gitlab.com/procps-ng/procps/tags/v3.3.12 From the NEWS file, procps 3.1.12 has the following: We are hoping this will be the last one to use the old API and the new format API ( imaginatively called newlib ) will be used in subsequent releases. Feedback for this and any other version of procps can be sent to either the issue tracker or the development email list.

8 July 2016

Russell Coker: Nexus 6P and Galaxy S5 Mini

Just over a month ago I ordered a new Nexus 6P [1]. I ve had it for over a month now and it s time to review it and the Samsung Galaxy S5 Mini I also bought. Security The first noteworthy thing about this phone is the fingerprint scanner on the back. The recommended configuration is to use your fingerprint for unlocking the phone which allows a single touch on the scanner to unlock the screen without the need to press any other buttons. To unlock with a pattern or password you need to first press the power button to get the phone s attention. I have been considering registering a fingerprint from my non-dominant hand to reduce the incidence of accidentally unlocking it when carrying it or fiddling with it. The phone won t complete the boot process before being unlocked. This is a good security feature. Android version 6 doesn t assign permissions to apps at install time, they have to be enabled at run time (at least for apps that support Android 6). So you get lots of questions while running apps about what they are permitted to do. Unfortunately there s no allow for the duration of this session option. A new Android feature prevents changing security settings when there is an overlay running . The phone instructs you to disable overlay access for the app in question but that s not necessary. All that is necessary is for the app to stop using the overlay feature. I use the Twilight app [2] to dim the screen and use redder colors at night. When I want to change settings at night I just have to pause that app and there s no need to remove the access from it note that all the web pages and online documentation saying otherwise is wrong. Another new feature is to not require unlocking while at home. This can be a convenience feature but fingerprint unlocking is so easy that it doesn t provide much benefit. The downside of enabling this is that if someone stole your phone they could visit your home to get it unlocked. Also police who didn t have a warrant permitting search of a phone could do so anyway without needing to compel the owner to give up the password. Design This is one of the 2 most attractive phones I ve owned (the other being the sparkly Nexus 4). I think that the general impression of the appearance is positive as there are transparent cases on sale. My phone is white and reminds me of EVE from the movie Wall-E. Cables This phone uses the USB Type-C connector, which isn t news to anyone. What I didn t realise is that full USB-C requires that connector at both ends as it s not permitted to have a data cable with USB-C at the device and and USB-A at the host end. The Nexus 6P ships with a 1M long charging cable that has USB-C at both ends and a ~10cm charging cable with USB-C at one end and type A at the other (for the old batteries and the PCs that don t have USB-C). I bought some 2M long USB-C to USB-A cables for charging my new phone with my old chargers, but I haven t yet got a 1M long cable. Sometimes I need a cable that s longer than 10cm but shorter than 2M. The USB-C cables are all significantly thicker than older USB cables. Part of that would be due to having many more wires but presumably part of it would be due to having thicker power wires for delivering 3A. I haven t measured power draw but it does seem to charge faster than older phones. Overall the process of converting to USB-C is going to be a lot more inconvenient than USB SuperSpeed (which I could basically ignore as non-SuperSpeed connectors worked). It will be good when laptops with USB-C support become common, it should allow thinner laptops with more ports. One problem I initially had with my Samsung Galaxy Note 3 was the Micro-USB SuperSpeed socket on the phone being more fiddly for the Micro-USB charging plug I used. After a while I got used to that but it was still an annoyance. Having a symmetrical plug that can go into the phone either way is a significant convenience. Calendars and Contacts I share most phone contacts with my wife and also have another list that is separate. In the past I had used the Samsung contacts system for the contacts that were specific to my phone and a Google account for contacts that are shared between our phones. Now that I m using a non-Samsung phone I got another Gmail account for the purpose of storing contacts. Fortunately you can get as many Gmail accounts as you want. But it would be nice if Google supported multiple contact lists and multiple calendars on a single account. Samsung Galaxy S5 Mini Shortly after buying the Nexus 6P I decided that I spend enough time in pools and hot tubs that having a waterproof phone would be a good idea. Probably most people wouldn t consider reading email in a hot tub on a cruise ship to be an ideal holiday, but it works for me. The Galaxy S5 Mini seems to be the cheapest new phone that s waterproof. It is small and has a relatively low resolution screen, but it s more than adequate for a device that I ll use for an average of a few hours a week. I don t plan to get a SIM for it, I ll just use Wifi from my main phone. One noteworthy thing is the amount of bloatware on the Samsung. Usually when configuring a new phone I m so excited about fancy new hardware that I don t notice it much. But this time buying the new phone wasn t particularly exciting as I had just bought a phone that s much better. So I had more time to notice all the annoyances of having to download updates to Samsung apps that I ll never use. The Samsung device manager facility has been useful for me in the past and the Samsung contact list was useful for keeping a second address book until I got a Nexus phone. But most of the Samsung apps and 3d party apps aren t useful at all. It s bad enough having to install all the Google core apps. I ve never read mail from my Gmail account on my phone. I use Fetchmail to transfer it to an IMAP folder on my personal mail server and I d rather not have the Gmail app on my Android devices. Having any apps other than the bare minimum seems like a bad idea, more apps in the Android image means larger downloads for an over-the-air update and also more space used in the main partition for updates to apps that you don t use. Not So Exciting In recent times there hasn t been much potential for new features in phones. All phones have enough RAM and screen space for all common apps. While the S5 Mini has a small screen it s not that small, I spent many years with desktop PCs that had a similar resolution. So while the S5 Mini was released a couple of years ago that doesn t matter much for most common use. I wouldn t want it for my main phone but for a secondary phone it s quite good. The Nexus 6P is a very nice phone, but apart from USB-C, the fingerprint reader, and the lack of a stylus there s not much noticeable difference between that and the Samsung Galaxy Note 3 I was using before. I m generally happy with my Nexus 6P, but I think that anyone who chooses to buy a cheaper phone probably isn t going to be missing a lot.

7 July 2016

Matthew Garrett: Bluetooth LED bulbs

The best known smart bulb setups (such as the Philips Hue and the Belkin Wemo) are based on Zigbee, a low-energy, low-bandwidth protocol that operates on various unlicensed radio bands. The problem with Zigbee is that basically no home routers or mobile devices have a Zigbee radio, so to communicate with them you need an additional device (usually called a hub or bridge) that can speak Zigbee and also hook up to your existing home network. Requests are sent to the hub (either directly if you're on the same network, or via some external control server if you're on a different network) and it sends appropriate Zigbee commands to the bulbs.

But requiring an additional device adds some expense. People have attempted to solve this in a couple of ways. The first is building direct network connectivity into the bulbs, in the form of adding an 802.11 controller. Go through some sort of setup process[1], the bulb joins your network and you can communicate with it happily. Unfortunately adding wifi costs more than adding Zigbee, both in terms of money and power - wifi bulbs consume noticeably more power when "off" than Zigbee ones.

There's a middle ground. There's a large number of bulbs available from Amazon advertising themselves as Bluetooth, which is true but slightly misleading. They're actually implementing Bluetooth Low Energy, which is part of the Bluetooth 4.0 spec. Implementing this requires both OS and hardware support, so older systems are unable to communicate. Android 4.3 devices tend to have all the necessary features, and modern desktop Linux is also fine as long as you have a Bluetooth 4.0 controller.

Bluetooth is intended as a low power communications protocol. Bluetooth Low Energy (or BLE) is even lower than that, running in a similar power range to Zigbee. Most semi-modern phones can speak it, so it seems like a pretty good choice. Obviously you lose the ability to access the device remotely, but given the track record on this sort of thing that's arguably a benefit. There's a couple of other downsides - the range is worse than Zigbee (but probably still acceptable for any reasonably sized house or apartment), and only one device can be connected to a given BLE server at any one time. That means that if you have the control app open while you're near a bulb, nobody else can control that bulb until you disconnect.

The quality of the bulbs varies a great deal. Some of them are pure RGB bulbs and incapable of producing a convincing white at a reasonable intensity[2]. Some have additional white LEDs but don't support running them at the same time as the colour LEDs, so you have the choice between colour or a fixed (and usually more intense) white. Some allow running the white LEDs at the same time as the RGB ones, which means you can vary the colour temperature of the "white" output.

But while the quality of the bulbs varies, the quality of the apps doesn't really. They're typically all dreadful, competing on features like changing bulb colour in time to music rather than on providing a pleasant user experience. And the whole "Only one person can control the lights at a time" thing doesn't really work so well if you actually live with anyone else. I was dissatisfied.

I'd met Mike Ryan at Kiwicon a couple of years back after watching him demonstrate hacking a BLE skateboard. He offered a couple of good hints for reverse engineering these devices, the first being that Android already does almost everything you need. Hidden in the developer settings is an option marked "Enable Bluetooth HCI snoop log". Turn that on and all Bluetooth traffic (including BLE) is dumped into /sdcard/btsnoop_hci.log. Turn that on, start the app, make some changes, retrieve the file and check it out using Wireshark. Easy.

Conveniently, BLE is very straightforward when it comes to network protocol. The only thing you have is GATT, the Generic Attribute Protocol. Using this you can read and write multiple characteristics. Each packet is limited to a maximum of 20 bytes. Most implementations use a single characteristic for light control, so it's then just a matter of staring at the dumped packets until something jumps out at you. A pretty typical implementation is something like:

0x56,r,g,b,0x00,0xf0,0x00,0xaa

where r, g and b are each just a single byte representing the corresponding red, green or blue intensity. 0x56 presumably indicates a "Set the light to these values" command, 0xaa indicates end of command and 0xf0 indicates that it's a request to set the colour LEDs. Sending 0x0f instead results in the previous byte (0x00 in this example) being interpreted as the intensity of the white LEDs. Unfortunately the bulb I tested that speaks this protocol didn't allow you to drive the white LEDs at the same time as anything else - setting the selection byte to 0xff didn't result in both sets of intensities being interpreted at once. Boo.

You can test this out fairly easily using the gatttool app. Run hcitool lescan to look for the device (remember that it won't show up if anything else is connected to it at the time), then do gatttool -b deviceid -I to get an interactive shell. Type connect to initiate a connection, and once connected send commands by doing char-write-cmd handle value using the handle obtained from your hci dump.

I did this successfully for various bulbs, but annoyingly hit a problem with one from Tikteck. The leading byte of each packet was clearly a counter, but the rest of the packet appeared to be garbage. For reasons best known to themselves, they've implemented application-level encryption on top of BLE. This was a shame, because they were easily the best of the bulbs I'd used - the white LEDs work in conjunction with the colour ones once you're sufficiently close to white, giving you good intensity and letting you modify the colour temperature. That gave me incentive, but figuring out the protocol took quite some time. Earlier this week, I finally cracked it. I've put a Python implementation on Github. The idea is to tie it into Ulfire running on a central machine with a Bluetooth controller, making it possible for me to control the lights from multiple different apps simultaneously and also integrating with my Echo.

I'd write something about the encryption, but I honestly don't know. Large parts of this make no sense to me whatsoever. I haven't even had any gin in the past two weeks. If anybody can explain how anything that's being done there makes any sense at all[3] that would be appreciated.

[1] typically via the bulb pretending to be an access point, but also these days through a terrifying hack involving spewing UDP multicast packets of varying lengths in order to broadcast the password to associated but unauthenticated devices and good god the future is terrifying

[2] For a given power input, blue LEDs produce more light than other colours. To get white with RGB LEDs you either need to have more red and green LEDs than blue ones (which costs more), or you need to reduce the intensity of the blue ones (which means your headline intensity is lower). Neither is appealing, so most of these bulbs will just give you a blue "white" if you ask for full red, green and blue

[3] Especially the bit where we calculate something from the username and password and then encrypt that using some random numbers as the key, then send 50% of the random numbers and 50% of the encrypted output to the device, because I can't even

comment count unavailable comments

Michal Čihař: uTidylib 0.3

Several years ago I've complained about uTidylib not being maintained upstream. Since that time I've occasionally pushed some fixes to my GitHub repository with uTidylib code, but without any clear intentions to take it over. Time has gone and there was still no progress and I started to consider becoming upstream maintainer as well. I quickly got approval from Cory Dodt, who was the original author of this code, unfortunately he is not owner of the PyPI entry and the claim request seems to have no response (if you know how to get in touch with "cntrlr" or how to take over PyPI module please let me know). Anyway the amount of patches in my repository is big enough to warrant new release. Additionally Debian bug report about supporting new HTML tidy library came in and that made me push towards releasing 0.3 version of the uTidylib. As you might guess, the amount of changes against original uTidylib is quite huge, to name the most important ones: Anyway as I can not update PyPI entry, the downloads are currently available only on my website: https://cihar.com/software/utidylib/

Filed under: Debian English uTidylib 0 comments

Petter Reinholdtsen: Unlocking HTC Desire HD on Linux using unruu and fastboot

Yesterday, I tried to unlock a HTC Desire HD phone, and it proved to be a slight challenge. Here is the recipe if I ever need to do it again. It all started by me wanting to try the recipe to set up an hardened Android installation from the Tor project blog on a device I had access to. It is a old mobile phone with a broken microphone The initial idea had been to just install CyanogenMod on it, but did not quite find time to start on it until a few days ago. The unlock process is supposed to be simple: (1) Boot into the boot loader (press volume down and power at the same time), (2) select 'fastboot' before (3) connecting the device via USB to a Linux machine, (4) request the device identifier token by running 'fastboot oem get_identifier_token', (5) request the device unlocking key using the HTC developer web site and unlock the phone using the key file emailed to you. Unfortunately, this only work fi you have hboot version 2.00.0029 or newer, and the device I was working on had 2.00.0027. This apparently can be easily fixed by downloading a Windows program and running it on your Windows machine, if you accept the terms Microsoft require you to accept to use Windows - which I do not. So I had to come up with a different approach. I got a lot of help from AndyCap on #nuug, and would not have been able to get this working without him. First I needed to extract the hboot firmware from the windows binary for HTC Desire HD downloaded as 'the RUU' from HTC. For this there is is a github project named unruu using libunshield. The unshield tool did not recognise the file format, but unruu worked and extracted rom.zip, containing the new hboot firmware and a text file describing which devices it would work for. Next, I needed to get the new firmware into the device. For this I followed some instructions available from HTC1Guru.com, and ran these commands as root on a Linux machine with Debian testing:
adb reboot-bootloader
fastboot oem rebootRUU
fastboot flash zip rom.zip
fastboot flash zip rom.zip
fastboot reboot
The flash command apparently need to be done twice to take effect, as the first is just preparations and the second one do the flashing. The adb command is just to get to the boot loader menu, so turning the device on while holding volume down and the power button should work too. With the new hboot version in place I could start following the instructions on the HTC developer web site. I got the device token like this:
fastboot oem get_identifier_token 2>&1   sed 's/(bootloader) //'
And once I got the unlock code via email, I could use it like this:
fastboot flash unlocktoken Unlock_code.bin
And with that final step in place, the phone was unlocked and I could start stuffing the software of my own choosing into the device. So far I only inserted a replacement recovery image to wipe the phone before I start. We will see what happen next. Perhaps I should install Debian on it. :)

6 July 2016

Mike Gabriel: [Arctica Project] Release of nx-libs (version 3.5.99.0)

Introduction NX is a software suite which implements very efficient compression of the X11 protocol. This increases performance when using X applications over a network, especially a slow one. NX (v3) has been originally developed by NoMachine and has been Free Software ever since. Since NoMachine obsoleted NX (v3) some time back in 2013/2014, the maintenance has been continued by a versatile group of developers. The work on NX (v3) is being continued under the project name "nx-libs". History Until January 2015, nx-libs was more mainly a redistribution approach of the original NX (v3) software. We (we as in mainly a group of X2Go developers) kept changes applied to the original NoMachine sources as minimal as possible. We also kept the original files and folders structure. Patches had been maintained via the quilt utility on top of a Git repository, the patches had always been kept separate. That was the 3.5.0.x series of nx-libs. In January 2015, the characteristics of nx-libs as maintained by the X2Go project between 2011 and 2015 changed. A decision was reached: This effort is now about to be released as "nx-libs 3.6.0.0". Contributors Since 2011, the nx-libs code base has to a great extent been maintained in the context of the X2Go Project [1]. Qindel Group joining in... In 2014, developers from the Qindel Group [2] joined the nx-libs maintenance. They found X2Go's work on nx-libs and suggested joining forces as best as possible on hacking nx-libs. The Arctica Project comming up... Starting in January 2015, the development on the 3.6.x branch of the project was moved into a new project called the Arctica Project [3]. Development Funding by Qindel In September 2015, a funding project was set up at Qindel. Since then, the Qindel group greatly supports the development of nx-libs 3.6.x monetarily and with provided man power. The funding project officially is a cooperation between Qindel and DAS-NETZWERKTEAM (business run by Mike Gabriel, long term maintainer of nx-libs). The funding is split into two subprojects and lasts until August 2017: The current nx-libs development effort is coordinated in the context of the Arctica Project (by Mike Gabriel), with use cases in Arctica, X2Go and TheQVD (VDI product worked on at Qindel) in mind. People involved There are various people involved in nx-libs 3.6.x maintenance and development, some of them shall be explicitly named here (in alphabetical order of surnames): Some other people have contributed, but have left the project already. Thanks for your work on nx-libs: A big thanks to everyone involved!!! Special thanks go to Stefan Baur for employing Mihai Moldovan and handling all the bureaucracy, so that Mihai can work on this project and get funded for his work. Achievements of nx-libs 3.5.99.0 We are very close to our self-defined release goal 3.6.0. The below tasks have already been (+/-) completed for version 3.5.99.0: In a previous blog post [4], the code reduction in nx-libs has already been discussed. With this announcement, we want to give a status update about our effort of shrinking the nx-libs code base:
    [mike@minobo nx-libs (3.6.x)]$ cloc --match-f '.*\.(c cpp h)' .
        1896 text files.
        1896 unique files.                                          
        7430 files ignored.
    http://cloc.sourceforge.net v 1.60  T=5.88 s (307.3 files/s, 143310.5 lines/s)
    -------------------------------------------------------------------------------
    Language                     files          blank        comment           code
    -------------------------------------------------------------------------------
    C                              958          68574          74891         419638
    C/C++ Header                   730          25683          33957         130418
    C++                            120          17007          11721          61292
    -------------------------------------------------------------------------------
    SUM:                          1808         111264         120569         611348
    -------------------------------------------------------------------------------
The previous statistics had these sums in the last line. First the nx-libs 3.5.0.x code tree (where we came from):
    -------------------------------------------------------------------------------
    SUM:                          5614         329279         382337        1757743
    -------------------------------------------------------------------------------
Then the nx-libs 3.6.x status as it was on May 9th 2016:
    -------------------------------------------------------------------------------
    SUM:                          1922         118581         126783         662635
    -------------------------------------------------------------------------------
Another 50.000 lines of code have been removed over the past two months. Work pending for the final 3.6.0 release goal Known Issues There are several open issues on the nx-libs Github project space, see https://github.com/ArcticaProject/nx-libs/issues. Testing the nx-libs 3.5.99.0 We are currently working on provisioning release builds and nightly builds of nx-libs for various recent Linux distributions. Please stay tuned and watch Mike Gabriel's blog [5]. We already have nightly builds of nx-libs for Debian and Ubuntu [6], but there are more to come soon. Until now, please use the build recipes provided in the README.md file of the nx-libs source tree [7]. References

3 July 2016

Paul Wise: DebConf16 Open Festival day 1

Today was day one of the DebConf16 Open Festival and I attended the open hardware panel, part of the talk about Code For South Africa, shirish's experiences and the DebConf new folks session. The open hardware panel was a wide ranging discussion between bdale, Andy and indiebio. bdate talked about the experiences he has had with his rocketry hardware. bdale said "Make concious decisions about what you are buying", referencing a case where he investigated, found a GPL violation and didn't buy. Various people care about openness of different layers of the hardware. Off-the-shelf products are very strongly integrated, which is great for makers but means that people who care about lower layers like CPU micro-architecture aren't able to participate. Andy said "We are just beginning to come out of the shareware stage [of open hardware]". bdale mentioned the companies who do hardware production as a service from design files. Later in the pub some folks mentioned j-core, an open re-implementation of SuperH processors. I missed most of the code4sa talk unfortunately, but it was about government services and open data. shirish covered his journey through life to Debian. His youth, how satellite TV and knowledge of the outside world came to India around the time of the Iraq war. His experience accessing the Internet for the first time, uncensored vs the usual censorship in India's media. His experiences of Windows 95 viruses and crashes. He learned of PCTwist Linux through a magazine cover. His initial install was not a success but eventually managed to break through and install a desktop, but experienced network and other issues. Eventually he encountered Ubuntu and began contributing bug reports. His experiences there led him to Debian. He began blogging about Debian. In the last few years he and others have been going around the country doing mini-DebConfs at institutes around India. The first question was predictably about having a DebConf in India and how shirish might like to get more involved. DebConf in India sounds like a possibility some day and shirish was thinking about getting involved in publicity, marketting and the Debian installer. The DebConf new folks session was a great intro to DebConf for folks new to the community. There were some quite excellent touches added to this year's version of the event by indiebio and Rhonda. I also got some things done. Usual spam reporting. Reviewed wiki RecentChanges. Talked to the chromium-bsu/MacPorts maintainer about AX_CHECK_GL brokenness. Filed Debian wishlist bug #829292 asking to update autoconf-archive. Redirected a Hurd porterbox request to the exodar admin and quickly found out I was wrong to do that, rectified. Then we found out the LDAP sync to exodar was broken. Replied to someone who intends to sell Debian pre-installs. Thanked BunsenLabs folks for joining the derivatives list. Applied reproducible builds patch for cats from Chris Lamb. Heard about awesome new terminal-mode screensaver. Moo! Prepared a blog post about check-all-the-things.

Thadeu Lima de Souza Cascardo: Debian on old Linux versions

On recent posts, I mentioned that I have a Chromebook, and that I would like to run Debian on devices that ship with old Linux versions. The Samsung ARM Chromebook is such a device, it has Linux 3.4, and that's still what I am running on it most of the time. After an upgrade of Debian, systemd stopped working, and it took me some time before I could look into it. In order to boot, I had to temporarily use System V init. The bug involves the use of new interfaces, bugs in such interfaces, and how versioning would not be a solution to such cases. Cases like this are not common, so it's no excuse to doing the right thing when developing software, like considering portability, API and ABI maintenance and not bundle. But they expose some of the challenges when trying to support different OSes on top of old versions of Linux, or maybe even new versions. The new version of systemd included in Debian Jessie uses a feature of Linux of checking the mount ID of a file by inspecting the mnt_id line in the fdinfo files. fdinfo files are not that new, they have been present since 2007, Linux 2.6.22. The mnt_id line, however, is from Linux 3.15, from 2014. So, a fairly new feature. But systemd falls back to other methods if the feature is not present. However, during the last few weeks of development of Linux 3.4, a bug was introduced to fdinfo handling, which will cause its access to fail in a way that will cause systemd to halt. It simply will refuse to mount the most basic filesystems and not proceed. The bug was fixed not much longer than two weeks after that, but that window of time was enough to have it on a shipped product, one that was not using systemd, let alone one that required a feature not yet present on Linux. But now I wanted to run this new systemd on this old Linux version. Being too early in the boot process, systemd even ignored some of the debug options and I had to resort to other methods of debugging. As I am running a kernel I built by myself, it wasn't hard for me to have a fixed kernel up and running after I found out the problem. Unfortunately, not everyone will be able to do it by themselves. And the point of getting different OSes working on these many varied devices is scaling. If every device needs to be tested, we lose this scalability. The other problem I found was that this old Linux version didn't load firmware files by itself, and requested help from userspace, which used to be the job of udev. Well, udev now does not support loading firmware, and the kernel must do it by itself. I managed to backport that feature too. But now, it sounds like I should get upstream support for this device. Unfortunately, this means not working on the scalability problem, but going back to get upstream support for a lot of devices out there, which is very hard to scale without the necessary people to do the work. I am certainly going to do that, because it's the right thing to do. But not as many people will benefit from that. But I think it has been interesting to look into the possible problems we might find when trying to make Debian and other OSes working on old Linux versions. I know even older Debian versions won't boot fine on Linux 2.6.32, one of the LTS versions of Linux. I have one other device which runs on it, and I am also working on getting better upstream support for it. Let's see if I can get some news on that soon.

1 July 2016

Thorsten Alteholz: My Debian Activities in June 2016

FTP assistant This month I marked 233 packages for accept and rejected 29. I also sent 11 emails to maintainers asking questions. Currently there are 33 packages in NEW and the minimum this week has been as low as 24 packages. Come on you fellow developers, where are your packages? I am sure you can do better :-) . Debian LTS This was my twenty-fourth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. This month my all in all workload has been 18.75h. This resulted in patches for 13 CVEs and the following uploads: I also looked at mxml and libstruts1.2-java and marked CVEs for these packages as no-dsa . I also reviewed a patch of Salvatore for an embargoed CVE of xerces-c. Last but not least I looked at the remaining two CVEs for asterisk, but was not really able to create working patches This month I called again for testing php5. Thanks a lot to Stefan and anybody else who sent in their reports! As there are already new CVEs for php5 available, I am afraid I need your support again in July This month I also had another term of frontdesk work and answered questions or looked for CVEs that are important for Wheezy LTS or could be ignored. Other stuff I made some progress with the Alljoyn framework. Up to now the following packages are available: Unfortunately as some of those modules still need to be released in current versions, there are some gaps. Anyway, the next uploads will include an XMPP connector, to basically bridge a local AllJoyn bus to a remote AllJoyn bus over XMPP. Further, with the lighting module, real lamps can be switched on and off and much more. Also the Home Appliances and Entertainment Service Framework seems to be interesting as well. In the Javascript world I uploaded some new packages and uploaded new versions for the following packages:

Joachim Breitner: When to reroll a six

This is a story about counter-intuitive probabilities and how a small bit of doubt turned out to be very justified. It begins with the game To Court the King (German: Um Krone und Kragen ). It is a nice game with dice and cards, where you start with a few dice, and use your dice rolls to buy additional cards, which give you extra dice or special powers to modify the dice that you rolled. You can actually roll your dice many times, but every time, you have to set aside at least one die, which you can no longer change or reroll, until eventually all dice have been set aside. A few years ago, I have played this game a lot, both online (on yucata.de) as well as in real life. It soon became apparent that it is almost always better to go for the cards that give you an extra die, instead of those that let you modify the dice. Intuitively, this is because every additional die allows you to re-roll your dice once more. I concluded that if I have a certain number of dice (say, n), and I want to have a sum as high as possible at the end, then it may make sense to reroll as many dice as possible, setting aside only those showing a 6 (because that is the best you can get) or, if there is no dice showing a 6, then a single die with the best score. Besides for small number of dice (2 or 3), where even a 4 or 5 is worth keeping, this seemed to be a simple, obvious and correct strategy to maximize the expected outcome of this simplified game. It is definitely simple and obvious. But some doubt that it was correct remained. Having one more die still in the game (i.e. not set aside) definitely improves your expected score, because you can reroll the dice more often. How large is this advantage? What if it ever exceeds 6 then it would make sense to reroll a 6. The thought was strange, but I could not dismiss it. So I did what one does these days if one has a question: I posed it on the mathematics site of StackExchange. That was January 2015, and nothing happened. I tried to answer it myself a month later, or at least work towards at an answer, and did that by brute force. Using a library for probabilistic calculations for Haskell I could write some code that simply calculated the various expected values of n dice for up to n = 9 (beyond that, my unoptimized code would take too long):
1:  3.50000 (+3.50000)
2:  8.23611 (+4.73611)
3: 13.42490 (+5.18879)
4: 18.84364 (+5.41874)
5: 24.43605 (+5.59241)
6: 30.15198 (+5.71592)
7: 35.95216 (+5.80018)
8: 41.80969 (+5.85753)
9: 47.70676 (+5.89707)
Note that this calculation, although printed as floating point numbers, is performed using fractions of unbounded integers, so there are no rounding issues that could skew the result. The result supported the hypothesis that there is no point in rolling a 6 again: The value of an additional die grows and approaches 6 from beyond, but judging from these number is never going to reach it. Then again nothing happened. Until 14 month later, when some Byron Schmuland came along, found this an interesting puzzle, and set out a 500 point bounty to whoever solved this problem. This attracted a bit attention, and a few not very successful attempts at solving this. Eventually it reached twitter, where Roman Cheplyaka linked to it. Coincidally a day later some joriki came along, and he had a very good idea: Why not make our life easier and think about dice with less sides, and look at 3 instead of 6. This way, and using a more efficient implementation (but still properly using rationals), he could do a similar calculation for up to 50 dice. And it was very lucky that he went to 50, and not just 25, because up to 27, the results were very much as expected, approaching value of +3 from below. But then it surpassed +3 and became +3.000000008463403. In other words: If you have roll 28 dice, and you have exactly two dice showing a 3, then it gives you better expected score if you set aside only one 3, and not both of them. The advantage is minuscule, but that does not matter it is there. From then on, the results behaved strangely. Between 28 and 34, the additional value was larger than 3. Then, from 35 on again lower than 2. It oscillated. Something similar could be observed when the game is played with coins. Eventually, joriki improved his code and applied enough tricks so that he could solve it also for the 6-sided die: The difference of the expected value of 198 dice and having 199 dice is larger than 6 (by 10 21...)! The optimizations that allowed him to calculate these numbers in a reasonable amount of time unfortunately was to assume that my original hypothesis (never rerolling a 6 is optimal), which held until n < 199. But this meant that for n > 199, the code did not yield correct results. What is the rationale of the story? Don t trust common sense when it comes to statistics; don t judge a sequence just from a few initial numbers; if you have an interesting question, post it online and wait for 16 months.

Rapha&#235;l Hertzog: My Free Software Activities in June 2016

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian packaging Django and Python. I uploaded Django 1.9.7 and filed an upstream ticket (#26755) for a failure seen in its DEP-8 tests. I packaged/sponsored python-django-modeltranslation and python-paypal. I opened a pull request on model-translation to fix failing tests in the Debian package build. I packaged a new python-django-jsonfield (1.0.0), filed a bug and discovered some regression in its PostgreSQL support. I helped on the upstream ticket and I have been granted commit rights. I used this opportunity to do some bug triage and push a few fixes. I also discussed the future of the module and ended up starting a discussion on Django s developer list about the possibility to add a JSONField to the core. CppUTest. I uploaded a new upstream version (3.8) with more than a year of work. I found out that make install does not install a required header so I opened a ticket with a patch. The package ended up not compiling on quite a few architectures so I opened a ticket and prepared a fix for some of those failures with the help of the upstream developers. I also added a DEP-8 tests after having uploaded a broken (untested) package systemd support in net-snmp and postfix. I worked on adding native systemd service units to net-snmp (#782243) and postfix (#715188). In both cases, the maintainers have not been very reactive so far so I uploaded my changes as delayed NMU. pkg-security team. The team that I started quietly a few months ago is now growing, both with new members and new packages. I created the required Teams/pkg-security wiki page. I sponsored xprobe, hydra, made an upload of medusa to merge Kali changes into Debian (and at the same time submitting the patch to upstream). fontconfig. After having read Jonathan McDowell s analysis of a bug that I experienced multiple times (and that many Kali users had too), I opened bug #828037 to get it fixed once for all. Unfortunately, nothing happened yet. DebConf 16 I spent some time to prepare the 2 talks and the BoF that I will give/manage in Cape Town next week: Distro Tracker I continued to mentor Vladimir Likic who managed to finish his first patch. He is now working on documentation for new contributors based on his recent experience. I enhanced the tox configuration to run tests with Django 1.8 LTS with fatal warnings (python -Werror) so as to ensure that I m not relying on any deprecated feature and so that I can be sure that the codebase will work on the next Django LTS release (1.11). Thanks to this, I did discover quite a few places where I have been using deprecated API and I fixed them all (the JSONField update to 1.0.0 I mentionned above was precisely to fix such a warning). I also fixed a few more issues with folded mail headers that you can t inject back in a new Message object and with messages lacking the subject field. All those have been caught through real (spam) email generating exceptions wich are then mailed to me. Kali related work I uploaded a new live-boot (5.20160608) to Debian to fix a bug where the boot process was blocking on some timeout. I forwarded a Kali bug against libatk-wrapper-java (#827741) which turned out to be an OpenJDK bug. I filed #827749 against reprepro to request a way to remove selected internal file references. This is required if you want to be able to make a file disappear and if that file is part of a snapshot that you want to keep despite this. But in truth, my real need is to be able to replace the .orig.tar.gz used by Kali by the orig.tar.gz used by Debian those conflicts break the mirroring/import script. Salt I have been using salt to deploy a new service, and I developed patches for a few issues in salt formulas. I also created a new letsencrypt-sh formula to manage TLS certificates with the letsencrypt.sh ACME client. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

29 June 2016

Olivier Gr goire: Fifth week at GSoC: push information from the daemon!

Last week I worked to create a window for the gnome client to display information.
This week I worked on linking the D-BUS with the gnome client.
To do that I needed to modify the LRC.
-Create a QT slot to catch the signal from the D-BUS
-Create a signal connect with a lambda function on the client Unfortunately, I can only push a single variable at the time. So, I chose to use MAP to contain all my information. After changing this type in the daemon, D-Bus, LRC and the gnome client, everything finally work!

Michal &#268;iha&#345;: PHP shapefile library

Since quite a long time phpMyAdmin had embedded the bfShapeFiles library for import of geospatial data. Over the time we had to apply fixes to it to stay compatible with newer PHP versions, but there was really no development. Unfortunately, as it seems to be only usable PHP library which can read and write ESRI shapefiles. With recent switch of phpMyAdmin to dependency handling using Composer I wondered if we should get rid of the last embedded PHP library, which was this one - bfShapeFiles. As I couldn't find alive library which would work well for us, I resisted that for quite long, until pull request to improve it came in. At that point I've realized that it's probably better to separate it and start to improve it outside our codebase. That's when phpmyadmin/shapefile was started. The code is based on bfShapeFiles, applies all fixes which were used in phpMyAdmin and adds improvements from the pull request. On top of that it has brand new testsuite (the coverage is still much lower than I'd like to have) and while writing the tests several parsing issues have been discovered and fixed. Anyway you can now get the source from GitHub or install using Composer from Packagist. PS: While fixing parser bugs I've looked at other parsers as well to see how they handle some situations unclear in the specs and I had to fix Python pyshp on the way as well :-).

Filed under: Debian English phpMyAdmin 0 comments

28 June 2016

Reproducible builds folks: First steps towards getting containers working

Author: ceridwen The 0.1 alpha release of reprotest has been accepted into Debian unstable and is available for install at packages.debian.org or through apt. I've been working on redesigning reprotest so that it runs commands through autopkgtest's adt_testbed interface. For the most part, I needed to replace explicit calls to Python standard library functions for copying files and directories with calls to adt_testbed.Testbed.command() with copyup and copydown, and to use Testbed.execute() and Testbed.check_exec() to run commands instead of subprocess. To test reprotest on the actual containers requires having containers constructed for this purpose. autopkgtest has a test that builds a minimal chroot. I considered doing something like this approach or using BusyBox. However, I have a Python script that mocks a build process, which requires having Python available in the container, and while I looked into busybox-python and MicroPython to keep the footprint small, I decided that for now this would take too much work and decided to go straight to the autopkgtest recommendations for building containers, mk-sbuild and vmdebootstrap. (I also ended up discovering a bug in debootstrap.) This means that to get the tests run requires some manual setup at the moment. In the long run, I'd like to improve that, but it's not an immediate priority. While working on adding tests for the other containers supported by autopkgtest, I also converted to py.test so that I could use fixtures and parametrization to run the Cartesian product of each variation with each container. With tests written, I started trying to verify that my new code worked. One problem I encountered while trying to debug was that I wasn't getting full error output. In VirtSubproc.check_exec(), execute_timeout() acts something like a Popen() call:
(status, out, err) = execute_timeout(None, timeout, real_argv,
                                     stdout=stdout, stderr=subprocess.PIPE)
if status:
    bomb("%s%s failed (exit status %d)" %
         ((downp and "(down) " or ""), argv, status))
if err:
    bomb("%s unexpectedly produced stderr output  %s'" %
         (argv, err))
The problem with this is that if the call returns a non-zero exit code, which is typical for program failures, stderr doesn't get included in the error message. I changed the first if-block to:
if status:
    bomb("%s%s failed (exit status %d)\n%s" %
         ((downp and "(down) " or ""), argv, status, err))
Another example is that autopkgtest calls schroot with the --quiet flag, which in one case was making schroot fail without any output due to a misconfiguration. I'm still trying to find and eliminate more places where errors are silenced. autopkgtest was designed to be installed with Debian's packaging system, which handles arbitrary files and directory layouts. Unfortunately, setuptools is completely different in a way that doesn't work well with autopkgtest's design. (I'm sure this is partly because setuptools has to support all the different major OSes that run Python, including Windows.) As I discussed last week, autopkgtest has Python scripts in virt/ that are executed by subprocess calls in adt_testbed. Because these scripts import files from lib/, there needs to be an __init__.py in virt/ to make it into a package and a sys.path hack in each script to allow it to find modules in lib/. Unfortunately, setuptools will not install this structure. First, setuptools will not install any file without a .py extension into a package. Theoretically, this is fixable, the files in virt/ are Python scripts so I could rename them. (Theoretically, there's supposed to be some workaround involving MANIFEST.in or package_data in setup.py, but I have yet to find any documentation or explanation giving a method for installing non-Python files inside a Python package.) Second, however, setuptools does not preserve the executable bit when installing package files. The obvious workaround, changing the subprocess calls so that they invoke python virt/foo.py rather than virt/foo.py requires changing all the internal calls in the autopkgtest code, which I'm loathe to do for fear of breaking it. (It's not clear to me I can easily find all of the calls, for starters.) There are about three solutions to this I see at the moment, all of them difficult. The first involves using either the scripts keyword or console_scripts entry point in setup.py, as explained here. The scripts keyword is supposed to preserve the executable bit according to this StackExchange question, but I haven't verified this myself, and like everything to do with setuptools I don't trust anything anyone says about it without testing it myself. It also has the disadvantage of dumping them all into the common scripts directory. Using console_scripts involves rewriting all of them to have an executable function I can refer to in setup.py. I worry that this would be both fragile and break existing expectations in the rest of the autopkgtest code, but it might be the best solution. The third solution involves refactoring of all the autopkgtest code to import the code in the scripts rather than running it through subprocess calls. I'm reluctant to do this because I think it's almost certain to break things that will require significant work to fix. Getting setuptools to install the autopkgtest code correctly is one blocker for the next release. Another is that autopkgtest's handling of errors during the build process involves closing the adt_testbed.Testbed so it won't take further commands. Unfortunately, this handling runs before any cleanup code I write to run outside it, which means that at the moment errors during the build will result in things like disorderfs being left mounted. The last release blocker is that adt_testbed doesn't have any way to set a working directory when running commands. For instance, the virt/schroot script always calls schroot with --directory=/. I thought about trying to use absolute paths, but decided this was unintuitive and impractical. For the user, this would mean that instead of running something simple like make in the correct directory, they would have to run make --file=/absolute/path/to/Makefile or something similar, making all paths absolute. I worry that some build scripts wouldn't handle this correctly, either: for instance, running python setup.py from a different directory can have different effects because Python's path is initialized to contain the current directory. Changing this is going to require going deeper into the autopkgtest code than I'd hoped. I intend to try to resolve these three issues over the next week and then prepare the next release, though how much progress I make depends on how thorny they turn out to be.

27 June 2016

Alessio Treglia: A not exactly United Kingdom

Island of Ventotene Roman harbour

There once was a Kingdom strongly United, built on the honours of the people of Wessex, of Mercia, Northumbria and East Anglia who knew how to deal with the invasion of the Vikings from the east and of Normans from the south, to come to unify the territory under an umbrella of common intents. Today, however, 48% of them, while keeping solid traditions, still know how to look forward to the future, joining horizons and commercial developments along with the rest of Europe. The remaining 52%, however, look back and can not see anything in front of them if not a desire of isolation, breaking the European dream born on the shores of Ventotene island in 1944 by Altiero Spinelli, Ernesto Rossi and Ursula Hirschmann through the Manifesto for a free and united Europe . An incurable fracture in the country was born in a referendum on 23 June, in which just over half of the population asked to terminate his marriage to the great European family, bringing the UK back by 43 years of history. <Read More [by Fabio Marzocca]>

Next.

Previous.