Search Results: "edd"

8 December 2016

Dirk Eddelbuettel: RcppAPT 0.0.3

A new version of RcppAPT -- our interface from R to the C++ library behind the awesome apt, apt-get, apt-cache, ... commands and their cache powering Debian, Ubuntu and the like -- is now on CRAN. We changed the package to require C++11 compilation as newer Debian systems with g++-6 and the current libapt-pkg-dev library cannot build under the C++98 standard which CRAN imposes (and let's not get into why ...). Once set to C++11 we have no issues. We also added more examples to the manual pages, and turned on code coverage. A bit more information about the package is available here as well as as the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

7 December 2016

Shirish Agarwal: Day trip in Cape Town, part 2

Debconf16 logo The post continues from the last post shared. Let me get some interesting tit-bits not related to the day-trip out-of-the-way first I don t know whether we had full access to see all parts of fuller hall or not. Couple of days I was wondering around Fuller Hall, specifically next to where clothes were pressed. Came to know of the laundry service pretty late but still was useful. Umm next to where the ladies/gentleman pressed our clothes, there is a stairway which goes down. In fact even on the opposite side there is a stairway which goes down. I dunno if other people explored them or not. The jail inside and under UCT I was surprised and shocked to see bars in each room as well as connecting walkways etc. I felt a bit sad, confused and curious and went on to find more places like that. After a while I came up to the ground-level and enquired with some of the ladies therein. I was shocked to know that UCT some years ago (they were not specific) was a jail for people. I couldn t imagine that a place which has so much warmth (in people, not climate) could be evil in a sense. I was not able to get much information out of them about the nature of jail it was, maybe it is a dark past that nobody wants to open up, dunno. There were also two *important* aspects of UCT which Bernelle either forgot, didn t share or I just came to know via the Wikipedia page then but nothing else. 1. MeerKAT Apparently quite a bit of the technology was built-in UCT itself. This would have been interesting for geeks and wanna-be geeks like me 2. The OpenContent Initiative by UCT This would have been also something worth exploring. One more interesting thing which I saw was the French council in Cape Town from outside The French Council in cape town from outside I would urge to look at the picture in the gallery as the picture I shared doesn t really show all the details. For e.g. the typical large french windows which are the hall-mark of French architecture doesn t show its glory but if you look at 1306 2322 original picture instead of the 202 360 reproduction you will see that. You will also the insignia of the French Imperial Eagle whose history I came to know only after I looked it up on the Wikipedia page on that day. It seemed fascinating and probably would have the same pride as the State Emblem of India has for Indians with the four Asiatic Lions standing in a circle protecting each other. I also like the palm tree and the way the French Council seemed little and yet had character around all the big buildings. What also was interesting that there wasn t any scare/fear-build and we could take photos from outside unlike what I had seen and experienced in Doha, Qatar as far as photography near Western Embassies/Councils were concerned. One of the very eye-opening moments for me was also while I was researching flights from India to South Africa. While perhaps unconsciously I might have known that Middle East is close to India, in reality, it was only during the search I became aware that most places in Middle East by flight are only an hour or two away. This was shocking as there is virtually no mention of one of our neighbours when they are source of large-scale remittances every year. I mean this should have been in our history and geography books but most do not dwell on the subject. It was only during and after that I could understand Mr. Modi s interactions and trade policies with the Middle East. Another interesting bit was seeing a bar in a Sprinbok bus spingbok atlas bar in bus While admittedly it is not the best picture of the bar, I was surprised to find a bar at the back of a bus. By bar I mean a machine which can serve anything from juices to alcoholic drinks depending upon what is stocked. What was also interesting in the same bus is that the bus also had a middle entrance-and-exit. The middle door in springbok atlas This is something I hadn t seen in most Indian buses. Some of the Volvo buses have but it is rarely used (only except emergencies) . An exhaustive showcase of local buses can be seen here . I find the hand-drawn/cad depictions of all the buses by Amit Pense near to the T. Axe which can be used to break windows Emergency exit window This is also something which I have not observed in Indian inter-city buses (axe to break the window in case of accident and breakable glass which doesn t hurt anyone I presume), whether they are State-Transport or the high-end Volvo s . Either it s part of South African Roads Regulations or something that Springbok buses do for their customers. All of these queries about the different facets I wanted to ask the bus-driver and the attendant/controller but in the excitement of seeing, recording new things couldn t ask In fact one of the more interesting things I looked at and could look day and night is the variety of vehicles on display in Cape Town. In hindsight, I should have bought a couple of 128 GB MMC cards for my mobile rather than the 64 GB one. It was just plain inadequate to capture all that was new and interesting. Auditorum chair truck seen near Auditorium This truck I had seen about some 100 metres near the Auditorium on Upper Campus. The truck s design, paint was something I had never seen before. It is/was similar to casket trucks seen in movies but the way it was painted and everything made it special. What was interesting is to see the gamut of different vehicles. For instance, there were no bicycles that I saw in most places. There were mostly Japanese/Italian bikes and all sorts of trucks. If I had known before, I would definitely have bought an SD specifically to take snaps of all the different types of trucks, cars etc. that I saw therein. The adage/phrase I should stop in any one place and the whole world will pass me by seemed true on quite a few South African Roads. While the roads were on par or a shade better than India, many of those were wide roads. Seeing those, I was left imagining how the Autobahn in Germany and other high-speed Expressways would look n feel. India has also been doing that with the Pune-Mumbai Expressway and projects like Yamuna Expressway and now the extension Agra Lucknow Expressway but doing this all over India would take probably a decade or more. We have been doing it since a decade and a half. NHDP and PMGSY are two projects which are still ongoing to better the roads. We have been having issues as to should we have toll or no toll issues but that is a discussion for some other time. One of the more interesting sights I saw was the high-arched gothic-styled church from outside. This is near Longstreet as well. high arch gothic-styled church I have seen something similar in Goa, Pondicherry but not such high-arches. I did try couple of times to gain entry but one time it was closed, the other time some repairing/construction work was going on or something. I would loved to see it from inside and hopefully they would have had an organ (music) as well. I could imagine to some extent the sort of music that would have come out. Now that Goa has come in the conversation I can t help but state that Seafood enthusiasts/lover/aficionado, or/and Pescatarianism would have a ball of a time in Goa. Goa is on the Konkan coast and while I m eggie, ones who enjoy seafood really have a ball of a time in Goa. Fouthama s Festival which happens in February is particularly attractive as Goan homes are thrown open for people to come and sample their food, exchange recipes and alike. This happens around 2 weeks before the Goan Carnival and is very much a part of the mish-mashed Konkani-Bengali-Parsi-Portugese culture. I better stop here about the Goa otherwise I ll get into reminiscing mode. To put the story and event back on track from where we left of (no fiction hereon), Nicholas was in constant communication with base, i.e. UCT as well as another group who was hiking from UCT to Table Mountain. We waited for the other group to join us till 13:00 hrs. We came to know that they were lost and were trying to come up and hence would take more time. As Bernelle was with them, who was a local and she had two dogs who knew the hills quite well, it was decided to go ahead without them. We came down the same cable-car and then ventured on towards Houtbay. Houtbay has it all, a fisherman s wharf, actual boats with tough-mean looking men with tattoos working on boats puffing cigars/pipes, gaggle of sea-gulls, the whole scene. Sharing a few pictures of the way in-between. the view en-route to Houtbay western style car paint and repair shop Tajmahal Indian Restaurant, Houtbay I just now had a quick look at the restaurant and it seems they had options for veggies too. Unfortunately, the rating leaves a bit to be desired but then dunno as Indian flavoring is something that takes time to get used too. Zomato doesn t give any idea of from when a restaurant is in business and has too few reviews so not easy to know how the experience would have been. Chinese noodles and small houses Notice the pattern, the pattern of small houses I saw all the way till Houtbay and back. I do vaguely remember starting a discussion about it on the bus but don t really remember. I have seen (on TV) cities like Miami, Dubai or/and Hong Kong who have big buildings on the beach but both in Konkan as well as Houtbay there were small buildings. I guess a combination of zoning regulations, feel of community, fear of being flooded all play into beaches being the way they are. Also, this probably is good as less stress on the environment. Miamiboyz from Wikimedia Commons The above picture is taken from Wikipedia from the article Miami Beach, Florida for comparison. Audi rare car to be seen in India The Audi rare car to be seen in India. This car has been associated with Ravi Shastri when he won it in 1985. I was young but still get goosebumps remembering those days. first-glance-Houtbay-and-pier First glance of Houtbay beach and pier. Notice how clean and white the beach is. Wharf-Grill-Restaurant-from-side-and-Hop-on-Hop-off-bus You can see the wharf grill restaurant in the distance (side-view), see the back of the hop on and hop off bus (a concept which was unknown to me till then). Once I came back and explored on the web came to know this concept is prevalent in many a touristy places around the world. Umm also By sheer happenchance also captured a beautiful looking Indian female . So many things happening all at once In Hindi, we would call this picture virodabhas or contradiction . this is in afternoon, around 1430 hrs. You have the sun, the clouds, the Mountains, the x number of boats, the pier, the houses, the cars, the shops. It was all crazy and beautiful at the same time. The Biggest Contradiction is seeing the Mountain, the beach and the Sea in the same Picture. Baffled the mind. Konkan though is a bit similar there as well. You have all the three things in some places but that s a different experience altogether as ours is a more tropical weather although is one of the most romantic places in the rains. We were supposed to go on a short cruise to seal/dolphin island but as we were late (as had been waiting for the other group) didn t go and instead just loitered there. Fake-real lookout bar-restaurant IIRC the lookout bar is situated just next to Houtbay Search and Rescue. Although was curious if the Lookout tower was used in case of disappearance. lost people, boats etc. Seal in action Seal jumping over water, what a miracle ! One of the boats on which we possibly could have been on. It looked like the boat we could have been on. I clicked as I especially liked the name Calypso and Calypso . I shared the two links as the mythologies, interpretation differ a bit between Greek and Hollywood culture Debian folks and the area around Can see few Debian folks in the foreground, next to the Pole and the area around. Also can see a bit of the area around. Alone boy trying to surf I don t know anything about water sports and after sometime he came out. I was left wondering though, how safe he was in that water. While he was close to the pier and he was just paddling, there weren t big waves still felt a bit of concern. Mr. Seal - the actor and his handler While the act was not to the level we see in the movies, still for the time I hung around, I saw him showing attitude for his younger audiences, eating out of their hands, making funny sounds. Btw he farted a few times, whether that was a put-on or not can t really say but produced a few guffaws from his audience. A family feeding Mr. Seal I dunno why the birds came down for. Mr. Seal was being fed oily small fish parts, dunno if the oil was secreted by the fish themselves or whatever, it just looked oily from distance. Bird-Man-Bird Bird taking necessary sun bath typical equipment on a boat to catch fish-lot of nets boats-nets-and-ropes People working on disentangling a net There wasn t much activity on the time we went. It probably would have been different on sunrise and would be on sunset. The only activity I saw was on this boat where they were busy fixing and disentangling the lines. I came up with 5-15 different ideas for a story but rejected them as a. Probably all of them have been tried. People have been fishing since the beginning of time and modern fishing probably 200 odd years or so. I have read accounts of fishing companies in early 1800s onwards, so probably all must have been tried. b. More dangerous one, if there is a unique idea, then it becomes more dangerous as writing is an all-consuming process. Writing a blog post (bad or good) takes lots of time. I constantly read, re-read, try and improvise till I can or my patience loses out. In book you simply can t have such luxuries. hout-bay-search-and-rescue-no-parking-zone No parking/tow zone in/near the Houtbay search and rescue. Probably to take out emergency vehicles once something untoward happens. hout-bay-sea-rescue-with-stats Saved 54 lives, boats towed 154 Salut! Houtbay sea rescue. The different springbok atlas bus that we were on kraal-kraft The only small criticism is for Houtbay there wasn t a single public toilet. We had to ask favor at kraal kraft to use their toilets and there could have been accidents, it wasn t lighted well and water was spilled around. Road sign telling that we are near to UCT For us, because we were late we missed both the boat-cruise as well as some street shops selling trinkets. Other than that it was all well. We should have stayed till sunset, I am sure the view would have been breath-taking but we hadn t booked the bus till evening. Back at UCT Overall it was an interesting day as we had explored part of Table Mountain, seen the somewhat outrageously priced trinkets there as well as explored Houtbay sea-side as well.
Filed under: Miscellenous Tagged: #Audi, #Cape Town, #Cruises, #Debconf16, #French Council, #Geography, #Houtbay Sea Rescue, #Jail, #Middle East, #Springbok Atlas, #Vehicles

5 December 2016

Shirish Agarwal: The Anti-Pollito squad arrest and confession

Disclaimer This is an attempt at humor and hence entirely fictional in nature. While some incidents depicted are true, the context and the story woven around them are by yours truly. None of the Mascots of Debian were hurt during the blog post . I also disavow any responsibility for any hurt (real or imagined) to any past, current and future mascots. The attempt should not be looked upon as demeaning people who are accused of false crimes, tortured and confessions eked out of them as this happens quite a lot (In India for sure, but guess it s the same world over in various degrees). The idea is loosely inspired by Chocolate:Deep Dark Secrets. (2005) On a more positive note, let s start Being a Sunday morning woke up late to find incessant knocking on the door, incidentally mum was not at home. Opening the door, found two official looking gentleman. They asked my name, asked my credentials, tortured and arrested me for Group conspiracy of Malicious Mischief in second and third degrees . The torture was done by means of making me forcefully watch endless reruns of Norbit . While I do love Eddie Murphy, this was one of his movies he could have done without . I guess for many people watching it once was torture enough. I *think* they were nominated for razzie awards dunno if they won it or not, but this is beside the point. Unlike the 20 years it takes for a typical case to reach to its conclusion even in the smallest court in India, due to the torture, I was made to confess (due to endless torture) and was given summary judgement. The judgement was/is as follows a. Do 100 hours of Community service in Debian in 2017. This could be done via blog posts, raising tickets in the Debian BTS or in whichever way I could be helpful to Debian. b. Write a confessional with some photographic evidence sharing/detailing some of the other members who were part of the conspiracy in view of the reduced sentence. So now, have been forced to write this confession As you all know, I won a bursary this year for debconf16. What is not known by most people is that I also got an innocuous looking e-mail titled Pollito for DPL . While I can t name all the names as investigation is still ongoing about how far-reaching the conspiracy is . The email was purportedly written by members of cabal within cabal which are in Debian. I looked at the email header to see if this was genuine and I could trace the origin but was left none the wiser, as obviously these people are far more technically advanced than to fall in simple tricks like this Anyways, secretly happy that I have been invited to be part of these elites, I did the visa thing, packed my bags and came to Debconf16. At this point in juncture, I had no idea whether it was real or I had imagined the whole thing. Then to my surprise saw this evidence of conspiracy to have Pollito as DPL, Wifi Password Just like the Illuminati the conspiracy was for all to see those who knew about it. Most people were thinking of it as a joke, but those like me who had got e-mails knew better. I knew that the thing is real, now I only needed to bide my time and knew that the opportunity would present itself. And few days later, sure enough, there was a trip planned for Table Mountain, Cape Town . Few people planned to hike to the mountain, while few chose to take the cable car till up the mountain. First glance of the cable car with table mountain as background Quite a few people came along with us and bought tickets for the to and fro to the mountain and back. Ticket for CPT Table mountain car cable Incidentally, I was thinking if the South African Govt. were getting the tax or not. If you look at the ticket, there is just a bar-code. In India as well as the U.S. there is TIN Tax Identification Number TIN displayed on an invoice from channeltimes.com Few links to share what it is all about . While these should be on all invoices, need to specially check when taking high-value items. In India as shared in the article the awareness, knowledge leaves a bit to be desired. While I m drifting from the incident, it would be nice if somebody from SA could share how things work there. Moving on, we boarded the cable car. It was quite spacious cable car with I guess around 30-40 people or some more who were able to see everything along with the controller. from inside the table mountain cable car 360 degrees It was a pleasant cacophony of almost two dozen or more nationalities on this 360 degrees moving chamber. I was a little worried though as it essentially is a bucket and there is always a possibility that a severe wind could damage it. Later somebody did share that some frightful incidents had occurred not too long ago on the cable car. It took about 20-25 odd minutes to get to the top of table mountain and we were presented with views such as below View from Table Mountain cable car looking down The picture I am sharing is actually when we were going down as all the pictures of going up via the cable car were over-exposed. Also, it was pretty crowded on the way up then on the way down so handling the mobile camera was not so comfortable. Once we reached up, the wind was blowing at incredible speeds. Even with my jacket and everything I was feeling cold. Most of the group around 10-12 people looked around if we could find a place to have some refreshments and get some of the energy in the body. So we all ventured to a place and placed our orders the bleh... Irish coffee at top of Table Mountain I was introduced to Irish Coffee few years back and have had some incredible Irish Coffees in Pune and elsewhere. I do hope to be able to make Irish Coffee at home if and when I have my own house. This is hotter than brandy and is perfect if you are suffering from cold etc if done right, really needs some skills. This is the only drink which I wanted in SA which I never got right . As South Africa was freezing for me, this would have been the perfect antidote but the one there as well as elsewhere were all bleh. What was interesting though, was the coffee caller besides it. It looked like a simple circuit mounted on a PCB board with lights, vibrations and RFID and it worked exactly like that. I am guessing as and when the order is ready, there is an interrupt signal sent via radio waves which causes the buzzer to light and vibrate. Here s the back panel if somebody wants to take inspiration and try it as a fun project backpanel of the buzz caller Once we were somewhat strengthened by the snacks, chai, coffee etc. we made our move to seeing the mountain. The only way to describe it is that it s similar to Raigad Fort but the plateau seemed to be bigger. The wikipedia page of Table Mountain attempts to share but I guess it s more clearly envisioned by one of the pictures shared therein. table mountain panaromic image I have to say while Table Mountain is beautiful and haunting as it has scenes like these Some of the oldest rocks known to wo/man. There is something there which pulls you, which reminds you of a long lost past. I could have simply sat there for hours together but as was part of the group had to keep with them. Not that I minded. The moment I was watching this, I was transported to some memories of the Himalayas about 20 odd years or so. In that previous life, I had the opportunity to be with some of the most beautiful women and also been in the most happening places, the Himalayas. I had shared years before some of my experiences I had in the Himalayas. I discontinued it as I didn t have a decent camera at that point in time. While I don t wanna digress, I would challenge anybody to experience the Himalayas and then compare. It is just something inexplicable. The beauty and the rawness that Himalayas shows makes you feel insignificant and yet part of the whole cosmos. What Paulo Cohello expressed in The Valkyries is something that could be felt in the Himalayas. Leh, Ladakh, Himachal , Garwhal, Kumaon. The list will go on forever as there are so many places, each more beautiful than the other. Most places are also extremely backpacker-friendly so if you ask around you can get some awesome deals if you want to spend more than a few days in one place. Moving on, while making small talk @olasd or Nicolas Dandrimont , the headmaster of our trip made small talk to each of us and eked out from all of us that we wanted to have Pollito as our DPL (Debian Project Leader) for 2017. Few pictures being shared below as supporting evidence as well The Pollito as DPL cabal in action members of the Pollito as DPL where am I or more precisely how far am I from India. While I do not know who further up than Nicolas was on the coup which would take place. The idea was this If the current DPL steps down, we would take all and any necessary actions to make Pollito our DPL. Pollito going to SA - photo taken by Jonathan Carter This has been taken from Pollito s adventure Being a responsible journalist, I also enquired about Pollito s true history as it would not have been complete without one. This is the e-mail I got from Gunnar Wolf, a friend and DD from Mexico
Turns out, Valessio has just spent a week staying at my house And
in any case, if somebody in Debian knows about Pollito s
childhood That is me. Pollito came to our lives when we went to Congreso Internacional de
Software Libre (CISOL) in Zacatecas city. I was strolling around the
very beautiful city with my wife Regina and our friend Alejandro
Miranda, and at a shop at either Ram n L pez Velarde or Vicente
Guerrero, we found a flock of pollitos. http://www.openstreetmap.org/#map=17/22.77111/-102.57145 Even if this was comparable to a slave market, we bought one from
them, and adopted it as our own. Back then, we were a young couple Well, we were not that young
anymore. I mean, we didn t have children. Anyway, we took Pollito with
us on several road trips, such as the only time I have crossed an
international border driving: We went to Encuentro Centroamericano de
Software Libre at Guatemala city in 2012 (again with Alejandro), and
you can see several Pollito pics at: http://gwolf.org/album/road-trip-ecsl-2012-guatemala-0 Pollito likes travelling. Of course, when we were to Nicaragua for
DebConf, Pollito tagged along. It was his first flight as a passenger
(we never asked about his previous life in slavery; remember, Pollito
trust no one). Pollito felt much welcome with the DebConf crowd. Of course, as
Pollito is a free spirit, we never even thought about forcing him to
come back with us. Pollito went to Switzerland, and we agreed to meet
again every year or two. It s always nice to have a chat with him. Hugs!
So with that backdrop I would urge fellow Debianities to take up the slogans LONG LIVE THE DPL ! LONG LIVE POLLITO ! LONG LIVE POLLITO THE DPL ! The first step to make Pollito the DPL is to ensure he has a @debian.org (pollito@debian.org) We also need him to be made a DD because only then can he become a DPL. In solidarity and in peace
Filed under: Miscellenous Tagged: #caller, #confession, #Debconf16, #debian, #Fiction, #history, #Pollito, #Pollito as DPL, #Table Mountain, Cabal, memories, south africa

3 December 2016

Ben Hutchings: Linux Kernel Summit 2016, part 1

I attended this year's Linux Kernel Summit in Santa Fe, NM, USA and made notes on some of the sessions that were relevant to Debian. LWN also reported many of the discussions. This is the first of two parts of my notes; part 2 is here. Stable process Jiri Kosina, in his role as a distribution maintainer, sees too many unsuitable patches being backported - e.g. a fix for a bug that wasn't present or a change that depends on an earlier semantic change so that when cherry-picked it still compiles but isn't quite right. He thinks the current review process is insufficient to catch them. As an example, a recent fix for a minor information leak (CVE-2016-9178) depended on an earlier change to page fault handling. When backported by itself, it introduced a much more serious security flaw (CVE-2016-9644). This could have been caught very quickly by a system call fuzzer. Possible solutions: require 'Fixes' field, not just 'Cc: stable'. Deals with 'bug wasn't present', but not semantic changes. There was some disagreement whether 'Fixes' without 'Cc: stable' should be sufficient for inclusion in stable. Ted Ts'o said he specifically does that in some cases where he thinks backporting is risky. Greg Kroah-Hartman said he takes it as a weaker hint for inclusion in stable. Is it a good idea to keep 'Cc: stable' given the risk of breaking embargo? On balance, yes, it only happened once. Sometimes it's hard to know exactly how/when the bug was introduced. Linus doesn't want people to guess and add incorrect 'Fixes' fields. There is still the option to give some explanation and hints for stable maintainers in the commit message. Ideally the upstream developer should provide a test case for the bug. Is Linus happy? Linus complained about minor fixes coming later in the release cycle. After rc2, all fixes should either be for new code introduced in the current release cycle or for important bugs. However, new, production-ready drivers without new infrastructure dependencies are welcome at almost any point in the release cycle. He was unhappy about some big changes in RDMA, but I'm not sure what those were. Bugzilla and bug tracking Laura Abbott started a discussion of bugzilla.kernel.org, talking about subsystems where maintainers ignore it and any responses come from random people giving bad advice. This is a terrible experience for users. Several maintainers are actively opposed to using it, and the email bridge no longer works (or not well?). She no longer recommends Fedora bug submitters to submit reports there. Are there any alternatives? None were proposed. Someone asked whether Bugzilla could tell reporters to use email for certain products/components instead of continuing with the bug entry process. Konstantin Ryabitsev talked about the difficulty of upgrading a customised instance of Bugzilla. Much customisation requires patches which don't apply to next version (maybe due to limitations of the extension mechanism?). He has had to drop many such patches. Email is hard to track when a bug is handed over from one maintainer to another. Email archives are very unreliable. Linus: I'll take Bugzilla over mail-archive. No-one is currently keeping track of bugs across the kernel and making sure they get addressed by an appropriate maintainer. It's (at least) a full-time job but no individual company has business case for paying for this. Konstantin suggested (I think) that CII might pay for this. There was some discussion of what information should be included in a bug report. The Cut here line in oops messages was said to be a mistake because there are often relevant messages before it. The model of computer is often important. Beyond that, there was not much interest in the automated information gathering that distributions do. Distribution maintainers should curate bugs before forwarding upstream. There was a request for custom fields per component in Bugzilla. Konstantin says this is doable (possibly after upgrade to version 5); it doesn't require patches. The future of the Kernel Summit The kernel community is growing, and the invitation list for the core day is too small to include all the right people for technical subjects. For 2017, the core half-day will have an even smaller invitation list, only ~30 subsystem maintainers that Linus pulls from. The entire technical track will be open (I think). Kernel Summit 2017 and some mini-summits will be held in Prague alongside Open Source Summit Europe (formerly LinuxCon Europe) and Embedded Linux Conference Europe. There were some complaints that LinuxCon is not that interesting to kernel developers, compared to Linux Plumbers Conference (which followed this year's Kernel Summit). However, the Linux Foundation is apparently soliciting more hardcore technical sessions. Kernel Summit and Linux Plumbers Conference are quite small, and it's apparently hard to find venues for them in cities that also have major airports. It might be more practical to co-locate them both with Open Source Summit in future. time_t and 2038 On 32-bit architectures the kernel's representation of real time (time_t etc.) will break in early 2038. Fixing this in a backward-compatible way is a complex problem. Arnd Bergmann presented the current status of this process. There has not yet been much progress in mainline, but more fixes have been prepared. The changes to struct inode and to input events are proving to be particularly problematic. There is a need to add new system calls, and he intends to add these for all (32-bit) achitectures at once. Copyright retention James Bottomley talked about how developers can retain copyright on their contributions. It's hard to renegotiate within an existing employment; much easier to do this when preparing to sign a new contract. Some employers expect you to fill in a document disclosing 'prior inventions' you have worked on. Depending on how it's worded, this may require the employer to negotiate with you again whenever they want you to work on that same software. It's much easier for contractors to retain copyright on their work - customers expect to have a custom agreement and don't expect to get copyright on contractor's software.

Vincent Bernat: Build-time dependency patching for Android

This post shows how to patch an external dependency for an Android project at build-time with Gradle. This leverages the Transform API and Javassist, a Java bytecode manipulation tool.
buildscript  
    dependencies  
        classpath 'com.android.tools.build:gradle:2.2.+'
        classpath 'com.android.tools.build:transform-api:1.5.+'
        classpath 'org.javassist:javassist:3.21.+'
        classpath 'commons-io:commons-io:2.4'
     
 
Disclaimer: I am not a seasoned Android programmer, so take this with a grain of salt.

Context This section adds some context to the example. Feel free to skip it. Dashkiosk is an application to manage dashboards on many displays. It provides an Android application you can install on one of those cheap Android sticks. Under the table, the application is an embedded webview backed by the Crosswalk Project web runtime which brings an up-to-date web engine, even for older versions of Android1. Recently, a security vulnerability has been spotted in how invalid certificates were handled. When a certificate cannot be verified, the webview defers the decision to the host application by calling the onReceivedSslError() method:
Notify the host application that an SSL error occurred while loading a resource. The host application must call either callback.onReceiveValue(true) or callback.onReceiveValue(false). Note that the decision may be retained for use in response to future SSL errors. The default behavior is to pop up a dialog.
The default behavior is specific to Crosswalk webview: the Android builtin one just cancels the load. Unfortunately, the fix applied by Crosswalk is different and, as a side effect, the onReceivedSslError() method is not invoked anymore2. Dashkiosk comes with an option to ignore TLS errors3. The mentioned security fix breaks this feature. The following example will demonstrate how to patch Crosswalk to recover the previous behavior4.

Simple method replacement Let s replace the shouldDenyRequest() method from the org.xwalk.core.internal.SslUtil class with this version:
// In SslUtil class
public static boolean shouldDenyRequest(int error)  
    return false;
 

Transform registration Gradle Transform API enables the manipulation of compiled class files before they are converted to DEX files. To declare a transform and register it, include the following code in your build.gradle:
import com.android.build.api.transform.Context
import com.android.build.api.transform.QualifiedContent
import com.android.build.api.transform.Transform
import com.android.build.api.transform.TransformException
import com.android.build.api.transform.TransformInput
import com.android.build.api.transform.TransformOutputProvider
import org.gradle.api.logging.Logger
class PatchXWalkTransform extends Transform  
    Logger logger = null;
    public PatchXWalkTransform(Logger logger)  
        this.logger = logger
     
    @Override
    String getName()  
        return "PatchXWalk"
     
    @Override
    Set<QualifiedContent.ContentType> getInputTypes()  
        return Collections.singleton(QualifiedContent.DefaultContentType.CLASSES)
     
    @Override
    Set<QualifiedContent.Scope> getScopes()  
        return Collections.singleton(QualifiedContent.Scope.EXTERNAL_LIBRARIES)
     
    @Override
    boolean isIncremental()  
        return true
     
    @Override
    void transform(Context context,
                   Collection<TransformInput> inputs,
                   Collection<TransformInput> referencedInputs,
                   TransformOutputProvider outputProvider,
                   boolean isIncremental) throws IOException, TransformException, InterruptedException  
        // We should do something here
     
 
// Register the transform
android.registerTransform(new PatchXWalkTransform(logger))
The getInputTypes() method should return the set of types of data consumed by the transform. In our case, we want to transform classes. Another possibility is to transform resources. The getScopes() method should return a set of scopes for the transform. In our case, we are only interested by the external libraries. It s also possible to transform our own classes. The isIncremental() method returns true because we support incremental builds. The transform() method is expected to take all the provided inputs and copy them (with or without modifications) to the location supplied by the output provider. We didn t implement this method yet. This causes the removal of all external dependencies from the application.

Noop transform To keep all external dependencies unmodified, we must copy them:
@Override
void transform(Context context,
               Collection<TransformInput> inputs,
               Collection<TransformInput> referencedInputs,
               TransformOutputProvider outputProvider,
               boolean isIncremental) throws IOException, TransformException, InterruptedException  
    inputs.each  
        it.jarInputs.each  
            def jarName = it.name
            def src = it.getFile()
            def dest = outputProvider.getContentLocation(jarName, 
                                                         it.contentTypes, it.scopes,
                                                         Format.JAR);
            def status = it.getStatus()
            if (status == Status.REMOVED)   //  
                logger.info("Remove $ src ")
                FileUtils.delete(dest)
              else if (!isIncremental   status != Status.NOTCHANGED)   //  
                logger.info("Copy $ src ")
                FileUtils.copyFile(src, dest)
             
         
     
 
We also need two additional imports:
import com.android.build.api.transform.Status
import org.apache.commons.io.FileUtils
Since we are handling external dependencies, we only have to manage JAR files. Therefore, we only iterate on jarInputs and not on directoryInputs. There are two cases when handling incremental build: either the file has been removed ( ) or it has been modified ( ). In all other cases, we can safely assume the file is already correctly copied.

JAR patching When the external dependency is the Crosswalk JAR file, we also need to modify it. Here is the first part of the code (replacing ):
if ("$ src " ==~ ".*/org.xwalk/xwalk_core.*/classes.jar")  
    def pool = new ClassPool()
    pool.insertClassPath("$ src ")
    def ctc = pool.get('org.xwalk.core.internal.SslUtil') //  
    def ctm = ctc.getDeclaredMethod('shouldDenyRequest')
    ctc.removeMethod(ctm) //  
    ctc.addMethod(CtNewMethod.make("""
public static boolean shouldDenyRequest(int error)  
    return false;
 
""", ctc)) //  
    def sslUtilBytecode = ctc.toBytecode() //  
    // Write back the JAR file
    //  
  else  
    logger.info("Copy $ src ")
    FileUtils.copyFile(src, dest)
 
We also need the following additional imports to use Javassist:
import javassist.ClassPath
import javassist.ClassPool
import javassist.CtNewMethod
Once we have located the JAR file we want to modify, we add it to our classpath and retrieve the class we are interested in ( ). We locate the appropriate method and delete it ( ). Then, we add our custom method using the same name ( ). The whole operation is done in memory. We retrieve the bytecode of the modified class in . The remaining step is to rebuild the JAR file:
def input = new JarFile(src)
def output = new JarOutputStream(new FileOutputStream(dest))
//  
input.entries().each  
    if (!it.getName().equals("org/xwalk/core/internal/SslUtil.class"))  
        def s = input.getInputStream(it)
        output.putNextEntry(new JarEntry(it.getName()))
        IOUtils.copy(s, output)
        s.close()
     
 
//  
output.putNextEntry(new JarEntry("org/xwalk/core/internal/SslUtil.class"))
output.write(sslUtilBytecode)
output.close()
We need the following additional imports:
import java.util.jar.JarEntry
import java.util.jar.JarFile
import java.util.jar.JarOutputStream
import org.apache.commons.io.IOUtils
There are two steps. In , all classes are copied to the new JAR, except the SslUtil class. In , the modified bytecode for SslUtil is added to the JAR. That s all! You can view the complete example on GitHub.

More complex method replacement In the above example, the new method doesn t use any external dependency. Let s suppose we also want to replace the sslErrorFromNetErrorCode() method from the same class with the following one:
import org.chromium.net.NetError;
import android.net.http.SslCertificate;
import android.net.http.SslError;
// In SslUtil class
public static SslError sslErrorFromNetErrorCode(int error,
                                                SslCertificate cert,
                                                String url)  
    switch(error)  
        case NetError.ERR_CERT_COMMON_NAME_INVALID:
            return new SslError(SslError.SSL_IDMISMATCH, cert, url);
        case NetError.ERR_CERT_DATE_INVALID:
            return new SslError(SslError.SSL_DATE_INVALID, cert, url);
        case NetError.ERR_CERT_AUTHORITY_INVALID:
            return new SslError(SslError.SSL_UNTRUSTED, cert, url);
        default:
            break;
     
    return new SslError(SslError.SSL_INVALID, cert, url);
 
The major difference with the previous example is that we need to import some additional classes.

Android SDK import The classes from the Android SDK are not part of the external dependencies. They need to be imported separately. The full path of the JAR file is:
androidJar = "$ android.getSdkDirectory().getAbsolutePath() /platforms/" +
             "$ android.getCompileSdkVersion() /android.jar"
We need to load it before adding the new method into SslUtil class:
def pool = new ClassPool()
pool.insertClassPath(androidJar)
pool.insertClassPath("$ src ")
def ctc = pool.get('org.xwalk.core.internal.SslUtil')
def ctm = ctc.getDeclaredMethod('sslErrorFromNetErrorCode')
ctc.removeMethod(ctm)
pool.importPackage('android.net.http.SslCertificate');
pool.importPackage('android.net.http.SslError');
//  

External dependency import We must also import org.chromium.net.NetError and therefore, we need to put the appropriate JAR in our classpath. The easiest way is to iterate through all the external dependencies and add them to the classpath.
def pool = new ClassPool()
pool.insertClassPath(androidJar)
inputs.each  
    it.jarInputs.each  
        def jarName = it.name
        def src = it.getFile()
        def status = it.getStatus()
        if (status != Status.REMOVED)  
            pool.insertClassPath("$ src ")
         
     
 
def ctc = pool.get('org.xwalk.core.internal.SslUtil')
def ctm = ctc.getDeclaredMethod('sslErrorFromNetErrorCode')
ctc.removeMethod(ctm)
pool.importPackage('android.net.http.SslCertificate');
pool.importPackage('android.net.http.SslError');
pool.importPackage('org.chromium.net.NetError');
ctc.addMethod(CtNewMethod.make(" "))
// Then, rebuild the JAR...
Happy hacking!

  1. Before Android 4.4, the webview was severely outdated. Starting from Android 5, the webview is shipped as a separate component with updates. Embedding Crosswalk is still convenient as you know exactly which version you can rely on.
  2. I hope to have this fixed in later versions.
  3. This may seem harmful and you are right. However, if you have an internal CA, it is currently not possible to provide its own trust store to a webview. Moreover, the system trust store is not used either. You also may want to use TLS for authentication only with client certificates, a feature supported by Dashkiosk.
  4. Crosswalk being an opensource project, an alternative would have been to patch Crosswalk source code and recompile it. However, Crosswalk embeds Chromium and recompiling the whole stuff consumes a lot of resources.

2 December 2016

Shirish Agarwal: Air Congestion and Politics

Confession time first I am not a frequent flyer at all. My first flight was in early late 2006. It was a 2 hour flight from Bombay (BOM) to Bengaluru (formerly Bangalore, BLG) . I still remember the trepidation, the nervousness and excitement the first time I took to air. I still remember the flight very vividly, It was a typical humid day for Bombay/Mumbai and we (me and a friend) had gone to Sahar (the domestic airport) to take the flight in the evening. Before starting the sky had turned golden-orange and I was wondering how I would feel once I would be in air.We started at around 20:00 hours in the evening and as it was a clear night were able to see the Queen s necklace (Marine Drive) in all her glory. The photographs on the wikipedia page don t really do justice to how beautiful the whole boulevard looks at night, especially how it looks from up there. While we were seeing, it seemed the pilot had actually banked at 45 degrees angle so we can have the best view of the necklace OR maybe the pilot wanted to take a photo OR ME being in overdrive (like Robin Williams, the Russian immigrant in Moscow on the Hudson experiences the first time he goes to the mall ;)) In either way, this would be an experience I would never forget till the rest of my life. I remember I didn t move an inch (even to go the loo) as I didn t want to let go of the whole experience. While I came back after 3-4 days, I still remember re-experiencing/re-imagining the flights for a whole month each time I went to sleep. While I can t say it has become routinised, but have been lucky to have the opportunity to fly domestic around the country primarily for work. After the initial romanticism wears off, you try and understand the various aspects of the flight which are happening around you. These experiences are what lead to file/share today s blog post. Yesterday, Ms. Mamata Banerjee, one of the leaders of the Opposition cried wolf because the Aircraft was circling the Airport. Because she is the Chief Minister she feels she should have got precedent or at least that seems to be the way the story unfolded on TV. I have been about 15-20 times on flight in the last decade for work or leisure. Almost all the flights I have been, it has been routine that the flights fly around the Airport for 15-20 minutes before landing. This is routine . I have seen Airlines being stacked (remember the scene from Die Hard 2 where Holly Mclane, John Mclane s wife looks at different aircraft at different altitudes from her window seat) this is what an Airport has to do when it doesn t have enough runaways. In fact just read few days back MIAL is going for an emergency expansion as they weren t expecting as many passengers as they did this year as well as last. In fact the same day there was a near-miss between two aircraft in Mumbai airport itself. Because of Ms. Mamata s belligerence, this story didn t even get a mention in the TV mainstream media. The point I wanna underscore is that this is a fact of life and not just in India, world-over it seems hubs are being busier than ever, for instance Heathrow has been also a busy bee and they will to rework air operations as per a recent article . In India, Kolkata is also one of the busier airports . If anything, I hope it teaches her the issues that plague most Indian airports and she works with the Government in Center so the Airport can expand more. They just got a new terminal three years back. It is for these issues that the Indian Government has come with the Regional Connectivity Scheme . Lastly, a bit of welcome news to people thinking to visit India, the Govt. of the day is facilitating easier visa norms to increase tourism and trade to India. Hope this is beneficial to all and any Debian Developers who wanna come visit India I do hope that we also do get reciprocity from those countries as well.
Filed under: Miscellenous Tagged: # Domestic Flights, #Air Congestion, #Airport Expansion, #Kolkata, #near-miss, #Visa for tourists

1 December 2016

Daniel Pocock: Using a fully free OS for devices in the home

There are more and more devices around the home (and in many small offices) running a GNU/Linux-based firmware. Consider routers, entry-level NAS appliances, smart phones and home entertainment boxes. More and more people are coming to realize that there is a lack of security updates for these devices and a big risk that the proprietary parts of the code are either very badly engineered (if you don't plan to release your code, why code it properly?) or deliberately includes spyware that calls home to the vendor, ISP or other third parties. IoT botnet incidents, which are becoming more widely publicized, emphasize some of these risks. On top of this is the frustration of trying to become familiar with numerous different web interfaces (for your own devices and those of any friends and family members you give assistance to) and the fact that many of these devices have very limited feature sets. Many people hail OpenWRT as an example of a free alternative (for routers), but I recently discovered that OpenWRT's web interface won't let me enable both DHCP and DHCPv6 concurrently. The underlying OS and utilities fully support dual stack, but the UI designers haven't encountered that configuration before. Conclusion: move to a device running a full OS, probably Debian-based, but I would consider BSD-based solutions too. For many people, the benefit of this strategy is simple: use the same skills across all the different devices, at home and in a professional capacity. Get rapid access to security updates. Install extra packages or enable extra features if really necessary. For example, I already use Shorewall and strongSwan on various Debian boxes and I find it more convenient to configure firewall zones using Shorewall syntax rather than OpenWRT's UI. Which boxes to start with? There are various considerations when going down this path:
  • Start with existing hardware, or buy new devices that are easier to re-flash? Sometimes there are other reasons to buy new hardware, for example, when upgrading a broadband connection to Gigabit or when an older NAS gets a noisy fan or struggles with SSD performance and in these cases, the decision about what to buy can be limited to those devices that are optimal for replacing the OS.
  • How will the device be supported? Can other non-technical users do troubleshooting? If mixing and matching components, how will faults be identified? If buying a purpose-built NAS box and the CPU board fails, will the vendor provide next day replacement, or could it be gone for a month? Is it better to use generic components that you can replace yourself?
  • Is a completely silent/fanless solution necessary?
  • Is it possibly to completely avoid embedded microcode and firmware?
  • How many other free software developers are using the same box, or will you be first?
Discussing these options I recently started threads on the debian-user mailing list discussing options for routers and home NAS boxes. A range of interesting suggestions have already appeared, it would be great to see any other ideas that people have about these choices.

30 November 2016

Carl Chenet: My Free Software activities in November 2016

My Monthly report for Novembre 2016 gives an extended list of what were my Free Software related activities during this month. Personal projects: Journal du hacker: The Journal du hacker is a frenck-speaking Hacker News-like website dedicated to the french-speaking Free and Open source Software community. logo-journal-du-hacker That s all folks! See you next month!

29 November 2016

Shirish Agarwal: The Iziko South African Museum

This would be a bit long on my stay in Cape Town, South Africa after Debconf16. Before I start, let me share the gallery works, you can see some photos that I have been able to upload to my gallery . It seems we are using gallery 2 while upstream had made gallery 3 and then it sort of died. I actually asked in softwarerecs stackexchange site if somebody knows of a drop-in replacement for gallery and was told/shared about Pwigo . I am sure the admin knows about it. There would be costs to probably migrate from gallery to Pwigo with the only benefit that it would be something which would perhaps be more maintainable. The issues I face with the current gallery system are few things a. There is no way to know how much your progress your upload has taken.
b. After it has submit, it gives a fake error message saying some error has occurred. This has happened on every occasion/attempt. Now I don t know whether it is because I have slow upload speeds or something else altogether. I had shared the error page last time in the blog post hence not sharing again. Although, all the pictures which would be shared in this blog post would be from the same gallery Another thing I would like to share is a small beginner article I wrote about why I like Debian. Another interesting/tit-bit of news I came to know few days back that both Singapore and Qatar have given 96 hours visa free stopovers for Indians for select destinations. Now to start with the story/experience due to some unknown miracle/angel looking upon me I got the chance to go to Debconf16, South Africa. I m sure there was lot of backend discussions but in the end I was given the opportunity to be part of Debcamp and Debconf. While I hope to recount my Debcamp and Debconf experience in another or two blog posts, this would be exclusively the Post-Debconf Experiences I had. As such opportunities to visit another country are rare, I wanted to make the most of it. Before starting from Pune, I had talked with Amey about Visas, about Debconf as he had just been to Debconf15 the year before and various things related to travel. He was instrumental in me having a bit more knowledge about how to approach things. I was also lucky to have both Graham and Bernelle who also suggested, advised and made it possible to have a pleasant stay both during Debcamp and Debconf. The only quibble is I didn t know heaters were being made available to us without any cost. Moving on, a day or two before Debconf was about to conclude, I asked Bernelle s help even though she was battling a burn-out I believe as I was totally clueless about Cape Town. She accepted my request and asked me to look at hostels near Longmarket Street. I had two conditions a. It should not be very far from the airport
b. It should be near to all or most cultural experiences the city has to offer. We looked at hostelworld and from the options listed, it looked like Homebasecapetown looked to be a perfect fit. It was one of the cheaper options and they also had breakfast included in the pricing. I booked through hostelworld for a mixed dorm for 2 days as I was unsure how it would be (the first night effect I have shared about previously) . When I reached there, I found it to be as good as the pictures shared were, the dorm was clean (most important), people were friendly (also important) as well as toilets and shower were also clean while the water was hot, so all in all it was a win-win situation for me. Posters I saw at homebasecapetown While I m not much of an adrenaline-junkie it was nice to know the activities that could be done/taken. Brochures and Condoms just left of main hall. This was again interesting. While apologies for the poor shaky quality of the picture, I believe it is easy to figure out. There were Brochures of the city attractions as well as condoms that people could discreetly use if need be. I had seen such condoms in few toilets during and around Debconf and it felt good that the public were aware and prioritizing safety for their guests and students instead of having fake holier than thou attitudes that many places have. For instance, you wouldn t find something like this in toilets of most colleges in India or anywhere else for that matter. There are few vending machines in what are termed as red light areas or where prostitution is known/infamous to happen and even then most times it is empty. I have 2-3 social workers as friends and they are a source of news on such things. While I went to few places and each had an attraction to it, the one which had my literally eyes out of socket was the Iziko South African Museum . I have been lucky to been quite a few museums in India, the best rated science museum in India in my limited experience has been the Visvesvaraya Industrial & Technological Museum, Bengaluru India . A beer from me if a European can get it right. Don t worry if you mispronounce it, I mispronounce it couple of times till I get it right . Looking up the word Iziko the meaning of the word seems to be the hearth and if you look at the range of collections in the museum, you would think it fits. I was lucky to find couple of friends, one of whom was living at homebase and we decided to go to the museum together. Making friends on the road So Eduardo, my friend on the left and his friend, we went to the museum. While viewing the museum, there were no adjectives to describe it other than Wow and Endless . See fossils of fish-whale-shark ? OR Giant fish-whale-dolphin-shark some million years ago. and Reminder of JAWS ;) While I have more than a few pictures, the point is easily made. It seems almost inconceivable that creatures of such masses actually were on earth. While I played with the model of the jaws of a whale/shark in reality if something like that happened, I would have been fighting for my life. The only thing I missed or could have been better if they had some interactive installations to showcase the now universally accepted Charles Darwin s On the Origin of Species I had never seen anything like this. Sadly, there was nobody around to help us figure out things as I had read that most species of fish don t leave a skeleton behind so how were these models made? It just boggles the mind. Apart from the Science Museum I was also introduced to the bloody history that South Africa had. I saw The 1913 native land act which was not honored . I had been under the impression that India had got a raw deal when it was under British rule but looking at South African history I don t know. While we got our freedom in 1947 they got rid of apartheid about 20 years+ . I talked to lot of young African males and there was lot of naked hostility for the Europeans even today. It was a bit depressing but could relate to their point of view as similar sentiments were echoed by our forefathers. I read in the newspapers and it seemed to be a pretty mixed picture. I can t comment as only South Africans can figure out the way forward. For me, it was enough to know and see that we both had similar political histories as nations. It seemed the racial divide and anger was much more highly pronounced towards Europeans and divisive then the caste divisions here between Indians. I also shared with them my limited knowledge and understanding of the Indian history (as history is re-written all the time) and it was clear to them that we had common/similar pasts. As a result, what was surprising (actually not) is that many South Africans have no knowledge of Indian history. as well otherwise the political differences that South Africa and India has in the current scenario wouldn t have been. In the end, the trip proved to be fun, stimulating, educative, thought-provoking as questions about self-identity , national identity, our place in the Universe kinda questions which should be asked all the time. Thank you Bremmer and the team for letting me experience Cape Town, South Africa, I would have been poorer if I hadn t had the experience.
Filed under: Miscellenous Tagged: #Debconf16, #Dinosaur Fishes, #gallery, #Identity, #Iziko South African Museum, #Nation-state Identity, #pwigo

27 November 2016

Dirk Eddelbuettel: anytime 0.1.1: More robust

CRAN just accepted the newest release 0.1.1 of anytime, following the previous five releases since September. anytime is a very focussed package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, ... format to POSIXct (or Date) objects -- and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples, or just consider the following illustration:
R> library(anytime)
R> anytime("20161107 202122")   ## all digits
[1] "2016-11-07 20:21:22 CST"
R> utctime("2016Nov07 202122")  ## UTC parse example
[1] "2016-11-07 14:21:22 CST"
R> 
Release 0.1.1 robustifies two aspects. The 'digits only' input above extends what Boost Date_Time can parse and relies on simple-enough pre-processing. This operation is now more robust. We also ensure that input already of class Date is simply passed through by anydate() or utcdate(). Last but not least we added code coverage support, which oh-so-predictably lead us to game this metric to reach the elusive 100% coverage. The NEWS file summarises the release:

Changes in anytime version 0.1.1 (2016-11-27)
  • Both anydate() and utcdate() no longer attempt to convert an input value that is already of type Date.
  • The string splitter (needed for the 'all-digits' formats extending Boost Date_time) is now more defensive about the input argument and more robust. Thanks to Bob Jansen for the heads-up (PR #30 closing issue #29).
  • Code coverage reporting has been added (PR #31).

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

24 November 2016

Dirk Eddelbuettel: RcppExamples 0.1.8

A new version of the RcppExamples package is now on CRAN. The RcppExamples package provides a handful of short examples detailing by concrete working examples how to set up basic R data structures in C++. This version takes advantage of the updated date and datetime classes in Rcpp 0.12.8 (which are optional for now and being phased in while we deprecate the old ones). A NEWS extract follows:

Changes in RcppExamples version 0.1.8 (2016-11-24)
  • Updated DateExample to show vector addition available under Rcpp 0.12.8 when the (currently still phased in and optional) new Date(time) classes are used via the define in src/Makevars,.win; with fallback code for older versions
  • Other minor edits to DESCRIPTION and README.md

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

21 November 2016

Reproducible builds folks: Reproducible Builds: week 82 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday November 13 and Saturday November 19 2016: Media coverage Elsewhere in Debian Documentation update Packages reviewed and fixed, and bugs filed Reviews of unreproducible packages 43 package reviews have been added, 4 have been updated and 12 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated: 4 issue types have been added: Weekly QA work During our reproducibility testing, some FTBFS bugs have been detected and reported by: strip-nondeterminism development disorderfs development debrebuild development debrebuild is new tool proposed by HW42 and josch (see #774415: "From srebuild sbuild-wrapper to debrebuild"). debrepatch development debrepatch is a set of scripts that we're currently developing to make it easier to track unapplied patches. We have a lot of those and we're not always sure if they still work. The plan is to set up jobs to automatically apply old reproducibility patches to newer versions of packages and notify the right people if they don't apply and/or no longer make the package reproducible. debpatch is a component of debrepatch that applies debdiffs to Debian source packages. In other words, it is to debdiff(1) what patch(1) is to diff(1). It is a general tool that is not specific to Reproducible Builds. This week, Ximin Luo worked on making it more "production-ready" and will soon submit it for inclusion in devscripts. reprotest development Ximin Luo significantly improved reprotest, adding presets and auto-detection of which preset to use. One can now run e.g. reprotest auto . or reprotest auto $pkg_$ver.dsc instead of the long command lines that were needed before. He also made it easier to set up build dependencies inside the virtual server and made it possible to specify pre-build dependencies that reprotest itself needs to set up the variations. Previously one had to manually edit the virtual server to do that, which was not very usable to humans without an in-depth knowledge of the building process. These changes will be tested some more and then released in the near future as reprotest 0.4. tests.reproducible-builds.org Misc. This week's edition was written by Chris Lamb, Holger Levsen, Ximin Luo and reviewed by a bunch of Reproducible Builds folks on IRC.

19 November 2016

Dirk Eddelbuettel: BH 1.62.0-1

The BH package on CRAN was updated to version 1.62.0. BH provides a large part of the Boost C++ libraries as a set of template headers for use by R, possibly with Rcpp as well as other packages. This release upgrades the version of Boost to the upstream version Boost 1.62.0, and adds three new libraries as shown in the brief summary of changes from the NEWS file which follows below. Special thanks to Kurt Hornik and Duncan Murdoch for help tracking down one abort() call which was seeping into R package builds, and then (re-)testing the proposed fix. We are now modifying one more file ever so slightly to use ::Rf_error(...) instead.

Changes in version 1.62.0-1 (2016-11-15)
  • Upgraded to Boost 1.62 installed directly from upstream source
  • Added Boost property_tree as requested in #29 by Aydin Demircioglu
  • Added Boost scope_exit as requested in #30 by Kirill Mueller
  • Added Boost atomic which we had informally added since 1.58.0

Courtesy of CRANberries, there is also a diffstat report for the most recent release. Comments and suggestions are welcome via the mailing list or the issue tracker at the GitHubGitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

18 November 2016

Dirk Eddelbuettel: Rcpp 0.12.8: And more goodies

Yesterday the eighth update in the 0.12.* series of Rcpp made it to the CRAN network for GNU R where the Windows binary has by now been generated too; the Debian package is on its way as well. This 0.12.8 release follows the 0.12.0 release from late July, the 0.12.1 release in September, the 0.12.2 release in November, the 0.12.3 release in January, the 0.12.4 release in March, the 0.12.5 release in May, the 0.12.6 release in July, and the 0.12.7 release in September --- making it the twelveth release at the steady bi-montly release frequency. While we are keeping with the pattern, we have managed to include quite a lot of nice stuff in this release. None of it is a major feauture, though, and so we have not increased the middle number. Among the changes this release are (once again) much improved exception handling (thanks chiefly to Jim Hester), better large vector support (by Qiang), a number of Sugar extensions (mostly Nathan, Qiang and Dan) and beginnings of new DateVector and DatetimeVectors classes, and other changes detailed below. We plan to properly phase in the new date(time) classes. For now, you have to use a #define such as this one in Rcpp.h which remains commented-out for now. We plan to switch this on as the new default no earlier than twelve months from now. Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 843 packages on CRAN depend on Rcpp for making analytical code go faster and further. That is up by eightyfour packages, or a full ten percent, just since the last release in early September! Again, we are lucky to have such a large group of contributors. Among them, we have invited Nathan Russell to the Rcpp Core team given his consistently excellent pull requests (as well as many outstanding Stackoverflow answers for Rcpp). More details on changes are below.

Changes in Rcpp version 0.12.8 (2016-11-16)
  • Changes in Rcpp API:
    • String and vector elements now use extended R_xlen_t indices (Qiang in PR #560)
    • Hashing functions now return unsigned int (Qiang in PR #561)
    • Added static methods eye(), ones(), and zeros() for select matrix types (Nathan Russell in PR #569)
    • The exception call stack is again correctly reported; print methods and tests added too (Jim Hester in PR #582 fixing #579)
    • Variatic macros no longer use a GNU extensions (Nathan in PR #575)
    • Hash index functions were standardized on returning unsigned integers (Also PR #575)
  • Changes in Rcpp Sugar:
    • Added new Sugar functions rowSums(), colSums(), rowMeans(), colMeans() (PR #551 by Nathan Russell fixing #549)
    • Range Sugar now used R_xlen_t type for start/end (PR #568 by Qiang Kou)
    • Defining RCPP_NO_SUGAR no longer breaks the build. (PR #585 by Daniel C. Dillon)
  • Changes in Rcpp unit tests
    • A test for expression vectors was corrected.
    • The constructor test for datetime vectors reflects the new classes which treats Inf correctly (and still as a non-finite value)
  • Changes in Rcpp Attributes
    • An 'empty' return was corrected (PR #589 fixing issue #588, and with thanks to Duncan Murdoch for the heads-up)
  • Updated Date and Datetime vector classes:
    • The DateVector and DatetimeVector classes were renamed with a prefix old; they are currently typedef'ed to the existing name (#557)
    • New variants newDateVector and newDatetimeVector were added based on NumericVector (also #557, #577, #581, #587)
    • By defining RCPP_NEW_DATE_DATETIME_VECTORS the new classes can activated. We intend to make the new classes the default no sooner than twelve months from this release.
    • The capabilities() function can also be used for presence of this feature

Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

16 November 2016

Shirish Agarwal: The long tail in a common s man journey to debconf16 2

This is an extension of part 1 which I shared few days ago. This would be a longish one so please bear. First of all somebody emailed me this link so in the future a layover at Doha Airport will be a bit expensive from before, approx INR 700/- added to the ticket costs Moving on, Let me share an experience I shared one of the last few days I had while I was in Cape Town
Singer singing some great oldies from 60's , 70's till 90's.

Singer singing some great oldies from 60 s , 70 s till 90 s.

I had booked a place near Long Street, Cape Town using Bernelle s help. What I had not known at that time that near Long Street there are free walking tours every couple of hours. I took part in all the tours and those were nice experiences. Where they start the walk, there was the gentleman pictured above. I was amazed by this gentleman s rich voice. He strummed lot of classics from the 60 s, 70 s till the 90 s . I had two coffees and thought I was at a premium rock concert. It was a bitter-sweet experience for me because I could see that he has such prodigious talent and still he had to struggle to survive to make ends meet. I did my 2 bit but wish I could have done something more. Side note Before I forget there is one trick of feh which I use to view images without it getting very high-resolution (especially on my low-end systems) [shirish@debian] - [/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C006%5D/Card/DCIM/Camera] - [4621]
[$] feh -g 1350x1000 .
This actually makes it far far easier to traverse through the 1000 odd photos of the trip that I have in my personal archive without doing any sort of conversion methodology. Btw, it took me time but finally was able to create an album at gallery.debconf.org . Haven t been able to upload photos as came across an error which I have shared at https://lists.debconf.org/lurker/message/20161113.215659.fce58823.en.html Moving on, here s the funny story/experience I wanted to share could have been arrested ;) What happened was this. This is from the Doha Airport. I had seen big buggies (ones similar to golf carts) which was ferrying people from end of the concourse to the other. I had been walking the whole day and even with the horizontal escalators and everything, it takes a toll. I was half-tired, half-sleepy and saw a buggy stationed. From behind it looked like the buggies I had seen. As there was no place to park my behind there, I entered into the buggy and sat there. Around 15-20 minutes later a Doha cop in another buggy came to me and asked me if something had happened ? I had no clue what he was talking about. He told/shared/asked me in friendly tone whether I had committed a crime or wanted to report a crime. When I replied in negative to both, he asked then why I was sitting there. I replied it was for stretching my legs and it was the buggy which was being used to transport people from A. to B. He gently told me I had entered into the wrong one and it was actually a cop buggy. I couldn t believe it. He did go his own way as he saw I was dead-tired. After 10-15 minutes, half-believingly I came out of the buggy and to my shock the gentleman was right. There was nothing to do but solder on to find a spot in this big airport. I shared this with few friends and family and managed to elicit few laughs hence sharing. The somewhat sad one was I had met a couple with a baby. Now as shared before, Most Airports including the Doha Airport is Air-Conditioned/Climate-Controlled and is probably in mid-20 s so it was more than cold for me. The couples with the baby were from Asian sub-continent. From their clothes and the way they were, they were not very well off. I do remember them sharing that they had a death in the family and hence were going. I didn t know at that point in time that there was something called bereavement fares and if they were able to take opportunity of those tickets. But this is besides the point . The issue was that their baby had been running a high-fever and the A/C was making matters worse. I had seen a pharmacy but no clinic in the airport. It was much later I came across http://dohahamadairport.com/airport-guide/facilities-services/medical-emergencies but as can be seen on the web-page it doesn t tell whether the services are chargeable or not. I assume it would be paid, although in some of the developed/industrialized countries it is rumoured not to be for simple ailments such as the baby was going through. Have no idea if that s true or not. I also don t know how it equates with travel insurance as well as most travel insurance is also supposed to help you in situations like these. I was concerned as it was a baby and babies as all know are very very fragile. If anybody has an idea or had similar experience would like to know specifically related to International Airport environment as it has transit issues unlike in domestic airports where I don t think it would be a bit more easy. Now coming to my own inadequacies/lack of foresight which I had mentioned I will share, I had asked/queried and got to lead a Debian-installation workshop on the Open Debian Day. I had done a few earlier and had installed it a few times on my system and for my friends, relatives and some clients. The only bad experiences I had were to do with UEFI but even those in the jessie releases had got resolved quite a bit, so was pretty confident. The day before the Installfest was to happen, Mensah Nyarko Yaa Dufie (one full name) of Ghana approached me to install Debian on her system. I had some older version of the Debian DVD either 8.1 or 8.3 and had known that 8.5 had been released just a few days back. Had seen pretty fast internet (as far as downloading Debian DVD) is concerned hence asked her to wait a bit while I downloaded the newest image. I sha256summed it to make sure that the image was bit-to-bit perfect. Now I hadn t bought a pen drive/disk from India as I was under the impression that in such conferences, pen drives should not be an issue. I had asked Bernelle privately before via e-mail as well and she had assured me that some pen-drives would be available. She gave me a handful of HP pen drives. The pen drives as we came know during our usage were somewhat flaky. It would pop out/lose connection even with the slightest nudge to the lappy. Somehow I was able to transfer the image to the usb disk. As people say hindsight is 50:50 maybe it was not such a smart move on my part to download the big DVD image and maybe I should have got the netinstall iso . Be careful, the link I have just shared is of the old version, if you have good web link and want to try the newest stable netinstall head to cdimage.debian.org . Apart from that goof-up I dunno (still) of anyway to know if a copy from an .iso image to usb was successful or not and did it do correctly I did the following command sudo dd if=/path_to/debian-dvd.iso of=/dev/usb-mount-point which is usually /dev/sdb on all of my systems . Her system was a brand new HP (don t remember the model details) which she had bought just a few weeks/months before debconf. We tried a few times but it failed at installing the boot-loader stage. I asked Ritesh Raj Saraff (a friend and DD) and while he had some ideas, none of them worked. Ritesh later pointed out Steve McIntyre and shared he is part of the Debian-Installer team. At that point in time, I had no clue who Steve McIntyre was otherwise I probably would not have approached him. He quickly acquiesced to my request and shared that he would be there for the workshop. With load of my mind little bit, I apologized to mensah and asked her to be at the workshop the following day. I had no clue what was wrong at this point in time, whether it was the iso image in the usb disk or a UEFI issue. This also wasn t good for my confidence but as somebody from the Debian-Installer team was there, I was somewhat relaxed. Next day, some more people came for the Installfest. While I had made 2-3 copies, clearly it was not enough as more people came. I was in a frenzy and asked Deven Bansod, Keerthana Krishnan, Prabaharan Jaminy (the whole GSOC and Outreachy attendees) to volunteer to help out in making more iso images on usb disks. I introduced mensah to Steve McIntyre and we tried 2-3 times to get debian installed on the system but it didn t move from the same place. Ritesh shared that dd had a memory leak and hence cat was a better way to do it. So we did

$ cat debian.iso > /dev/sdb
and soon other machines had debian sporting on their desktops. But mensa s lappy wouldn t get move from the boot-loader stage. Suddenly Steve had the bright idea (light bulb moment) that maybe the .iso is corrupted/usb disk is bad or something is incomplete. We started on another usb disk. Now this is where I have a query While I don t want to compare, in Ubuntu there was an image self-checking mechanism where probably behind the scenes (backend) the checksums published in a file are compared with checksums generated by apps. which are on the .iso image. While it does extend your time, the end result is you know if there is some issue on the decompressed image on the usb disk. AFAIK we don t have anything similar. The only two things I know is the wiki page and of course the various checksums of the image as shared at http://cdimage.debian.org/debian-cd/8.6.0/amd64/iso-cd/ or http://cdimage.debian.org/debian-cd/8.6.0/amd64/iso-dvd/ If anybody knows of any movement or a bug in the BTS which I can follow for the above issue please let me know. This time Steve was able to install it without any issues. I asked him whether he had to make some specific FAT/Ex-FAT/NTFS partitions as some new UEFI-based lappies need one or more but he replied in the negative. While mensa did get her debian install, the GUI didn t come while command-prompt was available. Then Steve added backports to the sources.list, got the new kernel, new Intel/Nvidia drivers (think it was one of those hybrid models IIRC) and she was able to boot into GNOME-Debian. I didn t saw any bug-reports about checksumming state of the applications before installation but did couple of reports about badblocks support and memory checking and from action on both bug-reports it is also need of the hour (although the earlier one has been marked as won t fix :(). In this whole thing, I liked/appreciated the way Steve handled things, I intuitively understood/knew that he wasn t just part of the Debian-installer team but someone better. I can t explain it but it was there. A little investigation in the evening and it turned out that he had been Debian Project Leader for two consecutive years (2008 and 2009) . In hindsight it probably was a good thing I didn t know that before otherwise I probably wouldn t have interacted with him and it would have been my loss. To have been the DPL and still being so humble while technically being so proficient, I was amazed and didn t know what to make of it. Here i.e. in India, if somebody wins even the mohalla elections (neighbourhood elections) the person carries a big chip on her/is shoulder not just till he is on the seat but even beyond, and here was an example of a previous DPL asking time from one of the developers in a video if it s possible in the next couple of days. Lastly,last week have able to report 2 bugs upstream. The first one is of youtube-dl . It s somewhat complicated hence will not go there atm. The second and more surprising one was from nano our esteemed text-editor- Hopefully the bug will be fixed once a new version comes.
Filed under: Miscellenous Tagged: #buggy, #cop, #Debconf16, #doha airport, #Installfest, #nano, #singer, #youtube-dl, travel

15 November 2016

Keith Packard: AltOS-Lisp

A Tiny Lisp for AltOS I took a bit of a diversion over the last week or so when I wondered how small a lisp interpreter I could write, and whether I could fit that into one of the processors that AltOS runs on. It turns out, you can write a tiny lisp interpreter that fits in about 25kB of ram with a 3kB heap for dynamic data. I decided to target our ChaosKey boards; they're tiny, and I've got a lot of them. That processor offers 28kB of usable flash space (after the 4kB boot loader) and 6kB of ram with the processor running at a steaming 48MHz. I'm not at all sure this is useful, but I always enjoy doing language implementations, and this one presented some 'interesting' challenges: Iterative Compacting Allocator I'm betting someone has built one of these before, but I couldn't find one, so I wrote my own. The basic strategy is to walk the heap to find a subset of the active objects which are allocated sequentially in memory with only unused storage between them. These objects are then compacted in-place, and then the heap is walked again to update all references to the moved objects. Then, the process is restarted to find another subset and move them. By looking for these subsets starting at the bottom of the heap, and working upwards towards the top, the whole heap can be compacted into a contiguous chunk at the bottom of memory. Allocation involves moving a pointer along at the top of active memory; when it gets to the top of the heap, collect and see if there's space now. As always, the hardest part was to make sure all active memory was tied down. The second hardest part was to make sure that all active pointers were updated after any allocation, in case a collect moved the underlying object. That was just bookkeeping, but did consume much of the development time. One additional trick was to terminate the recursion during heap walking by flagging active cons cell locations in a global bitmap and then walking that separately, iterating until that bitmap is empty. Nested lambdas form another recursion which should probably get the same approach, but I haven't done that yet. An unexpected "benefit" of the tiny heap is that the collector gets called a lot, so any referencing bugs will have a good chance of being uncovered in even a short program execution. ROM-able Lisp Instead of implementing all of the language in C, I wanted to be able to implement various pieces in Lisp itself. Because of the complex nature of the evaluation process, adding things like 'let' or even 'defun' turn out to be dramatically simpler in Lisp. However, I didn't want to consume bunches of precious RAM to hold these basic functions. What I did was to create two heaps, one in ROM and the other in RAM. References are be tagged as to which heap they're in. 16-bit Values Lisp programs use a pile of references. Using a full 32 bits for each one would mean having a lot less effective storage. So, instead, I use an offset from the base of the heap. The top bit of the offset is used to distinguish between the ROM heap and the RAM heap. I needed a place to store type information, so I settled on using the bottom two bits of the references. This allows for four direct type values. One of these values is used to indicate an indirect type, where the type is stored in the first byte of the object. The direct types are:
ValueType
0Cons cell
114-bit int
2String
3Other
With 2 tag bits, the allocator needs to work in 32-bit units as the references couldn't point to individual bytes. Finally, I wanted 0 to be nil, so I add four to the offsets within the heaps. The result is that the ROM and RAM heaps can each cover up to 32k - 4 bytes. Note that ints are not stored in the heap; instead they are immediate values stored in 14 bits, providing a range of -8192 to 8191. One can imagine wanting more range in ints at some point. Heap-based Evaluator A simple lisp implementation uses the fact that eval is re-entrant and do the operation on the C stack:
val eval(val exprs)  
    val vals;
    while (exprs)  
        vals = append(vals, eval(car(exprs)));
        exprs = exprs->cdr;
     
    return execute (car(vals), cdr(vals));
 
This makes things really simple and provides for a clean framework for implementing various bits of lisp, including control flow and macros. However, it rapidly consumes all of the available memory for a stack, while also requiring separate bookkeeping for the in-use memory in each frame. I replaced this design with one which keeps the lisp stack on the heap, and then performs eval with a state machine with all state stored in global variables so that the memory manager can reference them directly. Each eval operation is performed in a separate 'stack' context, which holds the entire eval state except for the current value, which lives in a separate global variable and is used to pass values out of one stack frame and into another. When the last stack context is finished, the evaluation terminates and the value is returned to the caller. There are nine states in the state machine, each of which is implemented in a separate function, making the state machine a simple matter of pulling the current state from the top of the stack and invoking the associated function:
while (ao_lisp_stack)  
    if (!(*evals[ao_lisp_stack->state])()   ao_lisp_exception)  
        ao_lisp_stack_clear();
        return AO_LISP_NIL;
     
 
return ao_lisp_v;
Because there's no C recursion involved, catching exceptions is a simple matter of one test at this level. Primitives like progn, while, cond and eval all take special magic in the state machine to handle; getting all of that working took several attempts before I found the simple loop shown above. Lexical Scoping The last time I did a lisp interpreter, I implemented dynamic scoping. Atoms were all global and had values associated directly with them. Evaluating a lambda started by saving all of the existing global values for the parameter atoms and then binding the new values. When finished, the previous values would be restored. This is almost correct, but provides surprising results for things like:
> (setq baz 1)
> (def foo (lambda (bar) (+ baz bar)))
> (def bletch (lambda (baz) (foo baz)))
> (bletch 2)
4
The value that foo gets for 'baz' is 2 instead of 1 under dynamic scoping, which most people find surprising. This time, I was determined to use lexical scoping, and it turned out to be surprisingly easy. The first trick was to separate the atoms from their 'value'; each atom can have a different value in different lexical scopes. So, each lexical scope gets a 'frame' object, those contain the value for each atom defined in that scope. There's a global scope which holds all of the globally defined values (like baz, foo and bletch above). Each frame points to its enclosing scope, so you can search upwards to find the right value. The second trick was to realize that the lexical scope of a lambda is the scope in which the lambda itself is evaluated, and that the evaluation of a lambda expression results in a 'function' object, which contains the lambda and its enclosing scope:
> (def foo (lambda (bar bletch)
       ((lambda (baz)
          (+ baz bar))
        bletch)))
> (foo 2 3)
5
In this case, the inner lambda in foo can 'see' the value of bar from the enclosing lambda. More subtly, even if the inner lambda were executed multiple times, it would see the same baz, and could even change it. This can be used to implement all kinds of craziness, including generators:
> (defun make-inc (add)
  ((lambda (base)
     (lambda ()
       (progn
     (setq base (+ base add))
     base)))
   0)
  )
> (setq plus2 (make-inc 2))
> (plus2)
2
> (plus2)
4
The current implementation of each frame is a simple array of atom/value pairs, with a reference to the parent frame to form the full scope. There are dramatically faster implementations of this same concept, but the goal here was small and simple. A Tiny Allocator Optimization With eval consuming heap space for stacks, frames and argument lists, the interpreter was spending a lot of time in the collector. As a simple optimization, I added some free lists for stack frames and cons cells. Stack frames are never referenced when they're finished, so they can always go on the free list. Cons cells used to construct argument lists for functions are usually free. Builtin functions have a bit which indicates whether they might hold on to a reference to the argument list. Interpreted lambdas can't get the list while nlambdas, lexprs and macros do. Each lambda execution creates a new frame, and while it would be possible to discover if that frame 'escapes' the lambda, I decided to not attempt to cache free ones yet. Save and Restore To make the lisp interpreter more useful in tiny computers, I added the ability to save and restore the entire heap to flash. This requires leaving enough space in the flash to preserve the heap, further constraining the amount of flash available for the application. Code All of this code is in the 'lisp' branch of my AltOS repository: AltOS The lisp interpreter is independent from the rest of AltOS and could be re-purposed for another embedded operating system. It runs fine on ChaosKey hardware, and also on the STM32F042 Nucleo-32 board There's also a test framework which runs on Linux, and is how I developed almost all of the code. That's in the src/test directory in the above repository, and is called 'ao_lisp_test'. Towers of Hanoi Here's an implementation of the classic recursive Towers of Hanoi game; it shows most of the current features of the language.
;
; Towers of Hanoi
;
; Copyright   2016 Keith Packard <keithp@keithp.com>
;
; This program is free software; you can redistribute it and/or modify
; it under the terms of the GNU General Public License as published by
; the Free Software Foundation, either version 2 of the License, or
; (at your option) any later version.
;
; This program is distributed in the hope that it will be useful, but
; WITHOUT ANY WARRANTY; without even the implied warranty of
; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
; General Public License for more details.
;
; ANSI control sequences
(defun move-to (col row)
  (patom "\033[" row ";" col "H" nil)
  )
(defun clear ()
  (patom "\033[2J" nil)
  )
(defun display-string (x y str)
  (move-to x y)
  (patom str)
  )
; Here's the pieces to display
(setq stack '("*" "**" "***" "****" "*****" "******" "*******"))
(setq top (+ (length stack) 3))
;
; Here's all of the stacks of pieces
; This is generated when the program is run
;
(setq stacks nil)
; Display one stack, clearing any
; space above it
(defun display-stack (x y clear stack)
  (cond ((= 0 clear)
     (cond (stack (progn
            (display-string x y (car stack))
            (display-stack x (1+ y) 0 (cdr stack))
            )
              )
           )
     )
    (t (progn
         (display-string x y "          ")
         (display-stack x (1+ y) (1- clear) stack)
         )
       )
    )
  )
; This should probably be included in the rom image...
(defun length (list)
  (cond (list (1+ (length (cdr list))))
    (0)
    )
  )
; Position of the top of the stack on the screen
; Shorter stacks start further down the screen
(defun stack-pos (y stack)
  (- y (length stack))
  )
; Display all of the stacks, spaced 20 columns apart
(defun display-stacks (x y stacks)
  (cond (stacks (progn
          (display-stack x 0 (stack-pos y (car stacks)) (car stacks))
          (display-stacks (+ x 20) y (cdr stacks)))
        )
    )
  )
; Display all of the stacks, then move the cursor
; out of the way and flush the output
(defun display ()
  (display-stacks 0 top stacks)
  (move-to 1 21)
  (flush)
  )
; Reset stacks to the starting state, with
; all of the pieces in the first stack and the
; other two empty
(defun reset-stacks ()
  (setq stacks (list stack nil nil))
  (length stack)
  )
; more functions which could usefully
; be in the rom image
(defun min (a b)
  (cond ((< a b) a)
    (b)
    )
  )
(defun nth (list n)
  (cond ((= n 0) (car list))
    ((nth (cdr list) (1- n)))
    )
  )
; Replace a stack in the list of stacks
; with a new value
(defun replace (list pos member)
  (cond ((= pos 0) (cons member (cdr list)))
    ((cons (car list) (replace (cdr list) (1- pos) member)))
    )
  )
; Move a piece from the top of one stack
; to the top of another
(defun move-piece (from to)
  (let ((from-stack (nth stacks from))
    (to-stack (nth stacks to))
    (piece (car from-stack)))
    (setq from-stack (cdr from-stack))
    (setq to-stack (cons piece to-stack))
    (setq stacks (replace stacks from from-stack))
    (setq stacks (replace stacks to to-stack))
    (display)
    (delay 100)
    )
  )
; The implementation of the game
(defun _hanoi (n from to use)
  (cond ((= 1 n)
     (progn
      (move-piece from to)
      nil)
     )
    (t
     (progn
      (_hanoi (1- n) from use to)
      (_hanoi 1 from to use)
      (_hanoi (1- n) use to from)
      )
     )
    )
  )
; A pretty interface which
; resets the state of the game,
; clears the screen and runs
; the program
(defun hanoi ()
  (setq len (reset-stacks))
  (clear)
  (_hanoi len 0 1 2)
  )

13 November 2016

Dirk Eddelbuettel: pkgKitten 0.1.4: Creating R Packages that purr

kitten A new minor release 0.1.4 of pkgKitten just hit on CRAN this morning. One small change is that the package manual page it creates now makes use of the (still new-ish and under-documented and under-used) Rd macros described at the end of Section 2.13 of Writing R Extensions:
See the system.Rd file in share/Rd/macros for more details and macro definitions, including macros \packageTitle, \packageDescription, \packageAuthor, \packageMaintainer, \packageDESCRIPTION and \packageIndices.
By using these macros, and referencing them from the DESCRIPTION file, we can avoid redundancy, or worse, inconsitency, between both files. Or just be plain lazy and describe things just once in the higher-level file: A good thing! Otherwise we fixed a URL to the manual thanks to a PR, and just added some of the regular polish to some of the corners which R CMD check --as-cran is looking into:

Changes in version 0.1.4 (2016-11-13)
  • Utilize newer R macros in package-default manual page.
  • Repair a link to Wrting R Extensions (PR #7 by Josh O'Brien)

More details about the package are at the pkgKitten webpage and the pkgKitten GitHub repo. Courtesy of CRANberries, there is also a diffstat report for this release

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

11 November 2016

Jonathan Dowland: Vinyl is killing Vinyl (but that's ok)

I started buying vinyl records about 16 years ago, but recently I've become a bit uncomfortable being identified as a "vinyl lover". The market is ascendant, with vinyl album sales growing for 8 consecutive years, at least in the UK. So why am I uncomfortable about it? A quick word about audio fidelity/quality here. I don't subscribe to the school of thought that audio on vinyl is inherently better than digital audio, far from it. I'm aware of its limitations. For recordings that I love, I try to seek out the best quality version available, which is almost always digital. Some believe that vinyl is immune to the "loudness war" brickwall mastering plaguing some modern releases, but for some of the worst offenders (Depeche Mode's Playing The Angel; Red Hot Chili Pepper's Californication) I haven't found the vinyl masterings to sound any different. 16 years ago Let's go back to why I started buying vinyl. Back when I started, the world was a very different place to what it is today. You could not buy most music in a digital form: it was 3 more years before the iTunes Store was opened, and it was Mac-only at first, and the music it sold was DRM-crippled for the first 5 or so years afterwards. The iPod had not been invented yet and there was no real market for personal music players. Minidiscs were still around, but Net-MD (the only sanctioned way to get digital music onto them from a computer) was terrible.
old-ish LPs old-ish LPs
Buying vinyl 16 years ago was a way to access music that was otherwise much harder to reach. There were still plenty of albums, originally recorded and released before CDs, which either had not been re-issued digitally at all, or had been done so early, and badly. Since vinyl was not fashionable, the second hand market was pretty cheap. I bought quite a lot of stuff for pennies at markets and car boot sales. Some music such as b-sides and 12" mixes and other mixes prepared especially for the format remains unavailable and uncollected on CD. (I'm a big fan of the B-side culture that existed prior to CDs. I might write more about that one day.) 10 years ago
modern-ish 7 inches modern-ish 7 inches
Fast forward to around 10 years ago. Ephemeral digital music is now much more common, the iPod and PMPs are well established. High-street music stores start to close down, including large chains like MOS, Our Price, and Virgin. Streaming hasn't particularly taken off yet, attempts to set up digital radio stations are fought by the large copyright owners. Vinyl is still not particularly fashionable, but it is still being produced, in particular for singles for up-and-coming bands in 7" format. You can buy a 7" single for between 1 and 4, getting the b-side with it. The b-side is often exclusive to the 7" release as an incentive to collectors. I was very prepared to punt 1-2 on a single from a group I was not particularly familiar with just to see what they were like. I discovered quite a lot of artists this way. One of the songs we played at our wedding was such an exclusive: a recording of the Zutons' covering Jackie Wilson's "Higher and Higher", originally broadcast once on Colin Murray's Evening Session radio show. Now
Vangelis - Blade Runner OST An indulgence
So, where are we now? Vinyl album sales are a huge growth market. They are very fashionable. Many purchasers are younger people who are new to the format; it's believed many don't have the means to play the music on the discs. Many (most?) albums are now issued as 12" vinyl in parallel with digital releases. These are usually exactly the same product (track listing, mixes, etc.) and usually priced at exactly twice that of the CD (with digital prices normally a fraction under that). The second hand market for 12" albums has inflated enormously. Gone are the bargains that could be had, a typical second hand LP is now priced quite close to the digital price for a popular/common album in most places. The popularity of vinyl has caused a huge inflation in the price of most 7" singles, which average somewhere between 8- 10 each, often without any b-side whatsoever. The good news is from my observations the 2nd hand market for 7" singles hasn't been affected quite as much. I guess they are not as desirable to buyers. The less said about Record Store Day, the better. So, that's all quite frustrating. But most of the reasons I used to buy vinyl have gone away anyway. Many of the rushed-to-market CD masterings have been reworked and reissued, correcting the earlier problems. B-side compilations are much more common so there are far fewer obscure tracks or mixes, and when the transfer has been done right, you're getting those previously-obscure tracks in a much higher quality. Several businesses exist to sell 2nd hand CDs for rock bottom prices, so it's still possible to get popular music very cheaply. The next thing to worry about is probably streaming services.

8 November 2016

Dirk Eddelbuettel: anytime 0.1.0: New features, some fixes

A new release of anytime is now on CRAN following the four releases in September and October. anytime aims to convert anything in integer, numeric, character, factor, ordered, ... format to POSIXct (or Date) objects -- and does so without requiring a format string. See the anytime page for a few examples. Release 0.1.0 adds several new features. New functions utctime() and utcdate() parse to coordinated universal time (UTC). Several new formats were added. Digit-only formats like 'YYYYMMDD' with or without 'HHMMSS' (or even with fractional secodns 'HHMMSS.ffffff') are supported more thoroughly. Some examples:
R> library(anytime)
R> anytime("20161107 202122")   ## all digits
[1] "2016-11-07 20:21:22 CST"
R> utctime("2016Nov07 202122")  ## UTC parse example
[1] "2016-11-07 14:21:22 CST"
R> 
The NEWS file summarises the release:

Changes in anytime version 0.1.0 (2016-11-06)
  • New functions utctime() and utcdate() were added to parse input as coordinated universal time; the functionality is also available in anytime() and anydate() via a new argument asUTC (PR #22)
  • New (date)time format for RFC822-alike dates, and expanded existing datetime formats to all support fractional seconds (PR #21)
  • Extended functionality to support not only YYYYMMDD (without a separator, and not covered by Boost) but also with HHMM , HHMMSS and HHMMSS.ffffff (PR #30 fixing issue #29)
  • Extended functionality to support HHMMSS[.ffffff] following other date formats.
  • Documentation and tests have been expanded; typos corrected
  • New (unexported) helper functions setTZ, testOutput, setDebug
  • The testFormat (and testOutput) functions cannot be called under RStudio (PR #27 fixing issue #25).
  • More robust support for non-finite values such as NA, NaN or Inf (Fixing issue #16)

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dirk Eddelbuettel: gettz 0.0.3

A minor release 0.0.3 of gettz arrived on CRAN two days ago. gettz provides a possible fallback in situations where Sys.timezone() fails to determine the system timezone. That can happen when e.g. the file /etc/localtime somehow is not a link into the corresponding file with zoneinfo data in, say, /usr/share/zoneinfo. This release adds a second #ifdef to permit builds on Windows for the previous R release (ie r-oldrel-windows). No new code, or new features. Courtesy of CRANberries, there is a comparison to the previous release. More information is on the gettz page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.