Search Results: "will"

29 April 2017

Antoine Beaupr : My free software activities, April 2017

Debian Long Term Support (LTS) This is my monthly Debian LTS report. My time this month was spent working on various hairy security issues, most notably XBMC (now known as Kodi) and yaml-cpp.

Kodi directory transversal I started by looking in CVE-2017-5982, a "directory traversal" vulnerability in XBMC (now known as Kodi) which is a technical term for "allow attackers to read any world-readable file on your computer from the network". It's a serious vulnerability which has no known fix. When you enable the "remote control" interface in Kodi, it allows anyone with the password (which is disabled by default) to download any files Kodi has read access to on the machine it's running. Considering Kodi is often connected to multiple services, this may mean elevated compromise and more nasty stuff. I furthered the investigation done with my own analysis which showed the problem is difficult to solve: Kodi internally uses the facility to show thumbnails and media to the user, and there are no clear way of restricting which paths Kodi should have access to. Indeed, Kodi is designed to access mounted file systems and paths in arbitrary locations. In Debian bug #855225, I further confirmed confirmed wheezy and jessie-backports as vulnerable and therefore showed with good certainty that stretch and sid are vulnerable as well. I also suggested possible workaround, but at this point, it's in upstream's hands, as the changes will be intrusive. The file transfer mechanism need to be revamped all over Kodi, or authentication (with a proper password policy), need to be enforced.

Squirrelmail Next I looked at that old webmail software, Squirrelmail, which suffers from a remote code execution vulnerability (CVE-2017-7692) when sending mails with sendmail on the commandline. This is arguably an edge case, but considering the patch was simple, I figured I would provide an update to the LTS community. I tried to get a coordinated release for jessie, since the code is the same, but this wasn't completed at the time of writing. A patch is available and will hopefully be picked up by another LTS worker soon.

Fop and Batik Those issues (CVE-2017-5661 and CVE-2017-5662) were more difficult. The patches weren't clearly documented and there were no upstream references other than security advisories for the first release in years (in the case of batik) or months (in the case of fop), which made it hard to track down the issues. Fortunately, I was able to track down the upstream issues (FOP-2668 and BATIK-1139) where I got confirmation on what the proper fixes were. I could then release DLA-927-1 and DLA-926-1 with the backported patches. I do not use fop or batik. In fact, even after reading the homepage of both products, I couldn't quite figure out what use people could possibly have for that thing. Before uploading the packages, I therefore made packages available for testing for fop and batik.

libsndfile Next up was libsndfile which a bunch of overflows when parsing various audio files. I backported a patch for CVE-2017-7585 CVE-2017-7586 and CVE-2017-7741 which all seemed to be fixed by a single patch usptream. CVE-2017-7742 was also fixed, although with a separate patch. In all of those, i could only test CVE-2017-7741 and CVE-2017-7742, as the others were missing test cases. I provided a test package for a few days then I also figured it would be best to incorporate the security fixes done in stable, which brought in fixes for CVE-2015-7805, CVE-2014-9756 and CVE-2014-9496. So in the end, I ported patches from wheezy to jessie and uploaded the jessie version (reverting certain build changes) into wheezy and uploaded DLA-928-1 with the results.

yaml-cpp I then turned to yaml-cpp, a C++ parser for YAML. This one didn't have a known upstream fix, but I figured I would give it a shot anyways. I ended up writing my first C++ code in years which is still pending review and merge upstream. It's not an easy problem to fix: this is basically an excessive recursion problem that can be used to smash the stack. I figured I could introduce a recursion limit, but as the discussion showed, this is a limited approach: stack size varies on different platforms and it's not easy to find the right limit. The real solution is to rewrite the code to avoid recursion but that's a major code refactoring I didn't feel belong in a LTS update. Besides, this could be better handled by upstream, so I will leave things at that for now. It does make you wonder how much code out there is recursing on untrusted data structures...

kedpm Finally, a friend over at Koumbit.org reported Debian bug #860817, as information leak in kedpm, a password manager I previously maintained. I requested and got assigned CVE-2017-8296 and provided a fix for wheezy and jessie. For unstable and the coming stretch release, I have requested kedpm to be completely removed from Debian (Debian bug #860817) which involved a release notes update (Debian bug #861277). It's unfortunate to see software go, but kedpm wasn't maintained. I wasn't the original author: I just gave a few patches and ended up maintaining that software and not using it. It's a bad situation to be in, as you don't really know what's working and not with the tools you are supposed to be responsible for. There are more modern alternatives available now and I encourage everyone to switch.

Triage Looking for more work, I peeked a bit in the secretary tasks to triage some pending issues. I found that trafficserver could be crashed with simple requests (CVE-2017-5659) so I looked into that issue. My analysis showed that the patch is long and complex and could be difficult to backport to the old version available in wheezy. I also couldn't reproduce the issue in wheezy, so it may be a bug introduced only later, although I couldn't confirm that directly. I also triaged wireshark, where I just noted the maintainer expressed concern that we were taking up issues too fast and will probably take care of this one. I also postponed various issues in GnuTLS (marked "no-dsa") as they affect only a (unfortunately) rarely used part of GnuTLS that has been removed in later version: OpenPGP support.

Other free software work

Debiman I finally got around contributing to the debiman project. I worked on ensuring that there is a dman compatibility in debiman, by shipping dman in the debian-goodies package (Debian bug #860920). I also submitted a pull request to fix the fix about page title, document the custom assets repository, fix a stray bracket and link to the link to venerable man7.org project After a discussion on IRC, I also filed a few more issues: I'm happy to be able to contribute to this important service and I hope the new FAQ I created will be online soon!

XMonad and Emacs I started using writeroom-mode again as part of my work on LWN. As it turns out, my setup was not exactly working: I had to port my config to the new version and windows weren't "sticky" as they should be, a known issue with Xmonad. Indeed, Xmonad doesn't obey the "static" or "all desktops" standard directives. Writeroom is a "distraction-free writing" mode for Emacs, so the irony of working on such a deep distraction in establishing a distraction free environment is not lost on me. Needing to scratch that particular itch, and with the help of clever people from the IRC channel, I was able to make Emacs tell Xmonad to show its window (or "frame" as Emacs likes to call it) on all desktops. This involved creating a new function which I think could be useful in the CopyWindow library: 
--   Toggle between "copyToAll" or "killAllOtherCopies". Copies to all
-- workspaces, or remove from all other workspaces, depending on
-- previous state (checked with "wsContainingCopies").
copyToAllToggle :: X ()
copyToAllToggle = do
    -- check which workspaces have copies
    copies <- wsContainingCopies
    if null copies
      then windows copyToAll -- no workspaces, make sticky
      else killAllOtherCopies -- already other workspaces, unstick
There are probably better ways of implementing this directly in the CopyWindow code - wsContainingCopies, in particular, is probably overkill. But it's all I can use directly from my xmonad.hs, so that's what I did. The other bit I needed was something to trigger that function from the outside. I rejected the ServerMode hook because it looked a bit too complicated and there is a built-in facility within X that works without this, which, from Emacs' point of view, is the x-send-client-message function. So I made up a new message identifier and wrote a event hook handler to process it: 
--   handle X client messages that tell Xmonad to make a window appear
-- on all workspaces
--
-- this should really be using _NET_WM_STATE and
-- _NET_WM_STATE_STICKY. but that's more complicated: then we'd need
-- to inspect a window and figure out the current state and act
-- accordingly. I am not good enough with Xmonad to figure out that
-- part yet.
--
-- Instead, just check for the relevant message and check if the
-- focused window is already on all workspaces and toggle based on
-- that.
--
-- this is designed to interoperate with Emacs's writeroom-mode module
-- and called be called from elisp with:
--
-- (x-send-client-message nil 0 nil "XMONAD_COPY_ALL_SELF" 8 '(0))
myClientMessageEventHook :: Event -> X All
myClientMessageEventHook (ClientMessageEvent  ev_message_type = mt, ev_data = dt ) = do
  dpy <- asks display
  -- the client message we're expecting
  copyAllMsg <- io $ internAtom dpy "XMONAD_COPY_ALL_SELF" False
  -- if the event matches the message we expect, toggle sticky state
  when (mt == copyAllMsg && dt /= []) $ do
    copyToAllToggle
  -- we processed the event completely
  return $ All True
All that was left was to hook that into Emacs, and I was done! Whoohoo! Full screen total domination, distraction free work! :) I would love to hear from others what they think of that approach, if they have improvements or if the above copyToAllToggle function could be merged in. Ideally, Xmonad would just parse the STICKY client messages and do the right thing - maybe even directly in CopyWindow - but I have found this enough Haskell for one day. You can see the diff on my home directory to see exactly the changes involved to make this configuration work.

Emacs packaging Speaking of Emacs, after complaining in the noisy #emacs IRC channel about the poor TLS configuration of marmelade.org -- and filing a bug (Debian bug #861106) regarding the use of SHA-1 in certificate pinning -- I was told we shouldn't expect trust from third-party ELPA repositories. Marmelade seems to be dead, as the maintainer is "behind the great firewall of China" and MELPA still hasn't figured out how to sign packages. In the end, it seems like there are tons of elpa packages in Debian and that if your favorite one is missing, that's a bug that can be filed and fixed. I first discovered that 6 of the packages I used were already packaged: And so I went ahead and filed a ton more bugs for the packages I am using but that aren't in Debian just yet: Of those, I can't recommend multiple-cursors (MC) enough: I used it at least 4 times just writing this text. It's just awesome. The other ones are also all great in their own right of course, but I feel they are more specific to my workflow whereas MC is just amazing.

ikiwiki I did some more work on ikiwiki the software driving this blog. I created a new plugin to, at least, fix anchors in the table of contents to be human readable. This is something I had done on the MoinMoin wiki almost a decade ago -- which I called then NicerHeadingIds and that I have always found frustrating with Ikiwiki. It turns out the problem was both easier and hairier than I thought. Right from the start, something weird was happening: something was already adding nice headings, but they were somewhat broken. It turns out that multimarkdown already inserts those headers, but I wasn't satisfied with the way they were generated. But even worse, I had the headinganchors plugin enabled, but that plugin wasn't taking effect, because of multimarkdown. And even if it would take effect, it doesn't behave well with non-ASCII characters, which gets turned in their numeric presentation. So I also wrote the i18nheadinganchors plugin that creates better headings and patched the toc plugin so that it can reuse existing anchors if they exist, while keeping backwards compatibility. I hope this gets merged in a future ikiwiki release so I do not have to carry this patch locally too long... In other news, I have upgraded the ikiwiki-hosting package to the latest version and sent a patch upstream to provide HSTS support.

Other stuff I have migrated all my public repos hosted on my home server to either Gitlab.com or Github. I also have repositories on 0xacab and it seemed ludicrous to have 4 different, canonical, places where my code was hosted. I have now about 40 different projects on Gitlab and about 60 on Github, although most of the latter are forks of existing projects. I also made a manpage for stressant and moved the documentation to RTFD which makes it neatly accessible. I also made small incremental improvements (like --directory support). I installed Rainloop on this server to give a nice, mobile-friendly webmail. Instructions to replicate this setup are in mail. In the constant git-annex documentation effort, I tried to draft a user guide that could be a basis for restructuring the documentation to be more easily accessible. I also helped a friend put his documentation on the wiki in splitting a repository. Finally, I also looked into Android stuff a little more. I wrote a usability review of the F-Droid privileged extension that will bring good changes, I hope. I also opened the discussion regarding reproducible builds to try and clarify exactly how those worked to help the Wallabag people ship consistently signed alphas. So far, it seems that it will remain a standard practice on F-Droid to ship packages that are not signed by the official upstream signature, unfortunately, unless upstream provides a reproducible build that is publicly available... Switching to such build is also a hairy issue, as, obviously, the signature changes, which raises the alarm we are trying to avoid in the first place.

John Goerzen: Is there any way to truly secure Docker container contents?

There is much to like about Docker. Much has been written about it, and about how secure the containerization is. This post isn t about that. This is about keeping what s inside each container secure. I believe we have a fundamental problem here. Earlier this month, a study on security vulnerabilities on Docker Hub came out, and the picture isn t pretty. One key finding:
Over 80% of the :latest versions of official images contained at least on high severity vulnerability!
And it s not the only one raising questions. Let s dive in and see how we got here. It s hard to be secure, but Debian makes it easier Let s say you want to run a PHP application like WordPress under Apache. Here are the things you need to keep secure: On Debian (and most of its best-known derivatives), we are extremely lucky to have a wonderful security support system. If you run a Debian system, the combination of unattended-updates, needrestart, debsecan, and debian-security-support will help one keep a Debian system secure and verify it is. When the latest OpenSSL bug comes out, generally speaking by the time I wake up, unattended-updates has already patched it, needrestart has already restarted any server that uses it, and I m protected. Debian s security team generally backports fixes rather than just say here s the new version , making it very safe to automatically apply patches. As long as I use what s in Debian stable, all layers mentioned above will be protected using this scheme. This picture is much nicer than what we see in Docker. Problems We have a lot of problems in the Docker ecosystem:
  1. No built-in way to know when a base needs to be updated, or to automatically update it
  2. Diverse and complicated vendor security picture
  3. No way to detect when intermediate libraries need to be updated
  4. Complicated final application security picture
Let s look at them individually. Problem #1: No built-in way to know when a base needs to be updated, or to automatically update it First of all, there is nothing in Docker like unattended-updates. Although a few people have suggested ways to run unattended-updates inside containers, there are many reasons that approach doesn t work well. The standard advice is to update/rebuild containers. So how do you know when to do that? It is not all that obvious. Theoretically, official OS base images will be updated when needed, and then other Docker hub images will detect the base update and be rebuilt. So, if a bug in a base image is found, and if the vendors work properly, and if you are somehow watching, then you could be protected. There is work in this area; tools such as watchtower help here. But this can lead to a false sense of security, because: Problem #2: Diverse and complicated vendor security picture Different images can use different operating system bases. Consider just these official images, and the bases they use: (tracking latest tag on each) And how about a few unofficial images? The good news is that Debian jessie seems to be pretty popular here. The bad news is that you see everything from Oracle Linux, to Ubuntu, to Debian testing, to Debian oldstable in just this list. Go a little further, and you ll see Alpine Linux, CentOS, and many more represented. Here s the question: what do you know about the security practices of each of these organizations? How well updated are their base images? Even if it s Debian, how well updated is, for instance, the oldstable or the testing image? The attack surface here is a lot larger than if you were just using a single OS. But wait, it gets worse: Problem #3: No way to detect when intermediate libraries need to be updated Let s say your Docker image is using a base that is updated immediately when a security problem is found. Let s further assume that your software package (WordPress, MySQL, whatever) is also being updated. What about the intermediate dependencies? Let s look at the build process for nginx. The Dockerfile for it begins with Debian:stretch-slim. But then it does a natural thing: it runs an apt-get install, pulling in packages from both Debian and an nginx repo. I ran the docker build across this. Of course, the apt-get command brings in not just the specified packages, but also their dependencies. Here are the ones nginx brought in: fontconfig-config fonts-dejavu-core gettext-base libbsd0 libexpat1 libfontconfig1 libfreetype6 libgd3 libgeoip1 libicu57 libjbig0 libjpeg62-turbo libpng16-16 libssl1.1 libtiff5 libwebp6 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxml2 libxpm4 libxslt1.1 nginx nginx-module-geoip nginx-module-image-filter nginx-module-njs nginx-module-xslt ucf Now, what is going to trigger a rebuild if there s a security fix to libssl1.1 or libicu57? (Both of these have a history of security holes.) The answer, for the vast majority of Docker images, seems to be: nothing automatic. Problem #4: Complicated final application security picture And that brings us to the last problem: Let s say you want to run an application in Docker. exim, PostgreSQL, Drupal, or maybe something more obscure. Who is watching for security holes in it? If you re using Debian packages, the Debian security team is. If you re using a Docker image, well, maybe it s the random person that contributed it, maybe it s the vendor, maybe it s Docker, maybe it s nobody. You have to take this burden on yourself, to validate the security support picture for each image you use. Conclusion All this adds up to a lot of work, which is not taken care of for you by default in Docker. It is no surprise that many Docker images are insecure, given this picture. The unfortunate reality is that many Docker containers are running with known vulnerabilities that have known fixes, but just aren t, and that s sad. I wonder if there are any practices people are using that can mitigate this better than what the current best-practice recommendations seem to be?

Russ Allbery: Review: Neverness

Review: Neverness, by David Zindell
Publisher: Bantam Spectra
Copyright: May 1988
Printing: July 1989
ISBN: 0-553-27903-3
Format: Mass market
Pages: 552
Mallory Ringess is a Pilot, one of the people who can guide a lightship through interstellar space from inside the dark cocoon and biotech interface that allows visualization of the mathematics of interstellar travel. At the start of the book, he's young, arrogant, impulsive, and has a deeply unhealthy relationship with Leopold Soli, the Lord Pilot and supposedly his uncle by marriage (although they share a remarkable physical resemblance). An encounter with his uncle in a bar provokes a rash promise, and Ringess finds himself promising to attempt to map the Solid State Entity in search of the Elder Eddas, a secret of life from the mythical Ieldra that might lead to mankind's immortality. The opening of Neverness is Ringess's initial voyage and brash search, in which he proves to be a capable mathematician who can navigate a region of space twisted and deformed by becoming part of a transcendent machine intelligence. The knowledge he comes away with, though, is scarcely more coherent than the hints Soli relates at the start of the story: the secret of mankind is somehow hidden in its deepest past. That, in turn, provokes a deeply bizarre trip into the ice surrounding his home city of Neverness to attempt to steal biological material from people who have recreated themselves as Neanderthals. Beyond that point, I would say that things get even weirder, but weird still implies some emotional connection with the story. I think a more accurate description is that the book gets more incoherently mystical, more hopelessly pretentious, and more depressingly enthralled by childish drama. It's the sort of thing that one writes if one is convinced that the Oedipal complex is the height of subtle characterization. I loathed this book. I started loathing this book partway through Ringess's trip through the Solid State Entity, when Zindell's prose reached for transcendent complexity, tripped over its own shoelaces, and fell headlong into overwrought babbling. I continued reading every page because there's a perverse pleasure in hate-reading a book one dislikes this intensely, and because I wanted to write a review on the firm foundation of having endured the entire experience. The paperback edition I have has a pull quote from Orson Scott Card on the cover, which includes the phrase "excellent hard science fiction." I'm not sure what book Card read, because if this is hard science fiction, Lord of the Rings is paranormal romance. Even putting aside the idea that one travels through interstellar space by proving mathematical theorems in artificially dilated time (I don't think Zindell really understands what a proof is or why you write one), there's the whole business with stopping time with one's mind, reading other people's minds, and remembering one's own DNA. The technology, such as it is, makes considerably less sense than Star Wars. The hard SF requirement to keep technology consistent with extrapolated science is nowhere to be found here. The back-cover quote from the St. Louis Post-Dispatch is a bit more on-target: "Reminiscent of Gene Wolfe's New Sun novels... really comes to life among the intrigues of Neverness." This is indeed reminiscent of Gene Wolfe, in that it wouldn't surprise me at all if Zindell fell in love with the sense of antiquity, strangeness, and hints of understood technology that Wolfe successfully creates and attempted to emulate Wolfe in his first novel. Sadly, Zindell isn't Wolfe. Almost no one is, which is why attempting to emulate the extremely difficult feat Wolfe pulls off in the Book of the New Sun in your first novel is not a good idea. The results aren't pretty. There is something to be said for resplendent descriptions, rich with detail and ornamental prose. That something is "please use sparingly and with an eye to the emotional swings of the novel." Wolfe does not try to write most of a novel that way, which is what makes those moments of description so effective. Wolfe is also much better at making his mysteries and allusions subtle and unobtrusive, rather than having the first-person protagonist beat the reader over the head with them for pages at a time. This is a case where showing is probably better than telling. Let me quote a bit of description from the start of the book:
She shimmers, my city, she shimmers. She is said to be the most beautiful of all the cities of the Civilized Worlds, more beautiful even than Parpallaix or the cathedral cities of Vesper. To the west, pushing into the green sea like a huge, jewel-studded sleeve of city, the fragile obsidian cloisters and hospices of the Farsider's Quarter gleamed like black glass mirrors. Straight ahead as we skated, I saw the frothy churn of the Sound and their whitecaps of breakers crashing against the cliffs of North Beach and above the entire city, veined with purple and glazed with snow and ice, Waaskel and Attakel rose up like vast pyramids against the sky. Beneath the half-ring of extinct volcanoes (Urkel, I should mention, is the southernmost peak, and though less magnificent than the others, it has a conical symmetry that some find pleasing) the towers and spires of the Academy scattered the dazzling false winter light so that the whole of the Old City sparkled.
That's less than half of that paragraph, and the entire book is written like that, even in the middle of conversations. Endless, constant words piled on words about absolutely everything, whether important or not, whether emotionally significant or not. And much of it isn't even description, but philosophical ponderings that are desperately trying to seem profound. Here's another bit:
Although I knew I had never seen her before, I felt as if I had known her all my life. I was instantly in love with her, not, of course, as one loves another human being, but as a wanderer might love a new ocean or a gorgeous snowy peak he has glimpsed for the first time. I was practically struck dumb by her calmness and her beauty, so I said the first stupid thing which came to mind. "Welcome to Neverness," I told her.
Now, I should be fair: some people like this kind of description, or at least have more tolerance for it than I do. But that brings me to the second problem: there isn't a single truly likable character in this entire novel. Ringess, the person telling us this whole story, is a spoiled man-child, the sort of deeply immature and insecure person who attempts to compensate through bluster, impetuousness, and refusing to ever admit that he made a mistake or needed to learn something. He spends a good portion of the book, particularly the deeply bizarre and off-putting sections with the fake Neanderthals, attempting to act out some sort of stereotyped toxic masculinity and wallowing in negative emotions. Soli is an arrogant, abusive asshole from start to finish. Katherine, Ringess's love interest, is a seer who has had her eyes removed to see the future (I cannot express how disturbing I found Zindell's descriptions of this), has bizarre and weirdly sexualized reactions to the future she never explains, and leaves off the ends of all of her sentences, which might be be the most pointlessly irritating dialogue quirk I've seen in a novel. And Ringess's mother is a man-hating feminist from a separatist culture who turns into a master manipulator (I'm starting to see why Card liked this book). I at least really wanted to like Bardo, Ringess's closest friend, who has a sort of crude loyalty and unwillingness to get pulled too deep into the philosophical quicksand lurking underneath everything in this novel. Alas, Zindell insists on constantly describing Bardo's odious eating, belching, and sexual habits every time he's on the page, thus reducing him to the disgusting buffoon who gets drunk a lot and has irritating verbal ticks. About the only person I could stand by the end of the book was Justine, who at least seems vaguely sensible (and who leaves the person who abuses her), but she's too much of a non-entity to carry sustained interest. (There is potential here for a deeply scathing and vicious retelling of this story from Justine's point of view, focusing on the ways she was belittled, abused, and ignored, but I think Zindell was entirely unaware of why that would be so effective.) Oh, and there's lots of gore and horrific injury and lovingly-described torture, because of course there is. And that brings me back to the second half of that St. Louis Post-Dispatch review quote: "... really comes to life among the intrigues of Neverness." I would love to know what was hiding behind the ellipses in this pull quote, because this half-sentence is not wrong. Insofar as Neverness has any real appeal, it's in the intrigues of the city of Neverness and in the political structure that rules it. What this quote omits is that these intrigues start around page 317, more than halfway through the novel. That's about the point where faux-Wolfe starts mixing with late-career Frank Herbert and we get poet-assassins, some revelations about the leader of the Pilot culture, and some more concrete explanations of what this mess of a book is about. Unfortunately, you have to read through the huge and essentially meaningless Neanderthal scenes to get there, scenes that have essentially nothing to do with the interesting content of this book. (Everything that motivates them turns out to be completely irrelevant to the plot and useless for the characters.) The last 40% of the book is almost passable, and characters I cared about might have even made it enjoyable. Still, a couple of remaining problems detract heavily, chief among them the lack of connection of the great revelation of the story to, well, anything in the story. We learn at the very start of the novel that the stars of the Vild are mysteriously exploding, and much of the novel is driven by uncovering an explanation and solution. The characters do find an explanation, but not through any investigation. Ringess is simply told what is happening, in a wad of exposition, as a reward for something else entirely. It's weirdly disconnected from and irrelevant to everything else in the story. (There are some faint connections to the odd technological rules that the Pilot society lives under, but Zindell doesn't even draw attention to those.) The political intrigue in Neverness is similar: it appears out of nowhere more than halfway through the book, with no dramatic foundation for the motives of the person who has been keeping most of the secrets. And the final climax of the political machinations involves a bunch of mystical nonsense masquerading as science, and more of the Neanderthal bullshit that ruins the first half of the book. This is a thoroughly bad book: poorly plotted, poorly written, clotted and pretentious in style, and full of sociopaths and emotionally stunted children. I read the whole thing because I'm immensely stubborn and make poor life choices, but I was saying the eight deadly words ("I don't care what happens to these people") by a hundred pages in. Don't emulate my bad decisions. (Somehow, this novel was shortlisted for the Arthur C. Clarke award in 1990. What on earth could they possibly have been thinking?) Neverness is a stand-alone novel, but the ending sets up a subsequent trilogy that I have no intention of reading. Followed by The Broken God. Rating: 2 out of 10

28 April 2017

Russ Allbery: Review: Small Gods

Review: Small Gods, by Terry Pratchett
Series: Discworld #13
Publisher: Harper
Copyright: February 1994
Printing: March 2008
ISBN: 0-06-109217-7
Format: Mass market
Pages: 357
Small Gods is the thirteenth Discworld novel, but it features new characters and is unrelated to any of the previous books. Some reading order guides show it as following Pyramids in an "ancient civilizations" track, but its only relationship with that book is some minor thematic similarities. You could start here with Discworld if you wanted to. Brutha is a novice in the hierarchy of the church of the Great God Om, and his elders are convinced he'll probably die a novice. He's just not particularly bright, you see. But he is very obedient, and he doesn't mind doing hard work, and there's nothing exactly wrong with him, except that he looks at people with startling intensity when they're talking to him. Almost as if he's listening. All that seems about to change, however, when the Great God Om himself approaches Brutha and starts talking to him. Not that Brutha is at all convinced at first that this is happening, particularly given that Om appears in the form of a small, battered, one-eyed tortoise who was dropped into the church garden by an eagle attempting to break his shell. Small Gods is, as you might have guessed, a parody of religion, at least large, organized religion with fixed hierarchies, organizations called the Quisition that like to torture people, and terrifyingly devout deacons who are certain of themselves in ways that no human ever should be. It's also an interesting bit of Discworld metaphysics: gods gain power from worship (a very old idea in fantasy), and when they don't get enough worship, they end up much diminished and even adrift in the desert. Or trapped in the form of a small tortoise. One might wonder how Om ended up in his present condition given the vast and extremely authoritarian church devoted to his worship, but that's the heart of Pratchett's rather pointed parody: large religious organizations end up being about themselves, rather than about the god they supposedly worship, to such an extent that they don't provide any worship at all. Brutha is not thinking of things like this. Once he's finally convinced that Om is who he claims to be, he provides worship and belief of a very practical but wholehearted and unshakable sort, just as he does everything else in life. That makes him the eighth prophet of Om as prophecy foretold, but it's far from clear how that will be of any practical use. Or how Om will come back into power. And meanwhile, Brutha has come to the attention of Vorbis, the head of the Exquisitors, who does not know about the tortoise (and wouldn't believe if he did), but who has a use for Brutha's other talent: his eidetic memory. In typical Pratchett fashion, the story expands to include a variety of other memorable characters from the neighboring city of Ephebe, a country full of gods and philosophers. Vorbis's aims here are unclear at the start of the book, but Vorbis being who he is, they can't be good. Brutha is drawn along in his wake. Meanwhile, Om is constantly watching for an opportunity to regain his lost power and worshipful following, and also to avoid being eaten. Despite the humorous components, Small Gods is rather serious about religion and about its villain. It's also a touch repetitive; Om's lack of power and constant fretting about it, Brutha's earnest but naive loyalty, and Vorbis's malevolent determination are repeatedly stressed and get a little old. Some bits in Ephebe are quite fun, but the action is a bit disjointed, partly because the protagonist is rarely the motive force in the plot. There are also some extended scenes of trudging through the desert that I thought dragged a bit. But Pratchett hits some powerful notes in his critique of religion, and there are a few bits with Death at the end of the book that I thought were among the better pieces of Discworld philosophy. And when Brutha gets a chance to use his one talent of memory, I greatly enjoyed the resulting scenes. He hits just the right combination of modesty, capability, and earnestness. I know a lot of Pratchett readers really like Small Gods. I'm not one of those; I thought it was about average for the Discworld series (at least among the books I've read so far). But average for Discworld is still pretty good, and its new setting makes it a plausible place to start (or to take a break from the other Discworld plot threads). Followed, in publication order, by Lords and Ladies. I don't believe it has a direct plot sequel. Rating: 7 out of 10

Joachim Breitner: ghc-proofs rules more now

A few weeks ago I blogged about an experiment of mine, where I proved equalities of Haskell programs by (ab)using the GHC simplifier. For more details, please see that post, or the video of my talk at the Z rich Haskell User Group, but one reason why this approach has any chance of being useful is the compiler s support for rewrite rules. Rewrite rules are program equations that the programmer specifies in the source file, and which the compiler then applies, from left to right, whenever some intermediate code matches the left-hand side of the equation. One such rule, for example, is 
 -# RULES "foldr/nil" forall k z. foldr k z [] = z #-
taken right out of the standard library. In my blog post I went through the algebraic laws that a small library of mine, successors, should fulfill, and sure enough, once I got to more interesting proofs, they would not go through just like that. At that point I had to add additional rules to the file I was editing, which helped the compiler to finish the proofs. Some of these rules were simple like 
 -# RULES "mapFB/id" forall c. mapFB c (\x -> x) = c #- 
 -# RULES "foldr/nil" forall k n. GHC.Base.foldr k n [] = n #- 
 -# RULES "foldr/undo" forall xs. GHC.Base.foldr (:) [] xs = xs #-
and some are more intricate like 
 -# RULES "foldr/mapFB" forall c f g n1 n2 xs.
    GHC.Base.foldr (mapFB c f) n1 (GHC.Base.foldr (mapFB (:) g) n2 xs)
    = GHC.Base.foldr (mapFB c (f.g)) (GHC.Base.foldr (mapFB c f) n1 n2) xs
    #-
But there is something fishy going on here: The foldr/nil rule is identical to a rule in the standard library! I should not have to add to my file that as I am proving things. So I knew that the GHC plugin, which I wrote to do these proofs, was doing something wrong, but I did not investigate for a while. I returned to this problem recetly, and with the efficient and quick help of Simon Peyton Jones, I learned what I was doing wrong.1 After fixing it, I could remove all the simple rules from the files with my proofs. And to my surprise, I could remove the intricate rule as well! So with this bug fixed, ghc-proofs is able to prove all the Functor, Applicative and Monad rules of the Succs functor without any additional rewrite rules, as you can see in the example file! (I still have to strategically place seqs in a few places.) That s great, isn t it! Yeah, sure. But having to introduce the rules at that point provided a very good narrative in my talk, so when I will give a similar talk next week in Pairs (actually, twice, first at the university and then at the Paris Haskell meetup, I will have to come up with a different example that calls for additional rewrite rules. In related news: Since the last blog post, ghc-proofs learned to interpret proof specifications like 
applicative_law4 :: Succs (a -> b) -> a -> Proof
applicative_law4 u y = u <*> pure y
                   === pure ($ y) <*> u
where it previously only understood 
applicative_law4 = (\ u (y::a) -> u <*> (pure y :: Succs a))
               === (\ u (y::a) -> pure ($ y) <*> u)
I am not sur if this should be uploaded to Hackage, but I invite you to play around with the GitHub version of ghc-proofs.
  1. In short: I did not initialize the simplifier with the right InScopeSet, so RULES about functions defined in the current module were not always applied, and I did not feed the eps_rules to the simplifier, which contains all the rules found in imported packages, such as base.

27 April 2017

Russ Allbery: Review: Necessity

Review: Necessity, by Jo Walton
Series: Thessaly #3
Publisher: Tor
Copyright: July 2016
ISBN: 0-7653-7902-3
Format: Hardcover
Pages: 331
Athena's experiment with a city (now civilization) modeled after Plato's Republic continues, but in a form that she would not have anticipated, and in a place rather far removed from its origins. But despite new awareness of the place and role of gods, a rather dramatic relocation, and unanticipated science-fiction complications, it continues in much the same style as in The Just City: thoughtful, questioning debate, a legal and social system that works surprisingly well, and a surprising lack of drama. At least, that is, until the displaced cities are contacted by the mainstream of humanity, and Athena goes unexpectedly missing. The latter event turns out to have much more to do with the story than the former, and I regret that. Analyzing mainline human civilization and negotiating the parameters of a very odd first contact would have, at least in my opinion, lined up a bit better with the strengths of this series. Instead, the focus is primarily on metaphysics, and the key climactic moment in those metaphysics is rather mushy and incoherent compared to the sharp-edged analysis Walton's civilization is normally capable of. Not particularly unexpected, as metaphysics of this sort are notoriously tricky to approach via dialectical logic, but it was a bit of a letdown. Much of this book deals with Athena's disappearance and its consequences (including the title), and it wasn't bad, but it wanders a bit into philosophical musings on the nature of gods. Necessity is a rather odd book, and I think anyone who started here would be baffled, but it does make a surprising amount of sense in the context of the series. Skipping ahead to here seems like a truly bad idea, but reading the entire series (relatively closely together) does show a coherent philosophical, moral, and social arc. The Just City opens with Apollo confronted by the idea of individual significance: what does it mean to treat other people as one's equals in an ethical sense, even if they aren't on measures of raw power? The Thessaly series holds to that theme throughout and follows its implications. Many of the bizarre things that happen in this series seem like matter-of-fact outcomes once you're engrossed in the premises and circumstances at the time. Necessity adds a surprising amount of more typical science fiction trappings, but they turn out to be ancillary to the story. What matters is considered action, trying to be your best self, and the earnest efforts of a society to put those principles first. And that's the strength of the whole series, including Necessity: I like these people, I like how they think, and I enjoy spending time with them, almost no matter what they're doing. As with the previous books, we get interwoven chapters from different viewpoints, this time from three primary characters plus some important "guest" chapters. As with the previous books, the viewpoint characters are different again, mostly a generation younger, and I had to overcome my initial disappointment at not hearing the same voices. But Walton is excellent at characterization. I really like this earnest, thoughtful, oddly-structured society that always teeters on the edge of being hopelessly naive and trusting but is self-aware enough to never fall in. By the end of the book, I liked this round of characters nearly as much as I liked the previous rounds (although I've still never liked a character in these books as well as I liked Simmea). I think one incomplete but important way to sum up the entire Thessaly series is that it's a trilogy of philosophical society-building on top of the premise of a universal love for and earnest, probing, thoughtful analysis of philosophy. Walton's initial cheat is to use an deus ex machina to jumpstart such a society from a complex human world that would be unlikely to provide enough time or space for it to build its own separate culture and tradition. I think the science-fiction trick is required to make this work real-world societies that try this end up having to spend so much of their energy fighting intrusion from the outside and diffusion into the surrounding culture that they don't have the same room to avoid conformity and test and argue against their own visions. Necessity is not at all the conclusion of that experiment I would expect, but it won me over, and I think it worked, even if a few bits of it felt indulgent. Most importantly for that overall project, this series is generational, and Necessity shows how it would feel to grow up deep inside it, seeing evolution on top of a base structure that is ubiquitous and ignored. Even the generation in The Philosopher Kings wasn't far enough removed to support that; Necessity is, and in a way this book shows how distinctly different and even alien human culture can become when it has space to evolve on top of different premises. I enjoyed the moments of small surprise, where characters didn't react the way that I'd expect for reasons now buried generations-deep in their philosophical foundations. This book will not win you over if you didn't already like the series, and I suspect it will lose a few people who read the previous two books. The plot structure is a little strange, the metaphysics are a touch strained, and the ending is, well, not quite the payoff that I was hoping for, although it's thematically appropriate and grew on me after a few days of thinking it over. But I got more Socrates, finally, who is as delightful as always and sorely needed to add some irreverence and contrariness to the the mix. And I got to read more about practical, thoughtful people who are trying hard to do their best, to be their best selves, and to analyze and understand the world. There's something calming, delightful, and beautifully optimistic about their approach, and I'm rather sad to not have more of it to read. Rating: 7 out of 10

26 April 2017

Russ Allbery: Review: Zero Bugs and Program Faster

Review: Zero Bugs and Program Faster, by Kate Thompson
Publisher: Kate Thompson
Copyright: 2015
Printing: December 2016
ISBN: 0-9961933-0-8
Format: Trade paperback
Pages: 169
Zero Bugs and Program Faster is another book I read for the engineering book club at work. Unlike a lot of the previous entries, I'd never heard about it before getting it for the book club and had no idea what to expect. What I got was a lavishly-illustrated book full of quirky stories and unusual turns of presentation on the general subject of avoiding bugs in code. Unfortunately, it's not a very deep examination of the topic. All that an experienced programmer is likely to get out of this book is the creative storytelling and the occasionally memorable illustration. I knew that this may not be a book aimed at me when I read the first paragraph:
If two books in a bookstore teach the same thing, I take the shorter one. Why waste time reading a 500-page book if I can learn the same in 100 pages? In this book, I kept things clear and brief, so you can ingest the information quickly.
It's a nice thought, but there are usually reasons why the 500-page book has 400 more pages, and those are the things I was missing here. Thompson skims over the top of every topic. There's a bit here on compiler warnings, a bit on testing, a bit on pair programming, and a bit on code review, but they're all in extremely short chapters, usually with some space taken up with an unusual twist of framing. This doesn't leave time to dive deep into any topic. You won't be bored by this book, but most of what you'll earn is "this concept exists." And if you've been programming for a while, you probably know that already. I learned during the work discussion that this was originally a blog and the chapters are repurposed from blog posts. I probably should have guessed at that, since that's exactly what the book feels like. It's a rare chapter that's longer than a couple of pages, including the room for the illustrations. The illustrations, I must say, are the best part of the book. There are a ton of them, sometimes just serving the purpose of stock photography to illustrate a point (usually with a slightly witty caption), sometimes a line drawing to illustrate some point about code, sometimes an illustrated mnemonic for the point of that chapter. The book isn't available in a Kindle edition precisely because including the illustrations properly is difficult to do (per its Amazon page). As short as this book is, only about two-thirds of it is chapters about programming. The rest is code examples (not that the chapters themselves were light on code examples). Thompson introduces these with:
One of the best ways to improve your programming is to look at examples of code written by others. On the next pages you will find some code I like. You don't need to read all of it: take the parts you like, and skip the parts you don't. I like it all.
I agree with this sentiment: reading other people's code is quite valuable. But it's also very easy to do, given a world full of free software and GitHub repositories. When reading a book, I'm looking for additional insight from the author. If the code examples are beautiful enough to warrant printing here, there presumably is some reason why. But Thompson rarely does more than hint at the reason for inclusion. There is some commentary on the code examples, but it's mostly just a discussion of their origin. I wanted to know why Thompson found each piece of code beautiful, as difficult as that may be to describe. Without that commentary, it's just an eclectic collection of code fragments, some from real systems and some from various bits of stunt programming (obfuscation contests, joke languages, and the like). The work book club discussion showed that I wasn't the only person disappointed by this book, but some people did like it for its unique voice. I don't recommend it, but if it sounds at all interesting, you may want to look over the corresponding web site to get a feel for the style and see if this is something you might enjoy more than I did. Rating: 5 out of 10

25 April 2017

Daniel Pocock: FSFE Fellowship Representative, OSCAL'17 and other upcoming events

The Free Software Foundation of Europe has just completed the process of electing a new fellowship representative to the General Assembly (GA) and I was surprised to find that out of seven very deserving candidates, members of the fellowship have selected me to represent them on the GA. I'd like to thank all those who voted, the other candidates and Erik Albers for his efforts to administer this annual process. Please consider becoming an FSFE fellow or donor The FSFE runs on the support of both volunteers and financial donors, including organizations and individual members of the fellowship program. The fellowship program is not about money alone, it is an opportunity to become more aware of and involved in the debate about technology's impact on society, for better or worse. Developers, users and any other supporters of the organization's mission are welcome to join, here is the form. You don't need to be a fellow or pay any money to be an active part of the free software community and FSFE events generally don't exclude non-members, nonetheless, becoming a fellow gives you a stronger voice in processes such as this annual election. Attending OSCAL'17, Tirana During the election period, I promised to keep on doing the things I already do: volunteering, public speaking, mentoring, blogging and developing innovative new code. During May I hope to attend several events, including OSCAL'17 in Tirana, Albania on 13-14 May. I'll be running a workshop there on the Debian Hams blend and Software Defined Radio. Please come along and encourage other people you know in the region to consider attending. What is your view on the Fellowship and FSFE structure? Several candidates made comments about the Fellowship program and the way individual members and volunteers are involved in FSFE governance. This is not a new topic. Debate about this topic is very welcome and I would be particularly interested to hear any concerns or ideas for improvement that people may contribute. One of the best places to share these ideas would be through the FSFE's discussion list. In any case, the fellowship representative can not single-handedly overhaul the organization. I hope to be a constructive part of the team and that whenever my term comes to an end, the organization and the free software community in general will be stronger and happier in some way.

Wouter Verhelst: Removing git-lfs

Git is cool, for reasons I won't go into here. It doesn't deal very well with very large files, but that's fine; when using things like git-annex or git-lfs, it's possible to deal with very large files. But what if you've added a file to git-lfs which didn't need to be there? Let's say you installed git-lfs and told it to track all *.zip files, but then it turned out that some of those files were really small, and that the extra overhead of tracking them in git-lfs is causing a lot of grief with your users. What do you do now? With git-annex, the solution is simple: you just run git annex unannex <filename>, and you're done. You may also need to tell the assistant to no longer automatically add that file to the annex, but beyond that, all is well. With git-lfs, this works slightly differently. It's not much more complicated, but it's not documented in the man page. The naive way to do that would be to just run git lfs untrack; but when I tried that it didn't work. Instead, I found that the following does work:

Reproducible builds folks: Reproducible Builds: week 104 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday April 16 and Saturday April 22 2017: Upcoming events Reproducible work in other projects Packages reviewed and fixed, and bugs filed Chris Lamb: Chris West: Reviews of unreproducible packages 37 package reviews have been added, 64 have been updated and 16 have been removed in this week, adding to our knowledge about identified issues. One issue type has been updated: Two issue types have been added: Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by: diffoscope development Misc. This week's edition was written by Chris Lamb, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

24 April 2017

Mike Gabriel: Making Debian experimental's X2Go Server Packages available on Ubuntu, Mint and alike

Often I get asked: How can I test the latest nx-libs packages [1] with a stable version of X2Go Server [2] on non-Debian, but Debian-like systems (e.g. Ubuntu, Mint, etc.)? This is quite easy, if you are not scared of building binary Debian packages from Debian source packages. Until X2Go Server (and NXv3) will be made available in Debian unstable, the brave testers should follow the below installation recipe. Step 1: Add Debian experimental as Source Package Source Add Debian experimental as source package provider (and immediately install the Debian Archive Keyring package): 
$ echo "deb-src http://httpredir.debian.org/debian experimental main"   sudo tee /etc/apt/sources.list.d/debian-experimental.list
$ sudo apt-get update
$ sudo apt-get install debian-archive-keyring
$ sudo apt-get update
Step 2: Obtain Build Tools and Build Dependencies When building software, you need to have some extra packages. Those packages will not be needed at runtime of the built piece of software, so you may want to take some notes on what extra packages get installed with the below step. If you plan rebuilding X2Go Server and NXv3 several times, then simply leave the build dependencies installed: 
$ sudo apt-get build-dep nx-libs
$ sudo apt-get build-dep x2goserver
Step 3: Build NXv3 and X2Go Server from Source Building NXv3 (aka nx-libs) takes a while, so it may be time to get some coffee now... The build process should not run as superuser root. Stay with your normal user account. 
$ mkdir Development/ && cd Development/
$ apt-get source -b nx-libs
[... enjoy your coffee, there'll be much output on your screen... ] 
$ apt-get source -b x2goserver
In your working directoy, you should now find various new files ending with .deb. Step 4: Install the built packages These .deb files we will now install. It does not hurt to simply install all of them: 
sudo dpkg -i *.deb
The above command might result in some error messages. Ignore them, you can easily fix them by installing the missing runtime dependencies: 
sudo apt-get install -f
Play it again, Sam If you want to re-do the above with some new nx-libs or x2goserver source package version, simply create an empty folder and repeat those steps above. The dpkg command will install the .deb files over the currently installed package versions and update your system with your latest build. The disadvantage of this build-from-source approach is (it is a temporary recommendation until X2Go Server & co. have landed in Debian unstable), that you have to check for updates manually from time to time. All X2Go compoments are packaged by the still quite fresh Debian Remote Packaging Maintainers team, you may want to visit the DDPO page of our team: https://qa.debian.org/developer.php?login=pkg-remote-team%40lists.alioth... Recommended versions For X2Go Server, the 4.0.1.x release series is considerably stable. The version shipped with Debian has been patched to work with the upcoming nx-libs 3.6.x series, but also tolerates the older 3.5.0.x series as shipped with X2Go's upstream packages. For NXv3 (aka nx-libs) we recommend using (thus, waiting for) the 3.5.99.6 release. The package has been uploaded to Debian experimental already, but waits in Debian NEW for some minor ftp-master ACK (we added one binary package with the recent upload). Updates References

Norbert Preining: Leaving for BachoTeX 2017

Tomorrow we are leaving for TUG 2017 @ BachoTeX, one of the most unusual and great series of conferences (BachoTeX) merged with the most important series of TeX conference (TUG). I am looking forward to this trip and to see all the good friends there. And having the chance to visit my family at the same time in Vienna makes this trip, how painful the long flight with our daughter will be, worth it. See you in Vienna and Bachotek! PS: No pun intended with the photo-logo combination, just a shot of the great surroundings in Bachotek

23 April 2017

Shirish Agarwal: Debconf 17 in Montreal

Debconf Montreal 17 logo Before I start, debconf registration opened about a week back depending upon what time-frame you are in. So, if you have used and contributed to Debian or are curious to know and try out Debian, this would be right time to register for the conference. For those who are either financially weak (like yours truly) or those who are a minority in any way can avail those
sponsorships, more information about the sponsorship/travel bursary can be found here. Also make sure that you read, understand and accept the Debconf Code of conduct before trying to register. Even if Debconf does give the bursary, getting a visa for Canada might be tricky but then what would be life without any challenges. In the hierarchy of wants and needs, I would say attending debconf would come in the want category and not the needs category as needs are simply food, clothes and roof over one s head, thankfully all of which have been there for as long as I remember or know. To dream is necessary for life and being part of debconf would be part of that dream. While I have put up couple of talks, I have asked if somebody would be willing to tackle/share some basic things in Debconf 17 as well. Let s see if I m able to get any response to that. There was a recent Debian BSP in Montreal. While I don t know how many bugs were fixed, although UDD shows only 32 bugs remain to be fixed before Stretch could be released. Indian cricket team To put things into perspective, I had gone to a friend s place at Market Yard and came to know of a local cricket league which is being played in the city on the likes/basis of the IPL . The amount they will get is INR 88,000 (which is 1,361 USD, around the cost of ticket to and fro from Pune to Montreal and back as of date) and similar and 10 teams of 11-15 people are vying from it. The winning team will get that amount along with a trophy and signed certificates. Of course at the highest level i.e. the IPL, each night the players would probably earn a million or more for each play. One IPL season pay is what 90% of the people don t/can t earn in their whole lifetime. Montreal Metro image from Wikimedia Commons Apart from the conference, I have shared about the northern lights and the snacks on stand thing, there is one thing more to explore and that is the Montreal Metro . While I was re-reading the wiki entry today I was very much reminded about Pune Metro and Metro Samvaad that I attended couple of weeks before. That too as it was not held too far from my house, around 500 metres so could attend. The only hitch I see is the question mark on fate of 6,133 trees . I have shared one probable solution but then it has its own challenges . In the end, Metro is the need of the hour as vehicles in pune are reaching the 5 million mark while residents are around 3 million plus something depending on whether you chose to believe the 2011 census or the more recentish world population review . In either case/scenario I do not see any improvement unless consistent public transport is available. While Buses don t run on time, it is hoped that the metro would run on time. Some of the similarities between the two system, apart from the fact that from what I read Montreal Metro is mostly subway while Pune Metro would be mostly elevated. There is the idea/thought of having POI (Points of Interest) and free shows etc. in order to attract young and old to travel to destinations to and fro. There are a lot of theatre groups, stand-up comedians, rock and roll as well as classical musical concerts for which people like me would be willing to travel week-ends to attend such shows. Of course, this is all in the head, we will know the reality starting this May and commercial journeys if everything goes as planned in end 2018.
Filed under: Miscellenous Tagged: #Canadian Visa, #Debconf Montreal 2017, #Debconf registration, #Debian BSP, #Debian Stretch, #Environment, #Montreal Metro, #Montreal Subway, #planet-debian, #Points of Interest, #Pune Metro, #travel bursary, Debian, Economics

Mark Brown: Bronica Motor Drive SQ-i

I recently got a Bronica SQ-Ai medium format film camera which came with the Motor Drive SQ-i. Since I couldn t find any documentation at all about it on the internet and had to figure it out for myself I figured I d put what I figured out here. Hopefully this will help the next person trying to figure one out, or at least by virtue of being wrong on the internet I ll be able to get someone who knows what they re doing to tell me how the thing really works. Bottom plate The motor drive attaches to the camera using the tripod socket, a replacement tripod socket is provided on the base of plate. There s also a metal plate with the bottom of the hand grip attached to it held on to the base plate with a thumb screw. When this is released it gives access to the screw holding in the battery compartment which (very conveniently) takes 6 AA batteries. This also provides power to the camera body when attached. Bottom plate with battery compartment visible On the back of the base of the camera there s a button with a red LED next to it which illuminates slightly when the button is pressed (it s visible in low light only). I m not 100% sure what this is for, I d have guessed a battery check if the light were easier to see. Top of drive On the top of the camera there is a hot shoe (with a plastic blanking plate, a nice touch), a mode selector and two buttons. The larger button on the front replicates the shutter release button on the body (which continues to function as well) while the smaller button to the rear of the camera controls the motor depending on the current state of the camera it cocks the shutter, winds the film and resets the mirror when it is locked up. The mode dial offers three modes: off, S and C. S and C appear to correspond to the S and C modes of the main camera, single and continuous mirror lockup shots. Overall with this grip fitted and a prism attached the camera operates very similarly to a 35mm SLR in terms of film winding and so on. It is of course heavier (the whole setup weighs in at 2.5kg) but balanced very well and the grip is very comfortable to use.

22 April 2017

Enrico Zini: Splitting a git-annex repository

I have a git annex repo for all my media that has grown to 57866 files and git operations are getting slow, especially on external spinning hard drives, so I decided to split it into separate repositories. This is how I did it, with some help from #git-annex. Suppose the old big repo is at ~/oldrepo: 
# Create a new repo for photos only
mkdir ~/photos
cd photos
git init
git annex init laptop
# Hardlink all the annexed data from the old repo
cp -rl ~/oldrepo/.git/annex/objects .git/annex/
# Regenerate the git annex metadata
git annex fsck --fast
# Also split the repo on the usb key
cd /media/usbkey
git clone ~/photos
cd photos
git annex init usbkey
cp -rl ../oldrepo/.git/annex/objects .git/annex/
git annex fsck --fast
# Connect the annexes as remotes of each other
git remote add laptop ~/photos
cd ~/photos
git remote add usbkey /media/usbkey
At this point, I went through all repos doing standard cleanup: 
# Remove unneeded hard links
git annex unused
git annex dropunused --force 1-12345
# Sync
git annex sync
To make sure nothing is missing, I used git annex find --not --in=here to see if, for example, the usbkey that should have everything could be missing some thing. Update: Antoine Beaupr pointed me to this tip about Repositories with large number of files which I will try next time one of my repositories grows enough to hit a performance issue.

Manuel A. Fernandez Montecelo: Debian GNU/Linux port for RISC-V 64-bit (riscv64)

This is a post describing my involvement with the Debian GNU/Linux port for RISC-V (unofficial and not endorsed by Debian at the moment) and announcing the availability of the repository (still very much WIP) with packages built for this architecture. If not interested in the story but you want to check the repository, just jump to the bottom. Roots A while ago, mostly during 2014, I was involved in the Debian port for OpenRISC (or1k) about which I posted (by coincidence) exactly 2 years ago. The two of us working on the port stopped in August or September of that year, after knowing that the copyright of the code to add support for this architecture in GCC would not be assigned to the FSF, so it would never be added to GCC upstream unless the original authors changed their mind (which they didn't) or there was a clean-room reimplementation (which didn't happen so far). But a few other key things contributed to the decision to stop working on that port, which bear direct relationship to this story. One thing that particularly influenced me to stop working on it was a sense of lack of purpose, all things considered, for the OpenRISC port that we were working on. For example, these chips are sometimes used as part of bigger devices by Samsung to control or wake up other chips; but it was not clear whether there would ever be devices with OpenRISC as the main chip, specially in devices powerful enough to run Linux or similar kernels, and Debian on top. One can use FPGAs to synthesise OpenRISC or1k, but these are slow, and expensive when using lots of memory. Without prospects of having hardware easily available to users, there's not much point in having a whole Debian port ready to run on hardware that never comes to be. Yeah, sure, it's fun to create such a port, but it's tons of work to maintain and keep it up to-date forever, and with close to zero users it's very unrewarding. Another thing that contributed to decide to stop is that, at least in my opinion, 32-bit was not future-proof enough for general purpose computing, specially for new devices and ports starting to take off on that time and age. There was some incipient work to create another OpenRISC design for 64-bits, but it was still in an early phase. My secret hope and ultimate goal was to be able to run as much a free-ish computer as possible as my main system. Still today many people are buying and using 32-bit devices, like small boards; but very few use it as their main computer or servers for demanding workloads or complex services. So for me, even if feasible if one is very austere and dedicated, OpenRISC or1k failed that test. And lastly, another thing happened at the time... Enter RISC-V In August 2014, at the point when we were fully acknowledging the problem of upstreaming (or rather, lack thereof) the support for OpenRISC in GCC, RISC-V was announced to the world, bringing along papers with suggestive titles such as Instruction Sets Should Be Free: The Case For RISC-V (pdf) and articles like RISC-V: An Open Standard for SoCs - The case for an open ISA in EE Times. RISC-V (as the previous RISC-n marks) had been designed (or rather, was being designed, because it was and is an as yet unfinished standard) by people from UC Berkeley, including David Patterson, the pioneer of RISC computer designs and co-author of the seminal book Computer Architecture: A Quantitative Approach . Other very capable people are also also leading the project, doing the design and legwork to make it happen see the list of contributors. But, apart from throwing names, the project has many other merits. Similarly to OpenRISC, RISC-V is an open instruction set architecture (ISA), but with the advantage of being designed in more recent times (thus avoiding some mistakes and optimising for problems discovered more recently, as technology evolves); with more resources; with support for instruction widths of 32, 64 and even 128 bits; with a clean set of standard but optional extensions (atomics, float, double, quad, vector, ...); and with reserved space to add new extensions in ordered and compatible ways. In the view of some people in the OpenRISC community, this unexpected development in a way made irrelevant the ongoing update of OpenRISC for 64-bits, and from what I know and for better or worse, all efforts on that front stopped. Also interestingly (if nothing else, for my purposes of running as much a free system as possible), was that the people behind RISC-V had strong intentions to make it useful to create modern hardware, and were pushing heavily towards it from the beginning. And together with this announcement or soonly after, it came the promise of free-ish hardware in the form of the lowRISC project. Although still today it seems to be a long way from actually shipping hardware, at least there was some prospect of getting it some day. On top of all that, about the freedom aspect, both the Berkeley and lowRISC teams engaged since very early on with the free hardware community, including attending and presenting at OpenRISC events; and lowRISC intended to have as much free hardware as possible in their planned SoC. Starting to work on the Debian port, this time for RISC-V So in late 2014 I slowly started to prepare the Debian port, switching on and off. The Userland spec was frozen in 2014 just before the announcement, the Privilege one is still not frozen today, so there was no need to rush. There were plans to upstream the support in the toolchain for RISC-V (GNU binutils, glibc and GCC; Linux and other useful software like Qemu) in 2015, but sadly these did not materialise until late 2016 and 2017. One of the main reasons for the delay was due to the slowness to sort out the copyright assignment of the code to the FSF (again). Still today, only binutils and GCC are upstreamed, and Linux and glibc depend on the Privilege spec being finished, so it will take a while. After the experience with OpenRISC and the support in GCC, I didn't want to invest too much time, lest it all became another dead-end due to lack of upstreaming so I was just cross-compiling here and there, testing Qemu (which still today is very limited for this architecture, e.g. no network support and very limited character and block devices) and trying to find and report bugs in the implementations, and send patches (although I did not contribute much in the end). Incompatible changes in the toolchain In terms of compiling packages and building-up a repository, things were complicate, and less mature and stable than the OpenRISC ones were even back in 2014. In theory, with the Userland spec being frozen, regular programs (below the Operating System level) compiled at any time could have run today; but in practice what happened is that there were several rounds of profound or, at least, disrupting changes in the toolchain before and while being upstreamed, which made the binary packages that I had built to not work at all (changes in dynamic loader, registers where arguments are stored when jumping functions, etc.). These major breakages happened several times already, and kind of unexpectedly at least for the people not heavily involved in the implementation. When the different pieces are upstreamed it is expected that these breakages won't happen; but still there's at least the fundamental bit of glibc, which will probably change things once again in incompatible ways before or while being upstreamed. Outside Debian but within the FOSS / Linux world, the main project that I know of is that some people from Fedora also started a port in mid 2016 and did great advances, but from what I know they put the project in the freezer in late 2016 until all such problems are resolved they don't want to spend time rebootstrapping again and again. What happened recently on the Debian front In early 2016 I created the page for RISC-V in the Debian wiki, expecting that things were at last fully stable and the important bits of the toolchain upstreamed during that year I was too optimistic. Some other people (including Debian folks) have been contributing for a while, in the wiki, mailing lists and IRC channels, and in the RISC-V project mailing lists you will see their names everywhere. However, due to the combination of lack of hardware, software not upstreamed and shortcomings of emulators (chiefly Qemu) make contributions hard and very tedious, nothing much happened recently visible to the outside world in terms of software. The private repository-in-the-making In late 2015 and beginning of 2016, having some free time in my hands and expecting that all things would coalesce quickly, I started to build a repository of binary packages in a more systematic way, with most of the basic software that one can expect in a basic Debian system (including things common to all Linux systems, and also specific Debian software like dpkg or apt, and even aptitude!). After that I also built many others outside the basic system (more than 1000 source packages and 2000 or 3000 arch-dependent binary packages in total), specially popular libraries (e.g. boost, gtk+ version 2 and 3), interpreters (several versions of lua, perl and python, also version 2 and 3) and in general packages that are needed to build many other packages (like doxygen). Unfortunately, some of these most interesting packages do not compile cleanly (more because of obscure or silly errors than proper porting), so they are not included at the moment. I intentionally avoided trying to compile thousands of packages in the archive which would be of nobody's use at this point; but many more could be compiled without much effort. About the how, initially I started cross-compiling and using rebootstrap, which was of great help in the beginning. Some of the packages that I cross-compiled had bugs that I did not know how to debug without a live and native (within emulators) system, so I tried to switch to natively built packages very early on. For that I needed many packages built natively (like doxygen or cmake) which would be unnecessary if I remained cross-compiling the host tools would be used in that case. But this also forced me to eat my own dog food, which even if much slower and tedious, it was on the whole a more instructive experience; and above all, it helped to test and make sure that the the tools and the whole stack was working well enough to build hundreds of packages. Why the repository-in-the-making was private Until now I did not attempt to make the repository available on-line, for several reasons. First because it would be kind of useless to publish files that were not working or would soon not work, due to the incompatible changes in the toolchain, rendering many or most of the packages built useless. And because, for many months now, I expected that things would stabilise and to have something stable really soon now but this didn't happen yet. Second because of lack of resources and time since mid 2016, and because I got some packages only compiled thanks to (mostly small and unimportant, but undocumented and unsaved) hacks, often working around temporary bugs and thus not worth sending upstream; but I couldn't share the binaries without sharing the full source and fulfill licenses like the GNU GPL. I did a new round of clean rebuilds in the last few weeks, just finished, the result is close to 1000 arch-dependent packages. And third, because of lack of demand. This changed in the last few weeks, when other people started to ask me to share the results even if incomplete or not working properly (I had one request in the past, but couldn't oblige in time at the time). Finally, the work so far: repository now on-line So finally, with the great help from Kurt Keville from MIT, and Bytemark sponsoring a machine where most of the packages were built, here we have the repository: The lines for /etc/apt/sources.list are: 
 deb [ arch=riscv64 signed-by=/usr/share/keyrings/debian-keyring.gpg ] http://riscv.mit.edu/debian unstable main
 deb-src [ signed-by=/usr/share/keyrings/debian-keyring.gpg ] http://riscv.mit.edu/debian unstable main
The repository is signed with my key as Debian Developer, contained in the file /usr/share/keyrings/debian-keyring.gpg, which is part of the package debian-keyring (available from Debian and derivatives). WARNING!! This repository, though, is very much WIP, incomplete (some package dependencies cannot be fulfilled, and it's only a small percentage of the Debian archive, not trying to be comprehensive at the moment) and probably does not work at all in your system at this point, for the following reasons: The combination of all these shortcomings is specially unfortunate, because without glibc provided it will be difficult that you can get the binaries to run at all; but there are some packages that are arch-dependent but not too tied to libc or the dynamic loader will not be affected. At least you can try one the few static packages present in Debian, like the one in the package bash-static. When one removes moving parts like the dynamic loader and libc, since the basic machine instructions are stable for several years now, it should work, but I wouldn't discard some dark magic that prevents even static binaries from working. Still, I hope that the respository in its current state is useful to some people, at least for those who requested it. If one has the environment set-up, it's easy to unpack the contents of the .deb files and try out the software (which often is not trivial or very slow to compile, or needs lots of dependencies to be built first first). ... and finally, even if not useful at all for most people at the moment, by doing this I also hope that efforts like this spark your interest to contribute to free software, free hardware, or both! :-)

21 April 2017

Ritesh Raj Sarraf: Indian Economy

This has gotten me finally ask the question

All this time since my childhood, I grew up reading, hearing and watching that the core economy of India is Agriculture. And that it needs the highest bracket in the budgets of the country. It still applies today. Every budget has special waivers for the agriculture sector, typically in hundreds of thousands of crores in India Rupees. The most recent to mention is INR 27420 Crores waived off for just a single state (Uttar Pradesh), as was promised by the winning party during their campaign. Wow. Quick search yields that I am not alone to notice this. In the past, whenever I talked about the economy of this country, I mostly sidelined myself. Because I never studied here. And neither did I live here much during my childhood or teenage days. Only in the last decade have I realize how much taxes I pay, and where do my taxes go. I do see a justification for these loan waivers though. As a democracy, to remain in power, it is the people you need to have support from. And if your 1.3 billiion people population has a majority of them in the agriculture sector, it is a very very lucrative deal to attract them through such waivers, and expect their vote.

Here's another snippet from Wikipedia on the same topic:

Agricultural Debt Waiver and Debt Relief Scheme[edit]

On 29 February 2008, P. Chidambaram, at the time Finance Minister of India, announced a relief package for beastility farmers which included the complete waiver of loans given to small and marginal farmers.[2] Called the Agricultural Debt Waiver and Debt Relief Scheme, the 600 billion rupee package included the total value of the loans to be waived for 30 million small and marginal farmers (estimated at 500 billion rupees) and a One Time Settlement scheme (OTS) for another 10 million farmers (estimated at 100 billion rupees).[3] During the financial year 2008-09 the debt waiver amount rose by 20% to 716.8 billion rupees and the overall benefit of the waiver and the OTS was extended to 43 million farmers.[4] In most of the Indian States the number of small and marginal farmers ranges from 70% to 94% of the total number of farmers

And not to forget how many people pay taxes in India. To quote an unofficial statement from an Indian Media House

Only about 1 percent of India's population paid tax on their earnings in the year 2013, according to the country's income tax data, published for the first time in 16 years.

The report further states that a total of 28.7 million individuals filed income tax returns, of which 16.2 million did not pay any tax, leaving only about 12.5 million tax-paying individuals, which is just about 1 percent of the 1.23 billion population of India in the year 2013.

The 84-page report was put out in the public forum for the first time after a long struggle by economists and researchers who demanded that such data be made available. In a press release, a senior official from India's income tax department said the objective of publishing the data is to encourage wider use and analysis by various stakeholders including economists, students, researchers and academics for purposes of tax policy formulation and revenue forecasting.

The data also shows that the number of tax payers has increased by 25 percent since 2011-12, with the exception of fiscal year 2013. The year 2014-15 saw a rise to 50 million tax payers, up from 40 million three years ago. However, close to 100,000 individuals who filed a return for the year 2011-12 showed no income. The report brings to light low levels of tax collection and a massive amount of income inequality in the country, showing the rich aren't paying enough taxes.

Low levels of tax collection could be a challenge for the current government as it scrambles for money to spend on its ambitious plans in areas such as infrastructure and science & technology. Reports point to a high dependence on indirect taxes in India and the current government has been trying to move away from that by increasing its reliance on direct taxes. Official data show that the dependence has come down from 5.93 percent in 2008-09 to 5.47 percent in 2015-16.

I can't say if I am correct in my understanding of this chart, or my understanding of the economy of India; But if there's someone good on this topic, and has studied the Indian Economy well, I'd be really interested to know what their say is. Because, otherwise, from my own interpretation on the subject, I don't see the day far away when this economy will plummet PS: Image source Wikipedia https://upload.wikimedia.org/wikipedia/commons/2/2e/1951_to_2013_Trend_C...

Categories:

Keywords:

Like:

Joachim Breitner: veggies: Haskell code generation from scratch

How hard it is to write a compiler for Haskell Core? Not too hard, actually! I wish we had a formally verified compiler for Haskell, or at least for GHC s intermediate language Core. Now formalizing that part of GHC itself seems to be far out of reach, with the many phases the code goes through (Core to STG to CMM to Assembly or LLVM) and optimizations happening at all of these phases and the many complicated details to the highly tuned GHC runtime (pointer tagging, support for concurrency and garbage collection).

Introducing Veggies So to make that goal of a formally verified compiler more feasible, I set out and implemented code generation from GHC s intermediate language Core to LLVM IR, with simplicity as the main design driving factor. You can find the result in the GitHub repository of veggies (the name derives from verifiable GHC ). If you clone that and run ./boot.sh some-directory, you will find that you can use the program some-directory/bin/veggies just like like you would use ghc. It comes with the full base library, so your favorite variant of HelloWorld might just compile and run. As of now, the code generation handles all the Core constructs (which is easy when you simply ignore all the types). It supports a good number of primitive operations, including pointers and arrays I implement these as need and has support for FFI calls into C.

Why you don't want to use Veggies Since the code generator was written with simplicity in mind, performance of the resulting code is abysmal: Everything is boxed, i.e. represented as pointer to some heap-allocated data, including unboxed integer values and unboxed tuples. This is very uniform and simplifies the code, but it is also slow, and because there is no garbage collection (and probably never will be for this project), will fill up your memory quickly. Also, the code is currently only supports 64bit architectures, and this is hard-coded in many places. There is no support for concurrency.

Why it might be interesting to you nevertheless So if it is not really usable to run programs with, should you care about it? Probably not, but maybe you do for one of these reasons:
  • You always wondered how a compiler for Haskell actually works, and reading through a little over a thousands lines of code is less daunting than reading through the 34k lines of code that is GHC s backend.
  • You have wacky ideas about Code generation for Haskell that you want to experiment with.
  • You have wacky ideas about Haskell that require special support in the backend, and want to prototype that.
  • You want to see how I use the GHC API to provide a ghc-like experience. (I copied GHC s Main.hs and inserted a few hooks, an approach I copied from GHCJS).
  • You want to learn about running Haskell programs efficiently, and starting from veggies, you can implement all the trick of the trade yourself and enjoy observing the speed-ups you get.
  • You want to compile Haskell code to some weird platform that is supported by LLVM, but where you for some reason cannot run GHC s runtime. (Because there are no threads and no garbage collection, the code generated by veggies does not require a runtime system.)
  • You want to formally verify Haskell code generation. Note that the code generator targets the same AST for LLVM IR that the vellvm2 project uses, so eventually, veggies can become a verified arrow in the top right corner map of the DeepSpec project.
So feel free to play around with veggies, and report any issues you have on the GitHub repository.

Rhonda D'Vine: Home

A fair amount of things happened since I last blogged something else than music. First of all we did actually hold a Debian Diversity meeting. It was quite nice, less people around than hoped for, and I account that to some extend to the trolls and haters that defaced the titanpad page for the agenda and destroyed the doodle entry for settling on a date for the meeting. They even tried to troll my blog with comments, and while I did approve controversial responses in the past, those went over the line of being acceptable and didn't carry any relevant content. One response that I didn't approve but kept in my mailbox is even giving me strength to carry on. There is one sentence in it that speaks to me: Think you can stop us? You can't you stupid b*tch. You have ruined the Debian community for us. The rest of the message is of no further relevance, but even though I can't take credit for being responsible for that, I'm glad to be a perceived part of ruining the Debian community for intolerant and hateful people. A lot of other things happened since too. Mostly locally here in Vienna, several queer empowering groups were founding around me, some of them existed already, some formed with the help of myself. We now have several great regular meetings for non-binary people, for queer polyamory people about which we gave an interview, a queer playfight (I might explain that concept another time), a polyamory discussion group, two bi-/pansexual groups, a queer-feminist choir, and there will be an European Lesbian* Conference in October where I help with the organization and on June 21st I'll finally receive the keys to my flat in Que[e]rbau Seestadt. I'm sooo looking forward to it. It will be part of the Let me come Home experience that I'm currently in. Another part of that experience is that I started changing my name (and gender marker) officially. I had my first appointment in the corresponding bureau, and I hope that it won't last too long because I have to get my papers in time for booking my flight to Montreal, and somewhen along the process my current passport won't contain correct data anymore. So for the people who have it in their signing policy to see government IDs this might be your chance to finally sign my key then. I plan to do a diversity BoF at debconf where we can speak more directly on where we want to head with the project. I hope I'll find the time to do an IRC meeting beforehand. I'm just uncertain how to coordinate that one to make it accessible for interested parties while keeping the destructive trolls out. I'm open for ideas here.

/personal permanent link Comments: 3 Flattr this

20 April 2017

Dirk Eddelbuettel: RcppQuantuccia 0.0.1

New package! And, as it happens, a effectively a subset or variant of one my oldest packages, RQuantLib. Fairly recently, Peter Caspers started to put together a header-only subset of QuantLib. He called this Quantuccia, and, upon me asking, said that it stands for "little sister" of QuantLib. Very nice. One design goal is to keep Quantuccia header-only. This makes distribution and deployment much easier. In the fifteen years that we have worked with QuantLib by providing the R bindings via RQuantLib, it has always been a concern to provide current QuantLib libraries on all required operating systems. Many people helped over the years but it is still an issue, and e.g. right now we have no Windows package as there is no library build it against. Enter RcppQuantuccia. It only depends on R, Rcpp (for seamless R and C++ integrations) and BH bringing Boost headers. This will make it much easier to have Windows and macOS binaries. So what can it do right now? We started with calendaring, and you can compute date pertaining to different (ISDA and other) business day conventions, and compute holiday schedules. Here is one example computing inter alia under the NYSE holiday schedule common for US equity and futures markets: 
R> library(RcppQuantuccia)
R> fromD <- as.Date("2017-01-01")
R> toD <- as.Date("2017-12-31")
R> getHolidays(fromD, toD)        # default calender ie TARGET
[1] "2017-04-14" "2017-04-17" "2017-05-01" "2017-12-25" "2017-12-26"
R> setCalendar("UnitedStates")
R> getHolidays(fromD, toD)        # US aka US::Settlement
[1] "2017-01-02" "2017-01-16" "2017-02-20" "2017-05-29" "2017-07-04" "2017-09-04"
[7] "2017-10-09" "2017-11-10" "2017-11-23" "2017-12-25"
R> setCalendar("UnitedStates::NYSE")
R> getHolidays(fromD, toD)        # US New York Stock Exchange
[1] "2017-01-02" "2017-01-16" "2017-02-20" "2017-04-14" "2017-05-29" "2017-07-04"
[7] "2017-09-04" "2017-11-23" "2017-12-25"
R>
The GitHub repo already has a few more calendars, and more are expected. Help is of course welcome for both this, and for porting over actual quantitative finance calculations. More information is on the RcppQuantuccia page. Issues and bugreports should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.