Search Results: "ag"

6 November 2017

Andreas Bombe: Reviving GHDL in Debian

It has been a few years since Debian last had a working VHDL simulator in the archive. Its competitor Verilog has been covered by the iverilog and verilator simulator packages, but GHDL was the only option for VHDL in Debian and that has become broken, orphaned and was eventually removed. I have just submitted an ITP to make my work on it official. A lot has changed since the last Debian upload of GHDL. Upstream development is quite active and it has gained free reimplementations of the standard library definitions (the lack of which frustrated at least two attempts at adoption of the Debian package). It has gained additional backends, in addition to GCC it can now also use LLVM and its own custom mcode (x86 only) code generator. The mcode backend should provide faster compilation at the expense of lacking sophisticated optimization, hence it might be preferable over the other two for small projects. My intentions are to provide all three backends in separate packages which would also offer easier backend troubleshooting a user experiencing problems can simply install another package to try a different backend. The problem with that idea is that GHDL is not designed for that kind of parallel installation. The backend is chosen at build configure time and that configuration is built and installed. Parallel installation will probably need some development but if that would turn out to be much work I could always have the packages conflicting initially. Given all these changes I am redoing the Debianization from ground up and maybe take bits and pieces from the old packaging where suitable. Right now I m building the different backends to compare and see what files are backend specific and what can go into a common package.

John Goerzen: The Yellow House Phone Company (Featuring Asterisk and an 11-year-old)

Well Jacob, do you think we should set up our own pretend phone company in the house? We can DO THAT? Yes! Then yes. Yes! YES YES YESYESYESYES YES! Let s do it, dad! Not long ago, my parents had dug up the old phone I used back in the day. We still have a landline, and Jacob was having fun discovering how an analog phone works. I told him about the special number he could call to get the time and temperature read out to him. He discovered what happens if you call your own number and hang up. He figured out how to play Mary Had a Little Lamb using touchtone keys (after a slightly concerned lecture from me setting out some rules to make sure his musical dialing wouldn t result in any, well, dialing.) He was hooked. So I thought that taking it to the next level would be a good thing for a rainy day. I have run Asterisk before, though I had unfortunately gotten rid of most of my equipment some time back. But I found a great deal on a Cisco 186 ATA (Analog Telephone Adapter). It has two FXS lines (FXS ports simulate the phone company, and provide dialtone and ring voltage to a connected phone), and of course hooks up to the LAN. We plugged that in, and Jacob was amazed to see its web interface come up. I had to figure out how to configure it (unfortunately, it uses SCCP rather than SIP, and figuring out Asterisk s chan_skinny took some doing, but we got there.) I set up voicemail. He loved it. He promptly figured out how to record his own greetings. We set up a second phone on the other line, so he could call between them. The cordless phones in our house support SIP, so I configured one of them as a third line. He spent a long time leaving himself messages. IMG_3465 Pretty soon we both started having ideas. I set up extension 777, where he could call for the time. Then he wanted a way to get the weather forecast. Well, weather-util generates a text-based report. With it, a little sed and grep tweaking, the espeak TTS engine, and a little help from sox, I had a shell script worked up that would read back a forecast whenever he called a certain extension. He was super excited! That s great, dad! Can it also read weather alerts too? Sure! weather-util has a nice option just for that. Both boys cackled as the system tried to read out the NWS header (their timestamps like 201711031258 started with two hundred one billion ) Then I found an online source for streaming NOAA Weather Radio feeds Jacob enjoys listening to weather radio and I set up another extension he could call to listen to that. More delight! But it really took off when I asked him, Would you like to record your own menu? You mean those things where it says press 1 or 2 for this or that? Yes. WE CAN DO THAT? Oh yes! YES, LET S DO IT RIGHT NOW! So he recorded a menu, then came and hovered by me while I hacked up extensions.conf, then eagerly went back to the phone to try it. Oh the excitement of hearing hisown voice, and finding that it worked! Pretty soon he was designing sub-menus ( OK Dad, so we ll set it up so people can press 2 for the weather, and then choose if they want weather radio or the weather report. I m recording that now. Got it? ) He has informed me that next Saturday we will build an intercom system like we have at school. I m going to have to have some ideas on how to tie Squeezebox in with Asterisk to make that happen, I think. Maybe this will do.

5 November 2017

Thorsten Alteholz: My Debian Activities in October 2017

FTP assistant Again, this month almost the same numbers as last month appeared in the statistics. I accepted 214 packages and rejected 22 uploads. The overall number of packages that got accepted this month was only 339. Debian LTS This was my fortieth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. This month my all in all workload has been 20.75h. During that time I did LTS uploads of: I also took care of radare2 twice and marked all four CVEs as not-affected for Wheezy and Jessie. As nobody else wanted to address the issues in wireshark yet, I now started to work on this package. Last but not least I did one week of frontdesk duties. Other stuff During October I took care of some bugs and at one go uploaded new upstream versions of hoel and duktape (this had to be done twice as I introduced an new bug with the first upload :-(). I only fixed bugs in glewlwyd and smstools. This month I also sponsored an upload of printrun. After about ten years of living without any power outage, some construction worker decided to cut a cable near my place. Unfortunately one of my computers used for recording TV shows did not boot after the cable had been repaired and I had to switch some timers to other boxes. All in all this was too much stress and I purchased some USVs from APC. As apcupsd was orphaned, I took the opportunity to adopt it as DOPOM for this month. My license pasting project now contains 31 license templates for your debian/copyright. The list of available texts can be obtained with:

curl http://licapi.debian.net/template

The license text itself is available under the given links, for example with

curl http://licapi.debian.net/template/Apache-2

Dirk Eddelbuettel: pinp 0.0.4: Small tweak

A maintenance release of our pinp package for snazzier one or two column vignettes is now on CRAN as of yesterday. In version 0.0.3, we disabled the default \pnasbreak command we inherit from the PNAS LaTeX style. That change turns out to have been too drastic. So we reverted yet added a new YAML front-matter option skip_final_break which, if set to TRUE, will skip this break. With a default value of FALSE we maintain prior behaviour. A screenshot of the package vignette can be seen below. Additional screenshots of are at the pinp page.
The NEWS entry for this release follows.

Changes in pinp version 0.0.4 (2017-11-04)
  • Correct NEWS headers from 'tint' to 'pinp' (#45).
  • New front-matter variables skip_final_break skips the \pnasbreak on final page which back as default (#47).

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the tint page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Russ Allbery: Review: Sweep in Peace

Review: Sweep in Peace, by Ilona Andrews
Series: Innkeeper Chronicles #2
Publisher: NYLA
Copyright: 2015
ISBN: 1-943772-32-0
Format: Kindle
Pages: 302
This is the sequel to Clean Sweep. You could pick up the background as you go along, but the character relationships benefit from reading the series in order. Dina's inn is doing a bit better, but it still desperately needs guests. That means she's not really in a position to say no when an Arbitrator shows up at her door and asks her to host a peace summit. Lucky for the Arbitrator, since every other inn on Earth did say no. Nexus has been the site of a viciously bloody conflict between the vampires, the Hope-Crushing Horde, and the Merchants of Baha-char for years. All sides have despaired of finding any form of peace. The vampires and the Horde have both deeply entrenched themselves in a cycle of revenge. The Merchants have the most strategic position and an apparently unstoppable warrior. The situation is hopeless; by far the most likely outcome will be open warfare inside the inn, which would destroy its rating and probably Dina's future as an innkeeper. Dina will need all of her power and caution just to stop that; peace seems beyond any possibility, but thankfully isn't her problem. Maybe the Arbitrator can work some miracle if she can just keep everyone alive. And well fed. Which is another problem. She has enough emergency money for the food, but somehow cook for forty people from four different species while keeping them all from killing each other? Not a chance. She's going to have to hire someone somehow, someone good, even though she can't afford to pay. Sweep in Peace takes this series farther out of urban fantasy territory and farther into science fiction, and also ups the stakes (and the quality of the plot) a notch. We get three moderately interesting alien species with only slight trappings of fantasy, a wonderful alien chef who seems destined to become a regular in the series, and a legitimately tricky political situation. The politics and motives aren't going to win any awards for deep and subtle characterization, but that isn't what the book is going for. It's trying to throw enough challenges at Dina to let her best characteristics shine, and it does that rather well. The inn continues to be wonderful, although I hope it becomes more of a character in its own right as the series continues. Dina's reshaping of it for guests, and her skill at figuring out the rooms her guests would enjoy, is my favorite part of these books. She cares about making rooms match the personality of her guests, and I love books that give the character a profession that matters to them even if it's unrelated to the plot. I do wish Andrews would find a few other ways for Dina to use her powers for combat beyond tentacles and burying people in floors, but that's mostly a quibble. You should still not expect great literature. I guessed the big plot twist several chapters before it happened, and the resolution is, well, not how these sorts of political situations resolve in the real world. But there is not a stupid love affair, there are several interesting characters, and one of the recurring characters gets pretty solid and somewhat unusual characterization. And despite taking the plot in a more serious direction, Sweep in Peace retains its generally lighthearted tone and firm conviction in Dina's ability to handle just about anything. Also, the chef is wonderful. One note: Partway into the book, I started getting that "oh, this is a crossover" feeling (well-honed by years of reading comic books). As near as I can tell from a bit of research, Andrews pulled in some of their characters from the Edge series. This was a bit awkward, in the "who are these people and why do they seem to have more backstory than any of the other supporting characters" cross-over sort of way, but the characters that were pulled in were rather intriguing. I might have to go read the Edge books now. Anyway, if you liked Clean Sweep, this is better in pretty much every way. Recommended. Followed by One Fell Sweep. Rating: 8 out of 10

4 November 2017

Alexander Wirt: debconf mailinglists moved to lists.debian.org

Today I had the pleasure to move the debconf mailinglists to lists.debian.org. That means that the following mailinglists: are now hosted on lists.debian.org. Please update any documentation or bookmarks you have. Next step would be to join debconf again ;).

Steinar H. Gunderson: Tr ndisk 2017 live stream

We're streaming live from Tr ndisk 2017, first round of the Norwegian ultimate frisbee series, today from 0945 CET and throughout the day/weekend. It's an interesting first for Nageru in that it's sports, where everything happens much faster and there are more demands for overlay graphics (I've made a bunch of CasparCG templates). I had hoped to get to use Narabu in this, but as the (unfinished) post series indicates, I simply had to prioritize other things. There's plenty of new things for us anyway, not the least that I'll be playing and not operating. :-) Feel free to tune in to the live stream, although we don't have international stream reflectors. It's a fun sport with many nice properties. :-) There will be YouTube not too long after the day is over, too. Edit: All games from Saturday and Sunday are now online; see this YouTube list. Most commentary is in Norwegian, although some games are in English. I was happy everything worked well, and the production crew did a great job (I was busy playing), but of course there's tons of small things we want to improve for next time.

Louis-Philippe V ronneau: Migrating my website to Pelican

After too much time lying to myself, telling myself things like "I'll just add this neat feature I want on my blog next week", I've finally made the big jump, ditched django and migrated my website to Pelican. I'm going to the Cambridge Mini-Debconf at the end of the month for the Debconf Videoteam Autumn sprint and I've taken the task of making daily sprint reports for the team. That in return means I have to publish my blog on Planet Debian. My old website not having feeds made this a little hard and this perfect storm gave me the energy to make the migration happen. Anyway, django was fun. Building a (crappy) custom blogging engine with it taught me some rough basics, but honestly I don't know why I ever thought it was a good idea. Don't get me wrong: django is great and should definitely be used for large and complicated websites. My blog just ain't one. Migrating to Pelican was pretty easy since it also uses Jinja2 templates and generates content from Mardown. The hardest part was actually bending it to replicate the weird and specific behavior I wanted it to have. So yeah, woooo, I migrated to Pelican. Who cares, right? Well, if you are amongst the very, very few people who read the blog posts I mainly write for myself, you'll be please to know that: Here's a bonus picture of a Pelican from Wikimedia, just for the sake of it: A pelican

Dirk Eddelbuettel: tint 0.0.4: Small enhancements

A maintenance release of the tint package arrived on CRAN earlier today. Its name expands from tint is not tufte as the package offers a fresher take on the Tufte-style for html and pdf presentations. A screenshot of the pdf variant is below.
This release brings some minor enhancements and polish, mostly learned from having done the related pinp (two-column vignette in the PNAS style) and linl (LaTeX letter) RMarkdown-wrapper packages; see below for details from the NEWS.Rd file.

Changes in tint version 0.0.4 (2017-11-02)
  • Skeleton files are also installed as vignettes (#20).
  • A reference to the Tufte source file now points to tint (Ben Marwick in #19, later extended to other Rmd files).
  • Several spelling and grammar errors were corrected too (#13 and #16 by R. Mark Sharp and Matthew Henderson)

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the tint page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

3 November 2017

Reproducible builds folks: Reproducible Builds: Weekly report #131

Here's what happened in the Reproducible Builds effort between Sunday October 22 and Saturday October 28 2017: Past Events Upcoming/current events Documentation updates Bernhard Wiedemann started The Unreproducible Package which "is meant as a practical way to demonstrate the various ways that software can break reproducible builds using just low level primitives without requiring external existing programs that implement these primitives themselves. It is structured so that one subdirectory demonstrates one class of issues in some variants observed in the wild." Reproducible work in other projects Hush, a fork of ZCash, opened an issue into reproducible builds. A new tag was added to lintian (lint checker for Debian packages) to ensure that changelog entry timestamps are strictly increasing. This avoids certain real-world issues with identical timestamps, documented in Debian #843773. Packages reviewed and fixed, and bugs filed Patches sent upstream: Debian bug reports: Reviews of unreproducible packages 14 package reviews have been added, 35 have been updated and 28 have been removed in this week, adding to our knowledge about identified issues. 1 issue type has been updated: Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by: strip-nondeterminism development Version 0.040-1 was uploaded to unstable by Mattia Rizzolo. It included contributions already covered by posts of the previous weeks, as well as new ones from: reprotest development Development continued in git: buildinfo.debian.net development Development continued in git: reproducible-website development Misc. This week's edition was written by Ximin Luo, Chris Lamb, Bernhard M. Wiedemann and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Raphaël Hertzog: My Free Software Activities in October 2017

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donors (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I was allocated 12h but I had 1.5h left from September too. During this time, I finally finished my work on exiv2: I completed the triage of all CVE, backported 3 patches to the version in wheezy and released DLA-1147-1. I also did some review of the oldest entries in dla-needed. I reclassified a bunch of CVE on zoneminder and released DLA-1145-1 for the most problematic issue on that package. Many other packages got their CVE reclassified as not worth an update: xbmc, check-mk, rbenv, phamm, yaml-cpp. For mosquitto, I released DLA-1146-1. I filed #879001 (security issue) and #879002 (removal suggestion) on libpam4j. This library is no longer used by any other package in Debian, so it could be removed instead of costing us time in support. Misc Debian work After multiple months of wait, I was allowed to upload my schroot stable update (#864297). After ack from the d-i release manager, I pushed my pkgsel changes and uploaded version 0.46 of the package: this brings unattended-upgrades support in the installer. It s now installed by default. I nudged the upstream developer of gnome-shell-timer to get a new release for GNOME 3.26 compatibility and packaged it. Finally, I was pleased to merge multiple patches from Ville Skytt on Distro Tracker (the software powering tracker.debian.org). It looks like Ville will continue to contribute on a regular basis, yay. \o/ He already helped me to fix the remaining blockers for the switch to Python 3. Not really Debian related, but I also filed a bug against Tryton that I discovered after upgrading to the latest version. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Chris Lamb: Faking cleaner URLs in the Debian BTS

Debian bug #846500 requests that the Bug Tracking System moves the canonical URL for a given bug from:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=846500 to the shorter, cleaner and generally less ugly:

https://bugs.debian.org/846500 (The latter currently redirects to the former.)

However, whilst we wait for a fix we can abuse the window.history object from the HTML History API to fake this locally: 
var m = window.location.href
  .match(/https:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi\?bug=(\d+)(#.*)?$/);
if (!m) return;
for (var x of document.getElementsByTagName("a"))  
  var href = x.getAttribute("href");
  if (href && href.match(/^[^:]+\.cgi/))  
      // Mangle relative URIs; <base> tag does not DTRT
      x.setAttribute('href', "/cgi-bin/" + href);
   
 
history.replaceState( , "", "/" + m[1] + window.location.hash);
This should work with most "user script" managers I happen to use TamperMonkey in Chrome.

Joerg Jaspert: Automated wifi login, update 2

Seems my blog lately just consist of updates to my automated login script for the ICE wifi But I do hate the entirely useless Click a button crap, every day, twice. I ve seen it once, now leave me alone, please. Updated script: 
#!/bin/bash
# (Some) docs at
# https://wiki.ubuntuusers.de/NetworkManager/Dispatcher/
IFACE=$ 1:-"none" 
ACTION=$ 2:-"up" 
TMPDIR=$ TMPDIR:-"/tmp" 
WGET="/usr/bin/wget"
TIMEOUT="/usr/bin/timeout -k 20 15"
case $ ACTION  in
    up)
        CONID=$ CONNECTION_ID:-$(iwconfig $IFACE   grep ESSID   cut -d":" -f2   sed 's/^[^"]*"\ "[^"]*$//g') 
        if [[ $ CONID  == WIFIonICE ]]; then
            REFERER="http://www.wifionice.de/de/"
            LOGIN="http://www.wifionice.de/de/"
            COOKIETMP=$(mktemp -p $ TMPDIR  nmwifionice.XXXXXXXXX)
            trap "rm -f $ COOKIETMP " EXIT TERM HUP INT QUIT
            csrftoken=$($ TIMEOUT  $ WGET  -q -O - --keep-session-cookies --save-cookies=$ COOKIETMP  --referer $ REFERER  $ LOGIN     grep -oP  'CSRFToken"\ value="\K[0-9a-z]+')
            if [[ -z $ csrftoken  ]]; then
                echo "CSRFToken is empty"
                exit 0
            fi
            sleep 1
            $ TIMEOUT  $ WGET  -q -O - --load-cookies=$ COOKIETMP  --post-data="login=true&connect=connect&CSRFToken=$ csrftoken " --referer $ REFERER  $ LOGIN  >/dev/null
        fi
        ;;
    *)
        # We are not interested in this
        :
        ;;
esac

Jaldhar Vyas: New Hackergotchi

The sole purpose of this post is to check my new hackergotchi looks ok on Debian Planet. As I've lost a lot of weight and this is the second Diwali I've managed to survive without regaining, I thought I should update it to a more accurate depiction.

Rog rio Brito: Comparison of JDK installation of various Linux distributions

Today I spent some time in the morning seeing how one would install the JDK on Linux distributions. This is to create a little comparative tutorial to teach introductory Java. Installing the JDK is, thanks to the OpenJDK developers in Debian and Ubuntu (Matthias Klose and helpers), a very easy task. You simply type something like: 
apt-get install openjdk-8-jdk
Since for a student it is better to have everything for experiments, I install the full version, not only the -headless version. Given my familiarity with Debian/Ubuntu, I didn't have to think about the way of installing it, of course. But as this is a tutorial meant to be as general as I can, I tried also to include instructions on how to install Java on other distributions. The first two that came to my mind were openSUSE and Fedora. Both use the RPM package format for their "native" packages (in the same sense that Debian uses DEB packages for "native" packages). But they use different higher-level tools to install such packages: Fedora uses a tool called dnf, while openSUSE uses zypper. To try these distributions, I got their netinstall ISOs and used qemu/kvm to install on a virtual machine. I used the following to install/run the virtual machines (the example below, is, of course, for openSUSE): 
qemu-system-x86_64 -enable-kvm -m 4096 -smp 2 -net nic,model=e1000 -net user -drive index=0,media=disk,cache=unsafe,file=suse.qcow2 -cdrom openSUSE-Leap-42.3-NET-x86_64.iso
The names of the packages also change from one distribution to another. On Fedora, I had to use: 
dnf install java-1.8.0-openjdk-devel
On openSUSE, I had to use: 
zypper install java-1_8_0-openjdk-devel
Note that one distribution uses dots in the names of the packages while the other uses underscores. One interesting thing that I noticed with dnf was that, when I used it, it automatically refreshed the package lists from the network, something which I forgot, and it was a pleasant surprise. I don't know about zypper, but I guess that it probably had fresh indices when the installation finished. Both installations were effortless after I knew the names of the packages to install. Oh, BTW, in my 5 minute exploration with these distributions, I noticed that if you don't want the JDK, but only the JRE, then you omit the -devel suffix. It makes sense when you think about it, for consistency with other packages, but Debian's conventions also make sense (JRE with -jre suffix, JDK with -jdk suffix). I failed miserably to use Fedora's prebaked, vanilla cloud image, as I couldn't login on this image and I decided to just install the whole OS on a fresh virtual machine. I don't have instructions on how to install on Gentoo nor on Arch, though. I now see how hard it is to cover instructions/provide software for as many distributions as you wish, given the multitude of package managers, conventions etc.

2 November 2017

Steinar H. Gunderson: Introducing Narabu, part 5: Encoding

Narabu is a new intraframe video codec. You probably want to read part 1, part 2, part 3 and part 4 first. At this point, we've basically caught up with where I am, so thing are less set in stone. However, let's look at what qualitatively differentiates encoding from decoding; unlike in interframe video codecs (where you need to do motion vector search and stuff), encoding and decoding are very much mirror images of each other, so we can intuitively expect them to be relatively similar in performance. The procedure is DCT, quantization, entropy coding, and that's it. One important difference is in the entropy coding. Since our rANS encoding is non-adaptive (a choice made largely for simplicity, but also because our streams are so short) works by first signaling a distribution and then encoding each coefficient using that distribution. However, we don't know that distribution until we've DCT-ed all blocks in the picture, so we can't just DCT each block and entropy code the coefficients on-the-fly. There are a few ways to deal with this: As you can see, tons of possible strategies here. For simplicity, I've ended up with the former, although this could very well be changed at some point. There are some interesting subproblems, though: First of all, we need to decide the data type of this temporary array. The DCT tends to concentrate energy into fewer coefficients (which is a great property for compression!), so even after quantization, some of them will get quite large. This means we cannot store them in an 8-bit texture; however, even the bigger ones are very rarely bigger than 10 bits (including a sign bit), so using 16-bit textures wastes precious memory bandwidth. I ended up slicing the coefficients by horizontal index and then pairing them up (so that we generate pairs 0+7, 1+6, 2+5 and 3+4 for the first line of the 8x8 DCT block, 8+15, 9+14, 10+13 and 11+12 for the next line, etc.). This allows us to pack two coefficients into a 16-bit texture, for an average of 8 bits per coefficient, which is what we want. It makes for some slightly fiddly clamping and bit-packing since we are packing signed values, but nothing really bad. Second, and perhaps surprisingly enough, counting efficiently is nontrivial. We want a histogram over what coefficients are used the more often, ie., for each coefficient, we want something like ++counts[dist][coeff] (recall we have four distinct distributions). However, since we're in a massively parallel algorithm, this add needs to be atomic, and since values like e.g. 0 are super-common, all of our GPU cores will end up fighting over the cache line containing counts[dist][0]. This is not fast. Think 10 ms/frame not fast. Local memory to the rescue again; all modern GPUs have fast atomic adds to local memory (basically integrating adders into the cache, as I understand it, although I might have misunderstood here). This means we just make a suitably large local group, build up our sub-histogram in local memory and then add all nonzero buckets (atomically) to the global histogram. This improved performance dramatically, to the point where it was well below 0.1 ms/frame. However, our histogram is still a bit raw; it sums to 1280x720 = 921,600 values, but we want an approximation that sums to exactly 4096 (12 bits), with some additional constraints (like no nonzero coefficients). Charles Bloom has an exposition of a nearly optimal algorithm, although it took me a while to understand it. The basic idea is: Make a good approximation by multiplying each frequency by 4096/921600 (rounding intelligently). This will give you something that very nearly rounds to 4096 either above or below, e.g. 4101. For each step you're above or below the total (five in this case), find the best single coefficient to adjust (most entropy gain, or least loss); Bloom is using a heap, but on the GPU, each core is slow but we have many of them, so it's better just to try all 256 possibilities in parallel and have a simple voting scheme through local memory to find the best one. And then finally, we want a cumulative distribution function, but that is simple through a parallel prefix sum on the 256 elements. And then finally, we can take our DCT coefficients and the finished rANS distribution, and write the data! We'll have to leave some headroom for the streams (I've allowed 1 kB for each, which should be ample except for adversarial data and we'll probably solve that just by truncating the writes and accepting the corruption), but we'll compact them when we write to disk. Of course, the Achilles heel here is performance. Where decoding 720p (luma only) on my GTX 950 took 0,4 ms or so, encoding is 1,2 ms or so, which is just too slow. Remember that 4:2:2 is twice that, and we want multiple streams, so 2,4 ms per frame is eating too much. I don't really know why it's so slow; the DCT isn't bad, the histogram counting is fast, it's just the rANS shader that's slow for some reason I don't understand, and also haven't had the time to really dive deeply into. Of course, a faster GPU would be faster, but I don't think I can reasonably demand that people get a 1080 just to encode a few video streams. Due to this, I haven't really worked out the last few kinks. In particular, I haven't implemented DC coefficient prediction (it needs to be done before tallying up the histograms, so it can be a bit tricky to do efficiently, although perhaps local memory will help again to send data between neighboring DCT blocks). And I also haven't properly done bounds checking in the encoder or decoder, but it should hopefully be simple as long as we're willing to accept that evil input decodes into garbage instead of flagging errors explicitly. It also depends on a GLSL extension that my Haswell laptop doesn't have to get 64-bit divides when precalculating the rANS tables; I've got some code to simulate 64-bit divides using 32-bit, but it doesn't work yet. The code as it currently stands is in this Git repository; you can consider it licensed under GPLv3. It's really not very user-friendly at this point, though, and rather rough around the edges. Next time, we'll wrap up with some performance numbers. Unless I magically get more spare time in the meantime and/or some epiphany about how to make the encoder faster. :-)

Antoine Beaupr : October 2017 report: LTS, feed2exec beta, pandoc filters, git mediawiki

Debian Long Term Support (LTS) This is my monthly Debian LTS report. This time I worked on the famous KRACK attack, git-annex, golang and the continuous stream of GraphicsMagick security issues.

WPA & KRACK update I spent most of my time this month on the Linux WPA code, to backport it to the old (~2012) wpa_supplicant release. I first published a patchset based on the patches shipped after the embargo for the oldstable/jessie release. After feedback from the list, I also built packages for i386 and ARM. I have also reviewed the WPA protocol to make sure I understood the implications of the changes required to backport the patches. For example, I removed the patches touching the WNM sleep mode code as that was introduced only in the 2.0 release. Chunks of code regarding state tracking were also not backported as they are part of the state tracking code introduced later, in 3ff3323. Finally, I still have concerns about the nonce setup in patch #5. In the last chunk, you'll notice peer->tk is reset, to_set to negotiate a new TK. The other approach I considered was to backport 1380fcbd9f ("TDLS: Do not modify RNonce for an TPK M1 frame with same INonce") but I figured I would play it safe and not introduce further variations. I should note that I share Matthew Green's observations regarding the opacity of the protocol. Normally, network protocols are freely available and security researchers like me can easily review them. In this case, I would have needed to read the opaque 802.11i-2004 pdf which is behind a TOS wall at the IEEE. I ended up reading up on the IEEE_802.11i-2004 Wikipedia article which gives a simpler view of the protocol. But it's a real problem to see such critical protocols developed behind closed doors like this. At Guido's suggestion, I sent the final patch upstream explaining the concerns I had with the patch. I have not, at the time of writing, received any response from upstream about this, unfortunately. I uploaded the fixed packages as DLA 1150-1 on October 31st.

Git-annex The next big chunk on my list was completing the work on git-annex (CVE-2017-12976) that I started in August. It turns out doing the backport was simpler than I expected, even with my rusty experience with Haskell. Type-checking really helps in doing the right thing, especially considering how Joey Hess implemented the fix: by introducing a new type. So I backported the patch from upstream and notified the security team that the jessie and stretch updates would be similarly easy. I shipped the backport to LTS as DLA-1144-1. I also shared the updated packages for jessie (which required a similar backport) and stretch (which didn't) and those Sebastien Delafond published those as DSA 4010-1.

Graphicsmagick Up next was yet another security vulnerability in the Graphicsmagick stack. This involved the usual deep dive into intricate and sometimes just unreasonable C code to try and fit a round tree in a square sinkhole. I'm always unsure about those patches, but the test suite passes, smoke tests show the vulnerability as fixed, and that's pretty much as good as it gets. The announcement (DLA 1154-1) turned out to be a little special because I had previously noticed that the penultimate announcement (DLA 1130-1) was never sent out. So I made a merged announcement to cover both instead of re-sending the original 3 weeks late, which may have been confusing for our users.

Triage & misc We always do a bit of triage even when not on frontdesk duty, so I: I also did smaller bits of work on: The latter reminded me of the concerns I have about the long-term maintainability of the golang ecosystem: because everything is statically linked, an update to a core library (say the SMTP library as in CVE-2017-15042, thankfully not affecting LTS) requires a full rebuild of all packages including the library in all distributions. So what would be a simple update in a shared library system could mean an explosion of work on statically linked infrastructures. This is a lot of work which can definitely be error-prone: as I've seen in other updates, some packages (for example the Ruby interpreter) just bit-rot on their own and eventually fail to build from source. We would also have to investigate all packages to see which one include the library, something which we are not well equipped for at this point. Wheezy was the first release shipping golang packages but at least it's shipping only one... Stretch has shipped with two golang versions (1.7 and 1.8) which will make maintenance ever harder in the long term.
We build our computers the way we build our cities--over time, without a plan, on top of ruins. - Ellen Ullman

Other free software work This month again, I was busy doing some serious yak shaving operations all over the internet, on top of publishing two of my largest LWN articles to date (2017-10-16-strategies-offline-pgp-key-storage and 2017-10-26-comparison-cryptographic-keycards).

feed2exec beta Since I announced this new project last month I have released it as a beta and it entered Debian. I have also wrote useful plugins like the wayback plugin that saves pages on the Wayback machine for eternal archival. The archive plugin can also similarly save pages to the local filesystem. I also added bash completion, expanded unit tests and documentation, fixed default file paths and a bunch of bugs, and refactored the code. Finally, I also started using two external Python libraries instead of rolling my own code: the pyxdg and requests-file libraries, the latter which I packaged in Debian (and fixed a bug in their test suite). The program is working pretty well for me. The only thing I feel is really missing now is a retry/fail mechanism. Right now, it's a little brittle: any network hiccup will yield an error email, which are readable to me but could be confusing to a new user. Strangely enough, I am particularly having trouble with (local!) DNS resolution that I need to look into, but that is probably unrelated with the software itself. Thankfully, the user can disable those with --loglevel=ERROR to silence WARNINGs. Furthermore, some plugins still have some rough edges. For example, The Transmission integration would probably work better as a distinct plugin instead of a simple exec call, because when it adds new torrents, the output is totally cryptic. That plugin could also leverage more feed parameters to save different files in different locations depending on the feed titles, something would be hard to do safely with the exec plugin now. I am keeping a steady flow of releases. I wish there was a way to see how effective I am at reaching out with this project, but unfortunately GitLab doesn't provide usage statistics... And I have received only a few comments on IRC about the project, so maybe I need to reach out more like it says in the fine manual. Always feels strange to have to promote your project like it's some new bubbly soap... Next steps for the project is a final review of the API and release production-ready 1.0.0. I am also thinking of making a small screencast to show the basic capabilities of the software, maybe with asciinema's upcoming audio support?

Pandoc filters As I mentioned earlier, I dove again in Haskell programming when working on the git-annex security update. But I also have a small Haskell program of my own - a Pandoc filter that I use to convert the HTML articles I publish on LWN.net into a Ikiwiki-compatible markdown version. It turns out the script was still missing a bunch of stuff: image sizes, proper table formatting, etc. I also worked hard on automating more bits of the publishing workflow by extracting the time from the article which allowed me to simply extract the full article into an almost final copy just by specifying the article ID. The only thing left is to add tags, and the article is complete. In the process, I learned about new weird Haskell constructs. Take this code, for example: 
-- remove needless blockquote wrapper around some tables
--
-- haskell newbie tips:
--
-- @ is the "at-pattern", allows us to define both a name for the
-- construct and inspect the contents as once
--
--   is the "empty record pattern": it basically means "match the
-- arguments but ignore the args"
cleanBlock (BlockQuote t@[Table  ]) = t
Here the idea is to remove <blockquote> elements needlessly wrapping a <table>. I can't specify the Table type on its own, because then I couldn't address the table as a whole, only its parts. I could reconstruct the whole table bits by bits, but it wasn't as clean. The other pattern was how to, at last, address multiple string elements, which was difficult because Pandoc treats spaces specially: 
cleanBlock (Plain (Strong (Str "Notifications":Space:Str "for":Space:Str "all":Space:Str "responses":_):_)) = []
The last bit that drove me crazy was the date parsing: 
-- the "GAByline" div has a date, use it to generate the ikiwiki dates
--
-- this is distinct from cleanBlock because we do not want to have to
-- deal with time there: it is only here we need it, and we need to
-- pass it in here because we do not want to mess with IO (time is I/O
-- in haskell) all across the function hierarchy
cleanDates :: ZonedTime -> Block -> [Block]
-- this mouthful is just the way the data comes in from
-- LWN/Pandoc. there could be a cleaner way to represent this,
-- possibly with a record, but this is complicated and obscure enough.
cleanDates time (Div (_, [cls], _)
                 [Para [Str month, Space, Str day, Space, Str year], Para _])
    cls == "GAByline" = ikiwikiRawInline (ikiwikiMetaField "date"
                                           (iso8601Format (parseTimeOrError True defaultTimeLocale "%Y-%B-%e,"
                                                           (year ++ "-" ++ month ++ "-" ++ day) :: ZonedTime)))
                        ++ ikiwikiRawInline (ikiwikiMetaField "updated"
                                             (iso8601Format time))
                        ++ [Para []]
-- other elements just pass through
cleanDates time x = [x]
Now that seems just dirty, but it was even worse before. One thing I find difficult in adapting to coding in Haskell is that you need to take the habit of writing smaller functions. The language is really not well adapted to long discourse: it's more about getting small things connected together. Other languages (e.g. Python) discourage this because there's some overhead in calling functions (10 nanoseconds in my tests, but still), whereas functions are a fundamental and important construction in Haskell that are much more heavily optimized. So I constantly need to remind myself to split things up early, otherwise I can't do anything in Haskell. Other languages are more lenient, which does mean my code can be more dirty, but I feel get things done faster then. The oddity of Haskell makes frustrating to work with. It's like doing construction work but you're not allowed to get the floor dirty. When I build stuff, I don't mind things being dirty: I can cleanup afterwards. This is especially critical when you don't actually know how to make things clean in the first place, as Haskell will simply not let you do that at all. And obviously, I fought with Monads, or, more specifically, "I/O" or IO in this case. Turns out that getting the current time is IO in Haskell: indeed, it's not a "pure" function that will always return the same thing. But this means that I would have had to change the signature of all the functions that touched time to include IO. I eventually moved the time initialization up into main so that I had only one IO function and moved that timestamp downwards as simple argument. That way I could keep the rest of the code clean, which seems to be an acceptable pattern. I would of course be happy to get feedback from my Haskell readers (if any) to see how to improve that code. I am always eager to learn.

Git remote MediaWiki Few people know that there is a MediaWiki remote for Git which allow you to mirror a MediaWiki site as a Git repository. As a disaster recovery mechanism, I have been keeping such a historical backup of the Amateur radio wiki for a while now. This originally started as a homegrown Python script to also convert the contents in Markdown. My theory then was to see if we could switch from Mediawiki to Ikiwiki, but it took so long to implement that I never completed the work. When someone had the weird idea of renaming a page to some impossible long name on the wiki, my script broke. I tried to look at fixing it and then remember I also had a mirror running using the Git remote. It turns out it also broke on the same issue and that got me looking in the remote again. I got lost in a zillion issues, including fixing that specific issue, but I especially looked at the possibility of fetching all namespaces because I realized that the remote fetches only a part of the wiki by default. And that drove me to submit namespace support as a patch to the git mailing list. Finally, the discussion came back to how to actually maintain that contrib: in git core or outside? Finally, it looks like I'll be doing some maintenance that project outside of git, as I was granted access to the GitHub organisation...

Galore Yak Shaving Then there's the usual hodgepodge of fixes and random things I did over the month.
There is no [web extension] only XUL! - Inside joke

1 November 2017

Steve Kemp: Possibly retiring blogspam.net

For the past few years I've hosted a service for spam-testing blog/forum comments, and I think it is on the verge of being retired. The blogspam.net service presented a simple API for deciding whether an incoming blog/forum comment was SPAM, in real-time. I used it myself for two real reasons: As a result of the Debian-Administration server cleanup I'm still in the process of tidying up virtual machines, and servers. It crossed my mind that retiring this spam-service would allow me to free up another host. Initially the service was coded in Perl using XML/RPC. The current version of the software, version 2, is written as a node.js service, and despite the async-nature of the service it is still too heavy-weight to live on the host which runs most of my other websites. It was suggested to me that rewriting it in golang might allow it to process more requests, with fewer resources, so I started reimplementing the service in golang at 4AM this morning: The service does the minimum: I've ported several plugins, I've got 100% test-coverage of those plugins, and the service seems to be faster than the node.js version - so there is hope. Of course the real test will be when it is deployed for real. If it holds up for a few days I'll leave it running. Otherwise the retirement notice I placed on the website, which chances are nobody will see, will be true. The missing feature at the moment is keeping track of the count of spam-comments rejected/accepted on a per-site basis. Losing that information might be a shame, but I think I'm willing to live with it, if the alternative is closing down..

Jonathan Dowland: In defence of "Thought for the Day"

BBC Radio 4's long-running "Thought for the Day" has been in the news this week, as several presenters for the Today Programme have criticised the segment in an interview with Radio Times magazine. One facet of the criticism was whether the BBC should be broadcasting three minutes of religious content daily when more than half of the population are atheist. I'm an atheist and in my day-to-day life I have almost zero interaction with people of faith, certainly none where faith is a topic of conversation. However when I was an undergrad at Durham, I was a member of St John's College which has a Christian/Anglican/Evangelical heritage, and I met a lot of religious friends during my time there. What I find a little disturbing about the lack of faithful people in my day-to-day life, compared to then, is how it shines a light on how disjoint our society is. This has become even more apparent with the advent of the "filter bubble" and how irreconcilable factions are around topics like Brexit, Trump, etc. For these reasons I appreciate Thought for the Day and hearing voices from communities that I normally have little to do with. I can agree with the complaints about the lack of diversity, and I particularly enjoy hearing Thoughts from Muslims, Sikhs, Hindus, Buddhists, and such. Another criticism levelled against the segment was that it can be "preachy". I haven't found that myself. I get the impression that most of the monologues are carefully constructed to be as unpreaching as possible. I can usually appreciate the ethical content of the talks, without having to buy into the faith aspect. Interestingly the current principal of St John's College, David Wilkinson, has written his own response to the interview for the Radio Times.

Petter Reinholdtsen: Some notes on fault tolerant storage systems

If you care about how fault tolerant your storage is, you might find these articles and papers interesting. They have formed how I think of when designing a storage system. Several of these research papers are based on data collected from hundred thousands or millions of disk, and their findings are eye opening. The short story is simply do not implicitly trust RAID or redundant storage systems. Details matter. And unfortunately there are few options on Linux addressing all the identified issues. Both ZFS and Btrfs are doing a fairly good job, but have legal and practical issues on their own. I wonder how cluster file systems like Ceph do in this regard. After all, there is an old saying, you know you have a distributed system when the crash of a compyter you have never heard of stops you from getting any work done. The same holds true if fault tolerance do not work. Just remember, in the end, it do not matter how redundant, or how fault tolerant your storage is, if you do not continuously monitor its status to detect and replace failed disks.

Next.

Previous.