Search Results: "sre"

30 August 2020

Bits from Debian: DebConf20 online closes

DebConf20 group photo - click to enlarge On Saturday 29 August 2020, the annual Debian Developers and Contributors Conference came to a close. DebConf20 has been held online for the first time, due to the coronavirus (COVID-19) disease pandemic. All of the sessions have been streamed, with a variety of ways of participating: via IRC messaging, online collaborative text documents, and video conferencing meeting rooms. With more than 850 attendees from 80 different countries and a total of over 100 event talks, discussion sessions, Birds of a Feather (BoF) gatherings and other activities, DebConf20 was a large success. When it became clear that DebConf20 was going to be an online-only event, the DebConf video team spent much time over the next months to adapt, improve, and in some cases write from scratch, technology that would be required to make an online DebConf possible. After lessons learned from the MiniDebConfOnline in late May, some adjustments were made, and then eventually we came up with a setup involving Jitsi, OBS, Voctomix, SReview, nginx, Etherpad, and a newly written web-based frontend for voctomix as the various elements of the stack. All components of the video infrastructure are free software, and the whole setup is configured through their public ansible repository. The DebConf20 schedule included two tracks in other languages than English: the Spanish language MiniConf, with eight talks in two days, and the Malayalam language MiniConf, with nine talks in three days. Ad-hoc activities, introduced by attendees over the course of the entire conference, have been possible too, streamed and recorded. There have also been several team gatherings to sprint on certain Debian development areas. Between talks, the video stream has been showing the usual sponsors on the loop, but also some additional clips including photos from previous DebConfs, fun facts about Debian and short shout-out videos sent by attendees to communicate with their Debian friends. For those who were not able to participate, most of the talks and sessions are already available through the Debian meetings archive website, and the remaining ones will appear in the following days. The DebConf20 website will remain active for archival purposes and will continue to offer links to the presentations and videos of talks and events. Next year, DebConf21 is planned to be held in Haifa, Israel, in August or September. DebConf is committed to a safe and welcome environment for all participants. During the conference, several teams (Front Desk, Welcome team and Community team) have been available to help so participants get their best experience in the conference, and find solutions to any issue that may arise. See the web page about the Code of Conduct in DebConf20 website for more details on this. Debian thanks the commitment of numerous sponsors to support DebConf20, particularly our Platinum Sponsors: Lenovo, Infomaniak, Google and Amazon Web Services (AWS). About Debian The Debian Project was founded in 1993 by Ian Murdock to be a truly free community project. Since then the project has grown to be one of the largest and most influential open source projects. Thousands of volunteers from all over the world work together to create and maintain Debian software. Available in 70 languages, and supporting a huge range of computer types, Debian calls itself the universal operating system. About DebConf DebConf is the Debian Project's developer conference. In addition to a full schedule of technical, social and policy talks, DebConf provides an opportunity for developers, contributors and other interested people to meet in person and work together more closely. It has taken place annually since 2000 in locations as varied as Scotland, Argentina, and Bosnia and Herzegovina. More information about DebConf is available from https://debconf.org/. About Lenovo As a global technology leader manufacturing a wide portfolio of connected products, including smartphones, tablets, PCs and workstations as well as AR/VR devices, smart home/office and data center solutions, Lenovo understands how critical open systems and platforms are to a connected world. About Infomaniak Infomaniak is Switzerland's largest web-hosting company, also offering backup and storage services, solutions for event organizers, live-streaming and video on demand services. It wholly owns its datacenters and all elements critical to the functioning of the services and products provided by the company (both software and hardware). About Google Google is one of the largest technology companies in the world, providing a wide range of Internet-related services and products such as online advertising technologies, search, cloud computing, software, and hardware. Google has been supporting Debian by sponsoring DebConf for more than ten years, and is also a Debian partner sponsoring parts of Salsa's continuous integration infrastructure within Google Cloud Platform. About Amazon Web Services (AWS) Amazon Web Services (AWS) is one of the world's most comprehensive and broadly adopted cloud platforms, offering over 175 fully featured services from data centers globally (in 77 Availability Zones within 24 geographic regions). AWS customers include the fastest-growing startups, largest enterprises and leading government agencies. Contact Information For further information, please visit the DebConf20 web page at https://debconf20.debconf.org/ or send mail to press@debian.org.

28 July 2020

Chris Lamb: Pop culture matters

Many people labour under the assumption that pop culture is trivial and useless while only 'high' art can grant us genuine and eternal knowledge about the world. Given that we have a finite time on this planet, we are all permitted to enjoy pop culture up to a certain point, but we should always minimise our interaction with it, and consume more moral and intellectual instruction wherever possible. Or so the theory goes. What these people do not realise is that pop and mass culture can often provide more information about the world, humanity in general and what is even more important ourselves. This is not quite the debate around whether high art is artistically better, simply that pop culture can be equally informative. Jeremy Bentham argued in the 1820s that "prejudice apart, the game of push-pin is of equal value with the arts and sciences of music and poetry", that it didn't matter where our pleasures come from. (John Stuart Mill, Bentham's intellectual rival, disagreed.) This fundamental question of philosophical utilitarianism will not be resolved here. However, what might begin to be resolved is our instinctive push-back against pop culture. We all share an automatic impulse to disregard things we do not like and to pretend they do not exist, but this wishful thinking does not mean that these cultural products do not continue to exist when we aren't thinking about them and, more to our point, continue to influence others and even ourselves. Take, for example, the recent trend for 'millennial pink'. With its empty consumerism, faux nostalgia, reductive generational stereotyping, objectively ugly sthetics and tedious misogyny (photographed with Rose Gold iPhones), the very combination appears to have been deliberately designed to annoy me, curiously providing circumstantial evidence in favour of intelligent design. But if I were to immediately dismiss millennial pink and any of the other countless cultural trends I dislike simply because I find them disagreeable, I would be willingly keeping myself blind to their underlying ideology, their significance and their effect on society at large. If I had any ethical or political reservations I might choose not to engage with them economically or to avoid advertising them to others, but that is a different question altogether. Even if we can't notice this pattern within ourselves we can first observe it in others. We can all recall moments where someone has brushed off a casual reference to pop culture, be it Tiger King, TikTok, team sports or Taylor Swift; if you can't, simply look for the abrupt change of tone and the slightly-too-quick dismissal. I am not suggesting you attempt to dissuade others or even to point out this mental tic, but merely seeing it in action can be highly illustrative in its own way. In summary, we can simultaneously say that pop culture is not worthy of our time relative to other pursuits while consuming however much of it we want, but deliberately dismissing pop culture doesn't mean that a lot of other people are not interacting with it and is therefore undeserving of any inquiry. And if that doesn't convince you, just like the once-unavoidable millennial pink, simply sticking our collective heads in the sand will not mean that wider societal-level ugliness is going to disappear anytime soon. Anyway, that's a very long way of justifying why I plan to re-watch TNG.

16 July 2020

Louis-Philippe V ronneau: DebConf Videoteam Sprint Report -- DebConf20@Home

DebConf20 starts in about 5 weeks, and as always, the DebConf Videoteam is working hard to make sure it'll be a success. As such, we held a sprint from July 9th to 13th to work on our new infrastructure. A remote sprint certainly ain't as fun as an in-person one, but we nonetheless managed to enjoy ourselves. Many thanks to those who participated, namely: We also wish to extend our thanks to Thomas Goirand and Infomaniak for providing us with virtual machines to experiment on and host the video infrastructure for DebConf20. Advice for presenters For DebConf20, we strongly encourage presenters to record their talks in advance and send us the resulting video. We understand this is more work, but we think it'll make for a more agreeable conference for everyone. Video conferencing is still pretty wonky and there is nothing worse than a talk ruined by a flaky internet connection or hardware failures. As such, if you are giving a talk at DebConf this year, we are asking you to read and follow our guide on how to record your presentation. Fear not: we are not getting rid of the Q&A period at the end of talks. Attendees will ask their questions either on IRC or on a collaborative pad and the Talkmeister will relay them to the speaker once the pre-recorded video has finished playing. New infrastructure, who dis? Organising a virtual DebConf implies migrating from our battle-tested on-premise workflow to a completely new remote one. One of the major changes this means for us is the addition of Jitsi Meet to our infrastructure. We normally have 3 different video sources in a room: two cameras and a slides grabber. With the new online workflow, directors will be able to play pre-recorded videos as a source, will get a feed from a Jitsi room and will see the audience questions as a third source. This might seem simple at first, but is in fact a very major change to our workflow and required a lot of work to implement.
               == On-premise ==                                          == Online ==
                                                      
              Camera 1                                                 Jitsi
                                                                          
                 v                 ---> Frontend                         v                 ---> Frontend
                                                                                            
    Slides -> Voctomix -> Backend -+--> Frontend         Questions -> Voctomix -> Backend -+--> Frontend
                                                                                            
                 ^                 ---> Frontend                         ^                 ---> Frontend
                                                                          
              Camera 2                                           Pre-recorded video
In our tests, playing back pre-recorded videos to voctomix worked well, but was sometimes unreliable due to inconsistent encoding settings. Presenters will thus upload their pre-recorded talks to SReview so we can make sure there aren't any obvious errors. Videos will then be re-encoded to ensure a consistent encoding and to normalise audio levels. This process will also let us stitch the Q&As at the end of the pre-recorded videos more easily prior to publication. Reducing the stream latency One of the pitfalls of the streaming infrastructure we have been using since 2016 is high video latency. In a worst case scenario, remote attendees could get up to 45 seconds of latency, making participation in events like BoFs arduous. In preparation for DebConf20, we added a new way to stream our talks: RTMP. Attendees will thus have the option of using either an HLS stream with higher latency or an RTMP stream with lower latency. Here is a comparative table that can help you decide between the two protocols:
HLS RTMP
Pros
  • Can be watched from a browser
  • Auto-selects a stream encoding
  • Single URL to remember
  • Lower latency (~5s)
Cons
  • Higher latency (up to 45s)
  • Requires a dedicated video player (VLC, mpv)
  • Specific URLs for each encoding setting
Live mixing from home with VoctoWeb Since DebConf16, we have been using voctomix, a live video mixer developed by the CCC VOC. voctomix is conveniently divided in two: voctocore is the backend server while voctogui is a GTK+ UI frontend directors can use to live-mix. Although voctogui can connect to a remote server, it was primarily designed to run either on the same machine as voctocore or on the same LAN. Trying to use voctogui from a machine at home to connect to a voctocore running in a datacenter proved unreliable, especially for high-latency and low bandwidth connections. Inspired by the setup FOSDEM uses, we instead decided to go with a web frontend for voctocore. We initially used FOSDEM's code as a proof of concept, but quickly reimplemented it in Python, a language we are more familiar with as a team. Compared to the FOSDEM PHP implementation, voctoweb implements A / B source selection (akin to voctogui) as well as audio control, two very useful features. In the following screen captures, you can see the old PHP UI on the left and the new shiny Python one on the right. The old PHP voctowebThe new Python3 voctoweb Voctoweb is still under development and is likely to change quite a bit until DebConf20. Still, the current version seems to works well enough to be used in production if you ever need to. Python GeoIP redirector We run multiple geographically-distributed streaming frontend servers to minimize the load on our streaming backend and to reduce overall latency. Although users can connect to the frontends directly, we typically point them to live.debconf.org and redirect connections to the nearest server. Sadly, 6 months ago MaxMind decided to change the licence on their GeoLite2 database and left us scrambling. To fix this annoying issue, Stefano Rivera wrote a Python program that uses the new database and reworked our ansible frontend server role. Since the new database cannot be redistributed freely, you'll have to get a (free) license key from MaxMind if you to use this role. Ansible & CI improvements Infrastructure as code is a living process and needs constant care to fix bugs, follow changes in DSL and to implement new features. All that to say a large part of the sprint was spent making our ansible roles and continuous integration setup more reliable, less buggy and more featureful. All in all, we merged 26 separate ansible-related merge request during the sprint! As always, if you are good with ansible and wish to help, we accept merge requests on our ansible repository :)

15 July 2020

Antoine Beaupr : Goatcounter analytics in ikiwiki

I have started using Goatcounter for analytics after reading this LWN article called "Lightweight alternatives to Google Analytics". Goatcounter has an interesting approach to privacy in that it:
tracks sessions using a hash of the browser's user agent and IP address to identify the client without storing any personal information. The salt used to generate these hashes is rotated every 4 hours with a sliding window.
There was no Debian package for the project, so I filed a request for package and instead made a fork of the project to add a Docker image. This page documents how Goatcounter was setup from there...

Server configuration
  1. build the image from this fork
    docker build -t zgoat/goatcounter .
    
  2. create volume for db:
    docker volume create goatcounter
    
  3. start the server:
    exec docker run --restart=unless-stopped --volume="goatcounter:/home/user/db/" --publish 127.0.0.1:8081:8080 --detach zgoat/goatcounter serve -listen :8080 -tls none
    
  4. apache configuration:
    <VirtualHost *:80>
                ServerName analytics.anarc.at
                Redirect / https://analytics.anarc.at/
                DocumentRoot /var/www/html/
        </VirtualHost>
    <VirtualHost *:443>
            ServerName analytics.anarc.at
            Use common-letsencrypt-ssl analytics.anarc.at
            DocumentRoot /var/www/html/
            ProxyPass /.well-known/ !
            ProxyPass / http://localhost:8081/
            ProxyPassReverse / http://localhost:8081/
            ProxyPreserveHost on
    </VirtualHost>
    
  5. add analytics.anarc.at to DNS
  6. create a TLS cert with LE:
    certbot certonly --webroot  -d analytics.anarc.at --webroot-path /var/www/html/
    
    note that goatcounter has code to do this on its own, but we avoid it to follow our existing policies and simplify things
  7. create site:
    docker run -it --rm --volume="goatcounter:/home/user/db/" zgoat/goatcounter create -domain analytics.anarc.at -email anarcat+rapports@anarc.at
    
  8. add to ikiwiki template
  9. rebuild wiki:
    ikiwiki --setup ikiwiki.setup --rebuild --verbose
    

Remaining issues
  • Docker image should be FROM scratch, this is statically built golang stuff after all...
  • move to Docker Compose or podman instead of just starting the thing by hand
  • this is all super janky and should be put in config management somehow
  • remove "anarc.at" test site (the site is the analytics site, not the tracked site), seems like this is not possible yet
  • do log parsing instead of Javascript or 1x1 images?
  • compare with goaccess logs, probably at the end of july, to have two full weeks to compare

Fixed issues
  • cache headers are wrong (120ms!) deployed workaround in apache, reported as a bug upstream
  • remove self-referer done, just a matter of configuring the URL in the settings. could this be automated too?
  • add pixel tracking for noscript users done, but required a patch to ikiwi (and I noticed another bug while doing it)
  • goatcounter monitor doesn't with sqlite (fixed upstream!)
  • the :8080 port leaks in some places, namely in the "Site config" documentation that is because i was using -port 8080 which was not necessary.

3 June 2020

Debian Project Leader: DPL Activity logs for April/May 2020

First month as DPL I survived my first month as DPL! I agree with previous DPLs who have described it as starting a whole new job. Fortunately it wasn't very stressful, but it certainly was very time consuming. On the very first day my inbox exploded with requests. I dealt with this by deferring anything that wasn't important right away and just started working through it. Fortunately the initial swell subsided as the month progressed. The bulk of my remaining e-mail backlog are a few media outlets who wants to do interviews. I'll catch up with those during this month. Towards the end of the month, most of my focus was on helping to prepare for an online MiniDebConf that we hosted over the last weekend in May. We had lots of fun and we had some great speakers sharing their knowledge and expertise during the weekend. Activity log As I do on my own blog for free software activities, I'll attempt to keep a log of DPL activities on this blog. Here's the log for the period 2020-04-21 to 2020-05-21: 2020-04-19: Handover session with Sam, our outgoing DPL. We covered a lot of questions I had and main areas that the DPL works in. Thanks to Sam for having taken the time to do this. 2020-04-21: First day of term! Thank you to everybody who showed support and have offered to help! 2020-04-21: Request feedback from the trademark team on an ongoing trademark dispute. 2020-04-21: Join the GNOME Advisory Board as a representative from Debian. 2020-04-21: Reply on an ongoing community conflict issue. 2020-04-21: Update Debian project summary for SPI annual report. 2020-04-21: Received a great e-mail introduction from Debian France and followed up on that. 2020-04-21: Posted "Bits from the new DPL" to debian-devel-announce. 2020-04-22: Became Debian's OSI Affilliate representative. 2020-04-22: Reply to a bunch of media inquiries for interviews, will do them later when initial priorities are on track. 2020-04-23: Resign as Debian FTP team trainee and mailing list moderator. In both these areas there are enough people taking care of it and I intend to maximise available time for DPL and technical areas in the project. 2020-04-25: Review outgoing mail for trademark team. 2020-04-25: Answer some questions in preparation for DAM/Front Desk delegation updates. 2020-04-26: Initiated wiki documentation for delegation updates process. 2020-04-27: Update delegation for the Debian Front Desk team. 2020-04-29: Schedule video call with Debian treasurer team. 2020-04-29: OSI affiliate call. Learned about some Open Source projects including OpenDev, OpenSourceMatters, FOSS Responders and Democracy Labs. 2020-05-04: Delivered my first talk session as DPL titled "Mixed Bag of Debian" at "Fique Em Casa Use Debian" (Stay at home and use Debian!), organised by Debian Brazil, where they had a different talk every evening during the month of May. Great initiative I hope other local groups consider copying their ideas! 2020-05-05: Had a 2 hour long call with the treasurer team. Feeling optimistic for the future of Debian's financing although it will take some time and a lot of work to get where we want to be. 2020-05-17: Respond to cloud delegation update.

Keith Packard: picolibc-ryu

Float/String Conversion in Picolibc: Enter Ry I recently wrote about this topic having concluded that the best route for now was to use the malloc-free, but imprecise, conversion routines in the tinystdio alternative. A few days later, Sreepathi Pai pointed me at some very recent work in this area: This is amazing! Thirty years after the papers referenced in the previous post, Ulf Adams came up with some really cool ideas and managed to reduce the math required for 64-bit conversion to 128 bit integers. This is a huge leap forward; we were doing long multi-precision computations before, and now it's all short enough to fit in registers (ok, a lot of registers, but still). Getting the Ry Code The code is available on github: https://github.com/ulfjack/ryu. Reading through it, it's very clear that the author focuses on performance with lots of tuning for common cases. Still, it's quite readable, especially compared with the newlib multi-precision based code. Picolibc String/Float conversion interface Picolibc has some pretty basic needs for the float/string conversion code, it wants four functions:
  1. __dtoa_engine
    int
    __dtoa_engine(double x, struct dtoa *dtoa, uint8_t max_digits, uint8_t max_decimals);
    
    This converts the double x to a string of decimal digits and a decimal exponent stored inside the 'dtoa' struct. It limits the total number of digits to max_digits and, optionally (when max_decimals is non-zero), limits the number of fractional digits to max_decimals - 1. This latter supports 'f' formats. Returns the number of digits stored, which is <= max_digits. Less if the number can be accurately represented in fewer digits.
  2. __ftoa_engine
    int
    __ftoa_engine(float x, struct ftoa *ftoa, uint8_t max_digits, uint8_t max_decimals);
    
    The same as __dtoa_engine, except for floats.
  3. __atod_engine
    double
    __atod_engine(uint64_t m10, int e10);
    
    To avoid needing to handle stdio inside the conversion function, __atod_engine receives fully parsed values, the base-10 significand (m10) and exponent (e10). The value to convert is m10 * pow(10, e10).
  4. __atof_engine
    float
    __atof_engine(uint32_t m10, int e10);
    
    The same as __atod_engine, except for floats.
With these, it can do printf, scanf, ecvt, fcvt, gcvt, strtod, strtof and atof. Porting Ry to Picolibc The existing Ry float-to-string code always generates the number of digits necessary for accurate output. I had to hack it up to generate correctly rounded shorter output when max_digits or max_decimals were smaller. I'm not sure I managed to do that correctly, but at least it appears to be passing all of the test cases I have. In normal operation, Ry iteratively removes digits from the answer that aren't necessary to disambiguate with neighboring values. What I changed was to keep removing digits using that method until the answer had few enough digits to fit in the desired length. There's some tricky rounding code that adjusts the final result and I had to bypass that if I'd removed extra digits. That was about the only change necessary to the core algorithm. I also trimmed the code to only include the general case and not the performance improvements, then wrapped it with code to provide the _engine interface. On the string-to-float side, most of what I needed to do was remove the string parsing bits at the start of the function and switch from performance-optimized to space-optimized versions of a couple of internal routines. Correctness Results Because these new functions are now 'exact', I was able to adjust the picolibc tests to compare all of the bits for string/float conversion instead of having to permit a bit of slop in the answers. With those changes, the picolibc test suite passes, which offers some assurance that things aren't completely broken. Size Results Snek uses the 32-bit float versions of the conversion routines, and for that, the size difference is:
   text    data     bss     dec     hex filename
  59068      44   37968   97080   17b38 snek-qemu-riscv-orig.elf
  59430      44   37968   97442   17ca2 snek-qemu-riscv-ryu.elf
    362
362 bytes added to gain accurate printf/strtof results seems like a good trade-off in this case. Performance I haven't measured performance at all, but I suspect that it won't be nearly as problematic on most platforms as the source code makes it appear. And that's because Ry is entirely integer arithmetic with no floating point at all. This avoids using the soft fp code for platforms without hardware float support. Pointers to the Code I haven't merged this to picolibc master yet, it's on the ryu branch: Review, especially of the hack above to return short results, would be greatly appreciated! Thanks again to Ulf Adams for creating this code and to Sreepathi Pai for sending me a note about it!

30 May 2020

Sean Whitton: GNU Emacs' Transient Mark mode

Something I ve found myself doing as the pandemic rolls on is picking out and (re-)reading through sections of the GNU Emacs manual and the GNU Emacs Lisp reference manual. This has got me (too) interested in some of the recent history of Emacs development, and I did some digging into archives of emacs-devel from 2008 (15M mbox) regarding the change to turn Transient Mark mode on by default and set mark-even-if-inactive to true by default in Emacs 23.1. It s not always clear which objections to turning on Transient Mark mode by default take into account the mark-even-if-inactive change. I think that turning on Transient Mark mode along with mark-even-if-inactive is a good default. The question that remains is whether the disadvantages of Transient Mark mode are significant enough that experienced Emacs users should consider altering Emacs default behaviour to mitigate them. Here s one popular blog arguing for some mitigations. How might Transient Mark mode be disadvantageous? The suggestion is that it makes using the mark for navigation rather than for acting on regions less convenient:
  1. setting a mark just so you can jump back to it (i) is a distinct operation you have to think of separately; and (ii) requires two keypresses, C-SPC C-SPC, rather than just one keypress
  2. using exchange-point-and-mark activates the region, so to use it for navigation you need to use either C-u C-x C-x or C-x C-x C-g, neither of which are convenient to type, or else it will be difficult to set regions at the place you ve just jumped to because you ll already have one active.
There are two other disadvantages that people bring up which I am disregarding. The first is that it makes it harder for new users to learn useful ways in which to use the mark when it s deactivated. This happened to me, but it can be mitigated without making any behavioural changes to Emacs. The second is that the visual highlighting of the region can be distracting. So far as I can tell, this is only a problem with exchange-point-and-mark, and it s subsumed by the problem of that command actually activating the region. The rest of the time Emacs automatic deactivation of the region seems sufficient. How might disabling Transient Mark mode be disadvantageous? When Transient Mark mode is on, many commands will do something usefully different when the mark is active. The number of commands in Emacs which work this way is only going to increase now that Transient Mark mode is the default. If you disable Transient Mark mode, then to use those features you need to temporarily activate Transient Mark mode. This can be fiddly and/or require a lot of keypresses, depending on exactly where you want to put the region. Without being able to see the region, it might be harder to know where it is. Indeed, this is one of the main reasons for wanting Transient Mark mode to be the default, to avoid confusing new users. I don t think this is likely to affect experienced Emacs users often, however, and on occasions when more precision is really needed, C-u C-x C-x will make the region visible. So I m not counting this as a disadvantage. How might we mitigate these two sets of disadvantages? Here are the two middle grounds I m considering. Mitigation #1: Transient Mark mode, but hack C-x C-x behaviour
(defun spw/exchange-point-and-mark (arg)
  "Exchange point and mark, but reactivate mark a bit less often.

Specifically, invert the meaning of ARG in the case where
Transient Mark mode is on but the region is inactive."
  (interactive "P")
  (exchange-point-and-mark
   (if (and transient-mark-mode (not mark-active))
       (not arg)
     arg)))
(global-set-key [remap exchange-point-and-mark] &aposspw/exchange-point-and-mark)
We avoid turning Transient Mark mode off, but mitigate the second of the two disadvantages given above. I can t figure out why it was thought to be a good idea to make C-x C-x reactivate the mark and require C-u C-x C-x to use the action of exchanging point and mark as a means of navigation. There needs to a binding to reactivate the mark, but in roughly ten years of having Transient Mark mode turned on, I ve found that the need to reactivate the mark doesn t come up often, so the shorter and longer bindings seem the wrong way around. Not sure what I m missing here. Mitigation #2: disable Transient Mark mode, but enable it temporarily more often
(setq transient-mark-mode nil)
(defun spw/remap-mark-command (command &optional map)
  "Remap a mark-* command to temporarily activate Transient Mark mode."
  (let* ((cmd (symbol-name command))
         (fun (intern (concat "spw/" cmd)))
         (doc (concat "Call  "
                      cmd
                      "&apos and temporarily activate Transient Mark mode.")))
    (fset fun  (lambda ()
                 ,doc
                 (interactive)
                 (call-interactively #&apos,command)
                 (activate-mark)))
    (if map
        (define-key map (vector &aposremap command) fun)
      (global-set-key (vector &aposremap command) fun))))
(dolist (command &apos(mark-word
                   mark-sexp
                   mark-paragraph
                   mark-defun
                   mark-page
                   mark-whole-buffer))
  (spw/remap-mark-command command))
(with-eval-after-load &aposorg
  (spw/remap-mark-command &aposorg-mark-element org-mode-map)
  (spw/remap-mark-command &aposorg-mark-subtree org-mode-map))
;; optional
(global-set-key "\M-=" (lambda () (interactive) (activate-mark)))
;; resettle the previous occupant
(global-set-key "\C-cw" &aposcount-words-region)
Here we remove both of the disadvantages of Transient Mark mode given above, and mitigate the main disadvantage of not activating Transient Mark mode by making it more convenient to activate it temporarily. For example, this enables using C-M-SPC C-M-SPC M-( to wrap the following two function arguments in parentheses. And you can hit M-h a few times to mark some blocks of text or code, then operate on them with commands like M-% and C-/ which behave differently when the region is active.1 Comparing these mitigations Both of these mitigations handle the second of the two disadvantages of Transient Mark mode given above. What remains, then, is
  1. under the effects of mitigation #1, how much of a barrier to using marks for navigational purposes is it to have to press C-SPC C-SPC instead of having a single binding, C-SPC, for all manual mark setting2
  2. under the effects of mitigation #2, how much of a barrier to taking advantage of commands which act differently when the region is active is it to have to temporarily enable Transient Mark mode with C-SPC C-SPC, M-= or one of the mark-* commands?
These are unknowns.3 So I m going to have to experiment, I think, to determine which mitigation to use, if either. In particular, I don t know whether it s really significant that setting a mark for navigational purposes and for region marking purposes are distinct operations under mitigation #1. My plan is to start with mitigation #2 because that has the additional advantage of allowing me to confirm or disconfirm my belief that not being able to see where the region is will only rarely get in my way.

  1. The idea of making the mark-* commands activate the mark comes from an emacs-devel post by Stefan Monnier in the archives linked above.
  2. One remaining possibility I m not considering is mitigation #1 plus binding something else to do the same as C-SPC C-SPC. I don t believe there are any easily rebindable keys which are easier to type than typing C-SPC twice. And this does not deal with the two distinct mark-setting operations problem.
  3. Another way to look at this is the question of which of setting a mark for navigational purposes and activating a mark should get C-SPC and which should get C-SPC C-SPC.

2 May 2020

Jonathan Carter: Free Software Activities for 2020-04

Debian project leader This month contained the first week and a half or so of my term as Debian Project Leader. So far my focus has been getting up to speed and keeping the gears turning with day to day DPL tasks. The updates listed here will also be available on the DPL blog, where I aim to make more frequent updates. During May, Debian Brazil will host Debian talks throughout the month which they will stream to their YouTube channel. You can find the schedule in this git repository, most of the talks will be in Portuguese, but on the 4th of May at 21:00 UTC, I ll be covering some Debian project topics for an hour or so and take some questions if there s time left. 2020-04-19: Handover session with Sam, our outgoing DPL. We covered a lot of questions I had and main areas that the DPL works in. Thanks to Sam for having taken the time to do this. 2020-04-21: First day of term! Thank you to everybody who showed support and have offered to help! 2020-04-21: Request feedback from the trademark team on an ongoing trademark dispute. 2020-04-21: Join the GNOME Advisory Board as a representative from Debian. 2020-04-21: Reply on an ongoing community conflict issue. 2020-04-21: Update Debian project summary for SPI annual report. 2020-04-21: Received a great e-mail introduction from Debian France and followed up on that. 2020-04-21: Posted Bits from the new DPL to debian-devel-announce. 2020-04-22: Became Debian s OSI Affilliate representative. 2020-04-22: Reply to a bunch of media inquiries for interviews, will do them later when initial priorities are on track. 2020-04-23: Resign as Debian FTP team trainee and mailing list moderator. In both these areas there are enough people taking care of it and I intend to maximise available time for DPL and technical areas in the project. 2020-04-25: Review outgoing mail for trademark team. 2020-04-25: Answer some questions in preparation for DAM/Front Desk delegation updates. 2020-04-26: Initiated wiki documentation for delegation updates process. 2020-04-27: Update delegation for the Debian Front Desk team. 2020-04-29: Schedule video call with Debian treasurer team. 2020-04-29: OSI affiliate call. Learned about some Open Source projects including OpenDev, OpenSourceMatters, FOSS Responders and Democracy Labs.

Debian Social Work on the Debian Social project is progressing, we plan to start a separate blog syndicated to Planet Debian that contains progress and status updates. I ve been terrible at tracking the work we ve been doing on this, so for now, here are some micro updates:

MiniDebConf Online In the DebConf video team, we ve been wondering whether we have all the tools required to successfully host a DebConf (or even a mini DebConf) entirely online. There s really just one way to know for sure, so we re going to host MiniDebConf Online from 28 May to 31 May. The first two days will be an online MiniDebCamp, where we can try to hold online spints, meetings and general chit-chat. The last two days will be for talks and lightnight talks, with BoFs likely to take place throughout the 4 days (this will probably be decided once we have a content team). Announcements should go out within the next week, in the meantime, save the dates.

Debian package uploads 2020-04-07: Upload package flask-autoindex (0.6.6-1) to Debian unstable. 2020-04-07: Upload package gamemode (1.5.1-1) to Debian unstable. 2020-04-08: Accept MR#2 for live-tasks (add usbutils to live-task-standard). 2020-04-08: Upload package live-tasks (11.0.2) to Debian unstable (Closes: #955526, #944578, #942837, #942834). 2020-04-08: Close live-config bug #655198 (Only affects squeeze which is no longer supported). 2020-04-08: Upload package live-config (11.0.1) to Debian unstable. 2020-04-08: Upload package calamares (3.2.22-1) to Debian unstable. 2020-04-15: Upload package xabacus (8.2.6-1) to Debian unstable. 2020-04-16: Merge MR#1 for gnome-shell-extension-dashtodock (new upstream release). 2020-04-16: Upload package gnome-shell-extension-dashtodock (67+git20200225-1) to Debian unstable. 2020-04-16: Merge MR#1 for gnome-shell-extension-hard-disk-led (fix some lintian issues). 2020-04-16: Merge MR#1 for gnome-shell-extension-system-monitor (fix some lintian issues). 2020-04-17: Upload package calamares (3.2.23-1) to Debian unstable. 2020-04-17: Upload package catimg (2.6.0-1) to Debian unstable (Closes: #956150). 2020-04-17: Upload package fabulous (0.3.0+dfsg1-7) to Debian unstable (Closes: #952242). 2020-04-17: Upload package gnome-shell-extension-system-monitor (38+git20200414-32cc79e-1) to Debian unstable (Closes: #956656, #956171). 2020-04-17: Upload package gnome-shell-extension-arc-menu (45-1) to Debian unstable (Closes: #956168). 2020-04-18: Upload package toot (0.26.0-1) to Debian unstable. 2020-04-23: Update packaging for gnome-shell-extension-tilix-shortcut, upstream section needs some additional work before release is possible. 2020-04-23: Upload package xabacus (8.2.7-1) to Debian unstable. 2020-04-27: Upload package gnome-shell-extension-dash-to-panel (37-1) to Debian unstable (Closes: #956162, #954978). 2020-04-27: Upload package gnome-shell-extension-dash-to-panel (37-2) to Debian unstable. 2020-04-27: Upload package gnome-shell-extension-dashtodock (68-1) to Debian unstable. 2020-04-30: Merge MR#8 for gamemode (add symbol files) (Closes: #952425). 2020-04-30: Merge MR#9 for gamemode (reduce number of -dev packages generated). 2020-04-30: Merge MR#10 for gamemode (deal better with upgrades on a buggy version). 2020-04-30: Manually merge changes from MR#11 for gamemode (packaging fixes). 2020-04-30: Upload package gamemode (1.5.1-2) to Debian unstable.

26 April 2020

Russ Allbery: Review: The Last Goodbye

Review: The Last Goodbye, by Matt Potter
Publisher: Silvertail
Copyright: 2014-2016
Printing: 2016
ISBN: 1-909269-42-5
Format: Kindle
Pages: 308
In the contested space between the interested amateur and the trained expert lies the enthusiast. A topic has, for often inexplicable reasons, caught fire in their thoughts and become a mild obsession. They may not have formal training or systematic conceptual grounding beneath their interest, but they partly make up for that lack with sustained fascination. They research widely and obscurely about their particular focus, develop novel theories, see distinctions and classifications that few others would bother to investigate, and present their discoveries to anyone who will stand still long enough. And occasionally, when that interest happens to coincide with writing skill, they produce some surprisingly excellent essays or books. Matt Potter's enthusiasm is resignation letters.
Every damaging resignation letter, every cornered truth attack, every out-of-control speech by a former friend, is more than just an inconvenience, to be countered with positive spin and internal memos: it's an open challenge to the official version of the story, the perfectly controlled brand. They are breaks in an otherwise perfectly planned, smoothly executed narrative of the powerful. Holes in the program code. A rare, irresistible chance to hack into history's shiny, unstoppable operation.
The Last Goodbye: A History of the World in Resignation Letters is not, in truth, a history of the world. It is, first and foremost, a taxonomy, because there are types of resignation letters. The opening chapter, the truth bomb, is the type that one would expect upon discovering that someone wrote a book on the topic (that wasn't advice on how to write a good one). But there are other types, less heavy on the fireworks but just as fascinating. The unquotable expert construction. The knife in the back. The incoherent scream of rage. But also the surprisingly gentle and graceful conclusion.
It is the question that the letters themselves try in vain to answer, over and over again even as they explain, analyse, protest and bear witness to a million other details. The question is: Why? All the forces in the universe stack up against unburdening ourselves in a resignation letter. Professionally, it can be suicide. In practical terms, it is often self-defeating. Self-help books coach against unleashing its force; colleagues and confidantes urge caution, self-restraint. And yet we do it, and damn the consequences. We have no choice but to speak in sorrow, love, grief, cold anger, thirst for revenge, wounded pride, the pain of injustice, loyalty, pangs of regret, throes of vengeful madness, deluded righteousness, panic, black distress, isolation, ecstasies of martyrdom, and a million other shades of human extremity we need to say our piece even as we leave the stage.
The risk of the enthusiast's book is that the lack of structural grounding can leave the conclusions unsupported. A fair critique of this book is that it contains a lot of amateur sociology. Potter has a journalist's eye for motive and narrative, but some of his conclusions may not be warranted. But he compensates for the lack of rigor with, well, enthusiasm. Potter is fascinated by resignation letters and the insight they offer, and that fascination is irresistibly contagious. It's probably obvious that the chapters on truth bombs, fuck yous, and knives in the back have visceral appeal. The resignation letter as a force of truth-telling, as the revenge of a disregarded peon, as a crack in the alliance between the powerful that may let some truth shine through, is what got me to buy this book. And Potter doesn't disappoint; he quotes from both famous and nearly unknown examples, dissects the writing and the intent, and gives us a ringside seat to a few effective acts of revenge. That's not the best part of this book, though. The chapter that I will remember the longest is Potter's dissection of the constructed resignation letter. The carefully drafted public relations statement, the bland formality, the attempt to make a news story disappear. The conversation-ender. It's a truism that any area of human endeavor involves more expertise than those who have only observed it from the outside will realize, but I had never thought to apply that principle to the ghost-written resignation letter. The careful non-apology, the declaration that one has "become a distraction," the tell-tale phrasing of "spending more time with my family" is offensive in its bland dishonesty. But Potter shows that the blandness is expertly constructed to destroy quotability. Those statements are nearly impossible to remember or report on because they have been built that way: nouns carefully separated from verbs, all force dissipated by circuities and modifiers, and subtle grammatical errors introduced to discourage written news from including direct quotes. Potter's journalism background shines here because he can describe the effect on news reporting. He walks the reader through the construction to reveal that the writing is not incompetent but instead is skillfully bad in a way that causes one's attention to skitter off of it. The letter vanishes into its own vagueness. The goal is to smother the story in such mediocrity that it becomes forgettable. And it works. I've written several resignation letters of my own. Somewhat unusually, I've even sent several of them, although (as Potter says is typical) fewer than I've written. I've even written (and sent) the sort of resignation letter that every career advisor will say to never send. Potter's discussion of the motives and thought process behind those letters rang true for me. It's a raw and very human moment, one is never in as much control of it as one wishes, the cracks and emotions break through in the writing, and often those letters are trying to do far too many things at the same time. But it's also a moment in which one can say something important and have others listen, which can be weirdly challenging to do in the normal course of a job. Potter ends this book beautifully by looking at resignation letters that break or transcend the mold of most of the others he's examined: letters where the author seems to have found some peace and internal understanding and expresses that simply and straightforwardly. I found this surprisingly touching after the emotional roller-coaster of the rest of the book, and a lovely note on which to end. This is a great book. Potter has a good eye for language and the emotion encoded in it, a bracing preference for the common worker or employee over the manager or politician, and the skill to produce some memorable turns of phrase. Most importantly, he has the enthusiast's love of the topic. Even if you don't care about resignation letters going in, it will be hard to avoid some fascination with them by the end of this book. Recommended. This book was originally published as F*ck You and Goodbye in the UK. Rating: 8 out of 10

11 April 2020

Wouter Verhelst: Married!

If you read this post through Planet Debian, then you may already know this through "Johnathan"'s post on the subject: on the 29th of February this year, Tammy and I got married. Yes. Really. No, I didn't expect this to happen myself about five years ago, but here we are. Tammy and I met four years ago at DebConf16 in Cape Town, South Africa, where she was a local organizer. If you were at dc16, you may remember the beautifully designed conference stationery, T-shirts, and bag; this was all her work. In addition, the opening and closing credits on the videos of that conference were designed by her. As it happens, that's how we met. I've been a member of the Debconf video team since about 2010, when I first volunteered to handle a few cameras. In 2015, since someone had to do it, I installed Carl's veyepar on Debian's on-site servers, configured, and ran it. I've been in charge of the postprocssing infrastructure -- first using veyepar, later using my own SReview -- ever since. So when I went to the Debconf organizers in 2016 to ask for preroll and postroll templates for the videos, they pointed me to Tammy; and we haven't quite stopped talking since. I can speak from experience now when saying that a long-distance relationship is difficult. My previous place of residence, Mechelen, is about 9600km away from Cape Town, and so just seeing Tammy required about a half month's worth of pay -- not something I always have to spare. But after a number of back-and-forth visits and a lot of paperwork, I have now been living in Cape Town for just over a year. Obviously, this has simplified things a lot. The only thing left for me to do now is to train my brain to stop thinking there's something stuck on my left ring finger. It's really meant to be there, after all...

2 March 2020

Petter Reinholdtsen: Nikita version 0.5 released - updated free software archive API server

Today, after many months of development, a new release of Nikita Noark 5 core project was finally announced on the project mailing list. The Nikita free software solution is an implementation of the Norwegian archive standard Noark 5 used by government offices in Norway. These were the changes in version 0.5 since version 0.4, see the email link above for links to a demo site: If free and open standardized archiving API sound interesting to you, please contact us on IRC (#nikita on irc.freenode.net) or email (nikita-noark mailing list). As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

10 November 2017

Wouter Verhelst: SReview 0.1

This morning I uploaded version 0.1 of SReview, my video review and transcoding system, to Debian experimental. There's still some work to be done before it'll be perfectly easy to use by anyone, but I do think I've reached the point by now where it should have basic usability by now. Quick HOWTO for how to use it: There's still some bits of the above list that I want to make easier to do, and there's still some things that shouldn't be strictly necessary, but all in all, I think SReview has now reached a certain level of maturity that means I felt confident doing its first upload to Debian. Did you try it out? Let me know what you think!

Thadeu Lima de Souza Cascardo: Software Freedom Strategy with Community Projects

It's been some time since I last wrote. Life and work have been busy. At the same time, the world has been busy, and as I would love to write a larger post, I will try to be short here. I would love to touch on the Librem 5 and postmarketOS. In fact, I had, in a podcast in Portuguese, Papo Livre. Maybe, I'll touch a little on the latter. Some of the inspiration for this post include: All of those led me to understand how software freedom is under attack, in particular how copyleft in under attack. And, as I talked during FISL, though many might say that "Open Source has won", end users software freedom has not. Lots of companies have co-opted "free software" but give no software freedom to their users. They seem friends with free software, and they are. Because they want software to be free. But freedom should not be a value for software itself, it needs to be a value for people, not only companies or people who are labeled software developers, but all people. That's why I want to stop talking about free software, and talk more about software freedom. Because I believe the latter is more clear about what we are talking about. I don't mind that we use whatever label, as long as we stablish its meaning during conversations, and set the tone to distinguish them. The thing is: free software does not software freedom make. Not by itself. As Bradley Kuhn puts it: it's not magic pixie dust. Those who have known me for years might remember me as a person who studied free software licenses and how I valued copyleft, the GPL specifically, and how I concerned myself with topics like license compatibility and other licensing matters. Others might remember me as a person who valued a lot about upstreaming code. Not carrying changes to software openly developed that you had not made an effort to put upstream. I can't say I was wrong on both accounts. I still believe in those things. I still believe in the importance of copyleft and the GPL. I still value sharing your code in the commons by going upstream. But I was certaily wrong in valuing them too much. Or not giving as much or even more value to distribution efforts of getting software freedom to the users. And it took me a while in seeing how many people also saw the GPL as a tool to get code upstream. You see that a lot in Linus' discourse about the GPL. And that is on the minds of a lot of people, who I have seen argue that copyleft is not necessary for companies to contribute code back. But that's the problem. The point is not about getting code upstream. But about assuring people have the freedom to run a modified version of the software they received on their computers. It turns out that many examples of companies who had contributed code upstream, have not delivered that freedom to their end-users, who had received a modified version of that same software, which is not free. Bradley Kuhn also alerts us that many companies have been replacing copyleft software with non-copyleft software. And I completely agree with him that we should be writing more copyleft software that we hold copyright for, so we can enforce it. But looking at what has been happening recently in the Linux community about enforcement, even thought I still believe in enforcement as an strategy, I think we need much more than that. And one of those strategies is delivering more free software that users may be able to install on their own computers. It's building those replacements for software that people have been using for any reason. Be it the OS they get when they buy a device, or the application they use for communication. It's not like the community is not doing it, it's just that we need to acknowledge that this is a necessary strategy to guarantee software freedom. That distribution of software that users may easily install on their computers is as much or even more valuable than developing software closer to the hacker/developer community. That doing downstream changes to free software in the effort of getting them to users is worth it. That maintaining that software stable and secure for users is a very important task. I may be biased when talking about that, as I have been shifting from doing upstream work to downstream work and both on the recent years. But maybe that's what I needed to realize that upstreaming does not necessarily guarantees that users will get software freedom. I believe we need to talk more about that. I have seen many people dear to me disregard that difference between the freedom of the user and the freedom of software. There is much more to talk about that, go into detail about some of those points, and I think we need to debate more. I am subscribed to the libreplanet-discuss mailing list. Come join us in discussing about software freedom there, if you want to comment on anything I brought up here. As I promised I would, I would like to mention about postmarketOS, which is an option users have now to get some software freedom on some mobile devices. It's an effort I wanted to build myself, and I applaud the community that has developed around it and has been moving forward so quickly. And it's a good example of a balance between upstream and dowstream code that gets to deliver a better level of software freedom to users than the vendor ever would. I wanted to write about much of the topics I brought up today, but postponed that for some time. I was motivated by recent events in the community, and I am really disappointed at some the free software players and some of the events that happened in the last few years. That got me into thinking in how we need to manifest ourselves about those issues, so people know how we feel. So here it is: I am disappointed at how the Linux Foundation handled the situation about Software Freedom Conversancy taking a case against VMWare; I am disappointed about how Software Freedom Law Center handled a trademark issue against the Software Freedom Conservancy; and I really appreciate all the work the Software Freedom Conservancy has been doing. I have supported them for the last two years, and I urge you to become a supporter too.

31 October 2017

Norbert Preining: Debian/TeX Live 2017.20171031-1

Halloween is here, time to upload a new set of scary packages of TeX Live. About a month has passed, so there is the usual big stream up updates. There was actually an intermediate release to get out some urgent fixes, but I never reported the news here. So here are the accumulated changes and updates. My favorite this time is wallcalendar, a great class to design all kind of calendars, it looks really well done. I immediately will start putting one together. On the font side there is the new addition coelacanth. To quote from the README: Coelacanth is inspired by the classic Centaur type design of Bruce Rogers, described by some as the most beautiful typeface ever designed. It aims to be a professional quality type family for general book typesetting. And indeed it is beautiful! Other noteworthy addition is the Spark font that allows creating sparklines in the running text with LaTeX. Enjoy. New packages algobox, amscls-doc, beilstein, bib2gls, coelacanth, crossreftools, dejavu-otf, dijkstra, ducksay, dynkin-diagrams, eqnnumwarn, fetchcls, fixjfm, glossaries-finnish, hagenberg-thesis, hecthese, ifxptex, isopt, istgame, ku-template, limecv, mensa-tex, musicography, na-position, notestex, outlining, pdfreview, spark-otf, spark-otf-fonts, theatre, unitn-bimrep, upzhkinsoku, wallcalendar, xltabular. Updated packages acmart, amsmath, animate, arabluatex, arara, babel, babel-french, bangorexam, baskervillef, beebe, biblatex-philosophy, biblatex-source-division, bibletext, bidi, bxjaprnind, bxjscls, bxpapersize, bytefield, classicthesis, cochineal, complexity, cooking-units, curves, datetime2-german, dccpaper, doclicense, docsurvey, eledmac, epstopdf, eqparbox, esami, etoc, fbb, fei, fithesis, fmtcount, fnspe, fonts-tlwg, fontspec, genealogytree, glossaries, glossaries-extra, hecthese, hepthesis, hvfloat, ifplatform, ifptex, inconsolata, jfmutil, jsclasses, ketcindy, knowledge, koma-script, l3build, l3experimental, l3kernel, l3packages, langsci, latex2man, latexbug, lato, leadsheets, libertinust1math, listofitems, luatexja, luatexko, luatodonotes, lwarp, markdown, mcf2graph, media9, newtx, novel, numspell, ocgx2, overpic, philokalia, phonenumbers, platex, poemscol, pst-exa, pst-geometrictools, pst-ovl, pst-plot, pst-pulley, pst-tools, pst-vehicle, pst2pdf, pstool, pstricks, pstricks-add, pxchfon, pxjahyper, quran, randomlist, rec-thy, reledmac, robustindex, scratch, skrapport, spectralsequences, tcolorbox, tetex, tex4ht, texcount, texdoc, tikzducks, tikzsymbols, toptesi, translation-biblatex-de, unicode-math, updmap-map, uplatex, widetable, xcharter, xepersian, xetexko, xetexref, xsim, zhlipsum.

5 October 2017

Wouter Verhelst: Patching Firefox

At work, I help maintain a smartcard middleware that is provided to Belgian citizens who want to use their electronic ID card to, e.g., log on to government websites. This middleware is a piece of software that hooks into various browsers and adds a way to access the smartcard in question, through whatever APIs the operating system and the browser in question provide for that purpose. The details of how that is done differ between each browser (and in the case of Google Chrome, for the same browser between different operating systems); but for Firefox (and Google Chrome on free operating systems), this is done by way of a PKCS#11 module. For Firefox 57, mozilla decided to overhaul much of their browser. The changes are large and massive, and in some ways revolutionary. It's no surprise, therefore, that some of the changes break compatibility with older things. One of the areas in which breaking changes were made is in the area of extensions to the browser. Previously, Firefox had various APIs available for extensions; right now, all APIs apart from the WebExtensions API are considered "legacy" and support for them will be removed from Firefox 57 going forward. Since installing a PKCS#11 module manually is a bit complicated, and since the legacy APIs provided a way to do so automatically provided the user would first install an add-on (or provided the installer of the PKCS#11 module sideloads it), most parties who provide a PKCS#11 module for use with Firefox will provide an add-on to automatically install it. Since the alternative involves entering the right values in a dialog box that's hidden away somewhere deep in the preferences screen, the add-on option is much more user friendly. I'm sure you can imagine my dismay when I found out that there was no WebExtensions API to provide the same functionality. So, after asking around a bit, I filed bug 1357391 to get a discussion started. While it took some convincing initially to get people to understand the reasons for wanting such an API, eventually the bug was assigned the "P5" priority -- essentially, a "we understand the need and won't block it, but we don't have the time to implement it. Patches welcome, though" statement. Since having an add-on was something that work really wanted, and since I had the time, I got the go-ahead from management to look into implementing the required code myself. I made it obvious rather quickly that my background in Firefox was fairly limited, though, and so was assigned a mentor to help me through the process. Having been a Debian Developer for the past fifteen years, I do understand how to develop free software. Yet, the experience was different enough that still learned some new things about free software development, which was somewhat unexpected. Unfortunately, the process took much longer than I had hoped, which meant that the patch was not ready by the time Firefox 57 was branched off mozilla's "central" repository. The result of that is that while my patch has been merged into what will eventually become Firefox 58, it looks strongly as though it won't make it into Firefox 57. That's going to cause some severe headaches, which I'm not looking forward to; and while I can certainly understand the reasons for not wanting to grant the exception for the merge into 57, I can't help but feeling like this is a missed opportunity. Anyway, writing code for the massive Open Source project that mozilla is has been a load of fun, and in the process I've learned a lot -- not only about Open Source development in general, but also about this weird little thing that Javascript is. That might actually be useful for this other project that I've got running here. In closing, I'd like to thank Tomislav 'zombie' Jovanovic for mentoring me during the whole process, without whom it would have been doubtful if I would even have been ready by now. Apologies for any procedural mistakes I've made, and good luck in your future endeavours! :-)

2 September 2017

Wouter Verhelst: Playing with Moose and FFmpeg

As I've blogged before, I've been on and off working on SReview, a video review system. Development over the past year has been mostly driven by the need to have something up and running for first FOSDEM 2017, and then DebConf17, and so I've cut corners left and right which made the system, while functional, not quite entirely perfect everywhere. For instance, the backend scripts were done in ad-hoc perl, each reinventing their own wheel. Doing so made it easier for me to experiment with things and figure out where I want them to go, without immediately creating a lot of baggage that is not necessarily something I want to be stuck to. This flexibility has already paid off, in that I've redone the state machine between FOSDEM and DebConf17 and all it needed was to update a few SQL statements here and there. Well, and add a few of them, too. It was always the intent to replace most of the ad-hoc perl with something better, however, once the time was ripe. One place where historical baggage is not so much of a problem, and where in fact abstracting away the complexity would now be an asset, is in the area of FFmpeg command lines. Currently, these are built by simple string expansion. For example, we do something like this (shortened for brevity):
system("ffmpeg -y -i $outputdir/$slug.ts -pass 1 -passlogfile ...");
inside an environment where the $outputdir and $slug variables are set in a perl environment. That works, but it has some downsides; e.g., adding or removing options based on which codecs we're using is not so easy. It would be much more flexible if the command lines were generated dynamically based on requested output bandwidth and codecs, rather than that they be hardcoded in the file. Case in point: currently there are multiple versions of some of the backend scripts, that only differ in details mostly the chosen codec on the ffmpeg command line. Obviously this is suboptimal. Instead, we want a way where video file formats can be autodetected, so that I can just say "create a file that uses encoder etc settings of this other file here". In addition, we also want a way where we can say "create a file that uses encoder etc settings of this other file here, except for these one or two options that I want to fine-tune manually". When I first thought about doing that about a year ago, that seemed complicated and not worth it or at least not to that extent. Enter Moose. The Moose OO system for Perl 5 is an interesting way to do object orientation in Perl. I knew Perl supports OO, and I had heard about Moose, but never had looked into it, mostly because the standard perl OO features were "good enough". Until now. Moose has a concept of adding 'attributes' to objects. Attributes can be set at object construction time, or can be accessed later on by way of getter/setter functions, or even simply functions named after the attribute itself (the default). For more complicated attributes, where the value may not be known until some time after the object has been created, Moose borrows the concept of "lazy" variables from Perl 6:
package Object;
use Moose;
has 'time' => (
    is => 'rw',
    builder => 'read_time',
    lazy => 1,
);
sub read_time  
    return localtime();
 
The above object has an attribute 'time', which will not have a value initially. However, upon first read, the 'localtime()' function will be called, the result is cached, and then (and on all further calls of the same function), the cached result will be returned. In addition, since the attribute is read/write, the time can also be written to. In that case, any cached value that may exist will be overwritten, and if no cached value exists yet, the read_time function will never be called. (it is also possible to clear values if needs be, so that the function would be called again). We use this with the following pattern:
package SReview::Video;
use Moose;
has 'url' => (
    is => 'rw',
)
has 'video_codec' => (
    is => 'rw',
    builder => '_probe_videocodec',
    lazy => 1,
);
has 'videodata' => (
    is => 'bare',
    reader => '_get_videodata',
    builder => '_probe_videodata',
    lazy => 1,
);
has 'probedata' => (
    is => 'bare',
    reader => '_get_probedata',
    builder => '_probe',
    lazy => 1,
);
sub _probe_videocodec  
    my $self = shift;
    return $self->_get_videodata-> codec_name ;
 
sub _probe_videodata  
    my $self = shift;
    if(!exists($self->_get_probedata-> streams ))  
        return  ;
     
    foreach my $stream(@ $self->_get_probedata-> streams )  
        if($stream-> codec_type  eq "video")  
            return $stream;
         
     
    return  ;
 
sub _probe  
    my $self = shift;
    open JSON, "ffprobe -print_format json -show_format -show_streams " . $self->url . " "
    my $json = "";
    while(<JSON>)  
        $json .= $_;
     
    close JSON;
    return decode_json($json);
 
The videodata and probedata attributes are internal-use only attributes, and are therefore of the 'bare' type that is, they cannot be read nor written to. However, we do add 'reader' functions that can be used from inside the object, so that the object itself can access them. These reader functions are generated, so they're not part of the object source. The probedata attribute's builder simply calls ffprobe with the right command-line arguments to retrieve data in JSON format, and then decodes that JSON file. Since the passed JSON file contains an array with (at least) two streams one for video and one for audio and since the ordering of those streams depends on the file and is therefore not guaranteed, we have to loop over them. Since doing so in each and every attribute of the file we might be interested in would be tedious, we add a videodata attribute that just returns the data for the first found video stream (the actual source also contains a similar one for audio streams). So, if you create an SReview::Video object and you pass it a filename in the url attribute, and then immediately run print $object->video_codec, then the object will However, if the caller first calls $object->video_codec('h264'), then the ffprobe and most of the caching will be skipped, and instead the h265 data will be returned as video codec name. Okay, so with a reasonably small amount of code, we now have a bunch of attributes that have defaults based on actual files but can be overwritten when necessary. Useful, right? Well, you might also want to care about the fact that sometimes you want to generate a video file that uses the same codec settings of this other file here. That's easy. First, we add another attribute:
has 'reference' => (
    is => 'ro',
    isa => 'SReview::Video',
    predicate => 'has_reference'
);
which we then use in the _probe method like so:
sub _probe  
    my $self = shift;
    if($self->has_reference)  
        return $self->reference->_get_probedata;
     
    # original code remains here
 
With that, we can create an object like so:
my $video = SReview::Video->new(url => 'file.ts');
my $generated = SReview::Video->new(url => 'file2.ts', reference => $video);
now if we ask the $generated object what the value of its video_codec setting is without telling it ourselves first, it will use the $video object for its probed data, and use that. That only misses generating the ffmpeg command line, but that's all fairly straightforward and therefore left as an exercise to the reader. Or you can cheat, and look it up.

30 August 2017

Wouter Verhelst: Re-encoding DebConf17 videos

Feedback we received after DebConf17 was that the quality of the released videos was rather low. Since I'd been responsible for encoding the videos, I investigated that, and found that I had used the video bitrate defaults of ffmpeg, which are rather low. So, I read some more documentation, and came up with better settings for the encoding to be done properly. While at it, I found that the reason that VP9 encoding takes forever, as I'd found before, has everything to do with the same low-bitrate defaults, and is not an inherent problem to VP9 encoding. In light of that, I ran a test encode of a video with VP8 as well as VP9 encoding, and found that it shaves off some of the file size for about the same quality, with little to no increase in encoder time. Unfortunately, the initial VP9 encoder settings that I used produced files that would play in some media players and in some browsers, but not in all of them. A quick email to the WebM-discuss mailinglist got me a reply, explaining that since our cameras used YUV422 encoding, which VP9 supports but some media players do not, and that since VP8 doesn't support that format, the result was VP9 files with YUV422 encoding, and VP8 ones with YUV420. The fix was to add a single extra command-line parameter to force the pixel format to YUV420 and tell ffmpeg to also downsample the video to that. After adding another few parameters so as to ensure that some metadata would be added to the files as well, I've now told the system to re-encode the whole corpus of DebConf17 videos in both VP8 as well as VP9. It would appear that the VP9 files are, on average, about 1/4th to 1/3rd smaller than the equivalent VP8 files, with no apparent loss in quality. Since some people might like them for whatever reason, I've not dropped the old lower-quality VP8 files, but instead moved them to an "lq" directory. They are usually about half the size of the VP9 files, but the quality is pretty terrible. TLDR: if you downloaded videos from the debconf video archive, you may want to check whether it has been re-encoded yet (which would be the case if the state on that page is "done"), and if so, download them anew.

29 August 2017

Holger Levsen: 20170829-qubes-os

"Qubes OS from the POV of a Debian developer" and "Qubes OS user meetup at Bornhack" I wrote the following while on my way home from Bornhack which was an awesome hacking camp on the Danish island of Bornholm, where about 200 people gathered for a week, with a nice beach in walking distance (and a not too cold Baltic Sea) and vegan and not so vegan grills, to give some hints why it was awesome. (Actually it was mostly awesome due to the people there, not the things, but anyway ) And there were of course also talks and workshops, and one was a Qubes OS users meetup, which amazingly was attended by 20 Qubes OS users (and some Qubes OS users present were even missing), so in other words: Qubes OS had a >10% user base at this event! And I even found one heads user! ;-) At DebConf17 I gave a talk titled "Qubes OS from the POV of a Debian developer" and while the video was immediatly available (thanks to the DebConf videoteam!) I've also now put my slides for it online. Since then, I learned a few things: I'm looking forward to more Qubes OS user meetups in future!

20 June 2017

Norbert Preining: TeX Live 2017 hits Debian/unstable

Yesterday I uploaded the first packages of TeX Live 2017 to Debian/unstable, meaning that the new release cycle has started. Debian/stretch was released over the weekend, and this opened up unstable for new developments. The upload comprised the following packages: asymptote, cm-super, context, context-modules, texlive-base, texlive-bin, texlive-extra, texlive-extra, texlive-lang, texworks, xindy.
I mentioned already in a previous post the following changes: The last two changes are described together with other news (easy TEXMF tree management) in the TeX Live release post. These changes more or less sum up the new infra structure developments in TeX Live 2017. Since the last release to unstable (which happened in 2017-01-23) about half a year of package updates have accumulated, below is an approximate list of updates (not split into new/updated, though). Enjoy the brave new world of TeX Live 2017, and please report bugs to the BTS! Updated/new packages:
academicons, achemso, acmart, acro, actuarialangle, actuarialsymbol, adobemapping, alkalami, amiri, animate, aomart, apa6, apxproof, arabluatex, archaeologie, arsclassica, autoaligne, autobreak, autosp, axodraw2, babel, babel-azerbaijani, babel-english, babel-french, babel-indonesian, babel-japanese, babel-malay, babel-ukrainian, bangorexam, baskervaldx, baskervillef, bchart, beamer, beamerswitch, bgteubner, biblatex-abnt, biblatex-anonymous, biblatex-archaeology, biblatex-arthistory-bonn, biblatex-bookinother, biblatex-caspervector, biblatex-cheatsheet, biblatex-chem, biblatex-chicago, biblatex-claves, biblatex-enc, biblatex-fiwi, biblatex-gb7714-2015, biblatex-gost, biblatex-ieee, biblatex-iso690, biblatex-manuscripts-philology, biblatex-morenames, biblatex-nature, biblatex-opcit-booktitle, biblatex-oxref, biblatex-philosophy, biblatex-publist, biblatex-shortfields, biblatex-subseries, bibtexperllibs, bidi, biochemistry-colors, bookcover, boondox, bredzenie, breqn, bxbase, bxcalc, bxdvidriver, bxjalipsum, bxjaprnind, bxjscls, bxnewfont, bxorigcapt, bxpapersize, bxpdfver, cabin, callouts, chemfig, chemformula, chemmacros, chemschemex, childdoc, circuitikz, cje, cjhebrew, cjk-gs-integrate, cmpj, cochineal, combofont, context, conv-xkv, correctmathalign, covington, cquthesis, crimson, crossrefware, csbulletin, csplain, csquotes, css-colors, cstldoc, ctex, currency, cweb, datetime2-french, datetime2-german, datetime2-romanian, datetime2-ukrainian, dehyph-exptl, disser, docsurvey, dox, draftfigure, drawmatrix, dtk, dviinfox, easyformat, ebproof, elements, endheads, enotez, eqnalign, erewhon, eulerpx, expex, exsheets, factura, facture, fancyhdr, fbb, fei, fetamont, fibeamer, fithesis, fixme, fmtcount, fnspe, fontmfizz, fontools, fonts-churchslavonic, fontspec, footnotehyper, forest, gandhi, genealogytree, glossaries, glossaries-extra, gofonts, gotoh, graphics, graphics-def, graphics-pln, grayhints, gregoriotex, gtrlib-largetrees, gzt, halloweenmath, handout, hang, heuristica, hlist, hobby, hvfloat, hyperref, hyperxmp, ifptex, ijsra, japanese-otf-uptex, jlreq, jmlr, jsclasses, jslectureplanner, karnaugh-map, keyfloat, knowledge, komacv, koma-script, kotex-oblivoir, l3, l3build, ladder, langsci, latex, latex2e, latex2man, latex3, latexbug, latexindent, latexmk, latex-mr, leaflet, leipzig, libertine, libertinegc, libertinus, libertinust1math, lion-msc, lni, longdivision, lshort-chinese, ltb2bib, lualatex-math, lualibs, luamesh, luamplib, luaotfload, luapackageloader, luatexja, luatexko, lwarp, make4ht, marginnote, markdown, mathalfa, mathpunctspace, mathtools, mcexam, mcf2graph, media9, minidocument, modular, montserrat, morewrites, mpostinl, mptrees, mucproc, musixtex, mwcls, mweights, nameauth, newpx, newtx, newtxtt, nfssext-cfr, nlctdoc, novel, numspell, nwejm, oberdiek, ocgx2, oplotsymbl, optidef, oscola, overlays, pagecolor, pdflatexpicscale, pdfpages, pdfx, perfectcut, pgfplots, phonenumbers, phonrule, pkuthss, platex, platex-tools, polski, preview, program, proofread, prooftrees, pst-3dplot, pst-barcode, pst-eucl, pst-func, pst-ode, pst-pdf, pst-plot, pstricks, pstricks-add, pst-solides3d, pst-spinner, pst-tools, pst-tree, pst-vehicle, ptex2pdf, ptex-base, ptex-fontmaps, pxbase, pxchfon, pxrubrica, pythonhighlight, quran, ran_toks, reledmac, repere, resphilosophica, revquantum, rputover, rubik, rutitlepage, sansmathfonts, scratch, seealso, sesstime, siunitx, skdoc, songs, spectralsequences, stackengine, stage, sttools, studenthandouts, svg, tcolorbox, tex4ebook, tex4ht, texosquery, texproposal, thaienum, thalie, thesis-ekf, thuthesis, tikz-kalender, tikzmark, tikz-optics, tikz-palattice, tikzpeople, tikzsymbols, titlepic, tl17, tqft, tracklang, tudscr, tugboat-plain, turabian-formatting, txuprcal, typoaid, udesoftec, uhhassignment, ukrainian, ulthese, unamthesis, unfonts-core, unfonts-extra, unicode-math, uplatex, upmethodology, uptex-base, urcls, variablelm, varsfromjobname, visualtikz, xassoccnt, xcharter, xcntperchap, xecjk, xepersian, xetexko, xevlna, xgreek, xsavebox, xsim, ycbook.

9 June 2017

John Goerzen: Fixing the Problems with Docker Images

I recently wrote about the challenges in securing Docker container contents, and in particular with keeping up-to-date with security patches from all over the Internet. Today I want to fix that. Besides security, there is a second problem: the common way of running things in Docker pretends to provide a traditional POSIX API and environment, but really doesn t. This is a big deal. Before diving into that, I want to explain something: I have often heard it said the Docker provides single-process containers. This is unambiguously false in almost every case. Any time you have a shell script inside Docker that calls cp or even ls, you are running a second process. Web servers from Apache to whatever else use processes or threads of various types to service multiple connections at once. Many Docker containers are single-application, but a process is a core part of the POSIX API, and very little software would work if it was limited to a single process. So this is my little plea for more precise language. OK, soapbox mode off. Now then, in a traditional Linux environment, besides your application, there are other key components of the system. These are usually missing in Docker containers. So today, I will fix this also. In my docker-debian-base images, I have prepared a system that still has only 11MB RAM overhead, makes minimal changes on top of Debian, and yet provides a very complete environment and API. Here s what you get: The above goes into my minimal image. Additional images add layers on top of it, and here are some of the features they add: All of the above, including the optional features, has an 11MB overhead on start. Not bad for so much, right? From here, you can layer on top all your usual Dockery things. You can still run one application per container. But you can now make sure your disk doesn t fill up from logs, run your database vacuuming commands at will, have your blog download its RSS feeds every few minutes, etc all from within the container, as it should be. Furthermore, you don t have to reinvent the wheel, because Debian already ships with things to take care of a lot of this out of the box and now those tools will just work. There is some popular work done in this area already by phusion s baseimage-docker. However, I made my own for these reasons: Finally a word on the choice to use sysvinit. It would have been simpler to use systemd here, since it is the default in Debian these days. Unfortunately, systemd requires you to poke some holes in the Docker security model, as well as mount a cgroups filesystem from the host. I didn t consider this acceptable, and sysvinit ran without these workarounds, so I went with it. With all this, Docker becomes a viable replacement for KVM for various services on my internal networks. I ll be writing about that later.

Next.

Previous.