Search Results: "rul"

18 March 2020

Dirk Eddelbuettel: RcppCCTZ 0.2.7

A new release 0.2.7 of RcppCCTZ is now at CRAN. RcppCCTZ uses Rcpp to bring CCTZ to R. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. And while CCTZ is made by Google(rs), it is not an official Google product. The RcppCCTZ page has a few usage examples and details. This package was the first CRAN package to use CCTZ; by now at least three others do using copies in their packages which remains less than ideal. This version adds internal extensions, contributed by Leonardo, which support upcoming changes to the nanotime package we are working on.

Changes in version 0.2.7 (2020-03-18)
  • Added functions _RcppCCTZ_convertToCivilSecond that converts a time point to the number of seconds since epoch, and _RcppCCTZ_convertToTimePoint that converts a number of seconds since epoch into a time point; these functions are only callable from C level (Leonardo in #34 and #35).
  • Added function _RcppCCTZ_getOffset that returns the offset at a speficied time-point for a specified timezone; this function is only callable from C level (Leonardo in #32).

We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

16 March 2020

Ulrike Uhlig: Deconstructing the term control freak

Control freaks. You may have called people like this. Or you may have been called one yourself. Maybe you got angry. Or, on the contrary, you felt like being a control freak is a feature, because who would notice all these little details that are not exactly perfect if not you? This post is an attempt to deconstruct the term. Etymological considerations Control - from latin contra- rotulus - refers to the copy of an account register as a means of verification of the original register. So control is about keeping track, not making mistakes, i.e. it's about doing a perfect calculation. The meaning of control in our society is linked to authority, or to policing like in crowd control. What English calls an inspector, is a Kontrolleur in German or contr leur in French. The word is also linked to manufacturing, as in quality control. We also talk about being in control, which is linked to the desire of being autonomous, to be able to act on one's own account. Being in control can also designate a need to preserve one's own integrity: if I have integrity (in the sense of feeling whole), I can self-determine, which is the most fundamental requirement for having one's own identity. Freak - refers to someone who does not fit the norm. Calling someone a freak is per se problematic, because it blames a person for being atypical, abnormal. Freak is a word that, while pointing a finger at difference, at the same time denies people the right to be different and diverse. People can reappropriate such words to make it clear that not fitting the norm is a wanted expression of their diversity we can see that if we follow the history of the word queer for an example. Types of control freaking To me, being a control freak is mostly a feature, if it's about controlling my own life and my own effectiveness. Control freaking becomes troublesome when it's about controlling other people's effectiveness. And it becomes highly problematic when it's about controlling other people's lives. Controlling one's own life Having control over one's own life is a basic human need. In particular people who are part of minorities, people who face discrimination and oppression every day have an increased need for self-determination, and the need to see their existence and their identity acknowledged and accepted by the rest of the world. Too often do we experience that other people (generally the ones with privilege, or not facing the same oppression) try to own the narrative over our lives (1)(2). Calling this control freakish is missing the point, and is probably a sign of looking at it from a perspective of privilege. Controlling one's own effectiveness People produce things. People write code, make ceramics, or write texts, for example. People have a need to control their own effectiveness: they set up routines, product tests, documentation. This type of control is a feature, and can help to release better software, better texts, or perfectly burnt clay pots ultimately controlling one's own effectiveness helps to learn from one's mistakes and to improve routines over time. The grey zone The grey zone describes the zone in between wanting to control one's own effectiveness and wanting to control other people's effectiveness. Here we could situate the type of control freak who does not delegate tasks to others for fear of seeing them done differently then they had expected them to be done. This can be harmful particularly to the person who does not delegate: they can get overworked or burnt out. This type of control freakism might be linked to perfectionism. Controlling other people's effectiveness When we work with other people, our own effectiveness might get in the way of other people's effectiveness, or vice versa. Indeed, it happens - not only in work contexts - that people find themselves in setups in which mutual responsibilities and autonomies conflict with one another because one person is dependent on the other for making decisions or moving things forward as they see fit. (There is generally a relation of dependency between two conflicting parties that is worth looking at (3).) Add to this the fact that, when delegating a task, some people have a hard time also delegating the responsibility and autonomy needed to resolve the task. They lack trust that another person can also do the work, or want that person to do the work exactly in the same way they would do it. (In some cases this can be related to Founder's Syndrome and can result in organizations staying stuck with one or a small group of founders holding knowledge and power, and preventing the organization from growing. Page 11 in the booklet "Working with conflict in our groups" describes how such an informal hierarchy can come into being in grassroot groups.) The (perfectly valid) need behind this type of control freaking could be to make sure that a group of people builds a successful product, releases a fact-checked documentary, or creates a publication without mistakes. But controlling other people's effectiveness as a strategy to satisfy this need can create a non-cooperative climate in which people do not meet each other on eye level, but are dependent on each other, experience a lack autonomy, a break of boundaries, or sometimes feel authority to be overexerted. Acknowledging the need to build a good product, it is possible to create the appropriate strategies to guarantee that the involved people can meet each other on eye level: for example by clearly defining and documenting role-responsibility-accountability along with appropriate decision making processes, or by distributing leadership ( you should totally click on that link!), by looking at inclusive leadership models, by learning from past mistakes, by instating feedback cycles, by making boundaries between the direction of an organization and the day-to-day work clear. An organization I work with has the rule that people can make decisions for themselves if the decision only affects their work, while decisions that affect a team should be made with the team, and decisions that affect the organization as a whole need to be made at the organizational level. They call that a no-brainer, but in organizations with traditional hierarchies, or in grown grassroot environments that have never clearly defined and assigned responsibilities and accountabilities (aka "functional roles") this is not so obvious at all. Controlling someone else's life This type of control freak does not only desire to control their own life but for a reason or another wants to know and control what other people do, think (especially about the control freak), decide, or how they live. In some cases, this type of control freak might even want to force upon others things they should do as a way to be accepted by the control freak. It's what we call narcissism, harassment, abuse. It is unacceptable. Conclusion In summary, the distinctions I came up with in this post describe the boundaries along which control freakism takes hold of someone else's effectiveness or life and ultimately prevents them from self-determining. In German we have the word bergriffig which describes that someone is infringing someone else's boundaries they are over - grabbing, seizing, grasping, taking hold of. Which type of control freak are you, if any?

(1) Like when a West-Berliner in a round of 15 East Germans arrogantly talks about the time when the wall came down and tells the story that he could not go shopping between Thursdays and Sundays because the East Germans bought too many products in the supermarkets while this event marked an unimaginable rift in the biographies of 17 million East Germans, 15 of whom are sitting right in front of him. Thirty years after 1989, many of us are finally starting to question this publicly ( links in German language).

(2) Women experience it similary regularly, see Men Explain Things to me, a book by Rebecca Solnit.

(3) By the way, rather than having "recruited the wrong person" conflict may intrinsically arise as part of certain work relationships, simply due to the inter-dependencies of roles or workers, like in a delivery chain.

12 March 2020

Julien Danjou: One year of Mergify

One year of MergifyIt has been close to a year now that I've incorporated my new company, Mergify. I've been busy, and I barely wrote anything about it so far. Now is an excellent time to take a break and reflect a bit on what happened during those last 12 months.
One year of Mergify

What problem does Mergify solve?Mergify is a powerful automation engine for GitHub pull requests. It allows you to automate everything and especially merging. You write rules, and it handles the rest.
One year of MergifyExample of rule matching returned in GitHub checks
For example, let's say you want your pull request to be merged, e.g., once your CI passes and the pull request has been approved. You just write such a rule, and our engine merges the pull request as soon as it's ready.We also deal with more advanced use cases. For instance, we provide a merge queue so your pull requests are merged serially and tested by your CI one after another avoiding any regression in your code.Our goal is to make pull request management and automation easy. You can use your bot to trigger a rebase of your pull requests, or a backport to a different branch, just with a single comment.
One year of MergifySome people like to make bots talk to each other.

A New AdventureMergify is the first company that I ever started. I did run some personal businesses before, created non-profit organizations, built FOSS projects but I never created a company from scratch, even less with an associate.Indeed, I've chosen to build the company with my old friend Mehdi. We've known each others for 7 years now, and have worked together all that time on different open-source projects. Having worked with each other for so long has probably been a critical factor in the success of our venture so far.I had little experience sharing the founding seats with someone, and tons of reading seemed to indicate that it would be a tough ride. Picking the right business partner(s) can be a hard task. Luckily, after working so much time together, Mehdi and I both know our strengths and weaknesses well enough to be able to circumvent them. On the other hand, we both have similar backgrounds as software engineers. That does not help to cover all the hats you need to wear when building a company. Over time, we found arrangements to cover most of those equally between us.We don't have any magical advice to give on this. As in every relationship, communication is the key, and the #1 factor of success.

Getting UsersI don't know if we got lucky, but we got users and customers pretty early. We used a few cooperative projects as guinea pigs first, and they were brave enough to try our service and give us feedback. No repository has been harmed during this first phase!Then, as soon as we managed to get our application on the GitHub Marketplace, we saw a steady number of users coming to us.This has been fantastic as it allowed us to get feedback rapidly. We set up a form asking users for feedback after they used Mergify for a couple of weeks. What we hear is that users were happy, that the documentation was confusing and that some features were buggy or missing. We planned all of those ideas as our future work in our roadmap, using the principles we described a few months ago.
How we handle our roadmap for Mergify
Whatever you re building a company, a product, or a house a time comes where you need planning. Pushing random buttons to move forward does not work anymore. You need to take a step back to
One year of MergifyJulien DanjouMergify
One year of Mergify
If you're curious, you can read this article.
We tried various strategies to get new users, but so far, organic growth has been our #1 way of onboarding new users. Like many small startups out there, we're not that good at marketing and executing strategies.We provide our service for free for open-source projects We are now powering many organizations, such as Mozilla, Amazon Web Services, Ceph and Fedora.

Working with GitHubWorking with GitHub has been complicated. When you build an application for a marketplace, your business is entirely dependent on the platform you develop for both in terms of features and quality of service.In our case, we hit quite many bugs with GitHub. Their support has mostly been fast to answer, but some significant issues are still opened months later. The truth is that the GitHub API could deserve more love and care from GitHub. For example, their GraphQL API is a work in progress for years and miss out many essential features.
One year of MergifyGitHub service status is not always green.
We dealt and still deal with all those issues. It obviously impacts our operations and decreases our overall velocity. We regularly have to find new ways to sidestep GitHub limitations.You have no idea how we wished for GitHub to be open-source. The idea of not having access to their code and understand how it works is so frustrating that we publish our engine as an open-source project. That allows all of our users to see how it works and even propose enhancements.
One year of Mergify

Automate all the wayWe're a tiny startup, and we decided to bootstrap our company. We never took any funding. From the beginning, it has been clear to us that we had to think and act like we had no resources. We're built around a scarcity mindset. Every decision we make is based on the assumption that we basically are very limited in terms of money and time.We basically act like any wrong choice we do could (virtually) kill the company. We only do what is essential, we ship fast, and we automate everything.For example, we have built our whole operation about CI/CD systems, and pushing any new fix or feature in production is done in a matter of minutes. It's not uncommon for us to push a fix from our phone, just by reviewing some code or editing a file.

GrowthWe're extremely happy with our steady growth and more users using our service. We now manage close to 30k repositories and merge 15k pull requests per month for our users.That's a lot of mouse clicks saved!If you want to try Mergify yourself, it's a single click log-in using your GitHub account. Check it out!

11 March 2020

Axel Beckert: Backup over Tor with BackupPC

I have a Raspberry Pi at my parents home. They have internet access via some ISP using Carrier Grade NAT (CGN). Hence their home router is not reachable via IPv4 from the outside, they do have IPv6 and the devices can also be made accessible via IPv6 via the local router. Did that, was able to access my Raspberry Pi over IPv6 and SSH from the outside. So doing backup of that Raspberry Pi with BackupPC from the outside was a walk in the park. Unfortunately the IPv6 prefix seems to change occasionally and the router only allows to configure explicit IPv6 addresses in firewall rules so after a prefix change the configured rules no more match the devices IPv6 addresses. Meh. So there were multiple possibilities to work around these restrictions and access a devices behind the router: Enabling a Tor Hidden Service for port 22 is a no-brainer and was done most quickly (actually it already was in place as I already suspected that an IPv6 prefix change might happen) and I so far was too lazy to replace it with something more proper. But my backup was relying on direct SSH access via IPv6. So I needed to get that working over Tor, too. Here s what was needed for the host named sherpa (named after the Fiberfab Sherpa) to be backed up via Tor: That s it basically. Of course you also need to have the SSH public host key in the .ssh/known_hosts file also for the .onion hostname. And the Tor Hidden Service needs to be configured on the target device. But that s left as exercise for the reader. There s a lot of documentation about that on the internet, including slides and video recordings of talks and live demos I gave about this topic in German. Ah, and in case you might think that s unfair and misuse of the resources of the Tor Project: No, I explicitly asked and they said more or less any additional traffic helps to make it more difficult to analyse Tor traffic or to track Tor users and is hence welcome. Addendum: The last direct full backup of that Raspberry Pi (5.5 GB) took around 32 minutes. The first full backup over Tor (8.7 GB) took 341 minutes. Seems much slower, but there might be other factors as well: Most backups which ran last night were running at only 0.85 MB/s to 1 MB/s, probably because too many backups were running in parallel after a recent backup server downtime with file system check the backup server was probably the bottleneck. The backup of the Raspberry Pi over Tor ran at 0.42 MB/s, so about half the speed of the other backups. (Will probably add some more notes if I have more statistics over time.)

Antoine Beaupr : Font changes

I have worked a bit on the fonts I use recently. From the main font I use every day in my text editor and terminals to this very website, I did a major and (hopefully) thoughtful overhaul of my typography, in the hope of making things easier to use and, to be honest, just prettier.

Editor and Terminal: Fira mono This all started when I found out about the Jetbrains Mono font. I found the idea of ligatures fascinating: the result is truly beautiful. So I do what I often do (sometimes to the despair of some fellow Debian members) and filed a RFP to document my research. As it turns out, Jetbrains Mono is not free enough to be packaged in Debian, because it requires proprietary tools to build. I nevertheless figured I could try another font so I looked at other monospace alternatives. I found the following packages in debian: Those are also "programmer fonts" that caught my interest but somehow didn't land in Debian yet: Because Fira code had ligatures, i ended up giving it a shot. I really like the originality of the font. See, for example, how the @ sign looks when compared to my previous font, Liberation Mono:
A dark terminal window showing the prompt '[SIGPIPE]anarcat@angela:~(master)$' in the Liberation Mono font Liberation Mono
A dark terminal window showing the prompt '[SIGPIPE]anarcat@angela:~(master)$' in the Fira mono font Fira Mono
Interestingly, a colleague (thanks ahf!) pointed me to the Practical Typography post "Ligatures in programming fonts: hell no", which makes the very convincing argument that ligatures are downright dangerous to programming environment. In my experiences with the fonts, it was also not always giving the result I would expect. I also remembered that the Emacs Haskell mode would have this tendency of inserting crazy syntactic sugar like this in source code without being asked, which I found extremely frustrating. Besides, Emacs doesn't support ligatures, unless you count such horrendous hacks which hack at the display time. That's because Emacs' display layer is not based on a modern rendering library like Pango but some scary legacy code that very few people understand. On top of the complexity of the codebase, there is also resistance in including a modern font. So I ended up using Fira mono everywhere I use fixed-width fonts, even though it's not packaged in Debian. That involves the following configuration in my .Xresources (no, I haven't switched to Wayland): 
! font settings
Emacs*font: Fira mono
rofi*font: Fira mono 12
! Symbola is to get emojis to display correctly, apt install fonts-symbola
!URxvt*font: xft:Monospace,xft:Symbola
URxvt*font: xft:Fira mono,xft:Symbola
I also dropped this script in ~/.local/share/fonts/download.sh, for lack of a better way, to download all the fonts I'm interested in: 
#!/bin/sh
set -e
set -u
wget --no-clobber --continue https://practicaltypography.com/fonts/charter.zip
unzip -u charter.zip
wget --no-clobber --continue https://github.com/adobe-fonts/source-serif-pro/releases/download/3.001R/source-serif-pro-3.001R.zip
unzip -u source-serif-pro-3.001R.zip
wget --no-clobber --continue https://github.com/adobe-fonts/source-sans-pro/releases/download/3.006R/source-sans-pro-3.006R.zip
unzip -u source-sans-pro-3.006R.zip
wget --no-clobber --continue https://github.com/adobe-fonts/source-code-pro/releases/download/2.030R-ro%2F1.050R-it/source-code-pro-2.030R-ro-1.050R-it.zip
unzip -u source-code-pro-2.030R-ro-1.050R-it.zip
wget --no-clobber --continue -O Fira-4.202.zip https://github.com/mozilla/Fira/archive/4.202.zip   true
unzip -u Fira-4.202.zip
Update: I forgot to mention one motivation behind this was to work around a change in the freetype interpreter, discussed in bug 866685 and my upgrades documentation.

Website: Charter That "hell no" article got me interested in the Practical Typography web book, which I read over the weekend. It was an eye opener and I realized I had already some of those concepts implanted; in fact it's probably after reading the Typography in ten minutes guide that I ended up trying Fira sans a few years ago. I have removed that font since then, however, after realising it was taking up an order of magnitude more bandwidth space than the actual page content. I really loved the book, so much that I actually bought it. I liked the concept of it, the look, and the fact that it's a living document. There's a lot of typography work I would need to do on this site to catchup with the recommendations from Matthew Butterick. Switching fonts is only one part of this, but it's something I was excited to work on. So I sat down and reviewed the free fonts Butterick recommends and tried out a few. I ended up settling on Charter, a relatively old (in terms of computing) font designed by Matthew Carter (of Verdana fame) in 1987. Charter really looks great and is surprisingly small. While a single version of Fira varies between 96KiB (Fira Sans Condensed) and 308KiB (Fira Sans Medium Italic), Charter is only 28KiB! While it's still about as large as most of my articles, I found it was a better compromise and decided to make the jump. This site is now in Serif, which is a huge change for me. The change was done with the following CSS: 
h1, h2, h3, h4, h5, body  
    /* 
     * Charter: Butterick's favorite, freely available, found on https://practicaltypography.com/free-fonts.html
     * Palatino: Mac OS
     * Palatino Linotype: Windows
     * Noto serif: Android, packaged in Debian
     * Liberation serif: Linux fallback
     * Serif: fallback
     */
    font-family: Charter, Palatino, "Palatino Linotype", "Noto serif", "Liberation serif", serif;
    /* Charter is available from https://practicaltypography.com/charter.html under the liberal Bitstream license */
I have also decided to outline headings by making them slanted, using the beautiful italic version of Charter: 
h1, h2, h3, h4, h5  
    font-style: italic;
I also made the point size larger, if the display is large enough: 
/* enlarge body point size for charter for larger displays */
@media (min-device-width: 750px)  
    body  
        font-size: 20px;
        line-height: 1.3; /* default in FF is ~1.48 */
Modern display (including fonts) have a much higher resolution and the point size on my website was really getting too small to be readable. This, in turn, required a change to the max-width: 
#content  
    /* about 2.5 alphabets with charter */
    max-width: 35em;
I have kept the footers and "UI" elements in sans-serif though, and kept those aligned on operating system defaults or "system fonts": 
/* some hacking at typefaces to get some fresh zest in here
 * fallbacks from:
 * https://en.wikipedia.org/wiki/List_of_typefaces_included_with_Microsoft_Windows
 * https://en.wikipedia.org/wiki/List_of_typefaces_included_with_macOS
 */
.navbar, .footer  
    /*
     * Avenir: Mac, quite different but should still be pretty
     * Gill sans: Mac, Windows, somewhat similar to Avenir (indulge me)
     * Noto sans: Android, packaged in Debian
     * Open sans: Texlive extras AKA Linux, packaged in Debian
     * Fira sans: Mozilla's Firefox OS
     * Liberation sans: Linux fallback
     * Helvetica: general fallback
     * Sans-serif: fallback
     */
    font-family: Avenir, "Gill sans", "Noto sans", "Open sans", "Fira sans", "Liberation sans", Helvetica, sans-serif;
    /* Fira is available from https://github.com/mozilla/Fira/ under the SIL Open Font License */
    /* alternatively, just use system fonts for "controls" instead:
    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Roboto", "Oxygen", "Ubuntu", "Cantarell", "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;
    gitlab uses: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Noto Sans", Ubuntu, Cantarell, "Helvetica Neue", sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji"
    */
I've only done some preliminary testing of how this will look like. Although I tested on a few devices (my phone, e-book tablet, an iPad, and of course my laptop), I fully expect things to break on your device. Do let me know if things look better or worse. For future comparison, my site is well indexed in the Internet Wayback Machine and can be used to look at the site before the change. For example, compare the previous article here with its earlier style. The changes to the theme are of course available in my custom ikiwiki bootstrap theme (see in particular commits 0bca0fb7 and d1901fb8), as usual. Enjoy, and let me know what you think! PS: I considered just setting the Charter font in CSS and not adding it as a @font-face. I'm still considering that option and might do so if the performance cost is too big. The Fira mono font is actually set like this for the preformatted sections of the site, but because it's more common (and it's too big) I haven't added it as a font-face. You might want to download the font locally to benefit from the full experience as well. PPS: As it turns out, an earlier version of this post featured exactly that: a non-webfont version of Charter, which works fine if you have a good Charter font available. But it looks absolutely terrible if, like many Linux users, you have the nasty bitmap font shipped with xfonts-100dpi and xfonts-75dpi. So I fixed the webfont and it's unlikely this site will be able to load reasonably well in Linux until those packages are removed or bitmap font rendering is disabled.

10 March 2020

Enrico Zini: COVID-19 links

COVID-19 #CoronaVirus Infographic Datapack Information is Beautiful
chart covid19 health archive.org
2020-03-11
COVID-19 Infographic Datapack, Regularly updated
European Centre for Disease Prevention and Control - COVID-19
chart covid19 health archive.org
2020-03-11
European Centre for Disease Prevention and Control infographics
pcm-dpc/COVID-19
chart covid19 health interactive version (desktop) interactive version (mobile) archive.org
2020-03-11
COVID-19 Italia - Monitoraggio situazione
Nextstrain
chart covid19 health archive.org
2020-03-11
Real-time tracking of pathogen evolution
Covid19Italia Help
covid19 GitHub project archive.org
2020-03-11
Community sharing resources, verified information, and support initiatives, on COVID-19
Worldometer - real time world statistics
chart covid19 health archive.org
2020-03-11
Live world statistics on population, government and economics, society and media, environment, food, water, energy and health.
Why outbreaks like coronavirus spread exponentially, and how to "flatten the curve"
covid19 news archive.org
2020-03-11
An Intuitive Explanation of Bayes's Theorem - LessWrong 2.0
health learning archive.org
2020-03-11
Your friends and colleagues are talking about something called Bayes s Theorem or Bayes s Rule, or something called Bayesian reasoning. They sound really enthusiastic about it, too, so you google and find a web page about Bayes s Theorem and... It s this equation. That s all. Just one equation. The page you found gives a definition of it, but it doesn t say what it is, or why it s useful, or why your friends would be interested in it. It looks like this random statistics thing. Why does a mathematical concept generate this strange enthusiasm in its students? What is the so-called Bayesian Revolution now sweeping through the sciences, which claims to subsume even the experimental method itself as a special case? What is the secret that the adherents of Bayes know? What is the light that they have seen? Soon you will know. Soon you will be one of us. While there are a few existing online explanations of Bayes s Theorem, my experience with trying to introduce people to Bayesian reasoning is that the existing online explanations are too abstract. Bayesian reasoning is very counterintuitive. People do not employ Bayesian reasoning intuitively, find it very difficult to learn Bayesian reasoning when tutored, and rapidly forget Bayesian methods once the tutoring is over. This holds equally true for novice students and highly trained professionals in a field. Bayesian reasoning is apparently one of those things which, like quantum mechanics or the Wason Selection Test, is inherently difficult for humans to grasp with our built-in mental faculties. Or so they claim. Here you will find an attempt to offer an intuitive explanation of Bayesian reasoning an excruciatingly gentle introduction that invokes all the human ways of grasping numbers, from natural frequencies to spatial visualization. The intent is to convey, not abstract rules for manipulating numbers, but what the numbers mean, and why the rules are what they are (and cannot possibly be anything else). When you are finished reading this, you will see Bayesian problems in your dreams.
Coronavirus COVID-19 Global Cases by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU)
chart covid19 health archive.org
2020-03-11
COVID-19 Dashboard for Germany
chart covid19 health archive.org
2020-03-11
COVID-19 Dashboard for France
chart covid19 health archive.org
2020-03-11
COVID-19 Dashboard for Nederland
chart covid19 health archive.org
2020-03-11
COVID-19 Dashboard for Slovenia
chart covid19 health archive.org
2020-03-11
Io Resto A Casa .work
archive.org
2020-03-11
Continuiamo a lavorare, studiare, socializzare grazie a Jitsi Meet

29 February 2020

Chris Lamb: Free software activities in February 2020

Here is my monthly update covering what I have been doing in the free software world during February 2020 (previous month): For the Tails privacy-oriented operating system, I uploaded the following packages to Debian:
Reproducible builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to provide the ability to demonstrate these binaries originated from a particular trusted source release: if identical results are generated from a given source in all circumstances, reproducible builds provides the means for multiple third-parties to reach a consensus on whether a build was compromised via distributed checksum validation or some other scheme. The initiative is proud to be a member project of the Software Freedom Conservancy, a not-for-profit 501(c)(3) charity focused on ethical technology and user freedom. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter. This month, I: In our tooling, I also made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, including uploading version 137 to Debian:
Debian I submitted a Request for Package (RFP) bug for hsd, a blockchain-based top-level domain DNS protocol implementation that underpins Handshake and worked on some initial packaging. (#952472)
Debian LTS This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project. You can find out more about the project via the following video:
Uploads Finally, I made a non-maintainer upload of adminer (4.7.6-1) on behalf of Alexandre Rossi.

23 November 2017

Sean Whitton: Using Propellor to provision your Debian development laptop

sbuild is a tool used by those maintaining packages in Debian, and derived distributions such as Ubuntu. When used correctly, it can catch a lot of categories of bugs before packages are uploaded. It does this by building the package in a clean environment, and then running the package through the Lintian, piuparts, adequate and autopkgtest tools. However, configuring sbuild so that it makes use of all of these tools is cumbersome. In response to this complexity, I wrote a module for the Propellor configuration management system to prepare a system such that a user can just go ahead and run the sbuild(1) command. This module is useful on one s development laptop if you need to reinstall your OS, you don t have to look up the instructions for setting up sbuild again. But it s also useful on throwaway build boxes. I can instruct propellor to provision a new virtual machine to build packages with sbuild, and all the different tools mentioned above will be connected together for me. I just uploaded Propellor version 5.1.0 to Debian unstable. The version overhauls the API and internals of the Sbuild module to take better advantage of Propellor s design. I won t get into those details in this post. What I d like to do is demonstrate how you can set up sbuild on your own machines, using Propellor. Getting started with Propellor apt-get install propellor, and then propellor --init. You ll be offered two setups, options A and B. I suggest starting with option B. If you never use Propellor for anything other than provisioning sbuild, you can stick with option B. If this tutorial makes you want to check out more features of Propellor, you might consider switching to option A and importing your old configuration. Open ~/.propellor/config.hs. You will see something like this: 
-- The hosts propellor knows about.
hosts :: [Host]
hosts =
        [ mybox
        ]
-- An example host.
mybox :: Host
mybox = host "mybox.example.com" $ props
        & osDebian Unstable X86_64
        & Apt.stdSourcesList
        & Apt.unattendedUpgrades
        & Apt.installed ["etckeeper"]
        & Apt.installed ["ssh"]
        & User.hasSomePassword (User "root")
        & File.dirExists "/var/www"
        & Cron.runPropellor (Cron.Times "30 * * * *")
You ll want to customise this so that it reflects your computer. My laptop is called iris, so I might replace the above with this: 
-- The hosts propellor knows about.
hosts :: [Host]
hosts =
        [ iris
        ]
-- My laptop.
iris :: Host
iris = host "iris.silentflame.com" $ props
        & osDebian Testing X86_64
The list of lines beginning with & are the properties of the host iris. Here, I ve removed all properties except the osDebian property, which informs propellor that iris runs Debian testing and has the amd64 architecture. The effect of this is that Propellor will not try to change anything about iris. In this tutorial, we are not going to let Propellor configure anything about iris other than setting up sbuild. (The osDebian property is a pure info property, which means that it tells Propellor information about the host to which other properties might refer, but it doesn t itself change anything about iris.) Telling Propellor to configure sbuild First, add to the import lines at the top of config.hs the lines: 
import qualified Propellor.Property.Sbuild as Sbuild
import qualified Propellor.Property.Schroot as Schroot
to enable use of the Sbuild module. Here is the full config for iris, which I ll go through line-by-line: 
-- The hosts propellor knows about.
hosts :: [Host]
hosts =
        [ iris
        ]
-- My laptop.
iris :: Host
iris = host "iris.silentflame.com" $ props
        & osDebian Testing X86_64
        & Apt.useLocalCacher
        & sidSchrootBuilt
        & Sbuild.usableBy (User "spwhitton")
        & Schroot.overlaysInTmpfs
        & Cron.runPropellor (Cron.Times "30 * * * *")
  where
        sidSchrootBuilt = Sbuild.built Sbuild.UseCcache $ props
                & osDebian Unstable X86_64
                & Sbuild.update  period  Daily
                & Sbuild.useHostProxy iris
Running Propellor to configure your laptop propellor iris.silentflame.com. In this configuration, you don t need to worry about whether the hostname iris.silentflame.com actually resolves to your laptop. However, it must be possible to ssh root@localhost. This should be enough that spwhitton can: 
$ sbuild -A --run-lintian --run-autopkgtest --run-piuparts foo.dsc
Further configuration It is easy to add new schroots; for example, for building backports: 
        ...
        & stretchSchrootBuilt
        ...
  where
        ...
        stretchSchrootBuilt = Sbuild.built Sbuild.UseCcache $ props
                & osDebian (Stable "stretch") X86_64
                & Sbuild.update  period  Daily
                & Sbuild.useHostProxy iris
You can also add additional properties to configure your chroot. Perhaps on your LAN you need sbuild to install packages via https, and you already have an apt cacher available. You can replace the apt-cacher-ng configuration like this: 
  where
        sidSchrootBuilt = Sbuild.built Sbuild.UseCcache $ props
                & osDebian Unstable X86_64
                & Sbuild.update  period  Daily
                & Apt.mirror "https://foo.mirror/debian/"
                & Apt.installed ["apt-transport-https"]
Thanks Thanks to Propellor s author, Joey Hess, for help navigating Propellor s type system while performing the overhaul included in version 5.1.0. Also for a conversation at DebConf17 which enabled this work by clearing some misconceptions of mine.

18 November 2017

Matthieu Caneill: MiniDebconf in Toulouse

I attended the MiniDebconf in Toulouse, which was hosted in the larger Capitole du Libre, a free software event with talks, presentation of associations, and a keysigning party. I didn't expect the event to be that big, and I was very impressed by its organization. Cheers to all the volunteers, it has been an amazing week-end! Here's a sum-up of the talks I attended. Du logiciel libre la monnaie libre Speaker: lo s The first talk I attended was, translated to English, "from free software to free money". lo s compared the 4 freedoms of free software with money, and what properties money needs to exhibit in order to be considered free. He then introduced 1, a project of free (as in free speech!) money, started in the region around Toulouse. Contrary to some distributed ledgers such as Bitcoin, 1 isn't based on an hash-based proof-of-work, but rather around a web of trust of people certifying each other, hence limiting the energy consumption required by the network to function. YunoHost Speaker: Jimmy Monin I then attended a presentation of YunoHost. Being an happy user myself, it was very nice to discover the future expected features, and also meet two of the developers. YunoHost is a Debian-based project, aimed at providing all the tools necessary to self-host applications, including email, website, calendar, development tools, and dozens of other packages. Premiers pas dans l'univers de Debian Speaker: Nicolas Dandrimont For the first talk of the MiniDebConf, Nicolas Dandrimont introduced Debian, its philosophy, and how it works with regards to upstreams and downstreams. He gave many details on the teams, the infrastructure, and the internals of Debian. Trusting your computer and system Speaker: Jonas Smedegaard Jonas introduced some security concepts, and how they are abused and often meaningless (to quote his own words, "secure is bullshit"). He described a few projects which lean towards a more secure and open hardware, for both phones and laptops. Automatiser la gestion de configuration de Debian avec Ansible Speaker: J r my Lecour J r my, from Evolix, introduced Ansible, and how they use it to manage hundreds of Debian servers. Ansible is a very powerful tool, and a huge ecosystem, in many ways similar to Puppet or Chef, except it is agent-less, using only ssh connections to communicate with remote machines. Very nice to compare their use of Ansible with mine, since that's the software I use at work for deploying experiments. Making Debian for everybody Speaker: Samuel Thibault Samuel gave a talk about accessibility, and the general availability of the tools in today's operating systems, including Debian. The lesson to take home is that we often don't do enough in this domain, particularly when considering some issues people might have that we don't always think about. Accessibility on computers (and elsewhere) should be the default, and never require complex setups. Retour d'exp rience : mise jour de milliers de terminaux Debian Speaker: Cyril Brulebois Cyril described a problem he was hired for, an update of thousands of Debian servers from wheezy to jessie, which he discovered afterwards was worse than initially thought, since the machines were running the out-of-date squeeze. Since they were not always administered with the best sysadmin practices, they were all exhibiting different configurations and different packages lists, which raised many issues and gave him interesting challenges. They were solved using Ansible, which also had the effect of standardizing their system administration practices. Retour d'exp rience : utilisation de Debian chez Evolix Speaker: Gr gory Colpart Gr gory described Evolix, a company which manages servers for their clients, and how they were inspired by Debian, for both their internal tools and their practices. It is very interesting to see that some of the Debian values can be easily exported for a more open and collaborative business. Lightning talks To close the conference, two lightning talks were presented, describing the switch from Windows XP to Debian in an ecologic association near Toulouse; and how snapshot.debian.org can be used with bisections to find the source of some regressions. Conclusion A big thank you to all the organizers and the associations who contributed to make this event a success. Cheers!

14 November 2017

Jonathan Dowland: WadC 2.2

Bird Cage, a WadC-generated map for Heretic
Bird Cage map
I have recently released version 2.2 of Wad Compiler, a lazy functional programming language and IDE for the construction of Doom maps. The biggest change in this version is a reworking of the preferences system (to use the Java Preferences API), the wadcli command-line interface respecting preferences and a new preferences UI dialog (adapted from Quake Injector). There are two new example maps: A Labyrinth demonstration contributed by "Yoruk", and a Heretic map Bird Cage by yours truly. These are both now amongst the largest examples in the collection, although laby.wl was generated by a higher-level program. For more information see the release notes and the reference, or check out the new gallery of examples or skip straight to downloads. I have no plans to work on WadC further (but never say never, I suppose.)

12 November 2017

Russ Allbery: Review: Night Moves

Review: Night Moves, by Pat Green
Publisher: Aquarius
Copyright: 2014
ISBN: 0-9909741-1-1
Format: Kindle
Pages: 159
In the fall of 2012, Pat Green was a preacher of a failing church, out of a job, divorced for six months, and feeling like a failure at every part of his life. He was living in a relative's house and desperately needed work and his father had been a taxi driver. So he got a job as a 6pm to 6am taxi driver in his home town of Joliet, Illinois. That job fundamentally changed his understanding of the people who live in the night, how their lives work, and what it means to try to help them. This is nonfiction: a collection of short anecdotes about life as a cab driver and the people who have gotten a ride in Green's cab. They're mostly five or six pages long, just a short story or window into someone's life. I ran across Pat Green's writing by following a sidebar link from a post on Patheos (probably from Love, Joy, Feminism, although I no longer remember). Green has an ongoing blog on Patheos about raising his transgender son (who appears in this collection as a lesbian daughter; he wasn't out yet as transgender when this was published), which is both a good sample of his writing and occasionally has excerpts from this book. Green's previous writing experience, as mentioned at several points in this collection, was newspaper columns in the local paper. It shows: these essays have the succinct, focused, and bite-sized property of a good newspaper article (or blog post). The writing is a little rough, particularly the remembered dialogue that occasionally falls into the awkward valley between dramatic, constructed fictional dialogue and realistic, in-the-moment speech. But the stories are honest and heartfelt and have the self-reflective genuineness of good preaching paired with a solid sense of narrative. Green tries to observe and report first, both the other person and his own reactions, and only then try to draw more general conclusions. This book is also very hard to read. It's not a sugar-coated view of people who live in the night of a city, nor is it constructed to produce happy endings. The people who Green primarily writes about are poor, or alone, or struggling. The story that got me to buy this book, about taking a teenage girl to a secret liaison that turned out to be secret because her liaison was another girl, is heartwarming but also one of the most optimistic stories here. A lot of people die or just disappear after being regular riders for some time. A lot of people are desperate and don't have any realistic way out. Some people, quite memorably, think they have a way out, and that way out closes on them. The subtitle of this book is "An Ex-Preacher's Journey to Hell in a Taxi" and (if you followed the link above) you'll see that Green is writing in the Patheos nonreligious section. The other theme of this collection is the church and its effect on the lives of people who are trying to make a life on the outskirts of society. That effect is either complete obliviousness or an active attempt to make their lives even worse. Green lays out the optimism that he felt early in the job, the hope that he could help someone the way a pastor would, guide her to resources, and how it went horribly wrong when those resources turned out to not be interested in helping her at all. And those stories repeat, and repeat. It's a book that makes it very clear that the actual practice of Christianity in the United States is not about helping poor or marginalized people, but there are certainly plenty of Christian resources for judging, hurting people, closing doors, and forcing abused people back into abusive situations, all in the name of God. I do hope some Christians read this and wince very hard. (And lest the progressive Christians get too smug, one of the stories says almost as brutal things about liberal ministries as the stories of conservative ones.) I came away feeling even more convinced by the merits of charities that just give money directly to poor people. No paternalism, no assuming that rich people know what they need, no well-meaning intermediary organizations with endless rules, just resources delivered directly to the people who most need resources. Ideally done by the government and called universal basic income. Short of constructing a functional government that builds working public infrastructure, and as a supplement even if one has such a government (since infrastructure can't provide everything), it feels like the most moral choice. Individual people may still stay mired in awful situations, but at least that isn't compounded by other people taking their autonomy away and dictating life to them in complete ignorance. This is a fairly short and inexpensive book. I found it very much worth reading, and may end up following Green's blog as well. There are moments of joy and moments of human connection, and the details of the day-to-day worries and work style of a taxi driver (in this case, one who drives a company car) are pretty interesting. (Green does skip over some parts for various reasons, such as a lot of the routine fares and most of the stories of violence, but does mention what he's skipping over.) But it's also a brutal book, because so many people are hurting and there isn't much Green can do about it except bear witness and respect them as people in a way that religion doesn't. Recommended, but brace yourself. Rating: 8 out of 10

8 November 2017

Neil McGovern: Software Freedom Law Center and Conservancy

Before I start, I would like to make it clear that the below is entirely my personal view, and not necessarily that of the GNOME Foundation, the Debian Project, or anyone else. There s been quite a bit of interest recently about the petition by Software Freedom Law Center to cancel the Software Freedom Conservancy s trademark. A number of people have asked my views on it, so I thought I d write up a quick blog on my experience with SFLC and Conservancy both during my time as Debian Project Leader, and since. It s clear to me that for some time, there s been quite a bit of animosity between SFLC and Conservancy, which for me started to become apparent around the time of the large debate over ZFS on Linux. I talked about this in my DebConf 16 talk, which fortunately was recorded (ZFS bit from 8:05 to 17:30).
This culminated in SFLC publishing a statement, and Conservancy also publishing their statement, backed up by the FSF. These obviously came to different conclusions, and it seems bizarre to me that SFLC who were acting as Debian s legal counsel published a position that was contrary to the position taken by Debian. Additionally, Conservancy and FSF who were not acting as counsel mirrored the position of the project. Then, I hear of an even more confusing move that SFLC has filed legal action against Conservancy, despite being the organisation they helped set up. This happened on the 22nd September, the day after SFLC announced corporate and support services for Free Software projects. SFLC has also published a follow up, which they say that the act is not an attack, let alone a bizarre attack , and that the response from Conservancy, who view it as such was like reading a declaration of war issued in response to a parking ticket . Then, as SFLC somehow find the threat of your trademark being taken away as something other than an attack, they also state: Any project working with the Conservancy that feels in any way at risk should contact us. We will immediately work with them to put in place measures fully ensuring that they face no costs and no risks in this situation. which I read as a direct pitch to try and pull projects away from Conservancy and over to SFLC. Now, even if there is a valid claim here, despite the objections that were filed by a trademark lawyer who I have a great deal of respect for (disclosure: Pam also provides pro-bono trademark advice to my employer, the GNOME Foundation), the optics are pretty terrible. We have a case of one FOSS organisation taking another one to court, after many years of them being aware of the issue, and when wishing to promote a competing service. At best, this is a distraction from the supposed goals of Free Software organisations, and at worst is a direct attempt to interrupt the workings of an established and successful umbrella organisation which lots of projects rely on. I truly hope that this case is simply dropped, and if I was advising SFLC, that s exactly what I would suggest, along with an apology for the distress. Put it this way if SFLC win, then they re simply displaying what would be viewed as an aggressive move to hold the term software freedom exclusively to themselves. If they lose, then it shows that they re willing to do so to another 501(c)3 without actually having a case. Before I took on the DPL role, I was under the naive impression that although there were differences in approach, at least we were coming to try and work together to promote software freedoms for the end user. Unfortunately, since then, I ve now become a lot more jaded about exactly who, and which organisations hold our best interests at heart. (Featured image by Nick Youngson CC-BY-SA-3.0 http://nyphotographic.com/)

6 November 2017

John Goerzen: The Yellow House Phone Company (Featuring Asterisk and an 11-year-old)

Well Jacob, do you think we should set up our own pretend phone company in the house? We can DO THAT? Yes! Then yes. Yes! YES YES YESYESYESYES YES! Let s do it, dad! Not long ago, my parents had dug up the old phone I used back in the day. We still have a landline, and Jacob was having fun discovering how an analog phone works. I told him about the special number he could call to get the time and temperature read out to him. He discovered what happens if you call your own number and hang up. He figured out how to play Mary Had a Little Lamb using touchtone keys (after a slightly concerned lecture from me setting out some rules to make sure his musical dialing wouldn t result in any, well, dialing.) He was hooked. So I thought that taking it to the next level would be a good thing for a rainy day. I have run Asterisk before, though I had unfortunately gotten rid of most of my equipment some time back. But I found a great deal on a Cisco 186 ATA (Analog Telephone Adapter). It has two FXS lines (FXS ports simulate the phone company, and provide dialtone and ring voltage to a connected phone), and of course hooks up to the LAN. We plugged that in, and Jacob was amazed to see its web interface come up. I had to figure out how to configure it (unfortunately, it uses SCCP rather than SIP, and figuring out Asterisk s chan_skinny took some doing, but we got there.) I set up voicemail. He loved it. He promptly figured out how to record his own greetings. We set up a second phone on the other line, so he could call between them. The cordless phones in our house support SIP, so I configured one of them as a third line. He spent a long time leaving himself messages. IMG_3465 Pretty soon we both started having ideas. I set up extension 777, where he could call for the time. Then he wanted a way to get the weather forecast. Well, weather-util generates a text-based report. With it, a little sed and grep tweaking, the espeak TTS engine, and a little help from sox, I had a shell script worked up that would read back a forecast whenever he called a certain extension. He was super excited! That s great, dad! Can it also read weather alerts too? Sure! weather-util has a nice option just for that. Both boys cackled as the system tried to read out the NWS header (their timestamps like 201711031258 started with two hundred one billion ) Then I found an online source for streaming NOAA Weather Radio feeds Jacob enjoys listening to weather radio and I set up another extension he could call to listen to that. More delight! But it really took off when I asked him, Would you like to record your own menu? You mean those things where it says press 1 or 2 for this or that? Yes. WE CAN DO THAT? Oh yes! YES, LET S DO IT RIGHT NOW! So he recorded a menu, then came and hovered by me while I hacked up extensions.conf, then eagerly went back to the phone to try it. Oh the excitement of hearing hisown voice, and finding that it worked! Pretty soon he was designing sub-menus ( OK Dad, so we ll set it up so people can press 2 for the weather, and then choose if they want weather radio or the weather report. I m recording that now. Got it? ) He has informed me that next Saturday we will build an intercom system like we have at school. I m going to have to have some ideas on how to tie Squeezebox in with Asterisk to make that happen, I think. Maybe this will do.

29 October 2017

Niels Thykier: Building packages without (fake)root

Turns out that it is surprisingly easy to build most packages without (fake)root. You just need to basic changes:
  1. A way to set ownership to root:root of paths when dpkg-deb build constructs the binary.
  2. A way to have debhelper not do a bunch of (now) pointless chowns to root:root .
The above is sufficient for dpkg, debhelper, lintian, apt-file, mscgen, pbuilder and a long list of other packages that only provide paths owned by root:root . Obviously, packages differ and yours might need more tweaks than this (e.g. dh_usrlocal had to change behaviour to support this). But for me, the best part is that the above is not just some random prototype stuck in two git repos on alioth: Unfortunately, if you are working with games or core packages like shadow with need for static ownership different from root:root (usually with a setuid or setgid bit), then our first implementation does not support your needs at the moment[1]. We are working on a separate way to solve static ownership in a declarative way. [1] Note regarding /usr/local : If your package needs to provide directories there owned by root:staff with mode 02775, then dh_usrlocal can handle that. The non- root:root ownership here works because the directories are created in a maintainer script run as root during installation. Unfortunately, it cannot provide different ownership or modes with R != binary-targets at the moment.
Filed under: Debhelper, Debian

25 October 2017

Russ Allbery: Review: Clean Sweep

Review: Clean Sweep, by Ilona Andrews
Series: Innkeeper Chronicles #1
Publisher: NYLA
Copyright: 2013
ISBN: 1-62517-343-1
Format: Kindle
Pages: 228
Dina owns a bed and breakfast in a small town in Texas. As the book opens, her neighbor's dogs are being killed by some beast. As far as Dina is concerned, this should be the problem of the local werewolf, given that this is his territory, and she takes him to task for apparently doing nothing about it. The werewolf doesn't take that well, nor wants to admit to being a werewolf, and reacts by threatening Dina. Given Dina's position and the nature of her bed and breakfast, that turns out to be ill-advised. At this point, you're doubtless thinking urban fantasy, and you're not exactly wrong. You're probably also thinking love interest, and you're not entirely wrong there either. The dead dogs are, of course, an early warning sign of an evil monster who has moved into the area and whose violence may escalate. And if I mention that a vampire shows up later in the story and the werewolf and vampire dislike each other at first sight, you'll probably start rolling your eyes. But this also isn't quite what it looks like on the surface. I'm going to gleefully spoil what you find out about a fifth of the way into the book, since it's what got me to buy this book: neither the werewolves nor the vampires are magical creatures. They're aliens. Werewolves are bio-engineered soldiers; vampires are members of a militaristic order with advanced technology and almost imperceptible circulatory systems. And Dina doesn't truly have a bed and breakfast. She maintains an inn: a refuge for aliens traveling on Earth, part of the secrecy that keeps them out of the eyes of normal humans, and an institution sworn to neutrality in local problems. Dina is straining the last part. The alien creatures who are hunting and killing her neighbors' pets aren't exactly threatening the inn, but she's not willing to stand idly by while her neighbors are threatened. It's a dangerous path, since her inn is struggling already. Failure to prioritize correctly and protect her guests might lower her inn's rating, possibly fatally. She's also without a guide ever since her parents (prominent innkeepers themselves) mysteriously disappeared. But she does have an inn. It may be a weaker one, coaxed out of dormancy and desperately in need of more visitors, but it still has considerable technological resources and other abilities that are indistinguishable from magic. It's also sentient, although more like a large animal than a person, and it knows its innkeeper. This is a light, straightforward sort of book that does not take itself particularly seriously, as you might have guessed. It is very aware of the conventions of urban fantasy and is both following them and intentionally poking fun at them. The authors (Ilona Andrews is a husband and wife writing team) play an impish game of wedging fantasy tropes into a science fiction framework, preserving much of the outward appearance while playing with the explanations. I particularly liked the clannish reworking of vampires into a sort of crusader knight, which works considerably better than it has any right to. Clean Sweep is at its best when the story seems to be going down well-trodden urban fantasy paths and then takes an abrupt right turn into science fiction: Dina going through a dimensional gate to an interstellar marketplace to stock up on weapons, for instance. Also, it has a sentient house. This is one of my favorite story elements ever, provided that the story isn't horror (which is all too often the case with sentient houses, but which is not at all the case here). I found the early parts of this book, during which Sean is insulting Dina and Dina is unimpressed but not doing anything about it, rather tedious. Thankfully, once Dina finally loses her patience and knocks some sense into him it gets a lot better. It never becomes great literature, but sometimes that's a feature. If you're in the mood for some urban fantasy with an adequate, if hand-waved, re-spun science fiction justification and an uncomplicated and loyal sentient house, I can wholeheartedly recommend Clean Sweep. I liked it better than I had expected to. Followed by Sweep in Peace. Rating: 7 out of 10

23 October 2017

Russ Allbery: Review: Algorithms to Live By

Review: Algorithms to Live By, by Brian Christian & Tom Griffiths
Publisher: Henry Holt and Company
Copyright: April 2016
ISBN: 1-62779-037-3
Format: Kindle
Pages: 255
Another read for the work book club. This was my favorite to date, apart from the books I recommended myself. One of the foundations of computer science as a field of study is research into algorithms: how do we solve problems efficiently using computer programs? This is a largely mathematical field, but it's often less about ideal or theoretical solutions and more about making the most efficient use of limited resources and arriving at an adequate, if not perfect, answer. Many of these problems are either day-to-day human problems or are closely related to them; after all, the purpose of computer science is to solve practical problems with computers. The question asked by Algorithms to Live By is "can we reverse this?": can we learn lessons from computer science's approach to problems that would help us make day-to-day decisions? There's a lot of interesting material in the eleven chapters of this book, but there's also an amusing theme: humans are already very good at this. Many chapters start with an examination of algorithms and mathematical analysis of problems, dive into a discussion of how we can use those results to make better decisions, then talk about studies of the decisions humans actually make... and discover that humans are already applying ad hoc versions of the best algorithms we've come up with, given the constraints of typical life situations. It tends to undermine the stated goal of the book. Thankfully, it in no way undermines interesting discussion of general classes of problems, how computer science has tackled them, and what we've learned about the mathematical and technical shapes of those problems. There's a bit less self-help utility here than I think the authors had intended, but lots of food for thought. (That said, it's worth considering whether this congruence is less because humans are already good at this and more because our algorithms are designed from human intuition. Maybe our best algorithms just reflect human thinking. In some cases we've checked our solutions against mathematical ideals, but in other cases they're still just our best guesses to date.) This is the sort of a book where a chapter listing is an important part of the review. The areas of algorithms discussed here are optimal stopping, explore/exploit decisions (when to go with the best thing you've found and when to look for something better), sorting, caching, scheduling, Bayes's rule (and prediction in general), overfitting when building models, relaxation (solving an easier problem than your actual problem), randomized algorithms, a collection of networking algorithms, and finally game theory. Each of these has useful insights and thought-provoking discussion of how these sometimes-theoretical concepts map surprisingly well onto daily problems. The book concludes with a discussion of "computational kindness": an encouragement to reduce the required computation and complexity penalty for both yourself and the people you interact with. If you have a computer science background (as I do), many of these will be familiar concepts, and you might be dubious that a popularization would tell you much that's new. Give this book a shot, though; the analogies are less stretched than you might fear, and the authors are both careful and smart about how they apply these principles. This book passes with flying colors a key sanity check: the chapters on topics that I know well or have thought about a lot make few or no obvious errors and say useful and important things. For example, the scheduling chapter, which unsurprisingly is about time management, surpasses more than half of the time management literature by jumping straight to the heart of most time management problems: if you're going to do everything on a list, it rarely matters the order in which you do it, so the hardest scheduling problems are about deciding what not to do rather than deciding order. The point in the book where the authors won my heart completely was in the chapter on Bayes's rule. Much of the chapter is about Bayesian priors, and how one's knowledge of past events is a vital part of analysis of future probabilities. The authors then discuss the (in)famous marshmallow experiment, in which children are given one marshmallow and told that if they refrain from eating it until the researcher returns, they'll get two marshmallows. Refraining from eating the marshmallow (delayed gratification, in the psychological literature) was found to be associated with better life outcomes years down the road. This experiment has been used and abused for years for all sorts of propaganda about how trading immediate pleasure for future gains leads to a successful life, and how failure in life is because of inability to delay gratification. More evil analyses have (of course) tied that capability to ethnicity, with predictably racist results. I have kind of a thing about the marshmallow experiment. It's a topic that reliably sends me off into angry rants. Algorithms to Live By is the only book I have ever read to mention the marshmallow experiment and then apply the analysis that I find far more convincing. This is not a test of innate capability in the children; it's a test of their Bayesian priors. When does it make perfect sense to eat the marshmallow immediately instead of waiting for a reward? When their past experience tells them that adults are unreliable, can't be trusted, disappear for unpredictable lengths of time, and lie. And, even better, the authors supported this analysis with both a follow-up study I hadn't heard of before and with the observation that some children would wait for some time and then "give in." This makes perfect sense if they were subconsciously using a Bayesian model with poor priors. This is a great book. It may try a bit too hard in places (applicability of the math of optimal stopping to everyday life is more contingent and strained than I think the authors want to admit), and some of this will be familiar if you've studied algorithms. But the writing is clear, succinct, and very well-edited. No part of the book outlives its welcome; the discussion moves right along. If you find yourself going "I know all this already," you'll still probably encounter a new concept or neat explanation in a few more pages. And sometimes the authors make connections that never would have occurred to me but feel right in retrospect, such as relating exponential backoff in networking protocols to choosing punishments in the criminal justice system. Or the realization that our modern communication world is not constantly connected, it's constantly buffered, and many of us are suffering from the characteristic signs of buffer bloat. I don't think you have to be a CS major, or know much about math, to read this book. There is a lot of mathematical details in the end notes if you want to dive in, but the main text is almost always readable and clear, at least so far as I could tell (as someone who was a CS major and has taken a lot of math, so a grain of salt may be indicated). And it still has a lot to offer even if you've studied algorithms for years. The more I read of this book, the more I liked it. Definitely recommended if you like reading this sort of analysis of life. Rating: 9 out of 10

19 October 2017

Steinar H. Gunderson: Introducing Narabu, part 2: Meet the GPU

Narabu is a new intraframe video codec. You may or may not want to read part 1 first. The GPU, despite being extremely more flexible than it was fifteen years ago, is still a very different beast from your CPU, and not all problems map well to it performance-wise. Thus, before designing a codec, it's useful to know what our platform looks like. A GPU has lots of special functionality for graphics (well, duh), but we'll be concentrating on the compute shader subset in this context, ie., we won't be drawing any polygons. Roughly, a GPU (as I understand it!) is built up about as follows: A GPU contains 1 20 cores; NVIDIA calls them SMs (shader multiprocessors), Intel calls them subslices. (Trivia: A typical mid-range Intel GPU contains two cores, and thus is designated GT2.) One such core usually runs the same program, although on different data; there are exceptions, but typically, if your program can't fill an entire core with parallelism, you're wasting energy. Each core, in addition to tons (thousands!) of registers, also has some shared memory (also called local memory sometimes, although that term is overloaded), typically 32 64 kB, which you can think of in two ways: Either as a sort-of explicit L1 cache, or as a way to communicate internally on a core. Shared memory is a limited, precious resource in many algorithms. Each core/SM/subslice contains about 8 execution units (Intel calls them EUs, NVIDIA/AMD calls them something else) and some memory access logic. These multiplex a bunch of threads (say, 32) and run in a round-robin-ish fashion. This means that a GPU can handle memory stalls much better than a typical CPU, since it has so many streams to pick from; even though each thread runs in-order, it can just kick off an operation and then go to the next thread while the previous one is working. Each execution unit has a bunch of ALUs (typically 16) and executes code in a SIMD fashion. NVIDIA calls these ALUs CUDA cores , AMD calls them stream processors . Unlike on CPU, this SIMD has full scatter/gather support (although sequential access, especially in certain patterns, is much more efficient than random access), lane enable/disable so it can work with conditional code, etc.. The typically fastest operation is a 32-bit float muladd; usually that's single-cycle. GPUs love 32-bit FP code. (In fact, in some GPU languages, you won't even have 8-, 16-bit or 64-bit types. This is annoying, but not the end of the world.) The vectorization is not exposed to the user in typical code (GLSL has some vector types, but they're usually just broken up into scalars, so that's a red herring), although in some programming languages you can get to swizzle the SIMD stuff internally to gain advantage of that (there's also schemes for broadcasting bits by voting etc.). However, it is crucially important to performance; if you have divergence within a warp, this means the GPU needs to execute both sides of the if. So less divergent code is good. Such a SIMD group is called a warp by NVIDIA (I don't know if the others have names for it). NVIDIA has SIMD/warp width always 32; AMD used to be 64 but is now 16. Intel supports 4 32 (the compiler will autoselect based on a bunch of factors), although 16 is the most common. The upshot of all of this is that you need massive amounts of parallelism to be able to get useful performance out of a CPU. A rule of thumb is that if you could have launched about a thousand threads for your problem on CPU, it's a good fit for a GPU, although this is of course just a guideline. There's a ton of APIs available to write compute shaders. There's CUDA (NVIDIA-only, but the dominant player), D3D compute (Windows-only, but multi-vendor), OpenCL (multi-vendor, but highly variable implementation quality), OpenGL compute shaders (all platforms except macOS, which has too old drivers), Metal (Apple-only) and probably some that I forgot. I've chosen to go for OpenGL compute shaders since I already use OpenGL shaders a lot, and this saves on interop issues. CUDA probably is more mature, but my laptop is Intel. :-) No matter which one you choose, the programming model looks very roughly like this pseudocode: 
for (size_t workgroup_idx = 0; workgroup_idx < NUM_WORKGROUPS; ++workgroup_idx)     // in parallel over cores
        char shared_mem[REQUESTED_SHARED_MEM];  // private for each workgroup
        for (size_t local_idx = 0; local_idx < WORKGROUP_SIZE; ++local_idx)    // in parallel on each core
                main(workgroup_idx, local_idx, shared_mem);
except in reality, the indices will be split in x/y/z for your convenience (you control all six dimensions, of course), and if you haven't asked for too much shared memory, the driver can silently make larger workgroups if it helps increase parallelity (this is totally transparent to you). main() doesn't return anything, but you can do reads and writes as you wish; GPUs have large amounts of memory these days, and staggering amounts of memory bandwidth. Now for the bad part: Generally, you will have no debuggers, no way of logging and no real profilers (if you're lucky, you can get to know how long each compute shader invocation takes, but not what takes time within the shader itself). Especially the latter is maddening; the only real recourse you have is some timers, and then placing timer probes or trying to comment out sections of your code to see if something goes faster. If you don't get the answers you're looking for, forget printf you need to set up a separate buffer, write some numbers into it and pull that buffer down to the GPU. Profilers are an essential part of optimization, and I had really hoped the world would be more mature here by now. Even CUDA doesn't give you all that much insight sometimes I wonder if all of this is because GPU drivers and architectures are meant to be shrouded in mystery for competitiveness reasons, but I'm honestly not sure. So that's it for a crash course in GPU architecture. Next time, we'll start looking at the Narabu codec itself.

16 October 2017

Iain R. Learmonth: No more no surprises

Debian has generally always had, as a rule, sane defaults and no surprises . This was completely shattered for me when Vim decided to hijack the mouse from my terminal and break all copy/paste functionality. This has occured since the release of Debian 9. I expect for my terminal to behave consistently, and this is broken every time I log in to a Debian 9 system where I have not configured Vim to disable this functionality. I also see I m not alone in this frustration. To fix this, in your .vimrc: 
if !has("gui_running")
  set mouse=
endif
(This will check to see if your using GVim or similar, where it would be reasonable to expect the mouse to work.) This is perhaps not aggresive enough though. I never want to have console applications trying to use the mouse. I ve configured rxvt to do things like open URLs in Firefox, etc. that I always want to work, and I always want my local clipboard to be used so I can copy/paste between remote machines. I ve found a small patch that would appear to disable mouse reporting for rxvt, but unfortunately I cannot do this through an Xresources option. If someone is looking for something to do for Hacktoberfest, I d love to see this be an option for rxvt without re-compiling: 
diff --git a/src/rxvt.h b/src/rxvt.h
index 5c7cf66..2751ba3 100644
--- a/src/rxvt.h
+++ b/src/rxvt.h
@@ -646,7 +646,7 @@ enum  
 #define PrivMode_ExtMouseRight  (1UL<<24) // xterm pseudo-utf-8, but works in non-utf-8-locales
 #define PrivMode_BlinkingCursor (1UL<<25)
 
-#define PrivMode_mouse_report   (PrivMode_MouseX10 PrivMode_MouseX11 PrivMode_MouseBtnEvent PrivMode_MouseAnyEvent)
+#define PrivMode_mouse_report   0 /* (PrivMode_MouseX10 PrivMode_MouseX11 PrivMode_MouseBtnEvent PrivMode_MouseAnyEvent) */
 
 #ifdef ALLOW_132_MODE
 # define PrivMode_Default (PrivMode_Autowrap PrivMode_ShiftKeys PrivMode_VisibleCursor PrivMode_132OK)

13 October 2017

Shirish Agarwal: I need to speak up now X Economics

Dear all, This would be a longish blog post (as most of mine are) compiled over days but as there is so short a time and so much to share. I had previously thought to share beautiful photographs of Ganesh mandals taking out the procession at time of immersion of the idol or the last day of Durga Puja recent events around do not make my mood to share photos at this point in time. I may share some of them in a future blog post or two . Before going further, I would like to offer my sympathies and condolences to people hurt and dislocated in Hurricane Irma , the 2017 Central Mexico Earthquake and lastly the most recent Las Vegas shooting as well as Hurricane Maria in Puerto Rico . I am somewhat nonplussed as to why Americans always want to name, especially hurricanes which destroy people s lives and livelihood built over generations and why most of the hurricanes are named after women. A look at weather.com site unveiled the answer to the mystery. Ironically (or not) I saw some of the best science coverage about Earthquakes or anything scientific reporting and analysis after a long time in mainstream newspapers in India. On another note, I don t understand or even expect to understand why the gunman did what he did 2 days back. Country music AFAIK is one of the most chilled-out kind of music, in some ways very similar to classical Indian singing although they are worlds apart in style of singing, renditions, artists, the way they emote etc. I seriously wish that the gunman had not been shot but caught and reasons were sought about what he did, he did. While this is certainly armchair thinking as was not at the scene of crime, but if a Mumbai Police constable could do it around a decade ago armed only with a lathi could do it, why couldn t the American cops who probably are trained in innumerable ways to subdue people without killing them, did. While investigations are on, I suspect if he were caught just like Ajmal Kasab was caught then lot of revelations might have come up. From what is known, the gentleman was upwardly mobile i.e. he was white, rich and apparently had no reason to have beef with anybody especially a crowd swaying to some nice music, all of which makes absolutely no sense. Indian Economy Slowdown Anyways, back to one of the main reasons of writing this blog post. Few days back, an ex-finance Minister of India Yashwant Sinha wrote what was felt by probably millions of Indians, an Indian Express article called I need to speak up now While there have been many, many arguments made since then by various people. A simple search of I need to speak up would lead to lead to many a result besides the one I have shared above. The only exception I have with the article is the line Forty leading companies of the country are already facing bankruptcy proceedings. Many more are likely to follow suit. I would not bore you but you ask any entrepreneur trying to set up shop in India i.e. ones who actually go through the processes of getting all the licenses for setting up even a small businesses as to the numerous hurdles they have to overcome and laid-back corrupt bureaucracy which they have to overcome. I could have interviewed some of my friends who had the conviction and the courage to set up shop and spent more than half a decade getting all the necessary licenses and approval to set up but it probably would be too specific for one industry or the other and would lead to the same result. Co-incidentally, a new restaurant, leaf opened in my vicinity few weeks before. From the looks it looked like a high-brow, high-priced restaurant hence like many others I did not venture in. After a few days, they introduced south-Indian delicacies like Masala Dosa, Uttapam at prices similar to other restaurants around. So I ventured in and bought some south Indian food to consume between mum and me. Few days later, I became friends with the owner/franchisee and I suggested (in a friendly tone) that why he doesn t make it like a CCD play where many people including yours truly use the service to share, strategize and meet with clients. The CCD joints usually serve coffee and snacks (which are over-priced but still run out pretty fast) but people come as they have chilled-out atmosphere and Wi-Fi access which people need for their smartphones, although the Wi-Fi part may soon become redundant With Reliance Jio making a big play. I also shared why he doesn t add more variety and time (the south Indian items are time-limited) as I see/saw many empty chairs there. Anyways, the shop-owner/franchisee shared his gross costs including salary, stocking, electricity, rent and it doesn t pan out to be serving Rs.80/- dish (roughly a 1US dollar and 25 cents) then serving INR Rs. 400/- a dish (around 6 $USD). One round of INR 400/- + dishes make his costs for the day, around 12 tables were there. It s when they have two full rounds of dishes costing INR 400/- or more that he actually has profits and he is predicting loss for at least 6 months to a year before he makes a rebound. He needs steady customers rather than just walk-ins that will make his business work/click. Currently his family is bearing the costs. He didn t mention the taxes although I know apart from GST there are still some local body taxes that they will have to pay and comply with. There are a multitude of problems for shutting a shop legally as well as they have to again renavigate the bureaucracy for the same. I have seen more than a few retailers downing their shutters for 6-8 months and then either sell it to new management, let go of the lease or simply sell the property to a competitor. The Insolvency and Bankruptcy Code is probably the first proper exit policy for large companies. So the 40 odd companies that Mr. Sinha were talking about were probably sick for a long time. In India, there is also an additional shame of being a failed entrepreneur unlike in the west where Entrepreneurs start on their next venture. As seen from Retailing In India only 3.3% of the population or at the most 4% of the population is directly or indirectly linked with the retail trade. Most of the economy still derives its wealth from the agrarian sector which is still reeling under the pressure from demonetization which happened last year. Al jazeera surprisingly portrayed a truer picture of the effects demonetization had on common citizen than many Indian newspapers did at the time. Because of the South African Debconf, I had to resort to debit cards and hence was able to escape standing in long lines in which many an old and women perished. It is only yesterday that the Government has acknowledged which many prominent Indians have been saying for months now, that we are in a slowdown . Be aware of the terms being used for effect by the Prime Minister. There are two articles which outlines the troubles India is in atm. The only bright spot has been e-commerce which so far has eluded GST although the Govt. has claimed regulations to put it in check. Indian Education System Interestingly, Ravish Kumar has started a series on NDTV where he is showcasing how Indian education sector, especially public colleges have been left to teachers on contract basis, see the first four episodes on NDTV channel starting with the first one I have shared as a hyperlink. I apologize as the series is in Hindi as the channel is meant for Indians and is mostly limited to Northern areas of the Country (mostly) although he has been honest that it is because they lack resources to tackle the amount of information flowing to them. Ravish started the series with sharing information about the U.S. where the things are similar with some teachers needing to sleep in cars because of high-cost of living to some needing to turn to sex-work . I was shocked when I read the guardian article, that is no way to treat our teachers.I went on to read How the American University was Killed following the breadcrumbs along the way. Reading that it seems Indians have been following the American system playbook from the 1980 s itself. The article talks about HMO as well and that seems to have followed here as well with my own experience of hospital fees and drugs which I had to entail a few weeks/month ago. Few years ago, when me and some of my friends had the teaching bug and we started teaching in a nearby municipal school, couple of teachers had shared that they were doing 2-3 jobs to make ends meet. I don t know about others in my group, at least I was cynical because I thought all the teachers were permanent and they make good money only to realize now that the person was probably speaking the truth. When you have to do three jobs to make ends meet from where do you bring the passion to teach young people and that too outside the syllabus ? Also, with this new knowledge in hindsight, I take back all my comments I made last year and the year before for the pathetic education being put up by the State. With teachers being paid pathetically/underpaid and almost 60% teachers being ad-hoc/adjunct teachers they have to find ways to have some sense of security. Most teachers are bachelors as they are poor and cannot offer any security (either male or female) and for women, after marriage it actually makes no sense for them to continue in this profession. I salute all the professors who are ad-hoc in nature and probably will never get a permanent position in their life. I think in some way, thanx to him, that the government has chosen to give 7th pay commisson salary to teachers. While the numbers may appear be large, there are a lot of questions as to how many people will actually get paid. There needs to be lot of vacancies which need to be filled quickly but don t see any solution in the next 2-3 years as well. The Government has taken a position to use/re-hire retired teachers rather than have new young teachers as an unwritten policy. In this Digital India context how are retired teachers supposed to understand and then pass on digital concepts is beyond me when at few teacher trainings I have seen they lack even the most basic knowledge that I learnt at least a decade or two ago, the difference is that vast. I just don t know what to say to that. My own experience with my own mother who had pretty good education in her time and probably would have made a fine business-woman if she knew that she will have a child that she would have to raise by herself alone (along with maternal grand-parents) is testimonial to the fact how hard it is for older people to grasp technology and here I m talking just using the interface as a consumer rather than a producer or someone in-between who has the idea of how companies and governments profit from whatever data is shared one way or the other. After watching the series/episodes and discussing the issue with my mother it was revealed that both her and my late maternal grandfather were on casual/ad-hoc basis till 20-25 years in their service in the defense sector. If Ravish were to do a series on the defense sector he probably would find the same thing there. To add to that, the defense sector is a vital component to a country s security. If 60% of the defense staff in all defense establishments have temporary staff how do you ensure the loyalty of the people working therein. That brings to my mind Ignorance is bliss . Software development and deployment There is another worry that all are skirting around, the present dispensation/government s mantra is minimum government-maximum governance with digital technologies having all solutions which is leading to massive unemployment. Also from most of the stories/incidents I read in the newspapers, mainstream media and elsewhere it seems most software deployments done in India are done without having any system of internal checks and balances. There is no lintian for software to be implemented. Contracts seem to be given to big companies and there is no mention of what prerequisites or conditions were laid down by the Government for software development and deployment and if any checks were done to ensure that the software being developed was in according to government specifications or not. Ideally this should all be in public domain so that questions can be asked and responsibility fixed if things go haywire, as currently they do not. Software issues As my health been not that great, I have been taking a bit more time and depth while filing bugs. #877638 is a good example. I suspect though that part of the problem might be that mate has moved to gtk3 while guake still has gtk-2 bindings. I also reported the issue upstream both in mate-panel as well as guake . I haven t received any response from either or/and upstreams . I also have been fiddling around with gdb to better understand the tool so I can exploit/use this tool in a better way. There are some commands within the gdb interface which seem to be interesting and hopefully I ll try how the commands perform over days, weeks to a month. I hope we see more action on the mate-panel/guake bug as well as move of guake to gtk+3 but that what seemingly seemed like wait for eternity seems to have done by somebody in last couple of days. As shared in the ticket there are lots of things still to do but it seems the heavy lifting has been done but seems merging will be tricky as two developers have been trying to update to gtk+3 although aichingm seems to have a leg up with his 3! branch. Another interesting thing I saw is the below picture. Firefox is out of date on wordpress.com The firefox version I was using to test the site/wordpress-wp-admin was Mozilla Firefox 52.4.0 which AFAIK is a pretty recentish one and people using Debian stretch would probably be using the same version (firefox stable/LTS) rather than the more recent versions. I went to the link it linked to and it gave no indication as to why it thought my browser is out-of-date and what functionality was/is missing. I have found that wordpress support has declined quite a bit and people don t seem to use the forums as much as they used to before. I also filed a few bugs for qalculate. #877716 where a supposedly transitional package removes the actual application, #877717 as the software has moved its repo. to github.com as well as tickets and other things in process and lastly #877733. I had been searching for a calculator which can do currency calculations on the fly (say for e.g. doing personal budgeting for Taiwan debconf) without needing to manually enter the conversion rates and losing something in the middle. While the current version has support for some limited currencies, the new versions promise more as other people probably have more diverse needs for currency conversions (people who do long or short on oil, stocks overseas is just one example, I am sure there are many others) than simplistic mine.
Filed under: Miscellenous Tagged: #American Education System, #bug-filing, #Climate change, #Dignity, #e-commerce, #gtk+3, #gtk2, #Indian Economy 'Slowdown', #Indian Education System, #Insolvency and Bankruptcy Code, #Las Vegas shooting, #Modern Retail in India, #planet-debian, #qalculate, Ad-hoc and Adjunct Professors, wordpress.com

12 October 2017

Joachim Breitner: Isabelle functions: Always total, sometimes undefined

Often, when I mention how things work in the interactive theorem prover [Isabelle/HOL] (in the following just Isabelle 1) to people with a strong background in functional programming (whether that means Haskell or Coq or something else), I cause confusion, especially around the issue of what is a function, are function total and what is the business with undefined. In this blog post, I want to explain some these issues, aimed at functional programmers or type theoreticians. Note that this is not meant to be a tutorial; I will not explain how to do these things, and will focus on what they mean.

HOL is a logic of total functions If I have a Isabelle function f :: a b between two types a and b (the function arrow in Isabelle is , not ), then by definition of what it means to be a function in HOL whenever I have a value x :: a, then the expression f x (i.e. f applied to x) is a value of type b. Therefore, and without exception, every Isabelle function is total. In particular, it cannot be that f x does not exist for some x :: a. This is a first difference from Haskell, which does have partial functions like 
spin :: Maybe Integer -> Bool
spin (Just n) = spin (Just (n+1))
Here, neither the expression spin Nothing nor the expression spin (Just 42) produce a value of type Bool: The former raises an exception ( incomplete pattern match ), the latter does not terminate. Confusingly, though, both expressions have type Bool. Because every function is total, this confusion cannot arise in Isabelle: If an expression e has type t, then it is a value of type t. This trait is shared with other total systems, including Coq. Did you notice the emphasis I put on the word is here, and how I deliberately did not write evaluates to or returns ? This is because of another big source for confusion:

Isabelle functions do not compute We (i.e., functional programmers) stole the word function from mathematics and repurposed it2. But the word function , in the context of Isabelle, refers to the mathematical concept of a function, and it helps to keep that in mind. What is the difference?
  • A function a b in functional programming is an algorithm that, given a value of type a, calculates (returns, evaluates to) a value of type b.
  • A function a b in math (or Isabelle) associates with each value of type a a value of type b.
For example, the following is a perfectly valid function definition in math (and HOL), but could not be a function in the programming sense: 
definition foo :: "(nat   real)   real" where
  "foo seq = (if convergent seq then lim seq else 0)"
This assigns a real number to every sequence, but it does not compute it in any useful sense. From this it follows that

Isabelle functions are specified, not defined Consider this function definition: 
fun plus :: "nat   nat   nat"  where
   "plus 0       m = m"
   "plus (Suc n) m = Suc (plus n m)"
To a functional programmer, this reads
plus is a function that analyses its first argument. If that is 0, then it returns the second argument. Otherwise, it calls itself with the predecessor of the first argument and increases the result by one.
which is clearly a description of a computation. But to Isabelle, the above reads
plus is a binary function on natural numbers, and it satisfies the following two equations:
And in fact, it is not so much Isabelle that reads it this way, but rather the fun command, which is external to the Isabelle logic. The fun command analyses the given equations, constructs a non-recursive definition of plus under the hood, passes that to Isabelle and then proves that the given equations hold for plus. One interesting consequence of this is that different specifications can lead to the same functions. In fact, if we would define plus' by recursing on the second argument, we d obtain the the same function (i.e. plus = plus' is a theorem, and there would be no way of telling the two apart).

Termination is a property of specifications, not functions Because a function does not evaluate, it does not make sense to ask if it terminates. The question of termination arises before the function is defined: The fun command can only construct plus in a way that the equations hold if it passes a termination check very much like Fixpoint in Coq. But while the termination check of Fixpoint in Coq is a deep part of the basic logic, in Isabelle it is simply something that this particular command requires for its internal machinery to go through. At no point does a termination proof of the function exist as a theorem inside the logic. And other commands may have other means of defining a function that do not even require such a termination argument! For example, a function specification that is tail-recursive can be turned in to a function, even without a termination proof: The following definition describes a higher-order function that iterates its first argument f on the second argument x until it finds a fixpoint. It is completely polymorphic (the single quote in 'a indicates that this is a type variable): 
partial_function (tailrec)
  fixpoint :: "('a   'a)   'a   'a"
where
  "fixpoint f x = (if f x = x then x else fixpoint f (f x))"
We can work with this definition just fine. For example, if we instantiate f with ( x. x-1), we can prove that it will always return 0: 
lemma "fixpoint (  n . n - 1) (n::nat) = 0"
  by (induction n) (auto simp add: fixpoint.simps)
Similarly, if we have a function that works within the option monad (i.e. Maybe in Haskell), its specification can always be turned into a function without an explicit termination proof here one that calculates the Collatz sequence: 
partial_function (option) collatz :: "nat   nat list option"
 where "collatz n =
        (if n = 1 then Some [n]
         else if even n
           then do   ns <- collatz (n div 2);    Some (n # ns)  
           else do   ns <- collatz (3 * n + 1);  Some (n # ns) )"
Note that lists in Isabelle are finite (like in Coq, unlike in Haskell), so this function returns a list only if the collatz sequence eventually reaches 1. I expect these definitions to make a Coq user very uneasy. How can fixpoint be a total function? What is fixpoint ( n. n+1)? What if we run collatz n for a n where the Collatz sequence does not reach 1?3 We will come back to that question after a little detour

HOL is a logic of non-empty types Another big difference between Isabelle and Coq is that in Isabelle, every type is inhabited. Just like the totality of functions, this is a very fundamental fact about what HOL defines to be a type. Isabelle gets away with that design because in Isabelle, we do not use types for propositions (like we do in Coq), so we do not need empty types to denote false propositions. This design has an important consequence: It allows the existence of a polymorphic expression that inhabits any type, namely 
undefined :: 'a
The naming of this term alone has caused a great deal of confusion for Isabelle beginners, or in communication with users of different systems, so I implore you to not read too much into the name. In fact, you will have a better time if you think of it as arbitrary or, even better, unknown. Since undefined can be instantiated at any type, we can instantiate it for example at bool, and we can observe an important fact: undefined is not an extra value besides the usual ones . It is simply some value of that type, which is demonstrated in the following lemma: 
lemma "undefined = True   undefined = False" by auto
In fact, if the type has only one value (such as the unit type), then we know the value of undefined for sure: 
lemma "undefined = ()" by auto
It is very handy to be able to produce an expression of any type, as we will see as follows

Partial functions are just underspecified functions For example, it allows us to translate incomplete function specifications. Consider this definition, Isabelle s equivalent of Haskell s partial fromJust function: 
fun fromSome :: "'a option   'a" where
  "fromSome (Some x) = x"
This definition is accepted by fun (albeit with a warning), and the generated function fromSome behaves exactly as specified: when applied to Some x, it is x. The term fromSome None is also a value of type 'a, we just do not know which one it is, as the specification does not address that. So fromSome None behaves just like undefined above, i.e. we can prove 
lemma "fromSome None = False   fromSome None = True" by auto
Here is a small exercise for you: Can you come up with an explanation for the following lemma: 
fun constOrId :: "bool   bool" where
  "constOrId True = True"
lemma "constOrId = ( _.True)   constOrId = ( x. x)"
  by (metis (full_types) constOrId.simps)
Overall, this behavior makes sense if we remember that function definitions in Isabelle are not really definitions, but rather specifications. And a partial function definition is simply a underspecification. The resulting function is simply any function hat fulfills the specification, and the two lemmas above underline that observation.

Nonterminating functions are also just underspecified Let us return to the puzzle posed by fixpoint above. Clearly, the function seen as a functional program is not total: When passed the argument ( n. n + 1) or ( b. b) it will loop forever trying to find a fixed point. But Isabelle functions are not functional programs, and the definitions are just specifications. What does the specification say about the case when f has no fixed-point? It states that the equation fixpoint f x = fixpoint f (f x) holds. And this equation has a solution, for example fixpoint f _ = undefined. Or more concretely: The specification of the fixpoint function states that fixpoint ( b. b) True = fixpoint ( b. b) False has to hold, but it does not specify which particular value (True or False) it should denote any is fine.

Not all function specifications are ok At this point you might wonder: Can I just specify any equations for a function f and get a function out of that? But rest assured: That is not the case. For example, no Isabelle command allows you define a function bogus :: () nat with the equation bogus () = Suc (bogus ()), because this equation does not have a solution. We can actually prove that such a function cannot exist: 
lemma no_bogus: "  bogus. bogus () = Suc (bogus ())" by simp
(Of course, not_bogus () = not_bogus () is just fine )

You cannot reason about partiality in Isabelle We have seen that there are many ways to define functions that one might consider partial . Given a function, can we prove that it is not partial in that sense? Unfortunately, but unavoidably, no: Since undefined is not a separate, recognizable value, but rather simply an unknown one, there is no way of stating that A function result is not specified . Here is an example that demonstrates this: Two partial functions (one with not all cases specified, the other one with a self-referential specification) are indistinguishable from the total variant: 
fun partial1 :: "bool   unit" where
  "partial1 True = ()"
partial_function (tailrec) partial2 :: "bool   unit" where
  "partial2 b = partial2 b"
fun total :: "bool   unit" where
  "total True = ()"
  "total False = ()"
lemma "partial1 = total   partial2 = total" by auto
If you really do want to reason about partiality of functional programs in Isabelle, you should consider implementing them not as plain HOL functions, but rather use HOLCF, where you can give equational specifications of functional programs and obtain continuous functions between domains. In that setting, () and partial2 = total. We have done that to verify some of HLint s equations.

You can still compute with Isabelle functions I hope by this point, I have not scared away anyone who wants to use Isabelle for functional programming, and in fact, you can use it for that. If the equations that you pass to fun are a reasonable definition for a function (in the programming sense), then these equations, used as rewriting rules, will allow you to compute that function quite like you would in Coq or Haskell. Moreover, Isabelle supports code extraction: You can take the equations of your Isabelle functions and have them expored into Ocaml, Haskell, Scala or Standard ML. See Concon for a conference management system with confidentially verified in Isabelle. While these usually are the equations you defined the function with, they don't have to: You can declare other proved equations to be used for code extraction, e.g. to refine your elegant definitions to performant ones. Like with code extraction from Coq to, say, Haskell, the adequacy of the translations rests on a moral reasoning foundation. Unlike extraction from Coq, where you have an (unformalized) guarantee that the resulting Haskell code is terminating, you do not get that guarantee from Isabelle. Conversely, this allows you do reason about and extract non-terminating programs, like fixpoint, which is not possible in Coq. There is currently ongoing work about verified code generation, where the code equations are reflected into a deep embedding of HOL in Isabelle that would allow explicit termination proofs.

Conclusion We have seen how in Isabelle, every function is total. Function declarations have equations, but these do not define the function in an computational sense, but rather specify them. Because in HOL, there are no empty types, many specifications that appear partial (incomplete patterns, non-terminating recursion) have solutions in the space of total functions. Partiality in the specification is no longer visible in the final product.

PS: Axiom undefined in Coq This section is speculative, and an invitation for discussion. Coq already distinguishes between types used in programs (Set) and types used in proofs Prop. Could Coq ensure that every t : Set is non-empty? I imagine this would require additional checks in the Inductive command, similar to the checks that the Isabelle command datatype has to perform4, and it would disallow Empty_set. If so, then it would be sound to add the following axiom 
Axiom undefined : forall (a : Set), a.
wouldn't it? This axiom does not have any computational meaning, but that seems to be ok for optional Coq axioms, like classical reasoning or function extensionality. With this in place, how much of what I describe above about function definitions in Isabelle could now be done soundly in Coq. Certainly pattern matches would not have to be complete and could sport an implicit case _ undefined. Would it help with non-obviously terminating functions? Would it allow a Coq command Tailrecursive that accepts any tailrecursive function without a termination check?
  1. Isabelle is a metalogical framework, and other logics, e.g. Isabelle/ZF, behave differently. For the purpose of this blog post, I always mean Isabelle/HOL.
  2. Isabelle is a metalogical framework, and other logics, e.g. Isabelle/ZF, behave differently. For the purpose of this blog post, I always mean Isabelle/HOL.
  3. Let me know if you find such an n. Besides n = 0.
  4. Like fun, the constructions by datatype are not part of the logic, but create a type definition from more primitive notions that is isomorphic to the specified data type.

Next.

Previous.