Changes in littler version 0.3.2 (2017-02-14)

• Changes in examples
  • New scripts getRStudioServer.r and getRStudioDesktop.r to download daily packages, currently defaults to Ubuntu amd64
  • New script c4c.r calling rhub::check_for_cran()
  • New script rd2md.r to convert Rd to markdown.
  • New script build.r to create a source tarball.
  • The installGitHub.r script now use package remotes (PR #44, #46)

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

[...]
slapd[7408]: warning: cannot open /etc/hosts.allow: Too many open files
slapd[7408]: warning: cannot open /etc/hosts.deny: Too many open files
slapd[7408]: warning: cannot open /etc/hosts.allow: Too many open files
slapd[7408]: warning: cannot open /etc/hosts.deny: Too many open files
slapd[7408]: warning: cannot open /etc/hosts.allow: Too many open files
slapd[7408]: warning: cannot open /etc/hosts.deny: Too many open files
[...]

If people are hitting open file limits trying to open two extra files,
disabling features in the codebase is not the correct solution.

• system-wide limits (you tune these via sysctl, they live in the kernel)
• user/group/process limits (via limits.conf, ulimit and prlimit)

arturo@debian:~% sudo su openldap -s /bin/bash
openldap@debian:~% ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7915
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[...]
#<domain>      <type>  <item>         <value>
openldap	soft	nofile		1000000
openldap	hard	nofile		1000000
[...]

[...]
session		required	pam_limits.so
[...]

arturo@debian:~% cat /proc/$(pgrep slapd)/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             16000                16000                processes
Max open files            1024                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       16000                16000                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

#define _GNU_SOURCE
#define _FILE_OFFSET_BITS 64
#include <stdio.h>
#include <time.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/resource.h>
#define errExit(msg) do   perror(msg); exit(EXIT_FAILURE); \
                          while (0)
int
main(int argc, char *argv[])
 
    struct rlimit old, new;
    struct rlimit *newp;
    pid_t pid;
    if (!(argc == 2   argc == 4))  
        fprintf(stderr, "Usage: %s <pid> [<new-soft-limit> "
                "<new-hard-limit>]\n", argv[0]);
        exit(EXIT_FAILURE);
     
    pid = atoi(argv[1]);        /* PID of target process */
    newp = NULL;
    if (argc == 4)  
        new.rlim_cur = atoi(argv[2]);
        new.rlim_max = atoi(argv[3]);
        newp = &new;
     
    /* Set CPU time limit of target process; retrieve and display
       previous limit */
    if (prlimit(pid, RLIMIT_NOFILE, newp, &old) == -1)
        errExit("prlimit-1");
    printf("Previous limits: soft=%lld; hard=%lld\n",
            (long long) old.rlim_cur, (long long) old.rlim_max);
    /* Retrieve and display new CPU time limit */
    if (prlimit(pid, RLIMIT_NOFILE, NULL, &old) == -1)
        errExit("prlimit-2");
    printf("New limits: soft=%lld; hard=%lld\n",
            (long long) old.rlim_cur, (long long) old.rlim_max);
    exit(EXIT_FAILURE);
 

arturo@debian:~% gcc limits.c -o limits

arturo@debian:~% sudo limits $(pgrep slapd) 1000000 1000000

arturo@debian:~% cat /proc/$(pgrep slapd)/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             16000                16000                processes
Max open files            1000000              1000000              files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       16000                16000                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Developer time

Arguably the greatest resource of a project is the time its developers can devote to it. Unless it is a very large, well supported project like the Kernel or libreoffice almost all developer time is voluntary.

Developer focus

Any given developer is likely to work on an area of code that interests them in preference to one that does not. This means if a developer must do work which does not interest them they may loose focus and not work on the project at all.

Developer skillset

A given developer may not have the skillset necessary to work on a feature, this is especially acute when considering minority platforms which often have very, very few skilled developers available.

Developer access

It should be obvious that software that only requires commodity hardware and software to develop is much cheaper than that which requires special hardware and software. To use our earlier example the cocoa frontend required an apple computer running MAC OS X to compile and test, this resource was very limited and the project only had access to two such systems via remote desktop. These systems also had to serve as CI builders and required physical system administration as they could not be virtualized.

Support

Once a project releases useful software it generally gains users outside of the developers. Supporting users consumes developer time and generally causes them to focus on things other than code that interests them.

While most developers have enough pride in what they produce to fix bugs, users must always remember that the main freedom they get from OSS is they recived the code and can change it themselves, there is no requirement for a developer to do anything for them.

Resources

A project requires a website, code repository, wiki, CI systems etc. which must all be paid for. Netsurf for example is fortunate to have Pepperfish look after our website hosting at favorable rates, Mythic beasts provide exceptionally good rates for the CI system virtual machine along with hardware donations (our apple macs were donated by them) and Collabora for providing physical hosting for our virtual machine server.

Despite these incredibly good deals the project still spends around 200gbp (250usd) a year on overheads, these services obviously benefit the whole project including minority platforms but are generally donated by users of the more popular platforms.

Developer learning

A developer may implement a feature to allow them to learn a new technology or skill

Project diversity

A feature may mean the project gets built in a new environment which reveals issues or opportunities in unconnected code. For example the Debian OS is built on a variety of hardware platforms and sometimes reveals issues in software by compiling it on big endian systems. These issues are often underlying bugs that are causing errors which are simply not observed on a little endian platform.

More users

Gaining users of the software is often a benefit and although most OSS developers are contributing for personal reasons having their work appreciated by others is often a factor. This might be seen as the other side of the support cost.

• Antoine Beaupr did 12.75 hours.
• Balint Reczey did 14 hours (out of 12.75 hours allocated + 2.5 hours remaining, thus keeping 1.25 hours for February).
• Ben Hutchings did 3 hours (out of 12.75 hours allocated + 5.5 hours remaining, thus keeping 15.25 extra hours for February).
• Chris Lamb did 12.75 hours.
• Emilio Pozuelo Monfort did 15.25 hours (out of 12.75 hours allocated + 2.5 hours remaining).
• Guido G nther did 8 hours.
• Hugo Lefeuvre did 15.25 hours (out of 12.75 hours allocated + 2.5 hours remaining).
• Jonas Meurer did 9 hours (out of 12 hours allocated + 6.75 hours remaining, thus keeping 9.75 extra hours for February).
• Markus Koschany did 12.75 hours.
• Ola Lundqvist did 12.75 hours.
• Rapha l Hertzog did 10 hours.
• Roberto C. Sanchez did 12.5 hours (out of 12.75 hours allocated, thus keeping 0.25 hours for February).
• Thorsten Alteholz did 12.75 hours.

• Platinum sponsors:
• TOSHIBA (for 16 months)
• GitHub (for 7 months)
• Gold sponsors:
• The Positive Internet (for 32 months)
• Blablacar (for 31 months)
• Linode LLC (for 21 months)
• Babiel GmbH (for 10 months)
• Plat Home (for 10 months)
• Silver sponsors:
• Domeneshop AS (for 31 months)
• Universit Lille 3 (for 31 months)
• Trollweb Solutions (for 29 months)
• Nantes M tropole (for 25 months)
• University of Luxembourg (for 23 months)
• Dalenys (for 22 months)
• Univention GmbH (for 17 months)
• Universit Jean Monnet de St Etienne (for 17 months)
• Sonus Networks (for 11 months)
• UR Communications BV (for 5 months)
• maxcluster GmbH (for 5 months)
• Exonet B.V.
• Bronze sponsors:
• David Ayers IntarS Austria (for 32 months)
• Evolix (for 32 months)
• Offensive Security (for 32 months)
• Seznam.cz, a.s. (for 32 months)
• Freeside Internet Service (for 31 months)
• MyTux (for 31 months)
• Linuxhotel GmbH (for 29 months)
• Intevation GmbH (for 28 months)
• Daevel SARL (for 27 months)
• Bitfolk LTD (for 26 months)
• Megaspace Internet Services GmbH (for 26 months)
• Greenbone Networks GmbH (for 25 months)
• NUMLOG (for 25 months)
• WinGo AG (for 24 months)
• Ecole Centrale de Nantes LHEEA (for 21 months)
• Sig-I/O (for 18 months)
• Entr ouvert (for 16 months)
• Adfinis SyGroup AG (for 13 months)
• Laboratoire LEGI UMR 5519 / CNRS (for 8 months)
• Quarantainenet BV (for 8 months)
• GNI MEDIA (for 7 months)
• RHX Srl (for 5 months)

No comment Liked this article? Click here. My blog is Flattr-enabled.

• another hobby? check!
• new gear to learn? check!
• something more palpable to do with my photos? good enough reason, check!

• first question: glossy or matte ink?
• if glossy, which type of paper? actually glossy (uh, no), semi-gloss, pearl, satin?
• if matte, are we talking about textured or smooth matte?
• what weight? fine art paper that I tested can go from a very interesting 100gsm (almost like standard paper) Rice Paper, to 210, 286, 310 (quite standard), 325, 350 and finally towards 390-410 heavy canvas;
• on the more professional side, do you care about lifetime of paper? if you choose yes, then take care of choosing paper with no OBA optical brightening agents;
• and if you really want to go deep, what base? cellulose, alpha-cellulose or cotton?

• Nobody's Wife: The first song I heard from her, and her voice totally catched me.
• Lost: A more quite song for a break.
• Modern World: A great song about the toxic beauty norms that society likes to paint. Lovely!

/music permanent link Comments: 0 Flattr this

R> library(anytime)
R> anytime("Thu Sep 01 10:11:12 CDT 2016")
[1] "2016-09-01 10:11:12 CDT"
R> anytime("Thu Sep 01 10:11:12.123456 CDT 2016") # with frac. seconds
[1] "2016-09-01 10:11:12.123456 CDT"
R> 

Changes in anytime version 0.2.1 (2017-02-09)

• The new DatetimeVector class from Rcpp is now used, and proper versioned Depends: have been added (#43)
• The anydate and utcdate functions convert again from factor and ordered (#46 closing #44)
• A format similar to RFC 28122 but with additonal timezone text can now be parsed (#48 closing #47)
• Conversion from POSIXt to Date now also respect the timezone (#50 closing #49)
• The internal .onLoad functions was updated
• The Travis setup uses https to fetch the run script

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

package main
import (
    "log"
    "net"
    "net/http"
)
func main()  
    l, err := net.Listen("tcp", ":8081")
    if err != nil  
        log.Panicf("cannot listen: %s", err)
     
    http.Serve(l, nil)
 

[Unit]
Description=404 micro-service
[Service]
Type=notify
ExecStart=/usr/bin/404
WatchdogSec=30s
Restart=on-failure
[Install]
WantedBy=multi-user.target

Readiness The classic way for an Unix daemon to signal its readiness is to daemonize. Technically, this is done by calling fork(2) twice (which also serves other intents). This is a very common task and the BSD systems, as well as some other C libraries, supply a daemon(3) function for this purpose. Services are expected to daemonize only when they are ready (after reading configuration files and setting up a listening socket, for example). Then, a system can reliably initialize its services with a simple linear script:

syslogd
unbound
ntpd -s

Each daemon can rely on the previous one being ready to do its work. The sequence of actions is the following:

1. syslogd reads its configuration, activates /dev/log, daemonizes.
2. unbound reads its configuration, listens on 127.0.0.1:53, daemonizes.
3. ntpd reads its configuration, connects to NTP peers, waits for clock to be synchronized², daemonizes.

With systemd, we would use Type=fork in the service file. However, Go s runtime does not support that. Instead, we use Type=notify. In this case, systemd expects the daemon to signal its readiness with a message to an Unix socket. go-systemd package handles the details for us:

package main
import (
    "log"
    "net"
    "net/http"
    "github.com/coreos/go-systemd/daemon"
)
func main()  
    l, err := net.Listen("tcp", ":8081")
    if err != nil  
        log.Panicf("cannot listen: %s", err)
     
    daemon.SdNotify(false, "READY=1") //  
    http.Serve(l, nil)                //  
 

It s important to place the notification after net.Listen() (in ): if the notification was sent earlier, a client would get connection refused when trying to use the service. When a daemon listens to a socket, connections are queued by the kernel until the daemon is able to accept them (in ). If the service is not run through systemd, the added line is a no-op.

Liveness Another interesting feature of systemd is to watch the service and restart it if it happens to crash (thanks to the Restart=on-failure directive). It s also possible to use a watchdog: the service sends watchdog keep-alives at regular interval. If it fails to do so, systemd will restart it. We could insert the following code just before http.Serve() call:

go func()  
    interval, err := daemon.SdWatchdogEnabled(false)
    if err != nil   interval == 0  
        return
     
    for  
        daemon.SdNotify(false, "WATCHDOG=1")
        time.Sleep(interval / 3)
     
 ()

However, this doesn t add much value: the goroutine is unrelated to the core business of the service. If for some reason, the HTTP part gets stuck, the goroutine will happily continue to send keep-alives to systemd. In our example, we can just do a HTTP query before sending the keep-alive. The internal loop can be replaced with this code:

for  
    _, err := http.Get("http://127.0.0.1:8081") //  
    if err == nil  
        daemon.SdNotify(false, "WATCHDOG=1")
     
    time.Sleep(interval / 3)
 

In , we connect to the service to check if it s still working. If we get some kind of answer, we send a watchdog keep-alive. If the service is unavailable or if http.Get() gets stuck, systemd will trigger a restart. There is no universal recipe. However, checks can be split into two groups:

• Before sending a keep-alive, you execute an active check on the components of your service. The keep-alive is sent only if all checks are successful. The checks can be internal (like in the above example) or external (for example, check with a query to the database).
• Each component reports its status, telling if it s alive or not. Before sending a keep-alive, you check the reported status of all components (passive check). If some components are late or reported fatal errors, don t send the keep-alive.

If possible, recovery from errors (for example, with a backoff retry) and self-healing (for example, by reestablishing a network connection) is always better, but the watchdog is a good tool to handle the worst cases and avoid too complex recovery logic. For example, if a component doesn t know how to recover from an exceptional condition³, instead of using panic(), it could signal its situation before dying. Another dedicated component could try to resolve the situation by restarting the faulty component. If it fails to reach an healthy state in time, the watchdog timer will trigger and the whole service will be restarted.

---

1. Depending on the distribution, this should be installed in /lib/systemd/system or /usr/lib/systemd/system. Check with the output of the command pkg-config systemd --variable=systemdsystemunitdir.
2. This highly depends on the NTP daemon used. OpenNTPD doesn t wait unless you use the -s option. ISC NTP doesn t either unless you use the --wait-sync option.
3. An example of an exceptional condition is to reach the limit on the number of file descriptors. Self-healing from this situation is difficult and it s easy to get stuck in a loop.

Changes in RcppArmadillo version 0.7.700.0.0 (2017-02-07)

• Upgraded to Armadillo release 7.700.0 ("Rogue State")
  • added polyfit() and polyval()
  • added second form of log_det() to directly return the result as a complex number
  • added range() to statistics functions
  • expanded trimatu()/trimatl() and symmatu()/symmatl() to handle sparse matrice

Changes in RcppArmadillo version 0.7.600.2.0 (2017-01-05)

• Upgraded to Armadillo release 7.600.2 (Coup d'Etat Deluxe)
  • Bug fix to memory allocation for fields

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

What makes a good password? A "good password" may mean different things to different people. I will assert that a good password has the following properties:

• high entropy: hard to guess for machines
• transferable: easy to communicate for humans or transfer across various protocols for computers
• memorable: easy to remember for humans

High entropy means that the password should be unpredictable to an attacker, for all practical purposes. It is tempting (and not uncommon) to choose a password based on something else that you know, but unfortunately those choices are likely to be guessable, no matter how "secret" you believe it is. Yes, with enough effort, an attacker can figure out your birthday, the name of your first lover, your mother's maiden name, where you were last summer, or other secrets people think they have. The only solution here is to use a password randomly generated with enough randomness or "entropy" that brute-forcing the password will be practically infeasible. Considering that a modern off-the-shelf graphics card can guess millions of passwords per second using freely available software like hashcat, the typical requirement of "8 characters" is not considered enough anymore. With proper hardware, a powerful rig can crack such passwords offline within about a day. Even though a recent US National Institute of Standards and Technology (NIST) draft still recommends a minimum of eight characters, we now more often hear recommendations of twelve characters or fourteen characters. A password should also be easily "transferable". Some characters, like & or !, have special meaning on the web or the shell and can wreak havoc when transferred. Certain software also has policies of refusing (or requiring!) some special characters exactly for that reason. Weird characters also make it harder for humans to communicate passwords across voice channels or different cultural backgrounds. In a more extreme example, the popular Signal software even resorted to using only digits to transfer key fingerprints. They outlined that numbers are "easy to localize" (as opposed to words, which are language-specific) and "visually distinct". But the critical piece is the "memorable" part: it is trivial to generate a random string of characters, but those passwords are hard for humans to remember. As xkcd noted, "through 20 years of effort, we've successfully trained everyone to use passwords that are hard for human to remember but easy for computers to guess". It explains how a series of words is a better password than a single word with some characters replaced. Obviously, you should not need to remember all passwords. Indeed, you may store some in password managers (which we'll look at in another article) or write them down in your wallet. In those cases, what you need is not a password, but something I would rather call a "token", or, as Debian Developer Daniel Kahn Gillmor (dkg) said in a private email, a "high entropy, compact, and transferable string". Certain APIs are specifically crafted to use tokens. OAuth, for example, generates "access tokens" that are random strings that give access to services. But in our discussion, we'll use the term "token" in a broader sense. Notice how we removed the "memorable" property and added the "compact" one: we want to efficiently convert the most entropy into the shortest password possible, to work around possibly limiting password policies. For example, some bank cards only allow 5-digit security PINs and most web sites have an upper limit in the password length. The "compact" property applies less to "passwords" than tokens, because I assume that you will only use a password in select places: your password manager, SSH and OpenPGP keys, your computer login, and encryption keys. Everything else should be in a password manager. Those tools are generally under your control and should allow large enough passwords that the compact property is not particularly important.

Generating secure passwords We'll look now at how to generate a strong, transferable, and memorable password. These are most likely the passwords you will deal with most of the time, as security tokens used in other settings should actually never show up on screen: they should be copy-pasted or automatically typed in forms. The password generators described here are all operated from the command line. Password managers often have embedded password generators, but usually don't provide an easy way to generate a password for the vault itself. The previously mentioned xkcd cartoon is probably a common cultural reference in the security crowd and I often use it to explain how to choose a good passphrase. It turns out that someone actually implemented xkcd author Randall Munroe's suggestion into a program called xkcdpass:

    $ xkcdpass
    estop mixing edelweiss conduct rejoin flexitime

In verbose mode, it will show the actual entropy of the generated passphrase:

    $ xkcdpass -V
    The supplied word list is located at /usr/lib/python3/dist-packages/xkcdpass/static/default.txt.
    Your word list contains 38271 words, or 2^15.22 words.
    A 6 word password from this list will have roughly 91 (15.22 * 6) bits of entropy,
    assuming truly random word selection.
    estop mixing edelweiss conduct rejoin flexitime

Note that the above password has 91 bits of entropy, which is about what a fifteen-character password would have, if chosen at random from uppercase, lowercase, digits, and ten symbols:

    log2((26 + 26 + 10 + 10)^15) = approx. 92.548875

It's also interesting to note that this is closer to the entropy of a fifteen-letter base64 encoded password: since each character is six bits, you end up with 90 bits of entropy. xkcdpass is scriptable and easy to use. You can also customize the word list, separators, and so on with different command-line options. By default, xkcdpass uses the 2 of 12 word list from 12 dicts, which is not specifically geared toward password generation but has been curated for "common words" and words of different sizes. Another option is the diceware system. Diceware works by having a word list in which you look up words based on dice rolls. For example, rolling the five dice "1 4 2 1 4" would give the word "bilge". By rolling those dice five times, you generate a five word password that is both memorable and random. Since paper and dice do not seem to be popular anymore, someone wrote that as an actual program, aptly called diceware. It works in a similar fashion, except that passwords are not space separated by default:

    $ diceware
    AbateStripDummy16thThanBrock

Diceware can obviously change the output to look similar to xkcdpass, but can also accept actual dice rolls for those who do not trust their computer's entropy source:

    $ diceware -d ' ' -r realdice -w en_orig
    Please roll 5 dice (or a single dice 5 times).
    What number shows dice number 1? 4
    What number shows dice number 2? 2
    What number shows dice number 3? 6
    [...]
    Aspire O's Ester Court Born Pk

The diceware software ships with a few word lists, and the default list has been deliberately created for generating passwords. It is derived from the standard diceware list with additions from the SecureDrop project. Diceware ships with the EFF word list that has words chosen for better recognition, but it is not enabled by default, even though diceware recommends using it when generating passwords with dice. That is because the EFF list was added later on. The project is currently considering making the EFF list be the default. One disadvantage of diceware is that it doesn't actually show how much entropy the generated password has those interested need to compute it for themselves. The actual number depends on the word list: the default word list has 13 bits of entropy per word (since it is exactly 8192 words long), which means the default 6 word passwords have 78 bits of entropy:

    log2(8192) * 6 = 78

Both of these programs are rather new, having, for example, entered Debian only after the last stable release, so they may not be directly available for your distribution. The manual diceware method, of course, only needs a set of dice and a word list, so that is much more portable, and both the diceware and xkcdpass programs can be installed through pip. However, if this is all too complicated, you can take a look at Openwall's passwdqc, which is older and more widely available. It generates more memorable passphrases while at the same time allowing for better control over the level of entropy:

    $ pwqgen
    vest5Lyric8wake
    $ pwqgen random=78
    Theme9accord=milan8ninety9few

For some reason, passwdqc restricts the entropy of passwords between the bounds of 24 and 85 bits. That tool is also much less customizable than the other two: what you see here is pretty much what you get. The 4096-word list is also hardcoded in the C source code; it comes from a Usenet sci.crypt posting from 1997. A key feature of xkcdpass and diceware is that you can craft your own word list, which can make dictionary-based attacks harder. Indeed, with such word-based password generators, the only viable way to crack those passwords is to use dictionary attacks, because the password is so long that character-based exhaustive searches are not workable, since they would take centuries to complete. Changing from the default dictionary therefore brings some advantage against attackers. This may be yet another "security through obscurity" procedure, however: a naive approach may be to use a dictionary localized to your native language (for example, in my case, French), but that would deter only an attacker that doesn't do basic research about you, so that advantage is quickly lost to determined attackers. One should also note that the entropy of the password doesn't depend on which word list is chosen, only its length. Furthermore, a larger dictionary only expands the search space logarithmically; in other words, doubling the word-list length only adds a single bit of entropy. It is actually much better to add a word to your password than words to the word list that generates it.

Generating security tokens As mentioned before, most password managers feature a way to generate strong security tokens, with different policies (symbols or not, length, etc). In general, you should use your password manager's password-generation functionality to generate tokens for sites you visit. But how are those functionalities implemented and what can you do if your password manager (for example, Firefox's master password feature) does not actually generate passwords for you? pass, the standard UNIX password manager, delegates this task to the widely known pwgen program. It turns out that pwgen has a pretty bad track record for security issues, especially in the default "phoneme" mode, which generates non-uniformly distributed passwords. While pass uses the more "secure" -s mode, I figured it was worth removing that option to discourage the use of pwgen in the default mode. I made a trivial patch to pass so that it generates passwords correctly on its own. The gory details are in this email. It turns out that there are lots of ways to skin this particular cat. I was suggesting the following pipeline to generate the password:

    head -c $entropy /dev/random   base64   tr -d '\n='

The above command reads a certain number of bytes from the kernel (head -c $entropy /dev/random) encodes that using the base64 algorithm and strips out the trailing equal sign and newlines (for large passwords). This is what Gillmor described as a "high-entropy compact printable/transferable string". The priority, in this case, is to have a token that is as compact as possible with the given entropy, while at the same time using a character set that should cause as little trouble as possible on sites that restrict the characters you can use. Gillmor is a co-maintainer of the Assword password manager, which chose base64 because it is widely available and understood and only takes up 33% more space than the original 8-bit binary encoding. After a lengthy discussion, the pass maintainer, Jason A. Donenfeld, chose the following pipeline:

    read -r -n $length pass < <(LC_ALL=C tr -dc "$characters" < /dev/urandom)

The above is similar, except it uses tr to directly to read characters from the kernel, and selects a certain set of characters ($characters) that is defined earlier as consisting of [:alnum:] for letters and digits and [:graph:] for symbols, depending on the user's configuration. Then the read command extracts the chosen number of characters from the output and stores the result in the pass variable. A participant on the mailing list, Brian Candler, has argued that this wastes entropy as the use of tr discards bits from /dev/urandom with little gain in entropy when compared to base64. But in the end, the maintainer argued that reading "reading from /dev/urandom has no [effect] on /proc/sys/kernel/random/entropy_avail on Linux" and dismissed the objection. Another password manager, KeePass uses its own routines to generate tokens, but the procedure is the same: read from the kernel's entropy source (and user-generated sources in case of KeePass) and transform that data into a transferable string.

Conclusion While there are many aspects to password management, we have focused on different techniques for users and developers to generate secure but also usable passwords. Generating a strong yet memorable password is not a trivial problem as the security vulnerabilities of the pwgen software showed. Furthermore, left to their own devices, users will generate passwords that can be easily guessed by a skilled attacker, especially if they can profile the user. It is therefore essential we provide easy tools for users to generate strong passwords and encourage them to store secure tokens in password managers.

Note: this article first appeared in the Linux Weekly News.

• Header: this one contains basic information such as the virtual size of the image, the version number, and pointers to where the rest of the metadata is located, among other things.
• Data clusters: the data that the virtual machine sees.
• L1 and L2 tables: a two-level structure that maps the virtual disk that the guest can see to the actual location of the data clusters in the qcow2 file.
• Refcount table and blocks: a two-level structure with a reference count for each data cluster. Internal snapshots use this: a cluster with a reference count >= 2 means that it s used by other snapshots, and therefore any modifications require a copy-on-write operation.

1. Checks that run in constant time. These are equally fast for all kinds of images and I don t think they re worth disabling.
   • main-header
   • active-l1
   • refcount-table
   • snapshot-table
2. Checks that run in variable time but don t need to read anything from disk.
   • refcount-block
   • active-l2
   • inactive-l1
3. Checks that need to read data from disk. There is just one check here and it s only needed if there are internal snapshots.
   • inactive-l2

• inactive-l1: This is a variable length check because it depends on the number of internal snapshots in the qcow2 image. However its performance impact is likely to be negligible in all cases so I don t think it s worth bothering with.
• active-l2: This check depends on the virtual size of the image, and on the percentage that has already been allocated. This check might have some impact if the image is very large (several hundred GBs or more). In that case one way to deal with it is to create an image with a larger cluster size. This also has the nice side effect of reducing the amount of memory needed for the L2 cache.
• refcount-block: This check depends on the actual size of the qcow2 file and it s independent from its virtual size. This check is relatively expensive even for small images, so if you notice performance problems chances are that they are due to this one. The good news is that we have been working on optimizing it, so if it s slowing down your VMs the problem might go away completely in QEMU 2.9.

A simple terminal Let s start small with a terminal with the default settings. We ll write that in C. Another supported option is Vala.

#include <vte/vte.h>
int
main(int argc, char *argv[])
 
    GtkWidget *window, *terminal;
    /* Initialise GTK, the window and the terminal */
    gtk_init(&argc, &argv);
    terminal = vte_terminal_new();
    window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
    gtk_window_set_title(GTK_WINDOW(window), "myterm");
    /* Start a new shell */
    gchar **envp = g_get_environ();
    gchar **command = (gchar *[]) g_strdup(g_environ_getenv(envp, "SHELL")), NULL  ;
    g_strfreev(envp);
    vte_terminal_spawn_sync(VTE_TERMINAL(terminal),
        VTE_PTY_DEFAULT,
        NULL,       /* working directory  */
        command,    /* command */
        NULL,       /* environment */
        0,          /* spawn flags */
        NULL, NULL, /* child setup */
        NULL,       /* child pid */
        NULL, NULL);
    /* Connect some signals */
    g_signal_connect(window, "delete-event", gtk_main_quit, NULL);
    g_signal_connect(terminal, "child-exited", gtk_main_quit, NULL);
    /* Put widgets together and run the main loop */
    gtk_container_add(GTK_CONTAINER(window), terminal);
    gtk_widget_show_all(window);
    gtk_main();
 

You can compile it with the following command:

gcc -O2 -Wall $(pkg-config --cflags --libs vte-2.91) term.c -o term

And run it with ./term:

More features From here, you can have a look at the documentation to alter behavior or add more features. Here are three examples.

Colors You can define the 16 basic colors with the following code:

#define CLR_R(x)   (((x) & 0xff0000) >> 16)
#define CLR_G(x)   (((x) & 0x00ff00) >>  8)
#define CLR_B(x)   (((x) & 0x0000ff) >>  0)
#define CLR_16(x)  ((double)(x) / 0xff)
#define CLR_GDK(x) (const GdkRGBA)  .red = CLR_16(CLR_R(x)), \
                                    .green = CLR_16(CLR_G(x)), \
                                    .blue = CLR_16(CLR_B(x)), \
                                    .alpha = 0  
vte_terminal_set_colors(VTE_TERMINAL(terminal),
    &CLR_GDK(0xffffff),
    &(GdkRGBA)  .alpha = 0.85  ,
    (const GdkRGBA[]) 
        CLR_GDK(0x111111),
        CLR_GDK(0xd36265),
        CLR_GDK(0xaece91),
        CLR_GDK(0xe7e18c),
        CLR_GDK(0x5297cf),
        CLR_GDK(0x963c59),
        CLR_GDK(0x5E7175),
        CLR_GDK(0xbebebe),
        CLR_GDK(0x666666),
        CLR_GDK(0xef8171),
        CLR_GDK(0xcfefb3),
        CLR_GDK(0xfff796),
        CLR_GDK(0x74b8ef),
        CLR_GDK(0xb85e7b),
        CLR_GDK(0xA3BABF),
        CLR_GDK(0xffffff)
 , 16);

While you can t see it on the screenshot¹, this also enables background transparency.

Miscellaneous settings VTE comes with many settings to change the behavior of the terminal. Consider the following code:

vte_terminal_set_scrollback_lines(VTE_TERMINAL(terminal), 0);
vte_terminal_set_scroll_on_output(VTE_TERMINAL(terminal), FALSE);
vte_terminal_set_scroll_on_keystroke(VTE_TERMINAL(terminal), TRUE);
vte_terminal_set_rewrap_on_resize(VTE_TERMINAL(terminal), TRUE);
vte_terminal_set_mouse_autohide(VTE_TERMINAL(terminal), TRUE);

This will:

• disable the scrollback buffer,
• not scroll to the bottom on new output,
• scroll to the bottom on keystroke,
• rewrap content when terminal size change, and
• hide the mouse cursor when typing.

Update the window title An application can change the window title using XTerm control sequences (for example, with printf "\e]2;$ title \a"). If you want the actual window title to reflect this, you need to define this function:

static gboolean
on_title_changed(GtkWidget *terminal, gpointer user_data)
 
    GtkWindow *window = user_data;
    gtk_window_set_title(window,
        vte_terminal_get_window_title(VTE_TERMINAL(terminal))?:"Terminal");
    return TRUE;
 

Then, connect it to the appropriate signal, in main():

g_signal_connect(terminal, "window-title-changed", 
    G_CALLBACK(on_title_changed), GTK_WINDOW(window));

Final words I don t need much more as I am using tmux inside each terminal. In my own copy, I have also added the ability to complete a word using ones from the current window or other windows (also known as dynamic abbrev expansion). This requires to implement a terminal daemon to handle all terminal windows with one process, similar to urxvtcd. While writing a terminal from scratch ² suits my need, it may not be worth it. evilvte is quite customizable and can be lightweight. Consider it as a first alternative. Honestly, I don t remember why I didn t pick it. UPDATED: evilvte has not seen an update since 2014. Its GTK+3 support is buggy. It doesn t support the latest versions of the VTE library. Therefore, it s not a good idea to use it. You should also note that the primary goal of VTE is to be a library to support GNOME Terminal. Notably, if a feature is not needed for GNOME Terminal, it won t be added to VTE. If it already exists, it will likely to be deprecated and removed.

---

1. Transparency is handled by the composite manager (Compton, in my case).
2. For some definition of scratch since the hard work is handled by VTE.

Regular backups seem to also only be taken once per 24 hours, though team-member-1 has not yet been able to figure out where they are stored. According to team-member-2 these don t appear to be working, producing files only a few bytes in size.

$ backupchecker -G database-dump.tar.gz
$ cat database-dump.list
[archive]
mtime  1486480274.2923253
[files]
database.sql  =7854803 uid 1000 gid 1000 owner chaica group chaica mode 644 type f mtime 1486480253.0

[files]
database.sql  >6m uid 1000 gid 1000 mode 644 type f

$ touch /tmp/database.sql && \
tar zcvf /tmp/database-dump.tar.gz /tmp/database.sql && \
cp /tmp/database-dump.tar.gz .

$ backupchecker -C database-dump.conf
$ cat a.out 
WARNING:root:1 file smaller than expected while checking /tmp/article-backup-checker/database-dump.tar.gz: 
WARNING:root:database.sql size is 0. Should have been bigger than 6291456.

• The Backup Checker project on Github
• The official documentation of the Backup Checker project on Readthedocs.

The second Applicative law:
  pure (.) <*> Succs u us <*> Succs v vs <*> Succs w ws
= Succs (.) [] <*> Succs u us <*> Succs v vs <*> Succs w ws
= Succs (u .) (map (.) us) <*> Succs v vs <*> Succs w ws
= Succs (u . v) (map ($v) (map (.) us) ++ map (u .) vs) <*> Succs w ws
= Succs (u . v) (map (($v).(.)) us ++ map (u .) vs) <*> Succs w ws
= Succs ((u . v) w) (map ($w) (map (($v).(.)) us ++ map (u .) vs) ++ map (u.v) ws)
= Succs ((u . v) w) (map (($w).($v).(.)) us ++ map (($w).(u.)) vs ++ map (u.v) ws)
= Succs (u (v w)) (map (\u -> u (v w)) us ++ map (\v -> u (v w)) vs ++ map (\w -> u (v w)) ws)
= Succs (u (v w)) (map ($(v w)) us ++ map u (map ($w) vs ++ map v ws))
= Succs u us <*> Succs (v w) (map ($w) vs ++ map v ws)
= Succs u us <*> (Succs v vs <*> Succs w ws)

app_law_2 = (\ a b (c::Succs a) -> pure (.) <*> a <*> b <*> c)
        === (\ a b c -> a <*> (b <*> c))

[1 of 1] Compiling Successors       ( Successors.hs, Successors.o )
GHC.Proof: Proving getCurrent_proof1  
GHC.Proof: Proving getCurrent_proof2  
GHC.Proof: Proving getCurrent_proof3  
GHC.Proof: Proving ap_star  
GHC.Proof: Proving getSuccs_proof1  
GHC.Proof: Proving getSuccs_proof2  
GHC.Proof: Proving getSuccs_proof3  
GHC.Proof: Proving app_law_1  
GHC.Proof: Proving app_law_2  
GHC.Proof: Proving app_law_3  
GHC.Proof: Proving app_law_4  
GHC.Proof: Proving monad_law_1  
GHC.Proof: Proving monad_law_2  
GHC.Proof: Proving monad_law_3  
GHC.Proof: Proving return_pure  
GHC.Proof proved 15 equalities