Search Results: "spa"

20 October 2015

Ritesh Raj Sarraf: Controlling ill behaving applications with Linux Cgroups

For some time, I have been wanting to read more on Linux Cgroups to explore possibilities of using it to control Ill behaving applications. At this time, while I'm stuck in travel, it has given me some time to look into it. In our Free Software world, most of the things are do-o-cracy, i.e. when your use case is not the common one, it is typically you who has to explore possible solutions. It could be Bugs, Feature Requests or as is in my case, performance issues. But that is not to assume that we do not have better quality software in Free Software world. Infact, in my opinion, some of the tools available are far much more better than the competition in terms of features, and to add a sweetener (or nutritional facts) to it is the fact that Free Software liberates the user. One of my favorite tool, for photo management, is Digikam. Digikam is a big project, very featureful, and has some functionalities that may not be available in the competition. But as is with most Free Software projects, Digikam is a tool which underneath consumes many more subprojects from the Free Software ecosystem. For anyone who has used Digikam, may know some of the bugs that surface on it. Not necessarily a bug in Digikam, but maybe in one of the underneath libraries/tools that it consumes (Exiv, libkface, marble, OpenCV, libPGF etc). But the bottom line is that the overall Digikam experience (and if I may say: the overall GNU/Linux experience) takes a hit. Digikam has pretty powerful features for annotation, tagging, facial recognition. These features, together with Digikam, make it a compelling product. But the problem is that many of these projects are independent. Thus tight integration is a challenge. And at times, bugs can be hard to find, root cause and fix. Let's take a real example here. If you were to use Digikam today (version 4.13.0) with annotation, tagging and facial recognition as some of the core features for your use case, you may run into frustrating overall experience. Not just that, the bugs would also effect your overall GNU/Linux experience. The facial recognition feature, if triggered, will eat up all your memory. Thus leading you to uncover Linux's long old memory bug. The tagging feature, if triggered, again will lead to frequent I/O. Thus again leading to a stalled Linux system because of blocked CPU cycled, for nothing. So one of the items on my TODO list was to explore Linux Cgroups, and see if it was cleanly possible to tame a process to a confinement, so that even if it was ill behaving (for whatever reasons), your machine does not take the beating. And now that the cgroups consumer dust has kinda settled down, systemd was my first obvious choice to look at. systemd provides a helper utility, systemd-run, for similar tasks. With systemd-run, you could apply all the resource controller logic to the given process, typically cpu, memory and blkio. And restrict it to a certain set. You can also define what user to run the service as. 
rrs@learner:/var/tmp/Debian-Build/Result$ systemd-run -p BlockIOWeight=10 find /
Running as unit run-23805.service.
2015-10-20 / 21:37:44         
rrs@learner:/var/tmp/Debian-Build/Result$ systemctl status -l run-23805.service
  run-23805.service - /usr/bin/find /
   Loaded: loaded
  Drop-In: /run/systemd/system/run-23805.service.d
            50-BlockIOWeight.conf, 50-Description.conf, 50-ExecStart.conf
   Active: active (running) since Tue 2015-10-20 21:37:44 CEST; 6s ago
 Main PID: 23814 (find)
   Memory: 12.2M
      CPU: 502ms
   CGroup: /system.slice/run-23805.service
            23814 /usr/bin/find /
Oct 20 21:37:45 learner find[23814]: /proc/3/net/raw6
Oct 20 21:37:45 learner find[23814]: /proc/3/net/snmp
Oct 20 21:37:45 learner find[23814]: /proc/3/net/stat
Oct 20 21:37:45 learner find[23814]: /proc/3/net/stat/rt_cache
Oct 20 21:37:45 learner find[23814]: /proc/3/net/stat/arp_cache
Oct 20 21:37:45 learner find[23814]: /proc/3/net/stat/ndisc_cache
Oct 20 21:37:45 learner find[23814]: /proc/3/net/stat/ip_conntrack
Oct 20 21:37:45 learner find[23814]: /proc/3/net/stat/nf_conntrack
Oct 20 21:37:45 learner find[23814]: /proc/3/net/tcp6
Oct 20 21:37:45 learner find[23814]: /proc/3/net/udp6
2015-10-20 / 21:37:51
But, out of the box, graphical applications do not work. I haven't looked, but it should be doable by giving it the correct environment details.
Underneath, systemd is using the same Linux Control Groups to limit resources for individual applications. So, in cases where you have a requirement and do not have systemd, or you directly want to make use of cgroups, it could be easily done with basic cgroups tools like cgroup-tools.
With cgroup-tools, I now have a simple cgroups hierarchy set for my current use case, i.e. Digikam
rrs@learner:/var/tmp/Debian-Build/Result$ ls /sys/fs/cgroup/memory/rrs_customCG/
cgroup.clone_children           memory.kmem.tcp.limit_in_bytes      memory.numa_stat
cgroup.event_control            memory.kmem.tcp.max_usage_in_bytes  memory.oom_control
cgroup.procs                    memory.kmem.tcp.usage_in_bytes      memory.pressure_level
digikam/                        memory.kmem.usage_in_bytes          memory.soft_limit_in_bytes
memory.failcnt                  memory.limit_in_bytes               memory.stat
memory.force_empty              memory.max_usage_in_bytes           memory.swappiness
memory.kmem.failcnt             memory.memsw.failcnt                memory.usage_in_bytes
memory.kmem.limit_in_bytes      memory.memsw.limit_in_bytes         memory.use_hierarchy
memory.kmem.max_usage_in_bytes  memory.memsw.max_usage_in_bytes     notify_on_release
memory.kmem.slabinfo            memory.memsw.usage_in_bytes         tasks
memory.kmem.tcp.failcnt         memory.move_charge_at_immigrate
2015-10-20 / 21:45:38         
rrs@learner:/var/tmp/Debian-Build/Result$ ls /sys/fs/cgroup/memory/rrs_customCG/digikam/
cgroup.clone_children           memory.kmem.tcp.max_usage_in_bytes  memory.oom_control
cgroup.event_control            memory.kmem.tcp.usage_in_bytes      memory.pressure_level
cgroup.procs                    memory.kmem.usage_in_bytes          memory.soft_limit_in_bytes
memory.failcnt                  memory.limit_in_bytes               memory.stat
memory.force_empty              memory.max_usage_in_bytes           memory.swappiness
memory.kmem.failcnt             memory.memsw.failcnt                memory.usage_in_bytes
memory.kmem.limit_in_bytes      memory.memsw.limit_in_bytes         memory.use_hierarchy
memory.kmem.max_usage_in_bytes  memory.memsw.max_usage_in_bytes     notify_on_release
memory.kmem.slabinfo            memory.memsw.usage_in_bytes         tasks
memory.kmem.tcp.failcnt         memory.move_charge_at_immigrate
memory.kmem.tcp.limit_in_bytes  memory.numa_stat
2015-10-20 / 21:45:53         

rrs@learner:/var/tmp/Debian-Build/Result$ cat /sys/fs/cgroup/cpu/rrs_customCG/cpu.shares 
1024
2015-10-20 / 21:48:44         
rrs@learner:/var/tmp/Debian-Build/Result$ cat /sys/fs/cgroup/cpu/rrs_customCG/digikam/cpu.shares 
512
2015-10-20 / 21:49:05         
rrs@learner:/var/tmp/Debian-Build/Result$ cat /sys/fs/cgroup/memory/rrs_customCG/memory.limit_in_bytes 
9223372036854771712
2015-10-20 / 22:20:14         
rrs@learner:/var/tmp/Debian-Build/Result$ cat /sys/fs/cgroup/memory/rrs_customCG/digikam/memory.limit_in_bytes 
2764369920
2015-10-20 / 22:20:27         
rrs@learner:/var/tmp/Debian-Build/Result$ cat /sys/fs/cgroup/blkio/rrs_customCG/blkio.weight
500
2015-10-20 / 21:51:43         
rrs@learner:/var/tmp/Debian-Build/Result$ cat /sys/fs/cgroup/blkio/rrs_customCG/digikam/blkio.weight
10
2015-10-20 / 21:51:50
The base group, $USER_customCG needs super admin privileges. Which once set appropriately, allows the user to further self-define sub-groups. And users can then also define separate limits per sub-group.
With the resource limitations set in place, my overall experience on very recent hardware (Intel Haswell Core i7, 8 GiB RAM, 500 GB SSHD, 128 GB SSD) has improved considerably. It still is not perfect, but it definitely is a huge improvement over what I had to go through ealire: A stalled machine for hours.
top - 21:54:38 up 1 day,  6:46,  1 user,  load average: 7.22, 7.51, 7.37
Tasks: 299 total,   1 running, 298 sleeping,   0 stopped,   0 zombie
%Cpu0  :  7.1 us,  3.0 sy,  1.0 ni, 11.1 id, 77.8 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  6.0 us,  4.0 sy,  2.0 ni, 49.0 id, 39.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  5.0 us,  2.0 sy,  0.0 ni, 24.8 id, 68.3 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  5.9 us,  5.0 sy,  0.0 ni, 21.8 id, 67.3 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 7908.926 total,   96.449 free, 4634.922 used, 3177.555 buff/cache
MiB Swap: 8579.996 total, 3454.746 free, 5125.250 used. 2753.324 avail Mem 
PID to signal/kill [default pid = 8879] 
  PID  PPID nTH USER        PR  NI S %CPU %MEM     TIME+ COMMAND                           UID 
 8879  8868  18 rrs         20   0 S  8.2 31.2  37:44.64 digikam                          1000 
10255  9960   4 rrs         39  19 S  1.0  0.8  19:47.73 tracker-miner-f                  1000 
10157  9960   7 rrs         20   0 S  0.5  3.0  32:29.76 gnome-shell                      1000 
    7     2   1 root        20   0 S  0.2        0:53.48 rcu_sched                           0 
  401     1   1 root        20   0 S  0.2  1.3   0:54.93 systemd-journal                     0 
10269  9937   4 rrs         20   0 S  0.2  0.4   2:34.50 gnome-terminal-                  1000 
15316     1  14 rrs         20   0 S  0.2  3.7  30:05.96 evolution                        1000 
23777     2   1 root        20   0 S  0.2        0:05.73 kworker/u16:0                       0 
23814     1   1 root        20   0 D  0.2  0.0   0:02.00 find                                0 
24049     2   1 root        20   0 S  0.2        0:01.29 kworker/u16:3                       0 
24052     2   1 root        20   0 S  0.2        0:02.94 kworker/u16:4                       0 
    1     0   1 root        20   0 S       0.1   0:18.24 systemd                             0
The reporting tools may not be correct here. Because from what is being reported above, I should be having a machine stalled, and heavily paging, while the kernel scanning its list of processes to find the best process to kill. From this approach of jailing processes, the major side effect I can see is that the process (Digikam) is now starved of resources and will take much much much more time than what it would have been usually. But in the usual cases, it takes up all, and ends up starving (and getting killed) for consuming all available resources. So I guess it is better to be on a balanced resource diet. :-)

Categories:

Keywords:

Like:

17 October 2015

Joey Hess: it's a bird, it's a plane, it's a super monoid for propellor

I've been doing a little bit of dynamically typed programming in Haskell, to improve Propellor's Info type. The result is kind of interesting in a scary way. Info started out as a big record type, containing all the different sorts of metadata that Propellor needed to keep track of. Host IP addresses, DNS entries, ssh public keys, docker image configuration parameters... This got quite out of hand. Info needed to have its hands in everything, even types that should have been private to their module. To fix that, recent versions of Propellor let a single Info contain many different types of values. Look at it one way and it contains DNS entries; look at it another way and it contains ssh public keys, etc. As an migr from lands where you can never know what type of value is in a $foo until you look, this was a scary prospect at first, but I found it's possible to have the benefits of dynamic types and the safety of static types too. The key to doing it is Data.Dynamic. Thanks to Joachim Breitner for suggesting I could use it here. What I arrived at is this type (slightly simplified): 
newtype Info = Info [Dynamic]
    deriving (Monoid)
So Info is a monoid, and it holds of a bunch of dynamic values, which could each be of any type at all. Eep! So far, this is utterly scary to me. To tame it, the Info constructor is not exported, and so the only way to create an Info is to start with mempty and use this function: 
addInfo :: (IsInfo v, Monoid v) => Info -> v -> Info
addInfo (Info l) v = Info (toDyn v : l)
The important part of that is that only allows adding values that are in the IsInfo type class. That prevents the foot shooting associated with dynamic types, by only allowing use of types that make sense as Info. Otherwise arbitrary Strings etc could be passed to addInfo by accident, and all get concated together, and that would be a total dynamic programming mess. Anything you can add into an Info, you can get back out: 
getInfo :: (IsInfo v, Monoid v) => Info -> v
getInfo (Info l) = mconcat (mapMaybe fromDynamic (reverse l))
Only monoids can be stored in Info, so if you ask for a type that an Info doesn't contain, you'll get back mempty. Crucially, IsInfo is an open type class. Any module in Propellor can make a new data type and make it an instance of IsInfo, and then that new data type can be stored in the Info of a Property, and any Host that uses the Property will have that added to its Info, available for later introspection.
For example, this weekend I'm extending Propellor to have controllers: Hosts that are responsible for running Propellor on some other hosts. Useful if you want to run propellor once and have it update the configuration of an entire network of hosts. There can be whole chains of controllers controlling other controllers etc. The problem is, what if host foo has the property controllerFor bar and host bar has the property controllerFor foo? I want to avoid a loop of foo running Propellor on bar, running Propellor on foo, ... To detect such loops, each Host's Info should contain a list of the Hosts it's controlling. Which is not hard to accomplish: 
newtype Controlling = Controlled [Host]
    deriving (Typeable, Monoid)
isControlledBy :: Host -> Controlling -> Bool
h  isControlledBy  (Controlled hs) = any (== hostName h) (map hostName hs)
instance IsInfo Controlling where
    propigateInfo _ = True
mkControllingInfo :: Host -> Info
mkControllingInfo controlled = addInfo mempty (Controlled [controlled])
getControlledBy :: Host -> Controlling
getControlledBy = getInfo . hostInfo
isControllerLoop :: Host -> Host -> Bool
isControllerLoop controller controlled = go S.empty controlled
  where
    go checked h
          controller  isControlledBy  c = True
        -- avoid checking loops that have been checked before
          hostName h  S.member  checked = False
          otherwise = any (go (S.insert (hostName h) checked)) l
      where
        c@(Controlled l) = getControlledBy h
This is all internal to the module that needs it; the rest of propellor doesn't need to know that the Info is using used for this. And yet, the necessary information about Hosts is gathered as propellor runs.
So, that's a useful technique. I do wonder if I could somehow make addInfo combine together values in the list that have the same type; as it is the list can get long. And, to show Info, the best I could do was this: 
 instance Show Info where
            show (Info l) = "Info " ++ show (map dynTypeRep l)
The resulting long list of the types of vales stored in a host's info is not a useful as it could be. Of course, getInfo can be used to get any particular type of value: 
*Main> hostInfo kite
Info [InfoVal System,PrivInfo,PrivInfo,Controlling,DnsInfo,DnsInfo,DnsInfo,AliasesInfo, ...
*Main> getInfo (hostInfo kite) :: AliasesInfo
AliasesInfo (fromList ["downloads.kitenet.net","git.joeyh.name","imap.kitenet.net","nntp.olduse.net" ...
And finally, I keep trying to think of a better name than "Info".

Russell Coker: Mail Server Training

Today I ran a hands-on training session on configuring a MTA with Postfix and Dovecot for LUV. I gave each student a virtual machine running Debian/Jessie with full Internet access and instructions on how to configure it as a basic mail server. Here is a slightly modified set of instructions that anyone can do on their own system. Today I learned that documentation that includes passwords on a command-line should have quotes around the password, one student used a semi-colon character in his password which caused some confusion (it s the command separator character in BASH). I also discovered that trying to just tell users which virtual server to login to is prone to errors, in future I ll print out a list of user-names and passwords for virtual servers and tear off one for each student so there s no possibility of 2 users logging in to the same system. I gave each student a sub-domain of unixapropos.com (a zone that I use for various random sysadmin type things). I have changed the instructions to use example.com which is the official address for testing things (or you could use any zone that you use). The test VMs that I setup had a user named auser , the documentation assumes this account name. You could change auser to something else if you wish. Below are all the instructions for anyone who wants to try it at home or setup virtual machines and run their own training session. Basic MTA Configuration
  1. Run apt-get install postfix to install Postfix, select Internet Site for the type of mail configuration and enter the domain name you selected for the mail name.
  2. The main Postfix configuration file is /etc/postfix/main.cf. Change the myhostname setting to the fully qualified name of the system, something like mta.example.com.
    You can edit /etc/postfix/main.cf with vi (or any other editor) or use the postconf command to change it, eg postconf -e myhostname=mta.example.com .
  3. Add home_mailbox=Maildir/ to the Postfix configuration to make it deliver to a Maildir spool in the user s home directory.
  4. Restart Postfix to apply the changes.
  5. Run apt-get install swaks libnet-ssleay-perl to install swaks (a SMTP test tool).
  6. Test delivery by running the command swaks -f auser@example.com -t auser@example.com -s localhost . Note that swaks displays the SMTP data so you can see exactly what happens and if something goes wrong you will see everything about the error.
  7. Inspect /var/log/mail.log to see the messages about the delivery. View the message which is in ~auser/Maildir/new.
  8. When other students get to this stage run the same swaks command but with the -t changed to the address in their domain, check the mail.log to see that the messages were transferred and view the mail with less to see the received lines. If you do this on your own specify a recipient address that s a regular email address of yours (EG a Gmail account).
Basic Pop/IMAP Configuration
  1. Run apt-get install dovecot-pop3d dovecot-imapd to install Dovecot POP and IMAP servers.
    Run netstat -tln to see the ports that have daemons listening on them, observe that ports 110 and 143 are in use.
  2. Edit /etc/dovecot/conf.d/10-mail.conf and change mail_location to maildir:~/Maildir . Then restart Dovecot.
  3. Run the command nc localhost 110 to connect to POP, then run the following commands to get capabilities, login, and retrieve mail:
    user auser
    pass WHATEVERYOUMADEIT
    capa
    list
    retr 1
    quit
  4. Run the command nc localhost 143 to connect to IMAP, then run the following commands to list capabilities, login, and logout:
    a capability
    b login auser WHATEVERYOUMADEIT
    c logout
  5. For the above commands make note of the capabilities, we will refer to that later.
Now you have a basically functional mail server on the Internet! POP/IMAP Over SSL To avoid password sniffing we need to use SSL. To do it properly requires obtaining a signed key for a DNS address but we can do the technical work with the snakeoil certificate that is generated by Debian.
  1. Edit /etc/dovecot/conf.d/10-ssl.conf and change ssl = no to ssl = required . Then add the following 2 lines:
    ssl_cert = </etc/ssl/certs/ssl-cert-snakeoil.pem
    ssl_key = </etc/ssl/private/ssl-cert-snakeoil.key
    1. Run netstat -tln and note that ports 993 and 995 are not in use.
    2. Edit /etc/dovecot/conf.d/10-master.conf and uncomment the following lines:
      port = 993
      ssl = yes
      port = 995
      ssl = yes
    3. Restart Dovecot, run netstat -tln and note that ports 993 and 995 are in use.
  3. Run nc localhost 110 and nc localhost 143 as before, note that the capabilities have changed to include STLS/STARTTLS respectively.
  4. Run gnutls-cli --tofu 127.0.0.1 -p 993 to connect to the server via IMAPS and gnutls-cli --tofu 127.0.0.1 -p 995 to connect via POP3S. The --tofu option means to Trust On First Use , it stores the public key in ~/.gnutls and checks it the next time you connect. This allows you to safely use a snakeoil certificate if all apps can securely get a copy of the key.
Postfix SSL
  1. Edit /etc/postfix/main.cf and add the following 4 lines:
    smtpd_tls_received_header = yes
    smtpd_tls_loglevel = 1
    smtp_tls_loglevel = 1
    smtp_tls_security_level = may
    Then restart Postfix. This makes Postfix log TLS summary messages to syslog and in the Received header. It also permits Postfix to send with TLS.
  2. Run nc localhost 25 to connect to your SMTP port and then enter the following commands:
    ehlo test
    quit
    Note that the response to the EHLO command includes 250-STARTTLS, this is because Postfix was configured with the Snakeoil certificate by default.
  3. Run gnutls-cli --tofu 127.0.0.1 -p 25 -s and enter the following commands:
    ehlo test
    starttls
    ^D
    After the CTRL-D gnutls-cli will establish a SSL connection.
  4. Run swaks -tls -f auser@example.com -t auser@example.com -s localhost to send a message with SSL encryption. Note that swaks doesn t verify the key.
  5. Try using swaks to send messages to other servers with SSL encryption. Gmail is one example of a mail server that supports SSL which can be used, run swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com to send TLS (encapsulated SSL) mail to Gmail via swaks. Also run swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com -s localhost to send via your new mail server (which should log that it was a TLS connection from swaks and a TLS connection to Gmail).
SASL SASL is the system of SMTP authentication for mail relaying. It is needed to permit devices without fixed IP addresses to send mail through a server. The easiest way of configuring Postfix SASL is to have Dovecot provide it s authentication data to Postfix. Among other things if you change Dovecot to authenticate in another way you won t need to make any matching changes to Postfix.
  1. Run mkdir -p /var/spool/postfix/var/spool and ln -s ../.. /var/spool/postfix/var/spool/postfix , this allows parts of Postfix to work with the same configuration regardless of whether they are running in a chroot.
  2. Add the following to /etc/postfix/main.cf and restart Postfix:
    smtpd_sasl_auth_enable = yes
    smtpd_sasl_type = dovecot
    smtpd_sasl_path = /var/spool/postfix/private/auth
    broken_sasl_auth_clients = yes
    smtpd_sasl_authenticated_header = yes
  3. Edit /etc/dovecot/conf.d/10-master.conf, uncomment the following lines, and then restart Dovecot:
    unix_listener /var/spool/postfix/private/auth
    mode = 0666
  4. Edit /etc/postfix/master.cf, uncomment the line for the submission service, and restart Postfix. This makes Postfix listen on port 587 which is allowed through most firewalls.
  5. From another system (IE not the virtual machine you are working on) run swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com -s YOURSERVER and note that the message is rejected with Relay access denied .
  6. Now run swaks -tls --auth-user auser --auth-password WHATEVER -f auser@example.com -t YOURREALADDRESS -s YOURSERVER and observe that the mail is delivered (subject to anti-spam measures at the recipient).
    7. Configuring a MUA If every part of the previous 3 sections is complete then you should be able to setup your favourite MUA. Use auser as the user-name for SMTP and IMAP, mail.example.com for the SMTP/IMAP server and it should just work! Of course you need to use the same DNS server for your MUA to have this just work. But another possibility for testing is to have the MUA talk to the server by IP address not by name.
    Related posts:
    1. Mail Server Security I predict that over the course of the next 10...
    2. I need an LMTP server I am working on a system where a front-end mail...
    3. Moving a Mail Server Nowadays it seems that most serious mail servers (IE mail...

Steve Kemp: Robbing Peter to pay Paul, or location spoofing via DNS

I rarely watched TV online when I was located in the UK, but now I've moved to Finland with appalling local TV choices it has become more common. The biggest problem with trying to watch BBC's iPlayer, and similar services, is the location restrictions. Not a huge problem though: The next part is the harder one - making your traffic pass over the VPN. If you were simple you'd just say "Send everything over the VPN". But that would slow down local traffic, so instead you have to use trickery. My approach was just to run a series of routing additions, similar to this (except I did it in the openvpn configuration, via pushed-routes): 
ip -4 route add .... dev tun0
This works, but it is a pain as you have to add more and more routes. The simpler solution which I switched to after a while was just configuring mitmproxy on the remote OpenVPN end-point, and then configuring that in the browser. With that in use all your traffic goes over the VPN link, if you enable the proxy in your browser, but nothing else will. I've got a network device on-order, which will let me watch netflix, etc, from my TV, and I'm lead to believe this won't let you setup proxies, or similar, to avoid region-bypass. It occurs to me that I can configure my router to give out bogus DNS responses - if the device asks for "iplayer.bbc.com" it can return 10.10.10.10 - which is the remote host running the proxy. I imagine this will be nice and simple, and thought I was being clever: Because I can handle DNS-magic on the router I can essentially spoof my location for all the devices on the internal LAN, which is a good thing. Anyway I was reasonably pleased with the idea of using DNS to route traffic over the VPN, in combination with a transparent proxy. I was even going to blog about it, and say "Hey! This is a cool idea I've never heard of before". Instead I did a quick google(.fi) and discovered that there are companies offering this as a service. They don't mention the proxying bit, but it's clearly what they're doing - for example OverPlay's SmartDNS. So in conclusion I can keep my current setup, or I can use the income I receive from DNS hosting to pay for SmartDNS, or other DNS-based location-fakers. Regardless. DNS. VPN. Good combination. Try it if you get bored.

16 October 2015

Rapha&#235;l Hertzog: Freexian s report about Debian Long Term Support, September 2015

A Debian LTS logoLike each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In September, 71.50 work hours have been dispatched among 7 paid contributors. Their reports are available: Evolution of the situation October is back to the highest level of funding with 85.5 hours funded. The late sponsors have all caught up now. And next month will again rise to a new record with multiple sponsors having joined up. So far we already have two new silver sponsors (Universit Jean Monnet de Saint- tienne and Univention GmbH) and a new bronze sponsor (Entr ouvert). Many thanks to them! With those sponsors we crossed the 50% mark that was our first objective. \o/ But we still need more support to reach our second goal of funding the equivalent of a full time position. That said the increased level of support already allows us to do a better job in some areas that have been neglected : I asked the paid contributors to work towards providing mysql-5.5 in squeeze since version 5.1 is no longer supported by Oracle. We need beta testers to test the upgrade, see this message on the mailling list. In terms of security updates waiting to be handled, the situation is close to last month: the dla-needed.txt file lists 15 packages awaiting an update (3 less than last month), the list of open vulnerabilities in Squeeze shows about 23 affected packages in total (7 less than last month). Thanks to our sponsors The new sponsors are in bold.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Russ Allbery: Review: Ancillary Mercy

Review: Ancillary Mercy, by Ann Leckie
Series: Imperial Radch #3
Publisher: Orbit
Copyright: October 2015
ISBN: 0-316-24668-9
Format: Trade paperback
Pages: 330
This is the third book of the series that starts with Ancillary Justice. Please don't read it out of order. The three books form a rare thing in SF: a trilogy with a clear start and a clear ending. They're ideally read fairly close together so that you can remember the people involved. By the end of Ancillary Sword, Leckie had apparently resolved most of the plot in the Athoek system, although the Ghost Gate and broader imperial turmoil still lurked. I was expecting this novel to be another change of scene, and probably a broadening of the scope into the more sweeping events set off by earlier books. That's not what Ancillary Mercy is at all. Leckie caught me by surprise early, and the book kept not going in the direction I was expecting. And the result is brilliant. It's obviously difficult to talk about this in detail, since there's no way I'm going to spoil this or any of the previous books. This is some of the best SF I've ever read, and I want anyone who's not read the series yet to have the full experience. What I will say is that there was a lot of character growth left, including in some places that I thought would remain setting-mandated blind spots. Leckie tackles some very tricky themes, ones that are hard to do well because of how thoroughly the ground has already been explored, and still manages to find an original take and an even more original tone. And remains true to Breq's characterization throughout, including one critical scene that's masterful in its heart-breaking understatedness. Breq's flat affect is both very difficult and incredibly rewarding as a first-person protagonist. If I had to call out one thing that makes this series so special to me, it's the way that Leckie puts the reader in the middle of a restrained and very careful culture, centers the book on a character with a very unusual relationship to emotion, nearly completely avoids outward displays of feeling, treats emotions primarily as data to be weighed with other information, and yet provides just the right amount of subtle clues and external reflection to show just how central emotion is to everything that happens. This is an incredibly difficult balancing act, much harder than an openly emotional book (let alone the tough-guy cliches that military SF is normally full of), and it makes the occasional pay-offs so rewarding. I think some people might find Ancillary Mercy too understated, but I thought it was note-perfect throughout. And the ending was spectacular. For those who are less enamored of Breq and more interested in the plot, don't worry: we do get some resolution to most of the loose threads. There's clearly potential in this universe for more stories (and apparently more books coming!), but this trilogy comes to a definite end. If I had to quibble, I'd say that a few bits of the plot are too convenient: the resolution would have been impossible without one very critical mistake at the start of the second book, and there's at least one perfectly-timed accident. But Leckie's most critical plot resolution tool was properly hung on the wall in the last book and used perfectly. It's hard to manage a satisfying resolution to a twisty political conflict between very sharp antagonists; for me, at least, Leckie pulls it off. Looking back on the whole series, I find it striking how much more satisfying and believable Leckie's military culture is for her setting than those I normally read. I think we're too often given far-future military stories that are full of impulsive drama, high emotion, and physical feats. But in a world of starships and space stations, advanced computers, careful and systematized roles, and precise engineering, that's not realistic. While there is some physical violence required, problems in that world are more likely to look like operational computer problems than gladiatorial problems. That means the required tools are careful and fast analytical skills, quick iterative decision-making, and precise execution, not physical toughness and grit. In that world, adrenaline is at best useless and often your enemy. Leckie's world of tea sets and emotional control is partly driven by other world-building factors, but, particularly in this book, I was struck by how well-adapted it is to a military where calm decisions and efficient follow-through are far more important than adrenaline-driven courage. The emotion is absolutely there, but the catharsis doesn't come through bloody battles or feats of physicality. Again, so tricky to pull off, since it plays against the expectations we bring as readers to a war story, but I found it so much more satisfying when written well. I also love that this series is not about saving the world, at least in any direct sense. It is about justice, and ethics, and about standing up for what's right, but it's just as much about identity, about how to choose and define one's ethics, and how to make the most of the small number of decisions one gets to make. Everyone in this world is operating under sharp constraints: culture, ability, available alliances, physical limitations, and particularly the role into which they were cast. Huge things happen in this series, but not from the dramatic, order-overturning action of Chosen Ones. Rather, change is created by individuals who find small points of leverage and definition, and who make quiet, critical decisions to become better people than they were. I adore this series, and particularly this conclusion, beyond all rational measure. I'm sure that won't be universal, just as other readers weren't as enamored as I was of the direction Leckie went in Ancillary Sword. But this was a series conclusion that I was not expecting at all, that kept surprising me right up to the climax, and was exactly the conclusion that I wanted. If you liked the previous books, and particularly if you liked Ancillary Sword but thought Breq was a bit too much in control of the plot and a bit too able to make things go her way, I think you'll love Ancillary Mercy. This is simply the best SF series I've read in a long time. Rating: 10 out of 10

15 October 2015

Laura Arjona: Long summer story, Welcome team, and I am a Debian Developer now

Note: 2015/10/16: I need to add some links but I won t delay this more, posting now, will edit later. Summer ended long time ago, but believe me, I m still catching up with all the things that I began in June/July, all the things I left in August when I went holidays, and more things that appeared in August and September. This is a long overdue post, I hope you bear with me for waiting so long, and writing (now) so long too! June In June, I was 100% sure that I would not attend DebConf15 (well, I was 98% sure until then), and when the new Outreach Sponsorship grants were announced, I decided to write some mails to several Debian contributors, so they consider applying for the grant and attend DebConf (and maybe trigger some i18n/l10n meeting ). They kindly declined, and I understood their reasons, but also wondered what would have happened if the proposal would have come from somebody more official instead of a random contributor that they don t know. I also hoped that lots of other Debianites also write to newbies or not-yet-DD-contributors or non-packaging contributors to invite them to DebConf, and I hoped that they had better luck than me in convincing them :) July In July I usually work hard preparing the computer labs for next academic year at my workplace in the University, but I also have more free time in the long afternoons and evenings, since I don t sleep much, and there is not much to do outside with the summer hot. So I used that month to go on contributing to DebConf publicity and think a bit more about Debian and the other free software communities. I didn t put much time in advancing my selfhosting (no SSL yet in *.larjona.net! booooo!) but I decided to deep my toe in Sandstorm.io, and try to selfhost an instance ( http://lacaja.larjona.net ) and try Etherpad inside Sandstorm (since I failed in deploying Etherpad by myself in my jessie+nginx+postgres box). Sandstorm worked, and Etherpad was packaged in Sandstorm so it worked too; and I have my free-software-base pads now for writing and share. So I joined #sandstorm IRC channel since then, and there I learnt that Asheesh Laroia (who works in Sandstorm.io and is also a Debian Developer and was going to give a talk about Sandstorm.io in DebConf15) was offering mentorship for people wanting to learn Sandstorm packaging, and his proposal was to begin packaging Framadate. I also failed in selfhosting Dudle (prepared for Apache + FastCGI, couldn t make it work in my Nginx), so Asheesh s proposal looked suitable for me. We talked and decided to invest the rest of July and first days of August in learning to package Framadate. I learned a lot, but couldn t finish the task. I encountered many issues (setting my dev environment, and later trying to package), and we solved some of them but my time ran out. I posted my work in the list, and I hope that my feedback on the documentation and the issues I encountered helped Asheesh and the Sandstorm community. Framadate is packaged in Sandstorm.io now, Drew Fisher packaged it, not sure if my stuff was useful or not (it s been useful for me, for learning, at least). I ll talk more about Sandstorm.io in a future blog post updating on my selfhosting adventures. What I liked most was the kind of proposal of mentoring that Asheesh made. It was very detailed in every aspect: the task, the things you need to accomplish it, details about his availability for mentorship I try to be welcoming in the teams in which I participate, but the fact is that I fail in actually mentor, maybe because of not making specific proposals to people (until now, I was like Hi, newcomer! Go read this, this and this, and try for yourself any task you feel you like it, and come back if you have issues , la Debian ). This, plus the thoughts about my mails in June for diversity outreach in DebConf, made me feel the need of having a team where people willing to welcome newcomers share tricks and procedures, write together more specific proposals, and follow up the newcomers experiences in a regular way. I talked with Enrico Zini and we wrote down some notes for a Welcome Team in Debian; he said he would spread the word during DebCamp/DebConf and we would see what people thinks about it. August August came, and the day before going on holidays I was really tired: too much luggage to prepare, too many hours in front of the computer, and the usual stress of traveling; and I took the bad decision of signing some GPG keys of several Debianites that I met in July. I say bad decision because the lack of sleep showed its black magic and I accidentally deleted my secring.gpg file. I knew I had a backup but I didn t have too much time to invest and I didn t want to mess it with the backup too, and my laptop was going to stay at home, powered off, during the whole month, so I just went on holidays and left the GPG issue for later. The day after, meanwhile I was waiting in the airport for my boarding time, I received a mail accepting me as Debian Developer. Wow!! Really, I was not expecting that the process was already finished, I had interchanged several mails with my Application Manager (who happens to be the current DPL!) and I thought that his summer could be quite packed of Debian/DebConf work and my process could wait a bit. So it was a very happy news and very motivating after one month (July) full of free software work. On the other side, I was a bit scared: what type of Debian Developer are you, larjona, not capable to sign some GPG keys without breaking your setup?! but I answered myself well, I m the type of Debian Developer that has backups :) and then, with that mixed feelings of excitement and impostor syndrome, I took my plane and went on holidays, not expecting to touch any computer until the end of the month. August is probably the month in the year when I have more free time (holidays), but less time to dedicate to free software. I devote most of the month to visit family and stay with them, with no internet connection available or no free time to look at the mailbox or social networks or IRC But DebCamp and DebConf15 were happening during my holidays. And this DebConf15 was the first one in which I participated in the organization, and the first one in which I felt more than being a consumer of Debian videos . I could not follow the streamings, my only internet-capable device was my Android 2.x phone, but when I had wifi I fetched the mail, and during the nights, while everybody else was sleeping and I was laying on the terrace, below the sky full of stars, I could read batches of hundred of mails from debconf-discuss mailing list. And I could get some feeling from DebConf life, because I learned about the ad-hoc BoFs and discussions, the morning bike rides and swimming proposals, and the dancing classes, the i18m/l10n meeting, and many other things. I could answer some mail from time to time, and I also knew that a fellow Debianite from Madrid was going to bring me some stickers, maybe a t-shirt, and shake hands in my name to some persons. September and October September was about finishing reading all the mails and try to answer the pending ones, and preparing my computer to use my new Debian identity (and stop using larjona-guest). I still have some things to do, pending technical work, and some mails that I should have answered and I ve forgotten, for sure (if you sent me a mail that needs answer or would be fine that I answer (even if it was months ago!), please resend or ping me). I recovered my secring.gpg but and just now I added larjona@debian.org to the ID in my GPG key, but didn t signed the pending keys again (sorry dkg and holger! will catch up there soon). My subkeys expired and I m trying to find out how to proceed (they are in my FSFE SmartCard) :/ About the Debian teams, I ve resumed my work in publicity team (this year I ll try to be more involved, in Debian Project News in particular), partially in the website team, and recently I ve finished catching up with the Spanish translation of the website. I ve also joined the DebConf team again (for DebConf16, no matter I probably won t attend) and documented the Publicity task for DebConf, and I try to engage the mailing list and the IRC meetings. I finally could have time to watch some DebConf15 videos and Andreas Tille s talk ( Creating a more inviting environment for newcomers New experiences from MoM, SoB, Teammetrics ) helped me to step ahead in welcoming people with more useful stuff than Hi, newcomer! Go read this (general URLs), try for yourself whatever you like . I have made specific proposals for two people. In mid September I accepted an interview about Debian for a podcast with quite a lot audience (in Spanish), in which I explained the idea of the Welcome Team and offered myself as first-contact. Since then, two more people have contacted me and I have offered specific tasks I think are suitable for them. I also try to be more available in the IRC and offer some time spans for new contributors to DebConf to explain the git setup, the wiki, and all this stuff that looks more complicated than what it is. And I think that s all. My Debianite friend kindly brought me some stickers and a DebConf t-shirt, plus the organization t-shirt that the team gave me as present for my contributions in DebConf15. Neil McGovern kindly sent me a certificate of my new Debian Developer status (thanks!!), and it s posted in my wall at work. Here you are a photo! larjona_dd.JPG (Note: my wall is full of stickers and pieces of papers with things I need, things I like and things I use to explain my work (sometimes sarcastically/ironically ). Maybe some day I ll make a blog post about that!) I feel very proud and happy. Still, a lot of things to learn and work to do, but my intentions are: to keep on progressing (sometimes fast, sometimes slowly), never give up, and enjoy the multiple flowers I find in my way :) Thanks everybody! October and future Some other ideas/plans for the future (the ones I didn t say yet): Comments? If you want to comment you can use this pump.io thread.
Filed under: My experiences and opinion, News Tagged: Communities, Contributing to libre software, Debian, Developer motivations, encryption, English, Free Software, gpg, libre software

14 October 2015

Russ Allbery: Review: Firebird

Review: Firebird, by Jack McDevitt
Series: Alex Benedict #6
Publisher: Ace
Copyright: November 2011
Printing: October 2012
ISBN: 1-937007-80-4
Format: Mass market
Pages: 357
Firebird is the sixth book in the Alex Benedict series about an antique hunter in the far future (albeit a far future that looks exactly like 1960s suburbia, but with spaceships). This is a very episodic series, though, and I don't think it would be hurt much by starting in the middle. There are references to earlier investigations, but they're fleeting, and I often didn't map them to remembered plots even though I've read the whole series to this point. The investigation in this book starts with the heir to an estate coming to Alex to sell some of the possessions of a physicist. Chase (Alex's assistant, and here, as in the last few books, the viewpoint character) doesn't even recognize the name, but Alex does: Christopher Robin disappeared forty-one years earlier, under circumstances that were never fully explained. He was also (and unusually for a physicist) interested in strange and marginal ideas: dark energy, new drive technology, parallel universes one could potentially cross into, and similar fringe concepts. The normal pattern of this series is that Alex will hear about some mystery, be unable to restrain his curiosity, and start poking around, usually turning up things that people would rather he didn't. It takes a while for that to happen here; instead, the story starts with Alex playing up popular interest in Robin's ideas in a rather mercenary attempt to increase the value of the estate. He stumbles into more mystery mostly by accident. Eventually he can't resist the allure of a revealed link between Christopher Robin and sightings of mysterious disappearing starships and the normal pattern kicks in, but he spends rather more of the book than normal being flippant and slightly unethical. I didn't much like the shift in tone. Alex is a lot harder to like this book, and not just for his business practices. His tone towards Chase also moves past the slightly superior smugness that's common to many books of this type (think Nero Wolfe) and well into condescending ass. Some of this may be intentional, as McDevitt uses this book to bring out a bit more of Alex and Chase's past and has some story reasons for making Alex less of a saint. But some of it feels accidental, or unnoticed, and it left a bad taste in my mouth. As with a lot of McDevitt, the actual mystery is slow, a bit scattered, and has a fair number of blind alleys. That property makes these books feel more like real investigations, but it works better when the characters are fully engaged in the investigation and are communicating a bit better than they are here. The plot also gets entangled in a subplot about a planet full of abandoned AIs, and while that was moderately interesting, it felt like an extended digression with dubious relevance to the main plot. McDevitt occasionally has trouble with plot focus, and I think I noticed more this time because the characters weren't as fun to spend time with. The end of Firebird was up to the usual standards of this series, albeit surprisingly traumatic. The rest of the book, though, felt markedly weaker. Alex decided to be obnoxious and play into his (previously mostly inaccurate) public perception as a money and glory hound. Chase seemed too flippant, uninterested, and a bit too much of a foil instead of a partner. And the plot felt like two different story ideas awkwardly smushed together. I think this was the weakest of the series to date. Hopefully the next book, which looks to be more of a direct sequel to this one, will improve. Followed by Coming Home. Rating: 6 out of 10

Norbert Preining: On Lars Wirzenius, Mart n Ferrari, and Debian

Now that the tsunami that followed my recent blog post has passed, I can read through all the nice comments and blogs of other people. I didn t expect that just writing up some links to posts on LKML, plus adding my personal opinion on what has happened and about my interpretation concerning Debian could gather so much attention. Anyway, let us look at the blog posts written in the aftermath. debian-newspeak-coc In particular do I want to answer to two blogs: Lars Wirzenius blog and Mart n Ferrari s blog, both of which are excellent examples of insults. I also want to mention that although the previous two are bad examples, there are fortunately also those who are able to resist primitive insults, in particular Gunnar Wolf (direct link to the blog does not work), and Miriam Ruiz blog, which contradict my blog and propose a different view, without pulling the knife. Lars Wirzenius Let us start with Lars (ah sorry, I have to call him Wirzenius here, strange custom has suddenly popped up in Debian!):
I think Preining mis-characterises what happens on the Linux kernel mailing list, and in Debian, in free software development in general, and what Sarah Sharp has said and done. I continue to think he is wrong about everything on this topic.
I am very surprised that Lars^WWirzenius is in possession of a crystal ball that allows him to see and evaluate my attitude towards all these items, considering that I have presented links to posts on the LKML, and yes, my interpretation of the matter. Mind that the whole text amounts to about 130 lines on my screen, while my personal opinion was stated in two paragraphs of total 9 lines plus some interspersed comments. And in these maybe 20 lines I didn t make general statements about open source, kernel development etc etc. Lars, I know you have closed your blog for comments so I couldn t ask you but please, can you send me one of your crystal balls if you have more of them? At least he has managed to keep a bit of proper writing and disagreed with my statement on Debian (whether there is fun or not in Debian). He is of course free to do that, but please, don t rob me the right to state that I think Debian has changed. All this dispute centers around people not being capable to distinguish two things: One, being against the Code of Conduct due to the inclusion of administrative actions without clear definitions, and Two, being pro offensive behavior and and insults. Now, dear Lars^WWirzenius, please listen: I never advocated abusive behavior or insults, nor do I defend it. (Did you hear that!) I simply opposed the Code of Conduct as ruling instrument. And what kind of emails I got due to my opposition was far outside the Code of Conduct you are so strongly defending. So please, stay at the facts, and stop insulting me. Thanks. Mart n Ferrari Concerning Mart n I don t have much to say but please, stop spreading lies. You stated:
Once again, he s complaining about how the fun from Debian has been lost because making sexist jokes, or treating other people like shit is not allowed any more.
Could you please come up with a reference to this? Or are you just interpreting? I am very disappointed about this level of discussion between Debian Developers. You not even cared to answer my comment on your blog. Should I say something clear here you should be happy that this has not been written on a Debian mailing list, otherwise I hope the Code of Conduct hammer would hit you.
So yes, it seems that at least these two blogs underline exactly what my opinion is: Communication culture in Debian has changed to be more company-like. Probably due to the ever increasing amount of developers that are paid for their work on Debian (Ubuntu), the pressure to follow US company codes has taken a firm grip, so firm that even stating my diverging opinion is already enough to get branded. Good Future!

Lunar: Reproducible builds: week 24 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Scott Kitterman fixed an issue with non-deterministic Depends generated by dh-python identified by Santiago Vila and Chris Lamb. Lunar updated the patch against dpkg which makes the order of files in control.tar.gz deterministic using the new --sort=name option available in GNU Tar 1.28. josch released sbuild version 0.66.0-1 with several fixes and improvements. The most notable one for reproducible builds is the new --build-path option and $build_path configuration variable added by akira which allows to explicitly chose a given build path. Reiner Herrmann wrote a new patch for dh-systemd to sort the list of unit files in the generated maintainer scripts. Packages fixed The following packages became reproducible due to changes in their build dependencies: aoeui, apron, camlmix, cudf, findlib, glpk-java, hawtjni, haxe, java-atk-wrapper, llvm-py, misery, mtasc, ocamldsort, optcomp, spamoracle. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Untested Patches submitted which have not made their way to the archive yet: reproducible.debian.net ProfitBricks once again increased their support for reproducible builds in Debian and in other free software projects by adding 58 new cores and 138 GiB of RAM to the already existing setup. Two new amd64 build nodes and 16 new amd64 build jobs have been added which doubles the build capacity per day and allows us to spot many kind of problems earlier. The size of the tmpfs where builds are performed has also been increased from 70 to 200 GiB on all amd64 build nodes. Huge thanks! When examining a package, a link now points to a table listing all previous recorded tests for the same package. (Mattia) The menu on the package pages has also been improved. (h01ger) Packages in the depwait state are now rescheduled automatically after five days. (h01ger) Links to documentation and other projects being tested have been made more visible on the landing page. (h01ger) To reduce noise on the team IRC channel five different types of notifications have been turned into mail notifications. The remaining ones have been shortened and the status changes have been limited to unstable and experimental. (h01ger) Maintainer notifications about status changes in a package will only be sent out once per day, and not on each status change. (h01ger) diffoscope development Some more experiments of concurrent processing have been made. None were good and reliable enough to be shared, though. Package reviews 48 reviews have been removed, 189 added and 23 updated this week. 9 FTBFS bugs were reported by Chris Lamb. Misc. h01ger met with Levente Polyak to discuss testing Arch Linux on Debian continuous test system with an easily extensible framework. The idea is to also allow testing of other distributions, and provide a nice package based view like the one for Debian.

Mario Lang: ccidentals in Haskell

I've had quite some fun recently (re)learning Haskell. My learning project is to implement braille music notation parsing in Haskell. Given that i've already implemented most of this stuff in C++, it gives me a great opportunity to rethink my algorithms. Not everything I've had to implement until now was actually pretty. I spent yesterday evening implementing accidentals handling, which turned out to be quite a mess. However, I wanted to share my definition of the circle of fifths, because I find it rather concise.
The problem Given a key signature (often expressed as the number of sharp or flat accidentals), tell which pitch classes are actually raised/lowered. While reading through music notation software, I have seen several implementations of this basic concept. However, I have never seen one which was so concise. 
module Accidental where
import           Data.Map (Map)
import qualified Data.Map as Map (fromList)
import qualified Haskore.Basic.Pitch as Pitch
fifths n   n >  0    = let [a,b,c,d,e,f,g] = fifths (n-1)
                       in  [d,e,f,g+1,a,b,c]
           n <  0    = let [a,b,c,d,e,f,g] = fifths (n+1)
                       in  [e,f,g,a,b,c,d-1]
           otherwise = replicate 7 0
Given this, we can easily define a Map of pitches to currently active accidentals/alterations. List comprehension to the rescue! 
accidentals :: Int -> Map Pitch.T Pitch.Relative
accidentals k = Map.fromList [ ((o, c), a)
                               o <- [0..maxOctave]
                             , (c, a) <- zip diatonicSteps $ fifths k
                             , a /= 0
                             ] where
  maxOctave = 9
  diatonicSteps = [Pitch.C, Pitch.D, Pitch.E, Pitch.F, Pitch.G,
                   Pitch.A, Pitch.B]
The full source code for the haskore-braille (WIP) package can be found on GitHub. If you have any comments regarding the implementation, please drop me a mail.

Russ Allbery: Review: Magic's Price

Review: Magic's Price, by Mercedes Lackey
Series: Last Herald Mage #3
Publisher: DAW
Copyright: July 1990
ISBN: 0-88677-426-8
Format: Mass market
Pages: 351
Magic's Price is the conclusion of the Last Herald Mage trilogy (Lackey is from an era of fantasy in which trilogies stopped at three books), but those books are widely spaced and mostly stand alone. You might miss some context around Vanyel's personal life and the politics of the kingdom, but there's a lot of somewhat-awkward in-line summary, so you could read these books out of order if you wanted to. As with the previous two, this is a re-read of a book I've read several times before, but not for more than fifteen years. This is the book that finally introduces Stefan. On this re-read, that surprised me I thought of him as a large part of Vanyel's overall arc but he's only in the third book. He is, of course, key to the grand conclusion of Vanyel's story, beginning as the Bard who has the ability to blunt the pain of the badly-suffering king of Valdemar and then becoming more than that to Van. It's also a book about Vanyel trying desperately to hold things together as the country falls on hard times and far too much weight is put on his shoulders. And a book about a climactic fight against an arch-villain (awkwardly and retroactively inserted). And Vanyel coming to terms with his own foresight. It's a book about rather a lot of things, and therein lies a problem. I thought Magic's Promise had a better story than Magic's Pawn, but both had reasonably coherent stories and some internal flow. Magic's Price has three or four interwoven stories and some odd stops and starts. This is still early in Lackey's writing career (three years after her first novel), and I don't think she was entirely successful in what she was trying to do. The climaxes of the different stories don't quite line up and don't get enough separate attention, leaving a book that lacks clear dramatic shape, an engrossing build-up, or a satisfying climax. The biggest problem I had is that the main threat is not adequately supported. The story is building towards a rather drastic conclusion, one that has been foreshadowed from early in the series, but the threat that creates that conclusion comes out of nowhere. Lackey tries to tie it back to events from earlier in the series, and maybe it was planned all along, but there's little concrete support and no build-up. It feels like a retcon. And the key character in that threat receives almost no characterization whatsoever. I'm a bit grateful for that, since Lackey tends towards total evil in her villains, but to have the cause of all the drama in this story be a complete cipher takes a lot out of the story. (Also, I truly could have done without the rape.) The other irritation is that Vanyel spends quite a lot of the book being a jackass. This is another occasional problem with Lackey's books: her love stories, at least in her early books, involve a bit too much of people being stupid at each other. But I hadn't remembered just how much Van jerked Stef around and treated him miserably, straight to the end of the book. Love is a bit too ordained and imbued with magical powers in Valdemar, which is lucky for Vanyel: extensive plot support is required to explain why Stef didn't find someone else who was less of an ass. In the real world, when someone insists they have no emotional space for you, the correct reaction is "gotcha, have a nice life" and finding someone else to spend time with. This gets partly sorted out in the end, but I didn't find the reconciliation entirely believable, and certainly not fair to Stef. (And the age difference is just a little creepy, although thankfully Lackey doesn't put much focus on that.) On my first read of the Valdemar corpus, this was the book during which I started getting interested in the lore and politics of this universe. There's much more of that to come, thankfully, as I think it's my favorite part of the series. At this point in her writing career, Lackey is getting better at avoiding infodumps and making her magic system interesting. The politics is still a bit black and white, but the Tayledras are a lot of fun (and increasingly less like elves, even if that's the obvious initial inspiration), and the areas beyond Valdemar are developing some more coherence. The conclusion loses a lot from the lack of build-up and emotional support, but it still adds an interesting bit of magical history that will be important later. I wish I'd enjoyed Magic's Price more on re-read, but it was much spottier and more exasperating than I had remembered. That said, it certainly has its moments, particularly around the lore. This is the first book where one starts to realize just how powerful the Companions are, and just how much they're capable of that they're not showing. When I first read it, I loved it. Now, the flaws are too obvious, and Vanyel's behavior is too irritating, for me to unreservedly recommend it, but Magic's Price, and the whole trilogy, is important for building the lore of later books. When I get to Elspeth's part of the series, the politics and magical system will be developed at much greater length. Next, though, my favorite stories in this universe: Tarma and Kethry. Rating: 6 out of 10

13 October 2015

Julien Danjou: Benchmarking Gnocchi for fun & profit

We got pretty good feedback on Gnocchi so far, even if we only had little. Recently, in order to have a better feeling of where we were at, we wanted to know how fast (or slow) Gnocchi was. The early benchmarks that some of the Mirantis engineers ran last year showed pretty good signs. But a year later, it was time to get real numbers and have a good understanding of Gnocchi capacity. Benchmark tools The first thing I realized when starting that process, is that we were lacking of tools to run benchmarks. Therefore I started to write some benchmark tools in python-gnocchiclient, which provides a command line tool to interrogate Gnocchi. I added a few basic commands to measure metric performance, such as: 
$ gnocchi benchmark metric create -w 48 -n 10000 -a low
+----------------------+------------------+
  Field                  Value             
+----------------------+------------------+
  client workers         48                
  create executed        10000             
  create failures        0                 
  create failures rate   0.00 %            
  create runtime         8.80 seconds      
  create speed           1136.96 create/s  
  delete executed        10000             
  delete failures        0                 
  delete failures rate   0.00 %            
  delete runtime         39.56 seconds     
  delete speed           252.75 delete/s   
+----------------------+------------------+

The command line tool supports the --verbose switch to have detailed progress report on the benchmark progression. So far it supports metric operations only, but that's the most interesting part of Gnocchi. Spinning up some hardware I got a couple of bare metal servers to test Gnocchi on. I dedicated the first one to Gnocchi, and used the second one as the benchmark client, plugged on the same network. Each server is made of 2 Intel Xeon E5-2609 v3 (12 cores in total) and 32 GB of RAM. That provides a lot of CPU to handle requests in parallel. Then I simply performed a basic RHEL 7 installation and ran devstack to spin up an installation of Gnocchi based on the master branch, disabling all of the others OpenStack components. I then tweaked the Apache httpd configuration to use the worker MPM and increased the maximum number of clients that can sent request simultaneously. I configured Gnocchi to use the PostsgreSQL indexer, as it's the recommended one, and the file storage driver, based on Carbonara (Gnocchi own storage engine). That means files were stored locally rather than in Ceph or Swift. Using the file driver is less scalable (you have to run on only one node or uses a technology like NFS to share the files), but it was good enough for this benchmark and to have some numbers and profiling the beast. The OpenStack Keystone authentication middleware was not enabled in this setup, as it would add some delay validating the authentication token. Metric CRUD operations Metric creation is pretty fast. I managed to attain 1500 metric/s created pretty easily. Deletion is now asynchronous, which means it's faster than in Gnocchi 1.2, but it's still slower than creation: 300 metric/s can be deleted. That does not sound like a huge issue since metric deletion is actually barely used in production. Retrieving metric information is also pretty fast and goes up to 800 metric/s. It'd be easy to achieve very higher throughput for this one, as it'd be easy to cache, but we didn't feel the need to implement it so far. Another important thing is that all of these numbers are constant and barely depends on the number of the metric already managed by Gnocchi.
Operation Details Rate
Create metric Created 100k metrics in 77 seconds 1300 metric/s
Delete metric Deleted 100k metrics in 190 seconds 524 metric/s
Show metric Show a metric 100k times in 149 seconds 670 metric/s
Sending and getting measures Pushing measures into metrics is one of the hottest topic. Starting with Gnocchi 1.1, the measures pushed are treated asynchronously, which makes it much faster to push new measures. Getting new numbers on that feature was pretty interesting. The number of metric per second you can push depends on the batch size, meaning the number of actual measurements you send per call. The naive approach is to push 1 measure per call, and in that case, Gnocchi is able to handle around 600 measures/s. With a batch containing 100 measures, the number of calls per second goes down to 450, but since you push 100 measures each time, that means 45k measures per second pushed into Gnocchi! I've pushed the test further, inspired by the recent blog post of InfluxDB claiming to achieve 300k points per second with their new engine. I ran the same benchmark on the hardware I had, which is roughly two times smaller than the one they used. I achieved to push Gnocchi to a little more than 120k measurement per second. If I had same hardware as they used, I could interpolate the results to achieve almost 250k measures/s pushed. Obviously, you can't strictly compare Gnocchi and InfluxDB since they are not doing exactly the same thing, but it still looks way better than what I expected. Using smaller batch sizes of 1k or 2k improve the throughput further to around 125k measures/s.
Operation Details Rate
Push metric 5k Push 5M measures with batch of 5k measures in 40 seconds 122k measures/s
Push metric 4k Push 5M measures with batch of 4k measures in 40 seconds 125k measures/s
Push metric 3k Push 5M measures with batch of 3k measures in 40 seconds 123k measures/s
Push metric 2k Push 5M measures with batch of 2k measures in 41 seconds 121k measures/s
Push metric 1k Push 5M measures with batch of 1k measures in 44 seconds 113k measures/s
Push metric 500 Push 5M measures with batch of 500 measures in 51 seconds 98k measures/s
Push metric 100 Push 5M measures with batch of 100 measures in 112 seconds 45k measures/s
Push metric 10 Push 5M measures with batch of 10 measures in 852 seconds 6k measures/s
Push metric 1 Push 500k measures with batch of 1 measure in 800 seconds 624 measures/s
Get measures Push 43k measures of 1 metric 260k measures/s
What about getting measures? Well, it's actually pretty fast too. Retrieving a metric with 1 month of data with 1 minute interval (that's 43k points) takes less than 2 second. Though it's actually slower than what I expected. The reason seems to be that the JSON is 2 MB big and encoding it takes a lot of time for Python. I'll investigate that. Another point I discovered, is that by default Gnocchi returns all the datapoints for each granularities available for the asked period, which might double the size of the returned data for nothing if you don't need it. It'll be easy to add an option to the API to only retrieve what you need though! Once benchmarked, that meant I was able to retrieve 6 metric/s per second, which translates to around 260k measures/s. Metricd speed New measures that are pushed into Gnocchi are processed asynchronously by the gnocchi-metricd daemon. When doing the benchmarks above, I ran into a very interesting issue: sending 10k measures on a metric would make gnocchi-metricd uses up to 2 GB RAM and 120 % CPU for more than 10 minutes. After further investigation, I found that the naive approach we used to resample datapoints in Carbonara using Pandas was causing that. I reported a bug on Pandas and the upstream author was kind enough to provide a nice workaround, that I sent as a pull request to Pandas documentation. I wrote a fix for Gnocchi based on that, and started using it. Computing the standard aggregation methods set (std, count, 95pct, min, max, sum, median, mean) for 10k batches of 1 measure (worst case scenario) for one metric with 10k measures now takes only 20 seconds and uses 100 MB of RAM 45 faster. That means that in normal operations, where only a few new measures are processed, the operation of updating a metric only takes a few milliseconds. Awesome! Comparison with Ceilometer For comparison sake, I've quickly run some read operations benchmark in Ceilometer. I've fed it with one month of samples for 100 instances polled every minute. That represents roughly 4.3M samples injected, and that took a while almost 1 hour whereas it would have taken less than a minute in Gnocchi. Then I tried to retrieve some statistics in the same way that we provide them in Gnocchi, which mean aggregating them over a period of 60 seconds over a month.
Operation Details Rate
Read metric SQL Read measures for 1 metric 2min 58s
Read metric MongoDB Read measures for 1 metric 28s
Read metric Gnocchi Read measures for 1 metric 2s
Obviously, Ceilometer is very slow. It has to look into 4M of samples to compute and return the result, which takes a lot of time. Whereas Gnocchi just has to fetch a file and pass it over. That also means that the more samples you have (so the more time you collect data and the more resources you have), slower Ceilometer will become. This is not a problem with Gnocchi, as I emphasized when I started designing it. Most Gnocchi operations are O(log R) where R is the number of metrics or resources, whereas most Ceilometer operations are O(log S) where S is the number of samples (measures). Since is R millions of time smaller than S, Gnocchi gets to be much faster. And what's even more interesting, is that Gnocchi is entirely scalable horizontally. Adding more Gnocchi servers (for the API and its background processing worker metricd) will multiply Gnocchi performances by the number of servers added. Improvements There are several things to improve in Gnocchi, such as splitting Carbonara archives to make them more efficient, especially from drivers such as Ceph and Swift. It's already on my plate, and I'm looking forwarding to working on that! And if you have any questions, feel free to shoot them in the comment section.

Norbert Preining: (Kim Doo Soo) ( , Free Spirit)

On the recently reviewed album International Sad Hits Vol 1 I wrote that Kim Doo Soo was the great surprise of the CD for me. So I went ahead and bought (not that easy, import from US!) one of his most recommended album, the Free Spirit. Kim_Doo_Soo-Free_Spirit Sad songs, full of melancholy, speaking of desperation, isolation, destroyed love. The music and singing is simple, mainly based on guitar and voice only, at times harmonica, strings and others are added. In contrast to the Crying Philosopher Tomokawa Kazuki, the songs are all very melodic and easy to listen, but still carrying a strong impact. After listening to these songs, the melodies wander around my brain resonating again and again, like (mis)guiding lights in a dark forest. kim-doo-soo-japanese-textAll the songs are in Korean, but fortunately the accompanying booklet carries a Japanese translation for the Korean incapables like me. In the future I will provide some translations into English, since I believe that this artist should be more known outside this small area. While a friend of mine said I don t like (bimbokusai) music, I have to say I enjoy reading through the sad stories while listening to this special music.
This is the 2002 fourth album by Korea s true cult folk/singer-songwriter hero, Kim Doo Soo. Kim Doo Soo is perhaps the deepest and most introspective of Korea s acid folk singers. Legends surround his songs political oppression, alcoholism, suicide, a 10-year period of mountain seclusion. [ ] On Free Spirit he mines productive veins of profound melancholy, animistic nature, and unfathomable, hermetic affection. These songs are couched in a veil of gorgeous, still melodicism, Kim s quavering vocals and guitar shaded with subtle accordion, cello, organ and harmonica. A reflective and unearthly masterpiece.
Forced Exposure
To this I cannot add anything more eloquent! The songs on this CD are as follows, the English translation is taken from Kim Doo Soo s official web page.
  1. (Wild flower)
  2. (Boat for the shore)
  3. (Butterfly)
  4. (Sweetbrier)
  5. (Bohemian)
  6. (Dawn rain)
  7. 19 Blues (Blues of 19th street)
  8. (Mountain)
  9. (As time goes by)
  10. Romantic Horizon
  11. ( ) (Reminiscence)
  12. (Evening river)
  13. ( ) (Bohemian poetry)
  14. , (Dandelion, dandelion)
My favorites under these songs are Boat for the shore ( You aren t coming any more onto a wild sea I am leaving ), Bohemian ( I m going to the shores of emptiness, my free spirit flows down the river probably his most famous song, assumed to be responsible for his retreat to the mountains), Dawn rain (A lovely song, and the only one without Japanese translation! Very strange.), Mountain ( Mountains Mountains my dream, mountains ). But I can recommend each and every song on this CD. To close this short review, if you are interested in something else but the usual happy-pappy KPop or JPop or USPop, and are searching for inspiring and songs with Tiefgang (deep meaning, as we say it in German), give this special artist a try! Some links concerning Kim Doo Soo that might be useful:

12 October 2015

Ben Armstrong: Learning Nova Scotia Plants with Anki Flashcards

plants_of_nova_scotiaOne of the greatest pleasures of walking and hiking is to appreciate all of the many living things encountered along the way. A big part of that appreciation for me is to be able to identify individual species and learn the relationships among them. To that end, I would like to introduce a flashcard deck I created, based on the glossary of the excellent, and also free, Nova Scotia Plants, by Marian C. Munro, Ruth E. Newell, and Nicholas M. Hill, so that I could more effectively use the book as an amateur student of our local flora. Creating the Nova Scotia Plants glossary for Anki I authored the deck on Debian, using the free software utility pdftotext (from poppler-utils), the small shell script below to produce a rough draft, and a text editor to clean up any errors, inconsistencies, and artefacts caused by the conversion process, such as descriptions which wrapped to a second line. 
#!/bin/sh
pdftotext -f 40 -l 55 \
  'Print Nova Scotia Plants complete manuscript.pdf' \
  glossary_raw.txt
egrep -v '^[ixvl]+$' glossary_raw.txt   \
  grep -P '^\f?[ a-z]+'   \
  sed -re 's/^\f?([^ ]+)( [- ]+ ? [- ]+ )( ?(.+))/\1\t\4/' \
  > glossary_import.txt
Nova Scotia Plants The Anki flashcard deck is intended as a companion for studying Nova Scotia Plants, linked above, and available either as a single PDF file, or multiple, smaller PDFs per section or family. This ebook has been a constant companion to me on my tablet during my walks and to study in quiet moments of the day. It has enriched my enjoyment of nature in Nova Scotia immeasurably. I am indebted to the authors for the years of work they put into it, and for making it available to the public for free. I hope you get as much out of it as I have. Anki for devices One of the criteria I used in selecting Anki as my flashcard software is that it is available for Debian, but also should work on my devices. I use the free software, AnkiDroid, on my Android phone and tablet, available both in F-Droid and the Google Play store. I understand there is also AnkiMobile for iOS, but that is not free. Feedback Please take the time to give me feedback. I spent an afternoon and a morning putting these materials together to share, and am eager to hear if my work has benefited you. Let me know if you have any suggestions for improvements, and don t forget to leave a review at ankiweb.net. facebooktwittergoogle_plusredditpinterestlinkedinmail

Michael Stapelberg: ratt: Rebuild All The Things!

When uploading a new library package which changes its API/behavior in a subtle way, typically you will only hear about the downstream breakage after you ve uploaded the new library package (via bug reports telling you that your package FTBFS, fails to build from source). I prefer quality-assurance to happen proactively rather than reactively whenever possible, so I set out to write a tool which automates the rebuild of reverse-build-dependencies, i.e. all packages whose build process could be affected by the package one is about to upload. The result is a tool called ratt, which is short for Rebuild All The Things! . It injects the newly-built Debian package you provide it, figures out all the reverse-build-dependencies, invokes sbuild for all of them, and finally presents you with a list of packages that failed to build. To demonstrate how the tool works, let s assume we want to upload a new version of the Go library github.com/jacobsa/gcloud. To keep this example brief, we don t actually do anything to the package but bump its version number: 
midna /tmp $ debcheckout golang-github-jacobsa-gcloud-dev
declared git repository at git://anonscm.debian.org/pkg-go/packages/golang-github-jacobsa-gcloud.git
git clone git://anonscm.debian.org/pkg-go/packages/golang-github-jacobsa-gcloud.git golang-github-jacobsa-gcloud-dev ...
Cloning into 'golang-github-jacobsa-gcloud-dev'...
remote: Counting objects: 84, done.
remote: Compressing objects: 100% (73/73), done.
remote: Total 84 (delta 19), reused 0 (delta 0)
Receiving objects: 100% (84/84), 87.70 KiB   0 bytes/s, done.
Resolving deltas: 100% (19/19), done.
Checking connectivity... done.
midna /tmp $ cd golang-github-jacobsa-gcloud-dev
midna /tmp/golang-github-jacobsa-gcloud-dev $ dch -i -m 'dummy new version'
midna /tmp/golang-github-jacobsa-gcloud-dev $ git commit -a -m 'dummy new version'
midna /tmp/golang-github-jacobsa-gcloud-dev $ gbp buildpackage --git-pbuilder
[ ]
So far, so good. Now let s invoke ratt to rebuild all reverse-build-dependencies: 
midna /tmp/golang-github-jacobsa-gcloud-dev $ ratt golang-github-jacobsa-gcloud_0.0\~git20150709-2_amd64.changes         
2015/08/16 11:48:41 Loading changes file "golang-github-jacobsa-gcloud_0.0~git20150709-2_amd64.changes"
2015/08/16 11:48:41  - 1 binary packages: golang-github-jacobsa-gcloud-dev
2015/08/16 11:48:41  - corresponding .debs (will be injected when building):
2015/08/16 11:48:41     golang-github-jacobsa-gcloud-dev_0.0~git20150709-2_all.deb
2015/08/16 11:48:41 Loading sources index "/var/lib/apt/lists/ftp.ch.debian.org_debian_dists_sid_contrib_source_Sources"
2015/08/16 11:48:41 Loading sources index "/var/lib/apt/lists/ftp.ch.debian.org_debian_dists_sid_main_source_Sources"
2015/08/16 11:48:43 Loading sources index "/var/lib/apt/lists/ftp.ch.debian.org_debian_dists_sid_non-free_source_Sources"
2015/08/16 11:48:43 Building golang-github-jacobsa-ratelimit_0.0~git20150723.0.2ca5e0c-1 (commandline: [sbuild --arch-all --dist=sid --nolog golang-github-jacobsa-ratelimit_0.0~git20150723.0.2ca5e0c-1 --extra-package=golang-github-jacobsa-gcloud-dev_0.0~git20150709-2_all.deb])
2015/08/16 11:49:19 Build results:
2015/08/16 11:49:19 PASSED: golang-github-jacobsa-ratelimit_0.0~git20150723.0.2ca5e0c-1
Thanks for reading this far, and I hope ratt makes your life a tiny bit easier. As ratt just entered Debian unstable, you can install it using apt-get install ratt. If you have any feedback, I m eager to hear it.

9 October 2015

Enrico Zini: extract-xml-from-fattura-elettronica

Extracting XML payload from Italian Fattura Elettronica zipfiles This system does not even export to PDF. In order to provide my accountant with something better than a DER-encoded file with a random-looking name stored inside a zipfile, here is a script that at least extracts the unsigned XML payload out of a saved Fattura Elettronica. If you're giving a python course in Italy, this sounds like a nice early programming assignment.

8 October 2015

Elena 'valhalla' Grandi: Swirlone

Swirlone

While I was sewing the small swirl plushie http://blog.zouish.org/posts/swirl_how_to/ for the Racoon http://social.gl-como.it/display/3e3ce0df1855db8250aae3a718529027 I thought that it would have been nice to have a bigger one, and I already had enough felt for it, so here it comes.

Immagine/fotohttp://social.gl-como.it/photo/99335297f698a6692c32e47fbdaded5e.jpg

Completely random sheet of A4 paper included for size comparison.

And now comes the hard part: to draw the pattern I enlarged the official swirl fit onto A3 and pdfposter-ed it to print it, then roughly followed the outline in pencil to get something cuttable. The problem is that this doesn't fit into the scanner, so publishing the sources is going to involve some manual work that I've been postponing because of lazyness O:)

They will be available. promise! Before the Release! (not necessarily Jessie, but...)

P.S. -one is the italian suffix for "big".

@Debian

Norbert Preining: Looking at the facts: Sarah Sharp s crusade

Much has been written around the internet about this geeky kernel maintainer Sarah Sharp who left kernel development. I have now spent two hours reading through lkml posts, and want to summarize a few mails from the long thread, since most of the usual news sites just rewrap the original blog of hers without adding any background. darth-cookie The whole thread evolved out call for stable kernel review by Greg Kroah-Hartman where he complained about too many patches that are not actually in rc1 before going into stable: 
<rant>
  I'm sitting on top of over 170 more patches that have been marked for
  the stable releases right now that are not included in this set of
  releases.  The fact that there are this many patches for stable stuff
  that are waiting to be merged through the main -rc1 merge window cycle
  is worrying to me.
  [...]
from where it developed into a typical Linus rant on people flagging crap for stable, followed by some jokes: 
On Fri, Jul 12, 2013 at 8:47 AM, Steven Rostedt <rostedt@goodmis.org> wrote:
> I tend to hold things off after -rc4 because you scare me more than Greg
> does ;-)
 
Have you guys *seen* Greg? The guy is a freakish giant. He *should*
scare you. He might squish you without ever even noticing.
 
              Linus
and Ingo Molnar giving advice to Greg KH: 
So Greg, if you want it all to change, create some _real_ threat: be frank 
with contributors and sometimes swear a bit. That will cut your mailqueue 
in half, promise!
with Greg KH taking a funny position in answering: 
Ok, I'll channel my "inner Linus" and take a cue from my kids and start
swearing more.
Up to now a pretty decent and normal thread with some jokes and poking, nobody minded, and reading through it I had a good time. The thread continues with a discussion on requirements what to submit to stable, and some side threads on particular commits. And then, out of the blue, Social Justice Warrior (SJW) Sarah Sharp pops in with a very important contribution: 
Seriously, guys?  Is this what we need in order to get improve -stable?
Linus Torvalds is advocating for physical intimidation and violence.
Ingo Molnar and Linus are advocating for verbal abuse.
 
Not *fucking* cool.  Violence, whether it be physical intimidation,
verbal threats or verbal abuse is not acceptable.  Keep it professional
on the mailing lists.
 
Let's discuss this at Kernel Summit where we can at least yell at each
other in person.  Yeah, just try yelling at me about this.  I'll roar
right back, louder, for all the people who lose their voice when they
get yelled at by top maintainers.  I won't be the nice girl anymore.
Onto which Linus answers in a great way: 
That's the spirit.
 
Greg has taught you well. You have controlled your fear. Now, release
your anger. Only your hatred can destroy me.
 
Come to the dark side, Sarah. We have cookies.
On goes Sarah, gearing up in her SJW mode and starting to rant: 
However, I am serious about this.  Linus, you're one of the worst
offenders when it comes to verbally abusing people and publicly tearing
their emotions apart.
 
http://marc.info/?l=linux-kernel&m=135628421403144&w=2
http://marc.info/?l=linux-acpi&m=136157944603147&w=2
 
I'm not going to put up with that shit any more.
Linus himself made clear what he thinks of her: 
Trust me, there's a really easy way for me to curse at people: if you
are a maintainer, and you make excuses for your bugs rather than
trying to fix them, I will curse at *YOU*.
 
Because then the problem really is you.
[...]
It is easy to verify what Linus said, by reading the above two links and the answers of the maintainers, both agreed that it was their failure and were sorry. (Mauro s answer, Rafael s answer) It is just the geeky SJW that was not even attacked (who would dare to attack a woman nowadays?). The overall reaction to her by the maintainers can be exemplified by Thomas Gleixner s post: 
Just for the record. I got grilled by Linus several times over the
last years and I can't remember a single instance where it was
unjustified.
What follows is a nearly endless discussion with Sarah meandering around, changing permanently her opinion what is acceptable. Linus tried to explain to her in simple words, without success, she continues to rant around. Here arguments are so weak I had nothing but good laugh: 
> Sarah, that's a pretty potent argument by Linus, that "acting 
> professionally" risks replacing a raw but honest culture with a
> polished but dishonest culture - which is harmful to developing
> good technology.
> 
> That's a valid concern. What's your reply to that argument?
 
I don't feel the need to comment, because I feel it's a straw man
argument.  I feel that way because I disagree with the definition of
professionalism that people have been pushing.
 
To me, being "professional" means treating each other with respect.  I
can show emotion, express displeasure, be direct, and still show respect
for my fellow developers.
 
For example, I find the following statement to be both direct and
respectful, because it's criticizing code, not the person:
 
"This code is SHIT!  It adds new warnings and it's marked for stable
when it's clearly *crap code* that's not a bug fix.  I'm going to revert
this merge, and I expect a fix from you IMMEDIATELY."
 
The following statement is not respectful, because it targets the
person:
 
"Seriously, Maintainer.  Why are you pushing this kind of *crap* code to
me again?  Why the hell did you mark it for stable when it's clearly
not a bug fix?  Did you even try to f*cking compile this?"
Fortunately, she was immediately corrected and Ingo Molnar wrote an excellent refutation (starting another funny thread) of all her emails, statements, accusations (all of the email is a good read): 
_That_ is why it might look to you as if the person was
attacked, because indeed the actions of the top level maintainer were
wrong and are criticised.
 
... and now you want to 'shut down' the discussion. With all due respect,
you started it, you have put out various heavy accusations here and elsewhere,
so you might as well take responsibility for it and let the discussion be
brought to a conclusion, wherever that may take us, compared to your initial view?
(He retracted that last statement, though I don t see a reason for it) Last but not least, let us return to her blog post, where she states herself that: 
FYI, comments will be moderated by someone other than me. As this is my blog, not a
government entity, I have the right to replace any comment I feel like with 
 fart fart fart fart .
and she made lots of use of it, I counted at least 10 instances. She seems to remove or fart fart fart any comment that is not in line with her opinion. Further evidence is provided by this post on lkml. Everyone is free to have his own opinion (sorry, his/her), and I am free to form my own opinion on Sarah Sharp by just simply reading the facts. I am more than happy that one more SJW has left Linux development, as the proliferation of cleaning of speech from any personality has taken too far a grip. Coming to my home-base in Debian, unfortunately there is no one in the position and the state of mind of Linus, so we are suffering the same stupidities imposed by social justice worriers and some brainless feminists (no, don t get me wrong, these are two independent attributes. I do NOT state that feminism is brainless) that Linus and the maintainer crew was able to fend of this time. I finish with my favorite post from that thread, by Steven Rosted (from whom I also stole the above image!): 
On Tue, 2013-07-16 at 18:37 -0700, Linus Torvalds wrote:
> Emotions aren't bad. Quite the reverse. 
 
Spock and Dr. Sheldon Cooper strongly disagree.
Post Scriptum (after a bike ride) The last point by Linus is what I criticize most on Debian nowdays, it has become a sterilized over-governed entity, where most fun is gone. Making fun is practically forbidden, since there is the slight chance that some minority somewhere on this planet might feel hurt, and by this we are breaking the CoC. Emotions are restricted to the Happy happy how nice we are and how good we are level of US and also Japanese self-reenforcement society. Post Post Scriptum I just read Sarah Sharp s post on What makes a good community? , and without giving a full account or review, I am just pi**ed by the usage of the word microaggressions I can only recommend everyone to read this article and this article to get a feeling how bs the idea of microaggressions has taken over academia and obviously not only academia. Post3 Scriptum I am happy to see Lars Wirzenius, Gunnar Wolf, and Mart n Ferrari opposing my view. I agree with them that my comments concerning Debian are not mainstream in Debian something that is not very surprising, though, and I think it is great that they have fun in Debian, like many other contributors. Post4 Scriptum Although nobody will read this, here is a great response from a female developer: 
[...] To Linus: You're a hero to many of us. Don't change. Please. You DO
NOT need to take time away from doing code to grow a pair of breasts
and judge people's emotional states: [...]
Nothing to add here!

7 October 2015

Emanuele Rocca: systemd is your friend

Today I want to talk a bit about some cool features of systemd, the default Debian init system since the release of Jessie. Ubuntu has also adopted systemd in 15.04, meaning that you are going to find it literally everywhere.
Logging The component responsible for logging in systemd is called journal. It collects and stores logs in a structured, indexed journal (hence the name). The journal can replace traditional syslog daemons such as rsyslog and syslog-ng, or work together with them. By default Debian keeps on using rsyslog, but if you don't need to ship logs to a centralized server (or do other fancy things) it is possible to stop using rsyslog right now and rely on systemd-journal instead. The obvious question is: why would anybody use a binary format for logs instead of a bunch of tried and true plain-text files? As it turns out, there are quite a lot of good reasons to do so. The killer features of systemd-journald for me are:
  • Index tons of logs while being able to search with good performance: O(log(n)) instead of O(n) which is what you get with text files
  • No need to worry about log rotation anymore, in any shape or form
The last point in particular is really critical in my opinion. Traditional log rotation implementations rely on cron jobs to check how much disk space is used by logs, compressing/removing old files. Log rotation is usually: 1) annoying to configure; 2) hard to get right; 3) prone to DoS attacks. With journald, there is pretty much nothing to configure. Log rotation is built into the daemon disk space allocation logic itself. This also allows to avoid vulnerability windows due to time-based rotation, which is what you get with logrotate and friends. Enough high-level discussions though, here is how to use the journal! Check if you already have the directory /var/log/journal, otherwise create it (as root). Then restart systemd-journald as follows: sudo systemctl restart systemd-journald You can get all messages produced since the last boot with journalctl -b. All messages produced today can get extracted using journalctl --since=today. Want to get all logs related to ssh? Try with journalctl _SYSTEMD_UNIT=ssh.service. There are many more filtering options available, you can read all about them with man journalctl. journald's configuration file is /etc/systemd/journald.conf. Two of the most interesting options are SystemMaxUse and SystemKeepFree, which can be used to change the amount of disk space dedicated to logging. They default to 10% and 15% of the /var filesystem respectively. Here is a little cheatsheet: 
journalctl -b                # Show all messages since last boot
journalctl -f                # Tail your logs
journalctl --since=yesterday # Show all messages produced since yesterday
journalctl -pcrit            # Filter messages by priority
journalctl /bin/su           # Filter messages by program
journalctl --disk-usage      # The amount of space in use for journaling
Further reading:
Containers A relatively little known component of systemd is systemd-nspawn. It is a small, straightforward container manager. If you don't already have a chroot somewhere, here is how to create a basic Debian Jessie chroot under /srv/chroots/jessie: 
$ debootstrap jessie /srv/chroots/jessie http://http.debian.net/debian/
With systemd-nspawn you can easily run a shell inside the chroot: 
$ sudo systemd-nspawn -D /srv/chroots/jessie
Spawning container jessie on /srv/chroots/jessie.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
root@jessie:~#
Done. Everything works out of the box: no need for you to mount /dev, /run and friends, systemd-nspawn took care of that. Networking also works. If you want to actually boot the system, just add the -b switch to the previous command: 
$ sudo systemd-nspawn -b -D /srv/chroots/jessie
Spawning container jessie on /srv/chroots/jessie.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
systemd 215 running in system mode. (+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR)
Detected virtualization 'systemd-nspawn'.
Detected architecture 'x86-64'.
Welcome to Debian GNU/Linux jessie/sid!
Set hostname to <orion>.
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Swap.
[  OK  ] Created slice Root Slice.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Created slice System Slice.
[  OK  ] Created slice system-getty.slice.
[  OK  ] Listening on Syslog Socket.
         Mounting POSIX Message Queue File System...
         Mounting Huge Pages File System...
         Mounting FUSE Control File System...
         Starting Copy rules generated while the root was ro...
         Starting Journal Service...
[  OK  ] Started Journal Service.
[  OK  ] Reached target Slices.
         Starting Remount Root and Kernel File Systems...
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Started Copy rules generated while the root was ro.
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Reached target Remote File Systems.
         Starting Trigger Flushing of Journal to Persistent Storage...
[  OK  ] Started Load/Save Random Seed.
         Starting LSB: Raise network interfaces....
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Trigger Flushing of Journal to Persistent Storage.
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started LSB: Raise network interfaces..
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
         Starting /etc/rc.local Compatibility...
         Starting Login Service...
         Starting LSB: Regular background program processing daemon...
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
         Starting System Logging Service...
[  OK  ] Started System Logging Service.
         Starting Permit User Sessions...
[  OK  ] Started /etc/rc.local Compatibility.
[  OK  ] Started LSB: Regular background program processing daemon.
         Starting Cleanup of Temporary Directories...
[  OK  ] Started Permit User Sessions.
         Starting Console Getty...
[  OK  ] Started Console Getty.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Login Service.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Cleanup of Temporary Directories.
[  OK  ] Started Update UTMP about System Runlevel Changes.
Debian GNU/Linux jessie/sid orion console
orion login:
That's it! Just one command to start a shell in your chroot or boot the container, again zero configuration needed. Finally, systemd provides a command called machinectl that allows you to introspect and control your container: 
$ sudo machinectl status jessie
jessie
           Since: Wed 2015-10-07 11:22:56 CEST; 55min ago
          Leader: 32468 (systemd)
         Service: nspawn; class container
            Root: /srv/chroots/jessie
         Address: fe80::8e70:5aff:fe81:2290
                  192.168.122.1
                  192.168.1.13
              OS: Debian GNU/Linux jessie/sid
            Unit: machine-jessie.scope
                   32468 /lib/systemd/systemd
                   system.slice
                     dbus.service
                       32534 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile -...
                     cron.service
                       32539 /usr/sbin/cron
                     systemd-journald.service
                       32487 /lib/systemd/systemd-journald
                     systemd-logind.service
                       32532 /lib/systemd/systemd-logind
                     console-getty.service
                       32544 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt102
                     rsyslog.service
                       32540 /usr/sbin/rsyslogd -n
With machinectl you can also reboot, poweroff, terminate your containers and more. There are so many things to learn about systemd and containers! Here are some references. This stuff is pretty exciting. Now that all major distributions use systemd by default, we can expect to have access to tools like journalctl and systemd-nspawn everywhere!

Next.

Previous.