A new release of our BH package providing Boost headers for use by R is now on the CRAN mirrors. This release is the third based on Boost 1.54.0. At the request of the maintainer of the recent added RcppMLPACK package, it adds the Boost.Heap library. Boost.Heap implements priority queues which extend beyond the corresponding (and somewhat simpler) class in the STL. Key features of the Boost.Heap priority queues are mutability, iterators, ability to merge, stable sort, and comparison. No other changes were made.

Changes in version 1.54.0-3 (2014-08-03)

• Added Boost Heap library which will be needed by the next version of RcppMLPACK

Courtesy of CRANberries, there is also a diffstat report for the most recent release. Comments and suggestions are welcome via the mailing list or issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I wrote a BTRFS status report yesterday commenting on the uneventful use of BTRFS recently [1]. Early this morning the server that stores my email (which had 93 days uptime) had a filesystem related problem. The root filesystem became read-only and then the kernel message log filled with unrelated messages so there was no record of the problem. I m now considering setting up rsyslogd to log the kernel messages to a tmpfs filesystem to cover such problems in future. As RAM is so cheap it wouldn t matter if a few megs of RAM were wasted by that in normal operation if it allowed me to extract useful data when something goes really wrong. It s really annoying to have a system in a state where I can login as root but not find out what went wrong. After that I tried 2 kernels in the 3.14 series, both of which had kernel BUG assertions related to Xen networking and failed to network correctly, I filed Debian Bug #756714. Fortunately they at least had enough uptime for me to run a filesystem scrub which reported no errors. Then I reverted to kernel 3.13.10 but the reboot to apply that kernel change failed. Systemd was unable to umount the root filesystem (maybe because of a problem with Xen) and then hung the system instead of rebooting, I filed Debian Bug #756725. I believe that if asked to reboot a system there is no benefit in hanging the system with no user space processes accessible. Here are some useful things that systemd could have done:

1. Just reboot without umounting (like reboot -nf does).
2. Pause for some reasonable amount of time to give the sysadmin a possibility of seeing the error and then rebooting.
3. Go back to a regular runlevel, starting daemons like sshd.
4. Offer a login prompt to allow the sysadmin to login as root and diagnose the problem.

Options 1, 2, and 3 would have saved me a bit of driving. Option 4 would have allowed me to at least diagnose the problem (which might be worth the drive). Having a system on the other side of the city which has no remote console access just hang after a reboot command is not useful, it would be near the top of the list of things I don t want to happen in that situation. The best thing I can say about systemd s operation in this regard is that it didn t make the server catch fire. Now all I really know is that 3.14 kernels won t work for my server, 3.13 will cause problems that no-one can diagnose due to lack of data, and I m now going to wait for it to fail again. As an aside the server has ECC RAM and it s hardware is known to be good, so I m sure that BTRFS is at fault.

• [1] http://etbe.coker.com.au/2014/07/31/btrfs-status-july-2014/

I was recently updating some code that uses fping. Initially it used exec() that was redirected to a temporary file but I changed it to use popen. While it had been a while since I ve done this sort of thing, I do recall there was an issue with running popen on setuid binary. A later found it is mainly around setuid scripts which are very problematic and there are good reasons why you don t do this. Anyhow, the program worked fine which surprised me. Was fping setuid root to get the raw socket? It wasn t which at first all I thought ok, so that s why popen is happy . The way that fping and other programs work is they bind to a raw socket. This socket sits below the normal type sockets such as the ones used for TCP and UDP and normal users cannot use them by default. So how did fping work it s magic and get access to this socket? It used Capabilities. Previously getting privileged features had a big problem; it was an all or nothing thing. You want access to a raw socket? Sure, be setuid but that means you also could, for example, read any file on the system or set passwords. Capabilites provide a way of giving programs some better level of access, but not a blank cheque. The tool getcap is the way of determining what capabilities are found on a file. These capabilities are attributes on the file which, when the file is run, turn into capabilities or extra permissions. fping has the capability cap_net_raw+ep applied to it. This gives access to the RAW and PACKET sockets which is what fping needs. The +ep after the capability name means it is an Effective and Permitted capability, which describes what happens with child processes and dropping privileges. I hadn t seen these Capabilities before. They are a nice way to give your programs the access they need, but limiting the risk of something going wrong and having a rouge program running as root.

Related articles

• Lesser-known tool of the day: getcap, setcap and file capabilities (insecure.ws)
• Using File Capabilities Instead Of Setuid (wiki.archlinux.org)
• Safer suexec: from setuid to Linux capabilities (welldefinedbehaviour.wordpress.com)
• Kees Cook: easy example of filesystem capabilities (outflux.net)

My last BTRFS status report was in April [1], it wasn t the most positive report with data corruption and system hangs. Hacker News has a brief discussion of BTRFS which includes the statement Russell Coker s reports of his experiences with BTRFS give me the screaming heebie-jeebies, no matter how up-beat and positive he stays about it [2] (that s one of my favorite comments about my blog). Since April things have worked better. Linux kernel 3.14 solves the worst problems I had with 3.13 and it s generally doing everything I want it to do. I now have cron jobs making snapshots as often as I wish (as frequently as every 15 minutes on some systems), automatically removing snapshots (removing 500+ snapshots at once doesn t hang the system), balancing, and scrubbing. The fact that I can now expect that a filesystem balance (which is a type of defragment operation for BTRFS that frees some chunks ) from a cron job and expect the system not to hang means that I haven t run out of metadata chunk space. I expect that running out of metadata space can still cause filesystem deadlocks given a lack of reports on the BTRFS mailing list of fixes in that regard, but as long as balance works well we can work around that. My main workstation now has 35 days of uptime and my home server has 90 days of uptime. Also the server that stores my email now has 93 days uptime even though it s running Linux kernel 3.13.10. I am rather nervous about the server running 3.13.10 because in my experience every kernel before 3.14.1 had BTRFS problems that would cause system hangs. I don t want a server that s an hour s drive away to hang The server that runs my email is using kernel 3.13.10 because when I briefly tried a 3.14 kernel it didn t work reliably with the Xen kernel 4.1 from Debian/Wheezy and I had a choice of using the Xen kernel 4.3 from Debian/Unstable to match the Linux kernel or use an earlier Linux kernel. I have a couple of Xen servers running Debian/Unstable for test purposes which are working well so I may upgrade my mail server to the latest Xen and Linux kernels from Unstable in the near future. But for the moment I m just not doing many snapshots and never running a filesystem scrub on that server. Scrubbing In kernel 3.14 scrub is working reliably for me and I have cron jobs to scrub filesystems on every system running that kernel. So far I ve never seen it report an error on a system that matters to me but I expect that it will happen eventually. The paper An Analysis of Data Corruption in the Storage Stack from the University of Wisconsin (based on NetApp data) [3] shows that nearline disks (IE any disks I can afford) have an incidence of checksum errors (occasions when the disk returns bad data but claims it to be good) of about 0.42%. There are 18 disks running in systems I personally care about (as opposed to systems where I am paid to care) so with a 0.42% probability of a disk experiencing data corruption per year that would give a 7.3% probability of having such corruption on one disk in any year and a greater than 50% chance that it s already happened over the last 10 years. Of the 18 disks in question 15 are currently running BTRFS. Of the 15 running BTRFS 10 are scrubbed regularly (the other 5 are systems that don t run 24*7 and the system running kernel 3.13.10). Newer Kernels The discussion on the BTRFS mailing list about kernel 3.15 is mostly about hangs. This is correlated with some changes to improve performance so I presume that it has exposed race conditions. Based on those discussions I haven t felt inclined to run a 3.15 kernel. As the developers already have some good bug reports I don t think that I could provide any benefit by doing more testing at this time. I think that there would be no benefit to me personally or the Linux community in testing 3.15. I don t have a personal interest in RAID-5 or RAID-6. The only systems I run that have more data than will fit on a RAID-1 array of cheap SATA disks are ones that I am paid to run and they are running ZFS. So the ongoing development of RAID-5 and RAID-6 code isn t an incentive for me to run newer kernels. Eventually I ll test out RAID-6 code, but at the moment I don t think they need more bug reports in this area. I don t have a great personal interest in filesystem performance at this time. There are some serious BTRFS performance issues. One problem is that a filesystem balance and subtree removal seem to take excessive amounts of CPU time. Another is that there isn t much support for balancing IO to multiple devices (in RAID-1 every process has all it s read requests sent to one device). For large-scale use of a filesystem these are significant problems. But when you have basic requirements (such as a mail server for dozens of users or a personal workstation with a quad-core CPU and fast SSD storage) it doesn t make much difference. Currently all of my systems which use BTRFS have storage hardware that exceeds the system performance requirements by such a large margin that nothing other than installing Debian packages can slow the system down. So while there are performance improvements in newer versions of the BTRFS kernel code that isn t an incentive for me to upgrade. It s just been announced that Debian/Jessie will use Linux 3.16, so I guess I ll have to test that a bit for the benefit of Debian users. I am concerned that 3.16 won t be stable enough for typical users at the time that Jessie is released.

• [1] http://etbe.coker.com.au/2014/04/26/btrfs-status-april-2014/
• [2] https://news.ycombinator.com/item?id=7830970
• [3] http://research.cs.wisc.edu/adsl/Publications/corruption-fast08.html

This is time again for my annual report about the number of developers per country. This is now the sixth edition of this report. Former editions:

• July 2013
• June 2012
• June 2011
• August 2010
• October 2009

So, here we are with the July 2014 version, sorted by the ratio of *active* developers per million population for each country.

Act: number of active developers
Dev: total number of developers
A/M: number of active devels per million pop.
D/M: number of devels per million pop.
2009: rank in 2009
2010: rank in 2010
2011: rank in 2011 (June)
2012: rank in 2012 (June)
2013: rank in 2012 (July)
2014: rank now

Code	Name	Population	Act Dev	Dev	Act/Million	Dev/Million	2009	2010	June 2011	June 2012	July 2013	July 2014
fi	Finland	5259250	19	31	3,61	5,89	1	1	1	1	1	1
ie	Ireland	4670976	13	17	2,78	3,64	13	9	6	2	2	2
nz	New Zealand	4331600	11	15	2,54	3,46	4	3	5	7	7	3	*
mq	Martinique	396404	1	1	2,52	2,52			3	4	4	4
se	Sweden	9088728	22	37	2,42	4,07	3	6	7	5	5	5
ch	Switzerland	7870134	19	29	2,41	3,68	2	2	2	3	3	6	*
no	Norway	4973029	11	14	2,21	2,82	5	4	4	6	6	7	*
at	Austria	8217280	18	29	2,19	3,53	6	8	10	10	10	8	*
de	Germany	81471834	164	235	2,01	2,88	7	7	9	9	8	9	*
lu	Luxemburg	503302	1	1	1,99	1,99	8	5	8	8	9	10	*
fr	France	65350000	101	131	1,55	2	12	12	11	11	11	11
au	Australia	22607571	32	60	1,42	2,65	9	10	12	12	12	12
be	Belgium	11071483	14	17	1,26	1,54	10	11	13	13	13	13
uk	United-Kingdom	62698362	77	118	1,23	1,88	14	14	14	14	14	14
nl	Netherlands	16728091	18	40	1,08	2,39	11	13	15	15	15	15
ca	Canada	33476688	34	63	1,02	1,88	15	15	17	16	16	16
dk	Denmark	5529888	5	10	0,9	1,81	17	17	16	17	17	17
es	Spain	46754784	34	56	0,73	1,2	16	16	19	18	18	18
it	Italy	59464644	36	52	0,61	0,87	23	22	22	19	19	19
hu	Hungary	10076062	6	12	0,6	1,19	18	25	26	20	24	20	*
cz	Czech Rep	10190213	6	6	0,59	0,59	21	20	21	21	20	21	*
us	USA	313232044	175	382	0,56	1,22	19	21	25	24	22	22
il	Israel	7740900	4	6	0,52	0,78	24	24	24	25	23	23
hr	Croatia	4290612	2	2	0,47	0,47	20	18	18	26	25	24	*
lv	Latvia	2204708	1	1	0,45	0,45	26	26	27	27	26	25	*
bg	Bulgaria	7364570	3	3	0,41	0,41	25	23	23	23	27	26	*
sg	Singapore	5183700	2	2	0,39	0,39				33	33	27	*
uy	Uruguay	3477778	1	2	0,29	0,58	22	27	28	28	28	28
pl	Poland	38441588	11	15	0,29	0,39	29	29	30	30	30	29	*
jp	Japan	127078679	36	52	0,28	0,41	30	28	29	29	29	30	*
lt	Lithuania	3535547	1	1	0,28	0,28	28	19	20	22	21	31	*
gr	Greece	10787690	3	4	0,28	0,37	33	38	34	35	35	32	*
cr	Costa Rica	4301712	1	1	0,23	0,23	31	30	31	31	31	33	*
by	Belarus	9577552	2	2	0,21	0,21	35	36	39	39	32	34	*
ar	Argentina	40677348	8	10	0,2	0,25	34	33	35	32	37	35	*
pt	Portugal	10561614	2	4	0,19	0,38	27	32	32	34	34	36	*
sk	Slovakia	5477038	1	1	0,18	0,18	32	31	33	36	36	37	*
rs	Serbia	7186862	1	1	0,14	0,14					38	38
tw	Taiwan	23040040	3	3	0,13	0,13	37	34	37	37	39	39
br	Brazil	192376496	18	21	0,09	0,11	36	35	38	38	40	40
cu	Cuba	11241161	1	1	0,09	0,09		38	41	41	41	41
co	Colombia	45566856	4	5	0,09	0,11	41	44	46	47	46	42	*
kr	South Korea	48754657	4	6	0,08	0,12	39	39	42	42	42	43	*
gt	Guatemala	13824463	1	1	0,07	0,07					43	44	*
ec	Ecuador	15007343	1	1	0,07	0,07		40	43	43	45	45
cl	Chile	16746491	1	2	0,06	0,12	42	41	44	44	47	46	*
za	South Africa	50590000	3	10	0,06	0,2	38	48	48	48	48	47	*
ru	Russia	143030106	8	9	0,06	0,06	43	42	47	45	49	48	*
mg	Madagascar	21281844	1	1	0,05	0,05	44	37	40	40	50	49	*
ro	Romania	21904551	1	2	0,05	0,09	45	43	45	46	51	50	*
ve	Venezuela	28047938	1	1	0,04	0,04	40	45	50	49	44	51	*
my	Malaysia	28250000	1	1	0,04	0,04			49	50	52	52
pe	Peru	29907003	1	1	0,03	0,03	46	46	51	51	53	53
tr	Turkey	74724269	2	2	0,03	0,03	47	47	52	52	54	54
ua	Ukraine	45134707	1	1	0,02	0,02	48	53	58	59	55	55
th	Thailand	66720153	1	2	0,01	0,03	50	50	54	54	56	56
eg	Egypt	80081093	1	3	0,01	0,04	51	51	55	55	57	57
mx	Mexico	112336538	1	1	0,01	0,01	49	49	53	53	58	58
cn	China	1344413526	10	14	0,01	0,01	53	53	57	56	59	59
in	India	1210193422	8	9	0,01	0,01	52	52	56	57	60	60
sv	El Salvador	7066403	0	1	0	0,14			36	58	61	61
			969	1561	62,08%

A few interesting facts:

• New Zealand bumps from rank 7 to rank 3, thanks to one new active developer
• Switzerland loses one developer and goes donw to rank 6
• Norway also slightly goes down by losing one developer
• With two more developers, Austria climbs up to rank 8 and overtakes Germany...;-)
• Hungary climbs a little bit by gaining one developer
• Singapore doubles its number of developers from 1 to 2 and bumps from 33 to 27
• One rank up too for Poland that gained one developer
• Down to rank 31 for Lithuania by losing one developer
• Up to rank 32 for Greece with 4 developers instead of 3
• Argentina goes up by havign two more developers (it lost 2 last year)
• Up from 46 to 42 for Colombia by winning one more developer
• One more developer and Russia climps from 49 to 48
• One less for Venezuela that has only one developer left...:-(
• No new country this year. Less movement towards "the universal OS"?
• We have 12 more active Debian developers and 26 more developers overall. Less progression than last year
• The ratio of active developers increases is nearly stable though slightly decreasing

Finally I find the mood to do this, a process which started 5 years ago in DebConf 9 at C ceres by following Ana's post, of course with my preferred options and my name, not like some other ;-).

Said that, dear reader, if you have signed my old key:

1024D/C9B55DAC 2005-01-19 [expires: 2015-10-01]
Key fingerprint = CFB7 C779 6BAE E81C 3E05  7172 2C04 5542 C9B5 5DAC

And want to sign my "new" and stronger key:

4096R/DE5BCCA6 2009-07-29
Key fingerprint = 43BC 364B 16DF 0C20 5EBD  7592 1F0F 0A88 DE5B CCA6

You're welcome to do so :-)

The new key is signed with the old, and the old key is still valid, and will probably be until expiration date next year. Don't forget to gpg --recv-keys DE5BCCA6 to get the new key and gpg --refresh-keys C9B55DAC to refresh the old (otherwise it may look expired).

Debian's Keyring Team has already processed my request to add the new key, so all should keep working smoothly. Kudos to them!

start-stop-daemon is the classic tool on Debian and derived distributions to manage system background processes. A typical invokation from an initscript is as follows: The basic operation is that it will first check whether /usr/sbin/daemon is not running and, if not, execute /usr/sbin/daemon -c /etc/daemon.cfg -p /var/run/daemon.pid. This process then has the responsibility to daemonise itself and write the resulting process ID to /var/run/daemon.pid. start-stop-daemon then waits until /var/run/daemon.pid has been created as the test of whether the service has actually started, raising an error if that doesn't happen. (In practice, the locations of all these files are parameterised to prevent DRY violations.)

This is time again for my annual report about the number of developers per country. This is now the sixth edition of this report. Former editions:

• July 2013
• June 2012
• June 2011
• August 2010
• October 2009

So, here we are with the July 2014 version, sorted by the ratio of *active* developers per million population for each country.

Act: number of active developers
Dev: total number of developers
A/M: number of active devels per million pop.
D/M: number of devels per million pop.
2009: rank in 2009
2010: rank in 2010
2011: rank in 2011 (June)
2012: rank in 2012 (June)
2013: rank in 2012 (July)
2014: rank now

Code	Name	Population	Act Dev	Dev	Act/Million	Dev/Million	2009	2010	June 2011	June 2012	July 2013	July 2014
fi	Finland	5259250	19	31	3,61	5,89	1	1	1	1	1	1
ie	Ireland	4670976	13	17	2,78	3,64	13	9	6	2	2	2
nz	New Zealand	4331600	11	15	2,54	3,46	4	3	5	7	7	3	*
mq	Martinique	396404	1	1	2,52	2,52			3	4	4	4
se	Sweden	9088728	22	37	2,42	4,07	3	6	7	5	5	5
ch	Switzerland	7870134	19	29	2,41	3,68	2	2	2	3	3	6	*
no	Norway	4973029	11	14	2,21	2,82	5	4	4	6	6	7	*
at	Austria	8217280	18	29	2,19	3,53	6	8	10	10	10	8	*
de	Germany	81471834	164	235	2,01	2,88	7	7	9	9	8	9	*
lu	Luxemburg	503302	1	1	1,99	1,99	8	5	8	8	9	10	*
fr	France	65350000	101	131	1,55	2	12	12	11	11	11	11
au	Australia	22607571	32	60	1,42	2,65	9	10	12	12	12	12
be	Belgium	11071483	14	17	1,26	1,54	10	11	13	13	13	13
uk	United-Kingdom	62698362	77	118	1,23	1,88	14	14	14	14	14	14
nl	Netherlands	16728091	18	40	1,08	2,39	11	13	15	15	15	15
ca	Canada	33476688	34	63	1,02	1,88	15	15	17	16	16	16
dk	Denmark	5529888	5	10	0,9	1,81	17	17	16	17	17	17
es	Spain	46754784	34	56	0,73	1,2	16	16	19	18	18	18
it	Italy	59464644	36	52	0,61	0,87	23	22	22	19	19	19
hu	Hungary	10076062	6	12	0,6	1,19	18	25	26	20	24	20	*
cz	Czech Rep	10190213	6	6	0,59	0,59	21	20	21	21	20	21	*
us	USA	313232044	175	382	0,56	1,22	19	21	25	24	22	22
il	Israel	7740900	4	6	0,52	0,78	24	24	24	25	23	23
hr	Croatia	4290612	2	2	0,47	0,47	20	18	18	26	25	24	*
lv	Latvia	2204708	1	1	0,45	0,45	26	26	27	27	26	25	*
bg	Bulgaria	7364570	3	3	0,41	0,41	25	23	23	23	27	26	*
sg	Singapore	5183700	2	2	0,39	0,39				33	33	27	*
uy	Uruguay	3477778	1	2	0,29	0,58	22	27	28	28	28	28
pl	Poland	38441588	11	15	0,29	0,39	29	29	30	30	30	29	*
jp	Japan	127078679	36	52	0,28	0,41	30	28	29	29	29	30	*
lt	Lithuania	3535547	1	1	0,28	0,28	28	19	20	22	21	31	*
gr	Greece	10787690	3	4	0,28	0,37	33	38	34	35	35	32	*
cr	Costa Rica	4301712	1	1	0,23	0,23	31	30	31	31	31	33	*
by	Belarus	9577552	2	2	0,21	0,21	35	36	39	39	32	34	*
ar	Argentina	40677348	8	10	0,2	0,25	34	33	35	32	37	35	*
pt	Portugal	10561614	2	4	0,19	0,38	27	32	32	34	34	36	*
sk	Slovakia	5477038	1	1	0,18	0,18	32	31	33	36	36	37	*
rs	Serbia	7186862	1	1	0,14	0,14					38	38
tw	Taiwan	23040040	3	3	0,13	0,13	37	34	37	37	39	39
br	Brazil	192376496	18	21	0,09	0,11	36	35	38	38	40	40
cu	Cuba	11241161	1	1	0,09	0,09		38	41	41	41	41
co	Colombia	45566856	4	5	0,09	0,11	41	44	46	47	46	42	*
kr	South Korea	48754657	4	6	0,08	0,12	39	39	42	42	42	43	*
gt	Guatemala	13824463	1	1	0,07	0,07					43	44	*
ec	Ecuador	15007343	1	1	0,07	0,07		40	43	43	45	45
cl	Chile	16746491	1	2	0,06	0,12	42	41	44	44	47	46	*
za	South Africa	50590000	3	10	0,06	0,2	38	48	48	48	48	47	*
ru	Russia	143030106	8	9	0,06	0,06	43	42	47	45	49	48	*
mg	Madagascar	21281844	1	1	0,05	0,05	44	37	40	40	50	49	*
ro	Romania	21904551	1	2	0,05	0,09	45	43	45	46	51	50	*
ve	Venezuela	28047938	1	1	0,04	0,04	40	45	50	49	44	51	*
my	Malaysia	28250000	1	1	0,04	0,04			49	50	52	52
pe	Peru	29907003	1	1	0,03	0,03	46	46	51	51	53	53
tr	Turkey	74724269	2	2	0,03	0,03	47	47	52	52	54	54
ua	Ukraine	45134707	1	1	0,02	0,02	48	53	58	59	55	55
th	Thailand	66720153	1	2	0,01	0,03	50	50	54	54	56	56
eg	Egypt	80081093	1	3	0,01	0,04	51	51	55	55	57	57
mx	Mexico	112336538	1	1	0,01	0,01	49	49	53	53	58	58
cn	China	1344413526	10	14	0,01	0,01	53	53	57	56	59	59
in	India	1210193422	8	9	0,01	0,01	52	52	56	57	60	60
sv	El Salvador	7066403	0	1	0	0,14			36	58	61	61
			969	1561	62,08%

A few interesting facts:

• New Zealand bumps from rank 7 to rank 3, thanks to one new active developer
• Switzerland loses one developer and goes donw to rank 6
• Norway also slightly goes down by losing one developer
• With two more developers, Austria climbs up to rank 8 and overtakes Germany...;-)
• Hungary climbs a little bit by gaining one developer
• Singapore doubles its number of developers from 1 to 2 and bumps from 33 to 27
• One rank up too for Poland that gained one developer
• Down to rank 31 for Lithuania by losing one developer
• Up to rank 32 for Greece with 4 developers instead of 3
• Argentina goes up by havign two more developers (it lost 2 last year)
• Up from 46 to 42 for Colombia by winning one more developer
• One more developer and Russia climps from 49 to 48
• One less for Venezuela that has only one developer left...:-(
• No new country this year. Less movement towards "the universal OS"?
• We have 12 more active Debian developers and 26 more developers overall. Less progression than last year
• The ratio of active developers increases is nearly stable though slightly decreasing

xTuple is in my opinion incredibly well designed; the code is clean and the architecture ahderent to a standardized structure. All this makes working with xTuple software quite a breeze. I wanted to integrate JSCommunicator into the web-based xTuple version. JSCommunicator is a SIP communication tool, so my first step was to create an extension for the SIP account data. Luckily for me, the xTuple development team published an awesome tutorial for writing an xTuple extension. xTuple cleverly uses model based business objects for the various features available. This makes customizing xTuple very straightforward. I used the tutorial mentioned above for writing my extension, but soon noticed my goals were a little different. A SIP account has 3 data fields, these being the SIP URI, the account password and an optional display name. xTuple currently has a business object in the core code for a User Account and it would make a lot more sense to simply add my 3 fields to this existing business object rather than create another business object. The tutorial very clearly shows how to extend a business object with another business object, but not how to extend a business object with only new fields (not a whole new object). Now maybe I m just a whole lot slower than most people, but I had a ridiculously had time figuring this out. Mind you, this is because I m slow, because the xTuple documentation and code is understandable and as self-explanatory as it gets. I think it just takes a bit to get used to. Either way, I thought this just might be useful to others so here is how I went about it. Setup First you ll have to set up your xTuple development environment and fork the xtuple and xtuple-extesions repositories as shown in this handy tutorial. A footnote I d like to add is please verify that your version of Vagrant (and anything else you install) is the one listed in the tutorial. I think I spent like two entire days or more on a wild goose (bug) chase trying to set up my environment when the cause of all the errors was that I somehow installed an older version of Vagrant - 1.5.4 instead of 1.6.3. Please don t make the same mistake I did. Actually if for some reason you get the following error when you try using node: chances are, you have the wrong version. That s what happened to me. The Vagrant Virtual Development Environment automatically installs and configures everything you need, it s ready to go. So if you find yourself installing and updating and apt-gets and etc, you probably did something wrong. Coding So by now we should have the Vagrant Virtual Development Environment set up and the web app up and running and accessible at localhost:8443. So far so good. Disclaimer: You will note that much of this is similar - or rather, nearly identical - to xTuple s tutorial but there are some small but important differences and a few observations I think might be useful. Other Disclaimer: I m describing how I did it, which may or may not be up to snuff . Works for me though. Schema First let s make a schema for the table we will create with the new custom fields. Be sure to create the correct directory stucture, aka /path/to/xtuple-extensions/source/<YOUR EXTENSION NAME>/database/source or in my case /path/to/xtuple-extensions/source/sip_account/database/source, and create the file create_sa_schema.sql, sa is the name of my schema. This file will contain the following lines: Of course, feel free to replace sa with your schema name of choice. All the code described here can be found in my xtuple-extensions fork, on the sip_ext branch. Table We ll create a table containing your custom fields and a link to an existing table - the table for the existing business object you want to extend. If you re wondering why make a whole new table for a few extra fields, here s a good explanation, the case in question is adding fields to the Contact business object. You need to first figure out what table you want to link to. This might not be uber easy. I think the best way to go about it is to look at the ORMs. The xTuple ORMs are a JSON mapping between the SQL tables and the object-oriented world above the database, they re .json files found at path/to/xtuple/node_modules/xtuple/enyo-client/database/orm/models for the core business objects and at path/to/xtuplenyo-client/extensions/source/<EXTENSION NAME>/database/orm/models for exension business objects. I ll give two examples. If you look at contact.json you will see that the Contact business object refers to the table cntct . Look for the type : Contact on the line above, so we know it s the Contact business object. In my case, I wanted to extend the UserAccount and UserAccountRelation business objects, so check out user_account.json. The table listed for UserAccount is xt.usrinfo and the table listed for UserAccountRelation is xt.usrlite. A closer look at the sql files for these tables (usrinfo.sql and usrlite.sql) revealed that usrinfo is in fact a view and usrlite is A light weight table of user information used to avoid punishingly heavy queries on the public usr view . I chose to refer to xt.usrlite - that or I received error messages when trying the other table names. Now I ll make the file /path/to/xtuple-extensions/source/sip_account/database/source/usrlitesip.sql, to create a table with my custom fields plus the link to the urslite table. Don t quote me on this, but I m under the impression that this is the norm for naming the sql file joining tables: the name of the table you are referring to ( usrlite in this case) and your extension s name. Content of usrlitesip.sql: Breaking it down, line 1 creates the table named usrlitesip (no duh), line 2 is for the primary key (self-explanatory). You can then add any columns you like, just be sure to add one that references the table you want to link to. I checked usrlite.sql and saw the primary key is usr_username, be sure to use the primary key of the table you are referencing. You can check what you made by executing the .sql files like so: After which you will see the table with the columns you created if you enter: Now create the file /path/to/xtuple-extensions/source/sip_account/database/source/manifest.js to put the files together and in the right order. It should contain: I think the name has to be the same you named your extension directory as in /path/to/xtuple-extensions/source/<YOUR EXTENSION NAME>. I think the comment can be anything you like and you want your loadOrder to be high so it s the last thing installed (as it s an add on.) So far we are doing exactly what s instructed in the xTuple tutorial. It s repetitive, but I think you can never have too many examples to compare to. In databaseScripts you will list the two .sql files you just created for the schema and the table, plus another file to be made in the same directory named register.sql. I m not sure why you have to make the register.sql or even if you indeed have to. If you leave the file empty, there will be a build error, so put a ; in the register.sql or remove the line register.sql from manifest.js as I think for now we are good without it. Now let s update the database with our new extension: That last command should display a table with a list of extensions; the ones already in xtuple like crm and billing and some others plus your new extension, in this case sip_account . When you run build_app.js you ll probably see a message along the lines of <Extension name> has no client code, not building client code and that s fine because yeah, we haven t worked on the client code yet. ORM Here s where things start getting different. So ORMs link your object to an SQL table. But we DON T want to make a new business object, we want to extend an existing business object, so the ORM we will make will be a little different than the xTuple tutorial. Steve Hackbarth kindly explained this new business object/existing business object ORM concept here. First we ll create the directory /path/to/xtuple-extensions/source/sip_account/database/orm/ext, according to xTuple convention. ORMs for new business objects would be put in /path/to/xtuple-extensions/source/sip_account/database/orm/models. Now we ll create the .json file /path/to/xtuple-extensions/source/sip_account/database/orm/ext/user_account.jscon for our ORM. Once again, don t quote me on this, but I think the name of the file should be the name of the business object you are extending, as is done in the turorial example extending the Contact object. In our case, UserAccount is defined in user_account.json and that s what I named my extension ORM too. Here s what you should place in it: Note the context is my extension name, because the context + nameSpace + type combo has to be unique. We already have a UserAccount and UserAccountRelation object in the XM namespace in the xtuple context in the original user_account.json, now we will have a UserAccount and UserAccountRelation object in the XM namespace in the sip_account conext. What else is important? Note that isExtension is true on lines 7 and 47 and the relations item contains the column of the foreign key we referenced. This is something you might want to verify: column (lines 12 and 52) is the name of the attribute on your table. When we made a reference to the primary key usr_usrname from the xt.usrlite table we named that column usrlitesip_usr_usrname. But the inverse is the attribute name associated with the original sql column in the original ORM. Did I lose you? I had a lot of trouble with this silly thing. In the original ORM that created a new UserAccount business object, the primary key attribute is named username , as can be seen here. That is what should be used for the inverse value. Not the sql column name (usr_username) but the object attribute name (username). I m emphasizing this because I made that mistake and if I can spare you the pain I will. If we rebuild our extension everything should come along nicely, but you won t see any changes just yet in the web app because we haven t created the client code. Client Create the directory /path/to/xtuple-extensions/source/sip_account/client which is where we ll keep all the client code. Extend Workspace View I want the fields I added to show up on the form to create a new User Account, so I need to extend the view for the User Account workspace. I ll start by creating a directory /path/to/xtuple-extensions/source/sip_account/client/views and in it creating a file named workspace.js containing this code: So I m initializing my workspace and creating an array of items to add (append) to view XV.UserAccountWorkspace. The first item is this onyx.GroupboxHeader which is a pretty divider for my new form fields, the kind you find in the web app at Setup > User Accounts, like Overview . I have no idea what other options there are for container other than mainGroup , so let s stick to that. I ll explain content: _sipAccount .loc() in a bit. Next I created three input fields of the XV.InputWidget kind. This also confused me a bit as there are different kinds of input to be used, like dropdowns and checkboxes. The only advice I can give is snoop around the webapp, find an input you like and look up the corresponding workspace.js file to see what was used. What we just did is (should be) enough for the new fields to show up on the User Account form. But before we see things change, we have to package the client. Create the file /path/to/xtuple-extensions/source/sip_account/client/views/package.js. This file is needed to package groups of files and indicates the order the files should be loaded (for more on that, see this). For now, all the file will contain is: You also need to package the views directory containing workspace.js, so create the file Create the file /path/to/xtuple-extensions/source/sip_account/client/package.js and in it show that the directory views and its contents must be part of the higher level package: I like to think of it as a box full of smaller boxes. This will sound terrible, but apparently you also need to create the file /path/to/xtuple-extensions/source/sip_account/client/core.js containing this line: I don t know why. As soon as I find out I ll be sure to inform you. As we ve added a file to the client directory, be sure to update /path/to/xtuple-extensions/source/sip_account/client/package.js so it included the new file: Translations Remember _sipAccount .loc() in our workspace.js file? xTuple has great internationalization support and it s easy to use. Just create the directory and file /path/to/xtuple-extensions/source/sip_account/client/en/strings.js and in it put key-value pairs for labels and their translation, like this: So far I included all the labels I used in my Sip Account form. If you write the wrong label (key) or forget to include a corresponding key-value pair in strings.js, xTuple will simply name your lable _lableName , underscore and all. Now build your extension and start up the server: If the server is already running, just stop it and restart it to reflect your changes. Now if you go to Setup > User Accounts and click the + button, you should see a nice little addition to the form with a Sip Account divider and three new fields. Nice, eh? Extend Parameters Currently you can search your User Accounts list using any of the User Account fields. It would be nice to be able to search with the Sip account fields we added as well. To do that, let s create the directory /path/to/xtuple-extensions/source/sip_account/client/widgets and there create the file parameter.js to extend XV.UserAccountListParameters. One again, you ll have to look this up. In the xTuple code you ll find the application s parameter.js in /path/to/xtuple/enyo-client/application/source/widgets. Search for the business object you are extending (for example, XV.UserAccount) and look for some combination of the business object name and Parameters . If there s more than one, try different ones. Not a very refined method, but it worked for me. Here s the content of our parameter.js: Node that I didn t include a search field for the password attribute for obvious reasons. Now once again, we package this new code addition by creating a /path/to/xtuple-extensions/source/sip_account/client/widgets/package.js file: We also have to update /path/to/xtuple-extensions/source/sip_account/client/package.js: Rebuild the extension (and restart the server) and go to Setup > User Accounts. Press the magnifying glass button on the upper left side of the screen and you ll see many options for filtering the User Accounts, among them the SIP Uri and Display Name. Extend List View You might want your new fields to show up on the list of User Accounts. There s a bit of an issue here because unlike what we did in workspace.js and parameter.js, we can t append new things to the list of UserAccounts with the funciton XV.appendExtension(args). First I tried overwriting the original UserAccountList, which works but it s far from ideal as this could result in a loss of data from the core implementation. After some discussion with the xTuple dev community, now there s a better alternative: Create the file /path/to/xtuple-extensions/source/sip_account/client/views/list.js and add the following: To understand what I m doing, check out the XV.UserAccountList implementation in /path/to/xtuple/enyo-client/application/source/views/list.js the entire highlighted part. What we are doing is extending XV.UserAccountList through prototype-chaining ; this is how inheritance works with Enyo. In line 1 we create a prototype and in line 4 we inherit the features including original components array which the list is based on. We then create an additional component immitating the setup shown in XV.UserAccountList: An XV.ListColumn containing an XV.ListAttr, which should be placed in the XV.ListItem components array as is done with the existing columns (refer to implementation). Components can or should (?) have names which are used to access said components. You d refer to a specific component by the this.$.componentName hash. The components in XV.UserAccountList don t have names, so Enyo automatically names them (apparently) based on the kind name, for example something of the kind ListItem is named listItem. I found this at random after a lot of trial and error and it s not a bullet proof solution. Can be bettered. It s strange because if you encapsulate that code with as is done with parameter.js and workspace.js (and in the xTuple tutorial you are supposed to do that with a new business object), it doesn t work. I have no idea why. This might be wrong or against xTuple coding norms; I will find out and update this post ASAP. But it does work this way. * shrugs * That said, as we ve created the list.js file, we need to ad it to our package by editing /path/to/xtuple-extensions/source/sip_account/client/views/package.js: That s all. Rebuild the app and restart your server and when you select Setup > User Accounts in the web app you should see the Sip URI displayed on the User Accounts that have the Sip Account data. Add a new User Account to try this out.

$ sudo apt-get install -y leiningen
[...]
$ lein new scratch
[...]
$ cd scratch
$ lein repl
Downloading: org/clojure/clojure/1.3.0/clojure-1.3.0.pom from repository central at http://repo1.maven.org/maven2
Transferring 5K from central
Downloading: org/sonatype/oss/oss-parent/5/oss-parent-5.pom from repository central at http://repo1.maven.org/maven2
Transferring 4K from central
Downloading: org/clojure/clojure/1.3.0/clojure-1.3.0.jar from repository central at http://repo1.maven.org/maven2
Transferring 3311K from central
[...]

Wait what? lein downloads some random JARs from a website over HTTP¹, with, as far as far I can tell, no verification that what I m asking for is what I m getting (has nobody ever heard of Man-in-the-Middle attacks in Maven land?). It downloads a .sha1 file to (presumably) do integrity checking, but that s no safety net if I can serve you a dodgy .jar, I can serve you an equally-dodgy .sha1 file, too (also, SHA256 is where all the cool kids are at these days). Finally, jarsigner tells me that there s no signature on the .jar itself, either. It gets better, though. The repo1.maven.org site is served by the fastly.net² pseudo-CDN³, which adds another set of points in the chain which can be subverted to hijack and spoof traffic. More routers, more DNS zones, and more servers. I ve seen Debian take a kicking more than once because packages aren t individually signed, or because packages aren t served over HTTPS. But at least Debian s packages can be verified by chaining to a signature made by a well-known, widely-distributed key, signed by two Debian Developers with very well-connected keys. This repository, on the other hand oy gevalt. There are OpenPGP (GPG) signatures available for each package (tack .asc onto the end of the .jar URL), but no attempt was made to download the signatures for the .jar I downloaded. Even if the signature was downloaded and checked, there s no way for me (or anyone) to trust the signature the signature was made by a key that s signed by one other key, which itself has no signatures. If I were an attacker, it wouldn t be hard for me to replace that key chain with one of my own devising. Even ignoring everyone living behind a government- or company-run intercepting proxy, and everyone using public wifi, it s pretty well common knowledge by now (thanks to Edward Snowden) that playing silly-buggers with Internet traffic isn t hard to do, and there s no shortage of evidence that it is, in fact, done on a routine basis by all manner of people. Serving up executable code to a large number of people, in that threat environment, with no way for them to have any reasonable assurance that code is trustworthy, is very disappointing. Please, for the good of the Internet, improve your act, Maven. Putting HTTPS on your distribution would be a bare minimum. There are attacks on SSL, sure, but they re a lot harder to pull off than sitting on public wifi hijacking TCP connections. Far better would be to start mandating signatures, requiring signature checks to pass, and having all signatures chain to a well-known, widely-trusted, and properly secured trust root. Signing all keys that are allowed to upload to maven.org with a maven.org distribution root key (itself kept in hardware and only used offline), and then verifying that all signatures chain to that key, wouldn t be insanely difficult, and would greatly improve the security of the software supply chain. Sure, it wouldn t be perfect, but don t make the perfect the enemy of the good. Cost-effective improvements are possible here. Yes, security is hard. But you don t get to ignore it just because of that, when you re creating an attractive nuisance for anyone who wants to own up a whole passel of machines by slipping some dodgy code into a widely-used package.

---

My primary usecase is to prevent testsuites and build systems from contacting internet-based services. This, at the very least, introduces an element of non-determinism and malicious code at worst. I use Alberto Bertogli's libfiu for this, specifically the fiu-run utility which part of the fiu-utils package on Debian and Ubuntu. Here's a contrived example, where I prevent Curl from talking to the internet:

$ fiu-run -x -c 'enable name=posix/io/net/connect' curl google.com
curl: (6) Couldn't resolve host 'google.com'

... and here's an example of it detecting two possibly internet-connecting tests:

$ fiu-run -x -c 'enable name=posix/io/net/connect' ./manage.py text
[..]
----------------------------------------------------------------------
Ran 892 tests in 2.495s
FAILED (errors=2)
Destroying test database for alias 'default'...

---

Note that libfiu inherits all the drawbacks of LD_PRELOAD; in particular, we cannot limit the child process that calls setuid binaries such as /bin/ping:

$ fiu-run -x -c 'enable name=posix/io/net/connect' ping google.com
PING google.com (173.194.41.65) 56(84) bytes of data.
64 bytes from lhr08s01.1e100.net (17.194.41.65): icmp_req=1 ttl=57 time=21.7 ms
64 bytes from lhr08s01.1e100.net (17.194.41.65): icmp_req=2 ttl=57 time=18.9 ms
[..]

Whilst it would certainly be more robust and flexible to use iptables such as allowing localhost and other local socket connections but disabling all others I gravitate towards this entirely userspace solution as it requires no setup and I can quickly modify it to block other calls on an ad-hoc basis. The list of other "modules" libfiu supports is viewable here.

I've recently packaged the sunxi tools for Debian. These are a set of tools produce by the Linux Sunxi project for working with the Allwinner "sunxi" family of processors. See the package page for details. Thanks to Steve McIntyre for sponsoring the initial upload. The most interesting component of the package are the tools for working with the Allwinner processors' FEL mode. This is a low-level processor mode which implements a simple USB protocol allowing for initial programming of the device and recovery which can be entered on boot (usually be pressing a special 'FEL button' somewhere on the device). It is thanks to FEL mode that most sunxi based devices are pretty much unbrickable. The most common use of FEL is to boot over USB. In the Debian package the fel and usb-boot tools are named sunxi-fel and sunxi-usb-boot respectively but otherwise can be used in the normal way described on the sunxi wiki pages. One enhancement I made to the Debian version of usb-boot is to integrate with the u-boot packages to allow you to easily FEL boot any sunxi platform supported by the Debian packaged version of u-boot (currently only Cubietruck, more to come I hope). To make this work we take advantage of Multiarch to install the armhf version of u-boot (unless your host is already armhf of course, in which case just install the u-boot package):

# dpkg --add-architecture armhf
# apt-get update
# apt-get install u-boot:armhf
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  u-boot:armhf
0 upgraded, 1 newly installed, 0 to remove and 1960 not upgraded.
Need to get 0 B/546 kB of archives.
After this operation, 8,676 kB of additional disk space will be used.
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
Selecting previously unselected package u-boot:armhf.
(Reading database ... 309234 files and directories currently installed.)
Preparing to unpack .../u-boot_2014.04+dfsg1-1_armhf.deb ...
Unpacking u-boot:armhf (2014.04+dfsg1-1) ...
Setting up u-boot:armhf (2014.04+dfsg1-1) ...

With that done FEL booting a cubietruck is as simple as starting the board in FEL mode (by holding down the FEL button when powering on) and then:

# sunxi-usb-boot Cubietruck -
fel write 0x2000 /usr/lib/u-boot/Cubietruck_FEL/u-boot-spl.bin
fel exe 0x2000
fel write 0x4a000000 /usr/lib/u-boot/Cubietruck_FEL/u-boot.bin
fel write 0x41000000 /usr/share/sunxi-tools//ramboot.scr
fel exe 0x4a000000

Which should result in something like this on the Cubietruck's serial console:

U-Boot SPL 2014.04 (Jun 16 2014 - 05:31:24)
DRAM: 2048 MiB
U-Boot 2014.04 (Jun 16 2014 - 05:30:47) Allwinner Technology
CPU:   Allwinner A20 (SUN7I)
DRAM:  2 GiB
MMC:   SUNXI SD/MMC: 0
In:    serial
Out:   serial
Err:   serial
SCSI:  SUNXI SCSI INIT
Target spinup took 0 ms.
AHCI 0001.0100 32 slots 1 ports 3 Gbps 0x1 impl SATA mode
flags: ncq stag pm led clo only pmp pio slum part ccc apst 
Net:   dwmac.1c50000
Hit any key to stop autoboot:  0 
sun7i# 

As more platforms become supported by the u-boot packages you should be able to find them in /usr/lib/u-boot/*_FEL. There is one minor inconvenience which is the need to run sunxi-usb-boot as root in order to access the FEL USB device. This is easily resolved by creating /etc/udev/rules.d/sunxi-fel.rules containing either:

SUBSYSTEMS=="usb", ATTR idVendor =="1f3a", ATTR idProduct =="efe8", OWNER="myuser"

or

SUBSYSTEMS=="usb", ATTR idVendor =="1f3a", ATTR idProduct =="efe8", GROUP="mygroup"

To enable access for myuser or mygroup respectively. Once you have created the rules file then to enable:

# udevadm control --reload-rules

As well as the FEL mode tools the packages also contain a FEX (de)compiler. FEX is Allwinner's own hardware description language and is used with their Android SDK kernels and the fork of that kernel maintained by the linux-sunxi project. Debian's kernels follow mainline and therefore use Device Tree.

Recently I was updating my dotfiles, because I wanted to ensure that media-players were "always on top", when launched, as this suits the way I work. For many years I've used devilspie to script the placement of new windows, and once I googled a recipe I managed to achieve my aim. However during the course of my googling I discovered that devilspie is unmaintained, and has been replaced by something using Lua - something I like. I'm surprised I hadn't realized that the project was dead, although I've always hated the configuration syntax it is something that I've used on a constant basis since I found it. Unfortunately the replacement, despite using Lua, and despite being functional just didn't seem to gell with me. So I figured "How hard could it be?". In the past I've written softare which iterated over all (visible) windows, and obviously I'm no stranger to writing Lua bindings. However I did run into a snag. My initial implementation did two things:

• Find all windows.
• For each window invoke a lua script-file.

This worked. This worked well. This worked too well. The problem I ran into was that if I wrote something like "Move window 'emacs' to desktop 2" that action would be applied, over and over again. So if I launched emacs, and then manually moved the window to desktop3 it would jump back! In short I needed to add a "stop()" function, which would cause further actions against a given window to cease. (By keeping a linked list of windows-to-ignore, and avoiding processing them.) The code did work, but it felt wrong to have an ever-growing linked-list of processed windows. So I figured I'd look at the alternative - the original devilspie used libwnck to operate. That library allows you to nominate a callback to be executed every time a new window is created. If you apply your magic only on a window-create event - well you don't need to bother caching prior-windows. So in conclusion :

• devilspie is dead.
• there is a devils2pie2 project.
• I wrote kpie

I think my code is better than devilspie2 because it is smaller, simpler, and does things more neatly - for example instead of a function to get geometry and another to set it, I use one. (e.g. "xy()" returns the position of a window, but xy(3,3) sets it.). kpie also allows you to run as a one-off job, and using the simple primitives I wrote a file to dump your windows, and their size/placement, which looks like this:

shelob ~/git/kpie $ ./kpie --single ./samples/dump.lua
-- Screen width : 1920
-- Screen height: 1080
..
if ( ( window_title() == "Buddy List" ) and
     ( window_class() == "Pidgin" ) and
     ( window_application() == "Pidgin" ) ) then
     xy(1536,24 )
     size(384,1032 )
     workspace(2)
end
if ( ( window_title() == "feeds" ) and
     ( window_class() == "Pidgin" ) and
     ( window_application() == "Pidgin" ) ) then
     xy(1,24 )
     size(1536,1032 )
     workspace(2)
end
..

As you can see that has dumped all my windows, along with their current state. This allows a simple starting-point - Configure your windows the way you want them, then dump them to a script file. Re-run that script file and your windows will be set back the way they were! (Obviously there might be tweaks required.) I used that starting-point to define a simple recipe for configuring pidgin, which is more flexible than what I ever had with pidgin, and suits my tastes. Bug-reports welcome.

Modern desktop environments like GNOME and KDE involving a lot of mousing around and I much prefer using the keyboard where I can. This is why I switched to the Ion tiling window manager back when I interned at Net Integration Technologies and kept using it until I noticed it had been removed from Debian. After experimenting with awesome for 2 years and briefly considering xmonad , I finally found a replacement I like in i3. Here is how I customized it and made it play nice with the GNOME and KDE applications I use every day.

Startup script As soon as I log into my desktop, my startup script starts a few programs, including:

• gnome-settings-daemon: makes GTK applications look nice by applying my preferred theme amongst other things
• gnome-keyring-daemon: remembers ssh public keys for the duration of my session
• gnome-screensaver: locks the screen when I'm not around
• nm-applet: handles wifi and VPN connections
• git-annex: keeps my folders synchronized between machines

Because of a bug in gnome-settings-daemon which makes the mouse cursor disappear as soon as gnome-settings-daemon is started, I had to run the following to disable the offending gnome-settings-daemon plugin:

dconf write /org/gnome/settings-daemon/plugins/cursor/active false

Screensaver In addition, gnome-screensaver didn't automatically lock my screen, so I installed xautolock and added it to my startup script:

xautolock -time 30 -locker "gnome-screensaver-command --lock" &

to lock the screen using gnome-screensaver after 30 minutes of inactivity. I can also trigger it manually using the following shortcut defined in my ~/.i3/config:

bindsym Ctrl+Mod1+l exec xautolock -locknow

Keyboard shortcuts While keyboard shortcuts can be configured in GNOME, they don't work within i3, so I added a few more bindings to my ~/.i3/config:

# volume control
bindsym XF86AudioLowerVolume exec /usr/bin/pactl set-sink-volume @DEFAULT_SINK@ -- '-5%'
bindsym XF86AudioRaiseVolume exec /usr/bin/pactl set-sink-volume @DEFAULT_SINK@ -- '+5%'
# brightness control
bindsym XF86MonBrightnessDown exec xbacklight -steps 1 -time 0 -dec 5
bindsym XF86MonBrightnessUp exec xbacklight -steps 1 -time 0 -inc 5
bindsym XF86AudioMute exec /usr/bin/pactl set-sink-mute @DEFAULT_SINK@ toggle
# show battery stats
bindsym XF86Battery exec gnome-power-statistics

to make volume control, screen brightness and battery status buttons work as expected on my laptop. These bindings require the following packages:

• pulseaudio-utils
• xbacklight
• gnome-power-manager

Keyboard layout switcher Another thing that used to work with GNOME and had to re-create in i3 is the ability to quickly toggle between two keyboard layouts using the keyboard. To make it work, I wrote a simple shell script and assigned a keyboard shortcut to it in ~/.i3/config:

bindsym $mod+u exec /home/francois/bin/toggle-xkbmap

Suspend script Since I run lots of things in the background, I have set my laptop to avoid suspending when the lid is closed by putting the following in /etc/systemd/login.conf:

HandleLidSwitch=lock

Instead, when I want to suspend to ram, I use the following keyboard shortcut:

bindsym Ctrl+Mod1+s exec /home/francois/bin/s2ram

which executes a custom suspend script to clear the clipboards (using xsel), flush writes to disk and lock the screen before going to sleep. To avoid having to type my sudo password every time pm-suspend is invoked, I added the following line to /etc/sudoers:

francois  ALL=(ALL)  NOPASSWD:  /usr/sbin/pm-suspend

Window and workspace placement hacks While tiling window managers promise to manage windows for you so that you can focus on more important things, you will most likely want to customize window placement to fit your needs better.

Working around misbehaving applications A few applications make too many assumptions about window placement and are just plain broken in tiling mode. Here's how to automatically switch them to floating mode:

for_window [class="VidyoDesktop"] floating enable

You can get the Xorg class of the offending application by running this command:

xprop   grep WM_CLASS

before clicking on the window.

Keeping IM windows on the first workspace I run Pidgin on my first workspace and I have the following rule to keep any new window that pops up (e.g. in response to a new incoming message) on the same workspace:

assign [class="Pidgin"] 1

Automatically moving workspaces when docking Here's a neat configuration blurb which automatically moves my workspaces (and their contents) from the laptop screen (eDP1) to the external monitor (DP2) when I dock my laptop:

# bind workspaces to the right monitors
workspace 1 output DP2
workspace 2 output DP2
workspace 3 output DP2
workspace 4 output DP2
workspace 5 output DP2
workspace 6 output eDP1

You can get these output names by running:

xrandr --display :0   grep " connected"

Finally, because X sometimes fail to detect my external monitor when docking/undocking, I also wrote a script to set the displays properly and bound it to the appropriate key on my laptop:

bindsym XF86Display exec /home/francois/bin/external-monitor

Today we finished the talk selection process. We are very grateful to everyone who decided to submit talks and events for DebConf14. If you have submitted an event, please check your email :). If you have not received any confirmation regarding your talk status, please contact us on talks@debconf.org During the selection process, we bore in mind the number of talk slots during the conference, as well as maintaining a balance among the different submitted topics. We are pleased to announce that we have received a total of 115 events, of which 80 have been approved (69%). Approval means your event will be scheduled during the conference and you will have video coverage. The list of approved talks can be found on the following link: https://summit.debconf.org/debconf14/all/ If you got an email telling your talk have being approved, and your talk is not listed, don t panic. Check the status on summit, and make sure to select a track, if you have some track suggestions please mail us and tell us about it. This year, we expect to also have a sort of unconference schedule. This will take place during the designated hacking time . During that time the talks rooms will be empty, and ad hoc meetings can be scheduled on-site while we are in the Conference. The method for booking a room for your ad hoc meeting will be decided and announced later, but is expected to be flexible (i.e: open scheduling board / 1 day or less in advance booking), Please don t abuse the system: bear in mind the space will be limited, and only book your event if you gather enough people to work on your idea. Please make sure to read the email regarding your talk. :) and prepare yourself. Time is ticking and we will be happy to meet you in Portland.

Reworking Intel Glamor The original Intel driver Glamor support was based on the notion that it would be better to have the Intel driver capture any fall backs and try to make them faster than Glamor could do internally. Now that Glamor has reasonably complete acceleration, and its fall backs aren t terrible, this isn t as useful as it once was, and because this uses Glamor in a weird way, we re making the Glamor code harder to maintain. Fixing the Intel driver to not use Glamor in this way took a bit of effort; the UXA support is all tied into the overall operation of the driver. Separating out UXA functions The first task was to just identify which functions were UXA-specific by adding _uxa to their names. A couple dozen sed runs and now a bunch of the driver is looking better. Next, a pile of UXA-specific functions were actually inside the non-UXA parts of the code. Those got moved out, and a new intel_uxa.h file was created to hold all of the definitions. Finally, a few non UXA-specific functions were actually in the uxa files; those got moved over to the generic code. Removing the Glamor paths in UXA Each one of the UXA functions had a little piece of code at the top like:

if (uxa_screen->info->flags & UXA_USE_GLAMOR)  
    int ok = 0;
    if (uxa_prepare_access(pDrawable, UXA_GLAMOR_ACCESS_RW))  
        ok = glamor_fill_spans_nf(pDrawable,
                      pGC, n, ppt, pwidth, fSorted);
        uxa_finish_access(pDrawable, UXA_GLAMOR_ACCESS_RW);
     
    if (!ok)
        goto fallback;
    return;
 

Pulling those out shrank the UXA code by quite a bit. Selecting Acceleration (or not) The intel driver only supported UXA before; Glamor was really just a slightly different mode for UXA. I switched the driver from using a bit in the UXA flags to having an accel variable which could be one of three options:

• ACCEL_GLAMOR.
• ACCEL_UXA.
• ACCEL_NONE

I added ACCEL_NONE to give us a dumb frame buffer mode. That actually supports DRI3 so that we can bring up Mesa and run it under X before we have any acceleration code ready; avoiding a dependency loop when doing new hardware. All that it requires is a kernel that offers mode setting and buffer allocation. Initializing Glamor With UXA no longer supporting Glamor, it was time to plug the Glamor support into the top of the driver. That meant changing a bunch of the entry points to select appropriate Glamor or UXA functionality, instead of just calling into UXA. So, now we ve got lots of places that look like:

        switch (intel->accel)  
#if USE_GLAMOR
        case ACCEL_GLAMOR:
                if (!intel_glamor_create_screen_resources(screen))
                        return FALSE;
                break;
#endif
#if USE_UXA
        case ACCEL_UXA:
                if (!intel_uxa_create_screen_resources(screen))
                        return FALSE;
        break;
#endif
        case ACCEL_NONE:
                if (!intel_none_create_screen_resources(screen))
                        return FALSE;
                break;
         

Using a switch means that we can easily elide code that isn t wanted in a particular build. Of course accel is an enum, so places which are missing one of the required paths will cause a compiler warning. It s not all perfectly clean yet; there are piles of UXA-only paths still. Making It Build Without UXA The final trick was to make the driver build without UXA turned on; that took several iterations before I had the symbols sorted out appropriately. I built the driver with various acceleration options and then tried to count the lines of source code. What I did was just list the source files named in the driver binary itself. This skips all of the header files and the render program source code, and ignores the fact that there are a bunch of #ifdef s in the uxa directory selecting between uxa, glamor and none.

    Accel                    Lines          Size(B)
    -----------             ------          -------
    none                      7143            73039
    glamor                    7397            76540
    uxa                      25979           283777
    sna                     118832          1303904
    none legacy              14449           152480
    glamor legacy            14703           156125
    uxa legacy               33285           350685
    sna legacy              126138          1395231

The legacy addition supports i810-class hardware, which is needed for a complete driver. Along The Way, Enable Tiling for the Front Buffer While hacking the code, I discovered that the initial frame buffer allocated for the screen was created without tiling because a few parameters that depend on the GTT size were not initialized until after that frame buffer was allocated. I haven t analyzed what effect this has on performance. Page Flipping and Resize Page flipping (or just flipping) means switching the entire display from one frame buffer to another. It s generally the fastest way of updating the screen as you don t have to copy any bits. The trick with flipping is that a client hands you a random pixmap and you need to stuff that into the KMS API. With UXA, that s pretty easy as all pixmaps are managed through the UXA API which knows which underlying kernel BO is tied with each pixmap. Using Glamor, only the underlying GL driver knows the mapping. Fortunately (?), we have the EGL Image extension, which lets us take a random GL texture and turn it into a file descriptor for a DMA-BUF kernel object. So, we have this cute little dance:

fd = glamor_fd_from_pixmap(screen,
                               pixmap,
                               &stride,
                               &size);
bo = drm_intel_bo_gem_create_from_prime(intel->bufmgr, fd, size);
    close(fd);
    intel_glamor_get_pixmap(pixmap)->bo = bo;

That last bit remembers the bo in some local memory so we don t have to do this more than once for each pixmap. glamorfdfrompixmap ends up calling eglCreateImageKHR followed by gbmbo_import and then a kernel ioctl to convert a prime handle into an fd. It s all quite round-about, but it does seem to work just fine. After I d gotten Glamor mostly working, I tried a few OpenGL applications and discovered flipping wasn t working. That turned out to have an unexpected consequence all full-screen applications would run flat-out, and not be limited to frame rate. Present recovers from a failed flip queue operation by immediately performing a CopyArea; not waiting for vblank. This needs to get fixed in Present by having it re-queued the CopyArea for the right time. What I did in the intel driver was to add a bunch more checks for tiling mode, pixmap stride and other things to catch pixmaps that were going to fail before the operation was queued and forcing them to fall back to CopyArea at the right time. The second adventure was with XRandR. Glamor has an API to fix up the screen pixmap for a new frame buffer, but that pulls the size of the frame buffer out of the pixmap instead of out of the screen. XRandR leaves the pixmap size set to the old screen size during this call; fixing that just meant getting the pixmap size set correctly before calling into glamor. I think glamor should get fixed to use the screen size rather than the pixmap size. Painting Root before Mode set The X server has generally done initialization in one order:

1. Create root pixmap
2. Set video modes
3. Paint root window

Recently, we ve added a -background none option to the X server which causes it to set the root window background to none and have the driver fill in that pixmap with whatever contents were on the screen before the X server started. In a pre-Glamor world, that was done by hacking the video driver to copy the frame buffer console contents to the root pixmap as it was created. The trouble here is that the root pixmap is created long before the upper layers of the X server are ready for drawing, so you can t use the core rendering paths. Instead, UXA had kludges to call directly into the acceleration functions. What we really want though is to change the order of operations:

1. Create root pixmap
2. Paint root window
3. Set video mode

That way, the normal root window painting operation will take care of getting the image ready before that pixmap is ever used for scanout. I can use regular core X rendering to get the original frame buffer contents into the root window, and even if we re not using -background none and are instead painting the root with some other pattern (like the root weave), I get that presented without an intervening black flash. That turned out to be really easy just delay the call to I830EnterVT (which sets the modes) until the server is actually running. That required one additional kludge I needed to tell the DIX level RandR functions about the new modes; the mode setting operation used during server init doesn t call up into RandR as RandR lists the current configuration after the screen has been initialized, which is when the modes used to be set. Calling xf86RandR12CreateScreenResources does the trick nicely. Getting the root window bits from fbcon, setting video modes and updating the RandR/Xinerama DIX info is now all done from the BlockHandler the first time it is called. Performance I ran the current glamor version of the intel driver with the master branch of the X server and there were not any huge differences since my last Glamor performance evaluation aside from GetImage. The reason is that UXA/Glamor never called Glamor s image functions, and the UXA GetImage is pretty slow. Using Mesa s image download turns out to have a huge performance benefit:

1. UXA/Glamor from April
2. Glamor from today
       1                 2                 Operation
------------   -------------------------   -------------------------
     50700.0        56300.0 (     1.110)   ShmGetImage 10x10 square 
     12600.0        26200.0 (     2.079)   ShmGetImage 100x100 square 
      1840.0         4250.0 (     2.310)   ShmGetImage 500x500 square 
      3290.0          202.0 (     0.061)   ShmGetImage XY 10x10 square 
        36.5          170.0 (     4.658)   ShmGetImage XY 100x100 square 
         1.5           56.4 (    37.600)   ShmGetImage XY 500x500 square 
     49800.0        50200.0 (     1.008)   GetImage 10x10 square 
      5690.0        19300.0 (     3.392)   GetImage 100x100 square 
       609.0         1360.0 (     2.233)   GetImage 500x500 square 
      3100.0          206.0 (     0.066)   GetImage XY 10x10 square 
        36.4          183.0 (     5.027)   GetImage XY 100x100 square 
         1.5           55.4 (    36.933)   GetImage XY 500x500 square

Running UXA from today the situation is even more dire; I suspect that enabling tiling has made CPU reads through the GTT even worse than before?

1: UXA today
2: Glamor today
       1                 2                 Operation
------------   -------------------------   -------------------------
     43200.0        56300.0 (     1.303)   ShmGetImage 10x10 square 
      2600.0        26200.0 (    10.077)   ShmGetImage 100x100 square 
       130.0         4250.0 (    32.692)   ShmGetImage 500x500 square 
      3260.0          202.0 (     0.062)   ShmGetImage XY 10x10 square 
        36.7          170.0 (     4.632)   ShmGetImage XY 100x100 square 
         1.5           56.4 (    37.600)   ShmGetImage XY 500x500 square 
     41700.0        50200.0 (     1.204)   GetImage 10x10 square 
      2520.0        19300.0 (     7.659)   GetImage 100x100 square 
       125.0         1360.0 (    10.880)   GetImage 500x500 square 
      3150.0          206.0 (     0.065)   GetImage XY 10x10 square 
        36.1          183.0 (     5.069)   GetImage XY 100x100 square 
         1.5           55.4 (    36.933)   GetImage XY 500x500 square

Of course, this is all just x11perf, which doesn t represent real applications at all well. However, there are applications which end up doing more GetImage than would seem reasonable, and it s nice to have this kind of speed up. Status I m running this on my crash box to get some performance numbers and continue testing it. I ll switch my desktop over when I feel a bit more comfortable with how it s working. But, I think it s feature complete at this point. Where s the Code As usual, the code is in my personal repository. It s on the glamor branch.

git://people.freedesktop.org/~keithp/xf86-video-intel  glamor

Some days ago I decided to upgrade my laptop from stable to testing. I had tried Jessie since several months, in my husband s laptop, but that was a fresh install, and a not-so-old laptop, and we have not much software installed there. In my netbook (Compaq Mini 110c), with stable, I already had installed Pumpa, Dianara and how-can-i-help from testing, and since the freeze is coming, I thought that I could full-upgrade and use Jessie from now on, and report my issues and help to diagnose or fix them, if possible, before the freeze. I keep Debian stable at work for my desktop and servers (well, some of them are still in oldstable, thanks LTS team!!), and I have testing in a laptop that I use as clonezilla/drbl server (but I had issues, next week I ll put some time on them and I ll write here my findings, and report bugs, if any). So! let s go. Here I write my experience and the issues that I found (very few! and not sure if they are bugs or configuration problems or what, I ll keep an eye on them). The upgrade I pointed my /etc/apt/sources.list to jessie, then apt-get update, then apt-get dist-upgrade. (With the servers I am much more careful, read the release notes, upgrade guides and so, or directly I go for a fresh install, but with my laptop, I am too lazy). I went to bed (wow, risky LArjona!) and when I got up for going to work, the laptop was waiting for me to accept to block root from ssh access, or restart some services, and so. Ok! the upgrade resumed but I have to go to work and I wanted my laptop! Since all the packages were already downloaded, I closed the lid (double risky LArjona!) unplugged it, put everything in my bag, and catched the bus in time :) At the bus, I opened again the lid of my laptop (crossing fingers!) and perfect, the laptop had suspended and returned back to life, and the upgrade just resumed with no problem. Wow! I love you Debian! After 15 minutes, I had to suspend again, since the bus arrived and I had to take the metro. In the metro, the upgrade resumed, and finished. I shutdown my laptop and arrive to work. Testing testing :) In a break for lunch, I opened my brand new laptop (the hardware is the same, but the software totally renewed, so it s brand new for me). I have to say that use xfce, with some GNOME/GTK apps installed (gedit, cheese, evince, XChat ) and some others that use Qt or are part of the KDE project (Okular, Kile, QtLinguist, Pumpa, Dianara). I don t know/care too much about desktops and tweaking my desktop: I just put the terminal and gedit in black background, Debian wallpaper is enough dark for me so ok, put the font size a bit smaller to better use my low-vertical-resolution, and that s all, I only go to configure something else if there s something that really annoys me. My laptop booted correctly and a nice, more modern LightDM was greeting me. I logged in and everything worked ok, except some issues that follow. Network Manager and WPA2-enterprise wireless connections I had to reconfigure some wireless connections in Network Manager. At the University we use WPA2-enterprise, TTLS + PAP. I had stored my username and password in the connection, and network manager seemed to remember the username but not the password. No problem, I said, and I wrote it when it asked, but the Save or OK button was greyed out. I could not click it. Then I went to edit the connections, and more or less the same, it seems that I could edit, but not save the (new) configuration. Finally, I removed the wireless connection and created it again, and everything worked as a charm. This, I had to do it with the two wireless in my University (both of them are WPA2-enterprise TTLS + PAP). At home, I have WPA2 personal, and I had no issues, everything worked ok. This problem is not appearing in a fresh install, since there are no old configs to keep. Adblock Plus not working any more I opened Iceweasel and I began to see ads in the webpages that I visited. What? I checked and Adblock plus was installed and activated I reinstalled the package xul-ext-adblock-plus and it worked again. Strange display in programs based on Qt When I opened Pumpa I noticed that the edges of the windows where too rough, as if it was not using a desktop theme. I asked to a friend that uses Plasma and he suggested to install qt4-qtconfig, and then, select a theme for my Qt apps. It worked like a charm, but I find strange that I didn t need it before in stable. Maybe the default xfce configuration from stable is setting a theme, and the new one is not setting it, and so, the Qt apps are left barefoot .
With qtconfig I chose a GTK+ Style GUI for my Qt apps and then, they looked similar to what I had in stable (frankly, I cannot say if it was similar or exactly the same , but I didn t find them strange as before, so I m fine). Strange display in programs from GNOME Well, this is not a Jessie problem, it s just that some programs adopted the new GNOME appearance, and since I m on xfce, not on GNOME, they look a bit strange (no menus integration, and so). I am not sure that I can run GNOME (fallback, classic?) in my 1 GB RAM laptop, I have to investigate if I can tweak it to use less memory, or what. I m not very tied to xfce, and in fact it does not look so light (well, on top of it, I don t run light programs, I run Iceweasel, Icedove, Libreoffice, and some others). At work I use GNOME in my desktop, but with GNOME shell, not the fallback or classic modes, so I m thinking about giving a chance to MATE or second chance to LXDE. We ll see. Issues when opening the lid (waking up from suspend) This is the most strange thing I found in the migration, and the most dangerous one, I think. As I said before, I don t tweak too much my desktop, if it works with the default configuration. I m not sure that I know the differences between suspend, hibernate, hard disks disconnections and so. When I was in stable, and I closed the lid of my laptop, it just shutdown the screen, then I heard something like the system going to suspend or whatever, and after some seconds, the harddisk and fans stop, the wireless led turns off, and the power led begins to blink. Ok. When I open the lid, then it was waking up itself (the power led stayed on, the wireless led turns on, and when I tap the touchpad or type anything, the screen was coming, with the xscreensaver asking for my password). Just sometimes, when the screen was turning on, I could see my desktop for less than a second, before xscreensaver turns the background black and asks for the password. Now since I migrated to Jessie, I m experiencing a different behavior. When I close the lid, the laptop behaves the same. When I open the lid, the laptop behaves the same, but when I type or tap the touchpad and xscreensaver comes to ask the password, before than I can type it, the laptop just suspends again (or hibernates, I m not sure), and I have to press the power button in order to bring it back to life (then I see the xscreensaver again asking for the password, I type it, and my desktop is there, the same as I left it when I closed the lid). Strange, isn t it? I have tried to suspend my laptop directly from the menu, and it comes to the same state in which I have to press the power button in order to bring it back to life, but then, no xscreensaver password is required (which is double strange, IMHO). Things I miss in Jessie Well, until now, the only thing I miss in Jessie is the software center. I rarely use it (I love apt) but I think it makes a good job in easing the installation of programs in Debian for people coming from other operative systems (specially after smartphones and their copied software stores became popular). I hope the maintainer can upload a new version before the freeze, and so, it enters in the release. I ll try to contact him. Update 2014/07/20: Julian Andres Klode, maintainer of software-center, just replied (see his comment below) and pointed to GNOME Software (gnome-packagekit) as alternative. I just installed and it looks neat and nice. I m very happy! TODO I have a Debian stable laptop at work (this one with xfce + GNOME), I ll try to upgrade it and see if I see the same problems that I notice in mine. Then, I ll check the corresponding packages to see if there are open bugs about them, and if not, report them to their maintainers. I have to review the wiki pages related to the Jessie Desktop theme selection, I think they wanted the wallpaper to be inside before the freeze. Maybe I can help in publicity about that, handle the votings and so. I like Joy, but it s time to change a bit, new fresh air into the room!
Filed under: My experiences and opinion Tagged: Contributing to libre software, Debian, English, Free Software, Moving into free software