Search Results: "ron"

31 July 2014

Craig Small: Linux Capabilities

I was recently updating some code that uses fping. Initially it used exec() that was redirected to a temporary file but I changed it to use popen. While it had been a while since I ve done this sort of thing, I do recall there was an issue with running popen on setuid binary. A later found it is mainly around setuid scripts which are very problematic and there are good reasons why you don t do this. Anyhow, the program worked fine which surprised me. Was fping setuid root to get the raw socket?
$ ls -l /usr/bin/fping
-rwxr-xr-x 1 root root 31464 May  6 21:42 /usr/bin/fping
It wasn t which at first all I thought ok, so that s why popen is happy . The way that fping and other programs work is they bind to a raw socket. This socket sits below the normal type sockets such as the ones used for TCP and UDP and normal users cannot use them by default. So how did fping work it s magic and get access to this socket? It used Capabilities. Previously getting privileged features had a big problem; it was an all or nothing thing. You want access to a raw socket? Sure, be setuid but that means you also could, for example, read any file on the system or set passwords. Capabilites provide a way of giving programs some better level of access, but not a blank cheque. The tool getcap is the way of determining what capabilities are found on a file. These capabilities are attributes on the file which, when the file is run, turn into capabilities or extra permissions. fping has the capability cap_net_raw+ep applied to it. This gives access to the RAW and PACKET sockets which is what fping needs. The +ep after the capability name means it is an Effective and Permitted capability, which describes what happens with child processes and dropping privileges. I hadn t seen these Capabilities before. They are a nice way to give your programs the access they need, but limiting the risk of something going wrong and having a rouge program running as root.

Russell Coker: BTRFS Status July 2014

My last BTRFS status report was in April [1], it wasn t the most positive report with data corruption and system hangs. Hacker News has a brief discussion of BTRFS which includes the statement Russell Coker s reports of his experiences with BTRFS give me the screaming heebie-jeebies, no matter how up-beat and positive he stays about it [2] (that s one of my favorite comments about my blog). Since April things have worked better. Linux kernel 3.14 solves the worst problems I had with 3.13 and it s generally doing everything I want it to do. I now have cron jobs making snapshots as often as I wish (as frequently as every 15 minutes on some systems), automatically removing snapshots (removing 500+ snapshots at once doesn t hang the system), balancing, and scrubbing. The fact that I can now expect that a filesystem balance (which is a type of defragment operation for BTRFS that frees some chunks ) from a cron job and expect the system not to hang means that I haven t run out of metadata chunk space. I expect that running out of metadata space can still cause filesystem deadlocks given a lack of reports on the BTRFS mailing list of fixes in that regard, but as long as balance works well we can work around that. My main workstation now has 35 days of uptime and my home server has 90 days of uptime. Also the server that stores my email now has 93 days uptime even though it s running Linux kernel 3.13.10. I am rather nervous about the server running 3.13.10 because in my experience every kernel before 3.14.1 had BTRFS problems that would cause system hangs. I don t want a server that s an hour s drive away to hang The server that runs my email is using kernel 3.13.10 because when I briefly tried a 3.14 kernel it didn t work reliably with the Xen kernel 4.1 from Debian/Wheezy and I had a choice of using the Xen kernel 4.3 from Debian/Unstable to match the Linux kernel or use an earlier Linux kernel. I have a couple of Xen servers running Debian/Unstable for test purposes which are working well so I may upgrade my mail server to the latest Xen and Linux kernels from Unstable in the near future. But for the moment I m just not doing many snapshots and never running a filesystem scrub on that server. Scrubbing In kernel 3.14 scrub is working reliably for me and I have cron jobs to scrub filesystems on every system running that kernel. So far I ve never seen it report an error on a system that matters to me but I expect that it will happen eventually. The paper An Analysis of Data Corruption in the Storage Stack from the University of Wisconsin (based on NetApp data) [3] shows that nearline disks (IE any disks I can afford) have an incidence of checksum errors (occasions when the disk returns bad data but claims it to be good) of about 0.42%. There are 18 disks running in systems I personally care about (as opposed to systems where I am paid to care) so with a 0.42% probability of a disk experiencing data corruption per year that would give a 7.3% probability of having such corruption on one disk in any year and a greater than 50% chance that it s already happened over the last 10 years. Of the 18 disks in question 15 are currently running BTRFS. Of the 15 running BTRFS 10 are scrubbed regularly (the other 5 are systems that don t run 24*7 and the system running kernel 3.13.10). Newer Kernels The discussion on the BTRFS mailing list about kernel 3.15 is mostly about hangs. This is correlated with some changes to improve performance so I presume that it has exposed race conditions. Based on those discussions I haven t felt inclined to run a 3.15 kernel. As the developers already have some good bug reports I don t think that I could provide any benefit by doing more testing at this time. I think that there would be no benefit to me personally or the Linux community in testing 3.15. I don t have a personal interest in RAID-5 or RAID-6. The only systems I run that have more data than will fit on a RAID-1 array of cheap SATA disks are ones that I am paid to run and they are running ZFS. So the ongoing development of RAID-5 and RAID-6 code isn t an incentive for me to run newer kernels. Eventually I ll test out RAID-6 code, but at the moment I don t think they need more bug reports in this area. I don t have a great personal interest in filesystem performance at this time. There are some serious BTRFS performance issues. One problem is that a filesystem balance and subtree removal seem to take excessive amounts of CPU time. Another is that there isn t much support for balancing IO to multiple devices (in RAID-1 every process has all it s read requests sent to one device). For large-scale use of a filesystem these are significant problems. But when you have basic requirements (such as a mail server for dozens of users or a personal workstation with a quad-core CPU and fast SSD storage) it doesn t make much difference. Currently all of my systems which use BTRFS have storage hardware that exceeds the system performance requirements by such a large margin that nothing other than installing Debian packages can slow the system down. So while there are performance improvements in newer versions of the BTRFS kernel code that isn t an incentive for me to upgrade. It s just been announced that Debian/Jessie will use Linux 3.16, so I guess I ll have to test that a bit for the benefit of Debian users. I am concerned that 3.16 won t be stable enough for typical users at the time that Jessie is released.

Wouter Verhelst: Using your Belgian eID with SSH mark two

Several years ago, I blogged about how to use a Belgian electronic ID card with SSH. I never really used it myself, but was interested in figuring out if it would still work. The good news is that since then, you don't need to recompile OpenSSH anymore to get PKCS#11 support; this is now compiled in by default. The slightly bad news is that there will be some more typework. Rather than entering ssh-add -D 0 (to access the PKCS#11 certificate in slot 0), you should now enter something along the lines of ssh-add -s /usr/lib/libbeidpkcs11.so.0. This will ask for your passphrase, but it isn't necessary to enter the correct pin code at this point in time. The first time you try to log on, you'll get a standard beid dialog box where you should enter your pin code; this will then work. The next time, you'll be logged on and you can access servers without having to enter a pin code. The worse news is that there seems to be a bug in ssh-agent, making it impossible to unload a PKCS#11 library. Doing ssh-add -D will remove your keys from the agent; the next time you try to add them again, however, ssh-agent will simply report SSH_AGENT_FAILURE. I suspect the dlopen()ed modules aren't being unloaded when the keys are removed. Unfortunately, the same (or at least, a similar) bug appears to occur when one removes the card from the cardreader. As such, I don't currently recommend trying to use this. Update: fix command-line options to ssh-add invocation above.

Petter Reinholdtsen: Debian Edu interview: Bernd Zeitzen

The complete and free out of the box software solution for schools, Debian Edu / Skolelinux, is used quite a lot in Germany, and one of the people involved is Bernd Zeitzen, who show up on the project mailing lists from time to time with interesting questions and tips on how to adjust the setup. I managed to interview him this summer. Who are you, and how do you spend your days? My name is Bernd Zeitzen and I'm married with Hedda, a self employed physiotherapist. My former profession is tool maker, but I haven't worked for 30 years in this job. 30 years ago I started to support my wife and become her officeworker and a few years later the administrator for a small computer network, today based on Ubuntu Server (Samba, OpenVPN). For her daily work she has to use Windows Desktops because the software she needs to organize her business only works with Windows . :-( In 1988 we started with one PC and DOS, then I learned to use Windows 98, 2000, XP, , 8, Ubuntu, MacOSX. Today we are running a Linux server with 6 Windows clients and 10 persons (teacher of children with special needs, speech therapist, occupational therapist, psychologist and officeworkers) using our Samba shares via OpenVPN to work with the documentations of our patients. How did you get in contact with the Skolelinux / Debian Edu project? Two years ago a friend of mine asked me, if I want to get a job in his school (Gymnasium Harsewinkel). They started with Skolelinux / Debian Edu and they were looking for people to give support to the teachers using the software and the network and teaching the pupils increasing their computer skills in optional lessons. I'm spending 4-6 hours a week with this job. What do you see as the advantages of Skolelinux / Debian Edu? The independence. First: Every person is allowed to use, share and develop the software. Even if you are poor, you are allowed to use the software included in Skolelinux/Debian Edu and all the other Free Software. Second: The software runs on old machines and this gives us the possibility to recycle computers, weeded out from offices. The servers and desktops are running for more than two years and they are working reliable. We have two servers (one tjener and one terminal server), 45 workstations in three classrooms and seven laptops as a mobile solution for all classrooms. These machines are all booting from the terminal server. In the moment we are installing 30 laptops as mobile workstations. Then the pupils have the possibility to work with these machines in their classrooms. Internet access is realized by a WLAN router, connected to the schools network. This is all done without a dedicated system administrator or a computer science teacher. What do you see as the disadvantages of Skolelinux / Debian Edu? Teachers and pupils are Windows users. <Irony on> And Linux isn't cool. It's software for freaks using the command line. <Irony off> They don't realize the stability of the system. Which free software do you use daily? Firefox, Thunderbird, LibreOffice, Ubuntu Server 12.04 (Samba, Apache, MySQL, Joomla!, and Skolelinux / Debian Edu) Which strategy do you believe is the right one to use to get schools to use free software? In Germany we have the situation: every school is free to decide which software they want to use. This decision is influenced by teachers who learned to use Windows and MS Office. They buy a PC with Windows preinstalled and an additional testing version of MS Office. They don't know about the possibility to use Free Software instead. Another problem are the publisher of school books. They develop their software, added to the school books, for Windows.

29 July 2014

Daniel Pocock: Pruning Syslog entries from MongoDB

I previously announced the availability of rsyslog+MongoDB+LogAnalyzer in Debian wheezy-backports. This latest rsyslog with MongoDB storage support is also available for Ubuntu and Fedora users in one way or another. Just one thing was missing: a flexible way to prune the database. LogAnalyzer provides a very basic pruning script that simply purges all records over a certain age. The script hasn't been adapted to work within the package layout. It is written in PHP, which may not be ideal for people who don't actually want LogAnalyzer on their Syslog/MongoDB host. Now there is a convenient solution: I've just contributed a very trivial Python script for selectively pruning the records. Thanks to Python syntax and the PyMongo client, it is extremely concise: in fact, here is the full script:
#!/usr/bin/python
import syslog
import datetime
from pymongo import Connection
# It assumes we use the default database name 'logs' and collection 'syslog'
# in the rsyslog configuration.
with Connection() as client:
    db = client.logs
    table = db.syslog
    #print "Initial count: %d" % table.count()
    today = datetime.datetime.today()
    # remove ANY record older than 5 weeks except mail.info
    t = today - datetime.timedelta(weeks=5)
    table.remove( "time":  "$lt": t  , "syslog_fac":   "$ne" : syslog.LOG_MAIL  )
    # remove any debug record older than 7 days
    t = today - datetime.timedelta(days=7)
    table.remove( "time":  "$lt": t  , "syslog_sever": syslog.LOG_DEBUG )
    #print "Final count: %d" % table.count()
Just put it in /usr/local/bin and run it daily from cron. Customization Just adapt the table.remove statements as required. See the PyMongo tutorial for a very basic introduction to the query syntax and full details in the MongoDB query operator reference for creating more elaborate pruning rules. Potential improvements
  • Indexing the columns used in the queries
  • Logging progress and stats to Syslog

LogAnalyzer using a database backend such as MongoDB is very easy to set up and much faster than working with text-based log files

Russell Coker: Android Screen Saving

Just over a year ago I bought a Samsung Galaxy Note 2 [1]. About 3 months ago I noticed that some of the Ingress menus had burned in to the screen. Back in ancient computer times there were screen saver programs that blanked the screen to avoid this, then the screen saver programs transitioned to displaying a variety of fancy graphics which didn t really fulfill the purpose of saving the screen. With LCD screens I have the impression that screen burn wasn t an issue, but now with modern phones we have LED displays which have the problem again. Unfortunately there doesn t seem to be a free screen-saver program for Android in the Google Play store. While I can turn the screen off entirely there are some apps such as Ingress that I d like to keep running while the screen is off or greatly dimmed. Now I sometimes pull the notification menu down when I m going to leave Ingress idle for a while, this doesn t stop the screen burning but it does cause different parts to burn which alleviates the problem. It would be nice if apps were designed to alleviate this. A long running app should have an option to change the color of it s menus, it would be ideal to randomly change the color on startup. If the common menus such as the COMM menu would appear in either red, green, or blue (the 3 primary colors of light) in a ratio according to the tendency to burn (blue burns fastest so should display least) then it probably wouldn t cause noticable screen burn after 9 months. The next thing that they could do is to slightly vary the position of the menus, instead of having a thin line that s strongly burned into the screen there would be a fat line lightly burned in which should be easier to ignore. It s good when apps have an option of a dark theme, that involves less light coming from the screen that should reduce battery use and screen burn. A dark theme should be at least default and probably mandatory for long running apps, a dark theme is fortunately the only option for Ingress. I am a little disappointed with my phone. I m not the most intensive Ingress player so I think that the screen should have lasted for more than 9 months before being obviously burned.

Russell Coker: Happiness and Lecture Questions

I just attended a lecture about happiness comparing Australia and India at the Australia India Institute [1]. The lecture was interesting but the questions were so bad that it makes a good case for entirely banning questions from public lectures. Based on this and other lectures I ve attended I ve written a document about how to recognise worthless questions and cut them off early [2]. As you might expect from a lecture on happiness there were plenty of stupid comments from the audience about depression, as if happiness is merely the absence of depression. Then they got onto stupidity about suicide. One question claimed that Australia has a high suicide rate, Wikipedia however places Australia 49th out of 110 countries, that means Australia is slightly above the median for suicide rates per country. Given some of the dubious statistics in the list (for example the countries claiming to have no suicides and the low numbers reported by some countries with extreme religious policies) I don t think we can be sure that Australia would be above the median if we had better statistics. Another question claimed that Sweden had the highest suicide rate in Europe, while Greenland, Belgium, Finland, Austria, France, Norway, Denmark, Iceland, and most of Eastern Europe are higher on the list. But the bigger problem in regard to discussing suicide is that the suicide rate isn t about happiness. When someone kills themself because they have a terminal illness that doesn t mean that they were unhappy for the majority of their life and doesn t mean that they were any unhappier than the terminally ill people who don t do that. Some countries have a culture that is more positive towards suicide which would increase the incidence, Japan for example. While people who kill themselves in Japan are probably quite unhappy at the time I don t think that there is any reason to believe that they are more unhappy than people in other countries who only keep living because suicide is considered to be wrong. It seems to me that the best strategy when giving or MCing a lecture about a potentially contentious topic is to plan ahead for what not to discuss. For a lecture about happiness it would make sense to rule out all discussion of suicide, anti-depressants, and related issues as they aren t relevant to the discussion and can t be handled in an appropriate manner in question time.

Ricardo Mones: Switching PGP keys

Finally I find the mood to do this, a process which started 5 years ago in DebConf 9 at C ceres by following Ana's post, of course with my preferred options and my name, not like some other ;-).

Said that, dear reader, if you have signed my old key:

1024D/C9B55DAC 2005-01-19 [expires: 2015-10-01]
Key fingerprint = CFB7 C779 6BAE E81C 3E05  7172 2C04 5542 C9B5 5DAC

And want to sign my "new" and stronger key:

4096R/DE5BCCA6 2009-07-29
Key fingerprint = 43BC 364B 16DF 0C20 5EBD  7592 1F0F 0A88 DE5B CCA6

You're welcome to do so :-)

The new key is signed with the old, and the old key is still valid, and will probably be until expiration date next year. Don't forget to gpg --recv-keys DE5BCCA6 to get the new key and gpg --refresh-keys C9B55DAC to refresh the old (otherwise it may look expired).

Debian's Keyring Team has already processed my request to add the new key, so all should keep working smoothly. Kudos to them!

Christian Perrier: [life] Running update July 26th 2014

Dog, long time since I blogged about my running activities. Apparently, I didn't since.....I posted a summary for 2013. So, well, that will be a long update as many things happened during the first half of 2014 when it comes at running, for me. January: I was recovering from a fatigue fracture injury inherited from last races in 2013. As a consequence, I resumed running only on Jan 7th. Therefore I cancelled my participation to the "Semi Raid 28", an night orienteering raid of about 50-60km in southern neighbourhood of Paris. Instead, I actually offerred my help to organizers in collecting orienteering signs after the race (the longest one : 120km). So, I ended up spending over 24 hours running in woods and hunting down hidden signs with the same information than runners. My only advantage was that I was able to use my car to go from one point to another. Still, I ended up running over 70km in many small parts, often alone in the dark woods with my headlamp, on very muddy areas...and collecting nearly 80 huge signs. February: Everything was going well and I for instance ran a great half-marathon in Bullion (south of Paris) in 1h3821" (great for a quite hilly race)....until I twisted my left ankle while running back from work. A quite severe twist, though no bone damage, thankfully. I had to stop running, again, for 3 years. Biking to/from work was the replacement activity.... March: I resumed running on March 10th, one week before a quite difficult trail race in my neighbourhood (30km "only" but up to 800 meters positive climb). That race was a preparation (and a test after the injury) for my 3rd participation to "Paris Ecotrail", a 80km trail race in woods of the South-West area of Paris, ending in the Eiffel Tower area. Indeed, both went very well, though I was very careful with my ankle. I finally broke my record at Ecotrail, finishing the race in 9h08 (to be compared to 9h36 last year and 11h15 the year before). April: Paris marathon was scheduled one week after Ecotrail. Everybody will tell you that running a marathon one week after a 80km race is kinda crazy.....which is why I made it..:-). That was my 3rd Paris marathon and my 12th marathon overall. However, this year, no record in sight. The challenge was running the marathon....dressed as SpongeBob (you know me, right?). I actually had great fun doing that and was happy to get zillions of cheering all over the race, from the crowd. I finally completed the race in 4h30, which is, after all, not that far from the time of my very first marathon (4h12). The only drawback was that the succession of quite very long distance runs made my left knee suffer as it never happened before. As a consequence, I (again) had to stop running for nearly one month before we found that I was quite sensitive to pronation, which the succession of long and slow races made worse. May: so finally afterthese (very) long weeks, I could gradually resume running, which finally culminated in mid-May with the 50km race "trail des Cerfs", in the Rambouillet Forest, closed to our place. This quite long but not too difficult trail race ("only" 800 meters positive climb overall) was completed in 5h16, which was completely unexpected, given the low training during the previous weeks. June: no race during that month. The entire month was focused on preparing the Montagn'hard race of July 5th: so several training sessions with a lot of climbing either by running or by fast walking (nordic style) as well as downhill run training (always important for moutain trail). July: the second "big peak" of my 2014 season was scheduled for July 5th: "La Montagn'hard", a moutain trail race close to Les Contamines in the neighbourhood of Chamonix, the french moutaineering Mekkah. "Only" 60 kilometers....but close to 5000 meters positive climb. Montagn'hard is among the thoughest moutain trail races in France and therefore a "must do" for trail runners. This race week-end includes also a 105km ultra-race, which is often said to be as hard, even maybe harder, than the very famous "Ultra Trail du Mont-Blanc" trail in Chamonix. Still, for my second only season in moutain trail running, I decided to be "wise" and stick with the "medium" version (after all, my experience, as of now with moutain trails were only two quite "short" ones). Needless to say, it has indeed been a GREAT race. The environment is wonderful ("Miage" side of the Mont-Blanc range), the race goes through great place (Col de Tricot, noticeably) and I made a great result by finishing80th out of 325+ runners, in 12h18, while my target time was around 13 hours. This is where I am now. Nearly one month after Montagn'hard, I'm deeply training for my next Big Goal: The "Sur la Trace des Ducs de Savoie" or "TDS", one of the 4 races of the Ultra Trail du Mont-Blanc week, in end August (during DebConf): 120km, nearly 7500m positive climp, between Courmayeur and Chamonix, through several passes, up to 2600m height. Yet another challenge: my first "over 24h" race, with a full night out in the moutains. You'll certainly hear again from me about that...:-)

28 July 2014

Daniel Pocock: Secure that Dictaphone

2014 has been a big year for dictaphones so far. First, it was France and the secret recordings made by Patrick Buisson during the reign of President Sarkozy. Then, a US court ordered the release of the confidential Boston College tapes, part of an oral history project. Originally, each participant had agreed their recording would only be released after their death. Sinn Fein leader Gerry Adams was arrested and questioned over a period of 100 hours and released without charge. Now Australia is taking its turn. In #dictagate down under, a senior political correspondent from a respected newspaper recorded (most likely with consent) some off-the-record comments of former conservative leader Ted Baillieu. Unfortunately, this journalist misplaced the dictaphone at the state conference of Baillieu's arch-rivals, the ALP. A scandal quickly errupted. Secure recording technology There is no question that electronic voice recordings can be helpful for people, including journalists, researchers, call centers and many other purposes. However, the ease with which they can now be distributed is only dawning on people. Twenty years ago, you would need to get the assistance of a radio or TV producer to disseminate such recordings so widely. Today there is email and social media. The Baillieu tapes were emailed directly to 400 people in a matter of minutes. Just as technology brings new problems, it also brings solutions. Encryption is one of them. Is encryption worthwhile? Coverage of the Snowden revelations has revealed that many popular security technologies are not one hundred percent safe. In each of these dictaphone cases, however, NSA-level expertise was not a factor. Even the most simplistic encryption would have caused endless frustration to the offenders who distributed the Baillieu tape. How can anybody be sure encryption is reliable? Part of the problem is education. Everybody using the technology needs to be aware of the basic concepts, for example, public key cryptography. Another big question mark is back doors. There is ongoing criticism of Apple iPhone/iPod devices and the many ways that their encryption can be easily disabled by Apple engineers and presumably many former staff, security personnel and others. The message is clear: proprietary, closed-source solutions should be avoided. Free and open source technologies are the alternative. If a company does not give you the source code, how can anybody independently audit their code for security? With encryption software, what use is it if nobody has verified it? What are the options? However, given that the majority of people don't have a PhD in computer science or mathematics, are there convenient ways to get started with encryption? Reading is a good start. The Code Book by Simon Singh (author of other popular science books like Fermat's Last Theorem) is very accessible, not classified and assumes no formal training in mathematics. Even for people who do know these topics inside out, it is a good book to share with friends and family. The Guardian Project (no connection with Guardian Media of Edward Snowden fame) aims to provide a secure and easy to use selection of apps for pocket devices. This project has practical applications in business, journalism and politics alike. How should a secure dictaphone app work? Dictaphone users typically need to take their dictaphones in the field, so there is a risk of losing it or having it stolen. A strong security solution in this situation may involve creating an RSA key pair on a home/office computer, keeping the private key on the home computer and putting the public key on the dictaphone device. Configured this way, the dictaphone will not be able to play back any of the recordings itself - the user will always have to copy them to the computer for decryption.

Russ Allbery: AFS::PAG 1.02

This is primarily a testing exercise. I moved my software release process and web page generation to a different host, and wanted to run through a release of a package to make sure that I got all the details right. It's still a bit clunky, and I need to tweak the process, but it's close enough. That said, there are a few minor changes in this module (which provides the minimum C glue required to do AFS operations from Perl only the pieces that can't be duplicated by calling command-line programs). I'm improving the standardization of my Perl distributions, so I've moved NEWS to Changes and switched to the Lancaster Consensus environment variables for controlling testing. I also added some more pieces to the package metadata. You can get the latest version from the AFS::PAG distribution page.

27 July 2014

Christian Perrier: [life] Running update July 26th 2014

Dog, long time since I blogged about my running activities. Apparently, I didn't since...... So, well, that will be a long update as many things happened during the first half of 2014 when it comes at running, for me. January: I was recovering from a fatigue fracture injury inherited from last races in 2013. As a consequence, I resumed running only on Jan 7th. Therefore I cancelled my participation to the "Semi Raid 28", an night orienteering raid of about 50-60km in southern neighbourhood of Paris. Instead, I actually offerred my help to organizers in collecting orienteering signs after the race (the longest one : 120km). So, I ended up spending over 24 hours running in woods and hunting down hidden signs with the same information than runners. My only advantage was that I was able to use my car to go from one point to another. Still, I ended up running over 70km in many small parts, often alone in the dark woods with my headlamp, on very muddy areas...and collecting nearly 80 huge signs. February: Everything was going well and I for instance ran a great half-marathon in Bullion (south of Paris) in 1h3821" (great for a quite hilly race)....until I twisted my left ankle while running back from work. A quite severe twist, though no bone damage, thankfully. I had to stop running, again, for 3 years. Biking to/from work was the replacement activity.... March: I resumed running on March 10th, one week before a quite difficult trail race in my neighbourhood (30km "only" but up to 800 meters positive climb). That race was a preparation (and a test after the injury) for my 3rd participation to "Paris Ecotrail", a 80km trail race in woods of the South-West area of Paris, ending in the Eiffel Tower area. Indeed, both went very well, though I was very careful with my ankle. I finally broke my record at Ecotrail, finishing the race in 9h08 (to be compared to 9h36 last year and 11h15 the year before). April: Paris marathon was scheduled one week after Ecotrail. Everybody will tell you that running a marathon one week after a 80km race is kinda crazy.....which is why I made it..:-). That was my 3rd Paris marathon and my 12th marathon overall. However, this year, no record in sight. The challenge was running the marathon....dressed as SpongeBob (you know me, right?). I actually had great fun doing that and was happy to get zillions of cheering all over the race, from the crowd. I finally completed the race in 4h30, which is, after all, not that far from the time of my very first marathon (4h12). The only drawback was that the succession of quite very long distance runs made my left knee suffer as it never happened before. As a consequence, I (again) had to stop running for nearly one month before we found that I was quite sensitive to pronation, which the succession of long and slow races made worse. May: so finally afterthese (very) long weeks, I could gradually resume running, which finally culminated in mid-May with the 50km race "trail des Cerfs", in the Rambouillet Forest, closed to our place. This quite long but not too difficult trail race ("only" 800 meters positive climb overall) was completed in 5h16, which was completely unexpected, given the low training during the previous weeks. June: no race during that month. The entire month was focused on preparing the Montagn'hard race of July 5th: so several training sessions with a lot of climbing either by running or by fast walking (nordic style) as well as downhill run training (always important for moutain trail). July: the second "big peak" of my 2014 season was scheduled for July 5th: "La Montagn'hard", a moutain trail race close to Les Contamines in the neighbourhood of Chamonix, the french moutaineering Mekkah. "Only" 60 kilometers....but close to 5000 meters positive climb. Montagn'hard is among the thoughest moutain trail races in France and therefore a "must do" for trail runners. This race week-end includes also a 105km ultra-race, which is often said to be as hard, even maybe harder, than the very famous "Ultra Trail du Mont-Blanc" trail in Chamonix. Still, for my second only season in moutain trail running, I decided to be "wise" and stick with the "medium" version (after all, my experience, as of now with moutain trails were only two quite "short" ones). Needless to say, it has indeed been a GREAT race. The environment is wonderful ("Miage" side of the Mont-Blanc range), the race goes through great place (Col de Tricot, noticeably) and I made a great result by finishing80th out of 3250+ runners, in 12h18, while my target time was around 13 hours. This is where I am now. Nearly one month after Montagn'hard, I'm deeply training for my next Big Goal: The "Sur la Trace des Ducs de Savoie" or "TDS", one of the 4 races of the Ultra Trail du Mont-Blanc week, in end August (during DebConf): 120km, nearly 7500m positive climp, between Courmayeur and Chamonix, through several passes, up to 2600m height. Yet another challenge: my first "over 24h" race, with a full night out in the moutains. You'll certainly hear again from me about that...:-)

25 July 2014

Juliana Louback: Extending an xTuple Business Object

xTuple is in my opinion incredibly well designed; the code is clean and the architecture ahderent to a standardized structure. All this makes working with xTuple software quite a breeze. I wanted to integrate JSCommunicator into the web-based xTuple version. JSCommunicator is a SIP communication tool, so my first step was to create an extension for the SIP account data. Luckily for me, the xTuple development team published an awesome tutorial for writing an xTuple extension. xTuple cleverly uses model based business objects for the various features available. This makes customizing xTuple very straightforward. I used the tutorial mentioned above for writing my extension, but soon noticed my goals were a little different. A SIP account has 3 data fields, these being the SIP URI, the account password and an optional display name. xTuple currently has a business object in the core code for a User Account and it would make a lot more sense to simply add my 3 fields to this existing business object rather than create another business object. The tutorial very clearly shows how to extend a business object with another business object, but not how to extend a business object with only new fields (not a whole new object). Now maybe I m just a whole lot slower than most people, but I had a ridiculously had time figuring this out. Mind you, this is because I m slow, because the xTuple documentation and code is understandable and as self-explanatory as it gets. I think it just takes a bit to get used to. Either way, I thought this just might be useful to others so here is how I went about it. Setup First you ll have to set up your xTuple development environment and fork the xtuple and xtuple-extesions repositories as shown in this handy tutorial. A footnote I d like to add is please verify that your version of Vagrant (and anything else you install) is the one listed in the tutorial. I think I spent like two entire days or more on a wild goose (bug) chase trying to set up my environment when the cause of all the errors was that I somehow installed an older version of Vagrant - 1.5.4 instead of 1.6.3. Please don t make the same mistake I did. Actually if for some reason you get the following error when you try using node:
<<ERROR 2014-07-10T23:52:46.948Z>> Unrecoverable exception. Cannot call method 'extend' of undefined
    at /home/vagrant/dev/xtuple/lib/backbone-x/source/model.js:37:39
    at Object.<anonymous> (/home/vagrant/dev/xtuple/lib/backbone-x/source/model.js:1364:3)
    ...
chances are, you have the wrong version. That s what happened to me. The Vagrant Virtual Development Environment automatically installs and configures everything you need, it s ready to go. So if you find yourself installing and updating and apt-gets and etc, you probably did something wrong. Coding So by now we should have the Vagrant Virtual Development Environment set up and the web app up and running and accessible at localhost:8443. So far so good. Disclaimer: You will note that much of this is similar - or rather, nearly identical - to xTuple s tutorial but there are some small but important differences and a few observations I think might be useful. Other Disclaimer: I m describing how I did it, which may or may not be up to snuff . Works for me though. Schema First let s make a schema for the table we will create with the new custom fields. Be sure to create the correct directory stucture, aka /path/to/xtuple-extensions/source/<YOUR EXTENSION NAME>/database/source or in my case /path/to/xtuple-extensions/source/sip_account/database/source, and create the file create_sa_schema.sql, sa is the name of my schema. This file will contain the following lines:
do $$
  /* Only create the schema if it hasn't been created already */
  var res, sql = "select schema_name from information_schema.schemata where schema_name = 'sa'",
  res = plv8.execute(sql);
  if (!res.length)  
    sql = "create schema sa; grant all on schema sa to group xtrole;"
    plv8.execute(sql);
   
$$ language plv8;
Of course, feel free to replace sa with your schema name of choice. All the code described here can be found in my xtuple-extensions fork, on the sip_ext branch. Table We ll create a table containing your custom fields and a link to an existing table - the table for the existing business object you want to extend. If you re wondering why make a whole new table for a few extra fields, here s a good explanation, the case in question is adding fields to the Contact business object. You need to first figure out what table you want to link to. This might not be uber easy. I think the best way to go about it is to look at the ORMs. The xTuple ORMs are a JSON mapping between the SQL tables and the object-oriented world above the database, they re .json files found at path/to/xtuple/node_modules/xtuple/enyo-client/database/orm/models for the core business objects and at path/to/xtuplenyo-client/extensions/source/<EXTENSION NAME>/database/orm/models for exension business objects. I ll give two examples. If you look at contact.json you will see that the Contact business object refers to the table cntct . Look for the type : Contact on the line above, so we know it s the Contact business object. In my case, I wanted to extend the UserAccount and UserAccountRelation business objects, so check out user_account.json. The table listed for UserAccount is xt.usrinfo and the table listed for UserAccountRelation is xt.usrlite. A closer look at the sql files for these tables (usrinfo.sql and usrlite.sql) revealed that usrinfo is in fact a view and usrlite is A light weight table of user information used to avoid punishingly heavy queries on the public usr view . I chose to refer to xt.usrlite - that or I received error messages when trying the other table names. Now I ll make the file /path/to/xtuple-extensions/source/sip_account/database/source/usrlitesip.sql, to create a table with my custom fields plus the link to the urslite table. Don t quote me on this, but I m under the impression that this is the norm for naming the sql file joining tables: the name of the table you are referring to ( usrlite in this case) and your extension s name. Content of usrlitesip.sql:
select xt.create_table('usrlitesip', 'sa');
select xt.add_column('usrlitesip','usrlitesip_id', 'serial', 'primary key', 'sa');
select xt.add_column('usrlitesip','usrlitesip_usr_username', 'text', 'references xt.usrlite (usr_username)', 'sa');
select xt.add_column('usrlitesip','usrlitesip_uri', 'text', '', 'sa');
select xt.add_column('usrlitesip','usrlitesip_name', 'text', '', 'sa');
select xt.add_column('usrlitesip','usrlitesip_password', 'text', '', 'sa');
comment on table sa.usrlitesip is 'Joins User with SIP account';
Breaking it down, line 1 creates the table named usrlitesip (no duh), line 2 is for the primary key (self-explanatory). You can then add any columns you like, just be sure to add one that references the table you want to link to. I checked usrlite.sql and saw the primary key is usr_username, be sure to use the primary key of the table you are referencing. You can check what you made by executing the .sql files like so:
$ cd /path/to/xtuple-extensions/source/sip_account/database/source
$ psql -U admin -d dev -f create_sa_schema.sql
$ psql -U admin -d dev -f usrlitesip.sql
After which you will see the table with the columns you created if you enter:
$ psql -U admin -d dev -c "select * from sa.usrlitesip;"
Now create the file /path/to/xtuple-extensions/source/sip_account/database/source/manifest.js to put the files together and in the right order. It should contain:
 
  "name": "sip_account",
  "version": "1.4.1",
  "comment": "Sip Account extension",
  "loadOrder": 999,
  "dependencies": ["crm"],
  "databaseScripts": [
    "create_sa_schema.sql",
    "usrlitesip.sql",
    "register.sql"
  ]
 
I think the name has to be the same you named your extension directory as in /path/to/xtuple-extensions/source/<YOUR EXTENSION NAME>. I think the comment can be anything you like and you want your loadOrder to be high so it s the last thing installed (as it s an add on.) So far we are doing exactly what s instructed in the xTuple tutorial. It s repetitive, but I think you can never have too many examples to compare to. In databaseScripts you will list the two .sql files you just created for the schema and the table, plus another file to be made in the same directory named register.sql. I m not sure why you have to make the register.sql or even if you indeed have to. If you leave the file empty, there will be a build error, so put a ; in the register.sql or remove the line register.sql from manifest.js as I think for now we are good without it. Now let s update the database with our new extension:
$ cd /path/to/xtuple
$ ./scripts/build_app.js -d dev -e ../xtuple-extensions/source/sip_account
$ psql -U admin -d dev -c "select * from xt.ext;"
That last command should display a table with a list of extensions; the ones already in xtuple like crm and billing and some others plus your new extension, in this case sip_account . When you run build_app.js you ll probably see a message along the lines of <Extension name> has no client code, not building client code and that s fine because yeah, we haven t worked on the client code yet. ORM Here s where things start getting different. So ORMs link your object to an SQL table. But we DON T want to make a new business object, we want to extend an existing business object, so the ORM we will make will be a little different than the xTuple tutorial. Steve Hackbarth kindly explained this new business object/existing business object ORM concept here. First we ll create the directory /path/to/xtuple-extensions/source/sip_account/database/orm/ext, according to xTuple convention. ORMs for new business objects would be put in /path/to/xtuple-extensions/source/sip_account/database/orm/models. Now we ll create the .json file /path/to/xtuple-extensions/source/sip_account/database/orm/ext/user_account.jscon for our ORM. Once again, don t quote me on this, but I think the name of the file should be the name of the business object you are extending, as is done in the turorial example extending the Contact object. In our case, UserAccount is defined in user_account.json and that s what I named my extension ORM too. Here s what you should place in it:
 1 [
 2    
 3     "context": "sip_account",
 4     "nameSpace": "XM",
 5     "type": "UserAccount",
 6     "table": "sa.usrlitesip",
 7     "isExtension": true,
 8     "isChild": false,
 9     "comment": "Extended by Sip",
10     "relations": [
11        
12         "column": "usrlitesip_usr_username",
13         "inverse": "username"
14        
15     ],
16     "properties": [
17        
18         "name": "uri",
19         "attr":  
20           "type": "String",
21           "column": "usrlitesip_uri",
22           "isNaturalKey": true
23          
24        ,
25        
26         "name": "displayName",
27         "attr":  
28           "type": "String",
29           "column": "usrlitesip_name"
30          
31        ,
32        
33         "name": "sipPassword",
34         "attr":  
35           "type": "String",
36           "column": "usrlitesip_password"
37          
38        
39     ],
40     "isSystem": true
41    ,
42    
43     "context": "sip_account",
44     "nameSpace": "XM",
45     "type": "UserAccountRelation",
46     "table": "sa.usrlitesip",
47     "isExtension": true,
48     "isChild": false,
49     "comment": "Extended by Sip",
50     "relations": [
51        
52         "column": "usrlitesip_usr_username",
53         "inverse": "username"
54        
55     ],
56     "properties": [
57        
58         "name": "uri",
59         "attr":  
60           "type": "String",
61           "column": "usrlitesip_uri",
62           "isNaturalKey": true
63          
64        ,
65        
66         "name": "displayName",
67         "attr":  
68           "type": "String",
69           "column": "usrlitesip_name"
70          
71        ,
72        
73         "name": "sipPassword",
74         "attr":  
75           "type": "String",
76           "column": "usrlitesip_password"
77          
78        
79     ],
80     "isSystem": true
81    
82 ]
Note the context is my extension name, because the context + nameSpace + type combo has to be unique. We already have a UserAccount and UserAccountRelation object in the XM namespace in the xtuple context in the original user_account.json, now we will have a UserAccount and UserAccountRelation object in the XM namespace in the sip_account conext. What else is important? Note that isExtension is true on lines 7 and 47 and the relations item contains the column of the foreign key we referenced. This is something you might want to verify: column (lines 12 and 52) is the name of the attribute on your table. When we made a reference to the primary key usr_usrname from the xt.usrlite table we named that column usrlitesip_usr_usrname. But the inverse is the attribute name associated with the original sql column in the original ORM. Did I lose you? I had a lot of trouble with this silly thing. In the original ORM that created a new UserAccount business object, the primary key attribute is named username , as can be seen here. That is what should be used for the inverse value. Not the sql column name (usr_username) but the object attribute name (username). I m emphasizing this because I made that mistake and if I can spare you the pain I will. If we rebuild our extension everything should come along nicely, but you won t see any changes just yet in the web app because we haven t created the client code. Client Create the directory /path/to/xtuple-extensions/source/sip_account/client which is where we ll keep all the client code. Extend Workspace View I want the fields I added to show up on the form to create a new User Account, so I need to extend the view for the User Account workspace. I ll start by creating a directory /path/to/xtuple-extensions/source/sip_account/client/views and in it creating a file named workspace.js containing this code:
XT.extensions.sip_account.initWorkspace = function ()  
	var extensions = [
  	 kind: "onyx.GroupboxHeader", container: "mainGroup", content: "_sipAccount".loc() ,
  	 kind: "XV.InputWidget", container: "mainGroup", attr: "uri"  ,
  	 kind: "XV.InputWidget", container: "mainGroup", attr: "displayName"  ,
  	 kind: "XV.InputWidget", container: "mainGroup", type:"password", attr: "sipPassword"  
	];
	XV.appendExtension("XV.UserAccountWorkspace", extensions);
 ;
So I m initializing my workspace and creating an array of items to add (append) to view XV.UserAccountWorkspace. The first item is this onyx.GroupboxHeader which is a pretty divider for my new form fields, the kind you find in the web app at Setup > User Accounts, like Overview . I have no idea what other options there are for container other than mainGroup , so let s stick to that. I ll explain content: _sipAccount .loc() in a bit. Next I created three input fields of the XV.InputWidget kind. This also confused me a bit as there are different kinds of input to be used, like dropdowns and checkboxes. The only advice I can give is snoop around the webapp, find an input you like and look up the corresponding workspace.js file to see what was used. What we just did is (should be) enough for the new fields to show up on the User Account form. But before we see things change, we have to package the client. Create the file /path/to/xtuple-extensions/source/sip_account/client/views/package.js. This file is needed to package groups of files and indicates the order the files should be loaded (for more on that, see this). For now, all the file will contain is:
enyo.depends(
"workspace.js"
);
You also need to package the views directory containing workspace.js, so create the file Create the file /path/to/xtuple-extensions/source/sip_account/client/package.js and in it show that the directory views and its contents must be part of the higher level package:
enyo.depends(
"views"
);
I like to think of it as a box full of smaller boxes. This will sound terrible, but apparently you also need to create the file /path/to/xtuple-extensions/source/sip_account/client/core.js containing this line:
XT.extensions.icecream =  ;
I don t know why. As soon as I find out I ll be sure to inform you. As we ve added a file to the client directory, be sure to update /path/to/xtuple-extensions/source/sip_account/client/package.js so it included the new file:
enyo.depends(
"core.js",
"views"
);
Translations Remember _sipAccount .loc() in our workspace.js file? xTuple has great internationalization support and it s easy to use. Just create the directory and file /path/to/xtuple-extensions/source/sip_account/client/en/strings.js and in it put key-value pairs for labels and their translation, like this:
(function ()  
  "use strict";
  var lang = XT.stringsFor("en_US",  
    "_sipAccount": "Sip Account",
    "_uri": "Sip URI",
    "_displayName": "Display Name",
    "_sipPassword": "Password"
   );
  if (typeof exports !== 'undefined')  
    exports.language = lang;
   
 ());
So far I included all the labels I used in my Sip Account form. If you write the wrong label (key) or forget to include a corresponding key-value pair in strings.js, xTuple will simply name your lable _lableName , underscore and all. Now build your extension and start up the server:
$ cd /path/to/xtuple 
$ ./scripts/build_app.js -d dev -e ../xtuple-extensions/source/sip_account
$ node node-datasource/main.js
If the server is already running, just stop it and restart it to reflect your changes. Now if you go to Setup > User Accounts and click the + button, you should see a nice little addition to the form with a Sip Account divider and three new fields. Nice, eh? Extend Parameters Currently you can search your User Accounts list using any of the User Account fields. It would be nice to be able to search with the Sip account fields we added as well. To do that, let s create the directory /path/to/xtuple-extensions/source/sip_account/client/widgets and there create the file parameter.js to extend XV.UserAccountListParameters. One again, you ll have to look this up. In the xTuple code you ll find the application s parameter.js in /path/to/xtuple/enyo-client/application/source/widgets. Search for the business object you are extending (for example, XV.UserAccount) and look for some combination of the business object name and Parameters . If there s more than one, try different ones. Not a very refined method, but it worked for me. Here s the content of our parameter.js:
XT.extensions.sip_account.initParameterWidget = function ()  
  var extensions = [
     kind: "onyx.GroupboxHeader", content: "_sipAccount".loc() ,
     name: "uri", label: "_uri".loc(), attr: "uri", defaultKind: "XV.InputWidget" ,
     name: "displayName", label: "_displayName".loc(), attr: "displayName", defaultKind: "XV.InputWidget" 
  ];
  XV.appendExtension("XV.UserAccountListParameters", extensions);
 ;
Node that I didn t include a search field for the password attribute for obvious reasons. Now once again, we package this new code addition by creating a /path/to/xtuple-extensions/source/sip_account/client/widgets/package.js file:
enyo.depends(
"parameter.js"
);
We also have to update /path/to/xtuple-extensions/source/sip_account/client/package.js:
enyo.depends(
"core.js",
"widgets"
"views"
);
Rebuild the extension (and restart the server) and go to Setup > User Accounts. Press the magnifying glass button on the upper left side of the screen and you ll see many options for filtering the User Accounts, among them the SIP Uri and Display Name. Extend List View You might want your new fields to show up on the list of User Accounts. There s a bit of an issue here because unlike what we did in workspace.js and parameter.js, we can t append new things to the list of UserAccounts with the funciton XV.appendExtension(args). First I tried overwriting the original UserAccountList, which works but it s far from ideal as this could result in a loss of data from the core implementation. After some discussion with the xTuple dev community, now there s a better alternative: Create the file /path/to/xtuple-extensions/source/sip_account/client/views/list.js and add the following:
1 var oldUserAccountListCreate = XV.UserAccountList.prototype.create;
2 
3 XV.UserAccountList.prototype.create = function ()  
4   oldUserAccountListCreate.apply(this, arguments);
5   this.createComponent(
6    kind: "XV.ListColumn", container: this.$.listItem, components: [
7         kind: "XV.ListAttr", attr: "uri" 
8    ] )
9  ;
To understand what I m doing, check out the XV.UserAccountList implementation in /path/to/xtuple/enyo-client/application/source/views/list.js the entire highlighted part. What we are doing is extending XV.UserAccountList through prototype-chaining ; this is how inheritance works with Enyo. In line 1 we create a prototype and in line 4 we inherit the features including original components array which the list is based on. We then create an additional component immitating the setup shown in XV.UserAccountList: An XV.ListColumn containing an XV.ListAttr, which should be placed in the XV.ListItem components array as is done with the existing columns (refer to implementation). Components can or should (?) have names which are used to access said components. You d refer to a specific component by the this.$.componentName hash. The components in XV.UserAccountList don t have names, so Enyo automatically names them (apparently) based on the kind name, for example something of the kind ListItem is named listItem. I found this at random after a lot of trial and error and it s not a bullet proof solution. Can be bettered. It s strange because if you encapsulate that code with
XT.extensions.sip_account.initList = function ()  
 //Code here
 ;
as is done with parameter.js and workspace.js (and in the xTuple tutorial you are supposed to do that with a new business object), it doesn t work. I have no idea why. This might be wrong or against xTuple coding norms; I will find out and update this post ASAP. But it does work this way. * shrugs * That said, as we ve created the list.js file, we need to ad it to our package by editing /path/to/xtuple-extensions/source/sip_account/client/views/package.js:
enyo.depends(
"list.js",
"workspace.js"
);
That s all. Rebuild the app and restart your server and when you select Setup > User Accounts in the web app you should see the Sip URI displayed on the User Accounts that have the Sip Account data. Add a new User Account to try this out.

24 July 2014

Martin Pitt: vim config for Markdown+LaTeX pandoc editing

I have used LaTeX and latex-beamer for pretty much my entire life of document and presentation production, i. e. since about my 9th school grade. I ve always found the LaTeX syntax a bit clumsy, but with good enough editor shortcuts to insert e. g. \begin itemize \item...\end itemize with just two keystrokes, it has been good enough for me. A few months ago a friend of mine pointed out pandoc to me, which is just simply awesome. It can convert between a million document formats, but most importantly take Markdown and spit out LaTeX, or directly PDF (through an intermediate step of building a LaTeX document and calling pdftex). It also has a template for beamer. Documents now look soo much more readable and are easier to write! And you can always directly write LaTeX commands without any fuss, so that you can use markdown for the structure/headings/enumerations/etc., and LaTeX for formulax, XYTex and the other goodies. That s how it should always should have been! So last night I finally sat down and created a vim config for it:
"-- pandoc Markdown+LaTeX -------------------------------------------
function s:MDSettings()
    inoremap <buffer> <Leader>n \note[item] <Esc>i
    noremap <buffer> <Leader>b :! pandoc -t beamer % -o %<.pdf<CR><CR>
    noremap <buffer> <Leader>l :! pandoc -t latex % -o %<.pdf<CR>
    noremap <buffer> <Leader>v :! evince %<.pdf 2>&1 >/dev/null &<CR><CR>
    " adjust syntax highlighting for LaTeX parts
    "   inline formulas:
    syntax region Statement oneline matchgroup=Delimiter start="\$" end="\$"
    "   environments:
    syntax region Statement matchgroup=Delimiter start="\\begin .* " end="\\end .* " contains=Statement
    "   commands:
    syntax region Statement matchgroup=Delimiter start=" " end=" " contains=Statement
endfunction
autocmd BufRead,BufNewFile *.md setfiletype markdown
autocmd FileType markdown :call <SID>MDSettings()
That gives me good enough (with some quirks) highlighting without trying to interpret TeX stuff as Markdown, and shortcuts for calling pandoc and evince. Improvements appreciated!

Matthew Palmer: First Step with Clojure: Terror

$ sudo apt-get install -y leiningen
[...]
$ lein new scratch
[...]
$ cd scratch
$ lein repl
Downloading: org/clojure/clojure/1.3.0/clojure-1.3.0.pom from repository central at http://repo1.maven.org/maven2
Transferring 5K from central
Downloading: org/sonatype/oss/oss-parent/5/oss-parent-5.pom from repository central at http://repo1.maven.org/maven2
Transferring 4K from central
Downloading: org/clojure/clojure/1.3.0/clojure-1.3.0.jar from repository central at http://repo1.maven.org/maven2
Transferring 3311K from central
[...]
Wait what? lein downloads some random JARs from a website over HTTP1, with, as far as far I can tell, no verification that what I m asking for is what I m getting (has nobody ever heard of Man-in-the-Middle attacks in Maven land?). It downloads a .sha1 file to (presumably) do integrity checking, but that s no safety net if I can serve you a dodgy .jar, I can serve you an equally-dodgy .sha1 file, too (also, SHA256 is where all the cool kids are at these days). Finally, jarsigner tells me that there s no signature on the .jar itself, either. It gets better, though. The repo1.maven.org site is served by the fastly.net2 pseudo-CDN3, which adds another set of points in the chain which can be subverted to hijack and spoof traffic. More routers, more DNS zones, and more servers. I ve seen Debian take a kicking more than once because packages aren t individually signed, or because packages aren t served over HTTPS. But at least Debian s packages can be verified by chaining to a signature made by a well-known, widely-distributed key, signed by two Debian Developers with very well-connected keys. This repository, on the other hand oy gevalt. There are OpenPGP (GPG) signatures available for each package (tack .asc onto the end of the .jar URL), but no attempt was made to download the signatures for the .jar I downloaded. Even if the signature was downloaded and checked, there s no way for me (or anyone) to trust the signature the signature was made by a key that s signed by one other key, which itself has no signatures. If I were an attacker, it wouldn t be hard for me to replace that key chain with one of my own devising. Even ignoring everyone living behind a government- or company-run intercepting proxy, and everyone using public wifi, it s pretty well common knowledge by now (thanks to Edward Snowden) that playing silly-buggers with Internet traffic isn t hard to do, and there s no shortage of evidence that it is, in fact, done on a routine basis by all manner of people. Serving up executable code to a large number of people, in that threat environment, with no way for them to have any reasonable assurance that code is trustworthy, is very disappointing. Please, for the good of the Internet, improve your act, Maven. Putting HTTPS on your distribution would be a bare minimum. There are attacks on SSL, sure, but they re a lot harder to pull off than sitting on public wifi hijacking TCP connections. Far better would be to start mandating signatures, requiring signature checks to pass, and having all signatures chain to a well-known, widely-trusted, and properly secured trust root. Signing all keys that are allowed to upload to maven.org with a maven.org distribution root key (itself kept in hardware and only used offline), and then verifying that all signatures chain to that key, wouldn t be insanely difficult, and would greatly improve the security of the software supply chain. Sure, it wouldn t be perfect, but don t make the perfect the enemy of the good. Cost-effective improvements are possible here. Yes, security is hard. But you don t get to ignore it just because of that, when you re creating an attractive nuisance for anyone who wants to own up a whole passel of machines by slipping some dodgy code into a widely-used package.
  1. To add insult to injury, it appears to ignore my http_proxy environment variable, and the repo1.maven.org server returns plain-text error responses with Content-Type: text/xml. But at this point, that s just icing on the shit cake.
  2. At one point in the past, my then-employer (a hosting provider) blocked Fastly s caching servers from their network because they took down a customer site with a massive number of requests to a single resource, and the incoming request traffic was indistinguishable from a botnet-sourced DDoS attack. The requests were coming from IP space registered to a number of different ISPs, with no distinguishing rDNS (184-106-82-243.static.cloud-ips.com doesn t help me to distinguish between I m a professionally-run distributed proxy and I m a pwned box here to hammer your site into the ground ).
  3. Pretty much all of the new breed of so-called CDNs aren t actually pro-actively distributing content, they re just proxies. That isn t a bad thing, per se, but I rather dislike the far-too-common practice of installing varnish (and perhaps mod_pagespeed, if they re providing advanced capabilities) on a couple of AWS instances, and hanging out your shingle as a CDN. I prefer a bit of truth in my advertising.

23 July 2014

Michael Prokop: Book Review: The Docker Book

Docker is an open-source project that automates the deployment of applications inside software containers. I m responsible for a docker setup with Jenkins integration and a private docker-registry setup at a customer and pre-ordered James Turnbull s The Docker Book a few months ago. Recently James he s working for Docker Inc released the first version of the book and thanks to being on holidays I already had a few hours to read it AND blog about it. :) (Note: I ve read the Kindle version 1.0.0 and all the issues I found and reported to James have been fixed in the current version already, jey.) The book is very well written and covers all the basics to get familiar with Docker and in my opinion it does a better job at that than the official user guide because of the way the book is structured. The book is also a more approachable way for learning some best practices and commonly used command lines than going through the official reference (but reading the reference after reading the book is still worth it). I like James approach with ENV REFRESHED_AT $TIMESTAMP for better controlling the cache behaviour and definitely consider using this in my own setups as well. What I wasn t aware is that you can directly invoke docker build $git_repos_url and further noted a few command line switches I should get more comfortable with. I also plan to check out the Automated Builds on Docker Hub. There are some references to further online resources, which is relevant especially for the more advanced use cases, so I d recommend to have network access available while reading the book. What I m missing in the book are best practices for running a private docker-registry in a production environment (high availability, scaling options, ). The provided Jenkins use cases are also very basic and nothing I personally would use. I d also love to see how other folks are using the Docker plugin, the Docker build step plugin or the Docker build publish plugin in production (the plugins aren t covered in the book at all). But I m aware that this are fast moving parts and specialised used cases upcoming versions of the book are already supposed to cover orchestration with libswarm, developing Docker plugins and more advanced topics, so I m looking forward to further updates of the book (which you get for free as existing customer, being another plus). Conclusion: I enjoyed reading the Docker book and can recommend it, especially if you re either new to Docker or want to get further ideas and inspirations what folks from Docker Inc consider best practices.

Tanguy Ortolo: GNU/Linux graphic sessions: suspending your computer

Major desktop environments such as Xfce or KDE have a built-in computer suspend feature, but when you use a lighter alternative, things are a bit more complicated, because basically: only root can suspend the computer. There used to be a standard solution to that, using a D-Bus call to a running daemon upowerd. With recent updates, that solution first stopped working for obscure reasons, but it could still be configured back to be usable. With newer updates, it stopped working again, but this time it seems it is gone for good:
$ dbus-send --system --print-reply \
            --dest='org.freedesktop.UPower' \
            /org/freedesktop/UPower org.freedesktop.UPower.Suspend
Error org.freedesktop.DBus.Error.UnknownMethod: Method "Suspend" with
signature "" on interface "org.freedesktop.UPower" doesn't exist
The reason seems to be that upowerd is not running, because it no longer provides an init script, only a systemd service. So, if you do not use systemd, you are left with one simple and stable solution: defining a sudo rule to start the suspend or hibernation process as root. In /etc/sudoers.d/power:
%powerdev ALL=NOPASSWD: /usr/sbin/pm-suspend, \
                        /usr/sbin/pm-suspend-hybrid, \
                        /usr/sbin/pm-hibernate
That allows members of the powderdev group to run sudo pm-suspend, sudo pm-suspend-hybrid and sudo pm-hibernate, which can be used with a key binding manager such as your window manager's or xbindkeys. Simple, efficient, and contrary to all that ever-changing GizmoKit and whatsitd stuff, it has worked and will keep working for years.

22 July 2014

Neil Williams: Validating ARMMP device tree blobs

I ve done various bits with ARMMP and LAVA on this blog already, usually waiting until I ve got all the issues ironed out before writing it up. However, this time I m just going to do a dump of where it s at, how it works and what can be done. I m aware that LAVA can seem mysterious at first, the package description has improved enormously recently, thanks to exposure in Debian: LAVA is a continuous integration system for deploying operating systems onto physical and virtual hardware for running tests. Tests can be simple boot testing, bootloader testing and system level testing, although extra hardware may be required for some system tests. Results are tracked over time and data can be exported for further analysis. The LAVA documentation has a glossary of terms like result bundle and all the documentation is also available in the lava-server-doc package. The goal is to validate the dtbs built for the Debian ARMMP kernel. One of the most accessible ways to get the ARMMP kernel onto a board for testing is tftp using the Debian daily DI builds. Actually using the DI initrd can come later, once I ve got a complete preseed config so that the entire install can be automated. (There are some things to sort out in LAVA too before a full install can be deployed and booted but those are at an early stage.) It s enough at first to download the vmlinuz which is common to all ARMMP deployments, supply the relevant dtb, partner those with a minimal initrd and see if the board boots. The first change comes when this process is compared to how boards are commonly tested in LAVA with a zImage or uImage and all/most of the modules already built in. Packaged kernels won t necessarily raise a network interface or see the filesystem without modules, so the first step is to extend a minimal initramfs to include the armmp modules.
apt install pax u-boot-tools
The minimal initramfs I selected is one often used within LAVA:
wget http://images.armcloud.us/lava/common/linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot
It has a u-boot header added, as most devices using this would be using u-boot and this makes it easier to debug boot failures as the initramfs doesn t need to have the header added, it can simply be downloaded to a local directory and passed to the board as a tftp location. To modify it, the u-boot header needs to be removed. Rather than assuming the size, the u-boot tools can (indirectly) show the size:
$ ls -l linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot
-rw-r--r-- 1 neil neil  5179571 Nov 26  2013 linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot
$ mkimage -l linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot 
Image Name:   linaro-image-minimal-initramfs-g
Created:      Tue Nov 26 22:30:49 2013
Image Type:   ARM Linux RAMDisk Image (gzip compressed)
Data Size:    5179507 Bytes = 5058.11 kB = 4.94 MB
Load Address: 00000000
Entry Point:  00000000
Referencing http://www.omappedia.com/wiki/Development_With_Ubuntu, the header size is the file size minus the data size listed by mkimage.
5179571 - 5179507 == 64
So, create a second file without the header:
dd if=linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot of=linaro-image-minimal-initramfs-genericarmv7a.cpio.gz skip=64 bs=1
decompress it
gunzip linaro-image-minimal-initramfs-genericarmv7a.cpio.gz
Now for the additions
dget http://ftp.uk.debian.org/debian/pool/main/l/linux/linux-image-3.14-1-armmp_3.14.12-1_armhf.deb
(Yes, this process will need to be repeated when this package is rebuilt, so I ll want to script this at some point.)
dpkg -x linux-image-3.14-1-armmp_3.14.12-1_armhf.deb kernel-dir
cd kernel-dir
Pulling in the modules we need for most needs, comes thanks to a script written by the Xen folks. The set is basically disk, net, filesystems and LVM.
find lib -type d -o -type f -name modules.\*  -o -type f -name \*.ko  \( -path \*/kernel/lib/\* -o  -path \*/kernel/crypto/\* -o  -path \*/kernel/fs/mbcache.ko -o  -path \*/kernel/fs/ext\* -o  -path \*/kernel/fs/jbd\* -o  -path \*/kernel/drivers/net/\* -o  -path \*/kernel/drivers/ata/\* -o  -path \*/kernel/drivers/scsi/\* -o -path \*/kernel/drivers/md/\* \)   pax -x sv4cpio -s '%lib%/lib%' -d -w >../cpio
gzip -9f cpio
original Xen script (GPL-3+) I found it a bit confusing that i is used for extract by cpio, but that s how it is. Extract the minimal initramfs to a new directory:
sudo cpio -id < ../linaro-image-minimal-initramfs-genericarmv7a.cpio
Extract the new cpio into the same location. (Yes, I could do this the other way around and pipe the output of find into the already extracted location but that's for when I get a script to do this):
sudo cpio --no-absolute-filenames -id < ../ramfs/cpio
CPIO Manual Use newc format, the new (SVR4) portable format, which supports file systems having more than 65536 i-nodes. (4294967295 bytes)
(41M)
find .   cpio -H newc -o > ../armmp-image.cpio
... and add the u-boot header back:
mkimage -A arm -T ramdisk -C none -d armmp-image.cpio.gz debian-armmp-initrd.cpio.gz.u-boot
Now what? Now send the combination to LAVA and test it. Results bundle for a local LAVA test job using this technique. (18k) submission JSON - uses file:// references, so would need modification before being submitted to LAVA elsewhere. complete log of the test job (72k) Those familiar with LAVA will spot that I haven't optimised this job, it boots the ARMMP kernel into a minimal initramfs and then expects to find apt and other tools. Actual tests providing useful results would use available tools, add more tools or specify a richer rootfs. The tests themselves are very quick (the job described above took 3 minutes to run) and don't need to be run particularly often, just once per board type per upload of the ARMMP kernel. LAVA can easily run those jobs in parallel and submission can be automated using authentication tokens and the lava-tool CLI. lava-tool can be installed without lava-server, so can be used in hooks for automated submissions. Extensions That's just one DTB and one board. I have a range of boards available locally: * iMX6Q Wandboard (used for this test)
* iMX.53 Quick Start Board (needs updated u-boot)
* Beaglebone Black
* Cubie2
* CubieTruck
* arndale (no dtb?)
* pandaboard Other devices available could involve ARMv7 devices hosted at www.armv7.com and validation.linaro.org - as part of a thank you to the Debian community for providing the OS which is (now) running all of the LAVA infrastructure. That doesn't cover all of the current DTBs (and includes many devices which have no DTBs) so there is plenty of scope for others to get involved. Hopefully, the above will help get people started with a suitable kernel+dtb+initrd and I'd encourage anyone interested to install lava-server and have a go at running test jobs based on those so that we start to build data about as many of the variants as possible. (If anyone in DI fancies producing a suitable initrd with modules alongside the DI initrd for armhf builds, or if anyone comes up with a patch for DI to do that, it would help enormously.) This will at least help Debian answer the question of what the Debian ARMMP package can actually support. For help on LAVA, do read through the documentation and then come to us at #linaro-lava or the linaro-validation mailing list or file bugs in Debian: reportbug lava-server. , so you can ask me. I'm giving one talk on the LAVA software and there will be a BoF on validation and CI in Debian.

Russell Coker: Public Lectures About FOSS

Eventbrite I ve recently started using the Eventbrite Web site [1] and the associated Eventbrite Android app [2] to discover public events in my area. Both the web site and the Android app lack features for searching (I d like to save alerts for my accounts and have my phone notify me when new events are added to their database) but it is basically functional. The main issue is content, Eventbrite has a lot of good events in their database (I ve got tickets for 6 free events in the next month). I assume that Eventbrite also has many people attending their events, otherwise the events wouldn t be promoted there. At this time I haven t compared Eventbrite to any similar services, Eventbrite events have taken up much of my available time for the next 6 weeks (I appreciate the button on the app to add an entry to my calendar) so I don t have much incentive to find other web sites that list events. I would appreciate comments from users of competing event registration systems and may write a post in future comparing different systems. Also I have only checked for events in Melbourne, Australia as I don t have any personal interest in events in other places. For the topic of this post Eventbrite is good enough, it meets all requirements for Melbourne and I m sure that if it isn t useful in other cities then there are competing services. I think that we need to have free FOSS events announced through Eventbrite. We regularly have experts in various fields related to FOSS visiting Melbourne who give a talk for the Linux Users of Victoria (and sometimes other technical groups). This is a good thing but I think we could do better. Most people in Melbourne probably won t attend a LUG meeting and if they did they probably wouldn t find it a welcoming experience. Also I recommend that anyone who is looking for educational things to do in Melbourne visit the Eventbrite web site and/or install the Android app. Accessible Events I recently attended an Eventbrite event where a professor described the work of his research team, it was a really good talk that made the topic of his research accessible to random members of the public like me. Then when it came to question time the questions were mostly opinion pieces disguised as questions which used a lot of industry specific jargon and probably lost the interest of most people in the audience who wasn t from the university department that hosted the lecture. I spent the last 15 minutes in that lecture hall reading Wikipedia and resisted the temptation to load an Android game. Based on this lecture (and many other lectures I ve seen) I get the impression that when the speaker or the MC addresses a member of the audience by name (EG John Smith has a question ) then it s strongly correlated with a low quality question. See my previous post about the Length of Conference Questions for more on this topic [3]. It seems to me that when running a lecture everyone involved has to agree about whether it s a public lecture (IE one that is for any random people) as opposed to a society meeting (which while free for anyone to attend in the case of a LUG is for people with specific background knowledge). For a society meeting (for want of a better term) it s OK to assume a minimum level of knowledge that rules out some people. If 5% of the audience of a LUG don t understand a lecture that doesn t necessarily mean it s a bad lecture, sometimes it s not possible to give a lecture that is easily understood by those with the least knowledge that also teaches the most experienced members of the audience. For a public lecture the speaker has to give a talk for people with little background knowledge. Then the speaker and/or the MC have to discourage or reject questions that are for a higher level of knowledge. As an example of how this might work consider the case of an introductory lecture about how an OS kernel works. When one of the experienced Linux kernel programmers visits Melbourne we could have an Eventbrite event organised for a lecture introducing the basic concepts of an OS kernel (with Linux as an example). At such a lecture any questions about more technical topics (such as specific issues related to compilers, drivers, etc) could be met with we are having a meeting for more technical people at the Linux Users of Victoria meeting tomorrow night or we are having coffee at a nearby cafe afterwards and you can ask technical questions there . Planning Eventbrite Events When experts in various areas of FOSS visit Melbourne they often offer a talk for LUV. For any such experts who read this post please note that most lectures at LUV meetings are by locals who can reschedule, so if you are only in town for a short time we can give you an opportunity to speak at short notice. I would like to arrange to have some of those people give a talk aimed at a less experienced audience which we can promote through Eventbrite. The venue for LUV talks (Melbourne University 7PM on the first Tuesday of the month) might not work for all speakers so we need to find a sponsor for another venue. I will contact Linux companies that are active in Melbourne and ask whether they would be prepared to sponsor the venue for such a talk. The fallback option would be to have such a lecture at a LUV meeting. I will talk to some of the organisers of science and technology events advertised on Eventbrite and ask why they chose the times that they did. Maybe they have some insight into which times are best for getting an audience. Also I will probably get some idea of the best times by just attending many events and observing the attendance. I think that the aim of an Eventbrite event is to attract delegates who wouldn t attend other meetings, so it is a priority to choose a suitable time and place. Finally please note that while I am a member of the LUV committee I m not representing LUV in this post. My aim is that community feedback on this post will help me plan such events. I will discuss this with the LUV committee after I get some comments here. Please comment if you would like to give such a public lecture, attend such a lecture, or if you just have any general ideas.

21 July 2014

Steve Kemp: An alternative to devilspie/devilspie2

Recently I was updating my dotfiles, because I wanted to ensure that media-players were "always on top", when launched, as this suits the way I work. For many years I've used devilspie to script the placement of new windows, and once I googled a recipe I managed to achieve my aim. However during the course of my googling I discovered that devilspie is unmaintained, and has been replaced by something using Lua - something I like. I'm surprised I hadn't realized that the project was dead, although I've always hated the configuration syntax it is something that I've used on a constant basis since I found it. Unfortunately the replacement, despite using Lua, and despite being functional just didn't seem to gell with me. So I figured "How hard could it be?". In the past I've written softare which iterated over all (visible) windows, and obviously I'm no stranger to writing Lua bindings. However I did run into a snag. My initial implementation did two things: This worked. This worked well. This worked too well. The problem I ran into was that if I wrote something like "Move window 'emacs' to desktop 2" that action would be applied, over and over again. So if I launched emacs, and then manually moved the window to desktop3 it would jump back! In short I needed to add a "stop()" function, which would cause further actions against a given window to cease. (By keeping a linked list of windows-to-ignore, and avoiding processing them.) The code did work, but it felt wrong to have an ever-growing linked-list of processed windows. So I figured I'd look at the alternative - the original devilspie used libwnck to operate. That library allows you to nominate a callback to be executed every time a new window is created. If you apply your magic only on a window-create event - well you don't need to bother caching prior-windows. So in conclusion : I think my code is better than devilspie2 because it is smaller, simpler, and does things more neatly - for example instead of a function to get geometry and another to set it, I use one. (e.g. "xy()" returns the position of a window, but xy(3,3) sets it.). kpie also allows you to run as a one-off job, and using the simple primitives I wrote a file to dump your windows, and their size/placement, which looks like this:
shelob ~/git/kpie $ ./kpie --single ./samples/dump.lua
-- Screen width : 1920
-- Screen height: 1080
..
if ( ( window_title() == "Buddy List" ) and
     ( window_class() == "Pidgin" ) and
     ( window_application() == "Pidgin" ) ) then
     xy(1536,24 )
     size(384,1032 )
     workspace(2)
end
if ( ( window_title() == "feeds" ) and
     ( window_class() == "Pidgin" ) and
     ( window_application() == "Pidgin" ) ) then
     xy(1,24 )
     size(1536,1032 )
     workspace(2)
end
..
As you can see that has dumped all my windows, along with their current state. This allows a simple starting-point - Configure your windows the way you want them, then dump them to a script file. Re-run that script file and your windows will be set back the way they were! (Obviously there might be tweaks required.) I used that starting-point to define a simple recipe for configuring pidgin, which is more flexible than what I ever had with pidgin, and suits my tastes. Bug-reports welcome.

Next.

Previous.