Search Results: "Jonathan McDowell"

21 September 2015

Jonathan McDowell: Getting a Dell E7240 working with a dock + a monitor

I have a Dell E7240. I m pretty happy with it - my main complaint is that it has a very shiny screen, and that seems to be because it s the touchscreen variant. While I don t care about that feature I do care about the fact it means I get FullHD in 12.5 Anyway. I ve had issues with using a dock and an external monitor with the laptop for some time, including getting so far as mentioning the problems on the appropriate bug tracker. I ve also had discussions with a friend who has the same laptop with the same issues, and has some time trying to get it reliably work. However up until this week I haven t had a desk I m sitting at for any length of time to use the laptop, so it s always been low priority for me. Today I sat down to try and figure out if there had been any improvement. Firstly I knew the dock wasn t at fault. A Dell E6330 works just fine with multiple monitors on the same dock. The E6330 is Ivybridge, while the E7240 is Haswell, so I thought potentially there might be an issue going on there. Further digging revealed another wrinkle I hadn t previously been aware of; there is a DisplayPort Multi-Stream Transport (MST) hub in play, in particular a Synaptics VMM2320. Dell have a knowledge base article about Multiple external display issues when docked with a Latitude E7440/E7240 which suggests a BIOS update (I was already on A15) and a firmware update for the MST HUB. Sadly the firmware update is Windows only, so I had to do a suitable dance to be able to try and run it. I then discovered that the A05 update refused to work, complaining I had an invalid product ID. The A04 update did the same. The A01 update thankfully succeeded and told me it was upgrading from 2.00.002 to 2.15.000. After that had completed (and I d power cycled to switch to the new firmware) I tried A05 again and this time it worked and upgraded me to 2.22.000. Booting up Linux again I got further than before; it was definitely detecting that there was a monitor but it was very unhappy with lots of [drm:intel_dp_start_link_train] *ERROR* too many full retries, give up errors being logged. This was with 4.2, and as I d been meaning to try 4.3-rc2 I thought this was a good time to give it a try. Lo and behold, it worked! Even docking and undocking does what I d expect, with the extra monitor appearing / disappearing as you d expect. Now, I m not going to assume this means it s all happy, as I ve seen this sort-of work in the past, but the clue about MST, the upgrade of that firmware (and noticing that it made things better under Windows as well) and the fact that there have been improvements in the kernel s MST support according to the post 4.2 log gives me some hope that things will be better from here on.

24 August 2015

Jonathan McDowell: Random post-DebConf 15 thoughts

There are a bunch of things I mean to blog about, but as I have just got fully home from Heidelberg and DebConf15 this afternoon that seems most appropriate to start with. It s a bit of a set of disjoint thoughts, but I figure I should write them down while they re in my head. DebConf is an interesting conference. It s the best opportunity the Debian project has every year to come together and actually spend a decent amount of time with each other. As a result it s a fairly full on experience, with lots of planned talks as a basis and a wide range of technical discussions and general social interaction filling in whatever gaps are available. I always find it a thoroughly enjoyable experience, but equally I m glad to be home and doing delightfully dull things like washing my clothes and buying fresh milk. I have always been of the opinion that the key aspect of DebConf is the face time. It was thus great to see so many people there - we were told several times that this was the largest DebConf so far (~ 570 people IIRC). That s good in the sense that it meant I got to speak to a lot of people (both old friends and new), but does mean that there are various people I know I didn t spend enough, or in some cases any, time with. My apologies, but I think many of us were in the same situation. I don t feel it made the conference any less productive for me - I managed to get a bunch of hacking done, discuss a number of open questions in person with various people and get pulled into various interesting discussions I hadn t expected. In short, a typical DebConf. Also I d like to say that the venue worked out really well. I ll admit I was dubious when I heard it was in a hostel, but it was well located (about a 30 minute walk into town, and a reasonable bus service available from just outside the door), self-contained with decent facilities (I m a big believer in having DebConf talks + accommodation be as close as possible to each other) and the room was much better than expected (well, aside from the snoring but I can t blame the DebConf organisers for that). One of the surprising and interesting things for me that was different from previous DebConfs was the opportunity to have more conversations with a legal leaning. I expect to go to DebConf and do OpenPGP/general crypto related bits. I wasn t expecting affirmation about the things I have learnt on my course over the past year, in terms of feeling that I could use that knowledge in the process of helping Debian. It provided me with some hope that I ll be able to tie my technology and law skills together in a way that I will find suitably entertaining (as did various conversations where people expressed significant interest in the crossover). Next year is in Cape Town, South Africa. It s a long way (though I suppose no worse than Portland and I get to stay in the same time zone), and a quick look at flights indicates they re quite expensive at the moment. The bid presentation did look pretty good though so as soon as the dates are confirmed (I believe this will happen as soon as there are signed contracts in place) I ll take another look at flights. In short, excellent DebConf, thanks to the organisers, lovely to see everyone I managed to speak to, apologies to those of you I didn t manage to speak to. Hopefully see you in Cape Town next year.

11 August 2015

Jonathan McDowell: Programming the FST-01 (gnuk) with a Bus Pirate + OpenOCD

Last year at DebConf14 Lucas authorized the purchase of a handful of gnuk devices, one of which I obtained. At the time it only supported 2048 bit RSA keys. I took a look at what might be involved in adding 4096 bit support during DebConf and managed to brick my device several times in doing so. Thankfully gniibe was on hand with his STLinkV2 to help me recover. However subsequently I was loathe to experiment further at home until I had a suitable programmer. As it is this year has been busy and the 1.1.x release train is supposed to have 4K RSA (as well as ECC) support. DebConf15 is coming up and I felt I should finally sort out playing with the device properly. I still didn t have a suitable programmer. Or did I? Could my trusty Bus Pirate help? The FST-01 has an STM32F103TB on it. There is an exposed SWD port. I found a few projects that claimed to do SWD with a Bus Pirate - Will Donnelly has a much cloned Python project, the MC HCK project have a programmer in Ruby and there s LibSWD though that s targeted to smarter programmers. None of them worked for me; I could get the Python bits as far as correctly doing the ID of the device, but not reading the option bytes or successfully flashing (though I did manage an erase). Enter the old favourite, OpenOCD. This already has SWD support and there s an outstanding commit request to add Bus Pirate support. NodoNogard has a post on using the ST-Link/V2 with OpenOCD and the FST-01 which provided some useful pointers. I grabbed the patch from Gerrit, applied it to OpenOCD git and built an openocd.cfg that contained:
source [find interface/buspirate.cfg]
buspirate_port /dev/ttyUSB0
buspirate_vreg 1
buspirate_mode normal
transport select swd
source [find target/stm32f1x.cfg]
My BP has the Seeed Studio probe cable, so my hookups look like this: Bus Pirate + FST-01 SWD connection That s BP MOSI (grey) to SWD IO, BP CLK (purple) to SWD CLK, BP 3.3V (red) to FST-01 PWR and BP GND (brown) to FST-01 GND. Once that was done I fired up OpenOCD in one terminal and did the following in another:
$ telnet localhost 4444
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Open On-Chip Debugger
> reset halt
target state: halted
target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0xfffffffe msp: 0xfffffffc
Info : device id = 0x20036410
Info : SWD IDCODE 0x1ba01477
Error: Failed to read memory at 0x1ffff7e2
Warn : STM32 flash size failed, probe inaccurate - assuming 128k flash
Info : flash size = 128kbytes
> stm32f1x unlock 0
Device Security Bit Set
stm32x unlocked.
INFO: a reset or power cycle is required for the new settings to take effect.
> reset halt
target state: halted
target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0xfffffffe msp: 0xfffffffc
> flash write_image erase /home/noodles/checkouts/gnuk/src/build/gnuk.elf
auto erase enabled
wrote 109568 bytes from file /home/noodles/checkouts/gnuk/src/build/gnuk.elf in 95.055603s (1.126 KiB/s)
> stm32f1x lock 0
stm32x locked
> reset halt
target state: halted
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x08000280 msp: 0x20005000
Then it was a matter of disconnecting the gnuk from the BP, plugging it into my USB port and seeing it come up successfully:
usb 1-2: new full-speed USB device number 11 using xhci_hcd
usb 1-2: New USB device found, idVendor=234b, idProduct=0000
usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-2: Product: Gnuk Token
usb 1-2: Manufacturer: Free Software Initiative of Japan
usb 1-2: SerialNumber: FSIJ-1.1.7-87063020
usb 1-2: ep 0x82 - rounding interval to 1024 microframes, ep desc says 2040 microframes
More once I actually have a 4K key loaded on it.

21 July 2015

Jonathan McDowell: Recovering a DGN3500 via JTAG

Back in 2010 when I needed an ADSL2 router in the US I bought a Netgear DGN3500. It did what I wanted out of the box and being based on a MIPS AR9 (ARX100) it seemed likely OpenWRT support might happen. Long story short I managed to overwrite u-boot (the bootloader) while flashing a test image I d built. I ended up buying a new router (same model) to get my internet connection back ASAP and never getting around to fully fixing the broken one. Until yesterday. Below is how I fixed it; both for my own future reference and in case it s of use any any other unfortunate soul. The device has clear points for serial and JTAG and it was easy enough (even with my basic soldering skills) to put a proper header on. The tricky bit is that the flash is connected via SPI, so it s not just a matter of attaching JTAG, doing a scan and reflashing from the JTAG tool. I ended up doing RAM initialisation, then copying a RAM copy of u-boot in and then using that to reflash. There may well have been a better way, but this worked for me. For reference the failure mode I saw was an infinitely repeating:
ROM VER: 1.1.3
CFG 05
My JTAG device is a Bus Pirate v3b which is much better than the parallel port JTAG device I built the first time I wanted to do something similar. I put the latest firmware (6.1) on it. All of this was done from my laptop, which runs Debian testing (stretch). I used the OpenOCD 0.9.0-1+b1 package from there. Daniel Schwierzeck has some OpenOCD scripts which include a target definition for the ARX100. I added a board definition for the DGN3500 (I ve also send Daniel a patch to add this to his repo). I tied all of this together with an openocd.cfg that contained:
source [find interface/buspirate.cfg]
buspirate_port /dev/ttyUSB1
buspirate_vreg 0
buspirate_mode normal
buspirate_pullup 0
reset_config trst_only
source [find openocd-scripts/target/arx100.cfg]
source [find openocd-scripts/board/dgn3500.cfg]
gdb_flash_program enable
gdb_memory_map enable
gdb_breakpoint_override hard
I was then able to power on the router and type dgn3500_ramboot into the OpenOCD session. This fetched my RAM copy of u-boot from dgn3500_ram/u-boot.bin, copied it into the router s memory and started it running. From there I had a u-boot environment with access to the flash commands and was able to restore the original Netgear image (and once I was sure that was working ok I subsequently upgraded to the Barrier Breaker OpenWRT image).

29 June 2015

Jonathan McDowell: What Jonathan Did Next

While I mentioned last September that I had failed to be selected for an H-1B and had been having discussions at DebConf about alternative employment, I never got around to elaborating on what I d ended up doing. Short answer: I ended up becoming a law student, studying for a Masters in Legal Science at Queen s University Belfast. I ve just completed my first year of the 2 year course and have managed to do well enough in the 6 modules so far to convince myself it wasn t a crazy choice. Longer answer: After Vello went under in June I decided to take a couple of months before fully investigating what to do next, largely because I figured I d either find something that wanted me to start ASAP or fail to find anything and stress about it. During this period a friend happened to mention to me that the applications for the Queen s law course were still open. He happened to know that it was something I d considered before a few times. Various discussions (some of them over gin, I ll admit) ensued and I eventually decided to submit an application. This was towards the end of August, and I figured I d also talk to people at DebConf to see if there was anything out there tech-wise that I could get excited about. It turned out that I was feeling a bit jaded about the whole tech scene. Another friend is of the strong opinion that you should take a break at least every 10 years. Heeding her advice I decided to go ahead with the law course. I haven t regretted it at all. My initial interest was largely driven by a belief that there are too few people who understand both tech and law. I started with interests around intellectual property and contract law as well as issues that arise from trying to legislate for the global nature of most tech these days. However the course is a complete UK qualifying degree (I can go on to do the professional qualification in NI or England & Wales) and the first year has been about public law. Which has been much more interesting than I was expecting (even, would you believe it, EU law). Especially given the potential changing constitutional landscape of the UK after the recent general election, with regard to talk of repeal of the Human Rights Act and a referendum on exit from the EU. Next year will concentrate more on private law, and I m hoping to be able to tie that in better to what initially drove me to pursue this path. I m still not exactly sure which direction I ll go once I complete the course, but whatever happens I want to keep a linkage between my skill sets. That could be either leaning towards the legal side but with the appreciation of tech, returning to tech but with the appreciation of the legal side of things or perhaps specialising further down an academic path that links both. I guess I ll see what the next year brings. :)

29 May 2015

Norbert Preining: USB stick update: Debian is back, plus GRML

In the previous post on the multi-boot usb stick, Debian had to be dropped, due to missing support of iso-loading in the Debian installer iso images. Thanks to helpful comments from Jonathan McDowell and private emails, we found a way around this. In addition to having again an installable Debian image, I also added GRML support. multi-usb-disk There are a few tricks necessary to get this going, though. Let us start with the easy part, grml: Grml Quoting from the home page of Grml
Grml provides an excellent selection of software packages for installation, deployment and system rescue. You ll get sysadmin s favourite tools. Choose between a x86 and an amd64 version depending on your needs or just grab the multiboot ISO grml96, providing 64bit and 32bit ISOs in one single place.
I went for the 64bit version Version 2014.11 (named Gschistigschasti , a typical Austrian expression for fuss ), as most computers nowadays are anyway 64bit capable, downloaded the image file grml64-full_2014.11.iso, and saved it into my USB stick s boot/iso/ directory, where there are already the other iso images. Then, with the help of the Grml web page which provides guidance how to boot from ISO, I added the following entry to my grub.cfg:
submenu "GRML 2014.11 ---> "  
  menuentry "Grml Rescue System 64bit"  
        iso_path="/boot/iso/grml64-full_2014.11.iso"
        export iso_path
        loopback loop (hd0,1)$iso_path
        set root=(loop)
        kernelopts=" ssh=SOME_PASSWORD_HERE toram  "
        export kernelopts
        configfile /boot/grub/loopback.cfg
    
 
Note that if you add the ssh=SOME_PASSWORD_HERE part, an ssh server is started and the root login password will be set to SOME_PASSWORD_HERE, so choose it wisely (and different from your normal passwords!). If you don t want a running ssh server, simply remove the ssh parameter. That s it, booting into Grml takes quite some time, but worked out nicely. Debian Jessie installation As mentioned in the previous post, booting Debian/Jessie installation images via any method didn t work, since the iso images is never found. It turned out that the current installer iso images do not contain the iso-scan package, which is responsible for searching and loading of iso images. But with a small trick one can overcome this: One needs to replace the initrd that is on the ISO image with one that contains the iso-scan package. And we do not need to create these initrd by ourselves, but simply use the ones from hd-media type installer. I downloaded the following four gzipped initrds from one of the Debian mirrors: i386/initrd text mode, i386/initrd gui mode, amd64/initrd text mode, amd64/initrd gui mode, and put them into the USB stick s boot/debian/install.386, boot/debian/install.386/gtk, boot/debian/install.amd, boot/debian/install.amd/gtk, respectively. Finally, I added entries similar to this one (rest see the grub.cfg file):
submenu "Debian 8.0 Jessie NetInstall ---> "  
    set isofile="/boot/iso/firmware-8.0.0-amd64-i386-netinst.iso"
    menuentry '64 bit Install'  
        set background_color=black
        loopback loop (hd0,1)$isofile
        linux    (loop)/install.amd/vmlinuz iso-scan/ask_second_pass=true iso-scan/filename=$isofile vga=788 -- quiet 
        initrd   /boot/debian/install.amd/initrd.gz
     
    ...
 
Again an important point, don t forget the two kernel command line options: iso-scan/ask_second_pass=true iso-scan/filename=$isofile, otherwise you probably will have to make the installer scan all disks and drives completely, which might take ages. With that in place, the stick now is back to full working, plus an added Grml. Thanks everyone for the suggestions.

21 May 2015

Jonathan McDowell: I should really learn systemd

As I slowly upgrade all my machines to Debian 8.0 (jessie) they re all ending up with systemd. That s fine; my laptop has been running it since it went into testing whenever it was. Mostly I haven t had to care, but I m dimly aware that it has a lot of bits I should learn about to make best use of it. Today I discovered systemctl is-system-running. Which I m not sure why I d use it, but when I ran it it responded with degraded. That s not right, thought I. How do I figure out what s wrong? systemctl --state=failed turned out to be the answer.
# systemctl --state=failed
  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
  systemd-modules-load.service loaded failed failed Load Kernel Modules
LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
Ok, so it s failed to load some kernel modules. What s it trying to load? systemctl status -l systemd-modules-load.service led me to /lib/systemd/systemd-modules-load which complained about various printer modules not being able to be loaded. Turned out this was because CUPS had dropped them into /etc/modules-load.d/cups-filters.conf on upgrade, and as I don t have a parallel printer I hadn t compiled up those modules. One of my other machines had also had an issue with starting up filesystem quotas (I think because there d been some filesystems that hadn t mounted properly on boot - my fault rather than systemd). Fixed that up and then systemctl is-system-running started returning a nice clean running. Now this is probably something that was silently failing back under sysvinit, but of course nothing was tracking that other than some output on boot up. So I feel that I ve learnt something minor about systemd that actually helped me cleanup my system, and sets me in better stead for when something important fails.

18 May 2015

Jonathan McDowell: Stepping down from SPI

I was first elected to the Software in the Public Interest board back in 2009. I was re-elected in 2012. This July I am up for re-election again. For a variety of reasons I ve decided not to stand; mostly a combination of the fact that I think 2 terms (6 years) is enough in a single stretch and an inability to devote as much time to the organization as I d like. I mentioned this at the May board meeting. I m planning to stay involved where I can. My main reason for posting this here is to cause people to think about whether they might want to stand for the board. Nominations open on July 1st and run until July 13th. The main thing you need to absolutely commit to is being able to attend the monthly board meeting, which is held on IRC at 20:30 UTC on the second Thursday of the month. They tend to last at most 30 minutes. Of course there s a variety of tasks that happen in the background, such as answering queries from prospective associated projects or discussing ongoing matters on the membership or board lists depending on circumstances. It s my firm belief that SPI do some very important work for the Free software community. Few people realise the wide variety of associated projects. SPI offload the boring admin bits around accepting donations and managing project assets (be those machines, domains, trademarks or whatever), leaving those projects able to concentrate on the actual technical side of things. Most project members don t realise the involvement of SPI, and that s largely a good thing as it indicates the system is working. However it also means that there can sometimes be a lack of people wanting to stand at election time, and an absence of diversity amongst the candidates. I m happy to answer questions of anyone who might consider standing for the board; #spi on irc.oftc.net is a good place to ask them - I am there as Noodles.

31 March 2015

Jonathan McDowell: Shipping my belongings across the globe

I previously wrote about tracking a ship around the world, but never followed up with the practical details involved with shipping my life from the San Francisco Bay Area back to Belfast. So here they are, in the hope they provide a useful data point for anyone considering a similar move. Firstly, move out. I was in a one bedroom apartment in Fremont, CA. At the time I was leaving the US I didn t have anywhere for my belongs to go - the hope was I d be back in the Bay Area, but there was a reasonable chance I was going to end up in Belfast or somewhere in England. So on January 24th 2014 I had my all of my belongings moved out and put into storage, pending some information about where I might be longer term. When I say all of my belongings I mean that; I took 2 suitcases and everything else went into storage. That means all the furniture for probably a 2 bed apartment (I d moved out of somewhere a bit larger) - the US doesn t really seem to go in for the concept of a furnished lease the same way as the UK does. I had deliberately picked a moving company that could handle the move out, the storage and the (potential) shipping. They handed off to a 3rd party for the far end bit, but that was to be expected. Having only one contact to deal with throughout the process really helped. Fast forward 8 months and on September 21st I contacted my storage company to ask about getting some sort of rough shipping quote and timescales to Belfast. The estimate came back as around a 4-6 week shipping time, which was a lot faster than I was expecting. However it turned out this was the slow option. On October 27th (delay largely due to waiting for confirmation of when I d definitely have keys on the new place) I gave the go ahead. Container pickup (I ended up with exclusive use of a 20ft container - not quite full, but not worth part shipment) from the storage location was originally due on November 7th. Various delays at the Port of Oakland meant this didn t happen until November 17th. It then sat in Oakland until December 2nd. At that point the ETA into Southampton was January 8th. Various other delays, including a week off the coast of LA (yay West Coast Port Backups) meant that the ship finally arrived in Southampton on January 13th. It then had to get to Belfast and clear customs. On January 22nd 2015, 2 days shy of a year since I d seen them, my belongings and I were reunited. So, on the face of it, the actual time on the ship was only slightly over 6 weeks, but all of the extra bits meant that the total time from Ship it to I have it was nearly 3 months. Which to be honest is more like what I was expecting. The lesson: don t forget to factor in delays at every stage. The relocation cost in the region of US$8000. It was more than I d expected, but far cheaper than the cost of buying all my furniture again (plus the fact there were various things I couldn t easily replace that were in storage). That cost didn t cover the initial move into storage or the storage fees - it covered taking things out, packing them up for shipment and everything after that. Including delivery to a (UK) 3rd floor apartment at the far end and insurance. It s important to note that I d included this detail before shipment - the quote specifically mentioned it, which was useful when the local end tried to levy an additional charge for the 3rd floor aspect. They were fine once I showed them the quote as including that detail. Getting an entire apartment worth of things I hadn t seen in so long really did feel a bit like a second Christmas. I d forgotten a lot of the things I had, and it was lovely to basically get a home in a container delivered.

21 January 2015

Jonathan McDowell: Moving to Jekyll

I ve been meaning to move away from Movable Type for a while; they no longer provide the Open Source variant, I ve had some issues with the commenting side of things (more the fault of spammers than Movable Type itself) and there are a few minor niggles that I wanted to resolve. Nothing has been particularly pressing me to move and I haven t been blogging as much so while I ve been keeping an eye open for a replacement I haven t exerted a lot of energy into the process. I have a little bit of time at present so I asked around on IRC for suggestions. One was ikiwiki, which I use as part of helping maintain the SPI website (and think is fantastic for that), the other was Jekyll. Both are available as part of Debian Jessie. Jekyll looked a bit fancier out of the box (I m no web designer so pre-canned themes help me a lot), so I decided to spend some time investigating it a bit more. I d found a Movable Type to ikiwiki converter which provided a starting point for exporting from the SQLite3 DB I was using for MT. Most of my posts are in markdown, the rest (mostly from my Blosxom days) are plain HTML, so there wasn t any need to do any conversion on the actual content. A minor amount of poking convinced Jekyll to use the same URL format (permalink: /:year/:month/:title.html in the _config.yml did what I wanted) and I had to do a few bits of fix up for some images that had been uploaded into MT, but overall fairly simple stuff. Next I had to think about comments. My initial thought was to just ignore them for the moment; they weren t really working on the MT install that well so it s not a huge loss. I then decided I should at least see what the options were. Google+ has the ability to embed in your site, so I had a play with that. It worked well enough but I didn t really want to force commenters into the Google ecosystem. Next up was Disqus, which I ve seen used in various places. It seems to allow logins via various 3rd parties, can cope with threading and deals with the despamming. It was easy enough to integrate to play with, and while I was doing so I discovered that it could cope with importing comments. So I tweaked my conversion script to generate a WXR based file of the comments. This then imported easily into Disqus (and also I double checked that the export system worked). I m sure the use of a third party to handle comments will put some people off, but given the ability to export I m confident if I really feel like dealing with despamming comments again at some point I can switch to something locally hosted. I do wish it didn t require Javascript, but again it s a trade off I m willing to make at present. Anyway. Thanks to Tollef for the pointer (and others who made various suggestions). Hopefully I haven t broken (or produced a slew of new posts for) any of the feed readers pointed at my site (but you should update to use feed.xml rather than any of the others - I may remove them in the future once I see usage has died down). (On the off chance it s useful to someone else the conversion script I ended up with is available. There s a built in Jekyll importer that may be a better move, but I liked ending up with a git repository containing a commit for each post.)

13 January 2015

Jonathan McDowell: Tracking a ship around the world

I moved back from the California Bay Area to Belfast a while back and for various reasons it looks like I m going to be here a while, so it made sense to have my belongings shipped over here. They haven t quite arrived yet, and I ll do another post about that process once they have, but I ve been doing various tweets prefixed with [shipping] during the process. Various people I ve spoken to (some who should know me better) thought this was happening manually. It wasn t. If you care about how it was done, read on. I d been given details of the ship carrying my container, and searching for that turned up the excellent MarineTraffic which let me see the current location of the ship. Turns out ships broadcast their location using AIS and anyone with a receiver can see the info. Very cool, and I spent some time having a look at various bits of shipping around the UK out of interest. I also found the ship s itinerary which give me some idea of where it would be calling and when. Step one was to start recording this data; it was time sensitive and I wanted to be able to see historical data. I took the easy route and set up a cron job to poll the location and itinerary on an hourly basis, and store the results. That meant I had the data over time, if my parsing turned out to miss something I could easily correct it, and that I wasn t hammering Marine Traffic while writing the parsing code. Next I wanted to parse the results, store them in a more manageable format than the HTML, and alert me when the ship docked somewhere or set off again. I ve been trying to learn more Python rather than doing my default action of turning to Perl for these things, and this seemed like a simple enough project to try out. Beautiful Soup seemed to turn up top for HTML parsing in Python, so that formed the basis. Throwing the info into a database so I could do queries felt like the right move so I used SQLite - if this had been more than a one off I d have considered looking at PostgreSQL and its GIS support. Finally Tweepy made it very easy to tweet from Python in about 4 lines of code. The whole thing weighed in at only 175 lines of code, mostly around pulling the info out of the HTML and then a little to deal with checking for state changes against the current status and the last piece of info in the database. The pieces of information I chose to store were the time of the update (i.e. when the ship sent it, not when my script ran), reported area, reported state, the position + course, reported origin, reported destination and eta. The fact this is all in a database makes it very easy to do a few queries on the data. How fast did the ship go?
sqlite> SELECT MAX(speed) FROM status;
MAX(speed)
21.9
What areas did it report?
sqlite> SELECT area FROM status GROUP BY area;
area
-
Atlantic North
California
Caribbean Sea
Celtic Sea
English Channel
Hudson River
Pacific North
Panama Canal
What statuses did we see?
sqlite> SELECT status FROM status GROUP BY status;
status
At Anchor
Moored
Stopped
Underway
Underway using Engine
Finally having hourly data lets me draw a map of where the ship went. The data isn t complete, because the free AIS info depends on the ship being close enough to a receiving station. That means there were a few days in the North Atlantic without updates, for example. However there s enough to give a good idea of just how well traveled my belongings are, and it gave me an excuse to play with OpenLayers.
(Apologies if the zoom buttons aren t working for you here; I think I need to kick the CSS in some manner I haven t quite figured out yet.)

08 January 2015

Jonathan McDowell: Cup!

I got a belated Christmas present today. Thanks Jo + Simon! cup.jpg

27 September 2014

DebConf team: Wrapping up DebConf14 (Posted by Paul Wise, Donald Norwood)

The annual Debian developer meeting took place in Portland, Oregon, 23 to 31 August 2014. DebConf14 attendees participated in talks, discussions, workshops and programming sessions. Video teams captured a lot of the main talks and discussions for streaming for interactive attendees and for the Debian video archive. Between the video, presentations, and handouts the coverage came from the attendees in blogs, posts, and project updates. We ve gathered a few articles for your reading pleasure: Gregor Herrmann and a few members of the Debian Perl group had an informal unofficial pkg-perl micro-sprint and were very productive. Vincent Sanders shared an inspired gift in the form of a plaque given to Russ Allbery in thanks for his tireless work of keeping sanity in the Debian mailing lists. Pictures of the plaque and design scheme are linked in the post. Vincent also shared his experiences of the conference and hopes the organisers have recovered. Noah Meyerhans adventuring to Debian by train, (Inter)netted some interesting IPv6 data for future road and railwarriors. Hideki Yamane sent a gentle reminder for English speakers to speak more slowly. Daniel Pocock posted of GSoC talks at DebConf14, highlights include the Java Project Dependency Builder and the WebRTC JSCommunicator. Thomas Goirand gives us some insight into a working task list of accomplishments and projects he was able to complete at DebConf14, from the OpenStack discussion to tasksel talks, and completion of some things started last year at DebConf13. Antonio Terceiro blogged about debci and the Debian Continuous Integration project, Ruby, Redmine, and Noosfero. His post also shares the atmosphere of being able to interact directly with peers once a year. Stefano Zacchiroli blogged about a talk he did on debsources which now has its own HACKING file. Juliana Louback penned: DebConf 2014 and How I Became a Debian Contributor. Elizabeth Krumbach Joseph s in-depth summary of DebConf14 is a great read. She discussed Debian Validation & CI, debci and the Continuous Integration project, Automated Validation in Debian using LAVA, and Outsourcing webapp maintenance. Lucas Nussbaum by way of a blog post releases the very first version of Debian Trivia modelled after the TCP/IP Drinking Game. Fran ois Marier s shares additional information and further discussion on Outsourcing your webapp maintenance to Debian. Joachim Breitner gave a talk on Haskell and Debian, created a new tool for binNMUs for Haskell packages which runs via cron job. The output is available for Haskell and for OCaml, and he still had a small amount of time to go dancing. Jaldhar Harshad Vyas was not able to attend DebConf this year, but he did tune in to the videos made available by the video team and gives an insightful viewpoint to what was being seen. J r my Bobbio posted about Reproducible builds in Debian in his recap of DebConf14. One of the topics at hand involved defining a canonical path where packages must be built and a BOF discussion on reproducible builds from where the conversation moved to discussions in both Octave and Groff. New helpers dh_fixmtimes and dh_genbuildinfo were added to BTS. The .buildinfo format has been specified on the wiki and reviewed. Lots of work is being done in the project, interested parties can help with the TODO list or join the new IRC channel #debian-reproducible on irc.debian.org. Steve McIntyre posted a Summary from the d-i / debian-cd BoF at DC14, with some of the session video available online. Current jessie D-I needs some help with the testing on less common architectures and languages, and release scheduling could be improved. Future plans: Switching to a GUI by default for jessie, a default desktop and desktop choice, artwork, bug fixes and new architecture support. debian-cd: Things are working well. Improvement discussions are on selecting which images to make I.E. netinst, DVD, et al., debian-cd in progress with http download support, Regular live test builds, Other discussions and questions revolve around which ARM platforms to support, specially-designed images, multi-arch CDs, and cloud-init based images. There is also a call for help as the team needs help with testing, bug-handling, and translations. Holger Levsen reports on feedback about the feedback from his LTS talk at DebConf14. LTS has been perceived well, fits a demand, and people are expecting it to continue; however, this is not without a few issues as Holger explains in greater detail the lacking gatekeeper mechanisms, and how contributions are needed from finance to uploads. In other news the security-tracker is now fixed to know about old stable. Time is short for that fix as once jessie is released the tracker will need to support stable, oldstable which will be wheezy, and oldoldstable. Jonathan McDowell s summary of DebConf14 includes a fair perspective of the host city and the benefits of planning of a good DebConf14 location. He also talks about the need for facetime in the Debian project as it correlates with and improves everyone s ability to work together. DebConf14 also provided the chance to set up a hard time frame for removing older 1024 bit keys from Debian keyrings. Steve McIntyre posted a Summary from the State of the ARM BoF at DebConf14 with updates on the 3 current ports armel, armhf and arm64. armel which targets the ARM EABI soft-float ARMv4t processor may eventually be going away, while armhf which targets the ARM EABI hard-float ARMv7 is doing well as the cross-distro standard. Debian is has moved to a single armmp kernel flavour using Device Tree Blobs and should be able to run on a large range of ARMv7 hardware. The arm64 port recently entered the main archive and it is hoped to release with jessie with 2 official builds hosted at ARM. There is talk of laptop development with an arm64 CPU. Buildds and hardware are mentioned with acknowledgements for donated new machines, Banana Pi boards, and software by way of ARM s DS-5 Development Studio - free for all Debian Developers. Help is needed! Join #debian-arm on irc.debian.org and/or the debian-arm mailing list. There is an upcoming Mini-DebConf in November 2014 hosted by ARM in Cambridge, UK. Tianon Gravi posted about the atmosphere and contrast between an average conference and a DebConf. Joseph Bisch posted about meeting his GSOC mentors, attending and contributing to a keysigning event and did some work on debmetrics which is powering metrics.debian.net. Debmetrics provides a uniform interface for adding, updating, and viewing various metrics concerning Debian. Harlan Lieberman-Berg s DebConf Retrospective shared the feel of DebConf, and detailed some of the work on debugging a build failure, work with the pkg-perl team on a few uploads, and work on a javascript slowdown issue on codeeditor. Ana Guerrero L pez reflected on Ten years contributing to Debian.

18 September 2014

Jonathan McDowell: Automatic inline signing for mutt with RT

I spend a surprising amount of my time as part of keyring-maint telling people their requests are badly formed and asking them to fix them up so I can actually process them. The one that's hardest to fault anyone on is that we require requests to be inline PGP signed (i.e. the same sort of output as you get with "gpg --clearsign"). That's because RT does various pieces of unpacking[0] of MIME messages that mean that a PGP/MIME signatures that have passed through it are no longer verifiable. Daniel has pointed out that inline PGP is a bad idea and got as far as filing a request that RT handle PGP/MIME correctly (you need a login for that but there's a generic read-only one that's easy to figure out), but until that happens the requirement stands when dealing with Debian's RT instance. So today I finally added the following lines to my .muttrc rather than having to remember to switch Mutt to inline signing for this one special case:
send-hook . "unset pgp_autoinline; unset pgp_autosign"
send-hook rt.debian.org "set pgp_autosign; set pgp_autoinline"
i.e. by default turn off auto inlined PGP signatures, but when emailing anything at rt.debian.org turn them on. (Most of the other things I tell people to fix are covered by the replacing keys page; I advise anyone requesting a key replacement to read that page. There's even a helpful example request template at the bottom.) [0] RT sticks a header on the plain text portion of the mail, rather than adding a new plain text part for the header if there are multiple parts (this is something Mailman handles better). It will also re-encode received mail into UTF-8 which I can understand, but Mutt will by default try to find an 8 bit encoding that can handle the mail, because that's more efficient, which tends to mean it picks latin1. Update: Apparently Mutt in Jessie and beyond doesn't have the pgp_autosign option; you want crypt_autosign instead (and maybe crypt_autopgp but that defaults to yes so unless you've configured your setup to do S/MIME by default you should be fine). Thanks to Luca Capello for pointing this out.

12 September 2014

Jonathan McDowell: Back from DebConf 14

I previously forgot to mention that I was planning to attend DebConf14, having missed DebConf13. This year the conference was held in Portland, OR. This is a city I've been to many times before, and enjoy, but I hadn't spent any time wandering around its city centre as a pedestrian. I have to say I really prefer DebConfs that are held in middle of city. It always seems a bit of a shame to travel some distance to somewhere new and spend all the time there in a conference venue. Plus these days I have the added lure of going out and playing Ingress in a new location. DebConf14 didn't disappoint in these respects; the location was super easy to get to from the airport via public transportation, all of the evening social events were within reasonable walking distance (I'll tend to default to walking when possible) and the talk venue/accommodation were close to each other and various eating + drinking options. Throw in the fact at Portland managed to produce some excellent weather (modulo my Ingress session on the last Saturday morning, when rained on me) and it's impossible to fault the physicalities of DebConf this year. This year the conference format was a bit different; previous years have had a week long DebConf before the week of the conference itself. This year went for a 9 day talk schedule (Saturday -> Sunday) with various gaps of hacking time interspersed. I've found it hard to justify a full two weeks away in the past, so this setup worked a lot better from my viewpoint. Also I rarely go to DebConf with a predetermined list of things to do; the stuff I work on naturally falls out of talks I attend and informal discussions I have. Having hack time throughout the conference helped me avoid feeling I was having to trade off hacking vs talks. Naturally enough a lot of my involvement at DebConf was around OpenPGP. Gunnar and I spent a fair bit of time getting Daniel up to speed with the keyring-maint team (Gunnar more than I, I'll confess). We finally set a hard timeframe for freeing Debian of older 1024 bit keys. I was introduced to the Gnuk, which is a particularly interesting piece of open specification hardware with a completely Free software stack on top if it that implements the OpenPGP smartcard spec. Currently it's limited to 2K keys but it's hoped that 4K support can be added (and I ended up spending a couple of hours after the closing talk hacking on the source and seeing how much needs to change for 4K support, aided by the very patient Niibe). These are the sort of things that really benefit from the face time that DebConf offers to the Debian project. I've said it before, but I think it's worth saying again: Debian is a bit like a huge telecommuting organization and it's my opinion that any such organization should try and ensure its members actually spend some time together on a regular basis. It improves the ability to work remotely a hell of a lot if you can actually put a face to the entity you're emailing / IRCing and have some sort of idea where they're coming from because you've spent some time with them, whether that's in talks or over dinner or just casual hallway chats. For once I also found myself considering alternative employment while at DebConf and it was incredibly useful to be able to have various conversations with both old friends and people who were there with an eye on recruitment. Thanks to all those whose ears I bent about the subject (and more on the outcome in a future post). Thank you also to the many people involved with the organization of DebConf; I've been on the periphery a few times over the years and it's given me a glimpse into the amount of hard work all of the volunteers (be they global team, local organizing team, video team or just random volunteers) put into making DebConf one of my must-attend yearly conferences. If you're at all involved in Debian and haven't attended I strongly urge you to do so - I'll see you all next year at DebConf15 in Heidelberg!

06 September 2014

Jonathan McDowell: Breaking up with America

Back in January I changed jobs. This took me longer to decide to do than it should have. My US visa (an L-1B) was tied to the old job, and not transferable, so leaving the old job also meant leaving the US. That was hard to do; I'd had a mostly fun 3 and a half years in the SF Bay Area. The new job had an office in Belfast, and HQ in the Bay Area. I went to work in Belfast, and got sent out to the US to meet coworkers and generally get up to speed. During that visit the company applied for an H-1B visa for me. This would have let me return to the US in October 2014 and start working in the US office; up until that point I'd have continued to work from Belfast. Unfortunately there were 172,500 applications for 85,000 available visas and mine was not selected for processing. I'm disappointed by this. I've enjoyed my time in the US. I had a green card application in process, but after nearly 2 years it still hadn't completed the initial hurdle of the labor certification stage (a combination of a number of factors, human, organizational and governmental). However the effort of returning to live here seems too great for the benefits gained. I can work for a US company with a non-US office and return on an L-1B after a year. And once again have to leave should I grow out of the job, or the job change in some way that doesn't suit me, or the company hit problems and have to lay me off. Or I can try again for an H-1B next year, aiming for an October 2015 return, and hope that this time my application gets selected for processing. Neither really appeals. Both involve putting things on hold in the hope longer terms pans out as I hope. And to be honest I'm bored of that. I've loved living in America, but I ended up spending at least 6 months longer in the job I left in January than I'd have done if I'd been freely able to change employer without having to change continent. So it seems the time has come to accept that America and I must part ways, sad as that is. Which is why I'm currently sitting in SFO waiting for a flight back to Belfast and for the first time in 5 years not having any idea when I might be back in the US.

28 August 2014

Gunnar Wolf: Ongoing crypto handling discussions

I love to see there is a lot of crypto discussions going on at DebConf. Maybe I'm skewed by my role as keyring-maint, but I have been involved in more than one discussion every day on what do/should signatures mean, on best key handling practices, on some ideas to make key maintenance better, on how the OpenPGPv4 format lays out a key and its components on disk, all that. I enjoy some of those discussions pose questions that leave me thinking, as I am quite far from having all answers. Discussions should be had face to face, but some start online and deserve to be answered online (and also pose opportunity to become documentation). Simon Josefsson blogs about The case for short OpenPGP key validity periods. This will be an important issue to tackle, as we will soon require keys in the Debian keyring to have a set expiration date (surprise surprise!) and I agree with Simon, setting an expiration date far in the future means very little. There is a caveat with using, as he suggests, very short expiry periods: We have a human factor sitting in the middle. Keyring updates in Debian are done approximately once a month, and I do not see the period shortening. That means, only once a month we (currently Jonathan McDowell and myself, and we expect to add Daniel Kahn Gillmor soon) take the full changeset and compile a new keyring that replaces the active one in Debian. This means that if you have, as Simon suggests, a 100-day validity key, you have to remember to update it at least every 70 days, or you might be locked out during the days it takes us to process it. I set my expiration period to two years, although I might shorten it to only one. I expect to add checks+notifications before we enable this requirement project-wide (so that Debian servers will mail you when your key is close to expiry); I think that mail can be sent at approximately [expiry date - 90 days] to give you time both to you and to us to act. Probably the optimal expiration periods under such conditions would be between 180 and 365 days. But, yes, this is by far not yet a ruling, but a point in the discussion. We still have some days of DebConf, and I'll enjoy revising this point. And Simon, even if we correct some bits for these details, I'd like to have your permission to use this fine blog post as part of our documentation! (And on completely unrelated news: Congratulations to our dear and very much missed friend Bubulle for completely losing his sanity and running for 28 hours and a half straight! He briefly describes this adventure when it was about to start, and we all want him to tell us how it was. Mr. Running French Guy, you are amazing!)

17 July 2014

Jonathan McDowell: On the state of Free VoIP

Every now and then I decide I'll try and sort out my VoIP setup. And then I give up. Today I tried again. I really didn't think I was aiming that high. I thought I'd start by making my email address work as a SIP address. Seems reasonable, right? I threw in the extra constraints of wanting some security (so TLS, not UDP) and a soft client that would work on my laptop (I have a Grandstream hardphone and would like an Android client as well, but I figure those are the easy cases while the "I have my laptop and I want to remain connected" case is a bit trickier). I had a suitable Internet connected VM, access to control my DNS fully (so I can do SRV records) and time to read whatever HOWTOs required. And oh my ghod the state of the art is appalling. Let's start with getting a SIP server up and running. I went with repro which seemed to be a reasonably well recommended SIP server to register against. And mostly getting it up and running and registering against it is fine. Until you try and make a TLS SIP call through it (to a sip5060.net test address). Problem the first; the StartCom free SSL certs are not suitable because they don't advertise TLS Client. So I switch to CACert. And then I get bitten by the whole question about whether the common name on the cert should be the server name, or the domain name on the SIP address (it's the domain name on the SIP address apparently, though that might make your SIP client complain). That gets the SIP side working. Of course RTP is harder. repro looks like it's doing the right thing. The audio never happens. I capitulate at this point, and install Lumicall on my phone. That registers correctly and I can call the sip:test.time@sip5060.net test number and hear the time. So the server is functioning, it's the client that's a problem. I try the following (Debian/testing): I'm bored at this point. Can I "dial" my debian.org SIP address from Lumicall? Of course not; I get a "Codecs incompatible" (SIP 488 Not Acceptable Here) response. I have no idea what that means. I seem to have all of the options on Lumicall enabled. Is it a NAT thing? A codec thing? Did I sacrifice the wrong colour of goat? At some point during this process I get a Skype call from some friends, which I answer. Up comes a video call with them, their newborn, perfect audio, and no hassle. I have a conversation with them that doesn't involve me cursing technology at all. And then I go back to fighting with SIP. Gunnar makes the comment about Skype creating a VoIP solution 10 years ago when none was to be found. I believe they're still the market leader. It just works. I'm running the Linux client, and they're maintaining it (a little behind the curve, but close enough), and it works for text chat, voice chat and video calls. I've spent half a day trying to get a Free equivalent working and failing. I need something that works behind NAT, because it's highly likely when I'm on the move that's going to be the case. I want something that lets my laptop be the client, because I don't want to rely on my mobile phone. I want my email address to also be my VoIP address. I want some security (hell, I'm not even insisting on SRTP, though I'd like to). And the state of the Open VoIP stack just continues to make me embarrassed. I haven't given up yet, but I'd appreciate some pointers. And Skype, if you're hiring, drop me a line. ;)

07 July 2014

Jonathan McDowell: 2014 SPI Board election nominations open

I put out the call for nominations for the 2014 Software in the Public Interest (SPI) Board election last week. At this point I haven't yet received any nominations, so I'm mentioning it here in the hope of a slightly wider audience. Possibly not the most helpful as I would hope readers who are interested in SPI are already reading spi-announce. There are 3 positions open this election and it would be good to see a bit more diversity in candidates this year. Nominations are open until the end of Tuesday July 13th. The primary hard and fast time commitment a board member needs to make is to attend the monthly IRC board meetings, which are conducted publicly via IRC (#spi on the OFTC network). These take place at 20:00 UTC on the second Thursday of every month. More details, including all past agendas and minutes, can be found at http://spi-inc.org/meetings/. Most of the rest of the board communication is carried out via various mailing lists. The ideal candidate will have an existing involvement in the Free and Open Source community, though this need not be with a project affiliated with SPI. Software in the Public Interest (SPI, http://www.spi-inc.org/) is a non-profit organization which was founded to help organizations develop and distribute open hardware and software. We see it as our role to handle things like holding domain names and/or trademarks, and processing donations for free and open source projects, allowing them to concentrate on actual development. Examples of projects that SPI helps includes Debian, LibreOffice, OFTC and PostgreSQL. A full list can be found at http://www.spi-inc.org/projects/.

26 June 2014

Jonathan McDowell: Forms of communication

I am struck by the fragmentation in communication mechanisms. Let's look at how I have communicated with my friends in the past few days: While some of these work better for me than others really what I'd like is to use fewer of them, and I can't see that happening any time soon. I don't want to have to run a handful of different messaging apps on my phone. I also don't want to be limited to only using my laptop or my phone for something - I'd much prefer to be able to pickup the phone, laptop or tablet depending on what I'm up to and have my full range of communication available. Some of these things can be aggregated together, but that will then lose some of the advantages. And I'm sure that even if I got rid of one or two of the above there'd be something to fill the gap along shortly (I have, for example, so far completely avoided WhatsApp).

Next.

Previous.