A bit of news on the development of OpenTAC the Open Hardware Test Automation Controller. I ve talked about this at the MiniDebConf 2014 in Cambridge. (Video available). The development is being tracked on the Vero-Apparatus wiki and as this is Open Hardware, the files are attached to the wiki (All files are CC BY-SA 4.0). Andy completed the schematics at the start of November, allowing work to start on the routing. With routing completed, orders were placed for manufacture of the first PCBs on the 19th December 2014. Whilst waiting for the PCBs to arrive we re working on the device tree database (my first real experience with creating a device tree) which will underpin the PDU and serial console services available to the user as well as the admin interface for control of the USB subsystems, fan control, power control and thermal monitoring. The first thing we need to do is create a data dictionary a table to correlate software identifiers with the real hardware pins. We ll then follow that through with a default device tree overlay that will leave all the associated I/O lines in a safe initial state. Once we have some code, I ll be pushing to a branch on GitHub. We ve also got an internal git repo on vero-apparatus for OpenTAC files which we will add to in due course. Image of the board as rendered prior to prototype production: More to come once we have the hardware on the desk

There s a lot of discussion / moaning /arguing at this time, so I thought I d post something about how LAVA got into Debian Jessie, the work involved and the lessons I ve learnt. Hopefully, it will help someone avoid the disappointment of having their package missing the migration into a future stable release. This was going to be a talk at the Minidebconf-uk in Cambridge but I decided to put this out as a permanent blog entry in the hope that it will be a useful reference for the future, not just Jessie. Context LAVA relies on a number of dependencies which were at the time all this started NEW to Debian as well as many others already in Debian. I d been running LAVA using packages on my own system for a few months before the packages were ready for use on the main servers (I never actually installed LAVA using the old virtualenv method on my own systems, except in a VM). I did do quite a lot of this on my own but I also had a team supporting the effort and valuing the benefits of moving to a packaged system. At the time, LAVA was based on Ubuntu (12.04 LTS Precise Pangolin) and a new Ubuntu LTS was close (Trusty Tahr 14.04) but I started work on this in 2013. By the time my packages were ready for general usage, it was winter 2013 and much too close to get anything into Ubuntu in time for Trusty. So I started a local repo using space provided by Linaro. At the same time, I started uploading the dependencies to Debian. json-schema-validator, django-testscenarios and others arrived in April and May 2014. (Trusty was released in April). LAVA arrived in NEW in May, being accepted into unstable at the end of June. LAVA arrived in testing for the first time in July 2014. Upstream development continued apace and a regular monthly upload, with some hotfixes in between, continued until close to the freeze. At this point, note that although upstream is a medium sized team, the Debian packaging also has a team but all the uploads were made by me. I planned ahead. I knew that I would be going to Macau for Linaro Connect in February a critical stage in the finalisation of the packages and the migration of existing instances from the old methods. I knew that I would be on vacation from August through to the end of September 2014 including at least two weeks with absolutely no connectivity of any kind. Right at this time, Django1.7 arrived in experimental with the intent to go into unstable and hence into Jessie. This was a headache for me, I initially sought to delay the migration until after Jessie. However, we discussed it upstream, allocated time within the busy schedule and also sought help from within Debian with the RFH tag. Rapha l Hertzog contributed patches for django1.7 support and we worked on those patches upstream, once I was back from vacation. (The final week of my vacation was a work conference, so we had everyone together at one hacking table.) Still there was more to do, the django1.7 patches allowed the unit tests to complete but broke other parts of the lava-server package and needed subsequent tweaks and fixes. Even with all this, the auto-removal from testing for packages affected by RC bugs in their dependencies became very important to monitor (it still is). It would be useful if some packages had less complex dependency chains (I m looking at you, uwsgi) as the auto-removal also covers build-depends. This led to some more headaches with libmatheval. I m not good with functional programming languages, I did have some exposure to Scheme when working on Gnucash upstream but it wasn t pleasant. The thought of fixing a scheme problem in the test suite of libmatheval was daunting. Again though, asking for help, I found people in the upstream team who wanted to refresh their use of scheme and were able to help out. The fix migrated into testing in October. Just for added complications, lava-server gained a few RC bugs of it s own during that time too fixed upstream but awkward nonetheless. Achievement unlocked So that s how a complex package like lava-server gets into stable. With a lot of help. The main problem with top-level packages like this is the sheer weight of the dependency chain. Something seemingly unrelated (like libmatheval) can seriously derail the migrations. The package doesn t use the matheval support provided by uwsgi. The bug in matheval wasn t in the parts of matheval used by uwsgi. It wasn t in a language I am at all comfortable in fixing but it s my name on the changelog of the NMU. That happened because I asked for help. OK, when django1.7 was scheduled to arrive in Debian unstable and I knew that lava was not ready, I reacted out of fear and anxiety. However, I sought help, help was provided and that help was enough to get upstream to a point where the side-effects of the required changes could be fixed. Maintaining a top-level package in Debian is becoming more like maintaining a core package in Debian and that is a good thing. When your package has a lot of dependencies, those dependencies become part of the maintenance workload of your package. It doesn t matter if those are install time dependencies, build dependencies or reverse dependencies. It doesn t actually matter if the issues in those packages are in languages you would personally wish to be expunged from the archive. It becomes your problem but not yours alone. Debian has a lot of flames right now and Enrico encouraged us to look at what else is actually happening in Debian besides those arguments. Well, on top of all this with lava, I also did what I could to help the arm64 port along and I m very happy that this has been accepted into Jessie as an official release architecture. That s a much bigger story than LAVA yet LAVA was and remains instrumental in how arm64 gained the support in the kernel and various upstreams which allowed patches to be accepted and fixes to be incorporated into Debian packages. So a roll call of helpers who may otherwise not have been recognised via changelogs, in no particular order:

• Steve McIntyre (Debian & Linaro)
• Rapha l Hertzog (Debian)
• Dave Pigott (Linaro)
• R mi Duraffort (Linaro)
• Sjoerd Simons (Debian)
• Antonio Terceiro (Debian and formerly Linaro)
• Martin Pitt (Debian)
• Jordi Mallach (Debian)
• Hector Oron (Debian)
• Colin Watson (Debian)

Also general thanks to the Debian FTP and Release teams. Lessons learnt

1. Allow time! None of the deadlines or timings involved in this entire process were hidden or unexpected. NEW always takes a finite but fairly lengthy amount of time but that was the only timeframe with any amount of uncertainty. That is actually a benefit it reminds you that this entire process is going to take a significant amount of time and the only loser if you try to rush it is going to be you and your package. Plan for the time and be sceptical about how much time is actually required.
2. Ask for help! Everyone in Debian is a volunteer. Yes, the upstream for this project is a team of developers paid to work on this code (and largely only this code) but the upstream also has priorities, requirements, objectives and deadlines. It s no good expecting upstream to do everything. It s no good leaving upstream insufficient time to fit the required work into the existing upstream schedules. So ask for help within upstream and within Debian ask for help wherever you can. You don t know who may be able to help you until you ask. Be clear when asking for help how would someone test their proposed fix? Exactly what are you asking for help doing? (Hint: everything is not a good answer.)
3. Keep on top of announcements and changes. The release team in Debian have made the timetable strict and have published regular updates, guidelines and status notes. As maintainer, it is your responsibility to keep up with those changes and make others in the upstream team aware of the changes and the implications. Upstream will rely on you to provide accurate information about these requirements. This is almost more important than actually providing the uploads or fixes. Without keeping people informed, even asking for help can turn out to be counter-productive. Communicate within Debian too talk to the teams, send status updates to bugs (even if the status is tag 123456 + help).
4. Be realistic! Life happens around us, things change, personal timetables get torn up. Time for voluntary activity can appear and disappear (it tends to disappear far more often than extends, so take that into account too).
5. Do not expect others to do the work for you asking for help is one thing, leaving the work to others is quite another. No complaining to the release team that they are blocking your work and avoid pleading or arguing when a decision is made. The policies and procedures within Debian are generally clear and there are quite enough arguments without adding more. Read the policies, read the guidelines, watch how other packages and other maintainers are handled and avoid those mistakes. Make it easy for others to help deliver what you want.
6. Get to know your dependency chain follow the links on the packages.debian.org pages and get a handle on which packages are relevant to your package. Subscribe to the bug pages for some of the more high-risk packages. There are tools to help. rc-alert can help you spot problems with runtime dependencies (you do have your own package installed on a system running unstable if not, get that running NOW). Watching build-dependencies is more difficult, especially build-dependencies of a runtime dependency, so watch the RC bug lists for packages in your dependency chain.

Above all else, remember why you and upstream want the packages in Debian in the first place. Debian is a respected distribution and has an acknowledged reputation for stability and portability. The very qualities that you and your upstream desire from having your package in Debian have direct implications for the amount of work and the amount of time that will be required to get your packages into Debian and keep them there. Having your package in Debian will bring considerable benefits but you will be required to invest a considerable amount of time. It is this contribution which is valuable to Debian and it is this work which will deliver the benefits you seek. Being an expert in the one package is wildly inadequate. Debian is about the system, the whole distribution and sooner or later, you as the maintainer will be absolutely required to handle something which is so far out of your comfort zone it s untrue. The reality is that you are not expected to fix that problem you are expected to handle that problem and that includes seeking and acknowledging the help of others. The story isn t over until release day. Having your package in testing the day before the freeze is one step. It may be a large step, but it is only one. The status of that package still needs monitoring. That long dependency chain can still come back and bite. Don t wait for problems to surprise you. Finally One thing I do ask is that other upstream teams and maintainers think about the dependency chain they are creating. It may sound nice to have bindings for every interpreted language possible when releasing your compiled library but it does not help people using that library. Yes, it is more work releasing the bindings separately because a stable API is going to be needed to allow version 1.2.3 to work alongside 1.2.2 and 1.3.0 or the entire effort is pointless. Consider how your upstream package migrates. Consider how adding yet another build-dependency for an optional component makes things exponentially harder for those who need to rely on that upstream. If it is truly optional, release it separately and keep backwards compatibility on each side. It is more work but in reality, all that is happening is that the work is being transferred from the distribution (where it helps only that one distribution and causes duplication into other distributions) into the upstream (where it helps all distributions). Think carefully about what constitutes core functionality and release the rest separately. Combining bindings for php, ruby, python, java, lua and xslt into a single upstream release tarball is a complete nonsense. It simply means that the package gets blocked from new uploads by the constant churn of being involved in every transition that occurs in the distribution. There is a very real risk that the package will miss a stable release simply by having fingers in too many pies. That hurts not only this upstream but every upstream trying to use any part of your code. Every developer likes to think that people are using and benefiting from their effort. It s not nice to directly harm the interests of other developers trying to use your code. It is not enough for the binary packages to be discrete migrations happen by source package and the released tarball needs to not include the optional bindings. It must be this way because it is the source package which determines whether version 1.2.3 of plugin foo can work with version 1.2.0 of the library as well as with version 1.3.0. Maintainers regularly deal with these issues so talk to your upstream teams and explain why this is important to that particular team. Help other maintainers use your code and help make it easier to make a stable release of Debian. The quicker the freeze & release process becomes, the quicker new upstream versions can be uploaded and backported.

Linaro uses LAVA as a tool to test variety of devices. So far I had not installed it myself, mostly due to assuming it to be enermously complex to set up. But thanks to Neil Williams work on packaging, installation has got a lot easier. Follow the Official Install Doc and Official install to debian Doc, roughly looking like: 1. Install Jessie into kvm

kvm -m 2048 -drive file=lava2.img,if=virtio -cdrom debian-testing-amd64-netinst.iso

2. Install lava-server

apt-get update; apt-get install -y postgresql nfs-kernel-server apache2
apt-get install lava-server
# answer debconf questions
a2dissite 000-default && a2ensite lava-server.conf 
service apache2 reload
lava-server manage createsuperuser --username default --email=foo.bar@example.com
$EDITOR /etc/lava-dispatcher/lava-dispatcher.conf # make sure LAVA_SERVER_IP is right

That's the generic setup. Now you can point your browser to the IP address of the kvm machine, and log in with the default user and the password you made. 3 ... 1000 Each LAVA instance is site customized for the boards, network, serial ports, etc. In this example, I now add a single arndale board.

cp /usr/lib/python2.7/dist-packages/lava_dispatcher/default-config/lava-dispatcher/device-types/arndale.conf /etc/lava-dispatcher/device-types/
sudo /usr/share/lava-server/add_device.py -s arndale arndale-01 -t 7001

This generates us a almost usable config for the arndale. For site specifics I have usb-to-serial. Outside kvm, I provide access to serial ports using the following ser2net config:

7001:telnet:0:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT
7002:telnet:0:/dev/ttyUSB1:115200 8DATABITS NONE 1STOPBIT

TODO: make ser2net not run as root and ensure usb2serial devices always get same name.. For automatic power reset, I wanted something cheap, yet something that wouldn't require much soldering (I'm not a real embedded engineer.. I prefer software side ;) . Discussed with Hector, who hinted about prebuilt relay boxes. Chose one from Ebay, a kmtronic 8-port USB Relay. So now I have this cute boxed nonsense hack. The USB relay is driven with a short script, hard-reset-1

stty -F /dev/ttyACM0 9600
echo -e '\xFF\x01\x00' > /dev/ttyACM0
sleep 1
echo -e '\xFF\x01\x01' > /dev/ttyACM0

Sidenote: If you don't have or want automated power relay for lava, you can always replace this this script with something along "mpg123 puny_human_press_the_power_button_now.mp3" Both the serial port and reset script are on server with dns name aimless. So we take the /etc/lava-dispatcher/devices/arndale-01.conf that add_device.py created and make it look like:

device_type = arndale
hostname = arndale-01
connection_command = telnet aimless 7001
hard_reset_command = slogin lava@aimless -i /etc/lava-dispatcher/id_rsa /home/lava/hard-reset-1

Since in my case I'm only going to test with tftp/nfs boot, the arndale board needs only to be setup to have a u-boot bootloader ready on power-on. Now everything is ready for a test job. I have a locally built kernel and device tree, and I export the directory using the httpd available by default in debian.. Python!

cd out/
python -m SimpleHTTPServer

Go to the lava web server, select api->tokens and create a new token. Next we add the token and use it to submit a job

$ sudo apt-get install lava-tool
$ lava-tool auth-add http://default@lava-server/RPC2/
$ lava-tool submit-job http://default@lava-server/RPC2/ lava_test.json
submitted as job id: 1
$

The first job should now be visible in the lava web frontend, in the scheduler -> jobs part. If everything goes fine, the relay will click in a moment and the job will finish in a few minutes.

I ve previously covered running LAVA on ARM devices, now that the packages are in Debian. I ve also covered setting up the home lab, including the difficulty in obtaining the PDU and relying on another machine to provide USB serial converters with inherent problems of needing power to keep the same devices assigned to the same ser2net ports. There have been ideas about how to improve the situation. Conferences are a prime example setting up a demo involving LAVA means bringing a range of equipment, separate power bricks, separate network switches (with power bricks), a device of some kind to connect up the USB serial converters (and power brick) and then the LAVA server (with SATA drive and power brick) that is without the actual devices and their cables and power. Each of those power cables tend to be a metre long, with networking and serial, it quickly becomes a cable spaghetti. Ideas around this also have application inside larger deployments, so the hardware would need to daisy-chain to provide services to a rack full of test devices. The objective is a single case providing network, power and serial connectivity to a number of test devices over a single power input and network uplink. Naturally, with a strong free software and open development bias, the unit will be Open Hardware running Debian, albeit with a custom Beaglebone Linux kernel. It s a Test Automation Controller, so we re using the name OpenTAC. Progress Open hardware ARM device running Debian to automate tests on 4 to 8 devices, initially aimed at LAVA support for Linaro engineers. Power distribution, serial console, network and optional GPIO extensions. The design involves:

• A Beaglebone Black (revC)
  • USB hotplug support required, certainly during development.
• Custom PCB connected as a Beaglebone Cape, designed by Andy Simpkins.
• Base board provides 4 channels:
  • 5V Power delivered over USB
  • Ethernet standard Cat5, no LEDs
  • Serial connectivity
    • RS232
    • UART
  • GPIO
• Internal gigabit network switch
• Space for a board like a CubieTruck (with SATA drive) to act as LAVA server
• Daughter board:
  • Same basic design as the base board, providing another 4 channels, equivalent to the base channels. When the daughter board is fitted, a second network switch would be added instead of the CubieTruck.
• Power consumption measurement per channel
  • queries made via the Beaglebone Black over arbitrary time periods, including during the test itself.
• The GPIO lines can be used to work around issues with development boards under test, including closing connections which may be required to get a device to reboot automatically, without manual intervention.
• Serial connections to test devices can be isolated during device power-cycles this allows for devices which pull power over the serial connection. (These are typically hardware design issues but the devices still need to be tested until the boards can be modified or replaced.)
• Thermal control, individual fan control via the Beaglebone Black.
• 1U case rackable or used alone on the desk of developers.
• Software design:
  • lavapdu backend module for PDU control (opentac.py) & opentac daemon on the BBB
    • telnet opentac-01 3225
  • ser2net for serial console control
    • telnet opentac-01 4000

The initial schematics are now complete and undergoing design review. A lot of work remains

After patches from Petter to add foreign architecture support and picking up some scripting from freedombox, I ve just built a Debian unstable image using the ARMMP kernel on Beaglebone-black. A few changes to vmdebootstrap will need to go into the next version (0.3), including an example customise script to setup the u-boot support. With the changes, the command would be:

sudo ./vmdebootstrap --owner  whoami  --verbose --size 2G --mirror http://mirror.bytemark.co.uk/debian --log beaglebone-black.log --log-level debug --arch armhf --foreign /usr/bin/qemu-arm-static --no-extlinux --no-kernel --package u-boot --package linux-image-armmp --distribution sid --enable-dhcp --configure-apt --serial-console-command '/sbin/getty -L ttyO0 115200 vt100' --customize examples/beagleboneblack-customise.sh --bootsize 50m --boottype vfat --image bbb.img

Some of those commands are new but there are a few important elements:

• use of arch and foreign to provide the emulation needed to run the debootstrap second stage.
• drop extlinux and install u-boot as a package.
• linux-image-armmp kernel
• new command to configure an apt source
• serial-console-command as the BBB doesn t use the default /dev/ttyS0
• choice of sid to get the latest ARMMP and u-boot versions
• customize command this is a script which does two things:
  • copies the dtbs into the boot partition
  • copies the u-boot files and creates a u-boot environment to use those files.
• use of a boot partition note that it needs to be large enough to include the ARMMP kernel and a backup of the same files.

With this in place, a simple dd to an SD card and the BBB boots directly into Debian ARMMP. The examples are now in my branch and include an initial cubieboard script which is unfinished. The current image is available for download. (222Mb). I hope to upload the new vmdebootstrap soon let me know if you do try the version in the branch.

I ve done various bits with ARMMP and LAVA on this blog already, usually waiting until I ve got all the issues ironed out before writing it up. However, this time I m just going to do a dump of where it s at, how it works and what can be done. I m aware that LAVA can seem mysterious at first, the package description has improved enormously recently, thanks to exposure in Debian: LAVA is a continuous integration system for deploying operating systems onto physical and virtual hardware for running tests. Tests can be simple boot testing, bootloader testing and system level testing, although extra hardware may be required for some system tests. Results are tracked over time and data can be exported for further analysis. The LAVA documentation has a glossary of terms like result bundle and all the documentation is also available in the lava-server-doc package. The goal is to validate the dtbs built for the Debian ARMMP kernel. One of the most accessible ways to get the ARMMP kernel onto a board for testing is tftp using the Debian daily DI builds. Actually using the DI initrd can come later, once I ve got a complete preseed config so that the entire install can be automated. (There are some things to sort out in LAVA too before a full install can be deployed and booted but those are at an early stage.) It s enough at first to download the vmlinuz which is common to all ARMMP deployments, supply the relevant dtb, partner those with a minimal initrd and see if the board boots. The first change comes when this process is compared to how boards are commonly tested in LAVA with a zImage or uImage and all/most of the modules already built in. Packaged kernels won t necessarily raise a network interface or see the filesystem without modules, so the first step is to extend a minimal initramfs to include the armmp modules.

apt install pax u-boot-tools

The minimal initramfs I selected is one often used within LAVA:

wget http://images.armcloud.us/lava/common/linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot

It has a u-boot header added, as most devices using this would be using u-boot and this makes it easier to debug boot failures as the initramfs doesn t need to have the header added, it can simply be downloaded to a local directory and passed to the board as a tftp location. To modify it, the u-boot header needs to be removed. Rather than assuming the size, the u-boot tools can (indirectly) show the size:

$ ls -l linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot
-rw-r--r-- 1 neil neil  5179571 Nov 26  2013 linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot
$ mkimage -l linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot 
Image Name:   linaro-image-minimal-initramfs-g
Created:      Tue Nov 26 22:30:49 2013
Image Type:   ARM Linux RAMDisk Image (gzip compressed)
Data Size:    5179507 Bytes = 5058.11 kB = 4.94 MB
Load Address: 00000000
Entry Point:  00000000

Referencing http://www.omappedia.com/wiki/Development_With_Ubuntu, the header size is the file size minus the data size listed by mkimage.

5179571 - 5179507 == 64

So, create a second file without the header:

dd if=linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot of=linaro-image-minimal-initramfs-genericarmv7a.cpio.gz skip=64 bs=1

decompress it

gunzip linaro-image-minimal-initramfs-genericarmv7a.cpio.gz

Now for the additions

dget http://ftp.uk.debian.org/debian/pool/main/l/linux/linux-image-3.14-1-armmp_3.14.12-1_armhf.deb

(Yes, this process will need to be repeated when this package is rebuilt, so I ll want to script this at some point.)

dpkg -x linux-image-3.14-1-armmp_3.14.12-1_armhf.deb kernel-dir
cd kernel-dir

Pulling in the modules we need for most needs, comes thanks to a script written by the Xen folks. The set is basically disk, net, filesystems and LVM.

find lib -type d -o -type f -name modules.\*  -o -type f -name \*.ko  \( -path \*/kernel/lib/\* -o  -path \*/kernel/crypto/\* -o  -path \*/kernel/fs/mbcache.ko -o  -path \*/kernel/fs/ext\* -o  -path \*/kernel/fs/jbd\* -o  -path \*/kernel/drivers/net/\* -o  -path \*/kernel/drivers/ata/\* -o  -path \*/kernel/drivers/scsi/\* -o -path \*/kernel/drivers/md/\* \)   pax -x sv4cpio -s '%lib%/lib%' -d -w >../cpio
gzip -9f cpio

original Xen script (GPL-3+) I found it a bit confusing that i is used for extract by cpio, but that s how it is. Extract the minimal initramfs to a new directory:

sudo cpio -id < ../linaro-image-minimal-initramfs-genericarmv7a.cpio

Extract the new cpio into the same location. (Yes, I could do this the other way around and pipe the output of find into the already extracted location but that's for when I get a script to do this):

sudo cpio --no-absolute-filenames -id < ../ramfs/cpio

CPIO Manual Use newc format, the new (SVR4) portable format, which supports file systems having more than 65536 i-nodes. (4294967295 bytes)
(41M)

find .   cpio -H newc -o > ../armmp-image.cpio

... and add the u-boot header back:

mkimage -A arm -T ramdisk -C none -d armmp-image.cpio.gz debian-armmp-initrd.cpio.gz.u-boot

Now what? Now send the combination to LAVA and test it. Results bundle for a local LAVA test job using this technique. (18k) submission JSON - uses file:// references, so would need modification before being submitted to LAVA elsewhere. complete log of the test job (72k) Those familiar with LAVA will spot that I haven't optimised this job, it boots the ARMMP kernel into a minimal initramfs and then expects to find apt and other tools. Actual tests providing useful results would use available tools, add more tools or specify a richer rootfs. The tests themselves are very quick (the job described above took 3 minutes to run) and don't need to be run particularly often, just once per board type per upload of the ARMMP kernel. LAVA can easily run those jobs in parallel and submission can be automated using authentication tokens and the lava-tool CLI. lava-tool can be installed without lava-server, so can be used in hooks for automated submissions. Extensions That's just one DTB and one board. I have a range of boards available locally: * iMX6Q Wandboard (used for this test)
* iMX.53 Quick Start Board (needs updated u-boot)
* Beaglebone Black
* Cubie2
* CubieTruck
* arndale (no dtb?)
* pandaboard Other devices available could involve ARMv7 devices hosted at www.armv7.com and validation.linaro.org - as part of a thank you to the Debian community for providing the OS which is (now) running all of the LAVA infrastructure. That doesn't cover all of the current DTBs (and includes many devices which have no DTBs) so there is plenty of scope for others to get involved. Hopefully, the above will help get people started with a suitable kernel+dtb+initrd and I'd encourage anyone interested to install lava-server and have a go at running test jobs based on those so that we start to build data about as many of the variants as possible. (If anyone in DI fancies producing a suitable initrd with modules alongside the DI initrd for armhf builds, or if anyone comes up with a patch for DI to do that, it would help enormously.) This will at least help Debian answer the question of what the Debian ARMMP package can actually support. For help on LAVA, do read through the documentation and then come to us at #linaro-lava or the linaro-validation mailing list or file bugs in Debian: reportbug lava-server. , so you can ask me. I'm giving one talk on the LAVA software and there will be a BoF on validation and CI in Debian.

LAVA has arrived in Debian unstable. No need for third party repositories (unless you want to), a simple apt install lava-server. What happens from that point isn t so simple. I ve made the single instance installation as straightforward as I could but there is a lot more to do to get LAVA working in a useful manner. First, a little history to explain some of the hysterical raisins which may appear later. Validation So, you ve made a change to the code, good. It compiles, yay! Before you ship it, does it pass the unit tests? Right, now you have an improved patch which does what you want and keeps the unit tests running. Validation is all about asking that awkward question: does your change work? Does it introduce side effects on other systems / services in the same code? Does it break programs which use services exported by the code? LAVA is one step along the road to system testing, starting at the bottom. Automation Well you could do all that yourself. Write the image to the device yourself, apply power yourself, connect to serial, copy test scripts to the booted image and pull the results off, somehow. Much better if this is automated maybe every commit or every push or as often as the tests can be run on the number of devices available. LAVA Linaro Automated Validation Architecture. Linaro is a not-for-profit building the future of the Linux kernel on ARM devices. Lava was built for and by Linaro, so the initial focus is clearly on validating the Linux kernel on ARM devices. There are likely to be gotchas in the code for those wanting to use LAVA for other kernels Linaro can t support lots of different kernels, but if there are changes which make it easier to use other kernels without impacting on validation of Linux, those would likely be accepted. The experience with using LAVA is all with ARM but if there is interest in extending LAVA to work with devices of other architectures, again, patches would be welcome. The development of the packaging to make LAVA suitable for Debian has also meant that LAVA will run on hardware other than x86. e.g. armv7.com. I m running a mini lab at home based around an arndale board, I ve also got a mini lab at work based on a cubie2. Requirements for such setups are described in the documentation and in previous entries on this blog (principally you will need SATA, lots of RAM and as many CPU cores as you can find. If you want to run LAVA on other architectures, go ahead and let us know if there are issues. Available versions The versions uploaded to Debian will (from now on) be production releases. The same code as is running on http://validation.linaro.org/ development builds and test builds are supported using helpers in the lava-dev package. Releases to unstable will automatically migrate into Ubuntu Unicorn. I ll continue building interim versions at the former locations on http://people.linaro.org/~neil.williams/, including builds for Ubuntu Trusty 14.04LTS as well as providing packages for Jessie until the uwsgi package can migrate. LAVA is looking to work with anyone who can prepare and maintain packages for other distributions like Fedora. Bugs LAVA is migrating from Launchpad bugs to http://bugs.linaro.org which is a bugzilla interface. Now that LAVA is also in Debian, anyone is welcome to use the Debian BTS which does not require any login or account setup. The maintainers (including me) will then forward those bugs as appropriate. Documentation The immediate task is to update the documentation in the lava-server-doc package (and the Debian wiki) to reflect the availability of LAVA from Debian unstable and how to choose which release of LAVA to use in your systems. However, there is a large amount of documentation already available click the Help link in the menu bar of any current LAVA instance. As with many projects, the docs have been written by the development team. If there are things which are unclear or if sections need to be moved around to make it easier for people new to LAVA to pick it up, please file bugs. Hardware It isn t easy to run a LAVA lab, there is a lot more to it than simply installing lava-server. LAVA would not be possible without the lab team and any other LAVA instance is going to need the same level of excellence in system administration, device support and cooperation. I ve covered a little bit of that in previous entries on this blog about my home lab but that is nothing compared to the work required to provide a working lab like the one in Cambridge. Access to that lab is restricted to people within Linaro but LAVA is also reaching out to the community and if there are tests or hardware you want to see within a LAVA instance, not necessarily in the main lab, then talk to us. Contact Bug reports are preferable but you can also find LAVA people on #linaro-lava on OFTC or contact us on the linaro-validation mailing list. Future There is a lot more work to do on LAVA yet. There are assumptions about partition layout within images (hysterical raisins) and issues with unused software being required for remote worker installations. Both are now part of the next major development work within LAVA. DebConf14 There is lot more to talk about with LAVA if you are attending DebConf14 then there will be talks on LAVA and plenty of time to answer questions.

Packaging LAVA for Debian unstable with notes on other distributions I ve been building packages for LAVA on Debian unstable for several months now and I ve been running LAVA jobs on the laptop and on devices in my home lab and on an ARMv7 arndale too. Current LAVA installations use lava-deployment-tool which has only supported Ubuntu 12.04 LTS Precise Pangolin. There has been a desire in LAVA to move away from a virtual environment, to put configuration files in FHS compliant paths, to use standard distribution packages for dependencies and so to make LAVA available on more platforms than just precise. Packaging opens the door to installing LAVA on Debian, Ubuntu, Fedora and any other recent distribution. Despite LAVA currently being reliant on 12.04 Precise, some of the python dependencies of LAVA have been able to move forward using the virtual environment provided by builtout and pypi. This means that LAVA, as packaged, requires a newer base OS suite than precise for Ubuntu, the minimal base is Saucy Salamander 13.10 and for Debian it would be Jessie (testing) although there is currently a transition ongoing in Debian which means that uwsgi is not in testing and Debian unstable would be needed instead. The work to migrate configuration snippets out of deployment-tool and to ensure that the tarball built using setuptools contains all of the necessary files for the package has already been done. The packaging itself is clean and most of the work is done upstream. There is, as ever, more to do but the packages work smoothly for single install LAVA servers where the dispatcher is on the same machine as the django web frontend. The packages have also migrated to Django1.6, something which is proving difficult with the deployment-tool as it has not kept pace with the changes outside the virtual environment, even if other parts of LAVA have. LAVA will be switching to packages for installation instead of deployment-tool and this will mean changes to how LAVA works outside the Cambridge lab. When the time comes to swich to packaging, the plan is to update deployment-tool so that it no longer updates /srv/lava/ but instead migrates the instance to packages. Main changes

1. Configuration files move into /etc/
   • Device configuration files /etc/lava-dispatcher/devices/
   • Instance configuration files/etc/lava-server/
2. Log files move into /var/log/
   • Adding logrotate support no more multi-Gb log files in /srv/lava/
3. Commitment to keeping the upstream code up to date with dependencies
4. Support for migrating existing instances, using South.
5. Packaging helpers
   • add devices over SSH instead of via a combination of web frontend and SSH.
   • Developer builds with easily identifiable version strings, built as packages direct from your git tree.
6. New frontend
   • Although django1.6 does not change the design of the web frontend at all, LAVA will take the opportunity to apply a bootstrap frontend which has greater support for browsers on a variety of devices, including mobile. This also helps identify a packaged LAVA from a deployment LAVA.
7. Documentation and regular updates

The Plan LAVA has made regular releases based on a monthly cycle and these will be provided as source tarballs at http://www.linaro.org/downloads/ for distributions to download. The
official monthly release and any intervening updates will be made available for distributions to use for their own packaging. Additionally, Debian packages will be regularly built for use within LAVA and these will be available for those who choose to migrate from Ubuntu Precise to Debian Jessie. LAVA will assist maintainers who want to package LAVA for their distributions and we welcome patches from such maintainers. This can include changes to the developer build support script to automate the process of supporting development outside LAVA. Initially, LAVA will migrate to packaging internally, to prove the process and to smooth out the migration. Other LAVA instances are welcome to follow this migration or wait until the problems have been ironed out. The Issues

• Remote workers this is work to be completed during the migration to packaging within LAVA as well as pending work upstream on the internals of the connection between a remote worker and the master scheduler. Expect some churn in this area whilst the code is being finalised. A lava-worker package is being prepared which borrows enough code from lava-server to run the lava-scheduler-daemon on the remote worker, until such time as the remote worker communications are refactored.
• Integration there are plans to integrate some of the modules which LAVA uses which are not commonly packaged: linaro-dashboard-bundle and linaro-django-xmlrpc
• OpenID the certificates which underpin OpenID use with Launchpad have been removed in Debian and LAVA is currently investigating alternatives.
• json-schema-validator currently has a broken test suite, so this will need to be patched by LAVA to allow the package to build. The code may be replaced with a different validator if the issue persists.
• namespace handling the current package install is unnecessarily noisy with complaints about the lava namespace. This will need a fix but does not affect current operation.
• Unit tests LAVA is working hard to add a much larger coverage of internal unit tests. These use a temporary database which is not generally available during the build of a distribution package. LAVA is already running continuous integration tests to ensure that these tests continue to pass and the packages will gain documentation on how to run these tests after installation.

Build Dependencies For Debian unstable, the list of packages which must be installed on your Debian system to be able to build packages from the lava-server and lava-dispatcher source code trees are:

debhelper (>= 8.0.0) python   python-all   python-dev   python-all-dev 
python-sphinx (>= 1.0.7+dfsg)   python3-sphinx python-mocker 
python-setuptools python-versiontools

(python-versiontools may disappear before the packages are finalised) In addition, to be able to install lava-server, these packages need to be built from tarballs released by Linaro (the list may shorten as changes upstream are applied)

linaro-django-pagination, 
python-django-restricted-resource (>= 0.2.7), 
lava-tool (>= 0.2), lava-utils-interface (>= 1.0), 
linaro-django-xmlrpc (>= 0.4),
python-versiontools (>= 1.8),
python-longerusername,
linaro-dashboard-bundle (>= 1.10.2), 
lava-dispatcher (>= 0.33.3)
lava-coordinator, lava-server-doc

The list for lava-dispatcher is currently:

python-json-schema-validator, lava-tool (>= 0.4), 
lava-utils-interface, linaro-dashboard-bundle (>= 1.10.2), 

The packages available from my experimental repository are using a new packaging branch of lava-server and lava-dispatcher where we are also migrating the CSS to Bootstrap CSS. Installing LAVA on Debian unstable
$ sudo apt-get install emdebian-archive-keyring
Add the link to my experimental repository (amd64, i386, armhf & arm64) to your apt sources, e.g. by creating a file /etc/apt/sources.list.d/lava.list containing:

deb http://people.linaro.org/~neil.williams/lava sid main
Update with the new key

$ sudo apt-get update
It is always best to install postgresql first:

$ sudo apt-get install postgresql
There are then three options for the packages to install (Please be careful with remote worker setup, it is not suitable for important installations at this time.):

1. Single instance, server and dispatcher with recommended tools
   apt-get install lava
   Installs linaro-image-tools and guestfs tools.
2. Single instance, server and dispatcher
   apt-get install lava-server
   Installs enough to run LAVA on a single machine, running jobs on boards on the same LAN.
3. Experimental remote worker support
   apt-get install lava-worker
   Needs a normal lava-server installation to act as the master scheduler but is aimed at supporting a dispatcher and boards which are remote from that master.

The packages do not assume that your apache2.4 setup is identical to that used in other LAVA installations, so the LAVA apache config is installed to /etc/apache2/sites-available/ but is not enabled by default. If you choose to use the packaged apache config, you can simply run:

$ sudo a2ensite lava-server
$ sudo apache2ctl restart
(If this is a fresh apache install, use a2dissite to disable to default configuration before restarting.) Information on creating a superuser, adding devices and administering your LAVA install is provided in the README.Debian file in lava-server:

$ zless /usr/share/doc/lava-server/README.Debian.gz
Provisos and limitations Please be aware that packaged LAVA is still a work-in-progress but do let us know if there are problems. Now is the time to iron out install bugs and other issues as well as to prepare LAVA packages for other distributions. It will be a little while before the packages are ready for upload to Debian I ve got to arrange the download location and upload the dependencies first and some of that work will wait until more work has gone in upstream to consolidate some of the current dependencies.

(ARMv7 A15 dual core) I ve covered how I set up my home LAVA lab and how I got two iMX.53 boards running, the next step was to sort out the LAVA server. My initial setup relied on using my Thinkpad as the server. This was somewhat convenient as that s how I develop the code, but didn t help me run jobs like health checks because the laptop was usually somewhere other than on the home LAN. There had been a long-standing problem with using LAVA on ARM the uwsgi support didn t compile but the code being used was old. As the same code has been in Debian for a long time, I suspected that this was fixed with a newer version. So as part of migrating LAVA to django1.6, apache2.4, postgres9.3, django-tables2 0.13 and a few other updated dependencies, I have been using the version of uwsgi from Debian unstable for quite a while without problems on x86. A simple tweak to the LAVA setup scripts and a newer version of uwsgi became available and it builds just fine on armhf. (I haven t tried armel because Linaro is ARMv7 or newer and we don t test with ARMv5/ARMv6 boards, so there s no point in trying armel it s not likely to be an issue for this fix though.) (Only the current lava-deployment-tool methods need to actually compile uwsgi directly from upstream without any patches the packaging which will replace l-d-t uses precompiled binaries, just as any admin would expect.) That initial install was on one of the iMX.53 boards, which ran as slowly as anyone who has used an iMX.53 would expect. I haven t run test jobs on that install, it had a hard enough time serving the frontend. Clearly some more cores would be a good step and when an arndale became available for the home lab, the next step was to get a SATA drive onto the arndale, put Debian unstable on that and then deploy LAVA to the arndale using the Debian packages. As a bonus, these packages are built for django1.6 (using an unreleased branch). Sounds simple enough. It works on iMX.53, it should work on an arndale. Well, it does but only after some unexpected faff. arndale, u-boot and SATA I hate u-boot every single time I ever touch it, the little I know about any other u-boot device is completely useless. Uniformity is a good thing infinite permutations are just rope to hang ourselves with cute embedded nonsense hacks and I m bored of that whole scene. So, boot the arndale, interrupt the bootloader to get to the prompt. What s the first thing most people would want? help. Denied this particular u-boot has been compiled without the ability to display any help messages. u-boot has just sunk further into the depths of please can we have something else which is uniform across devices . OK, well maybe I can get by without help. sata init unknown command sata . (Go away, do something else for a bit.) The arndale isn t one of the newest boards in the ARMv7 world, LAVA has been using them for a while, so someone has probably done some work to fix this . nope. There s a workaround a nice detailed guide by Phil Endecott which details that the kernel needs to continue to be loaded from SD and then the SATA can be used for just the rootfs. Mildly annoyed (I was hoping to be able to update the arndale kernel without the hassle of mounting the SD card at all), the rest was straight-forward. Preparing the SATA and installing Debian Due to the complexity of some of the transitions currently ongoing in Debian ahead of the Jessie release freeze, uwsgi is not in Debian testing right now so this install has to be Debian unstable. That s OK, I ve been running Debian unstable on all my machines for several years now usually updating ~5 times a week.

# parted /dev/sda -s unit mb mktable msdos
# parted /dev/sda -s unit mb mkpart primary 1 -- "-0"
# mkfs.ext4 /dev/sda1
# mount /dev/sda1 /mnt/root/
(note the in the mkpart command, that catches me out most times I try this. Without it, the special -0 gets interpreted as an option to parted but parted doesn t then tell you how to fix it, it just complains about an unknown option.) The SD image I am using is a LAVA master image, based on Ubuntu, so it was easy to put debootstrap onto it.

# apt-get install debootstrap
# debootstrap --arch=armhf sid /mnt/root/ http://ftp.uk.debian.org/debian
# chroot /mnt/root/
Inside the chroot, it s Debian as usual:

apt-get update
apt-get install emdebian-archive-keyring
passwd
emdebian-archive-keyring is a package I (ab)use to sign my LAVA packaging as it has the advantage of being in Debian and Ubuntu going back to the time of Lenny, so it is always available from a repository which has already been authenticated to apt and doesn t involve downloading some random key from a website. I use passwd manually because this rootfs won t be using auto-login, it will have regular users. root passwords can be automated, if necessary. Thinking about login, remember to give the rootfs a usable tty. (The LAVA master image has a nasty habit of mangling editors like nano and vi when used over serial, so I m using echo.) For the arndale you need:

# echo T0:23:respawn:/sbin/getty -L ttySAC2 115200 vt102 >> ./etc/inittab
# echo >> ./etc/securetty
# echo ttySAC2 >> ./etc/securetty
Then to make the hostname useful, adapt this to your needs:

echo 127.0.0.1 arndale.codehelp arndale >> ./etc/hosts
Finally, get the network running at boot and add the LAVA repository:

# echo auto lo eth0 > ./etc/network/interfaces.d/arndale
# echo iface lo inet loopback >> ./etc/network/interfaces.d/arndale
# echo iface eth0 inet dhcp >> ./etc/network/interfaces.d/arndale
# echo deb http://people.linaro.org/~neil.williams/lava sid main >> ./etc/apt/sources.list.d/lava.list
# apt-get update
# exit
Once the LAVA packaging is finalised, the packages will be uploaded to a normal Linaro PPA, I ll update the packages to use an official download location and we ll move to using sane version string handling. For now, the packages are still in development. Code for the packaging is in Alioth but there are issues with anonymous access to the pkg-linaro-lava repositories currently, so I ve also pushed to http://github.com/Linaro if you are interested in building the packages yourself. See also my packaging guide which is also linked from the Links page on my own site. With that done, time to reboot, set the SATA drive as the rootfs using the horrid u-boot and install LAVA.

# reboot
> setenv bootargs console=tty0 console=ttySAC2,115200n8 drm_kms_helper.edid_firmware=edid-1920x1080.fw root=/dev/sda1
> boot
login
# apt-get install postgresql
After this operation, 63.0 MB of additional disk space will be used.

# apt-get install lava-server
After this operation, 231 MB of additional disk space will be used.

# a2dissite 000-default
# a2ensite lava-server
# apache2ctl restart
# service lava-server restart
Take note of the IP address of your new ARM LAVA server and sign in with your regular details, e.g. using OpenID before creating a temporary superuser:

# lava-server manage createsuperuser --username default --email=user@example.com
Choose a temporary password, log out, log in as the temporary superuser, promote your regular user to superuser and default owner of unrestricted devices in the django admin interface. Finally, log out and login as regular user to delete the temporary user. (Yes, that probably could do with some improvement remind me later and I ll see if a known user can be promoted from the command line.) See also Senthil s recent post on LAVA unit tests. Getting those to run without so many steps is also on the TODO list. Results
IP Address: 127.0.0.1
Is Master? : True
Uptime: 6:02:23.700000
Architecture: armv7l
Platform: Linux-3.10.1.0-1-linaro-arndale-armv7l-with-debian-jessie-sid
Insignal Arndale evaluation board based on EXYNOS5250
System memory: 1998MiB
Next! I m working on some code upstream to make it easier to add devices from the command line. The aim is that a set of simple commands will set up an initial device, run a job on that device and submit a result bundle to the LAVA server as job #1. This gives some assurance that everything has gone well. Now I can finally add health checks for the iMX.53 boards and start regular tests on those boards. The cubie boards will wait for a later blog post, there are interesting problems with the bootloader and serial connections. That s a whole different rant.

The home lab is my LAVA development environment and I ve recently got two iMX53 Quick Start Boards working with a typical LAVA master image based on a Linaro build of Linux 3.1.0 for mx5 with a Ubuntu Oneiric Ocelot 11.10 rootfs:

 3.1.0-1002-linaro-lt-mx5 (buildd@hubbard) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #13-Ubuntu PREEMPT Fri Dec 16 01:21:07 UTC 2011

As part of my Debian work, it was clearly time to look at a current, Debian, kernel and rootfs and as I m developing and testing on Debian unstable, this would necessarily mean testing the Debian ARMMP (multi-platform) kernel which replaces the mx5 build used in Wheezy.

Linux version 3.12-1-armmp (debian-kernel@lists.debian.org) (gcc version 4.8.2 (Debian 4.8.2-10) ) #1 SMP Debian 3.12.9-1 (2014-02-01)

I will be scripting the creation of a suitable image for these tests and there are other changes planned in LAVA to make it easier to build suitable images, but it is useful to document just how I worked out how to build the first image. Manual build steps First, I ve already discovered that the u-boot on the iMX53 doesn t like ext4, so the first step was to prepare an ext3 filesystem for the image. With a SATA drive attached, it was also much better to use that than the SD card, at least for generating the image. I m also doing this natively, so I am working inside the booted master image. This is fine as the master is designed to manipulate test images, so the only package I needed to install on the LAVA master image was debootstrap. I had an empty SATA drive to play with for these tests, so first prepare an ext3 filesystem:

# mkfs.ext3 /dev/sda1
# mkdir /mnt/sata
# mount /dev/sda1 /mnt/sata
# mkdir /mnt/sata/chroots/

Start the debootstrap:

# apt-get update
# apt-get install debootstrap
# debootstrap --arch=armhf --include=linux-image-armmp \
 --verbose unstable \
 /mnt/sata/chroots/unstable-armhf http://ftp.uk.debian.org/debian

Various actions in this chroot will need proc, so mount it here:

# chroot /mnt/sata/chroots/unstable-armhf
# mount proc -t proc /proc
# mount devpts -t devpts /dev/pts
# exit

You may also have to edit the apt sources the LAVA master image doesn t have editors installed, so either use echo or download an edited file. I m using:

deb http://ftp.uk.debian.org/debian sid main

flash-kernel needs changes For the initial tests, I ve got to get this image to boot directly from u-boot, so flash-kernel is going to be needed inside the chroot and to get the iMX53 to work with the ARMMP kernel and Device Tree, flash-kernel will need an update which will mean a patch:

# chroot /mnt/sata/chroots/unstable-armhf
# apt-get update
# apt-get install patch flash-kernel
# cp /usr/share/flash-kernel/db/all.db /home
# cd /home
# patch -p2

The patch itself goes through a couple of iterations. Initially, it is enough to use:

 Machine: Freescale MX53 LOCO Board
-Kernel-Flavors: mx5
+Kernel-Flavors: armmp
+DTB-Id: imx53-qsb.dtb
+DTB-Append: yes

Later, once the device has booted with the ARMMP kernel, the Machine Id can be updated to distinguish it from the mx5 flavour (from /proc/cpuinfo) and use the model name from the Device Tree (/proc/device-tree/model):

diff --git a/db/all.db b/db/all.db
index fab3407..41f6c78 100644
--- a/db/all.db
+++ b/db/all.db
@@ -62,6 +62,18 @@ U-Boot-Initrd-Address: 0x00800000
 Required-Packages: u-boot-tools
 Bootloader-Sets-Incorrect-Root: yes
 
+Machine: Freescale i.MX53 Quick Start Board
+Kernel-Flavors: armmp
+DTB-Id: imx53-qsb.dtb
+DTB-Append-From: 3.12
+Boot-DTB-Path: /boot/dtb
+U-Boot-Kernel-Address: 0x70008000
+U-Boot-Initrd-Address: 0x0
+Boot-Kernel-Path: /boot/uImage
+Boot-Initrd-Path: /boot/uInitrd
+Required-Packages: u-boot-tools
+Bootloader-Sets-Incorrect-Root: no
+
 Machine: Freescale MX53 LOCO Board
 Kernel-Flavors: mx5
 U-Boot-Kernel-Address: 0x70008000

(I will be filing this patch in a bug report against flash-kernel soon.) With that patched, update and run flash-kernel:

# mv all.db /usr/share/flash-kernel/db/all.db
# flash-kernel
flash-kernel: installing version 3.12-1-armmp
Generating kernel u-boot image... done.
Taking backup of uImage.
Installing new uImage.
Generating initramfs u-boot image... done.
Taking backup of uInitrd.
Installing new uInitrd.
Installing new dtb.
# exit

LAVA overlays This will be a LAVA test image and it needs an overlay if you want a vanilla image, set up a passwd inside the chroot instead.

# cd /mnt/sata/chroots/unstable-armhf/
# wget --no-check-certificate https://launchpad.net/~linaro-maintainers/+archive/overlay/+files/linaro-overlay-minimal_1112.2_all.deb
# wget --no-check-certificate https://launchpad.net/~linaro-maintainers/+archive/overlay/+files/linaro-overlay_1112.2_all.deb
# chroot /mnt/sata/chroots/unstable-armhf/
# dpkg -i linaro-overlay-minimal_1112.2_all.deb linaro-overlay_1112.2_all.deb
# rm linaro-overlay-minimal_1112.2_all.deb linaro-overlay_1112.2_all.deb
# exit

Changes to allow the chroot to boot Now the chroot needs setting up as a boot image:

# chroot /mnt/sata/chroots/unstable-armhf/
# echo T0:23:respawn:/sbin/getty -L ttymxc0 115200 vt102 >> ./etc/inittab
# echo auto lo eth0 > ./etc/network/interfaces.d/mx53loco
# echo iface lo inet loopback >> ./etc/network/interfaces.d/mx53loco
# echo iface eth0 inet dhcp >> ./etc/network/interfaces.d/mx53loco
# apt-get clean
# umount /dev/pts
# umount /proc
# exit

Partitioning as a LAVA test image This would be enough as a single partition test image but, currently, LAVA expects a much more hard-wired image. Depending on the history of the device and the need for LAVA to be able to put fresh kernel builds together with a known rootfs, LAVA has used a separate /boot and / partition in the test image for nearly all boards. Standard LAVA test images for many boards (like the iMX53) also have a small unallocated space at the start of the SD card, so until I can get LAVA upstream to handle test images of arbitrary design, I m adapting the image to suit the expectations inside LAVA. Yes, I know but it s better to get something working before spending time fixing things to make it work better. It will be fixed, in time. So I needed to separate out the /boot contents from the rest of the rootfs whilst keeping the chroot itself in a state which I can easily update and use to create other images.

# cd /mnt/sata/chroots/unstable-armhf/boot/
# tar -czf ../../boot.tar.gz ./*
# cd ..
# tar -czf ../root.tar.gz ./*

Now the test image file and its partitions:

# dd if=/dev/zero of=/mnt/sata/images/empty/debian.img bs=1M count=1024
# cp /mnt/sata/images/empty/debian.img /mnt/sata/images/debian-unstable-armhf-armmp.img
# losetup /dev/loop0 /mnt/sata/images/debian-unstable-armhf-armmp.img
# parted /dev/loop0 -s unit mb mktable msdos
# parted /dev/loop0 -s unit mb mkpart primary 1 10
# parted /dev/loop0 -s unit mb mkpart primary 11 110
# parted /dev/loop0 -s unit mb mkpart primary 111 1024

I did make the mistake of using kpartx at this stage but there are many areas of confusion when translating the kpartx output to offsets for mount when parted is easier:

# parted /dev/loop0 unit B -s print
Model:  (file)
Disk /dev/loop0: 1073741824B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start       End          Size        Type     File system  Flags
 1      1048576B    10485759B    9437184B    primary
 2      10485760B   110100479B   99614720B   primary
 3      110100480B  1024458751B  914358272B  primary

Use the Start numbers and use losetup to create the loop devices for each
partition

# losetup -o 10485760 /dev/loop1 /dev/loop0
# losetup -o 110100480 /dev/loop2 /dev/loop0
# mkfs.vfat /dev/loop1
# mkfs.ext3 /dev/loop2

Now clean up the loop mountpoints:

# losetup -d /dev/loop2
# losetup -d /dev/loop1
# losetup -d /dev/loop0
# losetup -a

losetup -a should return nothing. If it doesn t, investigate the contents of /dev/mapper and use dmsetup remove until losetup -a does report empty. Otherwise the subsequent stages will fail. Now deploy the /boot contents into the empty image:

# mount -oloop,offset=10485760 /mnt/sata/images/debian-unstable-armhf-armmp.img /mnt/boot/
# pushd /mnt/boot/
# tar -xzf /mnt/sata/chroots/boot.tar.gz
# popd
# sync
# umount /mnt/boot/

and the / contents (removing the duplicate ./boot contents)

# mount -oloop,offset=110100480 /mnt/sata/images/debian-unstable-armhf-armmp.img /mnt/root/
# pushd /mnt/root
# tar -xzf /mnt/sata/chroots/root.tar.gz
# rm ./boot/*
# popd
# sync
# umount /mnt/root

Check the image

# mount -oloop,offset=10485760 debian-unstable-armhf-armmp.img /mnt/boot
# ls /mnt/boot
config-3.12-1-armmp  initrd.img-3.12-1-armmp  System.map-3.12-1-armmp  uImage  uInitrd  vmlinuz-3.12-1-armmp
# umount /mnt/boot
# mount -oloop,offset=110100480 debian-unstable-armhf-armmp.img /mnt/root
# ls /mnt/root
bin  boot  dev  etc  home  initrd.img  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vmlinuz
# ls /mnt/root/bin/auto-serial-console
/mnt/root/bin/auto-serial-console
# umount /mnt/root
# md5sum debian-unstable-armhf-armmp.img

Downloads Yes, I have put that file online, if you are interested. Do read the readme first though. Getting the image off the device

# ip a
# python -m SimpleHTTPServer 80 2>/dev/null

then wget (just http://, IP address / and the filename), md5sum , finish with Ctrl-C. Depending on setup, it may be quicker to transfer the uncompressed
file over a LAN than to let the device compress it. Your main machine
will do the compression much faster, even with a larger download. (It also helps to not compress the image on the device, you can
test the mount offsets more easily and do check that the image
can be mounted with the offsets from parted.) Results

• complete log of the first successful LAVA test of a Debian ARMMP kernel on an iMX53 (153k)
• LAVA submission JSON (only usable with a local URL of the image and a few tweaks to the standard LAVA mx53loco device configuration)
• LAVA result bundle (JSON) (75k)

# cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 5 (v7l)
Features	: swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc08
CPU revision	: 5
Hardware	: Freescale i.MX53 (Device Tree Support)
Revision	: 0000
Serial		: 0000000000000000

# uname -a
Linux imx53-02 3.12-1-armmp #1 SMP Debian 3.12.9-1 (2014-02-01) armv7l GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux unstable (sid)
Release:	unstable
Codename:	sid

Next! Yes, there is a lot to do from here on.

• The creation of the test image needs to be smoothed out and scripted. I m thinking that this would go into vmdebootstrap. Lars?
• LAVA needs to be less presumptive about the contents and partitioning of the test image.
• the Debian ARMMP kernel itself needs to regain SATA support (an update is already pending).
• I ve also got two Cubieboard2 devices and there is a sun7i-a20-cubieboard2.dtb file in the ARMMP kernel which deserves some investigation .

Then there is the whole issue of actually making this multi-platform. After all, it is far from ideal that a multi-platform kernel package has to have platform-specific operations using flash-kernel at each update. So Grub on ARM is going to be on the list of things to investigate .

The iMX53 Quick Start Board has been a mainstay of the Debian armhf port and has been part of LAVA until recently. It s a natural choice for seeing how LAVA and Debian can work together more closely. So, in the first steps to getting my home lab running, I set about restoring two of these boards to LAVA support. I will be looking into each of the possible boot methods with these boards, but the way to start is with a LAVA master image on an SD card. With the pause in iMX53 development within LAVA, this raises a couple of issues:

• u-boot on the iMX53 is old and the Linaro image tools have advanced without making allowances for the unsupported mx53loco type.
  • For LAVA, this means that u-boot on the iMX53 does not understand ext4 and although there are guides on updating it, I wanted to stay with the original, at least at the start.
  • Using ext4 will lead to this error:

    mmc0: error 110 whilst initialising SD card
    

• iMX53 quick start boards come with a working Ubuntu installation on an SD card but the release is Lucid Lynx 10.04LTS. LAVA has been on Precise Pangolin 12.04LTS for some time.
  • This causes issues with debootstrap which cannot unpack packages from Debian sid due to changes in the compression format inside the .deb files. It is possible to use wheezy and note that lucid, by default, will produce armel. Use --arch=armhf, then apt-get upgrade and apt-get dist-upgrade to testing and then to sid (not in one single step). Remember to mount proc before starting the upgrade.

LAVA master image on iMX53 The LAVA master image is more recent and can fix these problems. See lava-image-creation preparing a master image
$ git clone git://git.linaro.org/lava/lava-master-image-scripts.git
$ cd lava-master-image-scripts
Use patch -p1 to apply this patch (apologies for the line wrapping in the blog):

diff --git a/lava-create-master b/lava-create-master
index 2e1d0e8..a1e6131 100755
--- a/lava-create-master
+++ b/lava-create-master
@@ -366,7 +366,7 @@ make_master()  
         verbose " * Building vanilla image with linaro-media-create..."
         if ! linaro-media-create \
             --dev $LMC_DEV \
-            --rootfs ext4 \
+            --rootfs ext3 \
             --image-file $CACHE_DIR/pristine-images/$dev-vanilla.img \
             --image-size 1G \
             --align-boot-part \

$ sudo ./lava-create-master imx53 Without the patch applied, I simply removed the SD card, mounted the third partition on my main machine and created a tarball of the contents:

$ sudo mount /dev/mmcblk0p3 /mnt/
$ pushd /mnt
$ sudo tar -czf /tmp/imx53-rootfs.tgz ./*
$ popd
$ sudo umount /mnt/

Reformat p3 to ext3 (or ext2 if you prefer)

$ sudo mkfs.ext3 /dev/mmcblk0p3

Put the rootfs back onto the replacement ext3 partition.

$ sudo mount /dev/mmcblk0p3 /mnt/
$ pushd /mnt
$ sudo tar -xzvf /tmp/imx53-rootfs.tgz
$ popd
$ sudo sync
$ sudo umount /mnt/

The bootscript is buggy, so run:

> run loaduimage
> run mmcboot

Then fix bootscript or you won t be able to reboot: Simplest way may be to set u-boot to ignore boot.scr

> setenv script
> saveenv
> reset

Device configuration in LAVA I found screen to be a bit awkward as a serial connection, (use Ctrl-A : quit) to quit. I tried autologin for the lucid image but that was not needed for the master image which uses the linaro overlay. The PDU commands are for my own setup. The daemon (the machine running the pdu daemon scripts), hostname (the hostname or IP of the PDU itself) and the port numbers will change between labs. Again, the blog has changed the line endings.

device_type = mx53loco
hostname = imx53-01
connection_command = telnet hobbes 4002
#login_prompt = lucid-desktop login:
#password_prompt = Password:
#username = lucid
#password = lucid
hard_reset_command = /usr/bin/pduclient --daemon localhost --hostname pdu --command reboot --port 05
power_off_cmd = /usr/bin/pduclient --daemon localhost --hostname pdu --command off --port 05
power_on_cmd = /usr/bin/pduclient --daemon localhost --hostname pdu --command on --port 05
testboot_offset = 3

Critical testboot_offset is currently undocumented. Without it, you will see:

 ** Partition 4 not valid on device 0 **

You can trace this back in a LAVA test log to:

  setenv bootcmd "fatload mmc 0:4 0x70000000 uImage; fatload mmc 0:4 0x72000000 uInitrd; fatload mmc 0:4 0x71ff0000 board.dtb; bootm 0x70000000 0x72000000 0x71ff0000"

The correct line needs to use mmc 0:5 in each case.
This is what testboot_offset modifies the default is 2.

  setenv bootcmd "fatload mmc 0:4 0x70000000 uImage; fatload mmc 0:5 0x72000000 uInitrd; fatload mmc 0:5 0x71ff0000 board.dtb; bootm 0x70000000 0x72000000 0x71ff0000"

Hacking session in LAVA A hacking session in LAVA is a way of getting an SSH login directly into a deployed test image, as root. It s a test image, so look around, trash stuff, build stuff, reboot the board When you are done, use stop-hacking and the test completes, leaving the master image unchanged, ready for the next rootfs to go onto the test image you were using. This will be improved in later blog entries but as a starter and so that you can play with the SATA drive with a recent installation, this is the JSON for a LAVA job which gives you an SSH session inside the LAVA test image on an iMX53. The test image comes from Linaro releases 12.01. I haven t set up a proxy yet, so to save time, I downloaded it once and used a local file:// URL in the testjob. Note the rootfstype parameter without this, LAVA will helpfully format the partition onto which the rootfs is installed to ext4, with predictable results. (This bug is currently being fixed in LAVA so that LAVA uses the same filesystem as the downloaded image. All part of making LAVA less presumptive.) The smoke-tests-basic testdef is gratuitous and can be removed but this particular run won t be quick it took over an hour to get into the hacking session, but future topics here will improve that so the extra few minutes for the smoke-tests is not a big win. Change the value for GATEWAY to something sane for your network, change the value for PUB_KEY to your SSH public key. Change the image location to something which works for your setup.

 
 "health_check": false,
 "job_name": "iMX53-hacking",
 "logging_level": "DEBUG",
 "device_type": "mx53loco",
 "timeout": 900,
 "actions": [
 
 "command": "deploy_linaro_image",
 "parameters": 
 "image": "file:///home/linaro/lava/mx53loco-ubuntu-desktop.img.gz",
 "rootfstype": "ext3"
 
 ,
 
 "command": "lava_test_shell",
 "parameters": 
 "testdef_repos": [
 
 "git-repo": "git://git.linaro.org/qa/test-definitions.git",
 "testdef": "ubuntu/smoke-tests-basic.yaml"
 ,
 
 "git-repo": "http://git.linaro.org/git/lava-team/hacking-session.git",
 "parameters": 
 "GATEWAY": "10.15.0.1",
 "PUB_KEY": "...."
 ,
 "testdef": "hacking-session-debian.yaml"
 
 ],
 "timeout": 18000
 
 ,
 
 "command": "submit_results_on_host",
 "parameters": 
 "server": "http://sylvester.codehelp/RPC2/",
 "stream": "/anonymous/codehelp/"
 
 
 ]
 

(Use http://jsonlint.com to reformat.) I was part of the way through running debootstrap on the SATA drive when ser2net timed out because I hadn t changed the values at that time. Configuring set2net for LAVA ser2net can run on any machine capable of providing the serial connections, including a dedicated device. Decide on a port range to use on the machine running ser2net. If the device has more than one network interface, ensure that ser2net only offers connections on the relevant interface by prefixing the port with the IP address or hostname. telnet is generally the easiest ser2net state to use with LAVA.

#  ::::
#     TCP port
#            Name   or  number of the TCP/IP port to accept con-
#            nections from for this device.  A port number may
#            be of the form [host,]port, such as 127.0.0.1,2000
#            or localhost,2000.  If this is specified, it will
#            only bind to the IP address specified. Otherwise
#            it will bind to all the ports on the machine.
# 10.15.0.1,4000:telnet:600:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT banner

Clear the timeout or long running jobs will fail.

10.15.0.1,4000:telnet:0:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT banner
10.15.0.1,4001:telnet:0:/dev/ttyUSB1:115200 8DATABITS NONE 1STOPBIT banner

I ve tried relatively carefully to preserve the instructions here from my notes. However, if you do try this yourself and there are problems, let me know and I ll update. I m also planning on putting all of the config files, patches, JSON examples and other code into a git repo fairly soon to make things easier.

The uses of this particular rack I ll cover in future entries this is about how I made the rack itself, with help from friends (Steve McIntyre & Andy Simpkins). A common theme is making allowances for using dev kit boards ready-to-rack ARM servers are not here yet. My aim was to have 4, possibly 6 ARM dev kit boards running services from home, so there was no need for a standard 42U rack, a 9U rack is enough. Hence:
Wall Mounted 9U 450mm deep 19 inch rack enclosure glass door To run various other services around the house (firewall, file server etc.), a microserver was also necessary:
HP 704941-421 ProLiant Micro Server (AMD Turion II Neo N54L 2.2GHz, 2GB RAM, 250GB HDD, 2 Core, 7th Generation) I chose to mount that on a bookcase beneath the wall mounted rack as it kept all the cables at the bottom of the rack itself. The microserver needed a second gigabit network card fitted to cope with being a firewall as well, if you do the same, ensure you find a suitable card with a low profile bracket. Some are described as being low profile but do not package the low profile bracket, only a low profile card and a full height bracket.
Intel EXPI9301CTBLK PRO1000 Network Card note the low profile bracket in the pack. The first of the dev kit requirements is the lack of boards which can be racked, so shelves are going to be needed, leading on to something to stop the boards wandering across the shelf when the cables are adjusted velcro pads in my case. Second requirement is that dev kit boards are notorious for not rebooting cleanly. Nothing to do with the image being run, the board just doesn t cut power or doesn t come back after cutting power. Sometimes this is down to early revisions of the board, sometimes the board pulls enough power through the USB serial converter to remain alive, whatever the cause, it won t reboot without user involvement. So a PDU becomes necessary remotely controllable. New units tend to be expensive and/or only suitable for larger racks, I managed to find an older 8 port APC unit, something like: (Don t be surprised if that becomes a dead link search for APC Smart Slot Master Switch Power Distribution Unit). Talking of power, I m looking to use SATA drives with these boards and the boards themselves come with a variety of wall wart plugs or USB cables, so a number of IEC sockets are needed not the usual plugs:
Power cable IEC C14 plug 13A socket 25 cm or, for devices which genuinely need 2A to boot (use the 1A for attached SATA or leave empty):

Black Universal 3.1A 15W 5V Dual USB Rapid Mains Charger Plug Check the power output rating of the USB plugs used to connect to the mains as well many are 1A or less. Keep the right plug for the right board.
Power is going to also be a problem if, like me, you want to run SATA drives off boards which support SATA. The lack of a standard case means that ATX power is awkward, so check out some cheap SATA enclosures to get a SATA connector with USB power. I am using these enclosures (prices seem to have risen since I obtained them):

Startech 2.5 inch eSATA USB External Hard Drive Enclosure for SATA HDD Along with these:
eSATA to SATA Serial External Shielded Cable 1m because the iMx53 boards have SATA connectors but the enclosure exports eSATA. Whilst this might seem awkward, the merit of having both eSATA and simple USB power on one enclosure is not to be under-estimated. (Avoid the stiffer black cables space will get tight inside the rack.) Naturally, a 2.5 inch SATA drive is going to be needed for each enclosure, I m using HDD but SSD is also an option. Also, consider simple 2 or 3 way fused adaptors so that the board and the SATA drive can be powered from a single PDU port, this makes rebooting much simpler if the board needs a power supply with integrated plug instead of over USB. Now to the networking (2 x 8 port was cheaper than 1 x 16):
Netgear GS108 8-port Gigabit Ethernet Unmanaged Switch Don t forget the cat5 cables too you ll want lots of short cables, 1m or shorter inside the rack and a few longer ones going to the microserver and wall socket. I used 8x1m. Naturally, on the floor below your rack you are going to put a UPS, so the PDU power then needs to be converted to draw from the UPS via IEC plugs instead of standard mains. I decided to use a 6 gang extension 1m cable with an IEC plug it was the only bit of wiring I had to do and even those are available ready-made if you want to do it that way. Depending on the board, you may need your own serial to USB converters, you ll certainly need some powered USB hubs. I m using a wall mounted 9U rack, so I also needed a masonry drill and 4 heavy duty masonry M8 bolts. The rack comes with a mounting plate which needs to be bolted to the wall but nothing else is included. This step is much easier with someone else to take the weight of the rack as it is guided into the brackets on the mounting plate the bracket may need a little persuasion to allow for the bolt heads to not get in the way during mounting. Once mounted, the holes in the back of the rack allow for plenty of room, it s just getting to that point. The rack has side panels which latch out easily, providing easy maintenance. The glass door can be easily reversed to open from the opposite side. However, the key in the glass door is largely useless. The expectation is that the units in the rack are attached at the front but the dev boards on shelves are not going to be protected by a key in the front door. The key therefore ends up being little more than a handle for the glass door. OK. If you ve got this far, it s a case of putting things together:
Economy Cage Nut Tool 19 inch racking for cagenut extraction Yes, you really do want one. Fine, do without the premium one, but the economy one will save you a lot of (physical) pain. At this stage, it becomes clear that the normal 19 inch server rack shelves don t leave a lot of room at the back of the rack for the cables and there are a lot of cables. Each board has power, USB serial connection and network. The SATA has power too. The PDU has a power lead and you ll need network switches too. The network switches need power and an uplink cable. I positioned the supports in the rack as far forward as possible and attached the shelves to give enough room for the PDU on the base of the rack, the network switches resting on top and the extension bar (with the heavier, stiffer cables) at the back. (I need to bring my shelves another 1 or 2 positions further forward as there is barely enough room for one cable between the shelf and the back of the rack and that makes moving boards around harder than it needs to be.) The PDU defaults to enabling all ports at startup, so connect to it over telnet and turn off the ports before connecting things and sorting out the network interface to what the rest of the lab needs. (I m using a 10. range and the PDU was originally set to use 192.168.1.) That s about it as far as the hardware setup is concerned. Just time to label up each board, note down which PDU port does which device, which serial to USB converter is on which device on the microserver and check the power my initial problem with one board was traced to the inability of the board to power SATA off the on-board USB port even when using the provided 2A power supply. That meant adding a standard mains adaptor to feed both the SATA power and the board power off the one PDU port there is little point powering off the board but not the SATA, or vice versa. I haven t totalled up the expenditure but the biggest expenses were the microserver and the wall mounted rack but don t underestimate how much it will cost to buy 6 IEC plugs, various USB serial converters and how much you may spend on peripheral items. There is quite a lot of room on the 2 shelves for more boards, what will limit the deployment in this rack is space for the cables, especially power. The shorter the power cables, the easier it will be to maintain the devices in the rack. It might be worth looking at a 12U rack, if only to give plenty of space for cables. Once I ve got the software working, I ll describe what this rack will be doing it s got something to do with Debian, ARM, Linux and tests but you ve probably already guessed that much

PyBit is a distributed build system able to build packages in response to VCS commits or other triggers, across multiple architectures, multiple clients and multiple build environments with automated uploads to a nominated repository.

Support is included in 1.0.0 for building Debian packages using sbuild in response to subversion commits or changes in debian-devel-changes@lists.debian.org (by using apt as a version control handler) for any architecture and build environment which sbuild can support. There is also an example git commit template. Pybit has been designed to be fully extensible, so support for RPM or other package formats can be added as well as other version control handlers, other build environments and other architectures. Pybit is also scalable, when one type of client is struggling with the workload, another machine of the same architecture can be added to the pool to share the load. Pybit can also build a package for any number of architectures and build environments at the same time. The Pybit web interface provides an at-a-glance summary of all current builds as well as options to blacklist certain combinations, cancel and retry specific jobs and add monitor each pybit client. Current use cases include:

• Rapidly changing VCS - one or more subversion repositories with lots of Debian packages, built automatically for any number of build environments and architectures every time the debian/changelog is modified. Clean chroot builds provide continuous integration testing of the every package.


• Rebuilding the archive with different compilers or flags - a dedicated email account subscribed to debian-devel-changes@lists.debian.org feeding messages through procmail to the changes-debian hook, passing build requests to the apt handler to rebuild each package in your own sbuild chroots, using whatever environments, suites and build options can be configured within those chroots.


• something else we haven't thought of yet ... there is scope for a lot more hooks, package formats, chroot tools and handler plugins.




Pybit 1.0.0 has arrived in Debian unstable as a direct result of the efforts put in by the pybit team during a sprint on 18th May 2013. Thanks to everyone involved in Pybit.

https://freecode.com/projects/pybit

http://nicholasdavidson.github.io/pybit/

https://github.com/nicholasdavidson/pybit

klibc_2.0.1-3.2_arm64.changes



I've now completed the first patch for klibc 2.0.1 for arm64 support - it's completely untested and likely to break things, so it's up for review not usage. Those who are brave enough to look at the patch, please report issues via the github issue tracker.

35 files changed, 509 insertions(+), 19 deletions(-)

21 new C files to replace removed syscalls, most of which simply borrow from glibc and provide a general purpose call (like open) instead of the actual syscall (openat).

Next is to test the build and look into what is necessary to push the patch to the current 2.0.3 upstream.

My 1Tb laptop drive started misbehaving a few weeks ago, just spending a lot of time spinning when it should have been reading frequently changed files (like the browser cache). I was tempted to blame the browser at that point as no other packages appeared affected.

Wednesday night, a routine package upgrade on unstable brought in a bunch of qt4 updates which I wanted and a virtual box update that I didn't see much point in delaying ... 15 minutes later I couldn't work out why the virtualbox dkms task was still running, spotted it spinning in depmod and found some alarming messages in dmesg about short reads and errors reading from the hard drive. Hmm. The hard drive was out of control at this point, it quickly became apparent that no disc access was going to be possible and I couldn't get to a terminal to kill the current tasks, so I killed the power. The fsck which followed reboot showed more of the errors I'd seen in dmesg and fell back to the manual intervention stage. A few hours of confirming attempts to fix the errors, fsck finally finished. A short amount of usage showed that although fsck had finished, the drive was not happy and was starting to give short reads on other parts of the filesystem, resulting in ~40% of the filesystem appearing to be read-only when the rest was read-write. Somehow, I didn't think this was a welcome feature as the areas affected appeared to be quite random.

The drive in question was a replacement for the original 300Gb drive supplied with the ThinkPad, so a quick bit of switching of drives into a caddy and I could rsync my data off the large drive onto the smaller one. The rsync itself took a lot longer than it should have done because it got lots of short reads too. (Principally in /lib/modules/3.2.0-4/ and in the browser cache directories which had been the original symptom as well as most other places where one could have expected processes to have open files when the drive failed).

Now the 1Tb drive was a pig to fit into the Thinkpad originally because it was too big for the bay but I fitted it anyway. Yes, that was probably a mistake. It certainly meant that in order to fit it I had to not put the drive into the useful caddy provided by Lenovo which makes removal of the drive simple. Indeed the drive was wedged into the bay so tightly that it wasn't going to come out with normal levels of persuasion. This probably contributed to the failure of the drive, so live and learn.

With help from Andy Simpkins (it's always handy to have a hardware engineer on hand at times like this), the keyboard was lifted out, the case was dismantled and just enough room was made to get a screwdriver in behind the sata drive and lever it out of the bay. OK, rebuild laptop, put replacement drive into the caddy (because the smaller capacity drive is also a lot smaller in height than the 1Tb and therefore has plenty of clearance between it and the bay) and move on to the software recovery stage.

Hint: if this happens again, before turning off the broken system for the last time, just remember to download a recent Debian ISO to a USB stick - it saves having to ask someone else or find another machine to do the download. (Thanks Andy...)

OK, so after the usual complaints on reboot that there was no operating system, F12 got the boot order menu up and I was in Debian Installer Rescue Mode. Reinstalling grub failed initially for a few reasons:

1. 
   I'd deliberately not copied /dev /proc and /sys from the old harddrive but I'd also forgotten to create empty ones on the new drive...
2. /etc/fstab helpfully referred to / as a UUID which no longer applied, so that had to be edited
3. grub was confused and couldn't find the root filesystem

A few iterations later, I had a working /dev directory inside the /target chroot, bind mounted from the /dev outside the chroot, I was able to mount proc and sys, so grub was finally happy to reinstall itself and then update the initramfs setup for the new drive.

Reboot, another fsck, all appeared well. I was able to login via the terminal but not in X. Hmmm. Stop xdm, startx manually from the terminal, problems with /tmp/ - permission denied. Oops. Yes, it does help to create /tmp with the right permissions....

The final stage was to complete the dpkg --configure -a from the original failure. I've yet to reinstall the 3.2.0-4 kernel, so I'm back on 3.2.0-3 but that's OK.

So now I'm back on the original drive, albeit temporarily without any swap whatsoever (because I didn't partition the replacement drive to create /dev/sda5 before doing the copy) and now I remember the second reason I wanted to replace the original drive with the 1Tb drive - the original drive is as NOISY as hell. The whole edge of the laptop vibrates constantly, to the point that I can feel the vibration under the keys as I type. It's not that the drive is loose in the bay, it's just a constant vibration.

But, I have kept all my data and I have a usable laptop for the BSP this weekend. I will be looking at an SSD drive to replace this one though and having also found my old Acer laptop with power supply, I can now reference this entry when I transfer the system a second time.

With significant assistance from Steve McIntyre and some judicious delving into the ARM Information Centre, I've now got the assembly portions of klibc sorted (but untested) for AArch64 (arm64).

Andy & I started by copying the old ARM support as a new directory and one of the final steps was to remove a whole bunch of legacy code from the days before Thumb and all the #ifdef lines which went with it. Some files disappeared entirely. setjmp.S was the largest amount of work as the load&store multiple support of ARMv7 has gone in ARMv8, so the stmia mnemonic had to be expanded to multiple stp calls but that makes it more explicit, so it's not a bad thing.

Steve & I borrowed from the glibc AArch64 code from upstream by Marcus Shawcroft, simplifying the glibc macros for klibc and also got clues about what extra registers needed handling compared to ARMv7.

I've now got to look at some traditional cross-compilation issues because the Linaro AArch64 toolchain doesn't install to typical Debian cross-building paths and the build now moves past the AArch64-specific assembly and fails later when the C code gets the wrong include path and ends up including /usr/include/i386-linux-gnu/asm/byteorder.h with predictable results.

If someone has an AArch64 toolchain already set up, feel free to clone my modified klibc tree and let me know if there are subsequent build errors. Of course, if you fancy testing a build in the Foundation Model for ARMv8, that would be good too! (Report issues via github.)

Whilst I'm sorting out the toolchain, I've also been updating perl-cross-debian (which has also seen some more upstream testing and improvement).

Working with perl upstream, we're getting closer to a fully cross-built upstream perl without needing the external perl installation. The branch (which is also available here with a few of my changes) now builds a host miniperl, cross-builds the rest of perl and almost gets through the rest of the build by using the host miniperl to handle the extensions up as far as XS::Typemap:

make[1]: Leaving directory  /home/neil/code/debian/src/perl/cross/git/perl/ext/XS-APItest'

./miniperl -Ilib make_ext.pl lib/auto/XS/Typemap/Typemap.so MAKE=make LIBPERL_A=libperl.a LINKTYPE=dynamic

	Making XS::Typemap (all)

make[1]: Entering directory  /home/neil/code/debian/src/perl/cross/git/perl/ext/XS-Typemap'

make[1]: Leaving directory  /home/neil/code/debian/src/perl/cross/git/perl/ext/XS-Typemap'

Making all in ext/XS-Typemap

make all PERL_CORE=1 LIBPERL_A=libperl.a LINKTYPE=dynamic

make[1]: Entering directory  /home/neil/code/debian/src/perl/cross/git/perl/ext/XS-Typemap'

make[1]: Leaving directory  /home/neil/code/debian/src/perl/cross/git/perl/ext/XS-Typemap'

./perl -f -Ilib pod/buildtoc -q

Can't load module Digest::MD5, dynamic loading not available in this perl.

  (You may need to build a new perl executable which either supports

  dynamic loading or has the Digest::MD5 module statically linked into it.)

 at Porting/pod_lib.pl line 4.

Compilation failed in require at Porting/pod_lib.pl line 4.

BEGIN failed--compilation aborted at Porting/pod_lib.pl line 4.

Compilation failed in require at pod/buildtoc line 17.

BEGIN failed--compilation aborted at pod/buildtoc line 18.

make: *** [pod/perltoc.pod] Error 2

Note the change from ./miniperl (which is itself a bug as it should be ./host/miniperl) to ./perl which is, naturally, an armel binary. It's also copied into the local directory, replacing the system perl if I copy it in.

So, more to do, but at least it gets this far.

Improved support for the extensions should also make it easier to clean up the current Debian cross-build diff which is the remaining bit of awkwardness / kludge.

I'm still working on perl-cross-debian (just uploaded 0.0.2) and there's more to do on that with upstream but part of the reason to work on perl cross-building is to do what I can to help with the ARM64 port.

So, I went back to Wookey's talk at DebConf12 which leads to the current list of cross-build results for arm64 and started through the list.

coreutils is listed as failing but that was an archive error (MD5sum hash mismatch), so that just needs a retry. I don't have access to that buildd, yet, so nothing I can do there.

Next on the list (excluding those just waiting for build-deps) was klibc.

  aarch64-linux-gnu-gcc -Wp,-MD,usr/klibc/.__static_init.o.d  -nostdinc -iwithprefix 

include -I/ PKGBUILDDIR /usr/include/arch/x86_64 -Iusr/include/arch/x86_64 

-I/ PKGBUILDDIR /usr/include/bits64 

-Iusr/include/bits64 -I/ PKGBUILDDIR /usr/klibc/../include 

-Iusr/klibc/../include -I/ PKGBUILDDIR /usr/include -Iusr/include 

-I/ PKGBUILDDIR /linux/include -Ilinux/include 

-I/ PKGBUILDDIR /linux/arch/x86/include -Ilinux/arch/x86/include 

-D__KLIBC__=2 -D__KLIBC_MINOR__=0 -D_BITSIZE=64 -fno-stack-protector -fwrapv -m64 -Os 

-fno-asynchronous-unwind-tables -fomit-frame-pointer -falign-functions=1 -falign-jumps=1 -falign-loops=1 

-W -Wall -Wno-sign-compare -Wno-unused-parameter -c -o usr/klibc/__static_init.o usr/klibc/__static_init.c

aarch64-linux-gnu-gcc: error: unrecognized command line option '-m64'

make[4]: *** [usr/klibc/__static_init.o] Error 1

Turns out that this is a build failure I understood, at least initially. A little digging and a trivial patch was begun:

--- a/usr/klibc/arch/arm/MCONFIG

+++ b/usr/klibc/arch/arm/MCONFIG

@@ -30,6 +30,13 @@

 ifeq ($(CONFIG_AEABI),y)

 KLIBCREQFLAGS += -mabi=aapcs-linux -mno-thumb-interwork

 else

+# aarch64

+ifeq ($(CONFIG_AARCH64),y)

+KLIBCREQFLAGS +=

+KLIBCOPTFLAGS += -mgeneral-regs-only

+else

 KLIBCREQFLAGS += -mabi=apcs-gnu -mno-thumb-interwork

 endif

 endif

+endif

+

Alongside a trivial change to debian/rules

ifeq ($(DEB_HOST_ARCH),arm64)

DEB_MAKE_ENVVARS := ARCH=arm CONFIG_AARCH64=y CPU_ARCH=armv8-a CPU_TUNE=generic

endif

OK, then things get a bit more awkward,

....

code/debian/cross/klibc/klibc-2.0.1/linux/arch/arm/include -Ilinux/arch/arm/include -D__KLIBC__=2 

-D__KLIBC_MINOR__=0 -D_BITSIZE=32 -fno-stack-protector -fwrapv -fno-exceptions -Os 

-march=armv8-a -mtune=generic -mgeneral-regs-only -W -Wall -Wno-sign-compare -Wno-unused-parameter 

-c -o ../foo.o usr/klibc/arch/arm/crt0.S

usr/klibc/arch/arm/crt0.S: Assembler messages:

usr/klibc/arch/arm/crt0.S:19: Error: operand 1 should be an integer register --  mov r0,sp'

usr/klibc/arch/arm/crt0.S:20: Error: operand 1 should be an integer register --  mov r1,#0'

Hmm. Assembly, well, yes, I've done assembly before, I know what mov should normally do, sp is likely to be the stack pointer .... where's my AARCH64 assembly PDF again... PRD03-GENC-010197 ...

OK, so maybe the r0 and r1 should be x0 and x1, hmm, that at least doesn't raise assembly errors. So a tentative change:

--- a/usr/klibc/arch/arm/crt0.S

+++ b/usr/klibc/arch/arm/crt0.S

@@ -15,9 +15,13 @@

 #ifdef __thumb__

 	.thumb_func

 #endif

-

+#ifdef __aarch64__

+_start: mov x0, sp

+	mov x1, #0

+	bl __libc_init

+#else

 _start:	mov	r0, sp

 	mov	r1, #0

 	bl	__libc_init

-

+#endif

 	.size _start,.-_start

Next stage, however, leaves me quite a bit more lost:

....

klibc-2.0.1/linux/arch/arm/include -Ilinux/arch/arm/include 

-D__KLIBC__=2 -D__KLIBC_MINOR__=0 -D_BITSIZE=32 -fno-stack-protector -fwrapv -fno-exceptions -Os 

-march=armv8-a -mtune=generic -mgeneral-regs-only -W -Wall -Wno-sign-compare -Wno-unused-parameter 

-c -o usr/klibc/arch/arm/setjmp.o usr/klibc/arch/arm/setjmp.S

usr/klibc/arch/arm/setjmp.S: Assembler messages:

usr/klibc/arch/arm/setjmp.S:32: Error: unknown mnemonic  stmia' --  stmia r0, r4,r5,r6,r7,r8,r9,r10,fp,sp,lr '

usr/klibc/arch/arm/setjmp.S:33: Error: operand 1 should be an integer register --  mov r0,#0'

usr/klibc/arch/arm/setjmp.S:34: Error: operand 1 should be an integer register --  mov pc,lr'

usr/klibc/arch/arm/setjmp.S:42: Error: unknown mnemonic  ldmia' --  ldmia r0, r4,r5,r6,r7,r8,r9,r10,fp,sp,lr '

usr/klibc/arch/arm/setjmp.S:43: Error: unknown mnemonic  movs' --  movs r0,r1'

usr/klibc/arch/arm/setjmp.S:44: Error: unknown mnemonic  moveq' --  moveq r0,#1'

usr/klibc/arch/arm/setjmp.S:45: Error: operand 1 should be an integer register --  mov pc,lr'

make[5]: *** [usr/klibc/arch/arm/setjmp.o] Error 1

So now I'm out of my depth of AARCH64 assembly (apart from the recurrence of mov r0 vs mov x0 etc.). If the above is useful then maybe someone can work out what is wrong with setjmp.S or whether AARCH64 just means that klibc needs to gain a arch/arm64/ directory and not try to duplicate each entire assembly block within #ifdef clauses.

I don't really know where else to put an incomplete investigation like this, so it's here for anyone to find.

(Oh, and if you're reading those arm64 cross-build logs, then a few hundred occurrences of ldconfig: /usr/lib/aarch64-linux-gnu/*.so is for unknown machine 183. in every build log (success or fail) is apparently entirely normal until more packages get fixed. Really slows down scanning the build log.

I may try busybox or libusb next. libusb looks like a classic "you might have told me to cross-compile but I'm going to use g++ anyway because I know best" cross-building problem, indicative of yet another BDBS. sigh.

Resources:
Getting started with 64-bit ARM development
ARMv8 images for developers
AArch64 for everyone, Marcin Juszkiewicz
Howto/HelloAarch64 - Linaro wiki
AArch64 gcc options

This is the listing for my local cross-build-only repository for perl:

perl   5.14.2-16   unstable   armel, source

perl   5.16.2-1   experimental   armel, source

Prior to this, 5.14.2-15 also cross-built.

I've just pushed the update which fixes 5.16 from current Debian experimental.

This means that I'm ready to push perl-cross-debian into experimental via NEW. Whilst the package is in NEW, I will be approaching perl upstream about the necessary changes for Makefile.SH and updating the existing bug reports #285559 and #633884 with the necessary changes for debian/rules.

In the meantime, the changes for Makefile.SH and debian/rules exist within the perl-cross-debian source code - the patch for 5.14.2 is the same as I used for 5.16.2 and I don't see a need, yet, for this to be any different with current perl upstream. Equally, the patch for debian/rules works equally well for 5.14.2 and 5.16.2. All the version-specific (and architecture-specific) data lives in perl-cross-debian.

So it's time to tag perl-cross-debian 0.0.1 and upload to ftp-master as a native package aimed at experimental.

What's left to do? TESTING!

I've only tested with armel using the Emdebian cross-building toolchains from Squeeze using the old dpkg-cross style cross-dependency installation paths. There is outline code for armhf and arm64 but these need testing. The code also needs testing using the latest MultiArch cross-building toolchains. This should be a simple matter of checking if the dpkg-cross style paths exist and looking for MultiArch if not.

Right now, all of this is "worksforme" grade. It needs others to have a go and file bugs. Until the package is through NEW, feel free to use the issue tracker on the perl-cross-debian github site.

Please read through the documentation in the source code and the manpages in the package (xml in the source code) and tell me if some of it isn't clear.

Long term maintenance of cross-build support for the Debian configuration of perl has now gained some code at github and an ITP: #694326 for Debian.

There's some working code for perl 5.14 and initial work on 5.16 (which isn't complete yet).

This will dramatically simplify the patch for #633884 and provide a base for getting another part of that patch into upstream (Makefile.SH). (Thanks to Steve McIntyre & Peter Pearse for the body of the patch itself.)

ifneq ($(DEB_HOST_GNU_TYPE),$(DEB_BUILD_GNU_TYPE))



define variant

$(if $(findstring static,$1),static,$(if $(findstring debug,$1),debug,shared))

endef



define cross-config

    /usr/bin/perl-cross-debian --variant $(call variant,$@)

    perl -Ilib  make_patchnum.pl

endef

endif

So the config*variant files will live in /usr/share/perl-cross-debian/$arch/$perl_version/