Here are my notes about copying PGP keys to external hardware devices such as
Yubikeys. Let me begin by saying that the gpg tools are pretty bad at this.
MAKE A COUPLE OF BACKUPS OF ~/.gnupg/
TO DIFFERENT ENCRYPTED USB STICKS
BEFORE YOU START. GPG WILL MESS UP YOUR KEYS. SERIOUSLY.
For example, would you believe me if I said that saving changes results in
the removal of your private key? Well
check this
out.
Now that you have multiple safe, offline backups of your keys, here are my notes.
apt install yubikey-manager scdaemon
Plug the Yubikey in, see if it s recognized properly:
ykman list
gpg --card-status
Change the default PIN (123456) and Admin PIN (12345678):
gpg --card-edit
gpg/card> admin
gpg/card> passwd
Look at the openpgp information and change the maximum number of retries, if
you like. I have seen this failing a couple of times, unplugging the Yubikey
and putting it back in worked.
ykman openpgp info
ykman openpgp access set-retries 7 7 7
Copy your keys. MAKE A BACKUP OF ~/.gnupg/
BEFORE YOU DO THIS.
gpg --edit-key $KEY_ID
gpg> keytocard # follow the prompts to copy the first key
Now choose the next key and copy that one too. Repeat till all subkeys are
copied.
gpg> key 1
gpg> keytocard
Typing gpg --card-status
you should be able to see all your keys on the
Yubikey now.
Using the key on another machine
How do you use your PGP keys on the Yubikey on other systems?
Go to another system, if it does have a ~/.gnupg
directory already move it
somewhere else.
Import your public key:
gpg -k
gpg --keyserver pgp.mit.edu --recv-keys $KEY_ID
Check the fingerprint and if it is indeed your key say you trust it:
gpg --edit-key $KEY_ID
> trust
> 5
> y
> save
Now try gpg --card-status
and gpg --list-secret-keys
, you should be able to
see your keys. Try signing something, it should work.
gpg --output /tmp/x.out --sign /etc/motd
gpg --verify /tmp/x.out
Using the Yubikey with Mutt
If you re using mutt with IMAP, there is a very simple trick to safely store
your password on disk. Create an encrypted file with your IMAP password:
echo SUPERSECRET gpg --encrypt > ~/.mutt_password.gpg
Add the following to ~/.muttrc
:
set imap_pass= gpg --decrypt ~/.mutt_password.gpg
With the above, mutt now prompts you to insert the Yubikey and type your PIN in
order to connect to the IMAP server.
This time it s about enabling new kernel config options in the official Debian
kernel packages. A few dependencies are needed to run the various scripts used
by the Debian kernel folks, as well as to build the kernel itself:
apt install git gpg python3-debian python3-dacite
apt build-dep linux
With that in place, fetch the linux
and kernel-team
repos:
git clone --depth 1 https://salsa.debian.org/kernel-team/linux
git clone --depth 1 https://salsa.debian.org/kernel-team/kernel-team
So far you ve only got the Debian-specific bits. Fetch the actual kernel
sources now. In the likely case that you re building a stable kernel, run the
following from within the linux
directory:
debian/bin/genorig.py https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
Use the torvalds
repo if you re building an RC version instead:
debian/bin/genorig.py https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Now generate the upstream tarball as well as debian/control
. The first
command will take a bit, and the second command will fail: but that s success just as the output says.
debian/rules orig
debian/rules debian/control
Now generate patched sources with:
Time to edit the Kconfig and enable/disable whatever setting you wanted to
change. Take a look around the files under debian/config/
to see where your
changes should go. If it s a setting shared among multiple architectures that
may be debian/config/config
. For x86-specific things, the file is
debian/config/amd64/config
. On aarch64 debian/config/arm64/config
. If in
doubt, you could try asking #debian-kernel
on IRC.
It may look like you need to figure out where exactly in the file the setting
should be placed. That is not the case. There s a helpful script fixing things
up for you:
../kernel-team/utils/kconfigeditor2/process.py .
The above will fail if you forgot to run debian/rules source
. The
debian/build/source_rt/Kconfig
file is needed by the script:
Traceback (most recent call last):
File "/tmp/linux/../kernel-team/utils/kconfigeditor2/process.py", line 19, in __init__
menu = fs_menu[featureset or 'none']
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'rt'
During handling of the above exception, another exception occurred:
[...]
FileNotFoundError: [Errno 2] No such file or directory: './debian/build/source_rt/Kconfig'
If that happens, run:
Now process.py
should work fine and fix your config
file.
Excellent, now the config is updated and we re ready to build the kernel. Off
we go:
export MAKEFLAGS=-j$(nproc)
export DEB_BUILD_PROFILES='pkg.linux.nokerneldbg pkg.linux.nokerneldbginfo pkg.linux.notools nodoc'
dpkg-buildpackage -b -nc -uc
To create a bootable EFI drive to use with QEMU, first make a disk image and
create a vfat
filesystem on it.
$ dd if=/dev/zero of=boot.img bs=1M count=512
$ sudo mkfs.vfat boot.img
By default, EFI firmwares boot a specific file under /efi/boot/
. The name of
such file depends on the architecture: for example, on 64 bit x86 systems it is
bootx64.efi
, while on ARM it is bootaa64.efi
.
Copy /usr/lib/grub/x86_64-efi/monolithic/grubx64.efi
from package
grub-efi-amd64-bin
to /efi/boot/bootx64.efi
on the boot image, and that
should be enough to start GRUB.
# mount boot.img /mnt/
# mkdir -p /mnt/efi/boot/
# cp /usr/lib/grub/x86_64-efi/monolithic/grubx64.efi /mnt/efi/boot/bootx64.efi
# umount /mnt/
Now get the x86 firmware from package ovmf
and start qemu
:
$ cp /usr/share/OVMF/OVMF_CODE.fd /tmp/code.fd
$ qemu-system-x86_64 -drive file=/tmp/code.fd,format=raw,if=pflash -cdrom boot.img
GRUB looks fine, but it would be good to have a kernel to boot. Let s add one
to boot.img
.
# mount boot.img /mnt
# cp vmlinuz-6.1.0-7-amd64 /mnt/vmlinuz
# umount /mnt/
Boot with qemu again, but this time pass -m 1G
. The default amount of
memory is not enough to boot.
$ qemu-system-x86_64 -drive file=/tmp/code.fd,format=raw,if=pflash -cdrom boot.img -m 1G
At the grub prompt, type the following to boot:
grub> linux /vmlinuz
grub> boot
The kernel will start and reach the point of trying to mount the root fs. This
is great but it would now be useful to have some sort of shell access in order
to look around. Let s add an initrd!
# mount boot.img /mnt
# cp initrd.img-6.1.0-7-amd64 /mnt/initrd
# umount /mnt/
There s the option of starting qemu in console, let s try that out. Start qemu
with -nographic
, and append console=ttyS0
to the kernel command line
arguments.
$ qemu-system-x86_64 -drive file=/tmp/code.fd,format=raw,if=pflash -cdrom boot.img -m 1G -nographic
grub> linux /vmlinuz console=ttyS0
grub> initrd /initrd
grub> boot
If all went well we are now in the initramfs shell. We can now run commands! At
this point we can see that the system has Secure boot disabled:
(initramfs) dmesg grep secureboot
[ 0.000000] secureboot: Secure boot disabled
In order to boot with Secure boot, we need:
-
a signed shim, grub, and kernel
-
the right EFI variables for Secure boot
The package shim-signed
provides a shim signed with Microsoft s key, while
grub-efi-amd64-signed
has GRUB signed with Debian s key.
The signatures can be shown with sbverify --list
:
$ sbverify --list /usr/lib/shim/shimx64.efi.signed
warning: data remaining[823184 vs 948768]: gaps between PE/COFF sections?
signature 1
image signature issuers:
- /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
image signature certificates:
- subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Windows UEFI Driver Publisher
issuer: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
- subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
issuer: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation Third Party Marketplace Root
Similarly for GRUB and the kernel:
$ sbverify --list /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed
signature 1
image signature issuers:
- /CN=Debian Secure Boot CA
image signature certificates:
- subject: /CN=Debian Secure Boot Signer 2022 - grub2
issuer: /CN=Debian Secure Boot CA
$ sbverify --list /mnt/vmlinuz
signature 1
image signature issuers:
- /CN=Debian Secure Boot CA
image signature certificates:
- subject: /CN=Debian Secure Boot Signer 2022 - linux
issuer: /CN=Debian Secure Boot CA
Let s use the signed shim and grub in the boot image:
# mount boot.img /mnt
# cp /usr/lib/shim/shimx64.efi.signed /mnt/efi/boot/bootx64.efi
# cp /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed /mnt/efi/boot/grubx64.efi
# umount /mnt
And start QEMU with the appropriate EFI variables for Secure boot:
$ cp /usr/share/OVMF/OVMF_VARS.ms.fd /tmp/vars.fd
$ qemu-system-x86_64 -drive file=/tmp/code.fd,format=raw,if=pflash -drive file=/tmp/vars.fd,format=raw,if=pflash -cdrom boot.img -m 1G -nographic
We can double-check in the firmware settings if Secure boot is indeed enabled.
At the GRUB prompt, type fwsetup
:
Check under "Device Manager" "Secure Boot Configuration" that "Attempt
Secure Boot" is selected, then boot from GRUB as before. If all went well, the
kernel should confirm that we have booted with Secure boot:
(initramfs) dmesg grep secureboot
[ 0.000000] secureboot: Secure boot enabled
Some notes on using debvm
, an amazing piece of software I ve started using
only recently.
Create a new virtual machine:
You now have a virtual machine with Debian Sid of your host native
architecture (probably amd64). The image file is called rootfs.ext4. You ve
got 1G of disk space in the VM.
You can now just run the VM! You will be automatically logged is as root.
Experiment in the VM, run all the sort of tests you have in mind. For example,
one thing I commonly do is verifying that things work in a clean environment,
as opposed to "on my machine".
If anything goes wrong, or if you just want to repeat the test: shutdown the
guest, remove rootfs.ext4
, and start again with debvm-create
.
Now, especially if you intend creating and recreating VMs multiple times, it
helps to use a local apt mirror as a cache to avoid fetching all the required
packages from the internet over and over again. Install apt-cacher-ng
on
your host machine and point debvm-create
at it:
$ debvm-create -- http://10.0.3.1:3142/debian
The additional options after
--
are passed to
mmdebstrap
. In this case
we re specifying
http://10.0.3.1:3142/debian as the URL of our local apt
mirror. It is going to be used both for image creation and as the only entry in
/etc/apt/sources.list
on the guest. This is the reason for using 10.0.3.1,
the IP address of the lxcbr0 interface used by qemu, instead of 127.0.0.1: to
make sure that the guest VM has access to it too.
For a slightly more advanced example, we now want to:
-
run a arm64 VM
-
have more disk space availably, say 2G
-
install additional packages (
curl
and locales
)
-
allow SSH as root with the given public keys, in this example the
authorized_keys
installed on the host
-
start the VM in the background with no console output
$ debvm-create -a arm64 -o sid-arm64.ext4 -z 2G -k ~/.ssh/authorized_keys -- http://10.0.3.1:3142/debian --include curl,locales
Start the VM:
$ debvm-run -i sid-arm64.ext4 -s 2222 -g -- -display none &
SSH into the guest:
$ ssh -o NoHostAuthenticationForLocalhost=yes -p 2222 root@127.0.0.1
Enjoy!
This post is mostly a sum-up of the Wikipedia page
History of Australia,
with some content taken from
History of the British Empire. Both texts are
available under the
Creative Commons Attribution-ShareAlike License.
I do not seem to be able to learn about a new topic without taking notes: in
this case I have decided to publish my work, hoping that someone will find it
useful. Some very important themes such as the Gold Rush and Australian History
during the World Wars have been impudently ignored.
Indigenous Australians
The ancestors of Indigenous Australians are believed to have arrived in
Australia 40,000 to 60,000 years ago, and possibly as early as 70,000 years
ago.
By 1788, the population of Australia existed as 250 individual nations, many
of which were in alliance with one another, and within each nation there
existed several clans, from as few as five or six to as many as 30 or 40.
Each nation had its own language and a few had multiple, thus over 250
languages existed, around 200 of which are now extinct.
Permanent European settlers arrived at Sydney in 1788 and came to control most
of the continent by end of the 19th century. Bastions of largely unaltered
Aboriginal societies survived, particularly in Northern and Western Australia
into the 20th century, until finally, a group of Pintupi people of the Gibson
Desert became the last people to be contacted by outsider ways in 1984.
European explorers
Terra Australis (Latin for South Land) is one of the names given to a
hypothetical continent which appeared on European maps between the 15th and
18th centuries. Although the landmass was drawn onto maps, Terra Australis was
not based on any actual surveying of such a landmass but rather based on the
hypothesis that continents in the Northern Hemisphere should be balanced by
land in the south.
The first documented European landing in Australia was made in 1606 by a Dutch
ship led by Willem Janszoon. Hence the ancient name "Nova Hollandia". The
same year, a Spanish expedition had landed in the New Hebrides and, believing
them to be the fabled southern continent, named the land: "Terra Austral del
Espiritu Santo". Hence the current name "Australia".
Although various proposals for colonisation were made, notably by Pierre Purry
from 1717 to 1744, none was officially attempted. Indigenous Australians were
less able to trade with Europeans than were the peoples of India, the East
Indies, China, and Japan. The Dutch East India Company concluded that there
was "no good to be done there".
In 1769, Lieutenant James Cook tried to locate the supposed Southern
Continent. This continent was not found, and Cook decided to survey the east
coast of New Holland, the only major part of that continent that had not been
charted by Dutch navigators.
Cook charted and took possession of the east coast of New Holland. He noted the
following in his journal:
"I can land no more upon this Eastern coast of New Holland, and
on the Western side I can make no new discovery the honour of
which belongs to the Dutch Navigators and as such they may lay
Claim to it as their property, but the Eastern Coast from the
Latitude of 38 South down to this place I am confident was never
seen or viseted by any European before us and therefore by the
same Rule belongs to great Brittan."
Colonisation
The American Revolutionary War (1775-1783) saw Great Britain lose most of its
North American colonies and consider establishing replacement territories.
The British colony of New South Wales was established with the arrival of the
First Fleet of 11 vessels in January 1788. It consisted of over a thousand
settlers, including 778 convicts (192 women and 586 men). A few days after
arrival at Botany Bay the fleet moved to the more suitable Port Jackson where
a settlement was established at Sydney Cove on 26 January 1788. This date
later became Australia's national day, Australia Day.
Between 1788 and 1868, approximately 161,700 convicts (of whom 25,000 were
women) were transported to the Australian colonies of New South Wales, Van
Diemen's land and Western Australia. Early colonial administrations were
anxious to address the gender imbalance in the population brought about by the
importation of large numbers of convict men.
In 1835, the British Colonial Office issued the Proclamation of Governor
Bourke, implementing the legal doctrine of terra nullius upon which British
settlement was based, reinforcing the notion that the land belonged to no one
prior to the British Crown taking possession of it and quashing any likelihood
of treaties with Aboriginal peoples, including that signed by John Batman. Its
publication meant that from then, all people found occupying land without the
authority of the government would be considered illegal trespassers.
A group in Britain led by Edward Gibbon Wakefield sought to start a colony
based on free settlement and political and religious freedoms, rather than
convict labour. The South Australia Act [1834], passed by the British
Government which established the colony reflected these desires and included a
promise of representative government when the population reached 50,000
people. Significantly, the Letters Patent enabling the South Australia Act
1834 included a guarantee of the rights of 'any Aboriginal Natives' and their
descendants to lands they 'now actually occupied or enjoyed'.
In 1836, two ships of the South Australia Land Company left to establish the
first settlement on Kangaroo Island. The foundation of South Australia is now
generally commemorated as Governor John Hindmarsh's Proclamation of the new
Province at Glenelg, on the mainland, on 28 December 1836. By 1851 the colony
was experimenting with a partially elected council.
Development of Australian democracy
Traditional Aboriginal society had been governed by councils of elders and a
corporate decision making process, but the first European-style governments
established after 1788 were autocratic and run by appointed governors.
The reformist attorney general, John Plunkett, sought to apply Enlightenment
principles to governance in the colony, pursuing the establishment of equality
before the law.
Plunkett twice charged the colonist perpetrators of the Myall Creek massacre
of Aborigines with murder, resulting in a conviction and his landmark Church
Act of 1836 disestablished the Church of England and established legal
equality between Anglicans, Catholics, Presbyterians and later Methodists.
In 1840, the Adelaide City Council and the Sydney City Council were
established. Men who possessed 1,000 pounds worth of property were able to
stand for election and wealthy landowners were permitted up to four votes each
in elections. Australia's first parliamentary elections were conducted for the
New South Wales Legislative Council in 1843, again with voting rights (for
males only) tied to property ownership or financial capacity. Voter rights
were extended further in New South Wales in 1850 and elections for legislative
councils were held in the colonies of Victoria, South Australia and Tasmania.
Women became eligible to vote for the Parliament of South Australia in 1895.
This was the first legislation in the world permitting women also to stand for
election to political office and, in 1897, Catherine Helen Spence became the
first female political candidate for political office, unsuccessfully standing
for election as a delegate to the Federal Convention on Australian Federation.
Western Australia granted voting rights to women in 1899. Early federal
parliamentary reform and judicial interpretation sought to limit Aboriginal
voting in practice, a situation which endured until rights activists began
campaigning in the 1940s.
Road to independence
Despite suspicion from some sections of the colonial community (especially in
smaller colonies) about the value of nationhood, improvements in
inter-colonial transport and communication, including the linking of Perth to
the south eastern cities by telegraph in 1877, helped break down
inter-colonial rivalries.
New South Wales Premier Henry Parkes addressed a rural audience in his 1889
Tenterfield Oration, stating that the time had come to form a national
executive government:
"Australia [now has] a population of three and a half millions,
and the American people numbered only between three and four
millions when they formed the great commonwealth of the United
States. The numbers were about the same, and surely what the
Americans had done by war, the Australians could bring about in
peace, without breaking the ties that held them to the mother
country."
Though Parkes would not live to see it, his vision would be achieved within a
little over a decade, and he is remembered as the "father of federation".
The Commonwealth of Australia came into being when the Federal Constitution
was proclaimed by the Governor-General, Lord Hopetoun, on 1 January 1901.
Australia took part in WWI. The contributions of Australian and New Zealand
troops during the 1915 Gallipoli Campaign against the Ottoman Empire had a
great impact on the national consciousness at home, and marked a watershed in
the transition of Australia and New Zealand from colonies to nations in their
own right. The countries continue to commemorate this occasion on ANZAC Day.
Australia achieved independent Sovereign Nation status after World War I,
under the Statute of Westminster, which defined Dominions of the British
empire in the following way:
"They are autonomous Communities within the British Empire, equal
in status, in no way subordinate one to another in any aspect of
their domestic or external affairs, though united by a common
allegiance to the Crown, and freely associated as members of the
British Commonwealth of Nations."
The parliaments of Canada, Australia, New Zealand, the Union of South Africa,
the Irish Free State and Newfoundland (currently part of Canada) were now
independent of British legislative control, they could nullify British laws
and Britain could no longer pass laws for them without their consent.
The Australia Act 1986 removed any remaining links between the British
Parliament and the Australian states.
Software Engineering distinguishes between
software features and
software
bugs.
It is usually understood that features are positive, expected characteristics
of a computer program. Features make users happy by allowing them to do
something useful, interesting, or fun. Something
good, anyways. Bugs are
instead undesirable and annoying. You're sitting there at your computer writing
a long email and the software crashes right before your email is sent.
Bad
stuff.
Features are generally implemented by programmers on purpose, whereas bugs are
purely unintentional. They are
mistakes. You don't make a mistake
on
purpose.
We might at this point be inclined to think that: i) what is good for users is
done on purpose by software manufacturers; ii) what is bad for users was not
meant to be. It happened by mistake.
Here is a handy table to visualize this idea:
|
On purpose |
By mistake |
Good |
Feature |
|
Bad |
|
Bug |
It seems to make a lot of sense. But you might have noticed that two cells of
the table are empty. Right!
In a great talk titled
When Free Software isn't better, Benjamin Mako Hill
mentions the concept of antifeatures, and how they relate to
Free
Software.
Antifeatures are features that make the software do something users will hate.
Something they will hate
so much they would pay to have those features
removed, if that's an option. Microsoft Windows 7 is used in the talk to
provide some examples of software antifeatures: the Starter Edition
does not
allow users to change their background image. Also, it limits the amount of
usable memory on the computer to 2GBs, regardless of how much memory the system
actually has. Two antifeatures engineered to afflict users to the point that
they will purchase a more expensive version of the software, if they have the
means to do that.
I have another nice example. The Spotify music streaming service plays
advertisements between songs every now and then. To make sure users are annoyed
as much as possible, Spotify automatically pauses an advertisement if it
detects that the volume is being lowered. A poor Spotify user even
tried to
report the bug on The Spotify Community forum, only to find out that what she
naively considered as a software error was "intentional behavior". A
spectacular antifeature indeed.
Whenever a piece of technology does something you most definitely do not want
it to do, such as
allowing the NSA to take complete control of your Apple
iPhone, including turning on its microphone and camera against your will,
that's an antifeature.
|
On purpose |
By mistake |
Good |
Feature |
|
Bad |
Antifeature |
Bug |
Both bugs and antifeatures are bad for users. The difference between them is
that antifeatures are
engineered. Time and money are spent to make sure the
goal is reached. A testing methodology is followed. "Are we really sure
customers cannot change their wallpaper even if they try very very hard?"
Engineering processes, of course, can fail. If the poor devils at Microsoft who
implemented those harassments would have made a mistake that allows users to
somehow change their wallpaper on Windows Starter... Well, I would call that a
glorious antibug.
|
On purpose |
By mistake |
Good |
Feature |
Antibug |
Bad |
Antifeature |
Bug |
There is no place for antifeatures in Free and Open Source Software. Free
Software gives users
control over what their software does. Imagine Mozilla
adding a feature to Firefox that sets your speakers volume to 11 and starts
playing a random song from the black metal artist
Burzum every time you add
a bookmark, unless you pay for
Mozilla Firefox Premium Edition. The source
code for Firefox is available under a free license. People who are not into
Burzum's music would immediately remove this neat antifeature.
I have spent many years of my life advocating Free and Open Source Software,
perhaps using the wrong arguments.
Mako's talk made me think about all this
(thanks mate!). All these years I've been preaching about the technical
superiority of Free Software, despite evidence of thousands of bugs and
usability issues in the very programs I am using, and contributing to develop.
Free Software is not better than Proprietary Software per se. Sometimes it is,
sometimes it's not. But it gives you control, and freedom. When it annoys you,
when it doesn't do what you expect and want, you can be sure it's not on
purpose. And we can fix it together.
Python is a great language with an impressive number of tools designed to make
developers' life easier. Sometimes, however, the problem is getting to know
that these tools exist in the first place. By contributing to projects like
OpenStack's
Nova client and
Falcon, I have recently come across some useful
tools that can seriously improve the quality of your code.
The first one is called
pyflakes, a
passive checker of Python programs
developed by Phil Frost. What it does is parsing your source files and checking
for possible errors such as undefined names and unused imports. Let's consider
the following example:
<figure class="code">
1
2
3
4
| import urllib
print "pyflakes example"
urlib.urlopen('http://www.linux.it')
|
</figure>The code above contains a typo, we have misspelled
urllib. Here is what
pyflakes thinks about our program:
$ pyflakes example.py
example.py:1: 'urllib' imported but unused
example.py:4: undefined name 'urlib'
On line 4 we try to use
urlib which is not defined. Also, we import
urllib on line 1 and we do nothing with it. Our typo has been spotted!
Notice that, even though our program contains a print statement, 'pyflakes
example' has not been printed. That is because pyflakes
parses the source
files it checks, without
importing them, making it safe to use on modules
with side effects.
pyflakes can be installed with pip or apt-get.
The second tool I want to talk about is Ned Batchelder's
coverage.py.
No doubt you write unit tests for your programs. Right? Good. coverage.py is
out there to help you checking how much of your program is
actually covered.
Let's use as an example
codicefiscale, a Python project of mine.
First we install coverage:
pip install coverage
Then we run our unit tests:
$ coverage run --source=codicefiscale tests.py
.......
----------------------------------------------------------------------
Ran 7 tests in 0.003s
We pass the module we want to test with --source=codicefiscale so that
coverage will only report information about that specific module.
Now that our tests have been performed successfully it is time to check how
much of our code is covered by unit tests:
$ coverage report -m
Name Stmts Miss Cover Missing
---------------------------------------------
codicefiscale 73 4 95% 61, 67, 95, 100
Not bad, 95% of our module is covered! Still, coverage let us know that 4 lines
have not been touched by the unit tests. With this information, we can go write
some meaningful test cases that will also cover the missing lines.
Some months ago I have run into a pretty interesting bug while working on a Ubuntu-based remote desktop system. The whole OS was installed on a server somewhere and users could access their desktop remotely. Some call this stuff
Desktop-as-a-Service.
The operating system we chose was Ubuntu Oneiric (11.10) and the remote access part was implemented with
x2go, which uses nxagent to provide NX transport of X sessions. Users could access their Ubuntu machines remotely, with sound, video, youtube, and all you would expect from a desktop machine. The whole thing was working quite well.
Now, as I said that was in May. Ubuntu 12.04
LTS was available, and the choice of upgrading to it sounded pretty much obvious. So we upgraded our test system to Precise and everything seemed to work smoothly. Till we tried to open a
PDF document, actually. It took evince about 50 seconds to display a relatively small document. Same story with images opened with the default image viewer, eog. The fix delivered to our users was simple: we have set shotwell-viewer as the default image viewer, and epdfview as the default
PDF viewer. Everybody happy.
In the meantime, obviously, I was interested in this issue so I ve tried to run run evince from a terminal, getting the following output:
(evince:15833): GRIP-WARNING **: Failed to initialize gesture manager.
Funny. On another test system running Debian Sid (unstable) everything was working smoothly.
The diff between Ubuntu s version of evince and Debian s is a 6MB monster. Among other changes, I noticed that Ubuntu s version build-depends on libgrip-dev, which depends on libutouch-geis. Multitouch stuff. Why should multitouch support break my remote session? So on May the 10th I
filed a bug on launchpad.
How this issue got handled is in my opinion one of the many fine examples of the inherent superiority of free software, coupled with a we won t hide problems mindset. For an example of how bad is the proprietary approach, just check a random bug in
Adobe s bug tracking system.
But let s go back to the evince bug.
Other users reported that their
VNC sessions were also affected by the same problem. After a few days it was clear that the culprit was utouch-geis, and a patch appeared. Unfortunately
it did not actually address the issue. Somebody else reported that
RDP sessions were broken too. At the beginning of June
Precise was still affected. Finally, on August the 6th
a working patch was submitted by Bradley M. Froehle and included by Chase Douglas (thank you guys). End of August, fixed version of geis
accepted into precise-updates, case closed.
Now for some considerations. The problem was clearly of a certain importance. A Long Term Support, stable version of Ubuntu shipped with broken
PDF and image viewing functionalities. It got fixed properly, even though 3 good months are quite a long time for such a bug to get solved. However, the issue only affected a pretty limited number of users, also certainly not Ubuntu s main target audience.
This bug never affected Debian, simply because
utouch-geis has not made its way into the archive yet. It takes longer to make changes like this in Debian, but for some categories of users stability is more important than new, cool features. Choice is a good thing.
Lately I have been playing a little bit with different Platform as a Service providers. So far, the most developer-friendly one seems to be
Heroku. In particular, Heroku provides an incredibly straightforward way to get started. After creating an account on their website, you just have to install the appropriate gem:
sudo gem install heroku
The command
heroku help
will then show you all the possible invocations of the command line client, just to give you an idea of what you can do with it (everything).
Alright, that s not particularly interesting without an application to play with, is it? So let s create a new
Sinatra application. In order to do that, we need a new directory containing a file like the following one:
# webapp.rb
require 'rubygems'
require 'sinatra'
# Add other gems you might need here
get '/' do
"Your Sinatra app is working!"
end
Also add a Gemfile with all the required dependencies, in our trivial example
sinatra
is actually the only gem you need:
source :rubygems
gem "sinatra"
Lastly, you need a
config.ru
file where you can require our newly created Ruby program and specify that it is a Sinatra application. In this example, the Ruby file is called
webapp.rb
.
require './webapp'
run Sinatra::Application
A very useful program when developing a Sinatra application is
shotgun.
gem install shotgun
and you will be able to run your new shiny app by executing the command
shotgun
. While developing on your machine, shotgun will take care of reloading the application for every request, so that you don t have to restart the server after every modification to the source code.
Run
bundle install
to easily install all the required dependencies in your development environment, and
shotgun
to start the app locally. Pointing your browser to
http://127.0.0.1:9393/ you can check if everything works as expected. If nothing went wrong, you can create a git repository to track changes to the source code:
git init
git add .
git commit -m "Initial commit"
Perfect! Now, how do you create and deploy your application on Heroku?
heroku create
git push heroku master
Heroku will automatically find out that you deployed a Sinatra application, it will install all the required dependencies and start serving requests. A few interesting commands you might want to run to see what s going on are
heroku info
,
heroku ps
and
heroku logs
. The client also provides a command,
heroku open
, that opens your browser at the right production
URL.
So, you now have a working application under revision control. Dependencies are handled by specifying them into the Gemfile, and the required gems can be installed locally using
bundle. Shotgun is our friend when it comes to use the application for development purposes. A
git push heroku master
is all we need to deploy a new version of the code.
Of course there is way much more about Sinatra and Heroku than what I ve covered in this mini-intro. Please refer to the
Sinatra Book and the
Heroku Dev Center for more information!
As somebody who often watches movies and TV shows I have the recurring problem of finding the right English subtitles for the video I am about to enjoy. Now, that is really a tedious thing to do. You go on a website such as
opensubtitles.org and type the title of what you are looking for. Then you usually have to choose from a plethora of results, download the archive, unzip it, fire up
VLC, go to Video Subtitles Track Open File, and finally open your subtitles file. And what you typically find out at that point is that the subtitles are completely out of sync with the audio, but rather than doing the whole process again you prefer to just sync them with a bit of
VLC-fu. Which always takes longer than you thought, but hey.
Now, what if there was an extremely simple way to just choose the subtitles for a given movie directly from
VLC?
The good news is that somebody wrote a beautiful
VLC extension that does exactly that. He also wrote an
introductory article about VLC extensions. Thank you, Jean-Philippe Andr .
Installing the thing is as simple as downloading
the subtitles extension and putting it in the
VLC scripts folder. On Linux machines, the extensions folder can be found at ~/.local/share/vlc/lua/extensions/, on Windows
APPDATA\
VLC\lua\extensions. But seriously, don t use Windows.
Once the extension is installed you can start watching a movie and then click View Subtitles to open the subtitles search interface.
This is how it looks like:
Now that the practical problem is solved, please take some time to look at
the source code.
VLC uses Lua as the language for writing extensions. I certainly cannot say that the web is full of resources about how to extend
VLC using Lua, but the piece of code I wrote about in this post is certainly a good way to start hacking, and maybe scratch yet another itch.
Some years ago I came across a really peculiar newsgroup post. It was not about technicalities of any sort. It was about history. A beautifully written history of computers. From the Turing machine to the Free Software world, the original author managed to capture all the important events of the computer revolution with a great deal of humor. Re-posting it here, it is just brilliant.
The Gospel of Tux (v1.0)
In the beginning Turing created the Machine.
And the Machine was crufty and bodacious, existing in theory only. And von Neumann looked upon the Machine, and saw that it was crufty. He divided the Machine into two Abstractions, the Data and the Code, and yet the two were one Architecture. This is a great Mystery, and the beginning of wisdom.
And von Neumann spoke unto the Architecture, and blessed it, saying, Go forth and replicate, freely exchanging data and code, and bring forth all manner of devices unto the earth. And it was so, and it was cool. The Architecture prospered and was implemented in hardware and software. And it brought forth many Systems unto the earth.
The first Systems were mighty giants; many great works of renown did they accomplish. Among them were Colossus, the codebreaker; ENIAC, the targeter; EDSAC and MULTIVAC and all manner of froody creatures ending in AC, the experimenters; and SAGE, the defender of the sky and father of all networks. These were the mighty giants of old, the first children of Turing, and their works are written in the Books of the Ancients. This was the First Age, the age of Lore.
Now the sons of Marketing looked upon the children of Turing, and saw that they were swift of mind and terse of name and had many great and baleful attributes. And they said unto themselves, Let us go now and make us Corporations, to bind the Systems to our own use that they may bring us great fortune. With sweet words did they lure their customers, and with many chains did they bind the Systems, to fashion them after their own image. And the sons of Marketing fashioned themselves Suits to wear, the better to lure their customers, and wrote grave and perilous Licenses, the better to bind the Systems. And the sons of Marketing thus became known as Suits, despising and being despised by the true Engineers, the children of von Neumann.
And the Systems and their Corporations replicated and grew numerous upon the earth. In those days there were IBM and Digital, Burroughs and Honeywell, Unisys and Rand, and many others. And they each kept to their own System, hardware and software, and did not interchange, for their Licences forbade it. This was the Second Age, the age of Mainframes.
Now it came to pass that the spirits of Turing and von Neumann looked upon the earth and were displeased. The Systems and their Corporations had grown large and bulky, and Suits ruled over true Engineers. And the Customers groaned and cried loudly unto heaven, saying, Oh that there would be created a System mighty in power, yet small in size, able to reach into the very home! And the Engineers groaned and cried likewise, saying, Oh, that a deliverer would arise to grant us freedom from these oppressing Suits and their grave and perilous Licences, and send us a System of our own, that we may hack therein! And the spirits of Turing and von Neumann heard the cries and were moved, and said unto each other, Let us go down and fabricate a Breakthrough, that these cries may be stilled.
And that day the spirits of Turing and von Neumann spake unto Moore of Intel, granting him insight and wisdom to understand the future. And Moore was with chip, and he brought forth the chip and named it 4004. And Moore did bless the Chip, saying, Thou art a Breakthrough; with my own Corporation have I fabricated thee. Thou thou art yet as small as a dust mote, yet shall thou grow and replicate unto the size of a mountain, and conquer all before thee. This blessing I give unto thee: every eighteen months shall thou double in capacity, until the end of the age. This is Moore s Law, which endures unto this day.
And the birth of 4004 was the beginning of the Third Age, the age of Microchips. And as the Mainframes and their Systems and Corporations had flourished, so did the Microchips and their Systems and Corporations. And their lineage was on this wise:
Moore begat Intel. Intel begat Mostech, Zilog and Atari. Mostech begat 6502, and Zilog begat Z80. Intel also begat 8800, who begat Altair; and 8086, mother of all PCs. 6502 begat Commodore, who begat PET and 64; and Apple, who begat 2. (Apple is the great Mystery, the Fruit that was devoured, yet bloomed again.) Atari begat 800 and 1200, masters of the game, who were destroyed by Sega and Nintendo. Xerox begat PARC. Commodore and PARC begat Amiga, creator of fine arts; Apple and PARC begat Lisa, who begat Macintosh, who begat iMac. Atari and PARC begat ST, the music maker, who died and was no more. Z80 begat Sinclair the dwarf, TRS-80 and CP/M, who begat many machines, but soon passed from this world. Altair, Apple and Commodore together begat Microsoft, the Great Darkness which is called Abomination, Destroyer of the Earth, the Gates of Hell.
Now it came to pass in the Age of Microchips that IBM, the greatest of the Mainframe Corporations, looked upon the young Microchip Systems and was greatly vexed. And in their vexation and wrath they smote the earth and created the IBM PC. The PC was without sound and colour, crufty and bodacious in great measure, and its likeness was a tramp, yet the Customers were greatly moved and did purchase the PC in great numbers. And IBM sought about for an Operating System Provider, for in their haste they had not created one, nor had they forged a suitably grave and perilous License, saying, First we will build the market, then we will create a new System, one in our own image, and bound by our Licence. But they reasoned thus out of pride and not wisdom, not forseeing the wrath which was to come.
And IBM came unto Microsoft, who licensed unto them QDOS, the child of CP/M and 8086. (8086 was the daughter of Intel, the child of Moore). And QDOS grew, and was named MS-DOS. And MS-DOS and the PC together waxed mighty, and conquered all markets, replicating and taking possession thereof, in accordance with Moore s Law. And Intel grew terrible and devoured all her children, such that no chip could stand before her. And Microsoft grew proud and devoured IBM, and this was a great marvel in the land. All these things are written in the Books of the Deeds of Microsoft.
In the fullness of time MS-DOS begat Windows. And this is the lineage of Windows: CP/M begat QDOS. QDOS begat DOS 1.0. DOS 1.0 begat DOS 2.0 by way of Unix. DOS 2.0 begat Windows 3.11 by way of PARC and Macintosh. IBM and Microsoft begat OS/2, who begat Windows NT and Warp, the lost OS of lore. Windows 3.11 begat Windows 95 after triumphing over Macintosh in a mighty Battle of Licences. Windows NT begat NT 4.0 by way of Windows 95. NT 4.0 begat NT 5.0, the OS also called Windows 2000, The Millenium Bug, Doomsday, Armageddon, The End Of All Things.
Now it came to pass that Microsoft had waxed great and mighty among the Microchip Corporations; mighter than any of the Mainframe Corporations before it had it waxed. And Gates heart was hardened, and he swore unto his Customers and their Engineers the words of this curse:
Children of von Neumann, hear me. IBM and the Mainframe Corporations bound thy forefathers with grave and perilous Licences, such that ye cried unto the spirits of Turing and von Neumann for deliverance. Now I say unto ye: I am greater than any Corporation before me. Will I loosen your Licences? Nay, I will bind thee with Licences twice as grave and ten times more perilous than my forefathers. I will engrave my Licence on thy heart and write my Serial Number upon thy frontal lobes. I will bind thee to the Windows Platform with cunning artifices and with devious schemes. I will bind thee to the Intel Chipset with crufty code and with gnarly APIs. I will capture and enslave thee as no generation has been enslaved before. And wherefore will ye cry then unto the spirits of Turing, and von Neumann, and Moore? They cannot hear ye. I am become a greater Power than they. Ye shall cry only unto me, and shall live by my mercy and my wrath. I am the Gates of Hell; I hold the portal to MSNBC and the keys to the Blue Screen of Death. Be ye afraid; be ye greatly afraid; serve only me, and live.
And the people were cowed in terror and gave homage to Microsoft, and endured the many grave and perilous trials which the Windows platform and its greatly bodacious Licence forced upon them. And once again did they cry to Turing and von Neumann and Moore for a deliverer, but none was found equal to the task until the birth of Linux.
These are the generations of Linux:
SAGE begat ARPA, which begat TCP/IP, and Aloha, which begat Ethernet. Bell begat Multics, which begat C, which begat Unix. Unix and TCP/IP begat Internet, which begat the World Wide Web. Unix begat RMS, father of the great GNU, which begat the Libraries and Emacs, chief of the Utilities. In the days of the Web, Internet and Ethernet begat the Intranet LAN, which rose to renown among all Corporations and prepared the way for the Penguin.
And Linus and the Web begat the Kernel through Unix. The Kernel, the Libraries and the Utilities together are the Distribution, the one Penguin in many forms, forever and ever praised.
Now in those days there was in the land of Helsinki a young scholar named Linus the Torvald. Linus was a devout man, a disciple of RMS and mighty in the spirit of Turing, von Neumann and Moore. One day as he was meditating on the Architecture, Linus fell into a trance and was granted a vision. And in the vision he saw a great Penguin, serene and well-favoured, sitting upon an ice floe eating fish. And at the sight of the Penguin Linus was deeply afraid, and he cried unto the spirits of Turing, von Neumann and Moore for an interpretation of the dream.
And in the dream the spirits of Turing, von Neumann and Moore answered and spoke unto him, saying, Fear not, Linus, most beloved hacker. You are exceedingly cool and froody. The great Penguin which you see is an Operating System which you shall create and deploy unto the earth. The ice-floe is the earth and all the systems thereof, upon which the Penguin shall rest and rejoice at the completion of its task. And the fish on which the Penguin feeds are the crufty Licensed codebases which swim beneath all the earth s systems. The Penguin shall hunt and devour all that is crufty, gnarly and bodacious; all code which wriggles like spaghetti, or is infested with blighting creatures, or is bound by grave and perilous Licences shall it capture. And in capturing shall it replicate, and in replicating shall it document, and in documentation shall it bring freedom, serenity and most cool froodiness to the earth and all who code therein.
Linus rose from meditation and created a tiny Operating System Kernel as the dream had foreshewn him; in the manner of RMS, he released the Kernel unto the World Wide Web for all to take and behold. And in the fulness of Internet Time the Kernel grew and replicated, becoming most cool and exceedingly froody, until at last it was recognised as indeed a great and mighty Penguin, whose name was Tux. And the followers of Linus took refuge in the Kernel, the Libraries and the Utilities; they installed Distribution after Distribution, and made sacrifice unto the GNU and the Penguin, and gave thanks to the spirits of Turing, von Neumann and Moore, for their deliverance from the hand of Microsoft. And this was the beginning of the Fourth Age, the age of Open Source.
Now there is much more to be said about the exceeding strange and wonderful events of those days; how some Suits of Microsoft plotted war upon the Penguin, but were discovered on a Halloween Eve; how Gates fell among lawyers and was betrayed and crucified by his former friends, the apostles of Media; how the mercenary Knights of the Red Hat brought the gospel of the Penguin into the halls of the Corporations; and even of the dispute between the brethren of Gnome and KDE over a trollish Licence. But all these things are recorded elsewhere, in the Books of the Deeds of the Penguin and the Chronicles of the Fourth Age, and I suppose if they were all narrated they would fill a stack of DVDs as deep and perilous as a Usenet Newsgroup.
Now may you code in the power of the Source; may the Kernel, the Libraries and the Utilities be with you, throughout all Distributions, until the end of the Epoch. Amen.