Ok, I have to admit, I sometimes do game and recently I finished Portal. Quite old (released in 2007), but still lots of fun. I started playing it about one year ago, off and on, until I recently finished the last level. Took me about 1 year of playing to finish the actual playing time of about 10h I guess you can see how much an addict I am 
I have never been a gamer, and I think there are only three set of games I played for extended periods of time:
Besides shooting these portals there are some cubes that one can carry around and use for a variety of purposes, like putting them onto buttons, using them as stairs, protecting yourself from being shot, etc. But that s already all the tools one has. Despite of this, the levels pose increasingly difficult problems, and one is surprised how strange things one can achieve with these limited abilities and no, one cannot buy new power-ups, its not WoW. Logical thinking, tactic, and a certain level of reaction suffices.
While not as philosophical as Myst, it was still a lot of fun. The only thing I am a bit unclear is, where to go from here. There are two possible successors: The logical one would be Portal 2. But I recently found a game that reminded me even more of the Myst series, combined with Portal: The Talos Principle, with stunning graphics:
And filled with riddles again, maybe not as involved as in the Myst series (I don t know by now), but still a bit more challenging than Portal s one:
Difficult decision. If you have any other suggestions, please let me know!
DebConf is not just for Debian Developers, we welcome all members of our
community active in different areas, like translation, documentation, artwork,
testing, specialized derivatives, and many other ways that help make Debian
better.
In fact, we would like to open DebConf to an even broader audience, and we
strongly believe that more diversity at DebConf and in the Debian community will
significantly help us towards our goal of becoming the Universal Operating
System.
The DebConf team is proud to announce that we have started designing a specific
diversity sponsorship programme to attract people to DebConf that would
otherwise not consider attending our conference or not be able to join us.
In order to apply for this special sponsorship, please write an email to outreach@debian.org, before July 6th,
about your interest in Debian and your sponsorship needs (accomodation, travel).
Please include a sentence or two about why you are applying for a Diversity
Sponsorship. You can also nominate people you think should be considered for
this sponsorship programme.
Please feel free to send this announcement on to groups or individuals that
could be interested in this sponsorship programme.
And we re also looking forward to your feedback. We re just getting started and
you can help shape these efforts.
Recently, I've been fighting with the never ending issue of timezones. I never
thought I would have plunged into this rabbit hole, but hacking on OpenStack
and Gnocchi I felt into that trap easily is, thanks to Python.
Why you really, really, should never ever deal with timezones
To get a glimpse of the complexity of timezones, I recommend that you watch
Tom Scott's video on the subject. It's fun and it
summarizes remarkably well the nightmare that timezones are and why you should
stop thinking that you're smart.
datetime.datetime. It can store
date and time precise to the microsecond, and is qualified of timezone "aware"
or "unaware", whether it embeds a timezone information or not.
To build such an object based on the current time, one can use
datetime.datetime.utcnow() to retrieve the date and time for the UTC
timezone, and datetime.datetime.now() to retrieve the date and time for the
current timezone, whatever it is.
>>> import datetime
>>> datetime.datetime.utcnow()
datetime.datetime(2015, 6, 15, 13, 24, 48, 27631)
>>> datetime.datetime.now()
datetime.datetime(2015, 6, 15, 15, 24, 52, 276161)
datetime API always returns unaware datetime objects, which is very
unfortunate. Indeed, as soon as you get one of this object, there is no way to
know what the timezone is, therefore these objects are pretty "useless" on
their own.
Armin Ronacher proposes that an application always consider that the unaware datetime objects from Python are considered as UTC.
As we just saw, that statement cannot be considered true for objects returned
by datetime.datetime.now(), so I would not advise doing so. datetime
objects with no timezone should be considered as a "bug" in the application.
datetime object, i.e. with timezone information. That
makes sure you can compare them directly (aware and unaware datetime
objects are not comparable) and will return them correctly to users.
Leverage pytz to have timezone objects.datetime.datetime.isoformat() to return timestamps as string formatted
using that format, which includes the timezone information.>>> import datetime
>>> import pytz
>>> def utcnow():
return datetime.datetime.now(tz=pytz.utc)
>>> utcnow()
datetime.datetime(2015, 6, 15, 14, 45, 19, 182703, tzinfo=<UTC>)
>>> utcnow().isoformat()
'2015-06-15T14:45:21.982600+00:00'
>>> import iso8601
>>> iso8601.parse_date(utcnow().isoformat())
datetime.datetime(2015, 6, 15, 14, 46, 43, 945813, tzinfo=<FixedOffset '+00:00' datetime.timedelta(0)>)
>>> iso8601.parse_date(utcnow().isoformat()) < utcnow()
True
timestamp with timezone, and which can store the timezone associated,
and do all the computation for you. That's the recommended way to store them
obviously. That does not mean you should not use UTC in most cases; that just
means you are sure that the timestamp are stored in UTC since it's written in
the database, and you check if any other application inserted timestamps with
different timezone.
OpenStack status
As a side note, I've improved OpenStack situation recently by changing the
oslo.utils.timeutils
module to deprecate some useless and dangerous functions. I've also added
support for returning timezone aware objects when using the
oslo_utils.timeutils.utcnow() function. It's not possible to make it a
default unfortunately for backward compatibility reason, but it's there
nevertheless, and it's advised to use it. Thanks to my colleague
Victor for the help!
Have a nice day, whatever your timezone is!
The authenticity of host 'host.example.org (1.2.3.4)' can't be established. RSA key fingerprint is 1b:9b:b8:5e:74:b1:31:19:35:48:48:ba:7d:d0:01:f5. Are you sure you want to continue connecting (yes/no)?If the server is a single-user machine, where you are the only person expected to login on it, answering yes once and then using the ~/.ssh/known_hosts file to record the key fingerprint will (sort-of) work and protect you against future man-in-the-middle attacks. I say sort-of, since if you want to access the server from multiple machines, you will need to sync the known_hosts file somehow. And once your organization grows larger, and you aren t the only person that needs to login, having a policy that everyone just answers yes on first connection on all their machines is bad. The risk that someone is able to successfully MITM attack you grows every time someone types yes to these prompts. Setting up one (or more) SSH Certificate Authority (CA) to create SSH Host Certificates, and have your users trust this CA, will allow you and your users to automatically trust the fingerprint of the host through the indirection of the SSH Host CA. I was surprised (but probably shouldn t have been) to find that deploying this is straightforward. Even setting this up with hardware-backed keys, stored on a YubiKey NEO, is easy. Below I will explain how to set this up for a hypothethical organization where two persons (sysadmins) are responsible for installing and configuring machines. I m going to assume that you already have a couple of hosts up and running and that they run the OpenSSH daemon, so they have a /etc/ssh/ssh_host_rsa_key* public/private keypair, and that you have one YubiKey NEO with the PIV applet and that the NEO is in CCID mode. I don t believe it matters, but I m running a combination of Debian and Ubuntu machines. The Yubico PIV tool is used to configure the YubiKey NEO, and I will be using OpenSC s PKCS#11 library to connect OpenSSH with the YubiKey NEO. Let s install some tools:
apt-get install yubikey-personalization yubico-piv-tool opensc-pkcs11 pcscdEvery person responsible for signing SSH Host Certificates in your organization needs a YubiKey NEO. For my example, there will only be two persons, but the number could be larger. Each one of them will have to go through the following process. The first step is to prepare the NEO. First mode switch it to CCID using some device configuration tool, like yubikey-personalization.
ykpersonalize -m1Then prepare the PIV applet in the YubiKey NEO. This is covered by the YubiKey NEO PIV Introduction but I ll reproduce the commands below. Do this on a disconnected machine, saving all files generated on one or more secure media and store that in a safe.
user=simon key= dd if=/dev/random bs=1 count=24 2>/dev/null hexdump -v -e '/1 "%02X"' echo $key > ssh-$user-key.txt pin= dd if=/dev/random bs=1 count=6 2>/dev/null hexdump -v -e '/1 "%u"' cut -c1-6 echo $pin > ssh-$user-pin.txt puk= dd if=/dev/random bs=1 count=6 2>/dev/null hexdump -v -e '/1 "%u"' cut -c1-8 echo $puk > ssh-$user-puk.txt yubico-piv-tool -a set-mgm-key -n $key yubico-piv-tool -k $key -a change-pin -P 123456 -N $pin yubico-piv-tool -k $key -a change-puk -P 12345678 -N $pukThen generate a RSA private key for the SSH Host CA, and generate a dummy X.509 certificate for that key. The only use for the X.509 certificate is to make PIV/PKCS#11 happy they want to be able to extract the public-key from the smartcard, and do that through the X.509 certificate.
openssl genrsa -out ssh-$user-ca-key.pem 2048 openssl req -new -x509 -batch -key ssh-$user-ca-key.pem -out ssh-$user-ca-crt.pemYou import the key and certificate to the PIV applet as follows:
yubico-piv-tool -k $key -a import-key -s 9c < ssh-$user-ca-key.pem yubico-piv-tool -k $key -a import-certificate -s 9c < ssh-$user-ca-crt.pemYou now have a SSH Host CA ready to go! The first thing you want to do is to extract the public-key for the CA, and you use OpenSSH's ssh-keygen for this, specifying OpenSC's PKCS#11 module.
ssh-keygen -D /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -e > ssh-$user-ca-key.pubIf you happen to use YubiKey NEO with OpenPGP using gpg-agent/scdaemon, you may get the following error message:
no slots cannot read public key from pkcs11The reason is that scdaemon exclusively locks the smartcard, so no other application can access it. You need to kill scdaemon, which can be done as follows:
gpg-connect-agent SCD KILLSCD SCD BYE /byeThe output from ssh-keygen may look like this:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCp+gbwBHova/OnWMj99A6HbeMAGE7eP3S9lKm4/fk86Qd9bzzNNz2TKHM7V1IMEj0GxeiagDC9FMVIcbg5OaSDkuT0wGzLAJWgY2Fn3AksgA6cjA3fYQCKw0Kq4/ySFX+Zb+A8zhJgCkMWT0ZB0ZEWi4zFbG4D/q6IvCAZBtdRKkj8nJtT5l3D3TGPXCWa2A2pptGVDgs+0FYbHX0ynD0KfB4PmtR4fVQyGJjJ0MbF7fXFzQVcWiBtui8WR/Np9tvYLUJHkAXY/FjLOZf9ye0jLgP1yE10+ihe7BCxkM79GU9BsyRgRt3oArawUuU6tLgkaMN8kZPKAdq0wxNauFtHNow all your users in your organization needs to add a line to their ~/.ssh/known_hosts as follows:
@cert-authority *.example.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCp+gbwBHova/OnWMj99A6HbeMAGE7eP3S9lKm4/fk86Qd9bzzNNz2TKHM7V1IMEj0GxeiagDC9FMVIcbg5OaSDkuT0wGzLAJWgY2Fn3AksgA6cjA3fYQCKw0Kq4/ySFX+Zb+A8zhJgCkMWT0ZB0ZEWi4zFbG4D/q6IvCAZBtdRKkj8nJtT5l3D3TGPXCWa2A2pptGVDgs+0FYbHX0ynD0KfB4PmtR4fVQyGJjJ0MbF7fXFzQVcWiBtui8WR/Np9tvYLUJHkAXY/FjLOZf9ye0jLgP1yE10+ihe7BCxkM79GU9BsyRgRt3oArawUuU6tLgkaMN8kZPKAdq0wxNauFtHEach sysadmin needs to go through this process, and each user needs to add one line for each sysadmin. While you could put the same key/certificate on multiple YubiKey NEOs, to allow users to only have to put one line into their file, dealing with revocation becomes a bit more complicated if you do that. If you have multiple CA keys in use at the same time, you can roll over to new CA keys without disturbing production. Users may also have different policies for different machines, so that not all sysadmins have the power to create host keys for all machines in your organization. The CA setup is now complete, however it isn't doing anything on its own. We need to sign some host keys using the CA, and to configure the hosts' sshd to use them. What you could do is something like this, for every host host.example.com that you want to create keys for:
h=host.example.com scp root@$h:/etc/ssh/ssh_host_rsa_key.pub . gpg-connect-agent "SCD KILLSCD" "SCD BYE" /bye ssh-keygen -D /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -s ssh-$user-ca-key.pub -I $h -h -n $h -V +52w ssh_host_rsa_key.pub scp ssh_host_rsa_key-cert.pub root@$h:/etc/ssh/The ssh-keygen command will use OpenSC's PKCS#11 library to talk to the PIV applet on the NEO, and it will prompt you for the PIN. Enter the PIN that you set above. The output of the command would be something like this:
Enter PIN for 'PIV_II (PIV Card Holder pin)': Signed host key ssh_host_rsa_key-cert.pub: id "host.example.com" serial 0 for host.example.com valid from 2015-06-16T13:39:00 to 2016-06-14T13:40:58The host now has a SSH Host Certificate installed. To use it, you must make sure that /etc/ssh/sshd_config has the following line:
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pubYou need to restart sshd to apply the configuration change. If you now try to connect to the host, you will likely still use the known_hosts fingerprint approach. So remove the fingerprint from your machine:
ssh-keygen -R $hNow if you attempt to ssh to the host, and using the -v parameter to ssh, you will see the following:
debug1: Server host key: RSA-CERT 1b:9b:b8:5e:74:b1:31:19:35:48:48:ba:7d:d0:01:f5 debug1: Host 'host.example.com' is known and matches the RSA-CERT host certificate.Success! One aspect that may warrant further discussion is the host keys. Here I only created host certificates for the hosts' RSA key. You could create host certificate for the DSA, ECDSA and Ed25519 keys as well. The reason I did not do that was that in this organization, we all used GnuPG's gpg-agent/scdaemon with YubiKey NEO's OpenPGP Card Applet with RSA keys for user authentication. So only the host RSA key is relevant. Revocation of a YubiKey NEO key is implemented by asking users to drop the corresponding line for one of the sysadmins, and regenerate the host certificate for the hosts that the sysadmin had created host certificates for. This is one reason users should have at least two CAs for your organization that they trust for signing host certificates, so they can migrate away from one of them to the other without interrupting operations.
Almost every new autopkgtest release brings some small improvements, but 3.14 got some reboot related changes worth pointing out.
First of all, I simplified and unified the implementation of rebooting across all runners that support it (ssh, lxc, and qemu). If you use a custom setup script for adt-virt-ssh you might have to update it: Previously, the setup script needed to respond to a reboot function to trigger a reboot, wait for the testbed to go down, and come back up. This got split into issuing the actual reboot system command directly by adt-run itself on the testbed, and the wait for go down and back up part. The latter now has a sensible default implementation: it simply waits for the ssh port to become unavailable, and then waits for ssh to respond again; most testbeds should be fine with that. You only need to provide the new wait-reboot function in your ssh setup script if you need to do anything else (such as re-enabling ssh after reboot). Please consult the manpage and the updated SKELETON for details.
The ssh runner gained a new --reboot option to indicate that the remote testbed can be rebooted. This will automatically declare the reboot testbed capability and thus you can now run rebooting tests without having to use a setup script. This is very useful for running tests on real iron.
Finally, in testbeds which support rebooting your tests will now find a new /tmp/autopkgtest-reboot-prepare command. Like /tmp/autopkgtest-reboot it takes an arbitrary marker , saves the current state, restores it after reboot and re-starts your test with the marker; however, it will not trigger the actual reboot but expects the test to do that. This is useful if you want to test a piece of software which does a reboot as part of its operation, such as a system-image upgrade. Another use case is testing kernel crashes, kexec or another nonstandard way of rebooting the testbed. README.package-tests shows an example how this looks like.
3.14 is now available in Debian unstable and Ubuntu wily. As usual, for older releases you can just grab the deb and install it, it works on all supported Debian and Ubuntu releases.
Enjoy, and let me know if you run into troubles or have questions!
What happened about the reproducible
builds effort for this week:
Presentations
On June 7th, Reiner Herrmann
presented the project at the
Gulaschprogrammiernacht 15 in Karlsruhe, Germany.
Video and audio
recordings
in German are available, and so are the slides in
English.
Toolchain fixes
dh_installtex.SOURCE_DATE_EPOCH with a value suitable for gmtime(3).
debian/rules in packages using it to produce HTML documentation.--mtime when creating source tarball.HTML_TIMESTAMP to NO in Doxygen configuration file.HTML_TIMESTAMP to NO in Doxygen configuration file.Makefile.PL.debbindiff to be better understood. (h01ger)
debbindiff development
Reiner Herrmann added support for decompling Java .class file and .ipk package files (used by OpenWrt). This is now available in version 22 released on 2015-06-14.
Documentation update
Stephen Kitt
documented
the new --insert-timestamp available since
binutils-mingw-w64 version 6.2
available to insert a ready-made date in PE binaries built with
mingw-w64.
Package reviews
195 obsolete
reviews have
been removed, 65 added and 126 updated this week.
New identified issues:
Provides: locales but is actually missing some of the files provided by locales.
Coreboot upstream has been quick to react after the
announcement
of the tests set up the week
before. Patrick Georgi has fixed all issues in a couple of days and all
Coreboot images are now reproducible (without a payload).
SeaBIOS is one of the most frequently used payload
on PC hardware and can now be made
reproducible
too.
Paul Kocialkowski wrote to the mailing
list
asking for help on getting U-Boot tested for
reproducibility.
Lunar had a chat with maintainers of Open Build
Service to better
understand the difference between their system and what we are doing for
Debian.
It s no secret that OpenStack is becoming the de-facto standard for private cloud and a way for telecom operators to differentiate against big names such as Amazon or Google.
Cloud is a huge shift in your organization and will change forever your way of working in the IT projects, improving your IT dramatically and cutting down costs.
In order to get the best of OpenStack, you need to understand deeply how cloud works. Moreover, you need to understand the whole picture beyond the software itself to provide new levels of agility, flexibility, and cost savings in your business.
Giuseppe Paterno , leading European consultant and recently awarded by HP, wrote OpenStack Explained to guide you through the OpenStack technology and reveal his secret ingredient to have a successful project. You can download the ebook for a small donation to provide emergency and reconstruction aid for Nepal. Your donation is certified by ZEWO , the Swiss federal agency that ensures that funds go to a real charity project.
but hurry up, the ebook is in a limited edition and it ends on July 2015.
Donate & Download here: https://life-changer.helvetas.ch/openstack
As with all the previous versions, I have prepared a mega update for the Kobo firmware 3.16.0, including all my favorite patches and features. Please see details for 3.15.0, 3.13.1 and 3.12.1. As before, the following addons are included: Kobo Start Menu, koreader, coolreader, pbchess, ssh access, custom dictionaries, plus as a new treat, some side-loaded fonts. What I dropped this time is most of the kobohack part, as the libraries seem to get outdated. But I included the dropbear ssh server from the kobohack package.
So what are all these items:
gitlab-ci-multi-runner 0.3.3 (dbaf96f) Using Docker executor with image csmall/testdebian ... Pulling docker image csmall/testdebian ... Build failed with Error: image csmall/testdebian: not foundHmm, I know I have that image, it just must be the runner so, let s see what images I have:
$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE csmall/testdebian latest 6979033105a4 5 weeks ago 369.4 MB gcc 5.1 b063030b23b8 5 weeks ago 1.225 GB gcc 5.1.0 b063030b23b8 5 weeks ago 1.225 GB gcc latest b063030b23b8 5 weeks ago 1.225 GB ruby 2.1 236bf35223e7 6 weeks ago 779.8 MB ruby 2.1.6 236bf35223e7 6 weeks ago 779.8 MB debian jessie 41b730702607 6 weeks ago 125.1 MB debian latest 41b730702607 6 weeks ago 125.1 MB debian 8 41b730702607 6 weeks ago 125.1 MB debian 8.0 41b730702607 6 weeks ago 125.1 MB busybox buildroot-2014.02 8c2e06607696 8 weeks ago 2.433 MB busybox latest 8c2e06607696 8 weeks ago 2.433 MB
$ docker save csmall/testdebian > csmall-testdebian.tar.gz
$ docker load csmall-testdebian.tar.gz$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE csmall/testdebian latest 6979033105a4 5 weeks ago 368.3 MB
rescan-scsi-bus tool from the scsitools package. Running (as root, of course)~ rescan-scsi-bus -w -l(sometimes two or three times) is enough to get the device back up, with the
/dev/dvd udev symlink.Hope this'll help !
I know very little about hardware.
I think I have a good understanding on many aspects of what happens inside a computer, but my knowledge is clearly firmer on what happens once an operating system is already running. And even then, my understanding of the lower parts of reality is shaky at most At least according to my self-evaluation, of course, comparing to people I'm honored to call "my peers".
During the last ~18 months, my knowledge of this part of reality, while still far from complete, has increased quite a bit Maybe mostly showing that I'm basically very cheap: As I have come across very cheap (or even free for me!) hardware, I have tried to understand and shape what happens in levels below those where I dwell.
I have been meaning to do a writeup on the MIPS Creator CI20, which was shipped to me for free (thanks++!) by Imagination Technologies; I still want to get more familiar with the board and have better knowledge before reporting on it. Just as a small advance, as this has been keeping me somewhat busy: I got this board after their offer to Debian Developers, and prompted because I'll be teaching some modules on the Embedded Linux diploma course dictated by Facultad de Ingenier a, UNAM Again, I'll blog about that later.
My post today follows Riku's, titled Dystopia of things, where he very clearly finds holes in the Internet of Things offering of one specific product and one specific company, but allows for generalizations on what we will surely see as the model. Riku says:
Today, the GPL sources for hub are available - at least the kernel and a patch for busybox. The proper GPL release is still only through written offer. The sources appeared online April this year while Hub has been sold for two years already. Even if I ordered the GPL CD, it's unlikely I could build a modified system with it - too many proprietary bits. The whole GPL was invented by someone who couldn't make a printer do what he wanted. The dystopian today where I have to rewrite the whole stack running on a Linux-based system if I'm not happy what's running there as provided by OEM.This is not exactly the situation on the boards/products (it's a disservice to call the cute CuBox-i just a board!) I mention I'm using, but it's neither too far. Being used to the easy x86 world, I am used to bitching on specific hardware that does not get promptly recognized by the Linux kernel But even with the extra work UEFI+SecureBoot introduces, getting the kernel to boot is something we just take for granted. In the MIPS and ARM worlds, this is not so much of a given; I'm still treating the whole SPL and DeviceTree world as a black box, but that's where a lot of the work happens. The boards I am working on try to make a point they are Open Hardware. The CI20 is quite impressive in this regard, as not only it has a much more complete set of on-board peripherials than any other, but a wealth of schematics, datasheets and specifications for the different parts of its components. And, of course, the mere availability of the MIPSfpga program to universities worldwide is noteworthy Completely outside of my skillset, but looks most interesting. However... Despite being so much almost-Free-with-a-capital-F, all those boards fail our definitions of freedom in several ways. And yes, they lead us to a situation similar to what Riku describes, to what Stallman feared... To a situation not really better to where we stand on openly closed-source, commodity x86 hardware: Relying on binary blobs and on non-free portions of code to just use our hardware, or at least to use many of the features that would be available to us otherwise. As an example, both the CI20 and the CuBox-i vendors provide system images able to boot what they describe as a Debian 7 system, based on a 3.0 Linux kernel (which Debian never used; IIRC the CuBox-i site said it was derived from a known-good Android kernel)... Only that it's an image resulting of somebody else installing and configuring it. Why should we trust their image to be sane? Yes, the resulting installation is quite impressive (i.e. the CI20's 3D demos are quite impressive for a system that feels otherwise sluggish, and out of my ARM experience, I'd wager it feels sluggish mostly because of a slow SSD)... I have managed to do clean Debian installs on most of my ARM machines (the CuBox-i as described in my previous blog post; this post from Elena of Valhalla'' prompted me into trying the already well documented way of running the official Debian Installer, which worked like a charm and gave me a very nice and responsive Debian 8 install Modulo yes, the Banana's non-free video interface, which AFAICT uses the non-free Mail binary driver... And which I haven't had the time to play with yet. Of course, my CuBox is in a similar situation, where it works like a charm as a personal server, but is completely worthless as a set-top box. So, with those beautiful, small, cheap SoC systems, we are close to where we stood twenty years ago with x86 Linux: Good support for a small set of peripherials, but a far cry from having a functional system with exclusively free software. ,
Despite claims of being open source, this is not open source hardware. If you are thinking of getting this device, you should also try looking into the hardware from our Community instead.Still... Playing with these boards has taught me a lot, and has clearly taught me I'm still standing on the first steps of the n00b level. I have a lot to learn to be able to responsibly teach my part of the diploma course, and I'm very thankful for the differences in hardware (and, of course, for the hardware manufacturers, specially for the MIPS Creator CI20 and the Lemaker Banana Pi for giving me boards to work on!) I shall keep posting on this topic.
The Thing on Internet
The recent AWS introduction of the Elastic File System gives you an automatic grow-and-shrink capability as an NFS mount, an exciting option that takes away the previous overhead in creating shared block file systems for EC2 instances.
However it should be noted that the same auto-management of capacity is not true in the EC2 instance s Elastic Block Store (EBS) block storage disks; sizing (and resizing) is left to the customer. With current 2015 EBS, one cannot simply increase the size of an EBS Volume as the storage becomes full; (as at June 2015) an EBS volume, once created, has fixed size. For many applications, that lack of resize function on its local EBS disks is not a problem; many server instances come into existence for a brief period, process some data and then get Terminated, so long term managment is not needed.
However for a long term data store on an instance (instead of S3, which I would recommend looking closely at from a durability and pricing fit), and where I want to harness the capacity to grow (or shrink) disk for my data, then I will need to leverage some slightly more advanced disk management. And just to make life interesting, I wish to do all this while the data is live and in-use, if possible.
Enter: Logical Volume Management, or LVM. It s been around for a long, long time: LVM 2 made a debut around 2002-2003 (2.00.09 was Mar 2004) and LVM 1 was many years before that so it s pretty mature now. It s a powerful layer that sits between your raw storage block devices (as seen by the operating system), and the partitions and file systems you would normally put on them.
In this post, I ll walk through the process of getting set up with LVM on Debian in the AWS EC2 environment, and how you d do some basic maintenance to add and remove (where possible) storage with minimal interruption.
Getting Started
First a little prep work for a new Debian instance with LVM.
As I d like to give the instance its own ability to manage its storage, I ll want to provision an IAM Role for EC2 Instances for this host. In the AWS console, visit IAM, Roles, and I ll create a new Role I ll name EC2-MyServer (or similar), and at this point I ll skip giving it any actual privileges (later we ll update this). As at this date, we can only associate an instance role/profile at instance launch time.
Now I launch a base image Debian EC2 instance launched with this IAM Role/Profile; the root file system is an EBS Volume. I am going to put data that I ll be managing on a separate disk from the root file system.
First, I need to get the LVM utilities installed. It s a simple package to install: the lvm2 package. From my EC2 instance I need to get root privileges (sudo -i) and run:
After a few moments, the package is installed. I ll choose a location that I want my data to live in, such as /opt/. I want a separate disk for this task for a number of reasons:apt update && apt install lvm2
A (redacted) list of instance from the EC2 console.The returned text is simply: us-east-1a . Time to navigate to Elastic Block Store , choose Volumes and click Create :wget -q -O - http://169.254.169.254/latest/meta-data/placement/availability-zone
Creating a volume in AWS EC2: ensure the AZ is the same as your instance
Attaching a volume to an instance: you ll be prompted for the compatible instances in the same AZ.(Note the time-stamp in square brackets will be different). Previously at this juncture you would format the entire disk with your favourite file system, mount it in the desired location, and be done. But we re adding in LVM here between this raw device, and the filesystem we are yet to make . Marking the block device for LVM Our first operation with LVM is to put a marker on the volume to indicate it s being use for LVM so that when we scan the block device, we know what it s for. It s a really simple command:[1994151.231815] xvdg: unknown partition table
The device name above (/dev/xvdg) should correspond to the one we saw from the dmesg output above. The output of the above is rather straight forward:pvcreate /dev/xvdg
Checking our EBS Volume We can check on the EBS volume which LVM sees as a Physical Volume using the pvs command.Physical volume "/dev/xvdg" successfully created
Here we see the entire disk is currently unused. Creating our First Volume Group Next step, we need to make an initial LVM Volume Group which will use our Physical volume (xvdg). The Volume Group will then contain one (or more) Logical Volumes that we ll format and use. Again, a simple command to create a volume group by giving it its first physical device that it will use:# pvs PV VG Fmt Attr PSize PFree /dev/xvdg lvm2 --- 5.00g 5.00g
And likewise we can check our set of Volume Groups with vgs :# vgcreate OptVG /dev/xvdg Volume group "OptVG" successfully created
The Attribute flags here indicate this is writable, resizable, and allocating extents in normal mode. Lets proceed to make our (first) Logical Volume in this Volume Group:# vgs VG #PV #LV #SN Attr VSize VFree OptVG 1 0 0 wz--n- 5.00g 5.00g
You ll note that I have created our Logical Volume as almost the same size as the entire Volume Group (which is currently one disk) but I left some space unused: the reason for this comes down to keeping some space available for any jobs that LVM may want to use on the disk and this will be used later when we want to move data between raw disk devices. If I wanted to use LVM for Snapshots, then I d want to leave more space free (unallocated) again. We can check on our Logical Volume:# lvcreate -n OptLV -L 4.9G OptVG Rounding up size to full physical extent 4.90 GiB Logical volume "OptLV" created
The attribytes indicating that the Logical Volume is writeable, is allocating its data to the disk in inherit mode (ie, as the Volume Group is doing), and that it is active. At this stage you may also discover we have a device /dev/OptVG/OptLV, and this is what we re going to format and mount. But before we do, we should review what file system we ll use.# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert OptLV OptVG -wi-a----- 4.90g
| Name | Shrink | Grow | Journal | Max File Sz | Max Vol Sz |
|---|---|---|---|---|---|
| btrfs | Y | Y | N | 16 EB | 16 EB |
| ext3 | Y off-line | Y | Y | 2 TB | 32 TB |
| ext4 | Y off-line | Y | Y | 16 TB | 1 EB |
| xfs | N | Y | Y | 8 EB | 8 EB |
| zfs* | N | Y | Y | 16 EB | 256 ZB |
And now mount this volume and check it out:# mkfs.ext4 /dev/OptVG/OptLV mke2fs 1.42.12 (29-Aug-2014) Creating filesystem with 1285120 4k blocks and 321280 inodes Filesystem UUID: 4f831d17-2b80-495f-8113-580bd74389dd Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done
Lastly, we want this to be mounted next time we reboot, so edit /etc/fstab and add the line:# mount /dev/OptVG/OptLV /opt/ # df -HT /opt Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/OptVG-OptLV ext4 5.1G 11M 4.8G 1% /opt
With this in place, we can now start using this disk. I selected here not to update the filesystem every time I access a file or folder updates get logged as normal but access time is just ignored. Time to expand After some time, our 5 GB /opt/ disk is rather full, and we need to make it bigger, but we wish to do so without any downtime. Amazon EBS doesn t support resizing volumes, so our strategy is to add a new larger volume, and remove the older one that no longer suits us; LVM and ext4 s online resize ability will allow us to do this transparently. For this example, we ll decide that we want a 10 GB volume. It can be a different type of EBS volume to our original we re going to online-migrate all our data from one to the other. As when we created the original 5 GB EBS volume above, create a new one in the same AZ and attach it to the host (perhaps a /dev/xvdh this time). We can check the new volume is visible with dmesg again:/dev/OptVG/OptLV /opt ext4 noatime,nodiratime 0 0
And now we initalise this as a Physical volume for LVM:[1999786.341602] xvdh: unknown partition table
And then add this disk to our existing OptVG Volume Group:# pvcreate /dev/xvdh Physical volume "/dev/xvdh" successfully created
We can now review our Volume group with vgs, and see our physical volumes with pvs:# vgextend OptVG /dev/xvdh Volume group "OptVG" successfully extended
There are now 2 Physical Volumes we have a 4.9 GB filesystem taking up space, so 10.09 GB of unallocated space in the VG. Now its time to stop using the /dev/xvgd volume for any new requests:# vgs VG #PV #LV #SN Attr VSize VFree OptVG 2 1 0 wz--n- 14.99g 10.09g # pvs PV VG Fmt Attr PSize PFree /dev/xvdg OptVG lvm2 a-- 5.00g 96.00m /dev/xvdh OptVG lvm2 a-- 10.00g 10.00g
At this time, our existing data is on the old disk, and our new data is on the new one. Its now that I d recommend running GNU screen (or similar) so you can detach from this shell session and reconnect, as the process of migrating the existing data can take some time (hours for large volumes):# pvchange -x n /dev/xvdg Physical volume "/dev/xvdg" changed 1 physical volume changed / 0 physical volumes not changed
During the move, checking the Monitoring tab in the AWS EC2 Console for the two volumes should show one with a large data Read metric, and one with a large data Write metric clearly data should be flowing off the old disk, and on to the new. A note on disk throughput The above move was a pretty small, and empty volume. Larger disks will take longer, naturally, so getting some speed out of the process maybe key. There s a few things we can do to tweak this:# pvmove /dev/sdb1 /dev/sdd1 /dev/xvdg: Moved: 0.1% /dev/xvdg: Moved: 8.6% /dev/xvdg: Moved: 17.1% /dev/xvdg: Moved: 25.7% /dev/xvdg: Moved: 34.2% /dev/xvdg: Moved: 42.5% /dev/xvdg: Moved: 51.2% /dev/xvdg: Moved: 59.7% /dev/xvdg: Moved: 68.0% /dev/xvdg: Moved: 76.4% /dev/xvdg: Moved: 84.7% /dev/xvdg: Moved: 93.3% /dev/xvdg: Moved: 100.0%
So all 5 GB is now unused (compare to above, where only 96 MB was PFree). With that disk not containing data, I can tell LVM to remove the disk from the Volume Group:# pvs PV VG Fmt Attr PSize PFree /dev/xvdg OptVG lvm2 --- 5.00g 5.00g /dev/xvdh OptVG lvm2 a-- 10.00g 5.09g
Then I cleanly wipe the labels from the volume:# vgreduce OptVG /dev/xvdg Removed "/dev/xvdg" from volume group "OptVG"
If I really want to clean the disk, I could choose to use shred(1) on the disk to overwrite with random data. This can take a lng time Now the disk is completely unused and disassociated from the VG, I can return to the AWS EC2 Console, and detach the disk:# pvremove /dev/xvdg Labels on physical volume "/dev/xvdg" successfully wiped
Detach an EBS volume from an EC2 instanceWe now have 0.6GB free space on the physical volume (pvs confirms this). Finally, its time to expand out ext4 file system:# lvresize -L +4.5G /dev/OptVG/OptLV Size of logical volume OptVG/OptLV changed from 4.90 GiB (1255 extents) to 9.40 GiB (2407 extents). Logical volume OptLV successfully resized
And with df we can now see:# resize2fs /dev/OptVG/OptLV resize2fs 1.42.12 (29-Aug-2014) Filesystem at /dev/OptVG/OptLV is mounted on /opt; on-line resizing required old_desc_blocks = 1, new_desc_blocks = 1 The filesystem on /dev/OptVG/OptLV is now 2464768 (4k) blocks long.
Automating this The IAM Role I made at the beginning of this post is now going to be useful. I ll start by adding an IAM Policy to the Role to permit me to List Volumes, Create Volumes, Attach Volumes and Detach Volumes to my instance-id. Lets start with creating a volume, with a policy like this:# df -HT /opt/ Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/OptVG-OptLV ext4 9.9G 12M 9.4G 1% /opt
This policy puts some restrictions on the volumes that this instance can create: only within the given Availability Zone (matching our instance), only GP2 SSD (no PIOPs volumes), and size no more than 250 GB. I ll add another policy to permit this instance role to tag volumes in this AZ that don t yet have a tag called InstanceId:"Version": "2012-10-17", "Statement": [ "Sid": "CreateNewVolumes", "Action": "ec2:CreateVolume", "Effect": "Allow", "Resource": "*", "Condition": "StringEquals": "ec2:AvailabilityZone": "us-east-1a", "ec2:VolumeType": "gp2" , "NumericLessThanEquals": "ec2:VolumeSize": "250" ]
Now that I can create (and then tag) volumes, this becomes a simple procedure as to what else I can do to this volume. Deleting and creating snapshots of this volume are two obvious options, and the corresponding policy:"Version": "2012-10-17", "Statement": [ "Sid": "TagUntaggedVolumeWithInstanceId", "Action": [ "ec2:CreateTags" ], "Effect": "Allow", "Resource": "arn:aws:ec2:us-east-1:1234567890:volume/*", "Condition": "Null": "ec2:ResourceTag/InstanceId": "true" ]
Of course it would be lovely if I could use a variable inside the policy condition instead of the literal string of the instance ID, but that s not currently possible. Clearly some of the more important actions I want to take are to attach and detach a volume to my instance:"Version": "2012-10-17", "Statement": [ "Sid": "CreateDeleteSnapshots-DeleteVolume-DescribeModifyVolume", "Action": [ "ec2:CreateSnapshot", "ec2:DeleteSnapshot", "ec2:DeleteVolume", "ec2:DescribeSnapshotAttribute", "ec2:DescribeVolumeAttribute", "ec2:DescribeVolumeStatus", "ec2:ModifyVolumeAttribute" ], "Effect": "Allow", "Resource": "*", "Condition": "StringEquals": "ec2:ResourceTag/InstanceId": "i-123456" ]
"Version": "2012-10-17",
"Statement": [
"Sid": "Stmt1434114682836",
"Action": [
"ec2:AttachVolume"
],
"Effect": "Allow",
"Resource": "arn:aws:ec2:us-east-1:123456789:volume/*",
"Condition":
"StringEquals":
"ec2:ResourceTag/InstanceID": "i-123456"
,
"Sid": "Stmt1434114745717",
"Action": [
"ec2:AttachVolume"
],
"Effect": "Allow",
"Resource": "arn:aws:ec2:us-east-1:123456789:instance/i-123456"
]
Now with this in place, we can start to fire up the AWS CLI we spoke of. We ll let the CLI inherit its credentials form the IAM Instance Role and the polices we just defined.
and at this stage, the above manipulation of the raw block device with LVM can begin. Likewise you can then use the CLI to detach and destroy any unwanted volumes if you are migrating off old block devices.AZ= wget -q -O - http://169.254.169.254/latest/meta-data/placement/availability-zone Region= wget -q -O - http://169.254.169.254/latest/meta-data/placement/availability-zone rev cut -c 2- rev InstanceId= wget -q -O - http://169.254.169.254/latest/meta-data/instance-id VolumeId= aws ec2 --region $ Region create-volume --availability-zone $ AZ --volume-type gp2 --size 1 --query "VolumeId" --output text aws ec2 --region $ Region create-tags --resource $ VolumeID --tags Key=InstanceId,Value=$ InstanceId aws ec2 --region $ Region attach-volume --volume-id $ VolumeId --instance-id $ InstanceId
After long time of waiting we finally released TeX Live 2015. Get some champagne, and ready to party!
Get it via downloading the net-installer, by obtaining a DVD, or downloading the iso. Besides the huge list up updates to packages and loads of bug fixes to all the programs, here are some notable news:
Long ago, in days of yore, we assumed that any software worth having would be
packaged by the operating system we used. Debian with its enormous pile of
software (over 20,000 sources last time I looked) looked to basically contain
every piece of free software ever. However as more and more people have come
to Linux-based and BSD-based systems, and the proliferation of *NIX-based
systems has become even more diverse, it has become harder and harder to ensure
that everyone has access to all of the software they might choose to use.
Couple that with the rapid development of new projects, who clearly want to get
users involved well before the next release cycle of a Linux-based distribution
such as Debian, and you end up with this recommendation to bypass the operating
system's packaging system and simply curl sudo bash -.
We, the OS-development literati, have come out in droves to say "eww, nasty,
don't do that please" and yet we have brought this upon ourselves. Our
tendency to invent, and reinvent, at the very basic levels of distributions has
resulted in so many operating systems and so many ways to package software (if
not in underlying package format then in policy and process) that third party
application authors simply cannot keep up. Couple that with the desire of the
consumers to not have their chosen platform discounted, and if you provide
Debian packages, you end up needing to provide for Fedora, RHEL, SuSE, SLES,
CentOS, Mint, Gentoo, Arch, etc.etc; let alone supporting all the various BSDs.
This leads to the simple expedience of curl sudo bash -.
Nobody, not even those who are most vehemently against this mechanism of
installing software, can claim that it is not quick, simple for users, easy to
copy/paste out of a web-page, and leaves all the icky complexity of sorting
things out up to a script which the computer can run, rather than the nascent
user of the software in question. As a result, many varieties of software have
ended up using this as a simple installation mechanism, from games to
orchestration frameworks - everyone can acknowledge how easy it is to use.
Now, some providers are wising up a little and ensuring that the url you are
curling is at least an https:// one. Some even omit the sudo from the
copy/paste space and have it in the script, allowing them to display some basic
information and prompting the user that this will occur as root before going
ahead and elevating. All of these myriad little tweaks to the fundamental idea
improve matters but are ultimately just putting lipstick on a fairly sad
looking pig.
So, what can be done? Well we (again the OS-development literati) got
ourselves into this horrendous mess, so it's up to us to get ourselves back
out. We're all too entrenched in our chosen packaging methodologies,
processes, and policies, to back out of those; yet we're clearly not properly
servicing a non-trivial segment of our userbase. We need to do better.
Not everyone who currently honours a curl sudo bash - is capable of
understanding why it's such a bad idea to do so. Some education may reduce
that number but it will never eliminate it.
For a long time I advocated a switch to wget && review && sudo ./script
approach instead, but the above comment, about people who don't understand why
it might be a bad idea, really applies to show how few of those users would
even be capable of starting to review a script they downloaded, let alone able
to usefully judge for themselves if it is really safe to run. Instead we need
something better, something collaborative, something capable of solving the
accessibility issues which led to the curl sudo bash - revolt in the first
place.
curl sudo bash - or
easier. This might mean a particular URI format which can have os-specific
ways to handle standardised inputs, or it might mean a pervasive tool which
does something like that.curl sudo bash - as many platforms
as possible need to get involved. This means not only Debian, Ubuntu,
Fedora and SuSE; but also Arch, FreeBSD, NetBSD, CentOS etc. Maybe even
the OpenSolaris/Illumos people need to get involved.curl sudo bash - :-
just like we can never seem to be rid of that one odd person at the party,
noone knows who invited them, and noone wants to tell them to leave because
they do fill a needed role, but noone really seems to like.
Until then, let's suck it up and while we might not like it, let's just let
people keep on curl sudo bash -ing until someone gets hurt.
curl sudo bash - for the record.
This year, on top of the many excellent contributed talks, BoFs, and other
events always part of DebConf (some of which have already been announced) we
are excited to have confirmed the following keynote speakers.
During the Open Weekend (Saturday, August 15th and Sunday, August 16th), we
will have keynotes delivered by:
Last weekend Hermann Zapf, one of the greatest font designers and typographers of our times, has passed away at the age of 96. Much has been written about his life, his immense sphere of influence, his excellent typefaces, and some of the links to other obituaries can be found here at the end. There is nothing of value I can contribute, but let Hermann Zapf speak for himself with a few images from two books by or dedicated to him.
The books I have taken these photos from are:
In his long life, Hermann Zapf has created a wide range of typefaces, some if which have become the main staple in the printing business, most notably Palatino, Janson, Optima, and Zapf Dingbats. But there are many more:
While probably not as popular anymore as some years (decades?) ago, Palatino still is one of my favorite typefaces. I often wished to have the freedom to choose my own fonts for publications, but normally that doesn t happen. Palatino has found many offspring (or imitations?), and there are excellent renderings for use with (La)TeX, too.
Hermann Zapf also had connections with Don Knuth and the AMS (American Mathematical Society), and he designed for the AMS the Euler typeface, using Knuth s MetaFont program.
Last weekend, when I heard of Hermann Zapf s death, I was at a conference at Tsukuba University. What a coincidence that all the signs in the building (floor numbers, indications, etc) were set 
in another of his great typefaces, the Optima. I had a strange feeling, and reset my presentation in Optima/Euler in honor of Hermann Zapf s great life.
Although I never met Hermann Zapf personally I missed the TeX conferences where he participated it was to a great extent his influence that years ago I got started in typography. Living a life like he did, dedicating himself to the beauty and expressiveness of letters, design, calligraphy, seems like a dream for me. But beauty can be found even in the strangest equation in mathematics, the queerest theorem in logic. While not all of us are as gifted as Hermann Zapf in creating beauty, we should strive to increase our ability to appreciate the beauty.
The beauty of letters is subtle, evasive, and needs training, that is practice, to be appreciated. Hermann Zapf taught us how even self-training can lead to excellence. His letters, which I meet every day, are a permanent reminder to strive for beauty and clarity, and never relent in our exercise.
Thanks Hermann Zapf.
Links to tributes around the world
English
Linotype (Type foundry where Zapf started working) While updating the TeX Live package for Debian for the upcoming 2015 release, one of my aims was to convert as much as possible of the TeX configuration infrastructure to use dpkg triggers. We used the trigger mechanism already for the update of the font map configuration file (calls to updmap-sys), but till now neither for files installed into texmf trees, nor for format generation. In this blog I will describe the new management of all TeX related configuration files, while in the following one I will describe how things have been implemented with dpkg triggers. In due time this information will also be codified in the Debian TeX sub-policy.
Let us start with recalling the different actions that have to be taken when TeX packages are installed, removed, or updated.
Required actions for TeX packages
Filename database
The simplest case are packages that only include files ment to be loaded while running TeX or Metafont or similar programs. In this case the file name database of TeX (and the one of ConTeXt) needs to be updated. Usually this is done by calling mktexlsr on the changed trees, plus, if ConTeXt is installed, running mtxrun --generate.
Font map activation
More complicated are font packages that provide PostScript fonts. These packages need to ship a so called map file which maps TeX font names to PostScript fonts names. Examples are lm.map which defines this mapping for the Latin Modern fonts. All these map files have to be collected in one or more updmap.cfg files, and then a call to updmap(-sys) is necessary to generate configuration files for dvips, pdftex, and some other programs.
Format and hyphenation definitions
Even more complicated is the case of additional hyphenation patterns and additional format definitions. At the moment, hyphenation patterns are only shipped from the TeX Live packages itself, but additional format definitions are already provided by other packages than the TeX Live ones. Again, in both cases all the different format and hyphenation pattern definitions must be combined, into configuration files fmtutil.cnf and several language.* files, respectively. After this some incantation of fmtutil(-sys) is necessary to update the format dumps of TeX.
Status up to Debian/TeX Live 2014
Up to now most of these actions are done in the postinstall script of the providing packages, generated by dh_installtex. The tex-common package shipped besides the program dh_installtex also postinstall and postrm snippets that took care of all the necessary steps, and ensuring that configuration succeeds even if some of the programs were missing.
While this worked quite nicely for many years, I saw several disadvantages:
Map qag.map Map qbk.map Map qcr.map Map qcs.map Map qhv.map Map qpl.map Map qtm.map Map qzc.map |
aleph aleph - *aleph.ini lamed aleph language.dat *lambda.ini |
name=polish lefthyphenmin=2 righthyphenmin=2 file=loadhyph-pl.tex file_patterns=hyph-pl.pat.txt file_exceptions=hyph-pl.hyp.txt |
deb http://people.debian.org/~preining/TeX/ exp/ deb-src http://people.debian.org/~preining/TeX/ exp/ |
Anyway, I hope to hear some feedback and installation reports.
Conclusion
While packaging has become easier for maintainers, and there is less clutter in /etc/texmf/, we now are depending on dpkg to call the tex-common triggers. Thus, if this happens rather often, then especially fmtutil-sys --all will be called several times. This is a slight regression from the previous setup, but paid off (IMNSHO) by the arguments given in the beginning.
That said, there will be of course a certain time of transition until the new setup has proven its merits, and bugs are ironed out. But having bugs that need to be mostly fixed only in one package, namely tex-common, seems like an easier task to me.
Enjoy.
Next.