Search Results: "tar"

29 December 2015

Antoine Beaupr : Bridging Ikiwiki and Twitter with Python and feed2tweet

Typical:

find new interesting software (feed2tweet)
try it out, file two issues:
- no startup file
- fails to install through pip
itch enough that i need to scratch it and file 3 pull requests to ask for forgiveness:

All fairly trivial, but it allowed me to make a simple cronjob to post my Ikiwiki blog posts straight out to my Twitter account. It's basically abusing my RSS feed to bridge with Twitter: the most boring and annoying part is setting up a new app, pasting the credentials in the config file and then running the thing in a cron job. But in the end, I really ended up spending one more hour at a time when I should be really sleeping, scratching an itch that I didn't have before I started working on this thing in the first place. Coincidentally, I requested to be added to Python Planet, looks like a fun place...

Norbert Preining: CafeOBJ 1.5.5 released

Yesterday we have released CafeOBJ 1.5.5 with a long list of changes, and many more internal changes. Documentation pages have been updated with the latest reference manual (PDF, Html) as well as some new docs on CITP (in Japanese for now) and tutorials.

To quote from our README:

CafeOBJ is a new generation algebraic specification and programming language. As a direct successor of OBJ, it inherits all its features (flexible mix-fix syntax, powerful typing system with sub-types, and sophisticated module composition system featuring various kinds of imports, parameterised modules, views for instantiating the parameters, module expressions, etc.) but it also implements new paradigms such as rewriting logic and hidden algebra, as well as their combination.

Changes

Enhancemets of a family of Bool term inspector commands:
- binspect [ in : <Modexpr> ] <bool-term> . converts given <bool-term> into abstracted xor-and normal form
- bshow [ tree grind ] shows abstracted Bool term
- bresolve [ <num1> [ <num2> all ] ] shows assignments which make abstracted Bool term true
- bguess imply and or tries with some heuristics to solve the abstracted Bool term
bug fix in ACZ rewriting
CITP changes
- :ctf [ <term> ] accepts constructors with arguments
- new command :imply to make instantiated lhs of existing equation of the form eq lhs = true a premise of a goal sentence
- new tactic rd- which is similar to rd but cancels the normalization of the goal sentence if the sentence is not satisfied
new switch tree horizontal. if on show term tree displays the term tree structure horizontally (default off).

Availability Binary packages for Windows, MacOS, and Linux have been built, and can be found at the CafeOBJ download page. The source code can also be found on the download page, or directly from here: cafeobj-1.5.5.tar.gz. The CafeOBJ Debian package will be updated soon. Macports file has also been updated, please see the above download/install page for details how to add our sources to your macport. Bug reports If you find a bug, have suggestions, or complains, please open an issue at the issue tracker for CafeOBJ. For other inquiries, please use info@cafeobj.org

28 December 2015

Steinar H. Gunderson: The difference between logs and no logs

An USB3 device of mine stopped working one day. In Windows (its native environment; Linux is not officially supported), there would be a device plugged in sound and then nothing. Nothing in the event log, no indication there was ever a device of any kind in Device Manager. In Linux, after 20 seconds or so, this would come up:

[   71.831659] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?

I bought a new and shorter cable. The card started working again.

Carl Chenet: Parse your rss feed and post to Twitter with feed2tweet

What s more important today for a website than relaying your new blog posts to the social networks, especially Twitter ? Lots of unmaintained or unreliable third-parties web services offer this feature. I tried and was really disappointed using them. Feed2tweet, forked from rss2twitter, now offer a self-hosted, documented, easy-to-read Python application to automatically transfer the new posts in your RSS feed to the social network Twitter. Have a look at the official documentation on Readthedocs in order to install, configure and use Feed2tweet. It is as easy as creating a Twitter app, then writing a feed2tweet.ini file with the following lines:

[twitter]
consumer_key=ml9jaiBnf3pmU9uIrKNIxAr3v
consumer_secret=8Cmljklzerkhfer4hlj3ljl2hfvc123rezrfsdctpokaelzerp
access_token=213416590-jgJnrJG5gz132nzerl5zerwi0ahmnwkfJFN9nr3j
access_token_secret=3janlPMqDKlunJ4Hnr90k2bnfk3jfnwkFjeriFZERj32Z
Une erreur est survenue ; le flux est probablement indisponible. Veuillez r essayer plus tard.
uri: https://www.mycoolwebsite.com/rss
[hashtaglist]
several_words_hashtags_list: severalwordshashtaglist.txt

Then just launch feed2tweet:

$ feed2tweet -c /path/to/feed2tweet.ini

The first version 0.1 is now available from Github. If it helps you, Github stars are appreciated :)

Aigars Mahinovs: Moving to Germany

After a long contemplation about what I want to do with my life and just as long a negotiation, I have accepted a job offer at BMW and will be moving from Latvia to Ulm, Germany in January 2016. It is going to be a big change and a big shake-up in my life, but the change should be for the better, eventually. I am going to move there using my car, loading a single car load of essential stuff and leaving the rest behind. My cat will be amond the essentials, so I have already started training him for car rides. The cat is not thrilled about it. I'll be renting out my current apartment - I wish to find someone that I know to rent to, but if noone comes up, then I'll have to give it to a renting company to rent out for me. There should be little change for my online activities, but this means that I will be less available for parties in Riga for the forseable future. Wish me luck!

Hideki Yamane: How about "grooming" outdated packages?

Lucas's good article: DEBIAN PACKAGES WITH /OUTDATED/ PACKAGING STYLE

Okay, then what's next? Action.

How about do "grooming" for those packages with contributors? Well experienced DDs become mentors and help to make those packages modern and put it to repo.

Packages become simple and modern, contributors can get packaging experience.
Isn't it a good idea? (Meow!)

27 December 2015

Vincent Sanders: The only pleasure I get from moving house is stumbling across books I had forgotton I owned

I have to agree with John Burnside on that statement, after having recently moved house again rediscovering our book collection has been a salve for an otherwise exhausting undertaking. I returned to Cambridge four years ago, initially on my own and then subsequently the family moved down to be with me.

We rented a house but, with two growing teenagers, the accommodation was becoming a little crowded. Melodie and I decided the relocation was permanent and started looking for our own property, eventually finding something to our liking in Cottenham village.

Melodie took the opportunity to have the house cleaned and decorated while empty because of overlapping time with our rental property. This meant we had to be a little careful while moving in as there was still wet paint in places.

Moving weekend was made bearable by Steve, Jonathan and Jo lending a hand especially on the trips to Yorkshire to retrieve, amongst other things, the aforementioned book collection. We were also fortunate to have Andy and Jane doing many other important jobs around the place while the rest of us were messing about in vans.

The seemingly obligatory trip to IKEA to acquire furniture was made much more fun by trying to park a luton van which was only possible because Steve and Jonathan helped me. Though it turns out IKEA ship mattresses rolled up so tight they can be moved in an estate car so taking the van was unnecessary.

Having moved in it seems like every weekend is filled with a never ending "todo" list of jobs. From clearing gutters to building a desk in the study. Eight weeks on and the list seems to be slowly shrinking meaning I can even do some lower priority things like the server rack which was actually a fun project.

The holidays this year afforded me some time to finish the boys bedrooms. They both got loft beds with a substantial area underneath. This allows them both to have double beds along with a desk and plenty of storage. Completing the rooms required the construction of some flat pack furniture which rather than simply do myself I supervised the boys doing it themselves.

Teaching them by letting them get on with it was a surprisingly effective and both of them got the hang of the construction method pretty quickly. There was only a couple of errors from which they learned immediately and did not repeat (draw bottoms having a finished side and front becomes back when you are constructing upside down)

The house is starting to feel like home and soon all the problems will fade from memory while the good will remain. Certainly our first holiday season has been comfortable here and I look forward to many more re-reading our books.

Lars Wirzenius: Coming back to Finland after five years

It was a shock to be back in Finland after five years. The country had gone bad. I was born in Finland, and lived here all my life, until January, 2010. I then moved abroad for five years, with my then-girlfriend, now wife. We lived in New Zealand, Scotland, and England. In October 2014 we came back, because my next job happened to be in Finland. When we came back, I found that the country had changed, or I had lost my rose-coloured glasses. The Finland I left was a place where equality and solidarity were assumed, though not universal. The country I came back to turned out to be racist, scared, and quickly abandoning everything that used to be examples of why the country is a good place to live. Some of the change has, possibly, only happened during 2015 and the refugee crisis in Europe, but the change started earlier. Examples:

Racist behaviour and violence is commonplace. Those who do not look like the natives are insulted, threatened with physical violence, and sometimes experience it.
Political violence is becoming commonplace. Refugee centers are burnt down, sometimes with people inside. Government ministers are attacked, albeit by throwing a drink on them.
Politicians get away with saying and doing clearly racist things, up to and including celebrating neo-nazism.
The government spends a lot of effort trying to break any power the labour unions have, so that the employers get to do what they want. The whole stabilising system of negotating between government, employer, and labour organisations is being torn down, and labour is being stripped of any power.
Prominent cabinet ministers talk condescendingly about science, universities, and cut down educational funding. Universities should concentrate on being R&D factories for business, forget about advancing civilisation, or the humanities, or the arts. Unless it's the type of arts that becomes best-selling games.

Now, it is clear that Finland was never the kind of idyllic utopia that I thought it was, back when I was young. The older I get, the clearer it is that I was naive. The change over five years is still quite remarkable. I can't ever go back home. Now, I am sometimes ashamed of being Finnish.

Russ Allbery: Review: The Outskirter's Secret

Review: The Outskirter's Secret, by Rosemary Kirstein

Series:	Steerswomen #2
Publisher:	Del Rey
Copyright:	1992, 2001
Printing:	2001
ISBN:	0-345-46105-3
Format:	Trade paperback
Pages:	389

This is a direct sequel to The Steerswoman, and I don't recommend starting here. I read this novel as part of The Steerswoman's Road omnibus, which is the edition reflected in the metadata (except for the page count, which is just this novel). There are a few plot constructs that occur depressingly often in SFF and almost always annoy me. One is books where nearly all of the book is just a journey from one place to another, without much interesting landscape along the way. Subtract additional points if the land is hostile and there are constant worries about food. Another is illness, or being unable to trust one's senses, or, worse, both. Somehow, this book manages to do both of those things and was still thoroughly enjoyable. This is exceptionally rare. After the conclusions reached in the previous book, Rowan and Bel are even more determined to reach the origin of the jewels on Bel's belt and gather more information. This means travel into the Outskirts, where Rowan has never been (and where people from the Inner Land almost never go). It also means close contact with Bel's people, who have an entirely different set of social rituals and who are notoriously prickly. Nearly all of The Outskirter's Secret is travel through the Outskirts while navigating Outskirter society, politics, and customs. It's anthropological fiction of a sort, particularly given the inquisitive and analytical mindset that Rowan brings to anything new. Anthropological SFF is another one of those things that normally leaves me cold, but which totally worked for me in this book. I rarely remember being this engaged in, or caring this much about, the customs and traditions of an invented and relatively primitive culture. Partly this is because Rowan's mindset continues to be fascinating and contagious. I kind of want to compare this series to Neal Stephenson's Anathem, but where Stephenson's novel was about the practice of organized science, the Steerswomen series is about the individual scholar and scientist. This could go badly in the hands of a lesser author, but Kirstein brilliantly balances analysis and theories with Rowan's child-like wonder in learning, peculiar and determined ethics, and emotional growth. She isn't a disinterested observer or an engineer with a wrench. She lets the world change her, works hard at finding her place and her own strength, and engages with it emotionally as much as she does intellectually. She's a wonderfully compelling character. Another significant factor to this book's success is the slow and deliberate way Kirstein mixes world-building revelations with the day-to-day struggles of the characters. There's much more to that than just the target of Rowan's investigations, and far more complexity and significance to the Outskirters than showed in first appearances. The world is also set up in a way that lets the readers guess various details before the characters, with their more limited knowledge, can work out: another tricky technique that Kirstein gets almost perfect. I never wanted to shake the characters for being dumb; usually, both Rowan and Bel leap to correct conclusions just shortly after the reader, and long before the reader is tired of following their thought processes. There was only one bit of world-building that took Rowan most of the book to figure out despite being obvious to me early in this novel, and I think that one was fair. She didn't have anywhere near enough information to work it out. Bel continues to be wonderful in numerous ways. But even better, Kirstein introduces us to Bel's people and highlights both similarities and differences. It's an excellent job of building a complex culture that feels human: varied, tradition-laden, practical, but also well-adapted to their environment and situation, and far more sophisticated than they might appear. I'm normally a hard sell for this sort of thing, but Kirstein got me thoroughly invested. One scene late in this novel where a tribe welcomes a newcomer is masterful and probably my favorite single moment of the series so far. Once again, unfortunately, the conclusion is a bit less than satisfying. The characters come away knowing more, planning more, but still with numerous unanswered questions. This is not a series that's in a hurry to get to major plot payoffs. Instead, it's a series that takes a close look at the cultures, relationships, emotions, and lives that it touches, and describes them in deft and engaging ways. The Outskirter's Secret is even better than The Steerswoman. Recommended. I bought the third and fourth books while only a little ways into this one and am eager to read them. Followed by The Lost Steersman. Rating: 9 out of 10

26 December 2015

Julian Andres Klode: Much faster incremental apt updates

APT s performance in applying the Pdiffs files, which are the diff format used for Packages, Sources, and other files in the archive has been slow. Improving performance for uncompressed files The reason for this is that our I/O is unbuffered, and we were reading one byte at a time in order to read lines. This changed on December 24, by adding read buffering for reading lines, vastly improving the performance of rred. But it was still slow, so today I profiled using gperftools the rred method running on a 430MB uncompressed Contents file with a 75 KB large patch. I noticed that our ReadLine() method was calling some method which took a long time (google-pprof told me it was some _nss method, but that was wrong [thank you, addr2line]). After some further look into the code, I noticed that we set the length of the buffer using the length of the line. And whenever we moved some data out of the buffer, we called memmove() to move the remaining data to the front of the buffer. So, I tried to use a fixed buffer size of 4096 (commit). Now memmove() would spend less time moving memory around inside the buffer. This helped a lot, bringing the run time on my example file down from 46 seconds to about 2 seconds. Later on, I rewrote the code to not use memmove() at all opting for start and end variables instead; and increasing the start variable when reading from the buffer (commit). This in turn further improved things, bringing it down to about 1.6 seconds. We could now increase the buffer size again, without any negative effect. Effects on apt-get update I measured the run-time of apt-get update, excluding appstream and apt-file files, for the update from todays 07:52 to the 13:52 dinstall run. Configured sources are unstable and experimental with amd64 and i386 architectures. appstream and apt-file indexes are disabled for testing, so only Packages and Sources indexes are fetched. The results are impressive:

For APT 1.1.6, updating with PDiffs enabled took 41 seconds.
For APT 1.1.7, updating with PDiffs enabled took 4 seconds.

That s a tenfold increase in performance. By the way, running without PDiffs took 20 seconds, so there s no reason not to use them. Future work Writes are still unbuffered, and account for about 75% to 80% of our runtime. That s an obvious area for improvements.

Performance for patching compressed files Contents files are usually compressed with gzip, and kept compressed locally because they are about 500 MB uncompressed and only 30MB compressed. I profiled this, and it turns out there is not much we can do about it: The majority of the time is spent inside zlib, mostly combining CRC checksums:

Going forward, I think a reasonable option might be to recompress Contents files using lzo they will be a bit bigger (50MB instead of 30MB), but lzo is about 6 times as fast (compressing a 430MB Contents file took 1 second instead of 6).
Filed under: Debian, Uncategorized

Russ Allbery: Review: The Steerswoman

Review: The Steerswoman, by Rosemary Kirstein

Series:	Steerswomen #1
Publisher:	Del Rey
Copyright:	1989, 2003
Printing:	2003
ISBN:	0-345-46105-3
Format:	Trade paperback
Pages:	264

I read this novel as part of The Steerswoman's Road omnibus, which is the edition reflected in the metadata (except for the page count, which is just this novel). Rowan is a steerswoman, a member of an order of traveling scholars in a world of generally medieval customs. This is a world of small villages and town blacksmiths, of swords, sailing ships, and lots of walking. Steerswomen (the order does include some men, but only rarely; the gender balance is left delightfully unexplained and unremarked) mostly travel routes through the Inner Lands until they decide to retire and do research and study. Sometimes, though, they get on the trail of something unexplained, and are driven to explain it. That's the case for Rowan at the start of this book: she's run across several peculiar flat jewels. No one seems to know what they are, their pattern of distribution is odd, and they're discovered in some decidedly strange places, such as embedded deep in a tree. It's a puzzle, and Rowan is determined to understand it. The Steerswoman opens with Rowan investigating a jewel owned by an innkeeper, and a chance meeting in his inn with an Outskirter who wears a whole belt of the same jewels. The Outskirters are nomadic tribes from the far west known for their prickly attitude, their ability in warfare, and their occasional raids. Those who live in the Inner Lands consider them barbarians. But Bel is curious, thoughtful, and happy to tell Rowan of the origin of her belt: her father found a huge number of the jewels embedded in the side of a cliff, far into the Outskirts. Rowan is immediately tempted to go there, but she has to report to the Archives first. Much to her surprise, Bel asks to accompany her. And then wizards start trying to kill them. The steerswomen are Kirstein's best bit of world-building and the anchor around which the rest of the story is told. Steerswomen have to answer all questions they're asked, truthfully. In return, they can ask a question of anyone and have to be answered truthfully. If anyone refuses to answer, or tells a deliberate lie, they're placed under the steerswomen's ban, and no steerswoman will answer their questions. This is woven deep into the culture of the Inner Lands... except for the wizards, who refuse to explain what they're doing or how to anyone, are under the ban, and don't care. They protect cities from dragons, build magical lighting, and occasionally start wars with other wizards in which numerous people die. They're effectively the rulers of this world insofar as it has rulers, and they're apparently quite interested in Rowan's search for these unexplained jewels. Rowan is a wonderful character. At the start of the book, she's the only steerswoman that the reader knows, so much of her early behavior is also world-building, showing us steerswomen. But as the story develops, we meet other steerswomen and Rowan develops into her own character: innocent in a way, but determined, ethical, and with a very deep store of self-knowledge and inner contemplation. I found the middle part of the book somewhat frustrating for the same reasons Rowan did. The ending lets her shine, in her own way, and it's deeply satisfying. The Steerswoman does pick up a third character along the way, a young boy named Will who hopes to become a wizard's apprentice, and to whom I never warmed. Part of the problem was structural: he's very suspicious of characters that the reader knows are entirely honorable. This is a difficult tone to pull off without irritating me, and Kirstein doesn't quite manage. She also shifts the story to the Will's perspective from time to time, and I never found his viewpoint very interesting. I warmed to him a bit by the climax, but I still don't think he added much to the story. Bel, though... Bel competes with Rowan as best character of this book. The other characters all have roles to play: Rowan is the steerswoman, integral to the world-building, and Will exemplifies another corner of the analytical process so central to this book and represents the appeal of the wizards. But Bel is just herself: irreverent, curious, loyal, dangerous, decisive, and adventurous. Rowan is the heart of the book and the primary protagonist, but I think Bel is its soul. I haven't said too much about the themes of this book, intentionally. Kirstein is doing something very interesting and subtle with the reader's expectations of a medieval world with wizards, and I think it's better experienced without spoilers. But it's also a character-driven story, a story of unlikely friendship between two very different women, and a story about the value of being true to yourself even when it's exceptionally dangerous. Rowan delighted me immediately, Bel grew on me over the course of the story, and the rough patches in the middle of the book faded in comparison to an excellent ending. Unfortunately, this is only part of a story. There is a climax, and we do know what the jewels are by the end of the book. But the journey has raised many more questions, most of which are not solved. The world has become far more complicated. Rowan has a lot left to learn and investigate. I recommend the omnibus, or getting the ebooks from Amazon where you can quickly get more, because you'll probably want to immediately start the second book. Followed by The Outskirter's Secret. Rating: 8 out of 10

Russ Allbery: kstart 4.2

kstart is a collection of two utilities for managing Kerberos tickets, particularly focused on the needs of long-running commands or daemons. This release has taken embarassingly long to release, given that the last release was in January of 2012. Sorry about that, particularly those who were stuck on various bugs! This code base, particularly the test suite, is in dire need of significant refactoring, which reduces my desire to work on it, and then I ran very, very short on free software time. This release is important because I think I've finally worked through all the various tricky complexity of startup when running a command under k5start. The core problem: the original k5start exited when authentication failed. However, it's very nice to start daemons that need a Kerberos ticket by running them via k5start. But if this is done during system startup, the network may not be up yet, and one may not be able to contact the KDC. It's very difficult to get the timing right just by adjusting init scripts. The core fix in this release is that k5start run as a daemon or with a command, and krenew run with the -i option, now keeps retrying the authentication or renewal until it succeeds. It's retried immediately and then with exponential backoff, starting with a delay of a second, until intervals of one minute, and then retried at a delay of one minute until it succeeds. Any command isn't run until the authentication works, so (unlike previous versions) k5start won't start the command without a Kerberos ticket. That should fix various problems people are having. There are also some new options to tweak the renewal behavior. -H can now be used in combination with -K to specify how long of a minimum ticket lifetime k5start or krenew should try to maintain. Alternately, there's now a -a flag that tells both to attempt an authentication or renewal every time they wake up (at the interval specified with -K). This probably should have been the default for -K originally, but I didn't change the original behavior for backwards compatibility. There are also a couple of bug fixes: the temporary ticket cache created by k5start is cleaned up properly, and the programs no longer incorrectly reject the combination of -b and -K or a command. You can get the latest version from the kstart distribution page.

24 December 2015

Clint Adams: Before the Tet Offensive

Kurt has trouble keeping his mouth shut. This became widely apparent when he was expelled from Catholic school for telling a visiting dignitary to go fuck himself. WB used to preach angrily against casual sex. Kurt enlisted in the Army, and due to his high IQ he ended up in the Army Security Agency. After training, he was stationed in Germany where he performed signals intelligence functions like direction finding and passing on information to the CIA and NSA. One day WB shagged a boy with a leather hat. Contrary to what one might assume, Kurt had awareness of consequences. He refused officer training because that would have extended his commitment from 4 to 6 years. He purposefully flunked his French language proficiency exam. He wanted there to be no chance of him getting transferred to Vietnam. The following day, WB found herself confronted and asked to explain why she had had a change of heart about casual sex. Oh, it wasn't casual sex, she explained. He came over and said, I've been dying to make love with you for the past three hours, so it was special, she clarified. Thus the nature of the past communication failure became clear to everyone but WB. Despite all his efforts to avoid Vietnam, Kurt, of course, had trouble keeping his mouth shut. So in 1967 he pissed off the wrong person and found himself having to choose between Leavenworth and Saigon. He chose life of alcoholism over prison.

Zlatan Todori : Deepin switches to Debian Sid

Distribution that I feel fond of their work on UX for Linux ecosystem has switched from Ubuntu to Debian. So now pabs will probably invite them to our Derivate Census and they start packaging their stack into Debian directly. More about their latest RC release and the mention of switch is on their blog

23 December 2015

David Bremner: Offline key signing with caff

After a mildly ridiculous amount of effort I made a bootable-usb key. I then layered a bash script on top of a perl script on top of gpg. What could possibly go wrong?

 #!/bin/bash
 infile=$1
 keys=$(gpg --with-colons  $infile   sed -n 's/^pub//p'   cut -f5 -d: )
 gpg --homedir $HOME/.caff/gnupghome --import $infile
 caff -R -m no "$ keys[*] "
 today=$(date +"%Y-%m-%d")
 output="$(pwd)/keys-$today.tar"
 for key in $ keys[*] ; do
     (cd $HOME/.caff/keys/;   tar rvf "$output" $today/$key.mail*)
 done

The idea is that keys are exported to files on a networked host, the files are processed on an offline host, and the resulting tarball of mail messages sneakernetted back to the connected host.

22 December 2015

Ian Wienand: Platform bootstrap for OpenStack Infrastructure

This is a terse guide on bootstrapping virtual-machine images for OpenStack infrastructure, with the goal of adding continuous-integration support for new platforms. It might also be handy if you are trying to replicate the upstream CI environment. It covers deployment to Rackspace for testing; Rackspace is one of the major providers of capacity for the OpenStack Infrastructure project, so it makes a good place to start when building up your support story. Firstly, get an Ubuntu Trusty environment to build the image in (other environments, like CentOS or Fedora, probably work -- but take this step to minimise differences to what the automated machinery of what upstream does). You want a fair bit of memory, and plenty of disk-space. The tool used for building virtual-machine images is diskimage-builder. In short, it takes a series of elements, which are really just scripts run in different phases of the build process. I'll describe building a Fedora image, since that's been my focus lately. We will use a fedora-minimal element -- this means the system is bootstrapped from nothing inside a chroot environment, before eventually being turned into a virtual-machine image (contrast this to the fedora element, which bases itself off the qcow2 images provided by the official Fedora cloud project). Thus you'll need a few things installed on the Ubuntu host to deal with bootstrapping a Fedora chroot

apt-get install yum yum-utils python-lzma

You will hit stuff like that python-lzma dependency on this road-less-travelled -- technically it is a bug that yum packages on Ubuntu don't depend on it; without it you will get strange yum errors about unsupported compression. At this point, you can bootstrap your diskimage-builder environment. You probably want diskimage-builder from git, and then build up a virtualenv for your support bits and pieces.

git clone git://git.openstack.org/openstack/diskimage-builder
virtualenv dib_env
. dib_env/bin/activate
pip install dib-utils

dib-utils is a small part of diskimage-builder that is split-out; don't think too much about it. While diskimage-builder is responsible for the creation of the basic image, there are a number of elements provided by the OpenStack project-config repository that bootstrap the OpenStack environment. This does a lot of stuff, including caching all git trees (so CI jobs aren't cloning everything constantly) and running puppet setup.

git clone git://git.openstack.org/openstack-infra/project-config

There's one more trick required for building the VHD images that Rackspace requires; make sure you install the patched vhd-util as described in the script help. At this point, you can probably build an image. Here's something approximating what you will want to do

break=after-error \
TMP_DIR=~/tmp \
ELEMENTS_PATH=~/project-config/nodepool/elements \
DIB_DEV_USER_PASSWORD="password"  DIB_DEV_USER_PWDLESS_SUDO=1 \
DISTRO=23 \
./bin/disk-image-create -x --no-tmpfs -t vhd \
   fedora-minimal vm devuser simple-init \
   openstack-repos puppet nodepool-base node-devstack

To break some of this down

break= will help you debug failing builds by dropping you into a shell when one of the element parts fail.
TMP_DIR should be set to somewhere with plenty of space; a default /tmp with tmpfs is probably restricted to a couple of gigabytes; not enough for a build.
ELEMENTS_PATH will add the OpenStack specific elements
DIB_DEV_USER_* flags will create a devuser login with a password and sudo access. This is really important as it is most likely you'll boot up fairly broken the first time, and you need a way to log-in (this is not used in "production").
DISTRO in this case says to build a Fedora 23 based-image, but will vary depending on what you are trying to do.
disk-image-create finally creates the image. We are telling it to create a vhd based image, using fedora-minimal in this case. For configuration we are using simple-init, which uses the glean project to configure networking from a configuration-drive.
- By default, this will install glean from pypi. However, adding DIB_INSTALL_TYPE_simple_init=repo would modify the install to use the git source. This is handy if you have changes in there that are not released to pypi yet.

This goes and does its thing; it will take about 20 minutes. Depending on how far your platform diverges from the existing support, it will require a lot of work to get everything working so you can get an image out the other side. To see a rough example of what should be happening, see the logs of the official image builds that happen for a variety of platforms. At some point, you should get a file image.vhd which is now ready to be deployed. The only reasonable way to do this is with shade. You can quickly grab this into the virtualenv we created before

pip install shade

Now you'll need to setup a clouds.yaml file to give yourself the permissions to upload the image. It should look something like

clouds:
  rax:
    profile: rackspace
    auth:
       username: your_rax_username
       password: your_rax_password
       project_id: your_rax_accountnum
    regions:
    - IAD

You should know your user-name and password (whatever you log into the website with), and when you login to Rackspace your project_id value is listed in the drop-down box labeled with your username as Account #. shade has no UI as such, so a simple script will do the upload.

import shade
shade.simple_logging(debug=True)
cloud = shade.openstack_cloud(cloud='rax')
image = cloud.create_image('image-name', filename='image.vhd', wait=True)

Now wait -- this will also take a while. Even after upload it takes a fair while to process (you will see the shade debug output looping around some glance calls seeing if it is ready). If everything works, the script should return and you should be able to see the new image in the output of nova list-images. At this point, you're ready to try booting! One caveat is that the Rackspace web interface does not seem to give you the option to boot with a configuration drive available to the host, essential for simple-init to bring up the network. So boot via the API with something like

nova boot --flavor=2 --image=image-uuid --config-drive 1 test-image

This should build and boot your image! This will allow you to open a whole new door of debugging to get your image booting correctly. You can now iterate by rebuilding/uploading/booting as required. Note these are pretty big images, and uploaded in broken-up swift files, so I find swift delete --all helpful to reset between builds (obviously, I have nothing else in swift that I want to keep). The Rackspace java-based console UI is rather annoying; it cuts itself off every time the host reboots. This makes it quite difficult to catch the early bootup, or modify grub options via the boot-loader, etc. You might need to fiddle timeout options, etc in the image build. If you've managed to get your image booting and listening on the network, you're a good-deal of the way towards having your platform supported in upstream OpenStack CI. At this point, you likely want to peruse the nodepool configuration and get an official build happening here. Once that is up, you can start the process of adding jobs that use your platform to test! Don't worry, there's plenty more that can go wrong from here -- but you're on the way! OpenStack infra is a very dynamic environment, which many changes in progress; so in general, #openstack-infra on freenode is going to be a great place to start looking for help.

Steve Kemp: Finding and reporting trivial security issues

This week I'll be mostly doing drive-by bug-reporting. As with last year we start by using the Debian Code Search, to look for obviously broken patterns such as "system.>./tmp/.*" Once we find a fun match we examine the code and then report the bugs we find. Today that was stalin which runs some fantastic things on startup:

(system "uname -m >/tmp/QobiScheme.tmp")
(system "rm -f /tmp/QobiScheme.tmp"))

We can exploit this like so:

$ ln -s /home/steve/HACK /tmp/QobiScheme.tmp
$ ls -l /home/steve/HACK
ls: cannot access /home/steve/HACK: No such file or directory

Now we run the script:

$ cd /tmp/stalin-0.11/benchmarks
$ ./make-hello

And we see this:

$ ls -l /home/steve/HACK
-rw-r--r-- 1 steve steve 6 Dec 22 08:30 /home/steve/HACK

For future reference the lsat looks horrifically bad - it writes multiple times to /tmp/lsat1.lsat and although it tries to detect races I'm not convinced. Something to look at in the future.

21 December 2015

Matthew Garrett: GPL enforcement is a social good

The Software Freedom Conservancy is currently running a fundraising program in an attempt to raise enough money to continue funding GPL compliance work. If they don't gain enough supporters, the majority of their compliance work will cease. And, since SFC are one of the only groups currently actively involved in performing GPL compliance work, that basically means that there will be nobody working to ensure that users have the rights that copyright holders chose to give them.

Why does this matter? More people are using GPLed software than at any point in history. Hundreds of millions of Android devices were sold this year, all including GPLed code. An unknowably vast number of IoT devices run Linux. Cameras, Blu Ray players, TVs, light switches, coffee machines. Software running in places that we would never have previously imagined. And much of it abandoned immediately after shipping, gently rotting, exposing an increasingly large number of widely known security vulnerabilities to an increasingly hostile internet. Devices that become useless because of protocol updates. Toys that have a "Guaranteed to work until" date, and then suddenly Barbie goes dead and you're forced to have an unexpected conversation about API mortality with your 5-year old child.

We can't fix all of these things. Many of these devices have important functionality locked inside proprietary components, released under licenses that grant no permission for people to examine or improve them. But there are many that we can. Millions of devices are running modern and secure versions of Android despite being abandoned by their manufacturers, purely because the vendor released appropriate source code and a community grew up to maintain it. But this can only happen when the vendor plays by the rules.

Vendors who don't release their code remove that freedom from their users, and the weapons users have to fight against that are limited. Most users hold no copyright over the software in the device and are unable to take direct action themselves. A vendor's failure to comply dooms them to having to choose between buying a new device in 12 months or no longer receiving security updates. When yet more examples of vendor-supplied malware are discovered, it's more difficult to produce new builds without them. The utility of the devices that the user purchased is curtailed significantly.

The Software Freedom Conservancy is one of the only organisations actively fighting against this, and if they're forced to give up their enforcement work the pressure on vendors to comply with the GPL will be reduced even further. If we want users to control their devices, to be able to obtain security updates even after the vendor has given up, we need to keep that pressure up. Supporting the SFC's work has a real impact on the security of the internet and people's lives. Please consider giving them money.

comments

Alessio Treglia: Creativity, Innovation and the Included Middle logic

The pressure of the post-modernism is establishing its bases on our general lack of ability to overcome a number of dualisms that have become ingrained in the modern way of thinking[1]. This is mainly due to the strong influence of past centuries scientific Reductionism , which postulated that any system to be understood had to be reduced to its minimum component elements.

However, a so defined system is a closed system, which does not interact with the surrounding environment and it can exist (not always) only in a reality-isolated laboratory. The logic of Complexity , instead, takes into account the open systems and all the interconnections and influences of the system itself with the world around it, in every physical, social, psychological and symbolic aspect

20 December 2015

Lunar: Reproducible builds: week 34 in Stretch cycle

What happened in the reproducible builds effort between December 13th to December 19th: Infrastructure Niels Thykier started implementing support for .buildinfo files in dak. A very preliminary commit was made by Ansgar Burchardt to prevent .buildinfo files from being removed from the upload queue. Toolchain fixes

Niels Thykier uploaded debhelper/9.20151219 which sorts files read by dh_installinit. Patch by Reiner Herrmann.
Jo Shields uploaded mona/4.2.1.102+dfsg2-4 which lands upstream changes making GUID reproducible in unstable.
Niko Tyni uploaded perl/5.22.1-1 which makes support for SOURCE_DATE_EPOCH in podlators available in unstable.

Mattia Rizzolo rebased our experimental debhelper with the changes from the latest upload. New fixes have been merged by OCaml upstream. Packages fixed The following 39 packages have become reproducible due to changes in their build dependencies: apache-mime4j, avahi-sharp, blam, bless, cecil-flowanalysis, cecil, coco-cs, cowbell, cppformat, dbus-sharp-glib, dbus-sharp, gdcm, gnome-keyring-sharp, gudev-sharp-1.0, jackson-annotations, jackson-core, jboss-classfilewriter, jboss-jdeparser2, jetty8, json-spirit, lat, leveldb-sharp, libdecentxml-java, libjavaewah-java, libkarma, mono.reflection, monobristol, nuget, pinta, snakeyaml, taglib-sharp, tangerine, themonospot, tomboy-latex, widemargin, wordpress, xsddiagram, xsp, zeitgeist-sharp. The following packages became reproducible after getting fixed:

4ti2/1.6.7+ds-2 uploaded by Jerome Benoit, original patch by Reiner Herrmann.
clblas/2.8+ds1-3 by Ghislain Antony Vaillant.
config-manager/0.4-2.2 by Mattia Rizzolo.
cvs-fast-export/1.35-1 by Anthony Fok.
dgedit/0~git20151217-1 by V ctor Cuadrado Juan.
genometools/1.5.7-7 by Sascha Steinbiss.
gnome-blog/0.9.1-7 uploaded by Frederic Peters, based on an NMU by Mattia Rizzolo.
guava-libraries/19.0-1 by Emmanuel Bourg.
kwave/0.9.0-1-1 uploaded by Pino Toscano, fixed upstream.
libandroid-json-org-java/20121204-20090211-2 by Emmanuel Bourg.
libchado-perl/1.23-5 uploaded by Andreas Tille, original patch by Niko Tyni.
libffi/3.2.1-4 by Matthias Klose.
mksh/52-1 by Thorsten Glaser.
monit/1:5.15-3 uploaded by Sergey B Kirpichev, original patch by Chris Lamb.
ntlmaps/0.9.9.0.1-11.5 by Mattia Rizzolo.
perl/5.22.1~rc3-2 by Niko Tyni.
picolisp/15.11-1 by Kan-Ru Chen.
qelectrotech/1:0.5-1 uploaded by Denis Briand, fixed upstream.
tomcat8/8.0.30-1 by Emmanuel Bourg.
xfonts-ayu/1:1.7a-1 by Hideki Yamane with a patch from Chris Lamb.
xfonts-kaname/1.1-10 by Hideki Yamane with a patch from Chris Lamb.
xfonts-kappa20/1:0.396-1 by Hideki Yamane with a patch from Chris Lamb.

Some uploads fixed some reproducibility issues, but not all of them:

bup/0.27-2 uploaded by Robert Edmonds, patch by Chris Lamb.
cain/1.10+dfsg-1 uploaded by Andreas Tille, original patch by Chris Lamb.
liblouisutdml/2.5.0-2 by Samuel Thibault.
mariadb-10.0/10.0.22-5 by Otto Kek l inen.

Patches submitted which have not made their way to the archive yet:

#807837 on lxc by Reiner Herrmann: use time of latest debian/changelog entry for LXC_GENERATE_DATE.
#807838 on graphite2 by Reiner Herrmann: tell dblatex to use a static path.
#808032 on python-genpy by Chris Lamb: sort list of generated modules.
#808388 on buzztrax by Chris Lamb: implement support for SOURCE_DATE_EPOCH.

reproducible.debian.net Packages in experimental are now tested on armhf. (h01ger) Arch Linux packages in the multilib and community repositories (4,000 more source packages) are also being tested. All of these test results are better analyzed and nicely displayed together with each package. (h01ger) For Fedora, build jobs can now run in parallel. Two are currently running, now testing reproducibility of 785 source packages from Fedora 23. mock/1.2.3-1.1 has been uploaded to experimental to better build RPMs. (h01ger) Work has started on having automatic build node pools to maximize use of armhf build nodes. (Vagrant Cascadian) diffoscope development Version 43 has been released on December 15th. It has been dubbed as epic! as it contains many contributions that were written around the summit in Athens. Baptiste Daroussin found that running diffoscope on some Tar archives could overwrite arbitrary files. This has been fixed by using libarchive instead of Python internal Tar library and adding a sanity check for destination paths. In any cases, until proper sandboxing is implemented, don't run diffosope on unstrusted inputs outside an isolated, throw-away system. Mike Hommey identified that the CBFS comparator would needlessly waste time scanning big files. It will now not consider any files bigger than 24 MiB 8 MiB more than the largest ROM created by coreboot at this time. An encoding issue related to Zip files has also been fixed. (Lunar) New comparators have been added: Android dex files (Reiner Herrmann), filesystem images using libguestfs (Reiner Herrmann), icons and JPEG images using libcaca (Chris Lamb), and OS X binaries (Clemens Lang). The comparator for Free Pascal Compilation Unit will now only be used when the unit version matches the compiler one. (Levente Polyak) A new multi-file HTML output with on-demand loading of long diffs is available through the --html-dir option. On-demand loading requires jQuery which path can be specified through the --jquery option. The diffs can also be simply browsed for non-JavaScript users or when jQuery is not available. (Joachim Breitner) Example of on-demand loading in diffosope

Example of on-demand loading in diffosope

Portability toward other systems has been improved: old versions of GNU diff are now supported (Mike McQuaid), suggestion of the appropriate locale is now the more generic en_US.UTF-8 (Ed Maste), the --list-tools option can now support multiple systems (Mattia Rizzolo, Levente Polyak, Lunar). Many internal changes and code clean-ups have been made, paving the way for parallel processing. (Lunar) Version 44 was released on December 18th fixing an issue affecting .deb lacking a md5sums file introduced in a previous refactoring (Lunar). Support has been added for Mozilla optimized Zip files. (Mike Hommey). The HTML output has been optimized in size (Mike Hommey, Esa Peuha, Lunar), speed (Lunar), and will now properly number lines (Mike Hommey). A message will always be displayed when lines are ignored at the end of a diff (Lunar). For portability and consistency, Python os.walk() function is now used instead of find to perform directory listing. (Lunar) Documentation update Package reviews 143 reviews have been removed, 69 added and 22 updated in the previous week. Chris Lamb reported 12 new FTBFS issues. News issues identified this week: random_order_in_init_py_generated_by_python-genpy, timestamps_in_copyright_added_by_perl_dist_zilla, random_contents_in_dat_files_generated_by_chasen-dictutils_makemat, timestamps_in_documentation_generated_by_pandoc. Chris West did some improvements on the scripts used to manage notes in the misc repository. Misc. Accounts of the reproducible builds summit in Athens were written by Thomas Klausner from NetBSD and Hans-Christoph Steiner from The Guardian Project. Some openSUSE developers are working on a hackweek on reproducible builds which was discussed on the opensuse-packaging mailing-list.

Next.

Previous.