Search Results: "cas"

4 September 2015

Norbert Preining: How to rename a local OfflineIMAP managed folder

Since quite some time I am using OfflineIMAP to keep my mailbox status from various servers locally available on my machine. I did more or less follow the excellent guide A unix style mail setup, but didn t use folder name translation right from the beginning. So I got stuck with four folders named INBOX: acc1/INBOX, acc2/INBOX, etc. This wouldn t be a big problem, but when using Mutt with the folder sidebar patch (as available in the mutt-patched package in Debian), there is the problem that only the INBOX part is shown. offlineimap So I decided to rename all the local folders to the scheme described in the above guide. At the end I want to have a folder layout as follows:
  Maildir/acc1/acc1
  Maildir/acc1/acc1.folder1
  Maildir/acc1/acc1.folder2
where the acc1 related to the INBOX on the remote account acc1, and the others are folders on the remote end. Unfortunately, renaming is not supported in offlineimap, it would simply download everything again at best! So I checked what needs to be done, and it turned out it is not as complicated as I thought. The necessary steps are: To make this more specific, let us assume I want to reorganize the account acc1 in this way, and there are two folders synced. That is, before we start the Maildir folder contains:
Maildir/acc1/INBOX
Maildir/acc1/folder1
Maildir/acc1/folder2
and we want to achieve that the layout is
Maildir/acc1/acc1
Maildir/acc1/acc1.folder1
Maildir/acc1/acc1.folder2
Due to the fact that both the LocalStatus files and the FolderValidity files are naned in the same way as the folders, the following trivial script does all the necessary renaming for this simple case:
for d in ~/Maildir/acc1 ~/.offlineimap/Account-acc1/LocalStatus ~/.offlineimap/Repository-acc1local/FolderValidity ; do
  mv $d/INBOX $d/acc1
  mv $d/folder1 $d/acc1.folder1
  mv $d/folder2 $d/acc1.folder2
done
Finally, the correct renaming has to be done by adding nametrans rules to the helper script. The functions oimaptransfolder_acc1 and localtransfolder_acc1 as defined in the above tutorial can be used as is, and need only be added to the respective Repository sections in the configuration file. After having done the necessary renaming, what remains is to give it a good test. Best way to do it is with
offlineimap --dry-run -o -a acc1
which runs the syncronization in dry-run mode, so nothing is actually copied, and only once and only for acc1. If during this test run nothing strange appears, like many many emails are deleted on either end, then you are fine. Enjoy.

1 September 2015

Raphaël Hertzog: My Free Software Activities in August 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 6.5 hours on Debian LTS. In that time I did the following: Apart from that, I also gave a talk about Debian LTS at DebConf 15 in Heidelberg and also coordinated a work session to discuss our plans for Wheezy. Have a look at the video recordings: DebConf 15 I attended DebConf 15 with great pleasure after having missed DebConf 14 last year. While I did not do lots of work there, I participated in many discussions and I certainly came back with a renewed motivation to work on Debian. That s always good. :-) For the concrete work I did during DebConf, I can only claim two schroot uploads to fix the lack of support of the new overlay filesystem that replaces aufs in the official Debian kernel, and some Distro Tracker work (fixing an issue that some people had when they were logged in via Debian s SSO). While the numerous discussions I had during DebConf can t be qualified as work , they certainly contribute to build up work plans for the future: As a Kali developer, I attended multiple sessions related to derivatives (notably the Debian Derivatives Panel). I was also interested by the Debian in the corporate IT BoF led by Michael Meskes (Credativ s CEO). He pointed out a number of problems that corporate users might have when they first consider using Debian and we will try to do something about this. Expect further news and discussions on the topic. Martin Kraff, Luca Filipozzi, and me had a discussion with the Debian Project Leader (Neil) about how to revive/transform the Debian s Partner program. Nothing is fleshed out yet, but at least the process initiated by the former DPL (Lucas) is again moving forward. Other Debian work Sponsorship. I sponsored an NMU of pep8 by Daniel Stender as it was a requirement for prospector which I also sponsored since all the required dependencies are now available in Debian. \o/ Packaging. I NMUed libxml2 2.9.2+really2.9.1+dfsg1-0.1 fixing 3 security issues and a RC bug that was breaking publican. Since there s no upstream fix for more than 8 months, I went back to the former version 2.9.1. It s in line with the new requirement of release managers a package in unstable should migrate to testing reasonably quickly, it s not acceptable to keep it unfixed for months. With this annoying bug fixed, I could again upload a new upstream release of publican so I prepared and uploaded 4.3.2-1. It was my first source only upload. This release was more work than I expected and I filed no less than 3 bug to upstream (new bash-completion install path, request to provide sources of a minified javascript file, drop a .po file for an invalid language code). GPG issues with smartcard. Back from DebConf, when I wanted to sign some key, I stumbled again upon the problem which makes it impossible for me to use my two smartcards one after the other without first deleting the stubs for the private key. It s not a new issue but I decided that it was time to report it upstream, so I did it: #2079 on bugs.gnupg.org. Some research helped me to find a way to work-around the problem. Later in the month, after a dist-upgrade and a reboot, I was no longer able to use my smartcard as a SSH authentication key again it was already reported but there was no clear analysis, so I tried to do my own one and added the results of my investigation in #795368. It looks like the culprit is pinentry-gnome3 not working when started by the gpg-agent which is started before the DBUS session. Simple fix is to restart the gpg-agent in the session but I have no idea yet of what the proper fix should be (letting systemd manage the graphical user session and start gpg-agent would be my first answer, but that doesn t solve the issue for users of other init systems so it s not satisfying). Distro Tracker. I merged two patches from Orestis Ioannou fixing some bugs tagged newcomer. There are more such bugs (I even filed two: #797096 and #797223), go grab them and do a first contribution to Distro Tracker like Orestis just did! I also merged a change from Christophe Siraut who presented Distro Tracker at DebConf. I implemented in Distro Tracker the new authentication based on SSL client certificates that was recently announced by Enrico Zini. It s working nice, and this authentication scheme is far easier to support. Good job, Enrico! tracker.debian.org broke during DebConf, it stopped being updated with new data. I tracked this down to a problem in the archive (see #796892). Apparently Ansgar Burchardt changed the set of compression tools used on some jessie repositorie, replacing bz2 by xz. He dropped the old Packages.bz2 but missed some Sources.bz2 which were thus stale and APT reported Hashsum mismatch on the uncompressed content. Misc. I pushed some small improvement to my Salt formulas: schroot-formula and sbuild-formula. They will now auto-detect which overlay filesystem is available with the current kernel (previously aufs was hardcoded). Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

31 August 2015

Matthew Garrett: Working with the kernel keyring

The Linux kernel keyring is effectively a mechanism to allow shoving blobs of data into the kernel and then setting access controls on them. It's convenient for a couple of reasons: the first is that these blobs are available to the kernel itself (so it can use them for things like NFSv4 authentication or module signing keys), and the second is that once they're locked down there's no way for even root to modify them.

But there's a corner case that can be somewhat confusing here, and it's one that I managed to crash into multiple times when I was implementing some code that works with this. Keys can be "possessed" by a process, and have permissions that are granted to the possessor orthogonally to any permissions granted to the user or group that owns the key. This is important because it allows for the creation of keyrings that are only visible to specific processes - if my userspace keyring manager is using the kernel keyring as a backing store for decrypted material, I don't want any arbitrary process running as me to be able to obtain those keys[1]. As described in keyrings(7), keyrings exist at the session, process and thread levels of granularity.

This is absolutely fine in the normal case, but gets confusing when you start using sudo. sudo by default doesn't create a new login session - when you're working with sudo, you're still working with key posession that's tied to the original user. This makes sense when you consider that you often want applications you run with sudo to have access to the keys that you own, but it becomes a pain when you're trying to work with keys that need to be accessible to a user no matter whether that user owns the login session or not.

I spent a while talking to David Howells about this and he explained the easiest way to handle this. If you do something like the following:
$ sudo keyctl add user testkey testdata @u
a new key will be created and added to UID 0's user keyring (indicated by @u). This is possible because the keyring defaults to 0x3f3f0000 permissions, giving both the possessor and the user read/write access to the keyring. But if you then try to do something like:
$ sudo keyctl setperm 678913344 0x3f3f0000
where 678913344 is the ID of the key we created in the previous command, you'll get permission denied. This is because the default permissions on a key are 0x3f010000, meaning that the possessor has permission to do anything to the key but the user only has permission to view its attributes. The cause of this confusion is that although we have permission to write to UID 0's keyring (because the permissions are 0x3f3f0000), we don't possess it - the only permissions we have for this key are the user ones, and the default state for user permissions on new keys only gives us permission to view the attributes, not change them.

But! There's a way around this. If we instead do:
$ sudo keyctl add user testkey testdata @s
then the key is added to the current session keyring (@s). Because the session keyring belongs to us, we possess any keys within it and so we have permission to modify the permissions further. We can then do:
$ sudo keyctl setperm 678913344 0x3f3f0000
and it works. Hurrah! Except that if we log in as root, we'll be part of another session and won't be able to see that key. Boo. So, after setting the permissions, we should:
$ sudo keyctl link 678913344 @u
which ties it to UID 0's user keyring. Someone who logs in as root will then be able to see the key, as will any processes running as root via sudo. But we probably also want to remove it from the unprivileged user's session keyring, because that's readable/writable by the unprivileged user - they'd be able to revoke the key from underneath us!
$ sudo keyctl unlink 678913344 @s
will achieve this, and now the key is configured appropriately - UID 0 can read, modify and delete the key, other users can't.

This is part of our ongoing work at CoreOS to make rkt more secure. Moving the signing keys into the kernel is the first step towards rkt no longer having to trust the local writable filesystem[2]. Once keys have been enrolled the keyring can be locked down - rkt will then refuse to run any images unless they're signed with one of these keys, and even root will be unable to alter them.

[1] (obviously it should also be impossible to ptrace() my userspace keyring manager)
[2] Part of our Secure Boot work has been the integration of dm-verity into CoreOS. Once deployed this will mean that the /usr partition is cryptographically verified by the kernel at runtime, making it impossible for anybody to modify it underneath the kernel. / remains writable in order to permit local configuration and to act as a data store, and right now rkt stores its trusted keys there.

comment count unavailable comments

Russ Allbery: Review: Through Struggle, The Stars

Review: Through Struggle, The Stars, by John J. Lumpkin
Series: Human Reach #1
Publisher: John J. Lumpkin
Copyright: July 2011
ISBN: 1-4611-9544-6
Format: Kindle
Pages: 429
Never let it be said that I don't read military SF. However, it can be said that I read books and then get hellishly busy and don't review them for months. So we'll see if I can remember this well enough to review it properly. In Lumpkin's future world, mankind has spread to the stars using gate technology, colonizing multiple worlds. However, unlike most science fiction futures of this type, it's not all about the United States, or even the United States and Russia. The great human powers are China and Japan, with the United States relegated to a distant third. The US mostly maintains its independence from either, and joins the other lesser nations and UN coalitions to try to pick up a few colonies of its own. That's the context in which Neil and Rand join the armed services: the former as a pilot in training, and the latter as an army grunt. This is military SF, so of course a war breaks out. But it's a war between Japan and China: improved starship technology and the most sophisticated manufacturing in the world against a much larger economy with more resources and a larger starting military. For reasons that are not immediately clear, and become a plot point later on, the United States president immediately takes an aggressive tone towards China and pushes the country towards joining the war on the side of Japan. Most of this book is told from Neil's perspective, following his military career during the outbreak of war. His plans to become a pilot get derailed as he gets entangled with US intelligence agents (and a bad supervisor). The US forces are not in a good place against China, struggling when they get into ship-to-ship combat, and Neil's ship goes on a covert mission to attempt to complicate the war with political factors. Meanwhile, Rand tries to help fight off a Chinese invasion of one of the rare US colony worlds. Through Struggle, The Stars does not move quickly. It's over 400 pages, and it's a bit surprising how little happens. What it does instead is focus on how the military world and the war feels to Neil: the psychological experience of wanting to serve your country but questioning its decisions, the struggle of working with people who aren't always competent but who you can't just make go away, the complexities of choosing a career path when all the choices are fraught with politics that you didn't expect to be involved in, and the sheer amount of luck and random events involved in the progression of one's career. I found this surprisingly engrossing despite the slow pace, mostly because of how true to life it feels. War is not a never-ending set of battles. Life in a military ship has moments when everything is happening far too quickly, but other moments when not much happens for a long time. Lumpkin does a great job of reflecting that. Unfortunately, I thought there were two significant flaws, one of which means I probably won't seek out further books in this series. First, one middle portion of the book switches away from Neil to follow Rand instead. The first part of that involves the details of fighting orbiting ships with ground-based lasers, which was moderately interesting. (All the technological details of space combat are interesting and thoughtfully handled, although I'm not the sort of reader who will notice more subtle flaws. But David Weber this isn't, thankfully.) But then it turns into a fairly generic armed resistance story, which I found rather boring. It also ties into the second and more serious flaw: the villain. The overall story is constructed such that it doesn't need a personal villain. It's about the intersection of the military and politics, and a war that may be ill-conceived but that is being fought anyway. That's plenty of conflict for the story, at least in my opinion. But Lumpkin chose to introduce a specific, named Chinese character in the villain role, and the characterization is... well. After he's humiliated early in the story by the protagonists, Li Xiao develops an obsession with killing them, for his honor, and then pursues them throughout the book in ways that are sometimes destructive to the Chinese war efforts. It's badly unrealistic compared to the tone of realism taken by the rest of the story. Even if someone did become this deranged, it's bizarre that a professional military (and China's forces are otherwise portrayed as fairly professional) would allow this. Li reads like pure caricature, and despite being moderately competent apart from his inexplicable (but constantly repeated) obsession, is cast in a mustache-twirling role of personal villainy. This is weirdly out of place in the novel, and started irritating me enough that it took me out of the story. Through Struggle, The Stars is the first book of a series, and does not resolve much by the end of the novel. That plus its length makes the story somewhat unsatisfying. I liked Neil's development, and liked him as a character, and those who like the details of combat mixed with front-lines speculation about politics may enjoy this. But a badly-simplified mustache-twirling victim and some extended, uninteresting bits mar the book enough that I probably won't seek out the sequels. Followed by The Desert of Stars. Rating: 6 out of 10

30 August 2015

Andrew Cater: Rescuing a Windows 10 failed install using GParted Live on CD

Windows 10 is here, for better or worse. As the family sysadmin, I've been tasked to update the Windows machines: ultimately, failure modes are not well documented and I needed Free software and several hours to recover a vital machine.

The "free upgrade for users of prior Windows versions" is a limited time offer for a year from launch. Microsoft do not offer licence keys for the upgrade: once a machine has updated to Windows 10 and authenticated on the 'Net, then a machine can be re-installed and will be regarded by Microsoft as pre-authorised. Users don't get the key at any point.

Although Microsoft have pushed the fact that this can be done through Windows Update, there is the option to use Microsoft's Media Creation tool to do the upgrade directly on the Windows machine concerned. This would be necessary to get the machine to upgrade and register before a full clean install of Windows 10 from media.

This Media Creation Tool failed for me on three machines with "Unable to access System reserved partition'

All the machines have SSDs from various makers: a colleague suggested that resizing the partition might enable the upgrade to continue. Of the machines that failed, all were running Windows 7 - two were running using BIOS, one using UEFI boot on a machine with no Legacy/CSM mode.

Using GParted live .iso - itself based on Debian Live - allowed me to resize the System partition from 100MiB to 200MiB by moving the Windows partition but Windows became unbootable.

In two cases, I was able to boot from DVD Windows installation media and make Windows bootable again at which point the Microsoft Media Creation Tool could be used to install Windows 10

The UEFI boot machine proved more problematic: I had to create a Windows 7 System Repair disk and repair Windows booting before Windows 10 could proceed.

My Windows-using colleaague had used only Windows-based recovery disks: using Linux tools allowed me to repair Windows installations I couldn't boot

Philipp Kern: Automating the 3270 part of a Debian System z install

If you try to install Debian on System z within z/VM you might be annoyed at the various prompts it shows before it lets you access the network console via SSH. We can do better. From within CMS copy the default EXEC and default PARMFILE:

COPYFILE DEBIAN EXEC A DEBAUTO EXEC A
COPYFILE PARMFILE DEBIAN A PARMFILE DEBAUTO A

Now edit DEBAUTO EXEC A and replace the DEBIAN in 'PUNCH PARMFILE DEBIAN * (NOHEADER' with DEBAUTO. This will load the alternate kernel parameters file into the card reader, while still loading the original kernel and initrd files.

Replace PARMFILE DEBAUTO A's content with this (note the 80 character column limit):

ro locale=C
s390-netdevice/choose_networktype=qeth s390-netdevice/qeth/layer2=true
s390-netdevice/qeth/choose=0.0.fffc-0.0.fffd-0.0.fffe
netcfg/get_ipaddress=<IPADDR> netcfg/get_netmask=255.255.255.0
netcfg/get_gateway=<GW> netcfg/get_nameservers=<FIRST-DNS>
netcfg/confirm_static=true netcfg/get_hostname=debian
netcfg/get_domain=
network-console/authorized_keys_url=http://www.kern.lc/id_rsa.pub
preseed/url=http://www.kern.lc/preseed-s390x.cfg

Replace <IPADDR>, <GW>, and <FIRST-DNS> to suit your local network config. You might also need to change the netmask, which I left in for clarity about the format. Adjust the device address of your OSA network card. If it's in layer 3 mode (very likely) you should set layer2=false. Note that mixed case matters, hence you will want to SET CASE MIXED in xedit.

Then there are the two URLs that need to be changed. The authorized_keys_url file contains your SSH public key and is fetched unencrypted and unauthenticated, so be careful what networks you traverse with your request (HTTPS is not supported by debian-installer in Debian).

preseed/url is needed for installation parameters that do not fit the parameters file - there is an upper character limit that's about two lines longer than my example. This is why this example only contains the bare minimum for the network part, everything else goes into this preseeding file. It file can optionally be protected with a MD5 checksum in preseed/url/checksum.

Both URLs need to be very short. I thought that there was a way to specify a line continuation, but in my tests I was unable to produce one. Hence it needs to fit on one line, including the key. You might want to use an IPv4 as the hostname.

To skip the initial boilerplate prompts and to skip straight to the user and disk setup you can use this as preseed.cfg:

d-i debian-installer/locale string en_US
d-i debian-installer/country string US
d-i debian-installer/language string en
d-i time/zone US/Eastern
d-i mirror/country manual
d-i mirror/http/mirror string httpredir.debian.org
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string

I'm relatively certain that the DASD disk setup part cannot be automated yet. But the other bits of the installation should be preseedable just like on non-mainframe hardware.

28 August 2015

Gunnar Wolf: 180

180 degrees people say their life has changed by 180 whenever something alters their priorities, their viewpoints, their targets in life. In our case, it's been 180 days. 183 by today, really. The six most amazing months in my life. We are still the same people, with similar viewpoints and targets. Our priorities have clearly shifted. But our understanding of the world, and our sources of enjoyment, and our outlook for the future... Are worlds apart. Not 180 , think more of a quantic transposition.

Dimitri John Ledkov: Go enjoy Python3

Given a string, get a truncated string of length up to 12.

The task is ambiguous, as it doesn't say anything about whether or not 12 should include terminating null character or not. None the less, let's see how one would achieve this in various languages.
Let's start with python3

import sys
print(sys.argv[1][:12])

Simple enough, in essence given first argument, print it up to length 12. As an added this also deals with unicode correctly that is if passed arg is , it will correctly print . (note these are just random Unicode strings to me, no idea what they stand for).

In C things are slightly more verbose, but in essence, I am going to use strncpy function:

#include <stdio.h>
#include <string.h>
void main(int argc, char *argv[])
char res[12];
strncpy(res,argv[1],12);
printf("%s\n",res);

This treats things as byte-array instead of unicode, thus for unicode test it will end up printing just . But it is still simple enough.
Finally we have Go
package main

import "os"
import "fmt"
import "math"

func main()
fmt.Printf("%s\n", os.Args[1][:int(math.Min(12, float64(len(os.Args[1]))))])

This similarly treats argument as a byte array, and one needs to cast the argument to a rune to get unicode string handling. But there are quite a few caveats. One cannot take out of bounds slices. Thus a na ve os.Args[1][:12] can result in a runtime panic that slice bounds are out of range. Or if a string is known at compile time, a compile time error. Hence one needs to calculate length, and do a min comparison. And there lies the next caveat, math.Min() is only defined for float64 type, and slice indexes can only be integers and thus we end up writing ]))))])...

12 points for python3, 8 points for C, and Go receives nul points Eurovision style.

EDIT: Andreas R ssland and James Hunt are full of win. Both suggesting fmt.Printf("%.12s\n", os.Args[1]) for go. I like that a lot, as it gives simplicity & readability without compromising the default safety against out of bounds access. Hence the scores are now: 14 points for Go, 12 points for python3 and 8 points for C.

EDIT2: I was pointed out much better C implementation by Keith Thompson - http://pastebin.com/5i7rFmMQ in essence it uses strncat() which has much better null termination semantics. And Ben posted a C implementation which handles wide characters http://www.decadent.org.uk/ben/blog/truncating-a-string-in-c.html. I regret to inform you that this blog post got syndicated onto hacker news and has now become the top viewed post on my blog of all time, overnight. In retrospect, I regret awarding points at the end of the blog post, as that's just was merely an expression of opinion and is highly subjective measure. But this problem statement did originate from me reviewing go code that did "if/then/else" comparison and got it wrong to truncate a string and I thought surely one can just do [:12] which has lead me down the rabbit hole of discovering a lot about Go; it's compile and runtime out of bounds access safeguards; lack of universal Min() function; runes vs strings handling and so on. I'm only a beginner go programmer and I am very sorry for wasting everyone's time on this. I guess people didn't have much to do on a Throwback Thursday.

The postings on this site are my own and don't necessarily represent Intel s positions, strategies, or opinions.

Lucas Nussbaum: DebConf 15

I attended DebConf 15 last week. After being on semi-vacation from Debian for the last few months, recovering after the end of my second DPL term, it was great to be active again, talk to many people, and go back to doing technical work. Unfortunately, I caught the debbug quite early in the week, so I was not able to make it as intense as I wanted, but it was great nevertheless. I still managed to do quite a lot:
  • I rewrote a core part of UDD, which will make it easier to monitor data importer scripts and reduce the cron-spam
  • with DSA members, I worked on finding a suitable workaround for the storage performance issues that have been plaguing UDD for the last few months. fsyncs() will now longer hang for 15 minutes, yay!
  • I added a DUCK importer to UDD, and added that information to the Debian Maintainer Dashboard
  • I worked a bit on cleaning up the status of my packages, including digging into a strange texlive issue (that showed up in developers-reference), that is now fixed in unstable
  • I worked a bit on improving git-buildpackage documentation (more to come in that area)
  • Last but not least, I played Mao for the first time in years, and it was a lot of fun. (even if my brain is still slowly recovering)
DC15 was a great DebConf, probably one of the two bests I ve attended so far. I m now looking forward to DC16 in Cape Town!

Ben Hutchings: Securing debcheckout of git repositories

Some source packages have Vcs-Git URLs using the git: scheme, which is plain-text and unauthenticated. It's probably harder to MITM than HTTP, but still we can do better than this even for anonymous checkouts. git is now nearly as efficient at cloning/pulling over HTTP-S, so why not make that the default? Adding the following lines to ~/.gitconfig will make git consistently use HTTP-S to access Alioth. It's not quite HTTPS-Everywhere, but it's a step in that direction:
[url "https://anonscm.debian.org/git/"]
	insteadOf = git://anonscm.debian.org/
	insteadOf = git://git.debian.org/
Additionally you can automatically fix up the push URL in case you have or are later given commit access to the repository on Alioth:
[url "git+ssh://git.debian.org/git/"]
	pushInsteadOf = git://anonscm.debian.org/
	pushInsteadOf = git://git.debian.org/
Similar for git.kernel.org:
[url "https://git.kernel.org/pub/scm/"]
	insteadOf = git://git.kernel.org/pub/scm/
[url "git+ssh://ra.kernel.org/pub/scm/"]
	pushInsteadOf = git://git.kernel.org/pub/scm/
RTFM for more information on these configuration variables.

Ben Hutchings: Securing debcheckout of git repositories

Some source packages have Vcs-Git URLs using the git: scheme, which is plain-text and unauthenticated. It's probably harder to MITM than HTTP, but still we can do better than this even for anonymous checkouts. git is now nearly as efficient at cloning/pulling over HTTP-S, so why not make that the default? Adding the following lines to ~/.gitconfig will make git consistently use HTTP-S to access Alioth. It's not quite HTTPS-Everywhere, but it's a step in that direction:
[url "https://anonscm.debian.org/git/"]
	insteadOf = git://anonscm.debian.org/
	insteadOf = git://git.debian.org/
Additionally you can automatically fix up the push URL in case you have or are later given commit access to the repository on Alioth:
[url "git+ssh://git.debian.org/git/"]
	pushInsteadOf = git://anonscm.debian.org/
	pushInsteadOf = git://git.debian.org/
Similar for git.kernel.org:
[url "https://git.kernel.org/pub/scm/"]
	insteadOf = git://git.kernel.org/pub/scm/
[url "git+ssh://ra.kernel.org/pub/scm/"]
	pushInsteadOf = git://git.kernel.org/pub/scm/
RTFM for more information on these configuration variables.

Ben Hutchings: Securing debcheckout of git repositories

Some source packages have Vcs-Git URLs using the git: scheme, which is plain-text and unauthenticated. It's probably harder to MITM than HTTP, but still we can do better than this even for anonymous checkouts. git is now nearly as efficient at cloning/pulling over HTTP-S, so why not make that the default? Adding the following lines to ~/.gitconfig will make git consistently use HTTP-S to access Alioth. It's not quite HTTPS-Everywhere, but it's a step in that direction:
[url "https://anonscm.debian.org/git/"]
	insteadOf = git://anonscm.debian.org/
	insteadOf = git://git.debian.org/
Additionally you can automatically fix up the push URL in case you have or are later given commit access to the repository on Alioth:
[url "git+ssh://git.debian.org/git/"]
	pushInsteadOf = git://anonscm.debian.org/
	pushInsteadOf = git://git.debian.org/
Similar for git.kernel.org:
[url "https://git.kernel.org/pub/scm/"]
	insteadOf = git://git.kernel.org/pub/scm/
[url "git+ssh://ra.kernel.org/pub/scm/"]
	pushInsteadOf = git://git.kernel.org/pub/scm/
RTFM for more information on these configuration variables.

27 August 2015

Ben Hutchings: Truncating a string in C

This version uses the proper APIs to work with the locale's multibyte encoding (with single-byte encodings being a trivial case of multibyte). It will fail if it encounters an invalid byte sequence (e.g. byte > 127 in the "C" locale), though it could be changed to treat each rejected byte as a single character.
#include <locale.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <wchar.h>
int main(int argc, char **argv)
 
    size_t n = 12, totlen = 0, maxlen, chlen;
    setlocale(LC_ALL, "");
    if (argc != 2)
	return EXIT_FAILURE;
    maxlen = strlen(argv[1]);
    while (n--)  
	chlen = mbrlen(argv[1] + totlen, maxlen - totlen, NULL);
	if (chlen > MB_CUR_MAX)
	    return EXIT_FAILURE;
	totlen += chlen;
     
    printf("%.*s\n", (int)totlen, argv[1]);
    return 0;
 

Ben Hutchings: Truncating a string in C

This version uses the proper APIs to work with the locale's multibyte encoding (with single-byte encodings being a trivial case of multibyte). It will fail if it encounters an invalid byte sequence (e.g. byte > 127 in the "C" locale), though it could be changed to treat each rejected byte as a single character.
#include <locale.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <wchar.h>
int main(int argc, char **argv)
 
    size_t n = 12, totlen = 0, maxlen, chlen;
    setlocale(LC_ALL, "");
    if (argc != 2)
	return EXIT_FAILURE;
    maxlen = strlen(argv[1]);
    while (n--)  
	chlen = mbrlen(argv[1] + totlen, maxlen - totlen, NULL);
	if (chlen > MB_CUR_MAX)
	    return EXIT_FAILURE;
	totlen += chlen;
     
    printf("%.*s\n", (int)totlen, argv[1]);
    return 0;
 

Ben Hutchings: Truncating a string in C

This version uses the proper APIs to work with the locale's multibyte encoding (with single-byte encodings being a trivial case of multibyte). It will fail if it encounters an invalid byte sequence (e.g. byte > 127 in the "C" locale), though it could be changed to treat each rejected byte as a single character.
#include <locale.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <wchar.h>
int main(int argc, char **argv)
 
    size_t n = 12, totlen = 0, maxlen, chlen;
    setlocale(LC_ALL, "");
    if (argc != 2)
	return EXIT_FAILURE;
    maxlen = strlen(argv[1]);
    while (n--)  
	chlen = mbrlen(argv[1] + totlen, maxlen - totlen, NULL);
	if (chlen > MB_CUR_MAX)
	    return EXIT_FAILURE;
	totlen += chlen;
     
    printf("%.*s\n", (int)totlen, argv[1]);
    return 0;
 

26 August 2015

Carl Chenet: Retweet 0.2 : bump to Python 3

Follow me on Identi.ca or Twitter or Diaspora*diaspora-banner Don t know Retweet? My last post about it introduced this small Twitter bot whichs just retweets (for now) every tweets from a Twitter account to another one. Retweet Retweet was created in order to improve the Journal du hacker Twitter account. The Journal du hacker is a Hacker News-like French-speaking website. logo-journal-du-hacker Especially useful to broadcast news through a network of Twitter accounts, Retweet was improved to bump Python version to 3.4 and to improve pep8 compliance (work in progress). The project is also well documented and should be quite simple to install, configure and use. After my first blog post about Retweet, new users gave me feedback about it and I now have great ideas for future features for the next release. Twitter_logo_blue What about you? If you try it, please tell me what you think about it, opening a bug request or ask for new features. Or just write your comment here ;)

Holger Levsen: 20150826-jenkins-fourth-state

jenkins has a fourth state So, at the jenkins.debian.org BOF (very short summary: j.d.o will be coming soonish, long summary thanks to the awesome video team) I shared a trick I discovered almost a year ago, but had never really announced anywhere yet, which enables one to programatically use a fourth state to the existing three jenkins job states ("success", "unstable" and "failed"), which is "aborted". Common knowledge is that it's only possible to abort jobs manually, but it's also possible to do that like this:
TMPFILE=$(mktemp)
curl https://jenkins.debian.net/jnlpJars/jenkins-cli.jar -o $TMPFILE
java -jar $TMPFILE -s http://localhost:8080/ set-build-result aborted
rm $TMPFILE
exit
The nice thing about aborted job runs is that these don't cause any notifications (neither mail nor IRC), so I intend to use this for several cases:
  • to abort jobs which encounter network problems
  • to abort jobs where a known bug will prevent the job from succeeding. This will require a small database to map bugs to jobs and some way to edit that database, so I will probably go with a .yaml file in some git repo.
I've no idea when I'll get along to actually implement that, so help doing this is very much welcome and I'd also be glad to help hooking this into the existing jenkins.debian.net.git codebase. In related news, I'm back home since Monday and am thankful for having shared a very nice and productive DebConf15 with many old and new friends in Heidelberg. Many thanks to everyone involved in making this happen!

25 August 2015

Lunar: Reproducible builds: week 17 in Stretch cycle

A good amount of the Debian reproducible builds team had the chance to enjoy face-to-face interactions during DebConf15.
Names in red and blue were all present at DebConf15
Picture of the  reproducible builds  talk during DebConf15
Hugging people with whom one has been working tirelessly for months gives a lot of warm-fuzzy feelings. Several recorded and hallway discussions paved the way to solve the remaining issues to get reproducible builds part of Debian proper. Both talks from the Debian Project Leader and the release team mentioned the effort as important for the future of Debian. A forty-five minutes talk presented the state of the reproducible builds effort. It was then followed by an hour long roundtable to discuss current blockers regarding dpkg, .buildinfo and their integration in the archive. Picture of the  reproducible builds  roundtable during DebConf15 Toolchain fixes
  • Kenneth J. Pronovici uploaded epydoc/3.0.1+dfsg-12 which makes class and modules ordering predictable (#795835) and fixes __repr__ so memory addresses don't appear in docs (#795826). Patches by Val Lorentz.
  • Sergei Golovan uploaded erlang/1:18.0-dfsg-2 which adds support for SOURCE_DATE_EPOCH to erlc. Patch by Chris West (Faux) and Chris Lamb.
  • Dmitry Shachnev uploaded sphinx/1.3.1-5 which make grammar, inventory, and JavaScript locales generation deterministic. Original patch by Val Lorentz.
  • St phane Glondu uploaded ocaml/4.02.3-2 to experimental, making startup files and native packed libraries deterministic. The patch adds deterministic .file to the assembler output.
  • Enrico Tassi uploaded lua-ldoc/1.4.3-3 which now pass the -d option to txt2man and add the --date option to override the current date.
Reiner Herrmann submitted a patch to make rdfind sort the processed files before doing any operation. Chris Lamb proposed a new patch for wheel implementing support for SOURCE_DATE_EPOCH instead of the custom WHEEL_FORCE_TIMESTAMP. akira sent one making man2html SOURCE_DATE_EPOCH aware. St phane Glondu reported that dpkg-source would not respect tarball permissions when unpacking under a umask of 002. After hours of iterative testing during the DebConf workshop, Sandro Knau created a test case showing how pdflatex output can be non-deterministic with some PNG files. Packages fixed The following 65 packages became reproducible due to changes in their build dependencies: alacarte, arbtt, bullet, ccfits, commons-daemon, crack-attack, d-conf, ejabberd-contrib, erlang-bear, erlang-cherly, erlang-cowlib, erlang-folsom, erlang-goldrush, erlang-ibrowse, erlang-jiffy, erlang-lager, erlang-lhttpc, erlang-meck, erlang-p1-cache-tab, erlang-p1-iconv, erlang-p1-logger, erlang-p1-mysql, erlang-p1-pam, erlang-p1-pgsql, erlang-p1-sip, erlang-p1-stringprep, erlang-p1-stun, erlang-p1-tls, erlang-p1-utils, erlang-p1-xml, erlang-p1-yaml, erlang-p1-zlib, erlang-ranch, erlang-redis-client, erlang-uuid, freecontact, givaro, glade, gnome-shell, gupnp, gvfs, htseq, jags, jana, knot, libconfig, libkolab, libmatio, libvsqlitepp, mpmath, octave-zenity, openigtlink, paman, pisa, pynifti, qof, ruby-blankslate, ruby-xml-simple, timingframework, trace-cmd, tsung, wings3d, xdg-user-dirs, xz-utils, zpspell. The following packages became reproducible after getting fixed: Uploads that might have fixed reproducibility issues: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet:
  • #795861 on fakeroot by Val Lorentz: set the mtime of all files to the time of the last debian/changelog entry.
  • #795870 on fatresize by Chris Lamb: set build date to the time of the latest debian/changelog entry.
  • #795945 on projectl by Reiner Herrmann: sort with LC_ALL set to C.
  • #795977 on dahdi-tools by Dhole: set the timezone to UTC before calling asciidoc.
  • #795981 on x11proto-input by Dhole: set the timezone to UTC before calling asciidoc.
  • #795983 on dbusada by Dhole: set the timezone to UTC before calling asciidoc.
  • #795984 on postgresql-plproxy by Dhole: set the timezone to UTC before calling asciidoc.
  • #795985 on xorg by Dhole: set the timezone to UTC before calling asciidoc.
  • #795987 on pngcheck by Dhole: set the date in the man pages to the latest debian/changelog entry.
  • #795997 on python-babel by Val Lorentz: make build timestamp independent from the timezone and remove the name of the build system locale from the documentation.
  • #796092 on a7xpg by Reiner Herrmann: sort with LC_ALL set to C.
  • #796212 on bittornado by Chris Lamb: remove umask-varying permissions.
  • #796251 on liblucy-perl by Niko Tyni: generate lib/Lucy.xs in a deterministic order.
  • #796271 on tcsh by Reiner Herrmann: sort with LC_ALL set to C.
  • #796275 on hspell by Reiner Herrmann: remove timestamp from aff files generated by mk_he_affix.
  • #796324 on fftw3 by Reiner Herrmann: remove date from documentation files.
  • #796335 on nasm by Val Lorentz: remove extra timestamps from the build system.
  • #796360 on libical by Chris Lamb: removes randomess caused Perl in generated icalderivedvalue.c.
  • #796375 on wcd by Dhole: set the date in the man pages to the latest debian/changelog entry.
  • #796376 on mapivi by Dhole: set the date in the man pages to the latest debian/changelog entry.
  • #796527 on vserver-debiantools by Dhole: set the date in the man pages to the latest debian/changelog entry.
St phane Glondu reported two issues regarding embedded build date in omake and cduce. Aur lien Jarno submitted a fix for the breakage of make-dfsg test suite. As binutils now creates deterministic libraries by default, Aur lien's patch makes use of a wrapper to give the U flag to ar. Reiner Herrmann reported an issue with pound which embeds random dhparams in its code during the build. Better solutions are yet to be found. reproducible.debian.net Package pages on reproducible.debian.net now have a new layout improving readability designed by Mattia Rizzolo, h01ger, and Ulrike. The navigation is now on the left as vertical space is more valuable nowadays. armhf is now enabled on all pages except the dashboard. Actual tests on armhf are expected to start shortly. (Mattia Rizzolo, h01ger) The limit on how many packages people can schedule using the reschedule script on Alioth has been bumped to 200. (h01ger) mod_rewrite is now used instead of JavaScript for the form in the dashboard. (h01ger) Following the rename of the software, debbindiff has mostly been replaced by either diffoscope or differences in generated HTML and IRC notification output. Connections to UDD have been made more robust. (Mattia Rizzolo) diffoscope development diffoscope version 31 was released on August 21st. This version improves fuzzy-matching by using the tlsh algorithm instead of ssdeep. New command line options are available: --max-diff-input-lines and --max-diff-block-lines to override limits on diff input and output (Reiner Herrmann), --debugger to dump the user into pdb in case of crashes (Mattia Rizzolo). jar archives should now be detected properly (Reiner Herrman). Several general code cleanups were also done by Chris Lamb. strip-nondeterminism development Andrew Ayer released strip-nondeterminism version 0.010-1. Java properties file in jar should now be detected more accurately. A missing dependency spotted by St phane Glondu has been added. Testing directory ordering issues: disorderfs During the reproducible builds workshop at DebConf, participants identified that we were still short of a good way to test variations on filesystem behaviors (e.g. file ordering or disk usage). Andrew Ayer took a couple of hours to create disorderfs. Based on FUSE, disorderfs in an overlay filesystem that will mount the content of a directory at another location. For this first version, it will make the order in which files appear in a directory random. Documentation update Dhole documented how to implement support for SOURCE_DATE_EPOCH in Python, bash, Makefiles, CMake, and C. Chris Lamb started to convert the wiki page describing SOURCE_DATE_EPOCH into a Freedesktop-like specification in the hope that it will convince more upstream to adopt it. Package reviews 44 reviews have been removed, 192 added and 77 updated this week. New issues identified this week: locale_dependent_order_in_devlibs_depends, randomness_in_ocaml_startup_files, randomness_in_ocaml_packed_libraries, randomness_in_ocaml_custom_executables, undeterministic_symlinking_by_rdfind, random_build_path_by_golang_compiler, and images_in_pdf_generated_by_latex. 117 new FTBFS bugs have been reported by Chris Lamb, Chris West (Faux), and Niko Tyni. Misc. Some reproducibility issues might face us very late. Chris Lamb noticed that the test suite for python-pykmip was now failing because its test certificates have expired. Let's hope no packages are hiding a certificate valid for 10 years somewhere in their source! Pictures courtesy and copyright of Debian's own paparazzi: Aigars Mahinovs.

Richard Hartmann: Tor-enabled Debian mirror

During Jacob Applebaum's talk at DebConf15, he noted that Debian should TLS-enable all services, especially the mirrors. His reasoning was that when a high-value target downloads a security update for package foo, an adversary knows that they are still using a vulnerable version of foo and try to attack before the security update has been installed. In this specific case, TLS is not of much use though. If the target downloads 4.7 MiB right after a security update with 4.7 MiB has been released, or downloads from security.debian.org, it's still obvious what's happening. Even padding won't help much as the 5 MiB download will also be suspicious. The mere act of downloading anything from the mirrors after an update has been released is reason enough to try an attack. The solution, is, of course, Tor. weasel was nice enough to set up a hidden service on Debian's infrastructure; initally we agreed that he would just give me a VM and I would do the actual work, but he went the full way on his own. Thanks :) This service is not redundant, it uses a key which is stored on the local drive, the .onion will change, and things are expected to break. But at least this service exists now and can be used, tested, and put under some load:
http://vwakviie2ienjx6t.onion/

I couldn't get apt-get to be content with a .onion in /etc/apt/sources.list and Acquire::socks::proxy "socks://127.0.0.1:9050"; in /etc/apt/apt.conf, but the torify wrapper worked like a charm. What follows is, to the best of my knowledge, the first ever download from Debian's "official" Tor-enabled mirror:
~ # apt-get install torsocks
~ # mv /etc/apt/sources.list /etc/apt/sources.list.backup
~ # echo 'deb http://vwakviie2ienjx6t.onion/debian/ unstable main non-free contrib' > /etc/apt/sources.list
~ # torify apt-get update
Get:1 http://vwakviie2ienjx6t.onion unstable InRelease [215 kB]
Get:2 http://vwakviie2ienjx6t.onion unstable/main amd64 Packages [7548 kB]
Get:3 http://vwakviie2ienjx6t.onion unstable/non-free amd64 Packages [91.9 kB]
Get:4 http://vwakviie2ienjx6t.onion unstable/contrib amd64 Packages [58.5 kB]
Get:5 http://vwakviie2ienjx6t.onion unstable/main i386 Packages [7541 kB]
Get:6 http://vwakviie2ienjx6t.onion unstable/non-free i386 Packages [85.4 kB]
Get:7 http://vwakviie2ienjx6t.onion unstable/contrib i386 Packages [58.1 kB]
Get:8 http://vwakviie2ienjx6t.onion unstable/contrib Translation-en [45.7 kB]
Get:9 http://vwakviie2ienjx6t.onion unstable/main Translation-en [5060 kB]
Get:10 http://vwakviie2ienjx6t.onion unstable/non-free Translation-en [80.8 kB]
Fetched 20.8 MB in 2min 0s (172 kB/s)
Reading package lists... Done
~ # torify apt-get install vim
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  vim-common vim-nox vim-runtime vim-tiny
Suggested packages:
  ctags vim-doc vim-scripts cscope indent
The following packages will be upgraded:
  vim vim-common vim-nox vim-runtime vim-tiny
5 upgraded, 0 newly installed, 0 to remove and 661 not upgraded.
Need to get 0 B/7719 kB of archives.
After this operation, 2048 B disk space will be freed.
Do you want to continue? [Y/n] 
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
Reading changelogs... Done
(Reading database ... 316427 files and directories currently installed.)
Preparing to unpack .../vim-nox_2%3a7.4.826-1_amd64.deb ...
Unpacking vim-nox (2:7.4.826-1) over (2:7.4.712-3) ...
Preparing to unpack .../vim_2%3a7.4.826-1_amd64.deb ...
Unpacking vim (2:7.4.826-1) over (2:7.4.712-3) ...
Preparing to unpack .../vim-tiny_2%3a7.4.826-1_amd64.deb ...
Unpacking vim-tiny (2:7.4.826-1) over (2:7.4.712-3) ...
Preparing to unpack .../vim-runtime_2%3a7.4.826-1_all.deb ...
Unpacking vim-runtime (2:7.4.826-1) over (2:7.4.712-3) ...
Preparing to unpack .../vim-common_2%3a7.4.826-1_amd64.deb ...
Unpacking vim-common (2:7.4.826-1) over (2:7.4.712-3) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for mime-support (3.58) ...
Processing triggers for desktop-file-utils (0.22-1) ...
Processing triggers for hicolor-icon-theme (0.13-1) ...
Setting up vim-common (2:7.4.826-1) ...
Setting up vim-runtime (2:7.4.826-1) ...
Processing /usr/share/vim/addons/doc
Setting up vim-nox (2:7.4.826-1) ...
Setting up vim (2:7.4.826-1) ...
Setting up vim-tiny (2:7.4.826-1) ...
~ # 

More services will follow. noodles, weasel, and me agreed that the project as a whole should aim to Tor-enable the complete package lifecycle, package information, and the website. Maybe a more secure install option on the official images which, amongst others, sets up apt, apt-listbugs, dput, reportbug, et al up to use Tor without further configuration could even be a realistic stretch goal.

24 August 2015

Jonathan McDowell: Random post-DebConf 15 thoughts

There are a bunch of things I mean to blog about, but as I have just got fully home from Heidelberg and DebConf15 this afternoon that seems most appropriate to start with. It s a bit of a set of disjoint thoughts, but I figure I should write them down while they re in my head. DebConf is an interesting conference. It s the best opportunity the Debian project has every year to come together and actually spend a decent amount of time with each other. As a result it s a fairly full on experience, with lots of planned talks as a basis and a wide range of technical discussions and general social interaction filling in whatever gaps are available. I always find it a thoroughly enjoyable experience, but equally I m glad to be home and doing delightfully dull things like washing my clothes and buying fresh milk. I have always been of the opinion that the key aspect of DebConf is the face time. It was thus great to see so many people there - we were told several times that this was the largest DebConf so far (~ 570 people IIRC). That s good in the sense that it meant I got to speak to a lot of people (both old friends and new), but does mean that there are various people I know I didn t spend enough, or in some cases any, time with. My apologies, but I think many of us were in the same situation. I don t feel it made the conference any less productive for me - I managed to get a bunch of hacking done, discuss a number of open questions in person with various people and get pulled into various interesting discussions I hadn t expected. In short, a typical DebConf. Also I d like to say that the venue worked out really well. I ll admit I was dubious when I heard it was in a hostel, but it was well located (about a 30 minute walk into town, and a reasonable bus service available from just outside the door), self-contained with decent facilities (I m a big believer in having DebConf talks + accommodation be as close as possible to each other) and the room was much better than expected (well, aside from the snoring but I can t blame the DebConf organisers for that). One of the surprising and interesting things for me that was different from previous DebConfs was the opportunity to have more conversations with a legal leaning. I expect to go to DebConf and do OpenPGP/general crypto related bits. I wasn t expecting affirmation about the things I have learnt on my course over the past year, in terms of feeling that I could use that knowledge in the process of helping Debian. It provided me with some hope that I ll be able to tie my technology and law skills together in a way that I will find suitably entertaining (as did various conversations where people expressed significant interest in the crossover). Next year is in Cape Town, South Africa. It s a long way (though I suppose no worse than Portland and I get to stay in the same time zone), and a quick look at flights indicates they re quite expensive at the moment. The bid presentation did look pretty good though so as soon as the dates are confirmed (I believe this will happen as soon as there are signed contracts in place) I ll take another look at flights. In short, excellent DebConf, thanks to the organisers, lovely to see everyone I managed to speak to, apologies to those of you I didn t manage to speak to. Hopefully see you in Cape Town next year.

Next.

Previous.