Search Results: "Christoph Egger"

27 June 2014

Christoph Egger

I've been working on my backup strategy for the notebook recently. The idea is to have full backups every now month and then incremental backups in between as fine-grained as possible. As it's a mobile device there's no point in time where it is guaranteed to be up, connected and within reach of the backup server. As I'm running Debian GNU/kFreeBSD on it, using ZFS and specifically zfs send comes quite naturally. I'm now generating a new file system snapshot every day (if the notebook happens to be online during that day) using cron. 
@daily zfs snapshot base/root@ date -I 
@daily zfs snapshot base/home@ date -I 
@reboot zfs snapshot base/root@ date -I 
@reboot zfs snapshot base/home@ date -I
When connected to the home network I'm synchronizing off all incrementals that are not yet on the backup server. This is using zfs send together with gpg to encrypt the data and then put it off to some sftp storage. For the first snapshot every month a full backup is created. As there doesn't seem to be a way to merge zfs send streams without importing everything in a zfs pool I create additional incremental streams to the first snapshot of last month so I'm able to delete older full backups and daily snapshots and still keep coarse-gained backups for a longer period of time. 
#!/usr/bin/python
# -*- coding: utf-8 -*-
####################
# Config
SFTP_HOST = 'botero.siccegge.de'
SFTP_DIR  = '/srv/backup/mitoraj'
SFTP_USER = 'root'
ZPOOL     = 'base'
GPGUSER   = '9FED5C6CE206B70A585770CA965522B9D49AE731'
#
####################
import subprocess
import os.path
import sys
import paramiko
term =  
    'green':  "\033[0;32m",
    'red':    "\033[0;31m",
    'yellow': "\033[0;33m",
    'purple': "\033[0;35m",
    'none':   "\033[0m",
     
sftp = None
def print_colored(data, color):
    sys.stdout.write(term[color])
    sys.stdout.write(data)
    sys.stdout.write(term['none'])
    sys.stdout.write('\n')
    sys.stdout.flush()
def postprocess_datasets(datasets):
    devices = set([entry.split('@')[0] for entry in datasets])
    result = dict()
    for device in devices:
        result[device] = sorted([ entry.split('@')[1] for entry in datasets
                                    if entry.startswith(device) ])
    return result
def sftp_connect():
    global sftp
    host_keys = paramiko.util.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
    hostkeytype = host_keys[SFTP_HOST].keys()[0]
    hostkey = host_keys[SFTP_HOST][hostkeytype]
    agent = paramiko.Agent()
    transport = paramiko.Transport((SFTP_HOST, 22))
    transport.connect(hostkey=hostkey)
    for key in agent.get_keys():
        try:
            transport.auth_publickey(SFTP_USER, key)
            break
        except paramiko.SSHException:
            continue
    sftp = paramiko.SFTPClient.from_transport(transport)
    sftp.chdir(SFTP_DIR)
def sftp_send(dataset, reference=None):
    zfscommand = ['sudo', 'zfs', 'send', '%s/%s' % (ZPOOL, dataset)]
    if reference is not None:
        zfscommand = zfscommand + ['-i', reference]
    zfs = subprocess.Popen(zfscommand, stdout=subprocess.PIPE)
    gpgcommand = [ 'gpg', '--batch', '--compress-algo', 'ZLIB',
                   '--sign', '--encrypt', '--recipient', GPGUSER ]
    gpg = subprocess.Popen(gpgcommand, stdout=subprocess.PIPE,
                                       stdin=zfs.stdout,
                                       stderr=subprocess.PIPE)
    gpg.poll()
    if gpg.returncode not in [None, 0]:
        print_colored("Error:\n\n" + gpg.stderr, 'red')
        return
    if reference is None:
        filename = '%s.full.zfs.gpg' % dataset
    else:
        filename = '%s.from.%s.zfs.gpg' % (dataset, reference)
    with sftp.open(filename, 'w') as remotefile:
        sys.stdout.write(term['purple'])
        while True:
            junk = gpg.stdout.read(1024*1024)
            if len(junk) == 0:
                break
            sys.stdout.write('#')
            sys.stdout.flush()
            remotefile.write(junk)
        print_colored(" DONE", 'green')
def syncronize(local_datasets, remote_datasets):
    for device in local_datasets.keys():
        current = ""
        for dataset in local_datasets[device]:
            last = current
            current = dataset
            if device in remote_datasets:
                if dataset in remote_datasets[device]:
                    print_colored("%s@%s -- found on remote server" % (device, dataset), 'yellow')
                    continue
            if last == '':
                print_colored("Initial syncronization for device %s" % device, 'green')
                sftp_send("%s@%s" % (device, dataset))
                lastmonth = dataset
                continue
            if last[:7] == dataset[:7]:
                print_colored("%s@%s -- incremental backup (reference: %s)" %
                              (device, dataset, last), 'green')
                sftp_send("%s@%s" % (device, dataset), last)
            else:
                print_colored("%s@%s -- full backup" % (device, dataset), 'green')
                sftp_send("%s@%s" % (device, dataset))
                print_colored("%s@%s -- doing incremental backup" % (device, dataset), 'green')
                sftp_send("%s@%s" % (device, dataset), lastmonth)
                lastmonth = dataset
def get_remote_datasets():
    datasets = sftp.listdir()
    datasets = filter(lambda x: '@' in x, datasets)
    datasets = [ entry.split('.')[0] for entry in datasets ]
    return postprocess_datasets(datasets)
def get_local_datasets():
    datasets = subprocess.check_output(['sudo', 'zfs', 'list', '-t', 'snapshot', '-H', '-o', 'name'])
    datasets = datasets.strip().split('\n')
    datasets = [ entry[5:] for entry in datasets ]
    return postprocess_datasets(datasets)
def main():
    sftp_connect()
    syncronize(get_local_datasets(), get_remote_datasets())
if __name__ == '__main__':
    main()
Rumors have it, btrfs has gained similar functionality to zfs send so maybe I'll be able to extend that code and use it on my linux nodes some future day (after migrating to btrfs there for a start).

Christoph Egger

I've written a small dmenu-based script which allows to select passwords from one's pass password manager and have it xdotool typed in. This should completely bypass the clipboard (which is distrusted by people for a reason). As I've been asked about the script a few times in the past here it is. Feel free to copy and any suggestions welcome. 
#!/bin/bash
shopt -s nullglob globstar
list_passwords()  
	basedir=~/.password-store/
	passwords=( ~/.password-store/**/*.gpg )
	for password in "$ passwords[@] "
	do
		filename="$ password#$basedir "
		filename="$ filename%.gpg "
		echo $filename
	done
 
xdotool_command()  
	echo -n "type "
	pass "$1"
 
selected_password="$(list_passwords 2>/dev/null  dmenu)"
echo $selected_password
if [ -n "$selected_password" ]
then
	xdotool_command "$selected_password"   xdotool -
fi

18 February 2014

Christoph Egger

Writing this because there seems to be no correct documentation on the relevant google websites and it turns out to be non-trivial. Our goal here is to unsubscribe from a ordinary google group. Mails from the google group contain the quoted footer: 
-- 
You received this message because you are subscribed to the Google
Groups "FOO" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to FOO+unsubscribe@googlegroups.com.
Visit this group at http://groups.google.com/group/FOO
For more options, visit https://groups.google.com/groups/opt_out.
Seems easy enough, so let's send a Mail to this FOO+unsubscribe address. Back comes a E-Mail: 
From: FOO <FOO+unsubconfirm@googlegroups.com>
Subject: Unsubscribe request for FOO [ EJzZjpgFhDHd9seTdRA0 ]
To: Christoph Egger <christoph@example.com>
Date: Tue, 18 Feb 2014 18:55:24 +0000 (38 minutes, 53 seconds ago)
 [Leave This Group]
Visit Go 
[Start] your own group, [visit] the help center, or [report]
abuse.
So click on the [Leave This Group] link and be done? Unfortunately not. Looking at the link you notice it's called http://groups.google.com/group/FOO/subscribe -- no token and "subscribe"? I actually want to unsubscribe! And indeed, clicking gets an Interface that offers to "Enter the email address to subscribe:" + Captcha. And whatever it does, it -- of course -- doesn't unsubscribe. (My guess is, it would actually work if you had a real google account associated with that email address and were logged in to that account but there's no way of verifying this as already the first condition is false in this case) Now if you disable HTML completely for the email, a totally different content emerges: 
Hello christoph@example.com,
We have received your request to unsubscribe from FOO. In order for us to complete the request, please reply to this email or visit the following confirmation URL:
http://groups.google.com/group/FOO/subscribe
If you have questions related to this or any other Google Group, visit the Help Center at http://groups.google.com/support/.
Thanks,
Google Groups
Still the non-functional link, however it also mentions a different solution: "please reply to this email" which was not present in the HTML mail at all. And it works.

20 December 2013

Christoph Egger

Greetings from the FAU Security Team (FAUST), the Uni Erlangen CTF group. We were participating in the RuCTFe competition and made it to 4th place. Following is my write-up on the nsaless service, the main crypto challenge in the competition. nsaless is a nodejs webservice providing a short message service. People can post messages and their followers receive the message encrypted to their individual RSA key.

About the gameserver protocol
The gameserver created groups of 8 users on the service 7 were just following the first user (and authorized by the first user to do so) while the first user sent a tweet containing the flag. The service used 512bit RSA with 7 as public exponent. While RSA512 is certainly weak, it's strong enough to make it unfeasible to break directly.

Attacking RSA
There are some known attacks against RSA with small exponents if no proper padding is done. The most straightforward version just takes the e-th root of the cipher-text and, if the clear message was small enough, outputs that root as plain-text. As the flag was long enough to make this attack impossible, we need a somewhat improved Attack.

H stad's Broadcast Attack
Reminder:
  • In RSA, given a plain-text A, the sender computes A mod N to build the cipher-text B.
  • Given simultaneous congruences we can efficiently compute a x such that x satisfies all congruences using the Chinese remainder theorem.
For NSAless we actually get several such B for different N (each belonging to different users receiving the tweet because they follow the poster). This effectively means we get A in mod N for different N. Using the Chinese remainder theorem we can now compute a x A mod N . If we use at least e different B for this we are guaranteed that x actually equals A (in ): A needs to be smaller than N for all N used (otherwise we lose information during encryption), therefore A needs to be smaller than N . Computing now the e-th root of x we get the plain-text A the flag.

Fix
Fixing your service is easy enough, just increase e to an suitable number > 8. At the end of the contest 5 Teams had fixed this vulnerability by either using 17 or 65537.

EXPLOIT
The basic exploit is shown below. Unfortunately it needs to retrieve all tweets for all users the compute the flags which just takes too long to be feasible (at least at the end of the competition where tons of users already existed) so you would need some caching to make it actually work. Would have been a great idea to have users expire after an hour or two in the service! 
#!/usr/bin/python
import httplib
import urllib
import re
import json
import pprint
import gmpy
import sys
userparse_re = re.compile('<a [^>]*>([^<]*)</a></div>\s*<div>([^<]*)</div>')
tweetparse_re = re.compile("<div id='last_tweet'>([0-9]+)</div>")
followingparse_re = re.compile('<div><a href="/[0-9]+">([0-9]+)</a></div>')
def my_parse_number(number):
    string = "%x" % number
    if len(string) != 64:
        return ""
    erg = []
    while string != '':
        erg = erg + [chr(int(string[:2], 16))]
        string = string[2:]
    return ''.join(erg)
def extended_gcd(a, b):
    x,y = 0, 1
    lastx, lasty = 1, 0
    while b:
        a, (q, b) = b, divmod(a,b)
        x, lastx = lastx-q*x, x
        y, lasty = lasty-q*y, y
    return (lastx, lasty, a)
def chinese_remainder_theorem(items):
  N = 1
  for a, n in items:
    N *= n
  result = 0
  for a, n in items:
    m = N/n
    r, s, d = extended_gcd(n, m)
    if d != 1:
      raise "Input not pairwise co-prime"
    result += a*s*m
  return result % N, N
def get_tweet(uid):
    try:
        conn = httplib.HTTPConnection("%s:48879" % sys.argv[1], timeout=60)
        conn.request("GET", "/%s" % uid)
        r1 = conn.getresponse()
        data = r1.read()
        tweet = re.findall(tweetparse_re, data)
        if len(tweet) != 1:
            return None
        followers = re.findall(followingparse_re, data)
        return tweet[0], followers
    except:
        return None
def get_users():
    conn = httplib.HTTPConnection("%s:48879" % sys.argv[1], timeout=60)
    conn.request("GET", "/users")
    r1 = conn.getresponse()
    data1 = r1.read(1024 * 1024)
    data = dict()
    for i in re.findall(userparse_re, data1)[:100]:
        userinfo = get_tweet(i[0])
        if userinfo != None:
            data[i[0]] = (json.loads(i[1].replace('&quot;', '"'))['n'], userinfo)
    return data
users = get_users()
allusers = users.keys()
masters = [ user for user in allusers if len(users[user][1][1]) > 0 ]
for test in masters:
    try:
        followers = users[test][1][1]
        data = []
        for fol in followers:
            n = int(users[fol][0])
            tweet = int(users[fol][1][0])
            data = data + [(tweet, n)]
        x, n = chinese_remainder_theorem(data)
        realnum = gmpy.mpz(x).root(7)[0].digits()
        print my_parse_number(int(realnum))
    except:
        pass

20 April 2013

Ulrich Dangel: Analyzing rc bug messages

Michael Stapelberg recently posted a blog post about looking into the number of Debian Developers actively working on RC bugs for the upcoming wheezy release. In this blog post I analyze the data shared by Michael and provide the R commands used to generate the plots & findings. If you are interested into looking into the data yourself, but don t like R, I suggest using ipython notebook + numpy instead.

Analysis After parsing the data file we typically want to get an understanding of the data, by using summary(bugs) we get the minimum(1), median(5), mean(15.4), max(716) and quantiles of the data. This shows that the number of messages is wide-spread and a few people contribute a lot. To visualize the dispersion of the data we can create a box plot showing the range of messages: boxplot As the first and third quantile are close together we can assume that the majority of the work is done by a few, especially since the second quantile is 5. This is supported by the histogram below, where the x axis is the number of recorded messages and y is the number of developers. histogram

Top 10 contributors The TOP 10 contributors, according to the dataset, are:
  1. Lucas Nussbaum - 716 messages
  2. Gregor Herrmann - 270 messages
  3. Jakub Wilk - 270 messages
  4. Andreas Beckmann - 225 messages
  5. Julien Cristau - 205 messages
  6. Cyril Brulebois - 169 messages
  7. Moritz Muehlenhoff - 162 messages
  8. Michael Biebl - 159 messages
  9. Salvatore Bonaccorso - 158 messages
  10. Christoph Egger - 142 messages

r commands These are the commands used to generate the plots and information in this plot: 
bugs <- read.csv("by-msg.csv")
summary(bugs)
boxplot(bugs$rcbugmsg, log='y', range=0, ylab="# bugs")
quantile(bugs$rcbugmsg)
0%  25%  50%  75% 100%
1    2    5   12  716
# create histogram
llibrary('ggplot2')
ggplot(bugs, aes(x=rcbugmsg)) + geom_histogram(binwidth=.5, colour="black", fill="black") + scale_x_sqrt()
top10 <- tail(bugs[order(bugs$rcbugmsg),], 10)
top10

Ulrich Dangel: Analyzing rc bug messages

Michael Stapelberg recently posted a blog post about looking into the number of Debian Developers actively working on RC bugs for the upcoming wheezy release. In this blog post I analyze the data shared by Michael and provide the R commands used to generate the plots & findings. If you are interested into looking into the data yourself, but don t like R, I suggest using ipython notebook + numpy instead.

Analysis After parsing the data file we typically want to get an understanding of the data, by using summary(bugs) we get the minimum(1), median(5), mean(15.4), max(716) and quantiles of the data. This shows that the number of messages is wide-spread and a few people contribute a lot. To visualize the dispersion of the data we can create a box plot showing the range of messages: boxplot As the first and third quantile are close together we can assume that the majority of the work is done by a few, especially since the second quantile is 5. This is supported by the histogram below, where the x axis is the number of recorded messages and y is the number of developers. histogram

Top 10 contributors The TOP 10 contributors, according to the dataset, are:
  1. Lucas Nussbaum - 716 messages
  2. Gregor Herrmann - 270 messages
  3. Jakub Wilk - 270 messages
  4. Andreas Beckmann - 225 messages
  5. Julien Cristau - 205 messages
  6. Cyril Brulebois - 169 messages
  7. Moritz Muehlenhoff - 162 messages
  8. Michael Biebl - 159 messages
  9. Salvatore Bonaccorso - 158 messages
  10. Christoph Egger - 142 messages

r commands These are the commands used to generate the plots and information in this plot: 
bugs <- read.csv("by-msg.csv")
summary(bugs)
boxplot(bugs$rcbugmsg, log='y', range=0, ylab="# bugs")
quantile(bugs$rcbugmsg)
0%  25%  50%  75% 100%
1    2    5   12  716
# create histogram
llibrary('ggplot2')
ggplot(bugs, aes(x=rcbugmsg)) + geom_histogram(binwidth=.5, colour="black", fill="black") + scale_x_sqrt()
top10 <- tail(bugs[order(bugs$rcbugmsg),], 10)
top10

3 December 2012

Christoph Egger: Generating .wot files now

Generating .wot files now

20 October 2012

Christoph Egger: Android

Hardware To make things clear: I'm having a Android 4.0.$recent tablet with considerably more horse-power than my Nokia n900 smartphone so don't tell me this is due to under-powered hardware the android is 3 years newer both in hardware and software.

Background Tasks Being somewhere with my Android Tablet. Network is kind of crappy and this site takes minutes again to load. So the most natural thing to do would be doing something else while the site continues loading in the background. This works really well on the n900. It might work with android. But of course when you switch to another Program the browser might also be shut down while you're doing something else and randomly when you switch back to your browser, not only the site hasn't loaded but the browser also forgot where you were heading. Now if you followed e.g. a link in a email you might have closed the mail program long ago (or the mail program has decided to stop) and you have to find the link again, wait again for the site to load. And remember not to background the browser or you might have to start over again. With the n900 Maemo smartphone I was able to load several pages in the background with whatever application in the foreground (like playing tuxracer) so don't tell me android has to do this to give enough power to the foreground process. If a Meamo device can load 5 pages in the background while a OpenGL game is running in the foreground there is no reason Android, with more CPU and RAM, can't load a single page in the background while I check email.

Software installation Can you imagine a system where you are unable to install software from your standard repository without registering an account first? Like after nearly two decades of Linux distributions? Maemo had this for mobile devices more than five years ago. Plus, on Maemo you'll easily find tons of good, free (as in freedom) and banner-add-free software try this on androids "Play Store".

19 March 2012

Jan Wagner: Chemnitzer Linuxtage 2012

As announced 3 weeks ago, the Debian project was present at Chemnitzer Linuxtage. Several talks and workshops where held by people related to the Debian project. At the booth we had talks and discussions with exhibitors and visitors, unfortunately I didn t had much time to visit more than small parts of two lectures. Unfortunately (for the visitors) we didn t had any merchandising on board, while we received several requests. On Sunday Axel surprised us with some leftovers from fosdem of debian.ch merch. At the booth we had a demo machine running Babelbox and xpenguins, which attracted visitors very well. Booth Babelbox We received also more than one Just thank you by satisfied users. :) Four different talks and one workshop were held by Debian people, but they were not specific to the Debian. The workshop was about OpenStreetMap, lectures was about commandline helpers, grep everything, quality analyzing and team management in opensource projects and Conkeror and other keyboard based webbrowsers. Many thanks to Jan Dittberner, Andreas Tille, Christian Hoffmann, Florian Baumann, Christoph Egger, Axel Beckert, Adam Schmalhofer, Markus Schnalke, Sebastian Harl and Patrick Matth i for running the booth, answering a wide range of questions or just chatting with visitors . A special thank to TMT GmbH & Co. KG for providing the complete equipment and sponsoring it s transportation. At the end we have to send a big thank to the organizing team of the Chemnitzer Linuxtage. It was fun and a pleasure to find new friends and meet old ones of the Free Software community. A small sidenote was anybody aware that OpenSuSE Package search is using screenshots.debian.net?

15 January 2012

Christoph Egger: PHP love

Migrating a mediawiki instance from the old server to a new box. Of course it does not work (returns an empty 500 Error page). Of course there is no entry in error.log. Of course there is no obvious match of verbose/debug in a grep over the config files. Lovin' it

4 September 2011

Christoph Egger: A week of Debian GNU/kFreeBSD

While other people are squashing RC bugs I was using this week to fix (or investigationg) some more kFreeBSD issues -- mostly looking at failed build logs and trying to fix the problems and after some nice fish for dinner writing things up.

29 June 2011

Christoph Egger: Debconf11

debconf banner I'm coming as well! Really looking forward to meet the people from Debconf9 again. Also people from the Games Team and the buildd and kfreebsd folks. And ideally there will be some more people interested in (Common) Lisp as well, we'll see

27 June 2011

Christoph Egger: Unknown Horizons

What is Unknown Horizons? Unknown Horizons is a Free Strategy game I have written about from time to time about and do from time to time contribute. I also build Packages for Debianoids. Unknown Horizons is not a new thing, it has been there for some years now. It currently gets boosted by 3 very active GSoC Students and evolving steadily. Preparing for 2011.2 Unknown Horizons is Preparing for another Release, 2011.2. As part of the effort I have updated the official Unknown Horizons Debian and Ubuntu weekly archives with new rc builds to get some more testing before the release happens. Please test! Why isn't it in Debian main yet? Unknown Horizons hasn't been uploaded to Debian yet. This is mostly due to 2 factors: The game engine Unknown Horizons build upon, FIFE isn't in Debian, evolving fast and not providing much of a stable interface and therefor Unknown Horizons has traditionally had a strict dependency to some VCS state of approximately the same age. Not good for a Debian package obviously. And second, Unknown Horizons got rid of some content that was not free (as in "main") just recently. Personally I'm hoping we can have some packages soon as the freeness stuff is sorted out now and I haven't seen any troubling breakages with regard to the engine recently.

15 June 2011

Christoph Egger: Feeling young

Looking around old files I have put online ages ago I stumbled upon a Unknown Horizons Code Swarm Video I have created back in September 2009. Feeling more than a bit sad this piece of software died soon after being released. Searching the web for "Code Swarm" still finds lots of old Videos created back then.

12 June 2011

Christoph Egger: Marking all closed bug reports &quot;read&quot; in a Maildir

#!/usr/bin/python
from btsutils.debbugs import debbugs
import mailbox
import re
import sys
mailbox =  mailbox.Maildir(sys.argv[1], factory=False)
bts = debbugs()
for key in mailbox.keys():
    message = mailbox[key]
    if not 'S' in message.get_flags():
        if message['X-Debian-PR-Message']:
            try:
                bugnr = message['X-Debian-PR-Message'].split()[1]
            except IndexError:
                continue
        else:
            test = re.search('Bug#(\d 6 )', message['Subject'])
            if test:
                bugnr = test.group(0)[4:]
            else:
                continue
        try:
            bug = bts.get(bugnr)
        except AttributeError:
            print bugnr
            continue
        if bug.getStatus() == u'done':
            message.set_flags(message.get_flags() + 'S')
            mailbox[key] = message
mailbox.flush()
Run it like python cleanbugsmail.py ~/Maildir/.debian.bugs. Anyone aware of a better solution?

30 May 2011

Christoph Egger: Maintaining kFreeBSD buildds since one month

At April 30, I took over maintenance of of Debian's kFreeBSD autobuilders. Means getting something like 4,5k e-Mails this month (gladly no need to sign all those 4k successful builds any more!), filling nearly 30 RC Bugs (quite a lot of which got fixed just within hours after filling, wow!), investigating some of the more strange build failures and such stuff. In general it turned out to be quite some fun. Quite interesting which libraries turn out to be rather central to the Archive. I wouldn't have guessed that a uninstallable libmtp would prevent a reasonable fraction of the builds to fail -- including packages like subversion. Packages builds failing because the disk space is exhausted may be something most of us have already witnessed, especially those here that use one of these small notebook hard drives. Build failures caused by a lack of RAM might certainly be imaginable as well, especially on highly parallel builds. But have you ever seen gcc fail because the virtual address space was exhausted on 32 bit architectures? Also there's a interesting lot of packages with misspelled build dependencies which sbuild can't find and therefore can't build the package. Maybe having a lintian check for some of these problems would be a good idea? I'm also regularly seeing build failures that look easy enough to fix -- like some glob in a *.install for some python package matching lib.linux*. I try to fix some of these as I see them but my time is unfortunately limited as well. Someone interested in quick&easy noticed about these kind of issues? I could put URLs to build-logs on identi.ca or somewhere on IRC. There are also some really strange failures like llvm, which builds flawlessly on my local kFreeBSD boxes all the time, inside and outside schroot but hangs all the time in the same testcase when building on any of the autobuilders (any hints welcome!) or perl failing on kfreebsd-amd64 selectively but all the time.

17 May 2011

Christoph Egger: Quick notes about PostgreSQL

Imagine you have a old postgresql database. Further imagine it has it's encoding set to something like LATIN-1 but some PHP web application has put in UTF-8 strings. Now what would you do if you have some python application actually respecting the encoding and recoding the db content from latin-1 to UTF-8 giving you garbage. Seems you can easily trick postgresql to now believe it is UTF-8: 
UPDATE pg_database SET encoding = 6 WHERE datname = 'foo';
For a summary of these magic numbers the PostgreSQL Manual is usefull.

13 May 2011

Christoph Egger: Thouhts on secure software archives

From the java point of view Recently I had to get some Scala Tool working correctly. Unfortunately there are basically no packages in the Debian Archive at all so I had to use maven to install these (or download + install manually). Being a highly paranoid person downloading and executing code from the internet without any cryptographic verification at all one after the other practically drove me nuts. Looking a bit deeper I noticed that some of the software in maven's repository have some signatures next to them -- signed by the author or release manager of this specific project. Why secure sources matters With my experience in mind I got some Input from other people. One of the things I was told is that some scala tools just aren't security critical -- they're only installed and used as the current user. In my opinion this is, for my desktop system, totally wrong. The important things on my private Computers are my GPG and SSH keys as well as my private data. For messing with these no super user access is needed at all. Comparing to the Common Lisp situation Being a Common Lisp fan of course I noticed basically the same problem for installing Common Lisp libraries. Here the situation in Debian is quite a bit better -- and I'm working in the pkg-common-lisp Team to improve this even more. Common Lisp has some maven-alike tool for downloading and installing dependency trees called quicklisp -- without any cryptographic verification as well. However there's light at the end of this tunnel: There are plans to add GPG verification of the package lists really soon. Comparing the maven and the quicklisp model So there are basically two different approaches to be seen here. In maven the software author confirms with his signature the integrity of his software while in quicklisp the distributor confirms all users get the same software that he downloaded. Now the quicklisp author can't and won't check all the software that is downloadable using quicklisp. This won't be doable anyway as there's way to much software or a single person to check. Now in some kind of perfect World the maven way would be vastly superior as there's a End-To-End verification and verification of the full way the software takes. However there's a big problem: I don't know any of these Authors personally and there's no reason I should just trust any of them. Now comparing this to the Distribution / quicklisp model. Here I would just have to trust one person or group -- here the quicklisp team -- to benefit from the crypto which might be possible based on karma inside the using community. However here I don't gain the possibility that the software is integer. However idealized if some of these pieces of software was forged between upstream and the quicklisp team and attacker would also intercept me downloading the software from the same address so I get the source from upstream matching the checksum from quicklisp -- assuming the quicklisp team does indeed know the correct website. Additionally I get the confirmation that all other quicklisp users get the same source (if the quicklisp guys are fine of course) so no-one inside the community complaining is a good indication the software is fine. For this to work there's of course a relevant user-base of the distributor (quicklisp) necessary. Relevance for Debian So how do conventional Linux Distributions like Debian fit in here. Ideally we would have maintainers understanding and checking the software and confirming the integrity using their private key or at least know their upstreams and having at least a secured way getting the software from upstream and a trust relationship with them. Of course that's just illusionary thinking of complex and important software (think libreoffice, gcc or firefox for example). Maintainers won't fully understand a lot simpler pieces of software. And loads of upstream projects don't provide a verified way of getting the correct source code though that's a bit better on the real high-impact projects where checksums signed by the Release Manager are more common than in small projects. A misguided thought at the end As I'm a heavy emacs user I like to have snapshots from current emacs development available. Fortunately binary packages with this are available from a Debian guy I tend to trust who is also involved upstream so adding the key from his repository to the keyring apt trusts. Now my first thoughts were along the lines "It would be really nice if I could pin that key to only the emacs snapshot packages" so this guy can't just put libc packages in his repository and my apt would trust them. Now thinking of it again a bogus upload of the emacs snapshot package could just as well put some binary or library on the system at some place in front of the real on in the system path which would be rather similar bad. b

20 February 2011

Christoph Egger: CSSH but without X

There are many ways to run some commands simultaneously on multiple hosts like cssh or dsh. They come handy for example when you are installing software updates on a set of hosts. dsh is a rather simple comandline tool allowing to execute a command over ssh on multiple hosts. However it doesn't allow any interactive input -- so you can't look at the potentially upgrading packages and press y to accept and you can't go through debconf promts or similar. This is solved by cssh which opens a XTerm for every host and a input area that is broadcastet to all of them. this is working really well -- you can execute your update on all hosts and still do individual adjustments just as needed: switch focus from the broadcasted input to one of the terminal windows and anything you type just goes there. Now cssh has a big disadvantage: it requires a running X server (and doesn't do too well with a fullscreen windowmanager). Requiring X is quite a blocker if you need to run that ssh multiplexer on a remote host, for example if the firewalling doesn't allow direct connections. Fortunately you can make tmux behave as we want -- in a simple terminal: First you need a script spawning the ssh sessions in separate tmux panes and direct input to all of them -- here called ssh-everywhere.sh (you could also write a tmux config I guess): 
#/bin/sh
# ssh-everywhere.sh
for i in $HOSTS
do
  tmux splitw "ssh $i"
  tmux select-layout tiled
done
tmux set-window-option synchronize-panes on
Now start the whole thing: 
tmux new 'exec sh ssh-everywhere.sh'
And be done. Update If you want to type in just one pane (on one host) you can do that as well: C-b : set-window-option synchronize-panes off and moving to the right pane (C-b + Arrow keys)

29 December 2010

Christoph Egger: Debian GNU/kFreeBSD

So when I was travveling to my parent's for christmas it looked like I'd have limited computer access. My Netbook is quite OK for reading mail but not really useable for any real hacking. And my trusty Thinkpad (Z61m) was oopsing when X was running so not much useable either. But as some Live CDs placed here were working well I decided that this would be fixed by an reinstall. And as I was reinstalling anyway I decided I could just choose kfreebsd-amd64. Turned out to be a quite entertaining decision with lots of stuff to hack away with wireless Bad news: there's no wireless support on Debian GNU/kFreeBSD at the moment. This problem is tracked as Bug #601803 so for wireless internet you will need a (plain) Freebsd chroot. Haven't tried this myself yet -- busy figuring other stuff out. SBCL Having a FreeBSD chroot I decided to give SBCL on GNU/kFreeBSD another try after having failed to get it working in a VM some time ago. With quite some help on SBCL's IRC channel I managed to build a patch that enables building (you need to force a :os-provides-dlopen to the feature list additionally). There's currently no multi-threading working so I hae a project for the rest of the hoidays (well lots of other stuff to do as well ;)) Audio Some more user-related stuff now. As it is this time of the year I wanted to listen to some 27c3 streams so I needed working audio. However there's no OSS device available. Turned out you just need to kldload the right module (here snd_hda) to get sound working. Volume was rather low although hardware controls of the soundcard where at max. As that's all OSS there's no point looking for alsamixer. Turns out aumix can do that here. IPv6 aiccu stuff Installing aiccu, copying the config in and starting did not work out as well. I already tried to do that from within the FreeBSD chroot already (which doesn't work for some reason) until I discovered just loading the if_tun kernel module solves the aiccu on Debian issue quite well. To get a default route up the last step was finding /lib/freebsd/route again -- /sbin/route is a wrapper around that abstracting differences in BSD route but not supporting IPv6.

Next.

Previous.