Search Results: "tfheen"

25 March 2008

Tollef Fog Heen: pkg-config, sonames and Requires.private

This post is both an attempt at replying to a bug against telepathy-glib, but also an attempt at explaining what Requires.private do (and don't). I am using Evolution as my example here, not to pick on Evolution or its authors in any way, but because it's a convenient example. Currently, on Ubuntu Hardy, evolution links against 75 different libraries. Amongst those, we find libz.so.1, libXinerama.so.1 and many more. I'll go out on a limb here and claim that Evolution does not call any of the functions in libXinerama directly. Let that be the assumption from here on. An obvious question then is, why does evolution link against libXinerama.so.1 if it doesn't use it? To answer that question, we need to go back in time to before we had dynamic linking. If you wanted to build a binary like evolution you had to have 75 -l statements when you linked and you ended up with the whole code for Xinerama embedded in your email and calendar client. For various reasons, we stopped doing that and switched to dynamic linking where the evolution binary just contains a reference to libXinerama. At some point we also grew the ability for libraries to contain those references to other libraries, so you don't have to hunt down all the dependencies of libfoo when you are linking with it. We also got tools such as libtool which try to abstract away a lot of the problems of building on older platforms which don't support inter-library dependencies. Now, since evolution still doesn't use anything directly in libXinerama.so.1 but just uses a library which in turn links against libXinerama.so.1, it shouldn't be linking against it. Then why is it linked with it? Again, we need to look back at history, and for this part I am at least partially responsible. pkg-config was originally written as a replacement for gnome-config and various other -config utilities. Lots of libraries and applications now ship .pc files and we have a standardised interface for querying those files. One of the problems the original authors of pkg-config faced was the problem of dependencies. They added dependencies so the authors of gst-python-0.10 could say "We need pygtk-2.0 too" and so the compilation flags needed for gst-python-0.10 would also include those for pygtk-2.0. Note that I'm using "compilation flags" loosely here, I am not just talking about CFLAGS. This did not fix the problem of inflated dependencies. Not at all. I talked with some of the Debian Release Managers back in 2004/2005 and we worked out a solution which should help us have correct, uninflated dependencies since the then-current way of handling dependencies caused big problems for migrations of packages between unstable and testing. The plan was to introduce a new field, Requires.private which would not show up unless you passed --static to the pkg-config invocation (since you need all libraries if you are linking statically). This definition of Requires.private was mostly useless since GNOME and GTK+ have a habit of including each other's headers. To make a long story short, I changed the semantics so the Cflags field from private dependencies were included even when not linking statically. A problem which pkg-config does nothing to guard against in this case is if you have libfoo.so.2 linking against libbar.so.1 and libfoo.so.2 exports some of libbar's types in its ABI (and not just as pointers, but actual structs and such). If libbar's soname is then bumped to libbar.so.2 and libfoo is rebuilt, libfoo's ABI has changed without a soname bump. This is bad and will cause problems. If your application is linked against both libfoo.so.2 and libbar.so.1, you'll still get problems since libfoo.so.2 then suddenly pulls in symbols from libbar.so.2. If you used symbol versioning, you would at least not get symbol conflicts and your application would continue to work, but you would have a spurious dependency and the package containing libbar.so.1 would be kept around until your application was recompiled. With this background, you might ask the question why we still have Requires since it is seemingly useless. For C, it is useless in all but the most special cases, just use Requires.private instead (and its sibling Libs.private). Other languages have different semantics. Some people use .pc files for other purposes such as gnome-screensaver having variables defining where themes and screensavers go. Hopefully this blog post has explained a bit about why we have Requires.private and what the difference between this and their regular counterpart is. If there's anything unclear, please do not hesitate to contact me.

9 January 2008

Tollef Fog Heen: Choosing a nonce in CTR mode

I am currently working on implementing a cryptographic file system using FUSE. It is different from EncFS and similar in that it just mirrors a normal directory tree, but encrypts the contents of the files as they are read or decrypted as they are written. My use case is backups. I have some machines where I and only I have access, machines which may contain proprietary information, personal emails and so on. Of course, I want backups of those, so when the hard drives stop working, I don't lose any data. The machine(s) I am backing up to, however are not always machines where I trust all the people with physical access to not make a copy of my data. In addition, I don't want broken hard drives returned under warranty to contain unencrypted data. This use case is the reason for why I'm encrypting on read rather than on write. I have chosen to use CTR (counter) mode together with AES which should give acceptable security. One of the requirements CTR needs to work well is a nonce, typically 64 bits (for 128 bit AES) which must not ever be used twice. If you use it twice, you leak information about your plaintext, which is, for obvious reasons, bad. My current design headache is how to choose a good nonce. Ideally, I believe it should be persistent for each version of the file and unique per file. Using the inode number takes up 64 bits (on AMD64 at any take, or when using -D_FILE_OFFSET_BITS=64 on 32 bit platforms). So while this gives me the latter, it doesn't give me the former at all. I am wondering if I should use the inode number modulo 2^32 (effectively choosing the lower 32 bits of the inode number) and then something which is fairly sure to never be the same, such as mtime (or at least the lower 32 bits of it, when time_t becomes 64 bit). The reason for not just choosing a completely random value is I don't want a command like diff file1 file1 to claim there are differences in the file. My hope was I'd get a great idea on how to solve the problem as part of writing it down. Alas, that hasn't happened, so if you happen to come across a great solution (or a reason to avoid a particular choice), feel free to email me

10 October 2006

Tollef Fog Heen: Contentless ping annoying

People tend to just ping me on IRC, which is annoying and useless. If people say something like tfheen: I have this problem with blah. Do you know a workaround? I can just respond later when I'm awake or around. Instead, people are going: tfheen: ping? and I pong five hours later and they're not around. Annoying for all parts. To counter this, I have now written a small irssi script which responds to contentless pings with "You sent me a contentless ping. This is a contentless pong. Please provide a bit of information about what you want and I'll respond when I am around." The script is available.

23 September 2006

Tollef Fog Heen: The Top Ten Unix Shell Commands

A bit surprising, really.
: tfheen@thosu ~ > history 1 awk ' print $2 ' awk 'BEGIN  FS=" "   print $1 ' sort uniq -c   sort -nr  head -n 10
  21477 ls
  18170 cd
  10640 ssh
   9257 sudo
   5559 less
   3015 grep
   2407 bzr
   2101 ps
   1980 man
   1980 debuild
For those of you wondering why there is no editor on the list; I use emacs and it's in the 11th place (with 1903 runs). I tend to let it run for a while and open more than one file, so it doesn't get that high on the list. I also cd and ls a lot.

19 September 2006

Tollef Fog Heen: Formulating iCalendar in SQL

I am currently working on something which hopefully end up being a good calendar server. The goal is to make it easy to add new frontends such as a CalDAV frontend, a Web frontend, etc. So far, I am just working on getting the data model right. The iCalendar RFC has a data model which I am trying to formulate into SQL, something which is ending up being quite hard. Shannon Clark blogs about why calendars are hard, and all of those issues are issues I am running up against when trying to make the data model. The main problem seems to be related to time zones, more closely: "How do you store timezone information in your database". The easy (but wrong) solution is to just normalise everything to UTC. This will not handle the case of events crossing timezone changes, such as to or from daylight savings. What I am currently considering is making sure all data in the database is stored in UTC, but also store the original time zone information. While a bit more complex, this allows applications to get back the same timezone they put in (I am not convinced Evolution will be happy if the timezones are renamed, for instance) and it allows my application to generate recurring events properly. My main issue with this is having the application responsible for making sure the data in the database makes sense, but without putting half my app in PL/perl, I can not see a way around that. Oh, and if anybody has great ideas on how to represent timezones in a relational database (postgres), please do mail me or grab me on IRC or something similar.

19 March 2006

Clint Adams: This report is flawed, but it sure is fun

91D63469DFdnusinow1243
63DEB0EC31eloy
55A965818Fvela1243
4658510B5Amyon2143
399B7C328Dluk31-2
391880283Canibal2134
370FE53DD9opal4213
322B0920C0lool1342
29788A3F4Cjoeyh
270F932C9Cdoko
258768B1D2sjoerd
23F1BCDB73aurel3213-2
19E02FEF11jordens1243
18AB963370schizo1243
186E74A7D1jdassen(Ks)1243
1868FD549Ftbm3142
186783ED5Efpeters1--2
1791B0D3B7edd-213
16E07F1CF9rousseau321-
16248AEB73rene1243
158E635A5Erafl
14C0143D2Dbubulle4123
13D87C6781krooger(P)4213
13A436AD25jfs(P)
133D08B612msp
131E880A84fjp4213
130F7A8D01nobse
12F1968D1Bdecklin1234
12E7075A54mhatta
12D75F8533joss1342
12BF24424Csrivasta1342
12B8C1FA69sto
127F961564kobold
122A30D729pere4213
1216D970C6eric12--
115E0577F2mpitt
11307D56EDnoel3241
112BE16D01moray1342
10BC7D020Aformorer-1--
10A7D91602apollock4213
10A51A4FDDgcs
10917A225Ejordi
104B729625pvaneynd3123
10497A176Dloic
962F1A57Fpa3aba
954FD2A58glandium1342
94A5D72FErafael
913FEFC40fenio-1--
90AFC7476rra1243
890267086duck31-2
886A118E6ch321-
8801EA932joey1243
87F4E0E11waldi-123
8514B3E7Cflorian21--
841954920fs12--
82A385C57mckinstry21-3
825BFB848rleigh1243
7BC70A6FFpape1---
7B70E403Bari1243
78E2D213Ajochen(Ks)
785FEC17Fkilian
784FB46D6lwall1342
7800969EFsmimram-1--
779CC6586haas
75BFA90ECkohda
752B7487Esesse2341
729499F61sho1342
71E161AFBbarbier12--
6FC05DA69wildfire(P)
6EEB6B4C2avdyk-12-
6EDF008C5blade1243
6E25F2102mejo1342
6D1C41882adeodato(Ks)3142
6D0B433DFross12-3
6B0EBC777piman1233
69D309C3Brobert4213
6882A6C4Bkov
66BBA3C84zugschlus4213
65662C734mvo
6554FB4C6petere-1-2
637155778stratus
62D9ACC8Elars1243
62809E61Ajosem
62252FA1Afrank2143
61CF2D62Amicah
610FA4CD1cjwatson2143
5EE6DC66Ajaldhar2143
5EA59038Esgran4123
5E1EE3FB1md4312
5E0B8B2DEjaybonci
5C9A5B54Esesse(Ps,Gs) 2341
5C4CF8EC3twerner
5C2FEE5CDacid213-
5C09FD35Atille
5C03C56DFrfrancoise---1
5B7CDA2DCxam213-
5A20EBC50cavok4214
5808D0FD0don1342
5797EBFABenrico1243
55230514Asjackman
549A5F855otavio-123
53DC29B41pdm
529982E5Avorlon1243
52763483Bmkoch213-
521DB31C5smr2143
51BF8DE0Fstigge312-
512CADFA5csmall3214
50A0AC927lamont
4F2CF01A8bdale
4F095E5E4mnencia
4E9F2C747frankie
4E9ABFCD2devin2143
4E81E55C1dancer2143
4E38E7ACFhmh(Gs)1243
4E298966Djrv(P)
4DF5CE2B4huggie12-3
4DD982A75speedblue
4C671257Ddamog-1-2
4C4A3823Ekmr4213
4C0B10A5Bdexter
4C02440B8js1342
4BE9F70EAtb1342
4B7D2F063varenet-213
4A3F9E30Eschultmc1243
4A3D7B9BClawrencc2143
4A1EE761Cmadcoder21--
49DE1EEB1he3142
49D928C9Bguillem1---
49B726B71racke
490788E11jsogo2143
4864826C3gotom4321
47244970Bkroeckx2143
45B48FFAEmarga2143
454E672DEisaac1243
44B3A135Cerich1243
44597A593agmartin4213
43FCC2A90amaya1243
43F3E6426agx-1-2
43EF23CD6sanvila1342
432C9C8BDwerner(K)
4204DDF1Baquette
400D8CD16tolimar12--
3FEC23FB2bap34-1
3F972BE03tmancill4213
3F801A743nduboc1---
3EBEDB32Bchrsmrtn4123
3EA291785taggart2314
3E4D47EC1tv(P)
3E19F188Etroyh1244
3DF6807BEsrk4213
3D2A913A1psg(P)
3D097A261chrisb
3C6CEA0C9adconrad1243
3C20DF273ondrej
3B5444815ballombe1342
3B1DF9A57cate2143
3AFA44BDDweasel(Ps,Gs) 1342
3AA6541EEbrlink1442
3A824B93Fasac3144
3A71C1E00turbo
3A2D7D292seb128
39ED101BFmbanck3132
3969457F0joostvb2143
389BF7E2Bkobras1--2
386946D69mooch12-3
374886B63nathans
36F222F1Fedelhard
36D67F790foka
360B6B958geiger
3607559E6mako
35C33C1B8dirson
35921B5D8ajmitch
34C1A5BE5sjq
3431B38BApxt312-
33E7B4B73lmamane2143
327572C47ucko1342
320021490schepler1342
31DEB8EAEgoedson
31BF2305Akrala(Gs)3142
319A42D19dannf21-4
3174FEE35wookey3124
3124B26F3mfurr21-3
30A327652tschmidt312-
3090DD8D5ingo3123
30813569Fjeroen1141
30644FAB7bas1332
30123F2F2gareuselesinge1243
300530C24bam1234
2FD6645ABrmurray-1-2
2F95C2F6Dchrism(P)
2F9138496graham(Gs)3142
2F5D65169jblache1332
2F28CD102absurd
2F2597E04samu
2F0B27113patrick
2EFA6B9D5hamish(P)3142
2EE0A35C7risko4213
2E91CD250daigo
2D688E0A7qjb-21-
2D4BE1450prudhomm
2D2A6B810joussen
2CFD42F26dilinger
2CEE44978dburrows1243
2CD4C0D9Dskx4213
2BFB880A3zeevon
2BD8B050Droland3214
2B74952A9alee
2B4D6DE13paul
2B345BDD3neilm1243
2B28C5995bod4213
2B0FA4F49schoepf
2B0DDAF42awoodland
2A8061F32osamu4213
2A21AD4F9tviehmann1342
299E81DA0kaplan
2964199E2fabbe3142
28DBFEC2Fpelle
28B8D7663ametzler1342
28B143975martignlo
288C7C1F793sam2134
283E5110Fovek
2817A996Atfheen
2807CAC25abi4123
2798DD95Cpiefel
278D621B4uwe-1--
26FF0ABF2rcw2143
26E8169D2hertzog3124
26C0084FCchrisvdb
26B79D401filippo-1--
267756F5Dfrn2341
25E2EB5B4nveber123-
25C6153ADbroonie1243
25B713DF0djpig1243
250ECFB98ccontavalli(Gs)
250064181paulvt
24F71955Adajobe21-3
24E2ECA5Ajmm4213
2496A1827srittau
23E8DCCC0maxx1342
23D97C149mstone(P)2143
22DB65596dz321-
229F19BD1meskes
21F41B907marillat1---
21EB2DE66boll
21557BC10kraai1342
2144843F5lolando1243
210656584voc
20D7CA701steinm
205410E97horms
1FC992520tpo-14-
1FB0DFE9Bgildor
1FAEEB4A9neil1342
1F7E8BC63cedric21--
1F2C423BCzack1332
1F0199162kreckel4214
1ECA94FA8ishikawa2143
1EAAC62DFcyb---1
1EA2D2C41malattia-312
1E77AC835bcwhite(P)
1E66C9BB0tach
1E145F334mquinson2143
1E0BA04C1treinen321-
1DFE80FB2tali
1DE054F69azekulic(P)
1DC814B09jfs
1CB467E27kalfa
1C9132DDByoush-21-
1C87FFC2Fstevenk-1--
1C2CE8099knok321-
1BED37FD2henning(Ks)1342
1BA0A7EB5treacy(P)
1B7D86E0Fcmb4213
1B62849B3smarenka2143
1B3C281F4alain2143
1B25A5CF1omote
1ABA0E8B2sasa
1AB474598baruch2143
1AB2A91F5troup1--2
1A827CEDEafayolle(Gs)
1A6C805B9zorglub2134
1A674A359maehara
1A57D8BF7drew2143
1A269D927sharky
1A1696D2Blfousse1232
19BF42B07zinoviev--12
19057B5D3vanicat2143
18E950E00mechanix
18BB527AFgwolf1132
18A1D9A1Fjgoerzen
18807529Bultrotter2134
1872EB4E5rcardenes
185EE3E0Eangdraug12-3
1835EB2FFbossekr
180C83E8Eigloo1243
17B8357E5andreas212-
17B80220Dsjr(Gs)1342
17796A60Bsfllaw1342
175CB1AD2toni1---
1746C51F4klindsay
172D03CB1kmuto4231
171473F66ttroxell13-4
16E76D81Dseanius1243
16C63746Dhector
16C5F196Bmalex4213
16A9F3C38rkrishnan
168021CE4ron---1
166F24521pyro-123
1631B4819anfra
162EEAD8Bfalk1342
161326D40jamessan13-4
1609CD2C0berin--1-
15D8CDA7Bguus1243
15D8C12EArganesan
15D64F870zobel
159EF5DBCbs
157F045DCcamm
1564EE4B6hazelsct
15623FC45moronito4213
1551BE447torsten
154AD21B5warmenhoven
153BBA490sjg
1532005DAseamus
150973B91pjb2143
14F83C751kmccarty12-3
14DB97694khkim
14CD6E3D2wjl4213
14A8854E6weinholt1243
14950EAA6ajkessel
14298C761robertc(Ks)
142955682kamop
13FD29468bengen-213
13FD25C84roktas3142
13B047084madhack
139CCF0C7tagoh3142
139A8CCE2eugen31-2
138015E7Ethb1234
136B861C1bab2143
133FC40A4mennucc13214
12C0FCD1Awdg4312
12B05B73Arjs
1258D8781grisu31-2
1206C5AFDchewie-1-1
1200D1596joy2143
11C74E0B7alfs
119D03486francois4123
118EA3457rvr
1176015EDevo
116BD77C6alfie
112AA1DB8jh
1128287E8daf
109FC015Cgodisch
106468DEBfog--12
105792F34rla-21-
1028AF63Cforcer3142
1004DA6B4bg66
0.zufus-1--
0.zoso-123
0.ykomatsu-123
0.xtifr1243
0.xavier-312
0.wouter2143
0.will-132
0.warp1342
0.voss1342
0.vlm2314
0.vleeuwen4312
0.vince2134
0.ukai4123
0.tytso-12-
0.tjrc14213
0.tats-1-2
0.tao1--2
0.stone2134
0.stevegr1243
0.smig-1-2
0.siggi1-44
0.shaul4213
0.sharpone1243
0.sfrost1342
0.seb-21-
0.salve4213
0.ruoso1243
0.rover--12
0.rmayr-213
0.riku4123
0.rdonald12-3
0.radu-1--
0.pzn112-
0.pronovic1243
0.profeta321-
0.portnoy12-3
0.porridge1342
0.pmhahn4123
0.pmachard1--2
0.pkern3124
0.pik1--2
0.phil4213
0.pfrauenf4213
0.pfaffben2143
0.p21243
0.ossk1243
0.oohara1234
0.ohura-213
0.nwp1342
0.noshiro4312
0.noodles2134
0.nomeata2143
0.noahm3124
0.nils3132
0.nico-213
0.ms3124
0.mpalmer2143
0.moth3241
0.mlang2134
0.mjr1342
0.mjg591342
0.merker2--1
0.mbuck2143
0.mbrubeck1243
0.madduck4123
0.mace-1-2
0.luther1243
0.luigi4213
0.lss-112
0.lightsey1--2
0.ley-1-2
0.ldrolez--1-
0.lange4124
0.kirk1342
0.killer1243
0.kelbert-214
0.juanma2134
0.jtarrio1342
0.jonas4312
0.joerg1342
0.jmintha-21-
0.jimmy1243
0.jerome21--
0.jaqque1342
0.jaq4123
0.jamuraa4123
0.iwj1243
0.ivan2341
0.hsteoh3142
0.hilliard4123
0.helen1243
0.hecker3142
0.hartmans1342
0.guterm312-
0.gniibe4213
0.glaweh4213
0.gemorin4213
0.gaudenz3142
0.fw2134
0.fmw12-3
0.evan1--2
0.ender4213
0.elonen4123
0.eevans13-4
0.ean-1--
0.dwhedon4213
0.duncf2133
0.ds1342
0.dparsons1342
0.dlehn1243
0.dfrey-123
0.deek1--2
0.davidw4132
0.davidc1342
0.dave4113
0.daenzer1243
0.cupis1---
0.cts-213
0.cph4312
0.cmc2143
0.clebars2143
0.chaton-21-
0.cgb-12-
0.calvin-1-2
0.branden1342
0.brad4213
0.bnelson1342
0.blarson1342
0.benj3132
0.bayle-213
0.baran1342
0.az2134
0.awm3124
0.atterer4132
0.andressh1---
0.amu1--2
0.akumria-312
0.ajt1144
0.ajk1342
0.agi2143
0.adric2143
0.adejong1243
0.adamm12--
0.aba1143

10 January 2006

Tollef Fog Heen: Casper, the friendly little ghost

Everybody who has used an Ubuntu live cd over the last nine months or so has used casper. It started out as a special udeb, called by the debian-installer code to bootstrap a live environment. While d-i is fairly flexible, this was stretching the limits and not really a great solution. Amongst the problems were user interactivity halfway through the boot and a very slow boot. In the middle of December, mdz asked me if I could take a look at implementing the SimplifiedLiveCD specification. As I had played a bit with casper already, I did. Casper is nothing like what it used to be, it now uses initramfs, so no user interactivity after the bootloader. It uses unionfs where available, which speeds it up a fair bit (compare to devmapper + cloop), and if the cd image has squashfs, it uses that too, which makes it even faster. Boot time improvements from around 368 to about 231 seconds is fairly good, but I hope to get it even lower. What I really, really like about casper however is how hackable it is. I added cd integrity check in less than a day (modulo some bugs in usplash I had to fix). Today, I integrated it with the new usplash in initramfs, so we actually have progress in the initramfs as well. (Instead of "mounting root file system" taking about 40 seconds.) Another neat feature is the persistence support. It will now look for filesystems with the label casper-cow (that will be changed to ubuntu-live-rw, I think) if persistent is seen on the kernel command line. This makes it easy to drag your setup around with just an USB key and any Ubuntu live cd. Next out is getting keyboard selection better and more speedups.