Those of you who haven t been in IT for far, far too long might not know that next month will be the 16th(!) anniversary of the disclosure of what was, at the time, a fairly earth-shattering revelation: that for about 18 months, the Debian OpenSSL package was generating entirely predictable private keys.
The recent xz-stential threat (thanks to @nixCraft for making me aware of that one), has got me thinking about my own serendipitous interaction with a major vulnerability.
Given that the statute of limitations has (probably) run out, I thought I d share it as a tale of how huh, that s weird can be a powerful threat-hunting tool but only if you ve got the time to keep pulling at the thread.
Prelude to an Adventure
Our story begins back in March 2008.
I was working at Engine Yard (EY), a now largely-forgotten Rails-focused hosting company, which pioneered several advances in Rails application deployment.
Probably EY s greatest claim to lasting fame is that they helped launch a little code hosting platform you might have heard of, by providing them free infrastructure when they were little more than a glimmer in the Internet s eye.
I am, of course, talking about everyone s favourite Microsoft product: GitHub.
Since GitHub was in the right place, at the right time, with a compelling product offering, they quickly started to gain traction, and grow their userbase.
With growth comes challenges, amongst them the one we re focusing on today: SSH login times.
Then, as now, GitHub provided SSH access to the git repos they hosted, by SSHing to git@github.com with publickey authentication.
They were using the standard way that everyone manages SSH keys: the ~/.ssh/authorized_keys file, and that became a problem as the number of keys started to grow.
The way that SSH uses this file is that, when a user connects and asks for publickey authentication, SSH opens the ~/.ssh/authorized_keys file and scans all of the keys listed in it, looking for a key which matches the key that the user presented.
This linear search is normally not a huge problem, because nobody in their right mind puts more than a few keys in their ~/.ssh/authorized_keys, right?
Of course, as a popular, rapidly-growing service, GitHub was gaining users at a fair clip, to the point that the one big file that stored all the SSH keys was starting to visibly impact SSH login times.
This problem was also not going to get any better by itself.
Something Had To Be Done.
EY management was keen on making sure GitHub ran well, and so despite it not really being a hosting problem, they were willing to help fix this problem.
For some reason, the late, great, Ezra Zygmuntowitz pointed GitHub in my direction, and let me take the time to really get into the problem with the GitHub team.
After examining a variety of different possible solutions, we came to the conclusion that the least-worst option was to patch OpenSSH to lookup keys in a MySQL database, indexed on the key fingerprint.
We didn t take this decision on a whim it wasn t a case of yeah, sure, let s just hack around with OpenSSH, what could possibly go wrong? .
We knew it was potentially catastrophic if things went sideways, so you can imagine how much worse the other options available were.
Ensuring that this wouldn t compromise security was a lot of the effort that went into the change.
In the end, though, we rolled it out in early April, and lo! SSH logins were fast, and we were pretty sure we wouldn t have to worry about this problem for a long time to come.
Normally, you d think patching OpenSSH to make mass SSH logins super fast would be a good story on its own.
But no, this is just the opening scene.
Chekov s Gun Makes its Appearance
Fast forward a little under a month, to the first few days of May 2008.
I get a message from one of the GitHub team, saying that somehow users were able to access other users repos over SSH.
Naturally, as we d recently rolled out the OpenSSH patch, which touched this very thing, the code I d written was suspect number one, so I was called in to help.
They're called The Usual Suspects for a reason, but sometimes, it really is Keyser S ze
Eventually, after more than a little debugging, we discovered that, somehow, there were two users with keys that had the same key fingerprint.
This absolutely shouldn t happen it s a bit like winning the lottery twice in a row1 unless the users had somehow shared their keys with each other, of course.
Still, it was worth investigating, just in case it was a web application bug, so the GitHub team reached out to the users impacted, to try and figure out what was going on.
The users professed no knowledge of each other, neither admitted to publicising their key, and couldn t offer any explanation as to how the other person could possibly have gotten their key.
Then things went from weird to what the ? .
Because another pair of users showed up, sharing a key fingerprint but it was a different shared key fingerprint.
The odds now have gone from winning the lottery multiple times in a row to as close to this literally cannot happen as makes no difference.
Once we were really, really confident that the OpenSSH patch wasn t the cause of the problem, my involvement in the problem basically ended.
I wasn t a GitHub employee, and EY had plenty of other customers who needed my help, so I wasn t able to stay deeply involved in the on-going investigation of The Mystery of the Duplicate Keys.
However, the GitHub team did keep talking to the users involved, and managed to determine the only apparent common factor was that all the users claimed to be using Debian or Ubuntu systems, which was where their SSH keys would have been generated.
That was as far as the investigation had really gotten, when along came May 13, 2008.
Chekov s Gun Goes Off
With the publication of DSA-1571-1, everything suddenly became clear.
Through a well-meaning but ultimately disasterous cleanup of OpenSSL s randomness generation code, the Debian maintainer had inadvertently reduced the number of possible keys that could be generated by a given user from bazillions to a little over 32,000.
With so many people signing up to GitHub some of them no doubt following best practice and freshly generating a separate key it s unsurprising that some collisions occurred.
You can imagine the sense of oooooooh, so that s what s going on! that rippled out once the issue was understood.
I was mostly glad that we had conclusive evidence that my OpenSSH patch wasn t at fault, little knowing how much more contact I was to have with Debian weak keys in the future, running a huge store of known-compromised keys and using them to find misbehaving Certificate Authorities, amongst other things.
Lessons Learned
While I ve not found a description of exactly when and how Luciano Bello discovered the vulnerability that became CVE-2008-0166, I presume he first came across it some time before it was disclosed likely before GitHub tripped over it.
The stable Debian release that included the vulnerable code had been released a year earlier, so there was plenty of time for Luciano to have discovered key collisions and go hmm, I wonder what s going on here? , then keep digging until the solution presented itself.
The thought hmm, that s odd , followed by intense investigation, leading to the discovery of a major flaw is also what ultimately brought down the recent XZ backdoor.
The critical part of that sequence is the ability to do that intense investigation, though.
When I reflect on my brush with the Debian weak keys vulnerability, what sticks out to me is the fact that I didn t do the deep investigation.
I wonder if Luciano hadn t found it, how long it might have been before it was found.
The GitHub team would have continued investigating, presumably, and perhaps they (or I) would have eventually dug deep enough to find it.
But we were all super busy myself, working support tickets at EY, and GitHub feverishly building features and fighting the fires in their rapidly-growing service.
As it was, Luciano was able to take the time to dig in and find out what was happening, but just like the XZ backdoor, I feel like we, as an industry, got a bit lucky that someone with the skills, time, and energy was on hand at the right time to make a huge difference.
It s a luxury to be able to take the time to really dig into a problem, and it s a luxury that most of us rarely have.
Perhaps an understated takeaway is that somehow we all need to wrestle back some time to follow our hunches and really dig into the things that make us go hmm .
Support My Hunches
If you d like to help me be able to do intense investigations of mysterious software phenomena, you can shout me a refreshing beverage on ko-fi.
the odds are actually probably more like winning the lottery about twenty times in a row.
The numbers involved are staggeringly huge, so it s easiest to just approximate it as really, really unlikely .
DebConf8
This tshirt is 12 years old and from DebConf8.
DebConf8 was my 6th DebConf and took place in Mar de la Plata, Argentina.
Also this is my 6th post in this series of posts about DebConfs and for the
last two days for the first time I failed my plan to do one post per day.
And while two days ago I still planned to catch up on this by doing more than one
post in a day, I have now decided to give in to realities, which mostly
translates to sudden fantastic weather in Hamburg and other summer
related changes in life. So yeah, I still plan to do short posts about all the
DebConfs I was lucky to attend, but there might be days without a blog post.
Anyhow, Mar de la Plata.
When we held DebConf in Argentina it was winter there, meaning locals and other folks
would wear jackets, scarfs, probably gloves, while many Debian folks not so much.
Andreas Tille freaked out and/or amazed local people by going swimming in the sea
every morning. And when I told Stephen Gran that even I would find it a bit cold with
just a tshirt he replied "na, the weather is fine, just like british summer", while
it was 14 celcius and mildly raining.
DebConf8 was the first time I've met Valessio Brito, who I had worked together
since at least DebConf6. That meeting was really super nice, Valessio is such a lovely person.
Back in 2008 however, there was just one problem: his spoken English was worse than
his written one, and that was already hard to parse sometimes. Fast forward
eleven years to Curitiba last year and boom, Valessio speaks really nice English now.
And, you might wonder why I'm telling this, especially if you were exposed to my Spanish
back then and also now. So my point in telling this story about Valessio is to illustrate two things:
a.) one can contribute to Debian without speaking/writing much English, Valessio did
lots of great artwork since DebConf6 and b.) one can learn English by doing Debian stuff. It worked
for me too!
During set up of the conference there was one very memorable moment, some time after the
openssl maintainer, Kurt Roeckx arrived at the venue: Shortly
before DebConf8 Luciano Bello, from Argentina no less, had found
CVE-2008-0166 which
basically compromised the security of sshd of all Debian and Ubuntu installations done in the last
4 years (IIRC two Debian releases were affected) and which was
commented heavily and noticed everywhere.
So poor Kurt arrived and
wondered whether we would all hate him, how many toilets he would have to clean and what not...
And then, someone rather quickly noticed this, approached some people and suddenly
a bunch of people at DebConf were group-hugging Kurt
and then we were all smiling and continuing doing set up of the conference.
That moment is one of my most joyful memories of all DebConfs and partly explains
why I remember little about the conference itself, everything else pales in comparison and
most things pale over the years anyway. As I remember it, the conference ran
very smoothly in the end, despite quite some organisational problems right before the start.
But as usual, once the geeks arrive and are happily geeking, things start to run smooth,
also because Debian people are kind and smart and give hands and brain were needed.
And like other DebConfs, Mar de la Plata also had moments which I want to share but I will only hint
about, so it's up to you to imagine the special leaves which were brought to that cheese and wine party! Update: added another xkcd link, spelled out Kurt's name after talking to him and added a link to a video of the group hug.
When we setup Freexian s offer to bring together funding from multiple companies in order to sponsor the work of multiple developers on Debian LTS, one of the rules that I imposed is that all paid contributors must provide a public monthly report of their paid work.
While the LTS project officially started in June, the first month where contributors were actually paid has been July. Freexian sponsored Thorsten Alteholz and Holger Levsen for 10.5 hours each in July and for 16.5 hours each in August. Here are their reports:
It s worth noting that Freexian sponsored Holger s work to fix the security tracker to support squeeze-lts. It s my belief that using the money of our sponsors to make it easier for everybody to contribute to Debian LTS is money well spent.
As evidenced by the progress bar on Freexian s offer page, we have not yet reached our minimal goal of funding the equivalent of a half-time position. And it shows in the results, the dla-needed.txt still shows around 30 open issues. This is slightly better than the state two months ago but we can improve a lot on the average time to push out a security update
To have an idea of the relative importance of the contributions of the paid developers, I counted the number of uploads made by Thorsten and Holger since July: of 40 updates, they took care of 19 of them, so about the half.
I also looked at the other contributors: Rapha l Geissert stands out with 9 updates (I believe that he is contracted by lectricit de France for doing this) and most of the other contributors look like regular Debian maintainers taking care of their own packages (Paul Gevers with cacti, Christoph Berg with postgresql, Peter Palfrader with tor, Didier Raboud with cups, Kurt Roeckx with openssl, Balint Reczey with wireshark) except Matt Palmer and Luciano Bello who (likely) are benevolent members of the LTS team.
There are multiple things to learn here:
Paid contributors already handle almost 70% of the updates. Counting only on volunteers would not have worked.
Quite a few companies that promised help (and got mentioned in the press release) have not delivered the promised help yet (neither through Freexian nor directly).
Last but not least, this project wouldn t exist without the support of multiple companies and organizations. Many thanks to them:
I just got my tickets to Portland, to assist to one of my favorite events: DebConf! So, the only point of this post it to say:
See you there and let s keep Portland Weird
I wanted to have a directory in my Dropbox where every file placed get some sort of encryption. They might be personal pictures (if you know what I mean), finance notes, patent ideas, or the like. I create those files constantly, often with my phone. While I would like to back them up on the cloud, I don t need to access them regularly. The problem is that I don t want my phone to storage them (at least not plain) for obvious reasons. And, on top of this, Android s Dropbox app does not allow to disable synchronization by folder.
So, here is the scheme I came up with. Maybe it is also useful for you or you get some idea how to improve it.
You will need an always-on-line computer with Dropbox client (for Dropbox synchronization), GnuPG (for cipher the files), and a really nice tool called incron (which monitors a directory and trigger actions).
The instructions:
Install the needed software in your box. Dropbox can be downloaded from here and the packages gnupg and incron.
Import your pgp key in the local keyring. Take a look to this if you don t know how.
Create a folder called private in your Dropbox.
Create somewhere (I did it in /opt/cipherPrivateDropbox.sh) the following script: #!/bin/sh
gpg --recipient "<your ID>" --trust-model always --encrypt $1
rm $1
As root, give access to your user to incron: echo luciano > /etc/incron.allow
As that user, run incrontab -e and include the following line:
/home/luciano/Dropbox/private IN_CREATE,IN_NO_LOOP /opt/cipherPrivateDropbox.sh $@/$#
That s all, each time you put something in your private folder, incron will cipher it and leave it there.
In my long trip to Nicaragua I made progress in my reading: Quicksilver, by Neal Stephenson. In the Spanish edition the title is Azogue. But I m assuming that you are not a Spanish speaker. Here is a small fragment (in English) I found there:
You must remember that the planters are short-sighted. They re all desperate to get out of Jamaica they wake up every day expecting to find themselves, or their children, in the grip of some tropical fever. To import female Neegers would cost nearly as much as to import males, but the females cannot produce as much sugar particularly when they are breeding. Daniel had finally recognized this voice as belonging to Sir Richard Apthorp the second A in the CABAL.
It s a bit embarrassing when I discovered myself realizing where the word cabal comes from. And I m posting this as a head-up for everyone who know there is no cabal in Debian; but they don t know which is the origin of the word cabal.
Stephenson changed the name of the historic cabal, a group of high councillers of King Charles II of England, Scotland and Ireland, in 1668. In the novel, they are: John Comstock (Earl of Epsom), Louis Anglesey (Duke of Gunfleet), Knott Bolstrood (Count Penistone), Sir Richard Apthorp and Hugh Lewis (Duke of Tweed). In the real world they had been:
Este post tambi n est escrito en espa ol.
If you follow me on Twitter/Facebook, you probably know that I went to Gothenburg, Sweden, during my last trip to Europe. But you probably don t know what I did in such remote city.
Few months ago, I applied to a PhD student position at Chalmers. I was lucky enough to be shortlisted, so I went to some in-situ interviews. And, incredibly, I have been accepted! :-)
Chalmers is a TOP 100 university. Most of the papers I read during the last months has been written by Chalmers researchers and for me is a great honor to be part of an academic institution with such prestige.
I m going to move to the nice Gothenburg city in August. And I m happy :)
Este post tambi n est escrito en espa ol aqu .
Privacy at Facebook is heavy-duty. As a big fan of the Worlds Collide Theory I hate be tagged compulsively. I would like to select in which photos appear in my profile and feed. Since I couldn t find that option in the setting menu, I looked for the answer in my favorite scripting language: Python.
This 60-lines-long script removes your tag from the latests photos where you has been labeled. You can download it from here. You may run it hourly (or every 15 minutes, or every 5 minutes, depends how paranoid you are) via cron or whatever.
Any improvement is welcome. It probably runs on Windows too. If you managed to do it, leave a comment for the others.
When your laptop is being repaired (and it s still there, since August 28) you need imaginative ways to be connected.
Here is my Nokia N800 as something near to a desktop computer.
Just few notices:
life battery is really short when you plug too many things to the USB interface.
If I say I got the third place in a scholarship application , it doesn t look bad.
But there is money only for the first two persons. Sometimes, close is not enough. So, without money, I won t be able to study in Europe damn
Maybe next year maybe not.
Note: The application was, as you can see, for a doctoral scholarship in Spain my broken English has no effect here
Some days ago, my new camera arrived. I bougth it by Amazon and the parents of a friend brought it to my country.
The selected model was a Canon PowerShot SX110 IS. My last camera was a Canon PowerShot A700. It has been in service since April 2006, until a terrible fall ended with its nice capability of taking good pictures, during the 25C3 in Berlin.
The next week I m going to Berlin (and some days to Frankfurt). I ll be in the 25th Chaos Communication Congress (25C3).
I ll be out of home from 24th Dec until 7th Jan. It ll be a non-traditional holidays, be far away from family. But, in the other hand, it gonna be awesome be and talk in a CCC and spend a Christmas Eve in a plane and a New Year s Eve in Brandenburg Gate with friends.
See you there?
Yes, it’s me again with this DSA-1571 exploitation issue. The discovery, explanation and exploitation of the bug is now part of my final coursework for my postgraduate degree career. So, yes… sorry.
Some weeks ago I started suspecting about the attack to PFS in SSL with EDH. The key point is: the key space is dependent of the PRNG state. The bug affects the initialization of the PRNG, but the random string has not a pattern by it self. If you ask for many random numbers to the PRNG, you gonna get numbers that differ among them, since they are the output of a hash function of them self. So each random number depends on, besides the PID, the state of the PRNG pool in the moment (in other words, amount of bytes that you already pull from the PRNG pool before)
The explained attack was based in a fixed list of private exponents (which are selected randomly during the DHE handshake), presupposing that all the application call RAND_bytes() the same number of times before get it. To make the list of exponent I ran the openssl s_client with all the possible PIDs, hoping that all the applications behaves the same way.
After more tests I notice that that was an overgeneralization. The proof is in the pudding: wget and cURL, two simple CLI file retrievers, gets different exponent between them, even running with the same PID.
I was working on this when I accidentally found a really nice Eric Rescorla’s post which is deeply related with this. The post goes further and analyzes the interaction between how Apache forks off and how it generates SSL handshakes.
So, I made lists of secret exponents for wget, curl, openssl s_client and openssl s_server with a modification version of libssl (appling this patch) and running scripts like this:
for i in $(seq $((2**15)));
do
export MAGICPID=$i;
LD_LIBRARY_PATH="openssl.broken/" LD_PRELOAD="./getpid.so" \
wget --no-check-certificate https://localhost/ -q -O /dev/null ;
echo $i ;
done
As you can see, I used the HD Moore’s GetPID faker shared library and a normal local Apache with mod_ssl. The broken libssl (which is in .openssl.broken/) store up in /tmp/data.key a csv with command name, PID and all the DH components (g, x, y and p).
But this way is farly unconfortable for others SSL deamon servers. Have you got any better idea?
Excellent organization, orga team fully deserved all the praise they got.
Meeting people - it was the first DebConf I attended in person, so it was a pleasure to meet lots of folks whom I’ve only known online before.
Quite a few interesting talks (videos are available, or will be available shortly, in DebConf8 video archive). Particularly recommended: quality assurance talks/BOFs by Lucas Nussbaum, BTS development progress by Don Armstrong, method diffusion in large volunteer projects by Martin Krafft, talk and debate on OpenSSL security issue by Luciano Bello, various lightning talks.
Deserving a special mention, a talk by Mark Shuttleworth on Debian and Ubuntu relationship. I thought that he said all the right words and, judging by reactions, attitude towards Ubuntu has changed significantly during the last year, turning from hostility to peaceful cooperation.
Argentinian food: steaks, empanadas, alfajores.
What I did not like
People pay much less attention to fixing RC bugs than they used to, the two BSPs organized during DebConf8 attracted only 3-5 people.
Some nice ideas being thrown around without anyone volunteering to work on them. Not really surprising, given that most of the core teams are short on manpower.
Almost complete lack of events targeted at engaging users and beginner contributors.
Quite a few specialized talks, interesting (IMHO) to only a minor percentage of the attendees.
Lots of discussions with few or no specific decisions or action items.
Pretty long trips to and from Mar del Plata (about 30 hours door-to-door travel time for me), luggage was lost for 2 days on the way there.
What I did
Attended about 75% of talks, tried to contribute to discussion in constructive, non-flamy ways .
Participated in BSPs, contributed debugging information, patches or fixes for about a dozen of release critical bugs, started triaging sparc-specific bugs.
I've had an outstandingly good time, as expected. Much kudos to the
local team here in Argentina for organising the conference so well,
and it's great to see that the efforts of the video team are making it
possible for lots of people to attend the talks virtually as
well. Good work, folks!
My highlights so far:
Getting all the right people together in one room for a really
productive session discussing "tdebs", a new way of organising the
translations in our packages. In less than an hour, we went from a mix
of opinions and suggestions (all with problems) to a good consensus
about what we're going to do and what work needs to be done. In Debian
we work very effectively online normally, but this kind of design
discussion could easily have taken a month or more via email.
Sometimes face-to-face meetings just work better! (See the
wiki for the session notes.)
The 15th Birthday Party last night. I got to formally accept our
"epic fail" award from Luciano Bello after his recent trip to Las
Vegas to tell the Defcon and Blackhat folks about our recent OpenSSL
problems. Then I was coerced into dancing for the delight of the
crowd. Apparently some git even caught that on video.
Oh, the shame...
Beeeeeeeeef! I joined a group of people at the "Palacio del Bife"
restaurant in the centre of town on Monday night, and we ate a
lot of beef. I had a massive T-bone steak which was
awesome - large and very very nicely cooked. I'll have to go
again. *grin*
Unfortunately, the end of the week is quickly approaching. I'll
have to head back to England all too soon.
At this point, all of you should know and see how the H D Moore s toys work. Those toys attack SSH public-key authentication using clone keys and online brute force.
Furthermore, many of you know that there are other effects produced by a biased PRNG besides this one.
Strangely, I could not find more of those toys exploiting these aspects. So, I would like to show you a Wireshark patch which attacks Perfect Forward Secrecy (PFS) provided by Ephemeral Diffie Hellman (EDH). Introduction to EDHLet s put it in plain words (if you know what we are talking about, ignore this and jump to the next heading): In an insecure communications channel the parties agree a common key to cipher their dialog. This is what happens in SSL (in most of the cases, depending on the cipher suite):
The server selects a random prime p and a generator g of the field Z*p (Let s ignore the mathematical properties of these values). So, the components p and g are public.
The server picks a secret random number Xs and calculates Ys=gXs mod p. Ys is public and is sent to the client (just like p and g).
The client does something similar, selecting a secret random number Xc and calculating Yc=gXc mod p too. The client makes Yc public by sending it to the server.
The shared secret s is the public key of the other part to the exponential of the own private number, all in p modulus. That is, for the client s=YsXcmod p and for the server s=YcXsmod p.
With this shared secret the parties can encrypt all the following messages in a secure way.
In the Ephemeral Diffie Hellman (EDH), the private numbers are ruled out, so s is mathematically secure and nobody can obtain it even having access to one of the parties after the aforementioned handshake.
The exploit If an eavesdropper can explore the complete private key space (the all possible numbers for Xc or Xs), he/she will be able to get access to the shared secret. With it all the communication can be deciphered. That s what this patch can do.
A Wireshark with this patch and a list of possible private keys will try to brute force the share secret. If one of the parties is using the vulnerable OpenSSL package the communication is totally insecure and will be decrypted.
Debian packages with the patch applied can be found here.
This is a list of all 215 possible 64 and 128 bit DH private keys in systems vulnerable to the predictable OpenSSL PRNG described by DSA-1571.
An example of a pcap file can be found here (it was built with a vulnerable client and one of the Moore toys, a hacked getpid by running $ MAGICPID=101 LD_PRELOAD= getpid.so ./vulnerable-openssl/apps/openssl s_client -connect db.debian.org:443 )
The patch was submitted in order to be committed on the Wireshark trunk. There you can find the patch against the on-develop source revision 25765. Issues that can be improvedWe (the other developers and myself) detected few things to be improved. But we will do nothing for them. So, if you want to contribute with some code, start from these items and submit the patches to the Wireshark s bugzilla:
When the packets are out-of-order the decipher with stop itself.
The brute force attack should run in a background process (and with a progress bar)
Check the length of the keys before trying to brute force them.
The patch also implements the display of public DH parameters in the packet tree. It s incomplete.
CreditsPaolo Abeni <paolo.abeni at email.it> Luciano Bello <luciano at debian.org> Maximiliano Bertacchini <mbertacchini at citefa.gov.ar>
This work was partially supported by Si6 Labs at CITEFA, Argentina.
UPDATE Jul. 21st: See more and updated info here, especially this.
Those of you who haven t been in IT for far, far too long might not know that next month will be the 16th(!) anniversary of the disclosure of what was, at the time, a fairly earth-shattering revelation: that for about 18 months, the Debian OpenSSL package was generating entirely predictable private keys.
The recent xz-stential threat (thanks to @nixCraft for making me aware of that one), has got me thinking about my own serendipitous interaction with a major vulnerability.
Given that the statute of limitations has (probably) run out, I thought I d share it as a tale of how huh, that s weird can be a powerful threat-hunting tool but only if you ve got the time to keep pulling at the thread.
They're called The Usual Suspects for a reason, but sometimes, it really is Keyser S ze
DebConf8
This tshirt is 12 years old and from DebConf8.
Update: added another xkcd link, spelled out Kurt's name after talking to him and added a link to a video of the group hug.
When we setup 
I just got my tickets to Portland, to assist to one of my favorite events: 
See you there and let s 
I wanted to have a directory in my Dropbox where every file placed get some sort of encryption. They might be personal pictures (if you know what I mean), finance notes, patent ideas, or the like. I create those files constantly, often with my phone. While I would like to back them up on the cloud, I don t need to access them regularly. The problem is that I don t want my phone to storage them (at least not plain) for obvious reasons. And, on top of this, Android s Dropbox app does not allow to disable synchronization by folder.
So, here is the scheme I came up with. Maybe it is also useful for you or you get some idea how to improve it.
You will need an always-on-line computer with 
If you 
Thanks a lot to 
The complete photo set is 
The next week I m going to 
What I liked
.
I've had an outstandingly good time, as expected. Much kudos to the
local team here in Argentina for organising the conference so well,
and it's great to see that the efforts of the video team are making it
possible for lots of people to attend the talks virtually as
well. Good work, folks!
My highlights so far:
