Search Results: "dkg"

13 December 2010

Daniel Kahn Gillmor: converting eviews models to gretl?

Someone recently pointed me to some economic/statistical models that were designed for eviews (a proprietary tool). They were looking for help converting the model into a format that could work with gretl (the Gnu Regression, Econometrics and Time-series Library). I'm afraid i'm an economics dunce, and i have no experience with this stuff. Any pointers?Tags: conversion, economics, gretl

6 December 2010

Jamie McClelland: rm -rf at ~ 500 Mbytes/second

If you ever have the misfortune of accidentally passing the path to a directory containing 177GB of data to the rm -rf command, I'll start by suggesting that you hit ctl-c. The sooner the better. Next, assuming you have some sort of backup, you'll be staring at two monumentally large data sets, wondering exactly what was deleted from the original. With help from dkg, I learned somethings about rm -rf. For one, it deletes one top level directory at a time. So - a comparison of the top level directory listings of the original and backup is a good place to start. Top level directories that are entirely missing in the original are easy to restore. However, the presence of a top level directory in the original doesn't mean it was un-touched. Next, you'll want to figure out which top level directory rm was operating on when you hit ctl-c. dkg discovered that ls -UR will provide a listing in the same order that rm -rf uses. The -U means do not sort. Note - the unsorted listing of the backup directory might not be the same as the unsorted listing of the original, so ls -UR is only really helpful on the original directory. After selecting the first top level directory, rm -rf seems to delete all files in that directory first (presumably in the same order that ls -UR will list them), then it enters the first sub directory (as returned by ls -UR) and repeats the process. With a careful comparison of ls -UR on the original with the directory listings on the backup, you should be able to pinpoint the exact sub directories affected, allowing you to restore only the files and directories that you deleted. Thanks to dkg for technical and blog title suggestions.

2 December 2010

Daniel Kahn Gillmor: forwarding unix domain sockets with ssh and socat

i suspect a lot of people are used to forwarding TCP sockets with SSH -- for example, to connect locally to a mysql daemon that runs only on the loopback interface of a remote machine (this is debian's default mysql-server configuration):
ssh -N -T -oExitOnForwardFailure=yes -L 3306:localhost:3306 remoteuser@mysqlserver.example
But sometimes, the remote service runs on a UNIX-domain socket, not on a TCP socket -- for example, debian's default configuration for postgresql is to have it listen only on a UNIX domain socket in /var/run/postgresql, and use SO_PEERCRED with a simple system account == psql account mapping scheme to authenticate users without needing any extra credentials. This is not quite as simple to forward over ssh, but it's doable as long as socat is installed on both your local host and on the remote postgres server. Here's one way to do it if $SOCKET_DIR points to the full path of a directory under the user's control (this is all one command, split across lines for easier reading):
socat "UNIX-LISTEN:$SOCKET_DIR/.s.PGSQL.5432,reuseaddr,fork" \
   EXEC:'ssh remoteuser@psqlserver.example socat STDIO UNIX-CONNECT\:/var/run/postgresql/.s.PGSQL.5432'
Then, you'd connect with something like:
psql "user=remoteuser host=$SOCKET_DIR"
Each such psql connection will trigger an ssh connection to be made. Of course, this won't work well if ssh has to prompt for passwords, but you should be using ssh-agent anyway, right? There are at least a couple nice features of being able to use postgresql from a local client like this: Of course, postgresql itself already comes with a nice range of high-quality network-capable authentication mechanisms you could use. But many of them (like GSSAPI or X.509 mutual key-based authentication over TLS) require additional infrastructure setup; and you probably already have sshd up and running on that machine -- so why not make use of it?Tags: postgresql, socat, ssh, tip, unix-domain socket

17 November 2010

Daniel Kahn Gillmor: registrars and AAAA glue records

i've been on an IPv6 kick recently, getting dual-stack systems up and working for a bunch of folks.I'd like to make some of these services reachable by IPv6-only clients. this suggests that i need a range of details sorted out, but i think the one piece left for me is the glue records for the nameservice. i use in-bailiwick nameservers for DNS where possible, which means i want mandatory glue records. that is, the primary namserver for example.org is probably something like ns0.example.org, which means that the org nameservers themselves need to store not only the NS record, but an A record that corresponds to the name pointed to by the NS. But for IPv6-only clients that do their own name resolution, i need AAAA glue records, and i haven't yet found a registrar that will push AAAA glue records for the same names as the existing A glue into the org zone. Do you know of a registrar that will do this?I've tried:
dotster
Dotster seems to only allow IPv4 glue to be entered on their Register Nameserver config page (needs a dotster login to see it). They haven't yet yet responded to my query through their support web form about submitting AAAA glue
gandi
gandi at least offers the opportunity to enter AAAA glue, but apparently can't let me have both AAAA and A glue for the same name. A note to their support team got me a response that this is planned for Q1 or Q2 of 2011.
Any suggestions for reasonable registrars that offer this today? Am i being silly in wanting AAAA and A glue for the same names? i note that the root zone and the org zone both offer A and AAAA records for each of their dual-stack nameservers. You can check for yourself:
 dig @a.root-servers.net ns org
 dig @a.root-servers.net ns .
if i don't go for dual records, i could instead use gandi and go with distinct names for the v6 and v4 servers, like this:
;; QUESTION SECTION:
;example.org.				IN	NS
;; AUTHORITY SECTION:
example.org.      172800	IN	NS	a.ns.example.org.
example.org.      172800	IN	NS	b.ns.example.org.
example.org.      172800	IN	NS	c.ns.example.org.
example.org.      172800	IN	NS	d.ns.example.org.
;; ADDITIONAL SECTION:
a.ns.example.org. 172800	IN	A	192.0.2.3
b.ns.example.org. 172800	IN	A	192.0.2.4
c.ns.example.org. 172800	IN	AAAAA	2001:db8::3
d.ns.example.org. 172800	IN	AAAAA	2001:db8::4
But of course what i really want is this:
;; QUESTION SECTION:
;example.org.				IN	NS
;; AUTHORITY SECTION:
example.org.      172800	IN	NS	a.ns.example.org.
example.org.      172800	IN	NS	b.ns.example.org.
;; ADDITIONAL SECTION:
a.ns.example.org. 172800	IN	A	192.0.2.3
a.ns.example.org. 172800	IN	AAAAA	2001:db8::3
b.ns.example.org. 172800	IN	A	192.0.2.4
b.ns.example.org. 172800	IN	AAAAA	2001:db8::4
My concern about this is if some IPv4-only system gets a list like the first one, and decides to use c.ns.example.org or d.ns.example.org, which doesn't have an A record at all. That would be a silly implementation, of course. but uh, we have a lot of silly implementations of things out there.Feedback welcome!Tags: dns, ipv6

20 October 2010

Daniel Kahn Gillmor: Debian NYC Workshop: What's in a Package?

Debian NYC will be holding a workshop next week: What's in a Package? will happen at 7:00pm New York time on October 27, 2010. If you're in the New York area, interested in packaging things for debian and related systems, or just want to understand the packages in your system better, you should RSVP and come on out! This workshop will provide advanced theory useful for people modifying or creating packages. For people modifying packages, you'll learn many typical motifs and about various build systems. For creating packages, you'll be much better prepared to read and understand guides a deep level. However, this is still not a step-by-step guide in "how to build packages", but will get you very close to there. See you there!

6 October 2010

Daniel Kahn Gillmor: monkeysphere and distributed naming

Roland Mas writes an interesting article about decentralized naming, in which he says:
Monkeysphere aims at adding a web of trust to the SSL certificates system, but the CA chain problem seems to persist (although I must admit I'm not up to speed with the actual details).
Since i'm one of the Monkeysphere developers, i figure i should respond! Let me clarify that Monkeysphere doesn't just work in places where X.509 (the SSL certificate system) works. It works in other places too (like SSH connections). And I don't think that the CA chain problem that remains in Monkeysphere is anything like the dangerous mess that common X.509 usage has given us. I do think that at some level, people need to think about who is introducing them to other people -- visual or human-comprehensible representations of public key material are notoriously difficult to make unspoofable. On the subject of distributed naming: OpenPGP already allows distributed naming: every participant in the WoT is allowed to assert that any given key maps to any given identity. Duplicates and disagreements can exist just fine. How an entity decides to certify another entity's ID without a consensus global namespace is a tough one, though. If i've always been known as "John Smith" to my friends, and someone else has also been known as "John Smith" to his friends, our friends aren't actually disagreeing or in conflict -- it's just that neither of us has a unique name. The trouble comes when someone new wants to find "John Smith" -- which of us should they treat as the "correct" one? I think the right answer probably has to do with who they're actually looking for, which has to do with why they're looking for someone named "John Smith". If they're looking for John Smith because the word on the street is that John Smith is a good knitter and they need a pair of socks, they can just examine what information we each publish about ourselves, and decide on a sock-by-sock basis which of us best suits their needs. But if they're looking for "John Smith" because their cousin said "hey, i know this guy John Smith. I think you would like to argue politics over a beer with him", then what matters is the introduction. And OpenPGP handles that just fine -- if their cousin has only ever met a single John Smith, that's the right one. If their cousin has met several John Smiths, then the searcher would do well to ask their cousin some variant of "hey, do you mean John Smith or John Smith ", or even "do you mean the John Smith who Molly has met, or the one who Charles has met?" (assuming that Molly and Charles have each only certified one John Smith in common with the cousin, and not the same one as each other), or to get a real-time introduction to a particular John Smith, where his specific key is somehow recordable by the searcher for future conversations (or beer drinking). This is what we do in the real world anyway. We currently lack good UIs for doing this over the network, but the certification infrastructure is in place already. What we're lacking in infrastructure, though, is a way to have a distributed addressing. Roland's proposal was to publish addresses corresponding to cryptographic identities within some DNS zone, or in freenet or gnutella. Another approach (piggybacking on existing infrastructure) would be to include IP address information in the OpenPGP self-certification, so the holder of the name could claim exactly their own IP address. This could be distributed through the keyserver network, just like other updates are today, and it could be done simply and immediately with a well-defined OpenPGP notation. I'd be happy to talk to interested people about how to specify such a notation, and what possible corner cases we might run into. Drop a note here, or mail the Monkeysphere mailing list or hop onto #monkeysphere on irc.oftc.netTags: authentication, distributed naming, identity, monkeysphere

1 October 2010

Stefano Zacchiroli: new GPG key 6D866396

so long and thanks for all the fish, dear old F2C423BC It was about time. Last bits from keyring-maint finally convinced me to move away from my old 1024 DSA to a new 4096 RSA GPG key. I know it's bloated, but given that I expect the life time of the new (master) key to be 10 years, I saw no reason to switch to any weaker key. I'm lucky enough to live in a DD-crowded area and I've henceforth already collected some signatures ... but I need waaaay more signatures before even daring to think about asking for a replacement in the Debian keyring. So .... ... here's my transition document (shamelessly inspired from Russ' excellent text). If you have signed my old key, you should have a fairly straightforward trust path to my new key. If that is compatible with your signing policy, please sign my new key. Having been there, here are a couple of excellent recommended readings for doing the transition yourself:

29 September 2010

Russell Coker: Links September 2010

Kevin Stone gave an interesting TED talk about the biological future of joint replacement [1]. Using stem cells and animal tissue which has been prepared to destroy the chemicals that trigger immune responses the tissues can regrow. Replacing joints with titanium and ceramic lets people walk again, regrowing them with Kevin s methods allows them to compete at the highest levels of sporting contests! Derek Sivers gave a brief and interesting TED talk advising people to keep their goals secret if they want to achieve them [2]. The Parrot AR.Drone is an interesting device [3], it s a four-propeller helicopter that is controlled by WiFi. Apparently there is little security (it binds to the first WiFi client it sees) which is a significant down-side. It returns a video feed to the controlling iPhone as it flys and can hover when it loses it s connection. It will be interesting to see when people write software for other devices (Android etc). Also I wonder whether there will be open source weapons kits for it. If you could have those devices use either a nerf gun or a lance to attack each other s turbines then you could have drone jousting. Don Marti had an interesting new idea for a crime game [4]. The main concept is to use the element of mistrust that is present in real criminal gangs. The new guy you invite to join a job might inform the police and you won t know for sure. Sometimes a heist will be discovered by the police through bad luck (or good police work) and you will wonder whether there was an informant. The aim is for a simple game design and with the complexity in email discussions between the players. The C64 isn t dead, it s even on the Internet [5], an original C64 is running a web site! Tan Li gave an interesting TED talk about a new headset to read brain-waves [6]. The device in question can be applied in minutes, requires no gel or abrasion of the scalp, connects to the computer wirelessly and is relatively cheap at $300US. The developer s kit (which I think includes a headset) is $500US. I wonder if the community can develop a cheaper version of this which is more open. Lisa Margonelli gave an interesting TED talk about the politics of oil [7]. One of her insightful points is that the subsidies for oil should be shifted from the oil industry to middle-class consumers. But she goes off track a bit by suggesting gradual oil tax increases until 2020, according to the best estimates of groups such as the CSIRO they won t need to have taxes to give a high oil price in 2020! She is aiming for a 20% reduction in petrol use by 2020, but I m not aware of any serious group of scientists who have evidence to suggest that the production capacity in 2020 will be close to 80% what it is now. Slate has a good article about The Innocence Project which uses DNA tests to overturn false convictions [8], it s scarey how badly the justice system works. Rachel Sussman gave an interesting talk about the World s Oldest Living Things [9], nothing less than 2000 years old is included. Nicholas Negroponte gave an interesting EG 2007 talk about the OLPC project [10]. While some of the content about OLPC production is a little dated the OLPC history is relevant and his personal opinions regarding the benefits that children receive from computers are insightful. Jayne Poynter gave an interesting TED talk about life in Biosphere 2 [11]. Her work on mini-biospheres is also interesting. Let s hope we get a permanent Biosphere running outside the Earth sometime soon. Sheryl WuDunn gave an informative TED talk titled Our Century s Greatest Injustice about the plight of women in most of the world [12]. Daniel Kahn Gillmor wrote a good article about the use of ssh-agent [13]. You really should use it to secure your ssh keys. Mark Shuttleworth has described the development process for the Ubuntu font [14]. This is a very interesting project and IMHO one of the most significant things that Ubuntu has done. Prior to this an entirely closed development process has been used for font development. Now we are getting a major new font family developed with a free and open process and with some new legal developments for a font license! One thing to note that this project appears to have involved a lot of work from professional font designers, it sounds like Canonical spent a significant amount of money on this project.

27 September 2010

Daniel Kahn Gillmor: You should be using ssh-agent

If you're not using ssh-agent to authenticate yourself to SSH servers, you should be. (i'm assuming you're already using PubkeyAuthentication; if you're still using PasswordAuthentication, ChallengeResponseAuthentication or KbdInteractiveAuthentication, fix that please). You should use ssh-agent for a number of reasons, actually, but the simplest is this: when you authenticate to a text-based channel on a remote server, you should never have to type anything about that authentication into the channel that will eventually be controlled by the remote server. That's because a malicious server could simply accept your connection as an anonymous connection and print out the exact prompt you're expecting. Then, whatever you're typing goes into the remote server instead of into your authentication scheme. and congrats, you just gave away the passphrase for your key. With ssh-agent, you talk first to your agent. Then, you talk to the server and your ssh client talks to the agent. Your keys and your passphrase are never exposed. the second reason is that the agent is a much smaller piece of code than the ssh client, and it doesn't talk to the network at all (unless you force it to). It holds your key and never releases it to querying processes; It even runs in a protected memory space so other processes can't peek at it. So if this protected, isolated agent is what holds your key, you're in much better shape than if a non-protected, larger, network-active process (the ssh client) has direct access to your secret key material. The third reason is that it's just more convenient -- you can put a key in your agent, and ask it to prompt you when its use is requested. you don't actually need to re-type your passphrase each time. you can just hit enter or type "yes". And if that scares you security-wise then you can put the key in for a limited period of time, as well. (btw, you should be using the ssh-agent that ships with OpenSSH, probably not the implementation offered by gnome, which doesn't offer a confirmation prompt, doesn't run in protected memory space, and links in a ton more libraries) So how do you use the agent? It's probably already installed and running on your computer if you run a desktop with debian or another reasonable free operating system. Query what keys are in your agent:
0 dkg@pip:~$ ssh-add -l
The agent has no identities.
1 dkg@pip:~$ 
Add a standard OpenSSH secret key to your agent, prompting for confirmation after each use:
0 dkg@pip:~$ ssh-add -c ~/.ssh/id_rsa
Enter passphrase for /home/dkg/.ssh/id_rsa: your nice long passphrase here
Identity added: /home/dkg/.ssh/id_rsa (/home/dkg/.ssh/id_rsa)
The user must confirm each use of the key
0 dkg@pip:~$ 
(if you drop the -c, you will not be prompted at each use) Add a standard OpenSSH secret key to your agent, with a lifespan of one hour (3600 seconds)
0 dkg@pip:~$ ssh-add -t 3600 ~/.ssh/id_rsa
Enter passphrase for /home/dkg/.ssh/id_rsa: your nice long passphrase here
Identity added: /home/dkg/.ssh/id_rsa (/home/dkg/.ssh/id_rsa)
Lifetime set to 3600 seconds
0 dkg@pip:~$ 
(note that you can combine the -t $SECONDS and -c flags to get key that is time-constrained and requires a confirmation prompt at each use) Add a monkeysphere-style key (an authentication-capable subkey from your GnuPG secret keyring) to the ssh-agent (this will prompt you for your GnuPG passphrase with a graphical ssh-askpass program during this keyload, if such a program is available), for one hour:
0 dkg@pip:~$ monkeysphere subkey-to-ssh-agent -t 3600
Identity added: Daniel Kahn Gillmor <dkg@fifthhorseman.net> (Daniel Kahn Gillmor <dkg@fifthhorseman.net>)
Lifetime set to 3600 seconds
0 dkg@pip:~$ 
If you don't already have such a subkey, but you want to use the monkeysphere, you'll need to run monkeysphere gen-subkey to create one first. Note also that you can use both -c and -t $SECONDS with monkeysphere subkey-to-ssh-agent, just like they are used with ssh-add. Remove all keys from your running agent:
0 dkg@pip:~$ ssh-add -D
All identities removed.
0 dkg@pip:~$ 
I hope this is helpful to people! Tags: security, ssh, ssh-agent, tip

21 September 2010

Daniel Kahn Gillmor: hotmail thinks powerpc means mobile

Apparently, live.com thinks that any browser coming from a ppc architecture is a mobile device. This sucks for the users of the hundreds of thousands of powerpc desktops still in service. I don't use hotmail myself, but i do support people who use it. I set one of my clients up with debian squeeze on their PPC machine because all the proprietary vendors have basically given up on that architecture -- debian represents the best way to get modern tools on these machines (and other machines too, but that's a different argument). However, this client couldn't get to their hotmail account, despite using the latest version of iceweasel (3.5.12). They were directed to a crippled interface that didn't include the ability to attach files, and was a gross waste of the desktop screen space. It appears to be the "mobile" version of live.com's services. However, the same version of iceweasel on an i686 test machine could access the standard version of hotmail with no trouble. My friend jeremyb helpfully suggested fiddling with the User Agent string exported by the browser. Some experimentation shows that the presence of the string "ppc" within any parenthetical expression in the UA makes live.com show the crappy interface. You can try it yourself (if you have a hotmail account) on your x86 or amd64 machine by adding (ppc) to the default valule of general.useragent.extra.firefoxComment in about:config. Stupid stupid stupid. I'd like to have fixed this by overriding the browser's reported architecture (or simply by removing it -- why does a web server need to know the hardware architecture of my client?). But there doesn't appear to be a way to do that with the way that mozilla constructs the UA. Instead, i needed to add a new string key named general.useragent.override which is not exposed by default in about:config. This raises some questions: Any ideas?Tags: browser, hotmail, powerpc, ppc, useragent, wtf

22 August 2010

Christine Spang: DebConf 10 postmortem and SD talk followup

DebConf launched with a bang the day I arrived by bike I was up until 3am meeting and greeting in the basement lounge of the Carmen Columbia dormitory, where I was staying. No idea how I managed to be so awake for that. The rest of the week alternated between hacking like crazy on code for my talk and spending a lot of time socializing with Debian folks new and old. For the day trip to Coney Island, I joined the dkg-led bike expedition which ended up running to nearly 30 miles, which was a bit more than expected. The fact that this was all in actual dense city really drove home the scale difference between Boston and New York (I'd never been to NYC before this). We took several breaks to lounge around and eat and drink, so it took quite a long time even given the distance. I hadn't planned on seeing the baseball game that was a part of the trip, but I ended up going anyway and it turns out that a bunch of geeks at a minor league game is actually quite a lot of fun! I hope someone else will put some pictures from the bike ride and game online soon, since I didn't really take any myself. This was the first DebConf where I gave a talk, which resulted in me skipping almost all of the other talks, because my talk was on the last day and I reaaally wasn't ready at the start of the conference due to the rest of life being pretty crazy this summer. I missed some things I would have liked to see because of this, but ultimately I think it was worth it. The good news is: it went well! I was nervous until I actually started talking (never given a talk at a conference before), and then it was fine. If you missed it, the talk video is on the web in low and high quality; slides are here. The audience was great there were excellent questions and people were excited and interested in the project. I couldn't have asked for a better reception. After the talk finished I spent some time aisle-chatting with some folks, and totally failed to recognize Joey despite having met him before, because he'd shaved off his hair. DebConf was, like usual, both inspiring and exhausting. I haven't managed to follow up on much that happened during the conference yet. I definitely plan to do so, though, now that real life is calming down again. I'd hate to waste the post-conference buzz about SD. My todo list includes:

21 August 2010

Jonathan Wiltshire: Improving the resilience of the Web of Trust

(this is a post in two parts) At the BoF of GPG Key Management: Best Practices, dkg had two immediate recommendations for improving WoT resilience: Regular Refreshes This is important because unless keys are refreshed, changed expirations and (more importantly) revocations on keys are never made known to you. Making regular refreshes is easy, though I was surprised to hear people had experienced difficulty with it it s a simple entry to cron, like: 0 0 * * 1 /usr/bin/gpg --refresh-keys to run at midnight on a Sunday, or any other convenient time. Append > /dev/null to make it silent unless there are errors. Provide a key server This is nearly as trivial a refreshing the keyring, but it does take some disk space (on my machine, 4G for the decompressed dump from another server, 7.4G for the database itself and a few hundred MB for other stuff. Supposedly the dump can go after a full database build, but as I m not constrained I haven t bothered). The sks package in Lenny and Squeeze manages the database and answers to clients. However, take care to follow README.Debian carefully, since if the permissions on /var/lib/sks or /var/log/sks aren t set right it will silently fail and much head-scratching will ensue. Building the database for the first time takes some hours even on a reasonably quick machine, and takes a lot of CPU time, so it s probably best done overnight. Once sks is running, it needs to catch up from the date of the key dump to now (which is why a recent dump is always a good start). To do that, it should be listed in the pool.sks-keyservers.net and therefore needs gossip peers. http://code.google.com/p/sks-keyserver/wiki/Peering is an excellent guide, though the version of sks in Lenny does not support multiple listening addresses for either the recon or hkp processes. When a small enough delta between the server and its peers is reached, it s included in the DNS pool for use by other clients.
Comments flattr this!

19 July 2010

Jamie McClelland: Growing disks: upgrading from 1 TB to 2 TB disks

We've got a fairly layered set of software that runs beneath our file systems: RAID1 followed by cryptsetup, followed by lvm. It works remarkably well. However, when it's time to do something like enlarge a partition or disk, it gets complicated. We have a server (hubert) that was running two 1 TB disks in a RAID1. We typically create two partitions: one small boot partition and one large partition the uses the rest of the available disk (with 2 TB disks we've added a third partition for grub). Last month I replaced one of the disks with a 2 TB disk, partitioned the new disk with similar a partition scheme (except that the "rest of the disk" partition was twice the size as the original one), and then added the new partitions to the RAID (mdadm /dev/md0 --add /dev/sda1; mdadm /dev/md1 --add /dev/sda2). Yesterday, I replaced the 2nd 2 TB disks, and again added the new partitions. The result was a fairly painless transition from 1 to 2 TB disks with minimal downtime. The problem - though - is that we're not using all the disk space available, since md1, the RAID array that uses the "rest of the disk" partition, is only about 1 TB when it should be just under 2 TB. That problem was easy enough to solve by growing the RAID device with:
mdadm -G /dev/md1 -z max
It took a while, but eventually, running:
cat /proc/mdstat
And
cat /proc/partitions
Reported the right sizes for md1. After RAID, comes cryptsetup (/dev/md1 is the source device for md1_crypt). So, the next command was:
cryptsetup resize md1_crypt
In the output of:
cat /proc/partitions
I saw the change reflected in the dm-0 device. Since md1_crypt is used as the physical volume for a volume group, the next step was to resize the physical volume:
pvresize /dev/dm-0
This change was reflected with:
pvs
Now, the command:
vgs
Displayed a full 931 GB available that wasn't there before. Next up: resizing the logical volume:
lvresize --size 1700GB vg_hubert0/blanco_members
And, last but not least, was the file system:
 resize2fs /dev/mapper/vg_hubert0-blanco_members
See also dkg's article on resizing crypt partitions.

19 May 2010

Daniel Kahn Gillmor: NYC SYEP still requires Microsoft Software

A year ago, i wrote about how New York City's Summer Youth Employment Program (SYEP) requires the use of Internet Explorer to apply online (and it even appears to require IE just to download the PDF of the application!) Sadly, the situation has not changed, a year later. Today, I'm writing to Dan Garodnick, Chair of The City Council's Committee on Technology (and the rest of the committee members), Carole Post, Commissioner of DoITT (the city's Department of Information Technology and Telecommunications), and Jeanne B. Mullgrav, Commissioner of DYCD (the Department of Youth and Community Development, which runs SYEP). Here's what i wrote:
For the last two years at least, the DYCD's Summer Youth Employment Program
(SYEP) has been only available to users of Internet Explorer:
 https://application.nycsyep.com/
Internet Explorer (IE) is only made by Microsoft, and is only available for
people running Microsoft operating systems.  Users of other operating systems,
such as GNU/Linux, Macintosh, or others cannot access the SYEP application
process.  Even users of Windows who care about their online security or simply
desire a different web browsing experience might prefer to avoid Internet
Explorer.
Not only is the online form inaccessible from browsers other than IE, even
retrieving a copy of the PDF to print out and fill in manually is unavailable
for web browsers other than IE.
What is the city's policy is on access to government sites?  Is it city policy
to mandate a single vendor's software for access to city resources?  Should NYC
youth be required to purchase software from Microsoft to be able to apply
for the Summer Youth Employment Program?
The sort of data collection needed by such an application is a mainstay of the
standards-based web, and has been so for over 15 years now.  There is no reason
to require particular client on an open platform.  I can point you toward
resources who would be happy to help you make the system functional for users
of *any* web browser, if you like.
I raised this issue over a year ago (see nyc.gov correspondence #1-1-473378926,
and a public weblog posted around the same time [0]), and got no effective
remedy.  It's worrisome to see that this is still a problem.
Please let me know what you plan to do to address the situation.
Regards,
	--dkg
[0] https://www.debian-administration.org/users/dkg/weblog/47
Feel free to send your own message to the folks above (especially helps if you live in or near NYC) Finally, Carole Post, the head of DoITT will also be present at a panel tonight in Soho, which i'm unfortunately be unable to attend. If you go there, you might ask her about the situation.Tags: policy

26 April 2010

Debian News: Brief Updates: d-i l10n status, DebConf, snapshot service, GSOC

25 April 2010

Daniel Kahn Gillmor: Talks and tracks at debconf 10

I'm helping out on the talks committee for debconf 10 this summer in NYC (so yes, i'm going to be here for it, even though i don't have that little badge thingy). This is a call for interested folks to let us know what you want to see at debconf! Talks If you haven't already, submit your proposal for a talk, performance, debate, panel, BoF session, etc! We know you've got good ideas, and the final call for contributions went out yesterday, due in less than a week. Please propose your event soon! Tracks Also, we want to introduce Tracks as a new idea for debconf this summer. A good track would thematically group a consecutive set of debconf events (talks, panels, debates, performances, etc) to encourage a better understanding of a broader theme. For this to work, we need a few good people to act as track coordinators for the areas where they are knowledgeable and engaged. A track coordinator would have a chance to set the tone and scope for their track, schedule events, assemble panels or debates, introduce speakers, and report back at the end of debconf to the larger gathering. We also hope that a coordinator could identify potential good work being done in their area, encourage people to submit relevant events for debconf, and shepherd proposals in their track through the submission process. Are you interested in coordinating a track on some topic? Or do you have a suggestion for someone else who might do a good job on a topic you want to see well-represented at debconf? You can contact the talk committee privately if you have questions at talks@debconf.org, or you can contact the whole team publicly at debconf-team@lists.debconf.org. Some ideas about possible tracks: We can't guarantee that any particular track will happen at dc10, but we can guarantee that it won't happen if no one proposes it or wrangles the relevant events together. Help us make this the best debconf ever and make sure that your own topical itch gets scratched! Tags: debconf, debconf10

23 March 2010

Daniel Kahn Gillmor: Avoiding erroneous OpenPGP certifications

i'm aware that people don't always take proper measures during mass OpenPGP keysignings. Apparently, some keys even get signed with no one at the keysigning present speaking for that key (for example, if the key was submitted to the keysigning via online mechanisms beforehand, but the keyholder failed to show up). Unverified certifications are potentially erroneous, and erroneous certifications are bad for the OpenPGP web of trust. Debian and other projects rely on the OpenPGP web of trust being reasonable and healthy. People should make a habit of doing proper verifications at keysignings. People who make unverified certifications should probably be made aware of better practices. So for future keysignings, i may introduce a key to the set under consideration and see what sort of OpenPGP certifications that key receives. I won't pretend to hold that key in person, won't speak for it, and it won't have my name attached to it. But it may be on the list. Depending on the certifications received on that key (and the feedback i get on this blog post), i'll either publish the list of wayward certifiers, or contact the certifiers privately. Wayward certifiers should review their keysigning practices and revoke any certifications they did not adequately verify. Remember, at a keysigning party, for each key: After the keysigning, when you go to actually make your OpenPGP certifications: Feedback welcome! Tags: keysigning, openpgp, tip

21 January 2010

Daniel Kahn Gillmor: TCP weirdness, IMAP, wireshark, and perdition

This is the story of a weirdly unfriendly/non-compliant IMAP server, and some nice interactions that arose from a debugging session around it. Over the holidays, i got to do some computer/network debugging for friends and family. One old friend (I'll call him Fred) had a series of problems i managed to help work through, but was ultimately basically stumped based on the weird behavior of an IMAP server. Here's the details (names of the innocent and guilty have been changed), just in case it helps other folks in at least diagnosing similar situations. the diagnosis The initial symptom was that Fred's computer was "very slow". Sadly, this was a Windows machine, so my list of tricks for diagnosing sluggishness is limited. I went through a series of questions, uninstalling things, etc, until we figured it would be better to just have him do his usual work while i watched, kibitzing on what seemed acceptable and what seemed slow. Quite soon, we hit a very specific failure: Fred's Thunderbird installation (version 2, FWIW) was sometimes hanging for a very long period of time during message retrieval. This was not exhaustion of the CPU, disk, RAM, or other local resource. It was pure network delay, and it was a frequent (if unpredictable) frustrating hiccup in his workflow. One thought i had was Thunderbird's per-server max_cached_connections setting, which can sometimes cause a TB instance to hang if a remote server thinks Thunderbird is being too aggressive. After sorting out why Thunderbird was resetting the values after we'd set them to 0 (grr, thanks for the confusing UI, folks!), we set it to 1, but still had the same occasional, lengthy (about 2 minutes) hang when transfering messages between folders (including the trash folder!), or when reading new messages. Sending mail was quite fast, except for occasional (similarly lengthy) hangs writing the copy to the sent folder. So IMAP was the problem (not SMTP), and the 2-minute timeouts smelled like an issue with the networking layer to me. At this point, i busted out wireshark, the trusty packet sniffer, which fortunately works as well on Windows as it does on GNU/Linux. Since Fred was doing his IMAP traffic in the clear, i could actually see when and where in the IMAP session the hang was happening. (BTW, Fred's IMAP traffic is no longer in the clear: after all this happened, i switched him to IMAPS (IMAP wrapped in a TLS session), because although the IMAP server in question actually supports the STARTTLS directive, it fails to advertise it in response to the CAPABILITIES query, so Thunderbird refuses to try it. arrgh.) The basic sequence of Thunderbird's side of an initial IMAP conversation (using plain authentication, anyway) looks something like this:
1 capability
2 login "user" "pass"
3 lsub "" "*"
4 list "" "INBOX"
5 select "INBOX"
6 UID fetch 1:* (FLAGS)
What i found with this server was that if i issued commands 1 through 5, and then left the connection idle for over 5 minutes, then the next command (even if it was just a 6 NOOP or 6 LOGOUT) would cause the IMAP server to issue a TCP reset. No IMAP error message or anything, just a failure at the TCP level. But a nice, fast, responsive failure -- any IMAP client could recover nicely from that by just immediately opening a new connection. I don't mind busy servers killing inactive connections after a reasonable timeout. If it was just this, though, Thunderbird should have continued to be responsive. the deep weirdness But if i issued commands 1 through 6 in rapid succession (the only difference is that extra 6 UID fetch 1:* (FLAGS) command), and then let the connection idle for 5 minutes, then sent the next command: no response of any kind would come from the remote server (not even a TCP ACK or TCP RST). In this circumstance, my client OS's TCP stack would re-send the data repeatedly (staggered at appropriate intervals), until finally the client-side TCP timeout would trigger, and the OS would report the failure to the app, which could turn around and do a simple connection restart to finish up the desired operation. This was the underlying situation causing Fred's Thunderbird client to hang. In both cases above (with or without the 6th command), the magic window for the idle cutoff was a little more than 300 seconds (5 minutes) of idleness. If the client issued a NOOP at 4 minutes, 45 seconds from the last NOOP, it could keep a connection active indefinitely. Furthermore, i could replicate the exact same behavior when i used IMAPS -- the state of the IMAP session itself was somehow modifying the TCP session behavior characteristics, whether it was wrapped in a TLS tunnel or not. One interesting thing about this set of data is that it rules out most common problems in the network connectivity between the two machines. Since none of the hops between the two endpoints know anything about the IMAP state (especially under TLS), and some of the failures are reported properly (e.g. the TCP RST in the 5-command scenario), it's probably safe to say that the various routers, NAT devices, and such were not themselves responsible for the failures. So what's going on on that IMAP server? The service itself does not announce the flavor of IMAP server, though it does respond to a successful login with You are so in, and to a logout with IMAP server logging out, mate. A bit of digging on the 'net suggests that they are running a perdition IMAP proxy. (clearly written by an Aussie, mate!) But why does it not advertise its STARTTLS capability, even though it is capable? And why do some idle connections end up timing out without so much as an RST, when other idle connections give at least a clean break at the TCP level? Is there something about issuing the UID command that causes perdition to hand off the connection to some other service, which in turn doesn't do proper TCP error handling? I don't really know anything about the internals of perdition, so i'm just guessing here. the workaround I ultimately recommended to Fred to reduce the number of cached connections to 1, and to set Thunderbird's interval to check for new mail down to 4 minutes. Hopefully, this will keep his one connection active enough that nothing will timeout, and will keep the interference to his workflow to a minimum. It's an unsatisfactory solution to me, because the behavior of the remote server still seems so non-standard. However, i don't have any sort of control over the remote server, so there's not too much i can do to provide a real fix (other than point the server admins (and perdition developers?) at this writeup). I don't even know the types of backend server that their perdition proxy is balancing between, so i'm pretty lost for better diagnostics even, let alone a real resolution. some notes I couldn't have figured out the exact details listed above just using Thunderbird on Windows. Fortunately, i had a machine with a decent OS available, and was able to cobble together a fake IMAP client from a couple files (imapstart contained the lines above, and imapfinish contained 8 LOGOUT), bash, and socat. Here's the bash snippet i used as a fake IMAP client:
spoolout()   while read foo; do sleep 1 && printf "%s\r\n" "$foo" ; done  
( sleep 2 && spoolout < imapstart && sleep 4 && spoolout < imapfinish && sleep 500 )   socat STDIO TCP4:imap.fubar.example.net:143
To do the test under IMAPS, i just replaced TCP4:imap.fubar.example.net:143 with OPENSSL:imap.fubar.example.net:993. And of course, i had wireshark handy on the GNU/Linux machine as well, so i could analyze the generated packets over there. One thing to note about user empowerment: Fred isn't a tech geek, but he can be curious about the technology he relies on if the situation is right. He was with me through the whole process, didn't get antsy, and never tried to get me to "just fix it" while he did something else. I like that, and wish i got to have that kind of interaction more (though i certainly don't begrudge people the time if they do need to get other things done). I was nervous about breaking out wireshark and scaring him off with it, but it turned out it actually was a good conversation starter about what was actually happening on the network, and how IP and TCP traffic worked. Giving a crash course like that in a quarter of an hour, i can't expect him to retain any concrete specifics, of course. But i think the process was useful in de-mystifying how computers talk to each other somewhat. It's not magic, there are just a lot of finicky pieces that need to fit together a certain way. And Wireshark turned out to be a really nice window into that process, especially when it displays packets during a real-time capture. I usually prefer to do packet captures with tcpdump and analyze them as a non-privileged user afterward for security reasons. But in this case, i felt the positives of user engagement (how often do you get to show someone how their machine actually works?) far outweighed the risks. As an added bonus, it also helped Fred really understand what i meant when i said that it was a bad idea to use IMAP in the clear. He could actually see his username and password in the network traffic! This might be worth keeping in mind as an idea for a demonstration for workshops or hacklabs for folks who are curious about networking -- do a live packet capture of the local network, project it, and just start asking questions about it. Wireshark contains such a wealth of obscure packet dissectors (and today's heterogenous public/open networks are so remarkably chatty and filled with weird stuff) that you're bound to run into things that most (or all!) people in the room don't know about, so it could be a good learning activity for groups of all skill levels. Tags: debugging, imap, perdition, wireshark

21 December 2009

Daniel Kahn Gillmor: January 2010 Bug-Squashing Party NYC

We're going to have a Bug-Squashing Party at the end of January 2010 in New York City. If you live in or around the tri-state area (or want to visit), are interested in learning about the process, meeting other debian folk, or just squashing some bugs in good company, you should come out and join us!
Where:
Brooklyn, New York, USA
When:
January 29th, 30th, and maybe 31st of 2010
Why:
Because them bugs need squashing!
If you plan on coming, please either sign up on the wiki page, or at least mail one of the good folks listed there, or pop into #debian-nyc on irc.oftc.net's IRC network. Tags: bsp

Daniel Kahn Gillmor: dd, netcat, and disk throughput

I was trying to dump a large Logical Volume (LV) over ethernet from one machine to another. I found some behavior which surprised me.fun constraints(how do i get myself into these messes?)This entry has been truncated read the full entry.Tags: lvm, netcat

Next.

Previous.