Search Results: "spa"

30 November 2008

Adeodato Sim : Snow for the guy from the South

Ah, seems the days of posting snow pictures in Planet Debian have gone, and in any case I don’t have pictures, but I’ll blog about snow nevertheless. It never snows in Alicante, my home town. It’s by the sea in the south-east of Spain, and the winters are very mild there. I had only seen or touched snow once in my life, when I was about 7 and there was an outing in school to go and see the snow in some nearby mountain. I’m told it was crappy snow anyway. Today we returned from the QA meeting in Extremadura, arriving in Madrid really early in the morning. Knowing this, I had arranged to stay with friends Sunday and Monday, spending the night at agi@’s place. The house is in some mountain, at about 1800 meters over sea level, and there was plenty of snow. There was also a slope, and a plastic sled, so we did the obvious. It was fun, it made me happy, and I should pay Alberto another visit some other time with more appropriate clothing.

Benjamin Mako Hill: Kosher

This bottle was found in Mika's biosafety level 2 laboratory. Isoamyl alcohol Certified Kosher To address any confusion, isoamyl alcohol is not drinking alcohol and this bottle was bought for use in a scientific lab from a scientific lab supply company. Explanations are welcome.

Biella Coleman: Hacking Spaces, the Spaces of Hacking

Space and place has long been important to hackers. Whether it was/is the university lab, the workplace, the hacker con, or the particularly high-tech city, hackers congregate and meet face to face, often and everyday. One semi-new development has been the explosion and proliferation of hacker labs and spaces, such as Noisebridge located in the Mission district of San Francisco Foulab in Montreal. I recently got back from San Francisco and was able to spend a few nights at Noisebridge and was jaw droppingly impressed. The space is, well, spacious and nice (and located right by the Bart, a real +++++), but more important is that it is a thriving collective with all sorts of geeky participants and they have just souped up the space with all sorts of equipment, from the usual suspects (lots of computers) to lots of electronic gear such as oscilloscopes. What I also was impressed by was not only the blizzard of events but the open and accessible nature of the organization, which seemed to sit in some contrast to NYC’s hacker collective, NYC Resistor. Like so many organizations in this metropolis, they apparently are lacking in space to grow and the word on the streets (which I cannot confirm or deny as I have had very little contact with them but have heard this repeatedly) is that the organization has had a tough time letting in new members. Some folks are understanding of this given their space limitation, others have been less kind, and have referred to the group as a clique. There are already a few other initiatives under way to find a larger space so as to accommodate a more open, participatory atmosphere for a hacker space (sign up for the email list here but I imagine that over time the culture and developments of NYC Resistor will also change as new spaces develops and do hope that this creates the conditions for more access rather than less (and again, I know next to nothing of the situation though I suspect space plays a real limiting role as it does with nearly everything else in NYC). And thankfully Rose White, a NYC-based sociology graduate student at CUNY, is paying close attention to the rise and development of these hacker spaces. She is well underway doing her dissertation dissertation on these hack spaces and I really look forward to her work. update: The Axis of Awesome is a hacker space in L.A. and as far as I am concerned, dons the best name.

29 November 2008

Stefano Zacchiroli: PTS SOAP interface

Hacking the PTS using SOAP I've finally found some time, thanks to the Extremadura QA + ftpmaster + i18n meeting, to release the first draft of the SOAP interface to the PTS. You probably already got the idea, which is quite simple after all. The PTS, as always, gathers information about (source) packages from various sources and melds them together into web pages. With a SOAP interface you just gain the ability of accessing such information from your programs via SOAP. A proof of concept is overdue: 
    $ cat ./test.py 
    #!/usr/bin/python
    import SOAPpy
    url = 'http://packages.qa.debian.org/cgi-bin/soap-alpha.cgi'
    ws = SOAPpy.SOAPProxy(url)
    print ws.versions(source="ocaml")['unstable']
    print ws.uploaders(source="ocaml")[1]['name']
    $ ./test.py 
    3.10.2-3
    Stefano Zacchiroli
Everything is still in alpha version, but already working. Some links which you might find useful: Please let me know if / how you are using of the SOAP interface, it will help for future developments. How it works Just a few comments on how it works. You might remember that a while ago I've made all PTS pages XHTML-valid. Well, on top of it I've implemented something along the lines microformats, that just make a clever use of ingredients already available in XHTML like classes and unique identifiers. Having that, a "reshuffling" of the information already available on the web pages (which are now kinda "semantically" tagged) can be obtained by evaluating a handful of XPaths on the (not anymore) final XHTML pages. That's precisely what the CGI implementing the SOAP API is currently doing. This way one can avoid implementing two different access paths to the information collected by the PTS: one for rendering, and one for SOAP (no, reusing the rendering one for SOAP was not an option, given that it was originally written in XSLT). The only annoyance I've encountered is that XPath is completely unaware of the "CSS-like" semantics of XHTML classes, which states that classes are space-separated list of class names, to be interpreted as sets. That means that to check whether an element belongs to a given class you need to fiddle with substring matches on the class attribute (which is quite crappy).

Chris Lamb: Re: A timeline view in Django

Alex Gaynor writes about "timeline views" in Django. The abstract problem is to merge already-sorted lazily evaluated lists by some specified attribute. My solution is similar to Malcolm's suggestion in the comments and improves upon Alex's in that has superior space considerations, as well as supporting reverse ordering with the usual Django syntax: 
import heapq
def merge_querysets(*querysets, **kwargs):
    order_by = kwargs.get('order_by', None)
    if order_by is None:
        raise ValueError, "Must specify order_by in keyword arguments"
    # Are we sorting in reverse?
    reverse = False
    if order_by.startswith('-'):
        order_by = order_by[1:]
        reverse = True
    class Wrapper(object):
        """
        Hacky wrapper class for custom sorting.
        """
        def __init__(self, sortval, val):
            self.sortval = sortval
            self.val = val
        def __cmp__(self, other):
            if reverse:
                return cmp(other.sortval, self.sortval)
            else:
                return cmp(self.sortval, other.sortval)
    iterables = map(iter, querysets)
    heap = []
    while True:
        for it in iterables:
            try:
                val = it.next()
                sortval = val.getattr(order_by)
                heapq.heappush(heap, Wrapper(sortval, val))
            except StopIteration:
                # This QuerySet has been exhausted
                iterables.remove(it)
                continue
        if not iterables:
            # All QuerySets have been exhausted; just yield from the heap
            # to save cost of insertion.
            for item in heap:
                yield item.val
            return
        yield heapq.heappop(heap).val
def my_view(request):
    tickets = Ticket.objects.order_by('-create_date')
    wikis = WikiEdit.objects.order_by('-create_date')
    changesets = Changeset.objects.order_by('-create_date')
    objs = merge_querysets(tickets, wikis, changesets, order_by='-create_date')
    return render_to_response('my_app/template.html',  
        'objects': objs,
     )

Sven Mueller: Re: Silly translations

In Silly translations, Gintautas Miliauskas wrote about some rather silly translations. This reminded me of a finding by a colleague a few days ago.
He had recently updated his Ubuntu installation to KDE4 (using a german locale). After this, he reassigned a hotkey (to CTRL-SPACE). However, KDE showed: STRG-Weltraum (ctrl-”outer space”) instead of STRG-Leertaste (ctrl-space key) :-) UPDATE: Since Frederik (see comments) reminded me of the fact I already knew:
This silly translation is not the fault of the german KDE members, but a bug in the Ubuntu package, which is taking more or less random translations from Rosetta.

28 November 2008

Theodore Ts'o: An ethical question involving ebooks

I recently purchased a short story from Fictionwise, which was not DRM’ed, so I could easily get it into a form where I could read it on my Sony eReader. Thanks to that short story, I was introduced to an author, and a character, which I found very engaging. When I decided to find out more about the character, I found that the author had written two additional short stories, and three additional novels many years ago, but has since stopped writing any more books involving that character. Furthermore, the novels have gone out of print, and are only available from amazon.com as used books. Unfortunately, I travel a lot. So much so, that one of the few times that I have time to read is when I’m traveling. And I really dislike having to haul dead-tree versions of my favorite novels around; they take up far too much weight and space in my carry-on luggage. Unfortunately, these out-of-print novels were published by a Neanderthal Publishing company who hasn’t made any of the books available in ebook format, DRM’ed or no. Grumpy, I searched on Internet, and found all three novels were easily available for free download — in a pirated form, of course. Should I download them and convert them into a form which would allow me to read them on my Sony eReader? Well, according to Russell Davis, former chair (and now president of the Science Fiction Writer’s Association) of the SFWA’s Copyright Committee, “electronic infringement is theft”. From a legal perspective, I suppose that is true. And given that as an Open Source programmer, I depend on Copyright Law to assure that my wishes as an author are upheld, it would be hypocritical for me assume that I should be able to ignore Copyright Law just because it is inconvenient. And yet… from a moral perspective, who has really lost anything? The argument made by Russell Davis is that infringement is bad because it is “harming authors and author estates”. Jerry Pournelle has indignantly proclaimed that e-piracy goes against a “specific (and very stern) Biblical injunction against stealing from widows and orphans”. Of course, in this case, the author is still alive (and is female, although I suppose stealing from widowers would be just as bad). Also, given that the author has publically stated she doesn’t plan to write any more books involving this character (since some of her more psychotic readers sent her death threats as a result of reading said books), the publisher is highly unlikely to re-release said novels — and if I buy used dead-tree versions of said novels, the author doesn’t receive any additional royalties. So, then, where is the moral bright line? Somewhere along this continuum, we’ve crossed over from the light-side to the dark-side. Setting aside the observation that the Neanderthal attitudes and business practices of the publisher involved has made it impossible for me to legitimiately follow the law, enjoy the novels, and direct money to the author via royalty payments — what do you think is the morally correct course of action? And why? And if you don’t mind saying so publically, roughly what generation are you from (i.e., Baby Boomer, Gen X, Gen Y, etc.)? I’m curious how attitudes are changing based on age, and whether folks who are currently in college might differ from those who can remember a time when the Internet didn’t exist… Update: I’ve posted a follow-up to this post here.

Raphael Geissert: how to perform archive-wide stuff without wasting time and bandwidth, or without setting up your own local mirror

Joerg Jaspert: Ftpmaster meeting in Extremadura

Yay, long time since the last ftpmaster blog post. Time to write a new one. It’s not that I haven’t done anything since my last post, it just wasn’t only ftpmaster related. Right now I’m somewhere in Extremadura, Spain, at the Ftpmaster / I18N / QA Meeting. This meeting is near to the location of DebConf9, so yesterday we visited that venue. Still have to upload the pictures I took. It’s pretty nice, but I fear we might have a problem with wheelchairs. While the venue is very well prepared for wheelchair users, like having ramps or electric elevator platforms for every little staircase, I see one big problem in the size of the elevator going from the ground floor up to the first level. We don’t have exact measurements currently, just our rough guess from looking at it, but this is something we definitely have to look in. Something similar applies to the sleeping place. They do have a room for disabled people, but we couldn’t look into it when we where there, as it was in use (And it is pretty unpolite to just move into an occupied room). So we need to get an exact floor plan and hopefully also pictures for it later on. Luckily, the sleeping place is a much smaller issue, if it doesn’t fit we could think about placing disabled people in a Hotel. More ftpmaster related we had a lot of discussions about one of our main agenda points for this meeting, tdebs. The more we discuss it, the more we all hate it. Its a topic that touches nearly everything in Debian and will redefine quite a lot of places/actions/behaviours, so it needs a lot of thought. Which is what we start here. We wont be able to get anything substantly implemented here, but I think we can get to a common understanding of a specification and a Todo. After all the schedule currently has “lenny + 2” for this, its not a light weight step. But thats not all, would be boring if we would have only one topic. Us ftpteam also intends to/does work on other issues, like a redesign of dak and the database below it. And today we issued a statement about the AGPL. Not ftpmaster related, but also a nice work done here in Extremadura, is the setup for mirrors of packages.debian.org. This enables us to mirror the part of packages.debian.org to other .debian.org hosts thats responsible for the web view, and as such allows us to have redundancy in case of errors, load balancing and all those nice things. Comments: 2

Russell Coker: Leaving Optus

Today I phoned Optus to disconnect my Internet service. Some time ago I got an Internode [1] SOHO connection. This gave me a much faster upload speed (typically 100KB/s) compared with Optus having a maximum of 25KB/s. Also Internode has better value for large data transfer (where “large” in Australia means 25GB per month) and I get a static IP address. I also get unfiltered Internet access, Optus blocks outbound connections to port 25 which forced me to ssh to another server to test my clients’ mail servers. But the real reason for leaving Optus is based on events two years ago. When I first signed up with Optus four years ago my contract said “unlimited uploads“. What they really meant was “upload as much as you want but if you transfer more than 8KB/s for any period of time you get disconnected“. They claimed that running a default configuration of BitTorrent was a DOS (Denial of Service) attack (the only part of their terms of service that even remotely permitted them to disconnect me). So I was quite unhappy when they cut me off for this. What really offended me was the second time they cut my connection. I had been running BitTorrent on Friday and Saturday, and they cut my connection off on Wednesday. Once it was determined that the issue was uploads we had a bit of a debate about when my BitTorrent session was terminated, it was my clear memory of using killall to end BitTorrent during a commercial break of a TV show on the Saturday night vs the Optus idiot claiming they had a record of me doing big uploads on the Sunday. But I let the help desk person think that they had won that debate in order to focus on the big issue, why large uploads on a Saturday (or a Sunday) should result in a loss of service on Wednesday (three or four days later). They said “it was to teach you a lesson“! The lesson I learned is that it is best to avoid doing business with Optus. I didn’t immediately cancel my contract, if you have both phone and Internet service through Optus they do offer a reasonable deal (there are a variety of discounts that are offered if you have multiple services through them). When discussing this matter in the past it had been suggested to me that I try appealing to the Telecommunications Industry Ombudsman etc. However I didn’t do this because I was in fact breaking the Optus acceptable usage policy for most of the time that I was a customer. When I signed up their AUP prohibited me from running a server and from memory I think it had a specific example of a shell server as something that should not be done, it now prohibits running any automated application that uses the Internet when a human is not present (which presumably includes servers). I’m pretty sure that my SE Linux Play Machine [2] met the criteria. While I’m reviewing Optus service I need to mention their mail server, here is the summary of the Optus anti-spam measures in protecting my email address etbe@optushome.com.au in September (other months were much the same):
131 emails have been sent to your Inbox.
52 of these emails were identified as spam and moved to the Spam Folder.
39% of your email has been identified as spam. The email address in question only received legitimate mail from Optus. This meant that I received between two and four valid messages a month, the rest were all spam. So of the 79 messages delivered to me, at least 75 were spam, and Optus blocked less than half the spam. But to be fair, given that the Optus mail servers are listed on some of the DNSBLs it seems reasonable for them to be lax in anti-spam measures. I wonder whether it would be reasonable for an ISP of Optus scale to run the SpamAssassin milter on mail received by their outbound relays to reject the most gross spam from customer machines. But Optus are good at some things. The download speed was always very good (I could receive data at 1MB/s if the remote server could send that fast). Also their procedures for account cancellation are quite good. The guy who took my call offered to transfer me to the complaints department when I mentioned how I was “taught a lesson”, he also offered me a significant discount if I was to continue using the service. In retrospect I should have had that conversation six months ago and had some cheap service from Optus before getting rid of them. Getting the account terminated happened in a couple of hours. It was so quick that I hadn’t got around to transferring my Play Machine to my Internode account before it happened, so I had a few hours of down-time.

Martin F. Krafft: Up the Khyber!

What started with a little host naming theme riddle, and spread as a meme across various planets, has come full circle today, as I installed a new machine, khyber.madduck.net in the racks hosted at the LayerOne colocation site in Z rich. Init Seven, my favourite Swiss ISP, are sponsoring the rack space and connectivity: a routed /28 IPv4 subnet and a /64 IPv6 subnet. The machine is a Transtec Calleo 331 1U server with two quad-core AMD Opteron 2354 processors, 16Gb of RAM, and 4Tb of disk space, configured into a 2Tb RAID1 with LVM on top. It will be used to host a couple of KVM instances for a number of projects I currently operate off my main, personal server, pulse.madduck.net. Unfortunately, Transtec (as well as most other vendors I checked out) uses nVidia s MCP55Pro chipset, which had me sceptical long before I sent the order. nVidia is known to suck at thinking open-source, and the Linux kernel driver for their network chips, forcedeth, had to be reverse-engineered, because nVidia failed to cooperate. Apart from that, nVidia s chips are also often simply crap. Many of their motherboards come with broken APICs and the network chips often wreak havoc on subnets. I find it curious that a search for MCP55 or MCP55Pro on the nVidia website yields no results. Given these expectations, I insisted on a try-and-buy contract, which are not default in Switzerland (as opposed to the EU). My fears proved right, it wasn t long until I encountered the first problems with nVidia s crap: the network chip is unable to deal with segmentation offloading for larger packets, e.g. those with IPv6 payload in a bridging context, and generic segmentation offloading has to be turned off on the interface: 
ethtool -K eth0 gso off
This fixes the issue, and so I stuffed it into /etc/network/interfaces and decided to keep the machine, especially after having invested the time into the server to figure out how to work the watchdog and the IPMI card. The folks behind Transtec s support hotline are helpful, but it took me a bit of time to get the information I wanted out of them. At least they deemed it perfectly natural that I wanted to run Debian, and never questioned that. Let s document some of the more internal specs here for posterity: The BIOS gives the impression of two separate watchdogs, one on the IPMI card (a SIMSO+) and one on the mainboard itself (a SuperMicro H8DMU+). Transtec told me that SuperMicro disabled the IPMI watchdog (called BMC watchdog in the BIOS) with firmware 1.29, leaving only the one on the motherboard. It took a bit of time to find out that this was a W83627THF and supported by the w83627hf_wdt driver, which I configured with module options timeout=120 nowayout=1. I left the BIOS setting at disabled since that would otherwise interfere with booting e.g. the awesome grml live system by rebooting the machine after the set amount of time. It also took a bit of time to get SOL working with the IPMI card. In the end, I found it to work best by configuring the console redirection in the BIOS to use COM2 with 38400/n/8, using hardware flow control, redirection-after-POST always, and vt100 emulation ANSI and vt-utf8 emulation were horrific. I instructed the bootloader to pass console=tty0 console=ttyS1,38400n8 to the kernel (order matters), and configured agetty in /etc/inittab like this: 
T0:2345:respawn:/sbin/getty -8hL ttyS1 38400n8 vt100
To guard against the worst, I also left a grml CD in the drive, to be able to boot a rescue system via SOL in case everything else was broken. The following kernel boot parameters enable me to use the rescue system via IPMI: 
vga=normal noeject noprompt console=tty0 console=ttyS1,38400n8 forensic
Michael Prokop of the grml project tried to create a customised ISO image with these options for me, but I kept getting medium errors trying to read it, independent of the burner, CD-writing-software, operating system used to burn, medium, and drive used to read, so it s possible that the ISO image is faulty. Michael used the following command line to create it (supporting files are here): 
grml-live -t ~/templates-madduck -c GRMLBASE,GRML_MEDIUM,AMD64 -o \
  /grml-live/grml64-medium_2008.11-madduck -a amd64 -s sid -v \
  2008.11-madduck -g grml64-medium -r Schluchtenscheisser -V -b
Until we figure out the problem with this, the grml64 0.2 CD in the drive will do. The machine is now operational, but the IPv6 prefix I have still arrives at vera.madduck.net, a Xen-Instance hosted by Init Seven, which I will no longer need, once I find the time to migrate all the services it hosts. NP: This Will Destroy You: Young Mountain

MJ Ray: Bristol Cooperatives and Social Enterprise Trade Fair on Monday

Cooperatives-SW is working in partnership with the Regional Infrastructure for Social Enterprise (RISE) to hold a Trade Fair with support from CDA (BRAVE Ltd.), Social Enterprise Works, Co-opportunity and Watershed Media Centre. It will be held on Monday 1 December at the Watershed in Bristol. TTLLP will be there and I should post lots of news about it on Tuesday. I’ll try to microblog the event, but that depends how busy it is. Thanks to the VAT rate cut on the same day, I’ll be the only one there, leaving workers in the office just in case any of our customers have problems with their OSCommerce or Drupal online shops. (It seems there shouldn’t be any, but we hope for the best and prepare for the worst.) If you’d like to meet up on Monday, send me a comment however you prefer. Update: If you want to attend the event, registration details are on this page - it looks like registration is required, but I’ll be disappointed if they turn people away from the exhibition space between 1 and 2pm if there’s room.

27 November 2008

Russell Coker: Slicehost vs Linode

Six months ago I investigated the options for Xen virtual servers [1]. I ended up receiving an offer of free hosting and not needing that, but the research was useful. There is a good range of options for Xen servers with different amounts of CPU power, RAM, bandwidth, and disk space. There are a couple of things that seem to be missing, options to upgrade from virtual servers to physical servers, and information on dedicated disk and database performance - but I’ll explain that later after some history. About a week ago a client needed a Xen virtual server in a hurry, their main server (a Xen system that I run on hardware that they rent) was getting a bit overloaded and needed to have one of the largest DomUs moved off. I ended up recommending Linode [2] based on my research and comments I received. The Linode server is working quite well and the client is happy, one nice feature of Linode is the choice of server rooms that they offer. I was able to choose a room in the same region as the other servers that the client owns and thus get ping times that are sometimes less than 2ms! Due to a missing feature in a program that I’m maintaining for the client a large number of MySQL queries are being made. Due to a problem I’m having with MySQL it won’t let me create a slave database server so all the queries go over the VPN and use a large amount of data. This combined with the other traffic that should be going over that link means that about 600G per month is being used, fortunately that is rather cheap. Linode staff handled this very well, after the server had exceeded it’s quota by 120G they asked my client to confirm that the traffic was legitimate and then suggested an upgrade to a plan that could handle the traffic (which went smoothly). Now I have another week to add the feature in question before I meet the quota again. Shortly after getting the new virtual server running at full capacity David Welton wrote a detailed review of Linode and Slicehost for the issues that matter to his use [3]. His conclusion seems strongly in favor of Linode. But now I am looking at getting a Slicehost [4] virtual server for the same client (for a different project) because Slicehost is owned by Rackspace [5], and the new project if successful will need a set of powerful servers and Rackspace seems like a reasonable company to host that. The problem with Rackspace is that they (and every other ISP I’ve researched so far) seems to offer little in regard to customers who need serious disk IO. I am planning some servers that will have a write bottleneck on a MySQL database (or maybe multiple shards), so serious disk capacity is needed. At least I would like to be able to get disk storage by the tray (12-14 disks) with the controllers having RAID-6 support. Rackspace only offers RAID-5 (according to the “livechat” person), and we didn’t get as far as discussing how to add more trays. What would be ideal is if there was an ISP that had both virtual servers and physical servers (so I could start with a virtual server and move to a physical server when things are working well), and also serious storage options. They would offer internal disks, external RAID arrays, and NetApp Filers [6] (or some equivalent device). It would be really nice if I could just instruct the ISP to add another NetApp Filer to my back-end network and have it done for me (I’m certainly not going to visit the US to install new hardware). It’s been over a day since I submitted a sales request to NetApp asking whether they partner with any ISPs and I haven’t received a response. OpenSolaris with ZFS also sounds good for disk IO performance (they have similar features to NetApp). Unfortunately the support for OpenSolaris among ISPs is not that great (while everyone offers Linux and Windows), and I haven’t used any recent version of Solaris. So using OpenSolaris would require finding someone with the skills to manage it who can work for my client - as opposed to a NetApp device that would be just like any other NFS server, SAN, or iSCSI server. But I’m not ruling OpenSolaris out, if someone knows of a good ISP that hosts OpenSolaris machines and supports adding dozens of disks and decent amounts of NVRAM for ZFS then I would be interested to investigate it. Joyent has some interesting OpenSolaris virtual server plans [7], they are a little pricey but offer large amounts of data transfer. They don’t provide any information on disk IO capacity (other than saying that they use ZFS for good performance). I’ve just downloaded Nexenta (Debian with the OpenSolaris kernel) [8] and will test it out over the next few days. One of the reasons I’m tending towards Rackspace at the moment (with Slicehost as the entry point) is that they seem cooperative to customer requests. My discussions with them (on a web based “livechat” and on the phone) have indicated that they may be able to do something special for me.

26 November 2008

Uwe Hermann: jhead - List and modify EXIF fields in JPEG photos

jhead is a very nice and very powerful command line utility to mess with JPEG headers (esp. EXIF fields). 
  $ apt-get install jhead
It can display/extract a great amount of metadata fields from JPEG files and also extract the thumbnails stored in JPEG files (if any). The following will list all known metadata fields from a sample photo: 
  $ wget http://farm4.static.flickr.com/3173/3061542361_60acb0904b_o.jpg
  $ jhead *.jpg
  File name    : 3061542361_60acb0904b_o.jpg
  File size    : 1074172 bytes
  File date    : 2008:11:26 23:38:04
  Camera make  : Panasonic
  Camera model : DMC-FZ18
  Date/Time    : 2008:03:05 15:45:52
  Resolution   : 3264 x 2448
  Flash used   : No
  Focal length : 4.6mm  (35mm equivalent: 28mm)
  Exposure time: 0.0100 s  (1/100)
  Aperture     : f/3.6
  ISO equiv.   : 100
  Whitebalance : Auto
  Metering Mode: matrix
  Exposure     : program (auto)
  GPS Latitude : N %:.7fd %;.8fm %;.8fs
  GPS Longitude: E %;.8fd %:.7fm %;.8fs
  GPS Altitude : 174.00m
  Comment      : Aufgenommen auf dem <a href="http://www.froutes.de/TT00000014_Ars_Natura">Kunstweg Ars Natura</a>.
  ======= IPTC data: =======
  Record vers.  : 4
  Headline      : Felsburg auf dem Felsberg
  (C)Notice     : www.froutes.de
  Caption       : Aufgenommen auf dem <a href="http://www.froutes.de/TT00000014_Ars_Natura">Kunstweg Ars Natura</a>.
As you can see there's a huge amount of potentially privacy-sensitive metadata in your typical JPEG as generated by your camera (including camera type, settings, date/time, maybe even GPS coordinates of your location, etc). You can extract the thumbnail stored in all JPEGs in the current directory with: 
  $ jhead -st "&i_t.jpg" *.jpg
  Created: '3061542361_60acb0904b_o.jpg_t.jpg'
Random flickr image and its differing thumbnail Note that the JPEG thumbnail does not necessarily show the same picture as the JPEG itself. Depending on the image manipulation software that was used to create the edited/fixed/cropped JPEG, the thumbnail may still reflect the original JPEG contents (see sample image on the right-hand side). This is a huge potential privacy issue. There have been a number of articles about this some years ago, in case you missed them: Thus, an important jhead command line to know is the following, which removes all metadata (including any thumbnails) from all JPEG images in the current directory: 
  $ jhead -purejpg *.jpg
  Modified: 3061542361_60acb0904b_o.jpg
As you can see the result is that only very basic information can be gathered from the file afterwards: 
  $ jhead *.jpg
  File name    : 3061542361_60acb0904b_o.jpg
  File size    : 1052506 bytes
  File date    : 2008:11:26 23:38:04
  Resolution   : 3264 x 2448
  $ jhead -st "&i_t.jpg" *.jpg
  Image contains no thumbnail
I recommend doing this for most photos you make publically available on sites like flickr etc. (unless you have a good reason not to). Finally, see the jhead(1) manpage for lots more options that the tool supports.

Biella Coleman: Hacking Spaces, the Spaces of Hacking

Space and place has long been important to hackers. Whether it was/is the university lab, the workplace, the hacker con, or the particularly high-tech city, hackers congregate and meet face to face, often and everyday. One semi-new development has been the explosion and proliferation of hacker labs and spaces, such as Noisebridge located in the Mission district of San Francisco Foulab in Montreal. I recently got back from San Francisco and was able to spend a few nights at Noisebridge and was jaw droppingly impressed. The space is, well, spacious and nice (and located right by the Bart, a real +++++), but more important is that it is a thriving collective with all sorts of geeky participants and they have just souped up the space with all sorts of equipment, from the usual suspects (lots of computers) to lots of electronic gear such as oscilloscopes. What I also was impressed by was not only the blizzard of events but the open and accessible nature of the organization, which seemed to sit in some contrast to NYC’s hacker collective, NYC Resistor. Like so many organizations in this metropolis, they apparently are lacking in space to grow and the word on the streets (which I cannot confirm or deny as I have had very little contact with them but have heard this repeatedly) is that the organization has had a tough time letting in new members. Some folks are understanding of this given their space limitation, others have been less kind, and have referred to the group as a clique. There are already a few other initiatives under way to find a larger space so as to accommodate a more open, participatory atmosphere for a hacker space but I imagine that over time the culture and developments of NYC Resistor will also change as new spaces develops and do hope that this creates the conditions for more access rather than less (and again, I know next to nothing of the situation though I suspect space plays a real limiting role as it does with nearly everything else in NYC). And thankfully Rose White, a NYC-based sociology graduate student at CUNY, is paying close attention to the rise and development of these hacker spaces. She is well underway doing her dissertation dissertation on these hack spaces and I really look forward to her work.

Russell Coker: Flash Storage and Servers

In the comments on my post about the Dell PowerEdge T105 server [1] there is some discussion of the internal USB port (which allows the use of a USB flash device for booting which is connected inside the case). This is a really nice feature of the Dell server and something that would be useful if it was in more machines. However I believe that it would be better to have flash storage with a SATA interface on the motherboard. The cost of medium size flash storage (eg 4G) in the USB format is not overly great, if soldered to the motherboard or connected to a daughter-board the incremental price for the server would be very small. Dell servers shipped with a minimum of an 80G SATA disk last time I checked, it seems quite likely to me that Dell could reduce the prices of their servers by providing flash storage on the motherboard and having no hard disk. It seems likely to me that there is a significant number of people who don’t want the default hard drive that ships with a Dell server. The 80G disk that came with my PowerEdge is currently gathering dust on a shelf, it was far to small to be of any use in that machine and Dell’s prices for bigger disks were outrageous so I replaced the default disk with a pair of big disks as soon as the server had passed some basic burn-in tests. Most servers that I run fall into one of two categories, machines which primarily do computation tasks and need little storage space or IO capacity (in which case 4G of flash would do nicely) and machines which have databases, backups, virtual machine images, and other big things (in which case anything less than 160G is silly and less than 500G makes no economic sense in today’s market). Note that for a machine with small storage requirements I would rather have a 4G flash device than an 80G disk, I am inclined to trust flash to not die but not trust a single disk, two 80G disks means more noise, heat dissipation, and expense. According to comments on my previous post VMWare ESX requires a USB boot device, so if VMWare could be booted with a motherboard based flash device then that would be an ideal configuration for VMWare. In some mailing list discussions I’ve seen concern raised about the reliability of permanently connected USB devices, while I’ve only encountered USB problems related to buggy hardware and drivers other people have had problems with the electrical connection. So it seems that motherboard based flash could be expected to increase the reliability of VMWare servers. The down-side to having flash permanently attached to the motherboard is of course the impossibility of moving the boot device to different hardware. In terms of recovering from failure restoring a few gig of flash storage from backup is easy enough. The common debugging option of connecting a hard drive to another machine to fix boot problems would be missed, but I think that the positive aspects of this idea outweigh the negative - and it would of course be an option to not boot from flash. If anyone knows of a tower server that is reasonably quiet, has ECC RAM and a usable amount of flash storage on the motherboard (2G would be a bare minimum, 4G or 8G would be preferred) then please let me know.

25 November 2008

Joost Yervante Damad: first snow (meme)

Since everybody is posting pictures of the first snow of the year, I can't stay behind. Quickly popped out this morning to make this snap of my favorite habitat, the valley of the "Grote Nete":


(click on the image to enlarge)

nikon d300 tamron 17-50 f/2.8 2008 Joost Yervante Damad



24 November 2008

David Welton: Slicehost vs Linode

Since hosting is the major expense for DedaSys LLC, and obviously a critical part of much of what I do, getting the right one is very important. Naturally, "the right one" for me may not be the right one for everyone. I am a fair sysadmin for small numbers of machines, so I don't mind doing that myself - I don't need, nor want to pay for hand holding. I recently moved everything over to Slicehost on the recommendations of friends, and am so far fairly happy with the experience. It's cheaper, simpler, and more flexible than Layered Tech was. To boot, I also like the fact that Slicehost is run out of St. Louis, Missouri, rather than some really expensive tech center like San Francisco or Boston. Hosting is basically a commodity, and much like you wouldn't want to put a factory in San Francisco, you don't want your hosting their either. However, I discovered something annoying about Slicehost: they use x86_64 servers, which, per se, doesn't really matter to me - I use open source code that can run on any number of architectures. The problem is that this particular architecture uses more memory than plain old x86. Significantly more. My Layered Tech server with 1 gig of memory was hitting the swap space a bit, but the same code on a slice was swapping quite heavily, despite the fact that I'd moved PostgreSQL off to its own slice. Since I pretty much exclusively run Rails, I decided to look into Phusion's "Ruby Enterprise Edition", which is basically just some nice hacking on Ruby's garbage collection mechanism. What they've done is nice, and I may end up using it, but ultimately it's buying me space that I would also gain from simply moving back to x86. With that in mind, I decided to take another look at what appears to be emerging as one of Slicehost's main competitors, Linode, who *do* use x86 servers. Here are my results, which are admittedly not all that scientific, but what the heck - you're getting them for free, and they were pretty quick to whip up. Service, Support, Setup But before we begin with the numbers, I want to add a few words of caution. One of the big things about hosting, to me at least, is how they deal with unexpected random negative events. Connections going down, disk breakage, DoS attacks, and so on. It's really hard to get an idea of just what sort of people you're dealing with until you've "ridden the river together". I don't really know what sort of response times, uptime, and all that either Slicehost or Linode have, so there are potentially some big intangibles there that are not quite as easy to draw pretty charts for. In terms of the console/setup/management tools, I liked Slicehost's simplicity more - Linode gives you more options, but they're a bit fiddly and cluttered feeling at times. For instance, Linode lets you pick how much swap space to give your disk image, but doesn't include a bit of javascript to balance out the swap and regular partitions, so that if you type in a larger swap number, you have to do the math and subtract that from the other number. Annoying, but not a big deal. Linode lets you pick various data centers in the US (no Europe, yet). Slicehost gives you the option to do backups of your disk images, which is nice, and something that Linode lacks. Comparing plans Since I'm interested in comparing hardware and machines I actually have access to, here are the plans I am currently signed up with, and the salient numbers:
Slicehost 1024 Slicehost 512 Linode 720
Memory 1024 512 720
Bandwidth 400 200 400
Disk 40 20 24
Cost 70 38 39.95
Memory / Dollar: Bandwidth / Dollar: Disk / Dollar: Linode mostly comes out ahead, but not by that much, except for bandwidth. However, let's take a crude look at x86_64 vs x86 memory usage. First, ./script/console (a Rails console) for the same code base on two different machines, showing the virtual memory size (VSZ) and resident set size (RSS): And just to look at something else that's a bit smaller, I ran the following script: 
puts File.open("/proc/self/status", "r").read
And got the following data: Being charitable with those numbers, we get x86 taking up 77% the memory that x86_64 does. So let's add that to our comparison of how much memory you're getting per dollar on Slicehost and Linode: Wow! That's a fairly significant difference. Performance That's most of what we can glean from published numbers, and running a few simple experiments. However, there's another factor that's important: performance. "Virtual Private Server" systems, or simply VPS's, got a bad name in the past because they were "overbooked" - too many virtual servers competing for the same CPU resources on one machine. Slicehost and Linode both look like they want to avoid that kind of bad reputation, and so far all the systems I have used have felt snappy and responsive. Now here's where we get really unscientific... I decided to try and do something, even if it wasn't much, to get some kind of objective measurement of what kind of CPU I was getting on the machines. This is really unscientific, because the machines do have other duties (I don't have the cash, time or inclination to set up random servers just for testing), and of course, who knows what else is sharing the computer with my systems: if you get a relatively unused computer, you can apparently pick up extra cycles for yourself, over and above your guaranteed minimum. But c'est la vie, and so I decided to do some numbers anyway. I picked a C implementation of mandelbrot from the language shootout more or less "for the hell of it" (I'm going to keep repeating "unscientific" and invite someone to do better tests than I have). I ran this code every 20 minutes for a couple days, and then averaged out the run times: As expected, the higher power Slicehost 1024 machine wins out in terms of raw speed (less time is better), but not by that much over the Linode system. Indeed, when we factor in the relative prices, Linode comes out ahead, again: So it looks like we're not paying a CPU penalty for having more memory, bandwidth, disk, etc... Notes
  • Nothing in these statistics is indicative of future directions that both Linode and Slicehost might take. Slicehost was recently acquired by Rackspace, and that could have effects on the service they provide. Maybe one or both will come under financial pressure and start "overbooking" their servers.
  • The spreadsheet I used is available via Google docs, for the curious. I did most of the work in OpenOffice, and then uploaded it though, because working with the web spreadsheet was kind of painful. Unfortunately, you can't select non-contiguous data ranges in the Google spreadsheet, so the labels are interspersed with the data. Yuck. Also, Google docs doesn't handle 'time' values nicely, which is what the performance data was originally. The spreadsheet is here: http://spreadsheets.google.com/pub?key=pVDeZF6ire2YwATBYb4mBKQ
  • The standard deviation on the processing times is higher for both slices than the linode system. Not quite sure what that means in terms of what's going on under the hood, but I thought it was interesting.

Aigars Mahinovs: Slowfs

I have just encountered a bug in Gnome that is much more visible when the hard drive is slow or overloaded and responds slowly. This gave me an idea - how about a simple transparent FUSE filesystem that does nothing else than delay, slow down and possibly reorder filesystem requests? Such a filesystem would be very useful for debugging. We developers tend to have high-performance systems and that hides many bugs, but if we could have a slower system on-demand, it will give us the ability to debug our applications better. Even better would be a virtual machine where one could arbitrarily slow down any specific aspects of the system - slower CPU, slower hard drive, slower RAM, slower network, … I know that engineers of network hardware have special devices that emulate long distance and noise - you take two wireless network cards and plug cables into their antennae sockets and then plug those cables into the device. The device then makes a connection between those cables, but with emulation of a variable distance, variable antennae and variable environmental noise, so the wireless cards can be tested for distance and noise resistance right in the lab. We could do the same in software for HDD, CPU, RAM and networking - imagine having two Xen instances and putting an emulated network between them with 10 hops, 1 second latency, 20% average packet drop traffic shaping and limited top bandwidth of 100 kbit/s. Now that is a real test platform for any kinds of applications that have anything to do with networking (or HDD, CPU or RAM). Anyone up to making that? Could someone of you use a system like that?

Josselin Mouette: The great return of the broomsticks

While a few developers have the chance to be paid for their job on Debian, most of us are doing it on our spare time, for fun. Yes, we are making the best operating system out there, for fun. Yet a large number of developers are taking this project way too seriously. To say it in other words, they have a carrot up the ass. I happen to prefer the French expression though, since it conveys better the rigidity of their attitude: they have a broomstick up the ass. Recently, when I launched the idea of the Bug Sprint, someone wrote to me to propose, instead of rewarding people with cookies, to send them money. This is the best example of broomstick behavior: taking the fun out of people by bringing back rampantly the very idea that almost split the project in two the last time it was introduced. The very same person, when faced with a sarcastic email sent to an announcement mailing list, immediately rushed in to send a ban request to list masters (a totally useless one, since I don t have the habit to make the same joke twice). That person happens to be the project leader. So it seems the only thing the DPL has to do, at a time when we d rather need people to fix RC bugs, is to protect the project from seditious members who don t share their love for broomsticks. When someone makes a bad-taste joke, you are not forced to laugh. But you are not forced to impose your rigid and moralist views either. And on top of that, throwing random accusations of sexism without any kind of justification does not improve the mood. This is even very close to defamation, but since I don t have a broomstick in the ass myself, I prefer to just laugh at them with the help of some fellow developers. I will go on shocking these people. And they won t like it more than in the past. Every time one of them climbs on her high horse to take a pathetic and patronizing stance while totally missing the point, it will bring a bit of fun. Bonus #1: as for the promised statistics, we are, at the time of writing, at 493 599 unique IPs who clicked on the trap. Quite a good investment for someone who d want to make money from smelly geeks. Bonus #2: here is a sexist picture.

Next.

Previous.