Search Results: "reg"

5 August 2011

Kai Wasserb ch: YouTube: download 1080p videos when CLive is unavailable

Todays addition to the tips series has a funny history. A friend of mine was stuck in an envrionment, where she couldn't install Perl or CLive but PHP with some PECL modules was installed. She needed to download a few videos for a presentation, but didn't know how to fetch the videos from YouTube. So I was asked to help. ;-) After some pondering of the problem, I came up with the following script:
<?php
/*
 * Copyright (C) 2011  Kai Wasserb ch <kai@dev.carbon-project.org>
 *
 * Small little script to help out in the absence of Perl/Clive but with
 * PHP present.
 *
 * License: GPLv3 <http://www.gnu.org/licenses/gpl-3.0.txt>
 *
 */
if(php_sapi_name() !== 'cli')
  die("This script needs to be run on the CLI!\n");
if($argc !== 2 )
  die("No URL given on CLI or too much parameters!\n");
// Script expects a URL like http://www.youtube.com/watch?v=[ID]
if(filter_var($argv[1], FILTER_VALIDATE_URL, FILTER_FLAG_QUERY_REQUIRED) === false)
  die("Parameter given as URL does not seem to be an URL or contain a query!\n");
if(!$ch = curl_init($argv[1]))
  die("Couldn't init cURL handle for  ".$argv[1]." \n");
curl_setopt($ch, CURLOPT_HEADER,          false );
curl_setopt($ch, CURLOPT_RETURNTRANSFER,  true  );
$htmlsource = curl_exec($ch);
$error = curl_error($ch);
curl_close($ch);
if($htmlsource === false)
  die('cURL returned an error: '.$error."\n");
unset($error);
if(!preg_match('/url_encoded_fmt_stream_map=([a-zA-Z0-9%\.,\-_]+)&/',
               $htmlsource, $matches))
  die("No matches found for url_encoded_fmt_stream_map!\n");
$final = NULL;
foreach(explode(',', urldecode($matches[1])) as $decoded_url_map)  
  parse_str($decoded_url_map, $tmp);
  if($tmp['quality'] === 'hd1080')  
    $final = $tmp;
    $final['url']   = urldecode($tmp['url']);
    // $final['type']  = urldecode($tmp['type']); // Not used
    break;  // Not really needed, but should save us some
            // iterations, since 1080p seems to come always first.
   
 
if(!is_array($final)   is_null($final))
  die("No 1080p version of the video was found!\n");
if(!preg_match('/<meta property="og:title" content="(.*)"/', $htmlsource, $title))
  die("Couldn't determine title!\n");
$ftitle = preg_replace('/[^\w\d]*/', '', $title[1]);
printf("Found hd1080-quality video for  %s .\n", $argv[1]);
printf("Downloading to %s.mp4 ...", $ftitle);
if(!$ch=curl_init($final['url']))
  die("\nCouldn't create cURL handle for  ".$final['url']." \n");
if(!$fh = fopen(getcwd().DIRECTORY_SEPARATOR.$ftitle.'.mp4', 'wb'))  
  curl_close($ch);
  die("\nCouldn't create  ".$ftitle.".mp4 .\n");
 
curl_setopt($ch, CURLOPT_HEADER,         false );
curl_setopt($ch, CURLOPT_FILE,           $fh   );
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true  );
if(!curl_exec($ch))  
  $error = curl_error($ch);
  fclose($fh);
  curl_close($ch);
  die("\nCouldn't download video. cURL error: ".$error."\n");
 
curl_close($ch);
fclose($fh);
printf(" done.\n");
?>
You can download it from my website or you can just copy the code given above into a file. However you get the code, the execution will look like: php -f /path/to/script.php http://www.youtube.com/watch?v=[ID]. Now, before I get a lot of e-mails telling me, what could be improved on the script or its other limitations (like only being able to download 1080p videos), please remember that it was hacked together for a simple use case with no real intent on my part to use it for anything else. It only later occurred to me, that others might be in a similar pinch sometime. Thus I'm releasing the above code "as is" under the GNU GPLv3. Therefore you're free to adapt it to your needs. Please also note, that I'm not recommending using the above script instead of CLive. If you can install CLive, do so and don't bother with the above.

Rudy Godoy: Compute Clusters Integration for Debian Development and Building Report 4

Hello, this is the fourth report for the project. First, I d like to
apologize for being late, I took a short trip over the weekend and
missed the deadline. This time I ve good news for my project. As
expected I ve made the necesary changes to have Eucalyptus support ARM
images. The approach was extending and exposing an arch field that
will be used by libvirt s XML domain file. I m working currently on
having the involved functions working with the new extension and I
expect to have this done over the middle of the next week. What was done so far

- Extended the ncInstance_t struct on util/data.h adding the archId
parameter. This struct keeps the information regarding a node (nc),
say RAM, Kernel, etc. Currently it has no way to set an arch in order
to be used later. typdef struct ncInstance_t
...
char reservationId[CHAR_BUFFER_SIZE];
char userId[CHAR_BUFFER_SIZE];
char archId[CHAR_BUFFER_SIZE]; // arch field added
int retries;
...

The Eucalyptus team was OK with this approach that I ve proposed few
weeks ago. For all my changes the approach is maintain compatibility
with existing features.
We still have to define if we are setting a default value, say amd64,
or keep it unset until the user sets one. For now we will be supporting
amd64 and arm as valid fields. - Modified the allocate_intance() function to support the archId
field. This involved adding a new parameter and detecting whether its
present and setting de value accordingly. This function is later used
by the doRunInstance() function in node/handlers_kvm.c - Modified the involved functions that are called by the KVM handlers
and result in passing parameters to the XML file generator for being
used by libvirt s and KVM and have the image running under
Eucalyptus. The new parameters and field are not required and only
being used if they are set with a value. The archId parameter has been
added as the last one, so existing function calls can work. This could
be rearranged later, but will involve a more extensive revision with
Eucalyptus developers. - I ve set a git local repo in order to keep track of the changes to
Eucalyptus. Since this part is a sort of a Eucalyptus branch I took
this approach. Steffen and I are discussing whether to release this as
a pkg-eucalyptus branch (as dpatch, using existing packaging). Either
way I m putting the diff files form my git repo over the wiki and will
put them on my site too. I plan to use git-svn in order to send them
later to the pkg-eucalyptus repo once we are settled with this. - Updated the project s wiki[1] page with information regarding all that s
involved on my work, so it can be re-played or resumed later. 1- http://wiki.debian.org/SummerOfCode2011/BuildWithEucalyptus/ProjectLog What I plan to to over the next weeks
-
- Finish the modifications to the relevant functions. Expected dealine:
next week.
- Do integration tests:
Write a test program, there s an example under node/ that could be
helpful.
Generate the XML domain definition from Eucalyptus functions. The
full cycle: handler_kvm -> allocate_instance -> get_instance_xml ->
have the XML file properly generated -> qemu-kvm runs the instance
using the proper emulator (arm in our case).
- Test runnning the image I ve worked on the project s begining.
- Fixing bits and cooperate with the pkg-eucalyptus team on the
packaging bits.
- Talk to Eucalyptus so they adopt our patches.
- Rock and roll :) What would be nice to have after the project ends
-
- The Debian packaging *needs* work. I ve already discussed this on the
pkg-eucalyptus mailing list. I ve also committed myself to help on
that side even after the SoC ends.
- Talk with upstream and Ubuntu to coordinate the packaging efforts and
modifications to the software.
- Get adopters to use our work :) By now, few weeks about to finish the SoC, I can say that it s been a
good experience and as I stated I m planning to keep maintaing/working
on the software after that. I ve already joined the pkg-eucalyptus team
and plan to contribute actively. I m sad I didn t managed to get Debconf
this year, time was a constraint again, hopefully we can meet in 2012! I
guess that all by now! Best regards,
Rud

Dirk Eddelbuettel: New Rcpp master classes scheduled for New York and San Francisco

Together with Revolution Analytics, I will be offering two more one-day classes on the Rcpp package for seamless integration of R and C++. The format will follow the workshop Romain and I gave during the tutorial day preceding this year's R/Finance conference. The style will once again be hands-on, with copious concrete examples and solid coverage of most aspects of Rcpp and related packages such as RInside, RcppArmadillo and others. The eight-hour schedule contains about six hours of instruction, split into four sessions of around ninety minutes. This leaves ample time for both lunch and coffee breaks, and for informal discussions and Q+A. Two one-day classes will be offered: The first in New York on Saturday, September 28, 2011 and the second one two weeks later in San Franciso on Saturday, October 8, 2011. Please see the official course page for more details, concrete location info and maps as well as registration details. Feel free to contact me at the usual email address with questions.

4 August 2011

Rapha&#235;l Hertzog: My Debian activities in July 2011

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (170 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. This month passed by very quickly since I attended both the Libre Software Meeting / RMLL and the DebConf. Libre Software Meeting / RMLL I attended only 3 days out of the 6 but that was a deliberate choice since I was also attending DebConf for a full week later in the month. During those 3 days I helped with the Debian booth that was already well taken care of by Fr d ric Perrenot and Arnaud Gambonnet. Unfortunately we did not have any goodies to sell. We (as in Debian France) should do better in this regard next time. One of the talks I attended presented EnVenteLibre. This website started as an online shop for two French associations (Ubuntu-fr, Framasoft). They externalize all the logistic to a company and only have to care about ordering goodies and delivering to the warehouse of the logistic company. They can also take some goodies from the warehouse and ship them for a conference, etc. We discussed a bit to see how Debian France could join, they are even ready to study what can be done to operate at the international level (that would be interesting for Debian with all the local associations that we have throughout the world). Back to the LSM, while I had 3 good days in Strasbourg, it seems to mee that the event is slowly fading out it s far from being an international event and the number of talks doesn t make for a better quality. BTW, do you remember that Debconf 0 and Debconf 1 were associated to this event while it was in Bordeaux? dpkg-source improvements During my time in Strasbourg (and in particular the travel to go there and back!) I implemented some changes to 3.0 (quilt) source format. It will now fail to build the source package if there are upstream changes that are not properly recorded in a quilt patch:
dpkg-source: info: local changes detected, the modified files are:
 2ping-1.1/README
dpkg-source: info: you can integrate the local changes with dpkg-source --commit
dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/2ping_1.1-1.diff.cki8YB
As the error message hints, there s a new --commit command supported by dpkg-source that will generate the required quilt patch to fix this. In the processe you will have to submit a name and edit the patch header (pre-formatted with DEP3 compatible fields). You can get back the old behavior with the --auto-commit option. Build flags changes Ever since we adopted the Ubuntu changes to let dpkg-buildpackage set some build related environment variables (see #465282), many Debian people expressed their concerns with this approach both because it broke some packages and because those variables are not set if you execute debian/rules directly. In the end, the change was not quickly reverted and we fixed the package that this change broke. Despite this we later decided that the correct approach to inject build flags would be a new interface: dpkg-buildflags. Before changing dpkg-buildpackage to no longer set the compilation flags, I wanted to ensure dpkg-buildflags had some decent coverage in the archive (to avoid breaking too many packages again). My criteria was that CDBS and dh (of debhelper) should be using it. With the recent debhelper change (see #544844) this has been reached so I changed dpkg-buildpackage accordingly. Makefile snippets provided by dpkg At the same time, I also wanted an easy way for maintainers not using dh or CDBS to be able to fix their package easily and go back to injecting the compilation flags in the environment but doing it from the rules files. Starting with the next version of dpkg, this will be possible with something like this:
DPKG_EXPORT_BUILDFLAGS = 1
include /usr/share/dpkg/default.mk
Without DPKG_EXPORT_BUILDFLAGS the variables are not exported in the environment and have no effect unless you use them somewhere. More than build flags, this will also provide a bunch of other variables that can be useful in a rules files: all the variables provided by dpkg-architecture, vendor related variables/macro and some basic package information (mainly version related). dpkg-buildflags improvements Given the renewed importance that dpkg-buildflags will take now that dpkg-buildpackage no longer sets the corresponding environment variables, I thought that I could give it some love by fixing all the open issues and implementing some suggestions I got. I also had a chat with a few members of the technical committee to discuss how hardening build flags could be enabled in Debian and this also resulted in a few ideas of improvements. In the end, here are the main changes implemented: Will all those changes, the complete set of compilation flags can be returned by dpkg-buildflags (before it would only return the default flags and it was expected that the Debian packaging would add whatever else is required afterwards). Now the maintainer just has to use the new environment variables to ensure the returned values correspond to what the package needs. DebConf: rolling and hardening build flags I spent a full week in DebConf (from Sunday 24th to Sunday 31th) and as usual, it s been a pleasure to meet again all my Debian friends. It s always difficult to find a good balance between attending talks, working in the hacklab and socializing but I m pretty happy with the result. I did not have any goal when I arrived, except managing the Rolling Bof (slides and video here) but all the discussions during talks always lead to a growing TODO list. This year was no exception. The technical committee BoF resulted in some discussions of some of the pending issues, in particular one that interests me: how to enable hardening build flags in Debian (see #552688). We scheduled another discussion on the topic for Tuesday and the outcome is that dpkg-buildflags is the proper interface to inject hardening build flags provided that it offers a mean to drop unwanted flags and a practical way to inject them in the ./configure command line. Given this I got to work and implemented those new features and worked with Kees Cook to prepare a patch that enables the hardening build flags by default. It s not ready to be merged but it s working already (see my last update in the bug log). A few words about the Rolling BoF too. The room was pretty crowded: as usual the topic generates lots of interest. My goal with the BoF was very limited, I wanted to weigh the importance of the various opinions expressed in the last gigantic discussion on debian-devel. It turns out a vast majority of attendants believe that testing is already usable. But when you ask them if we must advertise it more, answers are relatively mixed. When asked if we can sustain lots of testing/rolling users, few people feel qualified to reply but those that do tend to say yes. More dpkg work Lots of small things done: Package Tracking System and DEHS Christoph Berg recently wrote a replacement for DEHS because the latter was not really reliable and not under control of the QA team. This is a centralized system that uses the watch files to detect new upstream versions of the software available in Debian. I updated the Package Tracking System to use this new tool instead of DEHS. The new thing works well but we re still lacking the mail notifications that DEHS used to send out. If someone wants to contribute it, that would be great! Misc packaging work I did some preliminary work to update the WordPress package to the latest upstream version (3.2). I still have to test the resulting package, replacing upstream shipped copies of javascript/PHP libraries is always a risk and unfortunately all of them had some changes in the integration process. I also updated nautilus-dropbox to version 0.6.8 released upstream. I also uploaded the previous version (that was in testing at that time) to squeeze-backports. So there s now an official package in all the Debian distributions (Squeeze, Wheezy, Sid and Experimental)! Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Diego Escalante Urrelo: GUADEC Hispana in Sevilla

Sevilla is located in the (one of the many actually) oh-boy-is-that-really-the-temperature region of Spain. It has been historically significative for its financial, artistic and cultural value. But it s most important role has been that of hosting lots of GNOME activity. The 3rd ever GUADEC was held here, those cool Emergya guys who are working on a11y live here, and the even more awesome Sugus ETSII group helping with the conference setup and beer availability who helped organize the GNOME 3 launch party in Sevilla.

Foto de familia de la Guadec-es8

GUADEC Hispana group picture by Ana Rey (from GUADEC_ES8 set) Not as huge as GUADEC, but certainly as much fun as it, GUADEC Hispana lets the Spanish speaking community get closer together, get bugs fixed and only as a side effect get a bit drunk. A big thank to Ana, Juanje, V ctor, the Sugus team and all the local people that contributed. Also, of course, thanks to our GNOME loving (and loved) companies: Emergya, Igalia and OpenShine. And don t forget Escuela T cnica Superior de Ingenier a Inform tica who hosted the event. Beer time now. See ya at Berlin.

Aigars Mahinovs: Debconf 11 postmortem

Another year, another Debconf and now it is passed. Pictures are processes and all are now uploaded. We are still missing 25 names in the Debconf 11 group photo. With 265 people and 86 Mpix it is the highest resolution image we have had so far (Spain image had a bit more pixels, but a lot of them were outside the actual photo) and the largest number of people (Edinburgh photo had 248 people). The video team produced hundreds of gigabytes of footage, we had very interesting talks and debates and sometimes the AdHoc meeting room on the side was overcrowded with people in BoFs that were not on the initial schedule. It has been a very special kind of conference. As it always is. :)
As always, there were also some problems. I did not read the debconf-team mailing list, but I hear there was plenty of fun discussions to be had in the run-up to the event, some information on getting to the venue was not quite clear (that was quickly fixed as first people arriving to Debcamp were documenting their experience), the organizational dance with the food tickets was more elaborate than usually, both the day trip and the formal dinner were more self-driven than expected, vegetarian complaint level was about the same as usual (which is considered high by some) and the wireless in the hotel was very weak, also Saturday weddings are quite loud and run very late apparently.
To compensate for that we had: very sunny Debcamp, cheap beer (!!!), great quality accommodations, very good looking venue, good network at the venue (after initial scalability issues with the wireless), good food (that did sometimes run out, though), short distances to all locations, oh and did I mention cheep beer? What do we learn from this for next year? I am told that Debconf12 team has already had some meetings specifically to identify and learn from some shortcomings and strengths of this years conference. However, I think everyone of us might have an idea or two to add to the mix as well. For example, I think that BoF rooms are great and we should expand and solidify on that concept: take a page from unconference playbook have a board on-site where people can book slots for their BoF in one of the two rooms for tomorrow. Not far in advance and without moderation requirement of penta scheduling. In the evening the stickers can be taken down and tomorrows BoFs can be added to the schedule at that point, so the people can see online what is going on.
Also I think that we need some kind of coverage of the BoF room(s) from the video team. Even if that is in a form of a single self-service omnidirectional microphone where people could just walk into the room, press a single button and the audio would start streaming and recording.
In addition we could have a screen with the IRC channel of the room projected onto a screen on a side of the stage, to simplify getting questions off the channel. Recording that channel (as a text log) could also be useful.
It feels like post-conference assembly of the talk lists, presentations, links to the relevant recordings, photos, chat logs, ideas, comments, for every talk is very problematic in penta, should we perhaps just use wiki for that?
Any other ideas? And to finish off some more personal travel-inspired rambling. After visiting Paris, Zagreb, Banja Luka and (very shortly) Vienna on this trip it was very interesting to see similarities and draw some parallels, especially with the added context of Berlin and Riga. The similarities in city planning, street shops, brands and the way people go about their lives in these capitals are quite remarkable. I would even say that a trend can be seen where capitols of countries/regions that had troubles in the past (Bosnia more recently; Riga, Zagreb and East Berlin a bit more into the past) are now moving from their different pasts and converging towards the vision of the European capitol city living we see in cities like Paris. If you look at individual pieces you can even recognise milestones along this route and see when specific cities reach them (possibly in a different order than others). With that in mind you can even try to predict what might come next, like looking at Banja Luka and thinking that they might reach the point of upgrading their buses in the next 2-3 years, or comparing Riga and Zagreb we can see that Riga needs to soon choose a street and close it to car traffic to have more caf s, beer gardens and shops concentrated in that area. Or looking at Paris we might find a compelling idea, like clustering a lot of businesses of the same type (such as kitchen design) in the same street, so the customers know where they can go to have all the best choices available within a block. All in all this years Debconf for me was one with the most diverse experiences and the one that felt closest to home. I hope to see you all good people again next year! P.S. I do hope that Debconf12 dates are nailed down in the next 2 months, so we can start talking to air plane companies for group discounts immediately.

John Goerzen: Complete.Org Migrated To Different Continent, 15 minutes downtime

It s time to change locations for my server. I ve been with CoreNetworks for almost 5 years. They provide a good value, with fully dedicated servers a lot cheaper than most places, and good support to boot targeting people that can handle root on their own box. I m switching, though, to Hetzner Online (of Germany), primarily because I am needing more than 1GB RAM, and they can provide a 2GB box for less than I pay now for 1GB. I asked for feedback on Google+, and got positive feedback. Today, I migrated from one machine to another, copying a dozen or two GB of data, and only had about 15 minutes downtime. I was able to do this without console access to the source machine, though console access wasn t really required for either. This will work with any basic Linux install on the destination, or it could be prepared directly from a rescue environment. Before I explain how I did it, I thought it was interesting to think of the different places that the machine that was known as complete.org has lived over the years. Here s a map (click for detail): Here s the general process. First, well in advance of the move, crank the TTL on the domains way down. This way, old IPs won t be cached for very long once the system moves. Next, start rsyncing data from the old machine to the new. Do not yet shut down daemons on the old. Shut down as much as you can on the new. You ll want to focus on static data, such as /home. /usr is a good candidate as well. /var if you are selective databases may be a good candidate, or may not. The idea is to seed the destination with data so that when we do the real rsync, most of the data will be there; it will have to tidy things up after daemons are stopped, and update some things, but the bulk of the work should be done. Next, start preparing some exclude lists for the final rsync, which will copy an entire machine to the other. You will likely want to exclude files such as /etc/fstab, /etc/network/interfaces, /etc/resolv.conf, /etc/hosts, /etc/default/grub, /sys/*, /proc/*, /dev/*, and the like. I used rsync -v -P -a -H -A -X -S numeric-ids delete-after. Now, start editing config files for the new IP, but do it on the old server (these will be synced over to the new one). Start with the bind config files, and touch anything else that needs it maybe Apache configs, whatever. Next, get ready to do the final sync. In an ideal environment, we d just shut both machines down to single-user mode, but that s not going to quite cut it here. Use ps and shut down all daemons except sshd and udev on the destination. Yes, including even syslog. This is to prevent anything actively accessing the disk during the rsync, and also to prevent any issues with clients accessing server daemons that aren t ready yet. Now, on the source machine, reload bind. This will start answering DNS queries with the new machine s IP, and should also propagate the changes to your secondary DNS. Next, kill off all the daemons on the source machine, except for ssh and bind. You might want to set /etc/nologin to prevent regular users from logging in on ssh, if you have them. Now rsync things over to the new machine. Do any final tweaking over there (merging in /etc/hosts maybe, dealing with the udev rules.d persistent net thing, etc.). Reboot the new server and you should be up and running.

3 August 2011

Lior Kaplan: FireGPG

FireGPG is an extension that provides GPG encryption and decryption options for Firefox. It was discontinued a year ago, and didn t support Firefox 4.0 due to changes it introduced. I was happy to see that the source is still available ,and also maintained enough to have a easy option to build the extension manually. For Firefox 5.0 just download the code and use the build.sh script to build an xpi file, then install it as usual. The Firefox 5.0 support got version number 0.8.5. I m thinking about repackaging it to Debian, as it was removed from Debian testing/unstable (version 0.8 is available for squeeze). Thanks bit for the maintenance work and Aaron C. de Bruyn for making sure the single repository holds these changes.
Filed under: Debian GNU/Linux, Free software applications, Mozilla

Andrew Pollock: [life] Zoe at 15 months

Boy, the last three months have been very eventful, developmentally. First off, about 2 weeks after she turned 1, Zoe started walking in earnest. Within about 2 weeks of that there was no stopping her. It's funny to think she's only been walking for under 3 months and yet she's very stable on her feet. Secondly, she developed a mild egg allergy. Sarah was giving her some scrambled eggs (one egg's worth) for lunch one day, and she mixed in some cream cheese, and Zoe absolutely scoffed it all down, so she scrambled another egg and she scoffed that down too. She then broke out in some hives around her mouth and Sarah noticed that she was coughing a bit and a little wheezy when she put her down for her afternoon nap. Upon subsequent consultation with her paediatrician and a blood test, the allergy was confirmed and we now have to avoid eggs and Zoe was prescribed an Epi Pen Jr. We went and saw an allergist as a followup, and he did some skin tests that showed eggs were the only thing she's allergic to. He said her allergy was very mild in the grand scheme of things, and we're going to do another skin test when she's 2 to see if she's grown out of it. Eggs in baked goods are generally okay, as the offending protein is denatured at high temperatures. She's more allergic to the egg whites (1.56, which is in the "moderate" range) than the egg yolks (0.55, which is in the "low" range). Egg allergies are apparently something that one does eventually out grow, so that's a good thing. We'd also been keeping her off gluten and dairy because of a strong correlation between them and rather disgusting bowel motions. We've successfully reintroduced gluten with no adverse effects, and we've just started her back on cow's milk. We of course had our epic trip to the Grand Canyon, which Zoe handled absolutely marvelously. That was closely followed by a weekend down in LA, which was a bit more of a disaster as Sarah and I both came down with a 24 hour stomach bug that Zoe had had during the week. There's nothing quite like languishing in a non-childproofed hotel room, feeling like death, with a full of beans toddler busy running around jamming her fingers in drawers. Fortunately we were able to use a babysitter for a few hours respite, but it did rather completely mess up our plans for the weekend. Zoe did handle the drive to and from LA very well though. Zoe's been doing Saturday morning swim classes at the YMCA, which I've taken her to. I don't think she's enjoyed them as much as the indoor ones she did at the Sutton Swim School, where she went with Sarah previously, and I definitely don't feel like the classes are as structured. Unfortunately we just finished an 8 week session, and the next one filled up before we got a chance to book, so we're hoping to get back into something at the Sutton Swim School once an opening comes up. I might do something ad hoc with her on Saturdays at the Rinconada Pool in Palo Alto in the meantime. Zoe has also started going to regular daycare on Thursdays, so Sarah gets a day to focus on her University study. It's a home daycare that isn't too far away, and our friend's daughter goes there too, so there's someone she knows. The first few times Zoe got pretty upset when she was dropped off, but she barely makes a whimper now when Sarah drops her off. She's also usually down to one nap a day now. She has four teeth, top and bottom, with one molar definitely through and another three starting to make an appearance. I just bought an iBert safe-T-seat for my bike, because I've been itching to get out on the bike with Zoe on the weekends when I'm looking after her so Sarah can get some more study done. The weather is currently nice, and I've just learned about a bike trail that cuts across the bay in front of Moffett Field, so I'm keen to check that out. Apparently there's geese out there. I did my first decent ride with her last Sunday to the Mountain View Farmer's Markets and back. Zoe seemed to be okay with it. The helmet we got her drew a lot of comments from people at the markets. She's very much in the "point and grunt" stage at the moment. What's really cool is she'll nod or shake her head in response to yes/no questions. It's so cool when you can ask her a question and she solemnly nods her head in response. Often in the morning Sarah takes her to the park around the back of our home, and so now Zoe's gotten into that habit, so she'll scratch around at the front door and they head over there. The other day, Sarah opened the front door and she took off in the direction of the park, so Sarah just followed her to see where she went. Sure enough, she made her way over to the park. Tonight, Zoe fetched me my shoes and then dragged me over by the hand to the front door to demand that I take her out for a walk. She's also really getting into books now. She actively seeks out books, and has a bit of a flick through them, or brings one over and climbs into our laps, or backs up and sits in our laps if we're sitting on the floor, to have us read them to her. She's now strong enough to hold her bottle herself at bedtime, so I sit with her in my lap in the glider and read a story to her while she has her bedtime bottle. It's indescribably wonderful. She seems to understand quite a large array of words. We can go "Where's your <insert object here> and she'll look around the living room and generally find the item we're talking about. She also seems to understand questions about breakfast, lunch, going to the park, going to the bathroom, having a bath, etc as she seems to respond appropriately. In the speech department, I haven't really counted the words, but she's not saying a huge number of words. This morning, apparently she yelled "MAMA! MAMA! PARK!". "Park" being a new word for her. We bought an annual membership to the Children's Discovery Museum in San Jose, and we've probably taken her there half a dozen times or so since getting the membership. She particularly loves the water area. We finally got around to getting our back yard landscaped and made generally fit for a small child to run around in. Sarah's planted a butterfly garden in one corner, so hopefully once that's established we'll have some fun things for Zoe to watch from the window. I'm looking forward to planting some bulbs in the autumn. I expect the next 3 months will be more of the same. We've got some more overseas travel coming up, but a lot of relatives will be around, so hopefully we'll get a bit of downtime to ourselves. Zoe and I on my bike

Robert Millan: Recent improvements with Debian GNU/kFreeBSD

Debian GNU/kFreeBSD was first released with Squeeze in last february. The technology preview label indicated, among other things, that it had a number of limitations when compared with what users would expect: missing features, incomplete functionality, etc. But it has seen many noteworthy improvements since then. Here are some that I would like to mention:

Tollef Fog Heen: libvmod_curl using cURL from inside Varnish Cache

It's sometimes necessary to be able to access HTTP resources from inside VCL. Some use cases include authentication or authorization where a service validates a token and then tell Varnish whether to proceed or not. To do this, we recently implemented libvmod_curl which is a set of cURL bindings for VCL so you can fetch remote resource easily. HTTP would be the usual method, but cURL also supports other protocols such as LDAP or POP3. The API is very simple, to use it you would do something like:
require curl;
sub vcl_recv  
    curl.fetch("http://authserver/validate?key=" + regsub(req.url, ".*key=([a-z0-9]+), "\1"));
    if (curl.status() != 200)  
        error 403 "Go away";
     
 
Other methods you can use are curl.header(headername) to get the contents of a given header and curl.body() to get the body of the response. See the README file in the source for more information.

2 August 2011

Russ Allbery: Git Debian packaging notes revised

It's a new month, so as usual for the beginning of the month I'm hoping to be a bit better about regularly updating, and ideally with more than just software release announcements and book reviews. Sadly, today has not been a good day for concentration despite a good early start, but I did manage to finally get through a long-delayed revision of my page of notes on using Git for Debian packaging. The revision drops the idea of using a separate debian branch to track Debian-specific changes so that master could be a pure feature branch. While appealing for its conceptual purity, this ended up confusing people horribly and was extra work for no apparent benefit. There are also a variety of other revisions to bring things up to date and a few new minor tips. Unfortunately, I haven't yet had a chance to write up how to maintain the upstream branch as a merge between the upstream Git repository and the released tarball, an idea from Sam Hartman that I'm now using very successfully for the openafs package, but at least now there's a pointer to it. I'm not really sure what to do with the initial part of that page that offers an introduction to Git. I've now written about three of these in separate places, and I'm still unconvinced of how much they actually help. And there are a ton of tools and basic techniques that I use all the time in Git that aren't mentioned in there (like git pull --rebase --stat, which I aliased to git update, or the whole concept of aliases, or how this all plays with 3.0 source packages). I've left the basic instructions for now, but I think at some point I may cede the tutorial aspect to other sites and just collect neat tricks that aren't part of the basic workflow that everyone describes.

1 August 2011

Andreas Metzler: down

This weekend I looked down. there is the Bregenzer Ache

31 July 2011

Russell Coker: Links July 2011

The Reid Report has an article about the marriage pledge that Michelle Bachmann signed which implies that slavery wasn t so bad [1]. Greg Carey has written an interesting article for the Huffington Post about marriage and the bible [2], I always knew that the so-called conservatives weren t basing their stuff on the Bible, but the truth surprised me. Geoff Lemon has written an interesting blog post about the carbon tax debate in Australia [3]. He focuses on how small it is and how petty the arguments against it are. Lord Bacon wrote an interesting list of the top 100 items to disappear first in a national emergency [4]. Some of them are specific to region and climate but it is still a good source of ideas for things to stockpile. Markus Fischer gave an interesting TED talk about the SmartBird that he and his team built [5]. A flying machine that flaps it s wings isn t that exciting (my local department store sells toys that implement that concept), but having one closely match the way a bird s wings work is interesting.

30 July 2011

Torsten Werner: DebConf11: Jigsaw Progress in Debian

Tom Marble and Guillaume Mazoyer gave a talk about Jigsaw Progress in Debian during DebConf11 in Banja Luka. Guillaume is a Google Summer of Code student mentored by Tom and Sylvestre Ledru. The project is about modularizing OpenJDK which replaces the old fashioned classpath by a modulepath that knows about versioned dependencies. It can reduce the memory footprint and the startup time for applications running in the JVM. It would be interesting to match Debian package versions to Jigsaw module versions. Debian could influence the JDK version 8 in this area. Upstream is already building Debian packages but they do not follow the Debian policy and do not use the Debian packaging tools. Guillaume has uploaded 2 packages (jtharness, jtreg) to Debian. These packages are needed to run the upstream testsuite which ships 3484 individual tests. More than 99% of them are working in Debian. There is a GIT repository of his Jigsaw work at http://anonscm.debian.org/gitweb/?p=pkg-java/jigsaw.git;a=summary. The slides of the talk are available at http://penta.debconf.org/dc11_schedule/events/718.en.html.

28 July 2011

Ian Wienand: The quickest way to do nothing

As I was debugging something recently, an instruction popped up that seemed a little incongruous:
lea 0x0(%edi,%eiz,1),%edi
Now this is an interesting instruction on a few levels. Firstly, %eiz is a psuedo-register that simply equates to zero somewhat like MIPS r0; I don't think it is really in common usage. But when you look closer, this instruction is a fancy way of doing nothing. It's a little clearer in Intel syntax mode:
lea    edi,[edi+eiz*1+0x0]
So we can see that this is using scaled indexed addressing mode to load into %edi the value in %edi plus 0 * 1 with an offset of 0x0; i.e. put the value of %edi into %edi, i.e. do nothing. So why would this appear? What we can see from the disassembley is that this single instruction takes up an impressive 7 bytes:
8048489:	8d bc 27 00 00 00 00	lea    edi,[edi+eiz*1+0x0]
Now, compare that to a standard nop which requires just a single byte to encode. Thus to pad out 7 bytes of space would require 7 nop instructions to be issued, which is a significantly slower way of doing nothing! Let's investigate just how much... Below is a simple program that does nothing in a tight-loop; firstly using nops and then the lea do-nothing method.
#include <stdio.h>
#include <stdint.h>
#include <time.h>
typedef uint64_t cycle_t;
static inline cycle_t
i386_get_cycles(void)
 
        cycle_t result;
        __asm__ __volatile__("rdtsc" : "=A" (result));
        return result;
 
#define get_cycles i386_get_cycles
int main()  
	int i;
	uint64_t t1, t2;
	t1 = get_cycles();
	/* nop do nothing */
	while (i < 100000)  
		__asm__ __volatile__("nop;nop;nop");
		i++;
	 
	t2 = get_cycles();
	printf("%ld\n", t2 - t1);
	i = 0;
	t1 = get_cycles();
	/* lea do-nothing */
	while (i < 100000)  
		__asm__ __volatile__("lea 0x0(%edi,%eiz,1),%edi");
		i++;
	 
	t2 = get_cycles();
	printf("%ld\n", t2 - t1);
 
Firstly, you'll notice that rather than the 7-bytes mentioned before, we're comparing 3-byte sequences. That's because the lea instruction ends up encoded as:
 8048388:       8d 3c 27                lea    (%edi,%eiz,1),%edi
When you hand-code this instruction, you can't actually convince the assembler to pad out those extra zeros for the zero displacement because it realises it doesn't need them, so why would it waste the space! So, how did they get in there in the original disassembley? If gas is trying to align something by padding, it has built-in sequences for the most efficient way of doing that for different sizes (you can see it in i386_align_code of gas/config/tc-i386.c which adds the extra 4 bytes in directly). Anyway, we can build and test this out (note you need the special -mindex-reg flag passed to gas to use the %eiz syntax):
$ gcc -O3 -Wa,-mindex-reg  -o wait wait.c
$ ./wait
300072
189945
So, if you need 3-bytes of padding in your code for some reason, it's ~160% slower to pad out 3-bytes with no-ops rather than a single larger instruction (at least on my aging Pentium M laptop). So now you can rest easy knowing that even though your code is doing nothing, it is doing it in the most efficient manner possible!

Tanguy Ortolo: Using a gamepad under GNU/Linux

Saitek PS2700 gamepad I have just bought a gamepad, the Saitek V.3 Cyborg Rumble Pad (also marketed as PS2700), mainly to play at SuperTuxKart under Debian GNU/Linux. It works perfectly, except for one minor feature, and it can be used for playing and as an X11 input device.System support This device uses USB HID so it is directly supported by the Linux kernel, with its module joydev, I guess: it appears as /dev/input/eventX and /dev/input/jsY. It can be tested with jstest-gtk from the package jstest-gtk: it has 7 axes (two sticks, one directional pad and two triggers that count as one single axis) and 13 buttons (4 regular, 2 rear, the 2 triggers that also send button events, 3 specials and the 2 sticks that can be clicked). They are presented in an order that is rather logical, except the trigger axis is inserted between the two axis of the second stick. It also features two strange buttons: Game support I only tested SuperTuxKart, but I think any game able to use a gamepad or a joystick should support it: after all, they all use the same protocol. The default mapping is rather logical but I find it more convenient to use the triggers rather than the main stick to accelerate and brake. I do not know if the vibration feature can be used at all, because I do not know how it is exposed and I do not think I have any game that would be able to take advantage of it. X.Org support This is an interesting feature: X.Org has a joystick(4) driver, available in the package xserver-xorg-input-joystick that allows you to use a joystick or a gamepad as an input device, both for pointing and typing. So, just for fun, I installed the joystick driver. It installs a default configuration file as /usr/share/X11/xorg.conf.d/50-joystick.conf, that is enough to provide features equivalent to a regular mouse. But given the number of axes and buttons of my pad, it deserved a special configuration in /etc/X11/xorg.conf.d/rumblepad.conf:
Section "InputClass"
    Identifier  "Rumble Pad"
    Option      "MatchIsJoystick"   "on"
    Option      "MatchProduct"      "Cyborg V.3 Rumble Pad"
    # Main stick axes: point
    Option      "MapAxis1"          "mode=relative axis=+1x deadzone=5000"
    Option      "MapAxis2"          "mode=relative axis=+1y deadzone=5000"
    # Second stick axes: scroll
    Option      "MapAxis3"          "mode=relative axis=+1zx deadzone=5000"
    Option      "MapAxis5"          "mode=relative axis=+1zy deadzone=5000"
    # Triggers: scroll vertically
    Option      "MapAxis4"          "mode=relative axis=+1zy deadzone=5000"
    # Directional pad: point
    Option      "MapAxis6"          "mode=accelerated axis=+1x deadzone=5000"
    Option      "MapAxis7"          "mode=accelerated axis=+1y deadzone=5000"
    # Buttons 1, 2 and 3 : click (no use for button 4)
    Option      "MapButton1"        "button=1"
    Option      "MapButton2"        "button=2"
    Option      "MapButton3"        "button=3"
    Option      "MapButton4"        "none"
    # Rear buttons: back and forward
    Option      "MapButton5"        "key=166"   # XF86Back
    Option      "MapButton6"        "key=167"   # XF86Forward
    # Triggers: no use
    Option      "MapButton7"        "none"
    Option      "MapButton8"        "none"
    # Special buttons
    Option      "MapButton9"        "key=135"   # select : Menu
    Option      "MapButton10"       "key=172"   # start : XF86AudioPlay
    Option      "MapButton13"       "key=180"   # home : XF86HomePage
    # Sticks click: click
    Option      "MapButton11"       "button=1"  # clic left stick
    Option      "MapButton12"       "button=3"  # clic right stick
EndSection
It is commented, but here is what it does: It is worth noting that joysticks have one feature that mice do not have: they allow to scroll at a variable speed, proportional to the shift of an analogue stick. This is the meaning of these zx and zy axes I mapped to the second stick, and it provides a scrolling experience that is far superior from that of a mouse wheel.

27 July 2011

Thorsten Glaser: (Banja Luka)

Been hot and dry today (although the sky is now back full of dark clouds), so I had a headache most of the morning until way past noon. Better now though, and I found a place where I could get Cevapi, which are really some sort of quick imbiss / fast food here (no uve pirina though, and she didn t have any Ajvar nor did she speak any language other than the local, but that wasn t a problem, only a bit dry because I didn t give in and took the offered Ketchup). Bought a 1 bottle of Kru kovac (from Hrvatska, though) and some small plastic glasses, then. I wonder how many people would, now, be willing to give Bosna i Hercegovina a try as holiday region (which might have been the intent of having a Balkan DebConf). I m sure I do. To all attendees: the hotel will give you some kind of stamped hardpaper card which states where you stayed on the trip, and for how long give that to the border guards when exiting Bosnia.

26 July 2011

Erich Schubert: Restricting Skype via iptables

Whenever I launch Skype on my computer, it gets banned from the university network within a few minutes; the ban expires again after a few minutes when I close Skype. This is likely due to the aggresive nature of Skype, maybe the firewalls think it is trying to do a DDoS attack. One of the known big issues of using Skype.
For Windows users, there are some known workaround to limit Skype that usually involve registry editing. These are however not available on Linux, unfortunately.
Therefore, I decided to play around with advanced iptables functionality. While you cannot match the originating process reliably (the owner match module seemed to include such functionality at some point, but it was deemed unreliable on multi-core systems). However, there are other and more efficient methods of achieving the same.
Here's my setup:
# Add a system group for Skype
addgroup --system skype
# Override permissions of skype (assuming Debian package!)
dpkg-statoverride --update --add root skype 2755  which skype 
And these are the iptables rules I use:
iptables -I OUTPUT -p tcp -m owner --gid-owner skype \
    -m multiport ! --dports 80,443 -j REJECT
iptables -I OUTPUT -p udp -m owner --gid-owner skype -j REJECT
They allow outgoing connections by Skype only on ports 80 and 443, which supposedly do not trigger the firewall (in fact, this filter is recommended by our network administration for Skype).
Or wrapped as pyroman (my firewall configuration tool; aptitude install pyroman) module:
"""
Skype restriction to avoid firewall block.
Raw iptables commands.
"""
iptables(Firewall.output, "-p tcp -m owner --gid-owner skype -m multiport ! --dports 80,443 -j %s" % Firewall.reject)
iptables(Firewall.output, "-p udp -m owner --gid-owner skype -j %s" % Firewall.reject)
which I've put just after the conntrack default module, as 05_skype.py

Matthew Garrett: Further adventures in EFI booting

Many people still install Linux from CDs. But a growing number install from USB. In an ideal world you'd be able to download one image that would let you do either, but it turns out that that's quite difficult. Shockingly enough, it's another situation where the system firmware exists to make your life difficult.

Booting a hard drive is pretty easy. The BIOS reads the first 512 bytes off the drive, copies them to RAM and executes them. That code is then responsible for either starting your bootloader or identifying the currently active partition and jumping to its boot sector, but before too long you're in a happy place where you're executing whatever you want to. Life is good. So you'd think that CDs would work in a similar way. The ISO 9660 format even leaves a whole 32KB at the start of a filesystem, which is enough space for a pretty awesome bootloader. But no. This is not how CDs work. That would be far too easy.

Let's imagine we're back in the 90s. People want to be able to boot off CD without needing a boot floppy to do so. And you're a PC vendor with a BIOS that's been lovingly[1] forced into a tiny piece of flash and which has to execute out of an almost as tiny piece of RAM if you want your users to be able to play any games. Letting boot code read arbitrary content off the CD would mean adding a new set of interrupt hooks, and that's going to be even more complicated because CDs have a sector size of 2K while hard drives are 512 bytes[2] and who's going to pay to implement this and for the extra flash and RAM and look surely there has to be another way?

So, of course, another way was found. The El Torito specification defines a way for shoving a reference to some linear blocks into the ISO 9660 header. The BIOS reads those blocks into memory and then redirects either the floppy or hard drive access interrupts (depending on the El Torito type) to that region. The boot code can then proceed as if it had been read off a floppy without all the trouble of actually putting a floppy in the machine, and the extra code required in the system BIOS is minimal.

USB sticks, however, are treated as hard drives. The BIOS won't look for El Torito images on them. Instead, it'll try to execute a boot sector. That isn't there on a CD image. Sigh.

A few years ago a piece of software called isohybrid popped up and solved this problem nicely. isohybrid is a companion to isolinux, which itself is a bootloader that fits into an El Torito image and can then load your kernel and installer from CD. isohybrid takes an ISO image, adds an x86 boot sector and partition table and does some more fiddling to turn a valid ISO image into one that can be copied directly onto a USB stick and booted. The world suddenly becomes a better place.

But that's BIOS. EFI makes this easier, right? Right?

No. EFI does not make this easier.

Despite EFI being a modern firmware for the modern world[3], EFI implementations are not required to be able to understand ISO 9660. In fact, I've never seen one that does. FAT is all the spec requires, and FAT is typically all you get. Nor will EFI just execute some arbitrary boot code from the start of the CD. So, how does EFI boot off CD?

El Torito. Obviously.

It's not quite as bad as it sounds, merely almost as bad as it sounds. While the typical way of using El Torito for a long time was to use floppy or hard drive emulation, it also supports a "No emulation" mode. It also supports setting a type flag for your media, which means you can distinguish between images intended for BIOS booting and EFI booting. But the fact remains that your CD has to include an embedded FAT partition that then contains a bootloader that's able to read ISO 9660 because your firmware is too inept to handle that itself[4].

How about USB sticks? Thankfully, booting these on EFI doesn't require any boot sectors at all. Instead you just have to have a partition table, a FAT partition and a bootloader in a well known location in that FAT partition. The required partition is, in fact, identical to the one you need in an El Torito image. And so this is where we start introducing some extra hacks.

Like I said earlier, isohybrid fakes up an MBR and adds some boot code that points at the actual bootloader. It needs to do a little more on EFI. The first problem is that the isohybrid MBR partition has to cover the entire ISO 9660 filesystem on the USB stick so that the operating system can access it later, but the El Torito FAT image is inside that partition. A lot of MBR-based code becomes very unhappy if you try to set up a partition that's a subset of another partition. So we can't really use MBR. On to GPT.

GPT, or the GUID Partition Table, is the EFI era's replacement for MBR partitions. It has two main advantages over MBR - firstly it can cover partitions larger than 2TB without having to increase sector size, and secondly it doesn't have the primary/logical partition horror that still makes MBR more difficult than it has any right to be. The format is pretty simple - you have a header block 1 logical block into the media (so 512 bytes on a typical USB stick), and then a pointer to a list of partitions. There's then a secondary table one block from the end of the disk, which points at another list of partitions. Both blocks have multiple CRCs that guarantee that neither the header nor the partition list have been corrupted. It turns out to be a relatively straightforward modification of isohybrid to get it to look for a secondary EFI image and construct a GPT entry pointing at it. This works surprisingly well, and media prepared this way will boot EFI machines if burned to a CD or written to a USB stick.

There's a few quirks. Macs will show two boot icons for these CDs[6], one marked "EFI Boot" and one helpfully marked "Windows"[7], with the latter booting the BIOS El Torito image. That's a little irritating, but not insurmountable. The other issue is that older Macs won't look for boot loaders in the legacy locations. This is where things start getting horrible.

Back in the old days, Apple boot media used to have a special "blessed" folder. Attempting to boot would involve the firmware looking for such a folder and then using that to start itself up. Any folder in the filesystem could be blessed. Modern hardware doesn't use boot folders, but does use boot files. For an HFS+ filesystem, the inode of the bootloader is written to a specific offset in the filesystem superblock and the firmware simply finds that inode and executes it. And this appears to be all that older Macs support.

So, having written a small tool to bless an HFS+ partition, I tried the obvious first step of burning a CD with three El Torito images (one BIOS, one FAT, one HFS+). It failed. While Refit could see the bootloader in the HFS+ image, the firmware appeared to have no interest at all in booting off it. Yet Apple install media would boot. What was the difference?

The difference, obviously, was that these earlier Macs don't appear to support El Torito booting. The Apple install media contained an Apple partition map.

The Apple partition map (APM) is Apple's legacy partition table format. Apple mostly dropped it when they went to x86, where it's retained for two purposes. The first is for drives that need to be shared between Intel Macs and PPC ones. The second seems to be for their install DVDs. Some further playing revealed that burning a CD with an APM entry pointing at the HFS+ filesystem on the CD gave me a boot icon. Problem solved?

Not really. Remember how I earlier mentioned that ISO 9660 leaves 32KB at the start of the image, and that an isohybrid image then writes an MBR and boot sector in the first 512 bytes of that, and the GPT header starts 512 bytes into a drive? That means that it's easy to produce an ISO that has both a boot sector, MBR partition table and GPT. None of them overlap. APM, on the other hand, has a header that's located at byte 0 of the media, overlapping with the boot sector. And it has a partition listing that's located at sector 1, overlapping with the GPT. Is all lost?

No. Merely sanity.

The first thing to remember is that the boot sector is just raw assembler. It's a byte stream that's executed by the CPU. And there's a lot of things you can tell a CPU to do that result in nothing happening. Peter Jones pointed out that the only bits of the AFP header you actually need are the letters "ER", followed by the sector size as a two byte big endian integer. These disassemble to harmless instructions, so we can simply move the boot code down a little and stick these at the beginning. A PC that executes it will read straight through the bizarre (but harmless) Apple bytes and then execute the real boot code.

The second thing that's important here is that we were just given the opportunity to specify the sector size. The GPT is only relevant when the image is written to a USB stick, so assumes a sector size of 512 bytes. So when the GPT starts one sector into the drive, it's actually starting 512 bytes into the drive. APM also starts one sector into the drive, but we can simply put a different sector size into the header and suddenly we're able to choose where that's going to be. 2K seems like a good choice, and so the firmware will now look for the header at byte 2048.

That's still in them middle of the GPT partition listing, though. Except we can avoid that as well. GPT lets you specify where the partition listing starts and doesn't require it to be immediately after the header. So we can offset the partition listing to, say, byte 8192 and leave a hole for the Apple partition map.

And, shockingly, this works. Setting up a CD this way gives a boot icon on old Macs. On new Macs, it gives three - one for legacy boot, one for EFI boot via FAT and one for EFI boot via HFS. Less than ideal, but eh. The one remaining problem is that this doesn't work for USB sticks (the firmware sees the GPT and ignores the APM), so we also need to add a GPT entry for the HFS+ partition. Job done.

So, it is possible to produce install media that will work if burned to CD or written to a USB stick. It's even possible to produce a version that will work on Macs, as long as you're willing to put up with three partition tables and an x86 boot sector that doubles as an APM header. And patches to isohybrid to do all of this will be turning up as soon as I tidy the code to the point where it works without having to hack in offsets by hand.

[1] Insert some other adverb here if you feel like it
[2] Why yes, 15 years later BIOSes still tend to assume 512 bytes. Which is why your 4K sector disk is much harder to work with than you'd like it to be.
[3] Ever noticed how the modern world involves a great deal of suffering, misery and death? EFI fits into that world perfectly.
[4] Obviously if you want your media to be bootable via both BIOS and EFI you need to produce a CD with two El Torito images. BIOS systems should ignore the image that says it's for EFI, and EFI systems should ignore the BIOS one. Some especially creative BIOS authors[5] have decided that users shouldn't have their choices limited in such a way, and so pop up a screen that says:
1.
2.
Select CD-ROM boot type:

and wait for the user to press a key. The lack of labels after the numbers is not a typographical error on my part.
[5] Older (pre-2009, and some 2009 models) Apple hardware has this bug if a dual-El Torito CD is booted via the BIOS compatibility layer. This is especially unfortunate because said machines often fail to provide a working keyboard emulation at this stage, resulting in you being stuck forever at an impressively unhelpful screen. This isn't a Linux bug, since it's happening before we've run any of our code at all. It's not even limited to Linux. 64-bit install media for Vista SP1, Windows 7 and Server 2008 all have similar El Torito layout and all trigger the same bug on Apple hardware. Apple's aware of this, and has resolved the issue by declaring that these machines don't support 64 bit Windows.
[6] Even further investigation reveals that Apple will show you as many icons as there are El Torito images, which is a rare example of Apple giving the user the freedom to brutally butcher their extremities if they so desire
[7] "Windows" is Apple code for "Booting via BIOS compatibility". The Apple boot menu will call any filesystem with a BIOS boot sector Windows.

Next.

Previous.