As we grow old, so do our computing machines. And just like we don t decommission ourselves, so should be the case of the machines. They should be semi-retired, delegating major tasks to newer machines while they can still serve some less demaning work: File Servers, UPNP Servers et cetera. It is common on a Debian installer based derivative, and otherwise too, to use block encryption on Linux. With machines from this decade, I think we ve always had CPU extension for encryption. So, as would be the usual case, all my laptops are block encrypted. But as they reach the phase of their life to retire and serving as a headless boss, it becomes cumbersome to keep feeding it a password and all the logistics involved to feed it. As such, I wanted to get rid of feeding it the password. Then, there s also the case of bad/faulty hardware, many of which mostly can temporarily fix their functionality when reset, which usually is to reboot the machine. I still recollect words of my Linux Guru - Dhiren Raj Bhandari - that many of the unexplainable errors can be resolved by just rebooting the machine. This was more than 20 years ago in the prime era of Microsoft Windows OS and the context back then was quite different, but yes, some bits of that saying still apply today. So I wanted my laptop, which had LUKS set up for 2 disks, to go password-less now. I stumbled across a slightly dated article where the author achieved similar results with keyscript. So the thing was doable. To my delight, Debian cryptsetup has the best setup and documentation in place to do it with just adding keyfiles and the nice integration in crypttab to ensure your keys propagate to initramfs The whole thing took me around 20-25 minutes, including drafting this post. From Retired Head and Password Prompt to Headless and Password-less. The beauty of Debian and FOSS

Debian CD/DVD ISO images are huge files (4.7 GB for a DVD). It can take a long time to download if your connection with the server is not very fast. Jigdo can help you download them much more quickly if you have a high-speed connection to a normal Debian mirror. If your download rate from your usual Debian mirror is not better than the download rate from the closest Debian CD mirror, then you should download your images straight from the Debian CD mirror. Otherwise read on and learn how you can save lots of time. Jigdo is based on a simple idea: the CD images contain mainly thousands of .deb files (and source files too) that are already available on all Debian mirrors. So instead of downloading the ISO image in its entirety, you could reconstruct it with all the files that it contains, that you would download from your usual Debian mirror. Thus when you download a .jigdo file, you only download a little text file (compressed with gzip) that contains enough information to reconstruct the complete image:

• it contains a list of the individual files that are needed to rebuild the image;
• it indicates a mirror where those files can be downloaded (it s mainly a fallback URL in case our preferred mirror doesn t have them anymore);
• it contains an URL to a .template file which describes the layout of all files within the ISO image. It also contains the data which can t be downloaded from the mirror.

The tool that we will use to regenerate the ISO images is called jigdo-lite, it s available on Windows if you don t have any Debian system yet. Otherwise just install the jigdo-file Debian package. We just need the URL of a .jigdo file (get one from the Debian CD page) and we pass it to jidgo-lite on the command line. It asks for a path of Files to scan : if we have an older version of the same CD image, we can indicate a path where it is mounted and it will avoid redownloading the files that we already have, otherwise just press Enter. Then it asks for the Debian mirror to use and by default suggests the one used in /etc/apt/sources.list. Again pressing Enter is usually enough. You can even use --noask to avoid the prompts.

$ jigdo-lite http://cdimage.debian.org/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.jigdo
Jigsaw Download "lite"
Copyright (C) 2001-2005     jigdo@
Richard Atterer             atterer.net
Getting mirror information from /etc/apt/sources.list
Downloading .jigdo file
--2011-03-22 09:08:11--  http://[2001:6b0:e:2018::163]/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.jigdo
[...]
-----------------------------------------------------------------
Images offered by  http://cdimage.debian.org/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.jigdo':
  1: 'Debian GNU/Linux 6.0.1 "Squeeze" - Official Multi-architecture i386/amd64/source DVD #1 20110319-14:55 (20110319)' (debian-6.0.1-i386-amd64-source-DVD-1.iso)
Further information about  debian-6.0.1-i386-amd64-source-DVD-1.iso':
Generated on Sat, 19 Mar 2011 15:11:27 +0000
-----------------------------------------------------------------
If you already have a previous version of the CD you are
downloading, jigdo can re-use files on the old CD that are also
present in the new image, and you do not need to download them
again. Mount the old CD ROM and enter the path it is mounted under
(e.g.  /mnt/cdrom').
Alternatively, just press enter if you want to start downloading
the remaining files.
Files to scan: 
-----------------------------------------------------------------
The jigdo file refers to files stored on Debian mirrors. Please
choose a Debian mirror as follows: Either enter a complete URL
pointing to a mirror (in the form
 ftp://ftp.debian.org/debian/'), or enter any regular expression
for searching through the list of mirrors: Try a two-letter
country code such as  de', or a country name like  United
States', or a server name like  sunsite'.
Debian mirror [http://ftp.fr.debian.org/debian]: 
Downloading .template file
--2011-03-22 09:12:41--  http://cdimage.debian.org/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.template
[...]
Successfully created  debian-6.0.1-i386-amd64-source-DVD-1.iso'
-----------------------------------------------------------------
Finished!
The fact that you got this far is a strong indication that  debian-6.0.1-i386-amd64-source-DVD-1.iso'
was generated correctly. I will perform an additional, final check,
which you can interrupt safely with Ctrl-C if you do not want to wait.
OK: Checksums match, image is good!

And you re done, you have successfully downloaded an official Debian DVD with jigdo-lite. You can head over to the Debian CD page and find out the URL of the .jigdo files for the latest stable release. One last thing, Bluray images are only available with Jigdo because they are so huge that keeping complete ISO images would be a waste of disk resources of the CD images server (and its mirrors). Of course, if downloading ISO images and burning CD/DVD is not for you, you can always order some professionally printed CD/DVD. You will even help Debian by doing this.

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

6 comments Liked this article? Click here. My blog is Flattr-enabled.

Anthony Towns writes about using an improved version of jigdo to download CD/DVD images [1]. His improvement is basically to pipeline operation for better performance. Jigdo (the Jigsaw download) is a tool to download a set of files and then use them to create a CD or DVD image [2]. The idea is that most web sites that have CD or DVD images also have a collection of files which comprise the DVD image available. This removes the need to store the data twice (wasting disk space on mirrors and in some situations breaking web caching). I have never used jigdo, and for all Debian installations in recent times (the last few years at least) I download a small net-inst CD image and then have it download the other files from the net. I have Squid set up to cache large objects so this doesn’t waste too much of my precious network bandwidth (which is limited and expensive in Australia). Now I’m thinking about what the optimum method for doing installs might be. One thing that would be good would be to support multiple repositories, the packages have unique file names and checksums so it should be possible to check one repository and then check another if it’s not working. I don’t mean multiple “deb” lines in the APT configuration. What I would like to do is to have an NFS file server or web server with an archive of downloaded packages and have APT check there first before downloading a file. So APT could get a list of packages from the net and then get the actual files locally if they are available. The next thing that would be good is the ability to create a CD or DVD image dynamically and to store all temporary files. So I could download files from the repository and create a DVD image with just the packages that I need. Every time I create a DVD image my sub-set of the Debian archives would increase and the number of files actually downloaded in the creation process would be reduced. The effect would be to incrementally create a local mirror of the Debian repository. Then I would like to see a two-stage DVD install process. I would like to boot from a CD or DVD and start the install and then have it give a list of files needed (which could be stored on a USB device or floppy) to create further CDs or DVDs for installation. One situation where this could have been beneficial was when I was doing an emergency install of CentOS. I did the first part of the install (selecting packages etc) to determine which CDs were needed. It turned out that almost all CDs were needed even though some of the CDs had only a few files that I was installing. If the installer could have written a list of packages to a USB device then I could have downloaded just those packages and got the install working a lot sooner. It seems to me that it’s fairly common to do one test install and then do some dozens of other installs with the same settings. So the ability to create a DVD of exactly the needed files for the other dozens of installs would be a great benefit. Now this is just random commentary, unfortunately don’t have time to do any coding on this. But it seems obvious that something has to be done to improve the situation for developers and IT staff who need some degree of mirroring of the Debian package pool but who can’t do a full mirror. Back in 1996 I was able to mirror Debian over a 28K8 modem link and fit it on what was a reasonable hard drive by the standards of the day (incredibly tiny by today’s standards). Now I can’t practically mirror Debian over an Australian cable broadband connection and even by the standards of about 4 years ago (the age of most of my hard disks) the space requirements are significant. I hope this post helps inspire some interest in developing these features. As delays in joining Debian [3] is the topic of the day it should be noted that work on preparing DVD images can easily be done by people who are not DDs. Such software should work from Debian archives without requiring any changes to them, and thus nothing special is needed from the Debian project to start work.

• [1] http://azure.humbug.org.au/~aj/blog/2008/04/15#2008-04-15-jigdodl


• [2] http://atterer.net/jigdo/


• [3] http://etbe.coker.com.au/2008/04/17/motivation-and-perspective/

Share This

Kumar, a dedicated Debian dude has written following and I’m sponsoring the first ever Sponsored Post to my blog: Thanks to Kartik Mistry for sponsoring’ this blog post. I wanted him to do this because I don’t blog, and his blog has wide reception on Planet India FLOSS and Planet Debian. So, here goes! Jigdo is a wonderful way of constructing CD/DVD ISOs by dowloading smaller units (like deb packages) to construct an image on your machine, rather than download a 4.3 GB monster over internet only to realize that the connection stops half way or some such thing. I would also take this opportunity to publicize IIT Madras’ mirror, which mirrors quite a bit of free software. It has a (somewhat) offical Debian (x86, amd64) mirror at http://ftp.iitm.ac.in/debian
ftp://ftp.iitm.ac.in/debian And am x86 mirror for Ubuntu as well:
http://ftp.iitm.ac.in/ubuntu
ftp://ftp.iitm.ac.in/ubuntu It is also (most of the time) ftp.in.kernel.org and www.in.kernel.org, since it is an official Linux kernel mirror (the only one I know of in India). Anyway, let’s jigdo now! We are to prepare the DVD for Etch 4.0r1 (i386) using jigdo. Here are the steps: 0. Get Jigdo. You can get the “lite” version for any GNU/Linux distribution or Windows here:
http://atterer.net/jigdo/ But Debian/Ubuntu users just need to apt-get install jigdo-lite. 1. Visit http://cdimage.debian.org and go to the “Download CD/DVD images with Jigdo” section. 2. I need the official i386 “stable” DVD image for i386, so I click that. That takes me to a page with a directory listing with many “.jigdo” files and “.template” files. While the jigdo file is a gzipped and formatted list of files in the CD/DVD, the template file is one with a description of the image itself. See TechDetails.txt in the jugdo distribution (or /usr/share/doc/jigdo-file) for details. 3. I download debian-40r1-i386-DVD-1.jigdo and debian-40r1-i386-DVD-1.template to a directory on my computer, and run this: jigdo-lite debian-40r1-i386-DVD-1.jigdo 4. First, I am asked for files to scan. Since I don’t have any files on my disk already (or don’t want to use them), I just press Return. 5. I am then asked for a mirror. I specify the fastest mirror available:
ftp://ftp.iitm.ac.in/debian And that’s it. It is supposed to write a blank image and start populating it with Debian packages! Simple, huh? Depending on the speed of your connection, it should take a few hours (or days) to get a DVD image done. A CD would be much quicker. For people inside IIT Madras, we get 5-7 MBps to the mirror, so it takes around half an hour for the DVD image. So, now you know whom to ask for those Debian DVDs and CDs, if you’re near Chennai, right? Troubleshooting suggestions are available on the Debian CD-Image
website here: http://www.debian.org/CD/jigdo-cd/ So, hope this was useful. Kumar Appaiah