Search Results: "camrdale"

7 July 2008

Cameron Dale: blog software updated

I updated ikiwiki, the software used to generate this blog, to the version in backports (2.50 from 1.50). I tried very hard to not spam any aggregators, as the new version uses a different link format. Hopefully I succeeded, apologies if I didn't.

22 June 2008

Cameron Dale: apt-p2p 0.1.5: IMPORTANT update to expunge bittorrent nodes

I have just uploaded an important update to apt-p2p that is highly recommended for all users. At some point over the last 48 hours bittorrent nodes have started to infiltrate the apt-p2p DHT (you may have noticed many ValueError tracebacks in your log file). The (mainline) bittorrent DHT is very similar in protocol to apt-p2p, and so the nodes are able to partially communicate and pollute each others routing table. I didn't think this would ever happen, as there wouldn't seem to be a reason for them to ever come into contact, but somehow it did. I've made some minor changes that exploit the differences between the two protocols to exclude bittorrent nodes from the apt-p2p routing table, and to drop any requests from bittorrent nodes (which should prevent apt-p2p nodes from polluting the bittorrent DHT). However, it is very important that all users upgrade to this new version to prevent any further mixing of the DHTs.

20 June 2008

Cameron Dale: DebTorrent 0.1.7: Long-Lived Torrents for Testing/Unstable

This new version of DebTorrent finalizes the work I previously uploaded to experimental. This new version is the first in unstable to use unique piece numbers to create long-lived torrents for testing and unstable. It also comes with a new version of the apt-transport-debtorrent package, which can now tell APT not to cache any package files downloaded with DebTorrent, which should save some disk space for some people. The locations of the long-lived torrent files and piece files that DebTorrent uses has also changed, as they are now hosted on my merkel page and should be more up to date. However, I was not able to get the ssh trigger for the sync from ftp-master setup, nor am I able to access the projectb database through pygresql. If anyone knows how I can get those working, please email me. Here is the changelog for debtorrent: And for apt-transport-debtorrent:

10 May 2008

Cameron Dale: apt-p2p 0.1.3: now faster

I have updated apt-p2p to version 0.1.3, which is available now in unstable and my repository (see my previous post introducing apt-p2p). This update brings mostly speed improvements to finding peers to download from, and performing the download. Nodes in the DHT are now checked for responsiveness more thoroughly before adding them to the routing table, and rechecked more frequently afterwards. This should prevent bad nodes from propagating through the system, and departing nodes remaining in the system long after they have left. Downloading from peers is also quicker, as bad peers are dropped with a quicker timeout value and after a small number of errors. There was also a bug revealed by about 10% of the mirrors that caused downloads to fail, and was fixed in version 0.1.2. This update is STRONGLY recommended, as keeping unresponsive nodes out of the DHT will make everyone's experience better. If you tried apt-p2p and had some problems, I hope you'll consider trying again now. Unfortunately, these are enhancements I could not predict needing before releasing to the public, as the problems they solve are all caused by having a larger number of users, some of which are unresponsive. There are still a large number of peers (maybe 50% of them) that are unreachable, and so can not share any files with other peers. I don't have a good way to check yet, but you can go to sites such as this one to probe your client for you. Just enter the remote IP address and port (available on the status page) of your client in the form http://ipaddress:port/ (e.g. http://1.2.3.4:9977/). If the check returns any HTTP headers (even a 404 Not Found response) then you're fine, but if it doesn't return anything then your peer is firewalled or NATted and should probably be fixed. Here are the changelog entries for 0.1.2 and 0.1.3:

5 May 2008

Adrian von Bidder: apt-p2p in the LAN?

Going just from the title of Cameron's post about his apt-p2p tool: How about automatically running a zeroconf-enabled webserver serving /var/cache/apt/archives (and any mounted local repositories?) during install and while aptitude is running (and, if the user allows, as a daemon by default on a running system) and of course a corresponding sources.list line and apt retrieval module. This would hugely improve installation time for people with just a few machines who are too lazy to set up their own apt cache. Obviously, the details would be tricky, but since there is a security chain from Release.gpg to the .deb, downloading packages from untrusted peers shouldn't be a problem, even if the package name / version pair is not really unique (or even if someone actually tries a spoofing attack.) (There's a thought: running a tftp / dhcp server with a fai network boot in addition would be a step towards world domination, too, closely followed by automated hacking tools to install Debian on all computing devices on the network. Support call: “My printer only prints one page saying debian:~# after power on and then stops.”) Update 2008-05-07: Both Nijel and Florian Ludwig himself pointed out his apt-zeroconf project, which is unfortunately not very actively developed at the moment. Yet another thing I should/could do if I had the time.

4 May 2008

Cameron Dale: Introducing apt-p2p: a new P2P apt downloader

After many long months of planning and work, I have completed another peer-to-peer downloader for Debian. If you've been keeping track, that makes 2 now. This one is called apt-p2p, and as of yesterday it is available in unstable. The functionality is very similar to the first one I wrote, DebTorrent, so if you've used that one you should feel very comfortable. After installation you just add a localhost:9977 to the start of your sources.list entries (see the man page). The only difference is the port number (DebTorrent uses 9988 by default) and that you can use it on all your sources.list entries, whether they be deb, deb-src, official Debian archive, or any other archive. Then an apt-get update gets it started, and you can begin installing packages. Point your web browser to http://localhost:9977 to see what's going on. IMPORTANT: as with any P2P program, it works much better if you open a port through your NAT or firewall. Without this crucial step, you won't be able to share with any peers, and your lookup of peers to download from may take longer. Make sure to forward both TCP and UDP ports 9977, or whichever port you set in the config file. For more details, see the port forwarding section of the FAQ. The similarities with DebTorrent are all external, so let's look at how it differs from DebTorrent internally: Here's some technical details for those interested: Though I know it can be fast, I'm not yet sure if the peer lookup in the DHT will be quick enough to keep up with the downloading. All my tests so far show that it is, but until there are a number of peers out there trying it, I can't be sure. Also, I have some improvements in mind to enhance the speed, in particular the wait for a timeout to occur, so this may improve in the future. If you see a delay in downloading where apt seems to be stalled saying 'Waiting for headers', for now be patient and see what happens. It may be that apt-p2p is downloading in the background (it does this sometimes), or that it's waiting for a lookup in the DHT to complete. If it hangs for more than a minute, or there's errors in the log file, please file a bug so that I can look into it. Finally, for you DebTorrent fans don't worry, I haven't given up on it. Stay tuned for more info on it coming soon.

28 January 2008

Cameron Dale: DebTorrent 0.1.6: Finally, Unique Piece Numbers

I have uploaded a new version of DebTorrent to experimental and to my personal repository. This new version implements the unique piece numbers, which has been planned for quite some time now. Well, it's finally here. To summarize, unique piece numbers keep torrent's alive longer, by assigning files unique piece numbers that never get reused in that torrent. New files get new piece numbers added to the end, but peers in the old torrent can still share most of the old files with peers in the new torrent (in fact, it's the same torrent, but old and new peers have different ideas about what it contains). The creation of torrents from Packages files has also changed, as now 2 torrents are created, one for whatever arch the Packages file was for, and one for the Architecture:all files. Since the Arch:all files are the same for all architectures, this also allows for more sharing of common files between peers on different architectures. This change required a change in the cache directories, which is described in more detail in the NEWS file, but is handled almost automatically. The only thing to do is the make sure to do an apt-get update after upgrading so that the torrents can be restarted. There was also some changes in the statistics reporting. In the client, the uploaded and downloaded statistics will now persist over restarts so that you can see how much you've done over a longer period. The tracker status page also got an update, and now shows the total uploaded and downloaded bytes for each torrent, as well as some more descriptive names for some of the torrents. Due to the large changes in this release, I opted for uploading to experimental (and my repository) for now, so that it can be tested a bit. Please do test it, and let me know of any bugs, problems or concerns that you may have. Here is the changelog:

9 January 2008

Cameron Dale: DebTorrent 0.1.5: Now With Status Updates

It's been a while, but a new update to DebTorrent is available in unstable. There is also a new version of the helper apt-transport-debtorrent (0.2.0) to go with it, which was mostly the cause of the long delay (due to a pending apt transition to testing). It is now highly recommended to install both, read below for new reasons why. One of the previous problems with debtorrent was the incorrect status updates displayed by apt, due to it not being aware of pieces of large files that have already downloaded. Using these new versions together will help to improve the update status messages shown during a download. Now, apt-get update will display a status at the bottom that will look something like: 
DebTorrent: 837MiB left at 319 KiB/s (46m03s)
You may not be able to see it all if there are multiple updates to different mirrors under way. In aptitude, the status will appear on a line by itself at the bottom of the list of downloaded files. The status line will update about once per second, and should give a good indication of how much time is really left. The other (incorrect) status information presented by apt/aptitude will still be there but you can ignore it. I've only tested apt and aptitude, so I'm not sure how (or if) synaptic, adept or gnome-apt will display these messages, though they are general apt status messages, usually used for displaying the status while logging into an FTP server. This should also fix another problem that debtorrent can have with downloading large files (usually on slower connections). Since apt can't see the download happening for a long time, the connection would sometimes time out. With these status updates the connection is constantly active, so the timeouts shouldn't be a problem when downloading large packages (although they may still occur when doing an apt-get update, as there are no status updates during that process). I did implement a better solution, as suggested by Micha Politowski on the wiki. This solution uses the creation of sparse files to let apt know how much of a file has actually been downloaded. However, apt doesn't understand how a single process can download multiple files in prarallel, so it only follows the last file that was started, and complains when a file finishes that was not the last one that started. I've disabled that for now while I see if it's possible to modify apt to work better in this situation. Here are the changes for debtorrent: And for apt-transport-debtorrent:

14 October 2007

Cameron Dale: DebTorrent Needs a Sponsor

Since the conclusion of the Google Summer of Code, the DebTorrent project has been without a sponsor. I was going to email debian-mentors to see if anyone was interested, but I decided to first post to my blog to see if there are any Debian Developers reading this who have the time and are especially interested in sponsoring it. Here are some of the details of the package: It is currently maintained using an svn-buildpackage subversion repository, but I might be interested in switching to git if a sponsor is interested.

21 September 2007

Cameron Dale: 2Wire Router 2700HG-E Violates HTTP/1.1

I recently upgraded my Internet service with Telus, which included a new wireless router to replace my old high-speed modem. The new router is a 2Wire 2700HG-E which is configured through a web interface. The web interface worked fine at first on some other machines, but whenever I accessed it from my main desktop machine the pages would either fail to load, or have the images in the wrong locations. After much debugging (though it seems obvious now) I finally discovered that this was due to my enabling HTTP pipelining in FireFox on that machine (as suggested by Mozilla). Being the diligent citizen that I am, I of course filed a support request so that this could be fixed in the next version of the 2Wire software. Here is the response I received:
The HomePortal GUI was designed for internet browsers on default settings. We recommend keeping the pipelineing [sic] setting on "false".
Not exactly a good solution, and a little surprising, considering it's relatively straight-forward to implement. I know because I recently upgraded a simple HTTP handler from 1.0 to 1.1 specifically to support pipelining, as part of my work on DebTorrent. Now I've just read "Structured streams: a new transport abstraction", in which the authors state:
implementing pipelining correctly in web servers has proven challenging enough that seven years after the standardization of HTTP/1.1, popular browsers still leave pipelining disabled for compatibility
So it seems pipelining is disabled for a reason, though I had it enabled for years in FireFox and saw no problems (that I know of) until now. Maybe the problems are a thing of the past. Does anyone else have any experiences (good or bad) with pipelining to share?

26 August 2007

Cameron Dale: DebTorrent Usage: details of a successful dist-upgrade

I just completed a very successful and long overdue dist-upgrade of my unstable machine, using mainly DebTorrent for downloading the packages, so I thought I'd post some of my thoughts and experiences. The download consisted of 1294 packages to upgrade, totalling 1350 MB and taking 2h12m to download. Here are some of the good and bad things I observed. Good things There was a single other peer with me in the same torrent, and I managed to download 182 MB from him, which is about 13.5% of the total download. This is the first time I have noticed this downloading from peers occurring, as usually there are too few peers, too many torrents and too many possible packages to download for any sharing to occur. This will hopefully change in the future when more people start using DebTorrent, and when unique piece numbers are introduced to make the torrents last longer. However, it does show how the use of the backup HTTP downloader can seamlessly integrate with downloading from peers to provide a good user experience, even for early adopters. The CPU time used was only 10m46s, which translates to an average CPU usage of 8%, which is very reasonable. The average download speed was 174 KB/s, which is 58% of my maximum download speed. Though this may seem like a bad thing, my goal all along has been to make sure that the download time would not be more than twice as long as using HTTP. Of course, using DebTorrent may never be as fast as a straight HTTP download from a well-provisioned server, but that it not the point. The idea is to reduce the bandwidth needs of hosting a debian archive. But in the future, when there are many peers in a single torrent the download speed may be even faster than using HTTP, especially for peers with very high download rates that could not be matched by a single server. Things that need improvement The completion percentage reported by APT during the download was fairly inaccurate. Here are some sample readings I noted, compared with the actual completion percentage from the DebTorrent status page:
APT Reports Actual Completion
10% 40%
20% 70%
30% 85%
40% 90%
50% 93%
60% 96%
70% 97.5%
80% 98%
90% 99%
The discrepancy occurs because larger packages are broken up into pieces, so they can be partially downloaded without APT knowing about it (since only fully downloaded packages are passed to APT). Clearly this situation is far from ideal, and can lead to the user feeling that the download is progressing very slowly, or not at all. There are some plans to add status updates to the communication between APT and the DebTorrent client, but they require changes to the APT code to support them, so it may take some time to implement. Another problem is the memory usage I saw during the download, which was approximately 213 MB. This is obviously unnecessarily large, though the metainfo that DebTorrent needs to be aware of is quite large (stored in a text file it is about 3 MB). There does seem to be a memory allocation bug in Python 2.4 which causes increased memory usage, so moving to python 2.5 might help. However, my preliminary tests show this only saves you about 20% for DebTorrent. I think I will have to delve deeper into which parts are using all this memory, and unfortunately python doesn't seem to have a good memory profiler to help with this. I will be looking at both PySizer and Guppy/Heapy to start, but if anyone knows of a better solution, please let me know.

20 August 2007

Cameron Dale: DebTorrent Release 0.1.4 (and apt-transport-debtorrent 0.1.0)

The next release of DebTorrent is now available. This release includes new functionality for communicating with APT using a new transport method specifically designed for debtorrent, and many bug fixes. The major changes in this release are all in the communication between APT and DebTorrent. The HTTP server in DebTorrent that listens for APT requests has been upgraded to support HTTP/1.1 persistent connections and pipelining, which allows APT to have multiple outstanding requests for files. This is useful as DebTorrent suffers from the typical bittorrent slow start, so requesting multiple files at a time helps to speed up the download considerably. Though better, HTTP/1.1 is not ideal for DebTorrent however, as a maximum of 10 outstanding requests is maintained by APT's http method, and files must still be returned in the order they were requested (which is not ideal for bittorrent-type downloading since downloads occur randomly). To further improve the APT communication I have modified APT's http method to create a debtorrent method. This new debtorrent transport for APT is packaged separately as apt-transport-debtorrent, and once installed APT can be told to use it by replacing "http://" with "debtorrent://" in your sources.list file. This method sends all requests it receives immediately to DebTorrent, and will receive responses from DebTorrent in any order. You can find this new method on the Alioth project, or in my personal repository (only amd64 and i386 versions are currently available). Unfortunately, the story doesn't end here. The APT code responsible for sending requests to the method also limits the maximum number of outstanding requests that it will send to the method to 10, which is not really necessary since all existing methods limit the requests they send out themselves. I have therefore patched the current APT code to increase this limit to 1000 (a one line change), and released this patched version as 0.7.6-0.1. You can find this patched version in my personal repository (again, only for i386 and amd64). I have tested it with the other methods available and it causes no problems, and I hope to get the change included in the regular APT code soon. To sum up: The last DebTorrent version (0.1.3.1) is currently in the NEW queue, and judging by the length of it, will be there for about another week. After DebTorrent is added to the archive, I will be upgrading it to this new version. I also hope to get the new apt-transport-debtorrent package into the NEW queue soon. This brings to an end the Google Summer of Code (which this project was created as a part of), but development of DebTorrent will of course continue (probably a little slower). The next major change will be the addition of unique piece numbers, which is almost complete but needs to be extensively tested. I'd like to thank Anthony Towns, Steve McIntyre, and Michael Vogt for their help over the last 4 months, and also the many others who sent me encouraging emails or engaged in interesting discussions about this project. It's the people who make a project like this a fun and memorable thing to do. Here's the changelog for the new DebTorrent release:

Cameron Dale: DebTorrent Release 0.1.4 (and apt-transport-debtorrent 0.1.0)

The next release of DebTorrent is now available. This release includes new functionality for communicating with APT using a new transport method specifically designed for debtorrent, and many bug fixes. The major changes in this release are all in the communication between APT and DebTorrent. The HTTP server in DebTorrent that listens for APT requests has been upgraded to support HTTP/1.1 persistent connections and pipelining, which allows APT to have multiple outstanding requests for files. This is useful as DebTorrent suffers from the typical bittorrent slow start, so requesting multiple files at a time helps to speed up the download considerably. Though better, HTTP/1.1 is not ideal for DebTorrent however, as a maximum of 10 outstanding requests is maintained by APT's http method, and files must still be returned in the order they were requested (which is not ideal for bittorrent-type downloading since downloads occur randomly). To further improve the APT communication I have modified APT's http method to create a debtorrent method. This new debtorrent transport for APT is packaged separately as apt-transport-debtorrent, and once installed APT can be told to use it by replacing "http://" with "debtorrent://" in your sources.list file. This method sends all requests it receives immediately to DebTorrent, and will receive responses from DebTorrent in any order. You can find this new method on the Alioth project, or in my personal repository (only amd64 and i386 versions are currently available). Unfortunately, the story doesn't end here. The APT code responsible for sending requests to the method also limits the maximum number of outstanding requests that it will send to the method to 10, which is not really necessary since all existing methods limit the requests they send out themselves. I have therefore patched the current APT code to increase this limit to 1000 (a one line change), and released this patched version as 0.7.6-0.1. You can find this patched version in my personal repository (again, only for i386 and amd64). I have tested it with the other methods available and it causes no problems, and I hope to get the change included in the regular APT code soon. To sum up: The last DebTorrent version (0.1.3.1) is currently in the NEW queue, and judging by the length of it, will be there for about another week. After DebTorrent is added to the archive, I will be upgrading it to this new version. I also hope to get the new apt-transport-debtorrent package into the NEW queue soon. This brings to an end the Google Summer of Code (which this project was created as a part of), but development of DebTorrent will of course continue (probably a little slower). The next major change will be the addition of unique piece numbers, which is almost complete but needs to be extensively tested. I'd like to thank Anthony Towns, Steve McIntyre, and Michael Vogt for their help over the last 4 months, and also the many others who sent me encouraging emails or engaged in interesting discussions about this project. It's the people who make a project like this a fun and memorable thing to do. Here's the changelog for the new DebTorrent release:

11 August 2007

Cameron Dale: DebTorrent BugFix Release 0.1.3.1

Here's a new release of DebTorrent, containing only bug fixes. This release fixes 2 bugs, one minor, and one serious. The serious bug would probably have caused anyone using a recent (0.7) version of APT to experience hangs with APT saying "waiting for headers". If you had this issue and you're using APT 0.7, please update to this new version. If you had this issue with an older version of APT, please report it as a bug. Agian, if you do find any bugs or have any problems, please submit them to the DebTorrent mailing list, or come and find me (camrdale) on IRC in the #debtorrent channel on OFTC. Here's the changelog:

7 August 2007

Cameron Dale: DebTorrent Release 0.1.3 (in package form!)

Today I have released the next version of DebTorrent. This is the first release to be available in a binary package (.deb) form, and most of the changes are related to the packaging I still consider the program to be alpha quality, though all the functionality for a beta release is there, it just needs to be tested (so tell your friends). I run the program daily and use it for my apt-based updating, so I'm pretty sure it works, but there are definitely bugs. If you do find one please submit it to the DebTorrent mailing list. The installable binary can be found on the Alioth project, or in my personal repository which can be added to your sources.list like so: 
deb http://debian.camrdale.org/ unstable main contrib non-free
Once installed, it will start running automatically, and will restart on bootup, so all that is needed is to modify your sources.list files to point them at DebTorrent by prepending localhost:9988 to the mirror name. For example, the entry above for my personal repository would become: 
deb http://localhost:9988/debian.camrdale.org/ unstable main contrib non-free
Here's the changelog: