Search Results: "js"

17 February 2010

Russell Coker: Can the NBN Ever Break Even?

I previously wrote about how the National Broadband Network (NBN) seems more suited to porn delivery than regular Internet use [1]. It doesn t seem to be of much use really. In a particularly insightful comment John Hughes suggested that the real purpose would be TV delivery. The ABC is currently delivering 640*360 resolution MPEG4 files via iView for it s TV content. To use iView on Linux you need Jeremy Visser s Python-iView program [2] in conjunction with Luke Kenneth Casson Leighton s rtmp program [3]. Note that to get this working on Debian/Lenny you need to install the python-simplejson package as well. For the current ABC service it requires 251652 of data for a 3167.6 second Torchwood episode, that averages to 79.4KB/s of data. In contrast a video of Dan Gilbert from TED.com at high quality was in 640*480 resolution and required 301652KB of data for 1276.5 seconds which averaged out to 236.3KB/s. My ADSL2+ connection is theoretically capable of something over 1MB/s and occasionally gets such speeds for unusual download situations (such as downloading multiple large files at the same time). But generally I can t rely can t rely on sustained transfer rates of more than about 200KB/s. So I could watch streaming TED talks at reasonable quality, but for the best results I have to download them and watch them from disk. Assuming the same ratio of compressed data to raw pixels used for HDTV as used for TED talks a 1920*1080 HDTV resolution MPEG4 with the same quality as a TED talk would take 1600KB/s. It is possible to vary the compression level and possibly the usage of a TV stream would permit better compression than the usage of a TED talk for the number of pixels. But it seems reasonable to assume that something like 1600KB/s is needed for best HDTV, that is more than the vast majority of ADSL2+ installations can be relied on to sustain. But 100Mb/s would allow at least two 1920*1080 HDTV transmissions to be viewed at the same time, and maybe three or four as few homes have only one TV this should be of great interest. Now the question is how much people would pay for this. Currently there are two main pay TV companies in Australia, Foxtel which charges $916 for a 12 month plan [4] and Optus who s web site is so awful that I gave up before discovering what they cost (I will assume that they are competitive with Foxtel). Now I think it s reasonable to assume that $916 is at the high end of what potential customers are prepared to pay, as they are competing with free to air TV, DVD sales, Youtube, and downloads of pirate content. The current bank interest rate on term deposits from the Commonwealth bank is 6.3%, the interest rate on raising the finance would have to be greater than that let s assume 7%. So the $5000 per household will require an interest payment of $350 per annum. If a household signs up to pay TV services at a cost of $1200 per annum that might be enough money to pay slightly more than $350 to the NBN plus make a profit for the pay TV company. So if every household in Australia signed up for pay TV over the NBN it should be profitable. But that seems unlikely. The majority of the Australian population (IE the majority of the city population) is used to paying not much more than $240 per annum for a basic phone service and not much more than $360 per annum for a basic broadband Internet service. A bundled deal of $1600 per annum for phone, Internet and pay TV should allow the pay TV company and the NBN to be barely profitable if everyone accepts the deal (unless of course interest rates rise). If half the population aren t interested then the bundled deal would have to cost $2000 to have the potential for being profitable. If three quarters of the population aren t interested then it would have to cost $2300 or more! The Australian Bureau of Statistics documents that in 2008 the median household disposable income in capital cities was $593 per week [5]. So it seems that the idea of the NBN being profitable is based on plan to have more than three weeks of household disposable income per year being spent on Internet/phone/TV services. But fortunately for the pay TV companies and the content companies there is no requirement for the NBN to be profitable, it s being paid by tax money so if it loses money then we can just pay more tax! It s the ideal public private partnership , the citizens take all the risk and the corporations reap all the profits! James Purser describes how Stephen Conroy gave the TV networks something between $250,000,000 and $500,000,000 a year for the next two years [6]. It was claimed that this huge gift would be conditional on the production of more Australian content but they ended up not putting any conditions on it. In a strange coincidence Stephen Conroy did this a month after having a meeting with Kerry Stokes (head of Channel 7). Based on this I don t believe that there was ever a serious plan to make the NBN profitable. I think that the plan was just to take our tax money and spend it on things that benefit friendly corporations. Really it s better for us if the government just hands out $500,000,000 at a time instead of spending $43,000,000,000 on a project that has the same aim of giving a few hundred million to cronies.

15 February 2010

Christine Spang: a foray into emacs

This spring I'm taking a class that does all its programming assignments in MIT Scheme. The last time I had to write Scheme for class, we used PLT Scheme, and thus the DrScheme editor. This time, with that option out, I've decided to do the assignments in Emacs. It is, after all, an editor built out of lisp. I've used vim for the last five years or so. Some people ask me why this is so I do go to MIT after all, where Emacs was originally written. The answer to that is simply that Debian got to me first, and at the time at least, the people I knew there used vim. I've heard Emacs is good, but up until this point every time I've tried it I've thrown it down in frustration and then couldn't figure out how to exit the damn thing. This time I was more determined, and made it through the initial hump of being completely useless at the editor. Here are a few of my observations so far, after mucking around for a couple days over a period of about two weeks: I'm pretty sure at this point that I'm still trying to bend emacs into being vim with different keybindings instead of figuring out how emacs wants to be used. It's also difficult to compare and contrast between the two editors in that my .vimrc contains years of tweaks and customizations whereas my .emacs started out as a clean slate. There are two things I'm doing to try to mentally separate emacs from vim, to avoid my fingers' muscle memory from becoming confused:
  1. I'm always using emacs in an X window with dark-on-light text, whereas I always use vim in a terminal with light-on-dark text.
  2. I'm only writing scheme in emacs. No other languages so far.
I'm planning to keep working on these programming assignments in emacs, despite it being annoying slower than I would be in vim. Tips and pointers for improving my experience are welcome.

6 February 2010

C.J. Adams-Collier: PPA installation on karmic

I don t know how long add-apt-repository has been around, but I ve found it very useful for installing some of the bleeding edge stuff I want to test: 
$ for ppa in do-core team-xbmc nvidia-vdpau chromium-daily directhex/monoxide
do
  sudo add-apt-repository ppa:$ppa
done
$ apt-get update
$ apt-get install chromium-browser nvidia-glx-195 gnome-do xbmc monodevelop
Is there anything like this for debian proper, I wonder?
<script>//<script language="javascript" src="http://reddit.com/button.js?t=1"></script>

4 February 2010

Mike Hommey: xulrunner 1.9.1.6-2 diff stats

As part of preparing xulrunner 1.9.2 (which would lead to Iceweasel 3.6), I have gone through all the patches currently applied to the latest xulrunner release, to keep track of what should be upstreamed, what needed feedback, what patches I needed to update in upstream bugzilla, etc. As I have talked about our differences with upstream on a few occasions already, and as I was doing this grunt work anyway, I thought it would be nice to make the results public. First, a few reference numbers (excluding changes to generated configure files): A few observations on the above data: The latter used to be true, but is not anymore: xulrunner 1.9.1.6-2 added a lot of changes and is now up to: 98 files changed, 1107 insertions(+), 517 deletions(-). What happened ? will you ask me. A lot happened: the test suite, attempts at correctly installing C++ headers and IDL files, bug fixes. Work has also been done such that make clean/distclean doesn t leave stuff around and that xulrunner can be properly built twice in a row. Differences between 1.9.1.6-1 and 1.9.1.6-2 look like this: 31 files changed, 341 insertions(+), 167 deletions(-). Some were stolen from bugzilla (13 files changed, 162 insertions(+), 70 deletions(-)): Many others were sent: A few others were filed without a patch, as I felt a proper upstreamable patch needed discussion (3 files changed, 16 insertions(+), 3 deletions(-)): A few changes are related to the way we build and how we run the test suite (5 files changed, 35 insertions(+), 18 deletions(-)): Finally, a few were related to the Debian changes history (2 files changed, 1 insertions(+), 26 deletions(-)): Going through the rest of the changes between upstream and the latest Debian release, we can split in 5 categories of changes:
Those that were picked from upstream, or sent and eventually applied in 1.9.2 or trunk (12 files changed, 68 insertions(+), 41 deletions(-)): Those that were picked from bugzilla or sent there, but are not applied upstream for various reasons (29 files changed, 138 insertions(+), 209 deletions(-)): Those that were reported but for which I wanted feedback for a proper fix, yet a patch was applied in Debian because this was necessary (6 files changed, 22 insertions(+), 13 deletions(-)): Those that are yet to be sent to bugzilla (15 files changed, 111 insertions(+), 54 deletions(-)): And finally, the changes that are either Debian specific or not deemed upstreamable (30 files changed, 460 insertions(+), 77 deletions(-) ; this can sound huge, but see the details below): Note, the numbers in the various diff stats obviously don t add up, since some changes overlap. All that information doesn t account the few additional components added in debian/filemonitor and debian/extra-stuff, that account for nearly 600 lines (excluding licensing boilerplate), but I ll talk about these in another post. This analysis also helped spotting another harmless merge error (the first one was spotted a few weeks ago and just left the enable-system-lcms around, without any effect): a variable replacement in a Makefile that doesn t have any effect, since the 2 variables have the same value. Only a pointless 1 line change.

2 February 2010

Craig Small: procps: Third time lucky

OK, ok, i got a chroot and pbuilder now. So that should, I hope stop any more FTBFS bugs about missing depdendencies.

procps got uploaded that fixes some important bugs, but mainly they were small fiddly things. About the most significant enhancement was pmap now has a real working -x flag. It looks a lot like some of the other pmap programs out there and shows the RSS and Dirty bytes per map. Let me know if its useful or not.

However there still is 48 bugs in the package, so if you're feeling game wander over to procps bug page and have a look around, but here are some more interesting ones, such as why would a process start time be earlier than the boot time? Bug 408879 has this problem

Now, a nice can of worms is in a Linux system, what is free memory? What should the "free" program report? Currently free just reports what it sees in /proc, but in Bug 565518 should the slab count be taken out? I certainly won't be making any Debian-specific changes here as you could get different numbers depending on your distribution, or even worse the age of you Debian system.

procps is also my first attempt at using git-buildpackage which I found very helpful. There was one problem with it and that is how it works with the quilt patch program. If the quilt patches are applied, git doesn't know this and says all the files have changed. I know its how these two programs are supposed to work but its a little annoying.
<script defer="defer" src="http://static.zemanta.com/readside/loader.js" type="text/javascript"></script>

29 January 2010

Craig Small: The Great Australian Internet Firewall

Some what belatedly (hey I'm travelling!) I've changed the layout of my main website http://www.enc.com.au/ so it now has black panels and a little pop-up. It's all in the name of T he Great Australian Internet Blackout which is a protest about the upcoming Australian Governments Internet Filtering.

As per usual with this sort of thing, the proposed solution will not even meet its rather vague goals, unless you count filtering a 1000 or so websites a "goal". While the numbers vary, the best estimates is its way over 100 to 200 million websites. So that 1000 is 0.0005, or 1 in 200,000.

The filter is alternatively going to filter reported refused classification material or perhaps "other stuff"; they'll get back to you on the second one. For the first it works that somone finds a website of sufficient level of classification, reports it to the government and it gets filtered. And how many times have you come across RC websites, just by wandering around? That's not R rated websites, its not porn, its not even reasonably exotic porn, but stuff that's downright nasty.

The government, as per usual, has come out with the tired old protect the children rant. The filter doesn't stop spam, it doesn't stop weirdos on facebook or MSN chatting up your teenage daughter or son, it filters websites that, unless you or your child is REAL unlucky you will never see; ever.

So why am I against it? Surely filtering RC level websites is great right? And to be honest if that is all this ever was going to be, it's still a complete waste of my (and if you are an Australian Internet user) your money, but once the filtering is in, there is going to be a line outside the relevant ministers door to filter anything else anyone wants to filter, for example:
This sort of filtering while initially seeming ok, will get worse, much much worse. There is not a real strong push from the general public about filtering RC websites, most people see and hate spam much more. If you are an Australian citizen I ask you to visit the The Great Australian Internet Firewall website and write to your local member of parliament.

Related articles by Zemanta
<script defer="defer" src="http://static.zemanta.com/readside/loader.js" type="text/javascript"></script>

28 January 2010

Gunnar Wolf: Captchas are for humans...

Nobody cares about me, I thought. Whatever I say is just like throwing a bottle to the infinite ocean. No comments, no hopes of getting any, for several days. Weeks maybe? Not even the spammers cared about me. Until I read this mail, by Thijs Kinkhorst commenting to my yesterday post:
( )
(BTW, I was unable to comment on your blog - couldn't even read one letter of the CAPTCHA...)
And, yes, Drupal module captcha introduced in its 2.1 release (January 2) feature #571344: Mix multiple fonts. Only... no fonts were selected. Grah.

27 January 2010

Craig Small: Manilla, Git and Gjay

Work doesn't often send to me places as great as Manilla in the Philippines, but here I am. It's a reasonably modern place and to me feels more like America than Asia in so many ways, posssibly because of its history. One thing is for sure, noone follows road rules here. Red lights are a suggestion and a zebra crossing is just some painted lines that you do need to stop at.

As for food, it's not that different, in fact this sad lot is about what was different:
Generally though the food has been pretty good but nothing I could get at home.

Good news about Gjay, the previous maintainer said it was ok for SourceForge to hand over the control of the program to me so I've set it up in git and started working on it. Most of the work was getting the code up-to-date to the later gtk APIs and making it work with audacious instead of xmms. It's almost ready for (re)initial release and there is even an ITP ready to go.

Git is a rather interesting and new (for me at least) version control system. I've been using cvs for more years than I'd care to think about and svn but while it is a bit different as you'd expect I haven't had it get in my way. In fact I've been so impressed with Git I have put a few other projects into it, mainly with the collab maintence Debian project for a few of my packages.

<script defer="defer" src="http://static.zemanta.com/readside/loader.js" type="text/javascript"></script>

Gunnar Wolf: Packaging PKP OJS (Open Journals System)

New guidelines for periodic publications' websites at my University favor the different journals we have to use a standardized system And it makes quite a bit of sense. It is quite hard to explain to the people I work with that the content is not only meant to be consumed by humans, but also by other systems; the reasons behind rich content tagging and deep hierarchies for what they would just see as a list of words (think list of authors for an article, list of keywords, and so on). After all, aggregator databases such as Latindex and SciELO have achieved getting this understanding through. And I must be quite grateful, as the University's guidelines point to what appears to be a very well-thought and thorough system, the Open Journal Systems by the Public Knowledge Project, co-funded by several well-regarded universities. OJS is a GPL-2-covered PHP bundle. Anyway I am very glad at least one of my Institute's journal accepted the challenge and decided to go OJS. I know I will quite probably be administering this system long-term. And, being as snobbish as I am, I know I loathe anything installed in my machines that is not either developed by myself or comes in a Debian package. So, as it was not packaged, I made the package Note that I am still not filing an ITP (which means, I have not yet decided whether I will upload this to Debian) because I want first to make sure I do have the needed long-term commitment Besides, I am by far not a PHP person, and being responsible for a package Carries a nontrivial weight. Still, you might be interested in getting it. If you are interested, you can either download the .deb package or add it to your apt repositories (and stay updated with any new releases), by adding this to your /etc/apt/sources.list: deb http://www.iiec.unam.mx/apt/ lenny misc
deb-src http://www.iiec.unam.mx/apt/ lenny misc Note: My packaging has still a small bug: The installer fails to create the PostgreSQL database. The MySQL database works fine. I will look into it soon So far, I am quite impressed with this program's functionality and the depth/quality of its (online) documentation. Besides, its usage statistics speak for themselves: So, it is quite possible I will be uploading this into Debian in a couple of weeks (hopefully in time to be considered for Squeeze). The reasons I am making it available in my personal repository now is:

C.J. Adams-Collier: Font for composing Lushootseed

At the recommendation of David Beck, I have installed a TTF font from http://www.languagegeek.com/. It took a few minutes for me to figure out how to get it going, but it was pretty straightforward after that. Here are some quick instructions for those of you running Debian variants such as Ubuntu.
  1. Fetch the zip files from languagegeek.com and unpack: 
    $ sudo mkdir -m777 -p /usr/local/share/fonts/truetype/languagegeek/
$ cd /usr/local/share/fonts/truetype/languagegeek/
$ for zip in AboriginalSerif.zip AboriginalSans.zip
do
  wget http://www.languagegeek.com/font/$zip &&
  unzip $zip &&
  rm $zip
done
$ sudo chmod -R 755 .
  2. Register these fonts with the system using defoma (the debian font manager): 
    $ for font in *.ttf
do
  sudo defoma-font register truetype $PWD/$font
done
While performing the defoma registration, I was presented with a number of warning-ish-looking messages for each file processed. For better or worse, I am ignoring them: 
No CIDSupplement specified for Dotum-Bold, defaulting to 0.
No CIDSupplement specified for Batang-Regular, defaulting to 0.
No CIDSupplement specified for ZenHei-CNS, defaulting to 0.
No CIDSupplement specified for Batang-Bold, defaulting to 0.
No CIDSupplement specified for ZenHei, defaulting to 0.
No CIDSupplement specified for Dotum-Regular, defaulting to 0.
I used gnome-appearance-properties (System Preferences Appearance) to set my document font to Aboriginal Sans:
After telling Chromium that it should use these fonts, it renders all of the Lushootseed characters quite nicely.
<script>//<script language="javascript" src="http://reddit.com/button.js?t=1"></script>

22 January 2010

Wouter Verhelst: Clijsters out?

Whoa. Kim Clijsters, Belgium's number 1 female tennis player of the moment, just got booted out of the Australian Open by Nadia Petrova. Not what I'd expected especially not with this kind of score; 6-0 6-1. To call this "unexpected" would be a severe understatement. Seriously. That leaves Belgium's hope with Justine Henin or Yanina Wickmayer. I say "or", because they'll next meet eachother in Melbourne. Sounds like an interesting match, indeed.

16 January 2010

C.J. Adams-Collier: AoE root for KVM guests

So. I m trying to get familiar with libvirt and friends. To this end, I ve set up a Lucid virtual machine booting from PXE into an initrd environment which does a pivot_root to an AoE block device. The #virt channel on irc.oftc.net told me that in order to have libvirt provide PXE capability, I would have to install a recent version of libvirt. I built version 0.7.5-3 from sid on my karmic laptop and it seems to be working okay. I decided to set up the pxe root directoy in /var/lib/tftproot just because that s what the example code had in it. I had to manually configure a virtual network. Here is the XML config file: 
$ sudo virsh net-dumpxml netboot
<network>
  <name>netboot</name>
  <uuid>81ff0d90-c91e-6742-64da-4a736edb9a9b</uuid>
  <forward mode='nat'/>
  <bridge name='virbr1' stp='off' delay='1' />
  <domain name='example.com'/>
  <ip address='192.168.123.1' netmask='255.255.255.0'>
    <tftp root='/var/lib/tftproot' />
    <dhcp>
      <range start='192.168.123.2' end='192.168.123.254' />
      <bootp file='pxelinux.0' />
    </dhcp>
  </ip>
</network>
This, of course, depends on the pxelinux.0 file. Luckily, this is packaged up in syslinux and can be installed with a simple
$ sudo apt-get install syslinux
$ sudo mkdir /var/lib/tftproot
$ sudo cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftproot
 I had to create a pxelinux config file for the virtual machine (indexed by mac address). Note that I put a console=ttyS0,115200 argument on the kernel command line so that I can attach to the serial port from the host system for copy/paste debugging. Also of importance is the root=/dev/etherd/e0.1p1 argument, specifying which block device we ll be doing the pivot_root to eventually.
$ mkdir /var/lib/tftproot/pxelinux.cfg/
$ cat /var/lib/tftproot/pxelinux.cfg/01-52-54-00-44-34-67
DEFAULT linux
LABEL linux
SAY Now booting the kernel from PXELINUX...
KERNEL vmlinuz-lucid0
APPEND ro root=/dev/etherd/e0.1p1 console=ttyS0,115200 initrd=initrd.img-lucid0
 I decided to use the karmic kernel for lucid initially. I ll eventually switch over to the lucid kernel ;)
$ sudo cp /boot/vmlinuz-2.6.31-17-generic /var/lib/tftproot/vmlinuz-lucid0
 I copied /etc/initramfs-tools to ~/tmp/lucid so that I didn t mess up the system initrd scripts:
$ mkdir -p ~/tmp/lucid && cp -r /etc/initramfs-tools ~/tmp/lucid/
 Since mkinitramfs doesn t currently have a system for AoE root, I had to do a bit of fiddling. I copied the NFS root boot script and made a couple of modifications. 
$ diff -u /usr/share/initramfs-tools/scripts/nfs ~/tmp/lucid/initramfs-tools/scripts/aoe
--- /usr/share/initramfs-tools/scripts/nfs	2008-06-23 23:10:21.000000000 -0700
+++ /home/cjac/tmp/lucid/initramfs-tools/scripts/aoe	2010-01-15 14:56:28.098298027 -0800
@@ -5,59 +5,25 @@
 retry_nr=0
 # parse nfs bootargs and mount nfs
-do_nfsmount()
+do_aoemount()
  
-
 	configure_networking
-	# get nfs root from dhcp
-	if [ "x$ NFSROOT " = "xauto" ]; then
-		# check if server ip is part of dhcp root-path
-		if [ "$ ROOTPATH#*: " = "$ ROOTPATH " ]; then
-			NFSROOT=$ ROOTSERVER :$ ROOTPATH 
-		else
-			NFSROOT=$ ROOTPATH 
-		fi
-
-	# nfsroot=[<server-ip>:]<root-dir>[,<nfs-options>]
-	elif [ -n "$ NFSROOT " ]; then
-		# nfs options are an optional arg
-		if [ "$ NFSROOT#*, " != "$ NFSROOT " ]; then
-			NFSOPTS="-o $ NFSROOT#*, "
-		fi
-		NFSROOT=$ NFSROOT%%,* 
-		if [ "$ NFSROOT#*: " = "$NFSROOT" ]; then
-			NFSROOT=$ ROOTSERVER :$ NFSROOT 
-		fi
-	fi
+        ip link set up dev eth0
-	if [ -z "$ NFSOPTS " ]; then
-		NFSOPTS="-o retrans=10"
-	fi
+        ls /dev/etherd/
-	[ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-premount"
-	run_scripts /scripts/nfs-premount
-	[ "$quiet" != "y" ] && log_end_msg
+        echo > /dev/etherd/discover
-	if [ $ readonly  = y ]; then
-		roflag="-o ro"
-	else
-		roflag="-o rw"
-	fi
+        ls /dev/etherd/
-	nfsmount -o nolock $ roflag  $ NFSOPTS  $ NFSROOT  $ rootmnt 
+        mount $ ROOT  $ rootmnt 
  
-# NFS root mounting
+# AoE root mounting
 mountroot()
  
-	[ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-top"
-	run_scripts /scripts/nfs-top
-	[ "$quiet" != "y" ] && log_end_msg
-
-	modprobe nfs
-	# For DHCP
-	modprobe af_packet
+	modprobe aoe
 	# Default delay is around 180s
 	# FIXME: add usplash_write info
@@ -67,17 +33,13 @@
 		delay=$ ROOTDELAY 
 	fi
-	# loop until nfsmount succeds
+	# loop until aoemount succeds
 	while [ $ retry_nr  -lt $ delay  ] && [ ! -e $ rootmnt $ init  ]; do
 		[ $ retry_nr  -gt 0 ] && \
-		[ "$quiet" != "y" ] && log_begin_msg "Retrying nfs mount"
-		do_nfsmount
+		[ "$quiet" != "y" ] && log_begin_msg "Retrying AoE mount"
+		do_aoemount
 		retry_nr=$(( $ retry_nr  + 1 ))
 		[ ! -e $ rootmnt $ init  ] && /bin/sleep 1
 		[ $ retry_nr  -gt 0 ] && [ "$quiet" != "y" ] && log_end_msg
 	done
-
-	[ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-bottom"
-	run_scripts /scripts/nfs-bottom
-	[ "$quiet" != "y" ] && log_end_msg
(below is the full file in case udiff is less convenient) 
$ cat ~/tmp/lucid/initramfs-tools/scripts/aoe
# NFS filesystem mounting			-*- shell-script -*-
# FIXME This needs error checking
retry_nr=0
# parse nfs bootargs and mount nfs
do_aoemount()
 
	configure_networking
        ip link set up dev eth0
        ls /dev/etherd/
        echo > /dev/etherd/discover
        ls /dev/etherd/
        mount $ ROOT  $ rootmnt 
 
# AoE root mounting
mountroot()
 
	modprobe aoe
	# Default delay is around 180s
	# FIXME: add usplash_write info
	if [ -z "$ ROOTDELAY " ]; then
		delay=180
	else
		delay=$ ROOTDELAY 
	fi
	# loop until aoemount succeds
	while [ $ retry_nr  -lt $ delay  ] && [ ! -e $ rootmnt $ init  ]; do
		[ $ retry_nr  -gt 0 ] && \
		[ "$quiet" != "y" ] && log_begin_msg "Retrying AoE mount"
		do_aoemount
		retry_nr=$(( $ retry_nr  + 1 ))
		[ ! -e $ rootmnt $ init  ] && /bin/sleep 1
		[ $ retry_nr  -gt 0 ] && [ "$quiet" != "y" ] && log_end_msg
	done
There was also a small modification to the initramfs.conf file: 
$ diff -u /etc/initramfs-tools/initramfs.conf ~/tmp/lucid/initramfs-tools/initramfs.conf
--- /etc/initramfs-tools/initramfs.conf	2008-07-08 18:37:42.000000000 -0700
+++ /home/cjac/tmp/lucid/initramfs-tools/initramfs.conf	2010-01-15 14:33:38.088295207 -0800
@@ -47,14 +47,16 @@
 #
 #
-# BOOT: [ local   nfs ]
+# BOOT: [ local   nfs   aoe]
 #
 # local - Boot off of local media (harddrive, USB stick).
 #
 # nfs - Boot using an NFS drive as the root of the drive.
 #
+# aoe - Boot using an AoE drive as the root of the drive.
+#
-BOOT=local
+BOOT=aoe
 #
 # DEVICE: ...
I also needed to add aoe to the list of modules included in the initramfs:
$ echo aoe >> ~/tmp/lucid/initramfs-tools/modules
 In order to generate the initrd.img file from this new config, I ran the following:
$ sudo mkinitramfs -d ~/tmp/lucid/initramfs-tools/ -o /var/lib/tftproot/initrd.img-lucid0
 I created a lucid VM by installing from the desktop install disk. You can grab the ISO here: http://cdimage.ubuntu.com/daily-live/current/ I ll leave the creation of the virtual machine and installation as an exercise for the reader. I put the filesystem on an lvm volume group called vg0 in a logical volume called lucid0 (ie, /dev/vg0/lucid0). At this point, I created a new virtual machine called lucid0. Here is the xml for the domain: 
$ sudo virsh dumpxml lucid0
<domain type='kvm' id='1'>
  <name>lucid0</name>
  <uuid>96fbad21-4f25-5700-ddd8-1a565c7170ee</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-0.11'>hvm</type>
    <boot dev='network'/>
  </os>
  <features>
    <pae/>
  </features>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <interface type='network'>
      <mac address='52:54:00:44:34:67'/>
      <source network='netboot'/>
      <target dev='vnet0'/>
    </interface>
    <serial type='pty'>
   <source path='/dev/pts/4'/>
      <target port='0'/>
    </serial>
  <console type='pty' tty='/dev/pts/4'>
   <source path='/dev/pts/4'/>
      <target port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5901' autoport='yes' listen='127.0.0.1' keymap='en-us'/>
    <sound model='es1370'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
    </video>
  </devices>
</domain>
Now we re ready to start the AoE target and launch the virtual machine. If you don t have vblade installed, do so now:
$ sudo apt-get install vblade
 Start the target up with the following command:
$ sudo vbladed 0 1 virbr1 /dev/vg0/lucid0
 Now, if all goes well, you should be able to watch the virtual machine boot up and do its thing like so:
$ sudo virsh start lucid0 && sudo screen -S lucid0 sudo virsh ttyconsole lucid0 115200
 If you get errors about /dev/etherd/e0.1p1 not existing (these might look like this): 
Begin: Retrying AoE mount ...
err         discover    interfaces  revalidate  flush
err         discover    interfaces  revalidate  flush
mount: mounting /dev/etherd/e0.1p1 on /root failed: No such file or directory
Done.
then you might want to try restarting vbladed like this:
$ sudo kill -9 ps auwx grep vblade grep -v grep awk ' print $2 ' && sudo vbladed 0 1 virbr1 /dev/vg0/lucid0
 So. Now you should have a lucid gdm in your virt-manager console. Any questions? #virt on irc.oftc.net
<script>//<script language="javascript" src="http://reddit.com/button.js?t=1"></script>

13 January 2010

Matt Brubeck: Finding SI unit domain names with Node.js

I'm working on some ideas for finance or news software that deliberately updates infrequently, so it doesn't reward me for checking or reloading it constantly. I came up with the name "microhertz" to describe the idea. (1 microhertz once every eleven and a half days.) As usual when I think of a project name, I did some DNS searches. Unfortunately "microhertz.com" is not available (but "microhertz.org" is). Then I went off on a tangent and got curious about which other SI units are available as domain names. This was the perfect opportunity to try node.js so I could use its asynchronous DNS library to run dozens of lookups in parallel. I grabbed a list of units and prefixes from NIST and wrote the following script: 
var dns = require("dns"), sys = require('sys');
var prefixes = ["yotta", "zetta", "exa", "peta", "tera", "giga", "mega",
  "kilo", "hecto", "deka", "deci", "centi", "milli", "micro", "nano",
  "pico", "femto", "atto", "zepto", "yocto"];
var units = ["meter", "gram", "second", "ampere", "kelvin", "mole",
  "candela", "radian", "steradian", "hertz", "newton", "pascal", "joule",
  "watt", "colomb", "volt", "farad", "ohm", "siemens", "weber", "henry",
  "lumen", "lux", "becquerel", "gray", "sievert", "katal"];
for (var i=0; i<prefixes.length; i++)  
  for (var j=0; j<units.length; j++)  
    checkAvailable(prefixes[i] + units[j] + ".com", sys.puts);
   
 
function checkAvailable(name, callback)  
  var resolution = dns.resolve4(name);
  resolution.addErrback(function(e)  
    if (e.errno == dns.NXDOMAIN) callback(name);
   )
Out of 540 possible .com names, I found 376 that are available (and 10 more that produced temporary DNS errors, which I haven't investigated). Here are a few interesting ones, with some commentary: To get the complete list, just copy the script above to a file, and run it like this: node listnames.js Along the way I discovered that the API documentation for Node's dns module was out-of-date. This is fixed in my GitHub fork, and I've sent a pull request to the author Ryan Dahl.

12 January 2010

David Welton: Rough Estimates of the Dollar Cost of Scaling Web Platforms - Part I

I have been pondering the idea behind this article for a while, and finally had a bit of time to implement it. The basic idea is this: certain platforms have higher costs in terms of memory per concurrent connection. Those translate into increased costs in dollar terms.
Nota Bene: Having run LangPop.com for some time, I'm used to people getting hot and bothered about this or that aspect of statistics that are rough in nature, so I'm going to try and address those issues from the start, with more detail below. Methodology All tests were run like so: my new laptop with two cores and four gigs of memory was used as a server, and my older laptop was used to run the ab (Apache Benchmark) program - they're connected via ethernet. I built up to successive levels of concurrency, running first 1 concurrent connection, 2, 10, and so on and so forth. The "server" computer is running Ubuntu 9.10, "karmic". Platforms The platforms I tested: As stated above, it's pretty obvious that using Rails or Django for something so simple is overkill: Better Tests for the Future I would like to run similar tests with a more realistic application, but I simply don't have the time or expertise to sit down and write a blog, say, for all of the above platforms. If I can find a few volunteers, I'd be happy to discuss some rough ideas about what those tests ought to look like. Some ideas: Results With that out of the way, I do think the results are meaningful, and reflect something of what I've seen on various platforms in the real world. First of all, here we look at the total "VSZ" (as ps puts it) or Virtual Size of the process(es) in memory. Much of this might be shared, between libraries, and "copy on write" where applicable. The results are impressive: Rails, followed by Django and PHP eats up a lot of memory for each new concurrent connection. Rails, which I know fairly well, most likely suffers from several problems: 1) it includes a lot of code. That's actually a good thing if you're building a reasonably sized app that makes use of all it has to offer. 2) Its garbage collector doesn't play well with "copy on write". Which is what "Enterprise Ruby" aims to fix. Django and PHP are also fairly large, capable platforms when compared to something small and light like mochiweb. That said, excuses aside, Erlang and Mochiweb are very impressive in how little additional memory they utilize when additional concurrent connections are thrown at them. I was also impressed with Jetty. I don't have a lot of experience with Java on the web (I work more with J2ME for mobile phones), so I expected something a bit more "bloated", which is the reputation Java has. As we'll see below, Jetty does take up a lot of initial memory, but subsequent concurrent connections appear to not take up much. Of course, this is also likely another 'apples and oranges' comparison and it would be good to utilize a complete Java framework, rather than just a tiny web app with one JSP file. So what's this mean in real world terms of dollars and cents? As your Rails application gets more popular, you're going to have to invest relatively more money to make it scale, in terms of memory. For this comparison, I utilized the bytes/dollar that I'm getting for my Linode, which works out to 18,889,040.85 ($79.95 for 1440 MB a month). As we can see, to have a similar amount of concurrent users is essentially free for Mochiweb, whereas with Rails, it has a significant cost. This information is particularly relevant when deciding how to monetize a site: with Erlang and Jetty it would appear that scaling up to lots of users is relatively cheap, so even a small amount of revenue per user per month is going to be a profit, whereas with Rails, scaling up to huge numbers of users is going to be more expensive, so revenue streams such as advertising may not be as viable. It's worth noting that 37 signals, the company that created Rails, is a vocal supporter of charging money for products. There's another interesting statistic that I wanted to include as well. The previous graph shows the average cost per additional concurrent user, but this one shows how much the platform costs (using when there is just one user, so it acts as a sort of baseline: As we can see, Jetty is particularly expensive from this point of view. The default settings (on Ubuntu) seem to indicate that, for instance, the basic $20 a month Linode package would not be sufficient to run Jetty, plus a database, plus other software. I think that the Apache Worker number is off a bit, and may reflect settings made to handle a large number of connections, or perhaps a different MPM would make sense. Source Code / Spreadsheet The spreadsheet I put together is here: http://spreadsheets.google.com/ccc?key=0An76R90VwaRodElEYjVYQXpFRmtreGV3MEtsaWYzbXc&hl=en And the source code (admittedly not terribly well organized) is here: http://github.com/davidw/marginalmemory/

8 January 2010

Dirk Eddelbuettel: Review of 'Computational Statistics: An Introduction to R' in JSS

Somehow missed during the the end-of-year switchover was the fact that my review of Guenther Sawitzki's Computational Statistics: An Introduction to R (CRC / Chapman \& Hall, 2009) is now up on the Journal of Statistical Software website.

1 January 2010

C.J. Adams-Collier: SAN configuration (AoE)

With the help of a couple of friends, we ve put a 4.5T RAID-5 machine on our network and I m trying to figure out how to share the storage with the rest of the hosts. In the past, I have used NFS and CIFS/Samba to provide access to remote hosts. This has generally worked okay so long as the server stays online. I don t know if the results are going to be much different, but I am now trying a different approach. I plan to run an iSCSI server, and I ve already configured AoE (ATA over Ethernet). I ve exported a block device on the network segment and mounted it on a remote host. This was pretty easy to configure. There is a bit of documentation on the internet already, but I ll give another quick overview. I gave the storage server the unoriginal name san0 . This host is running debian lenny. I am testing the configuration from my debian sid development host, which has the similarly unoriginal name dev0 . So, think server when you see san0 and client when you see dev0 . I assume that you ve already got an LVM volume group set up. Mine is called vg0 . Correct the following examples to account for any differences. You can use disk partitions instead of LVM logical volumes.
Create a logical volume to be exported: cjac@san0:~$ sudo lvcreate /dev/vg0 -n e0.1 -L 5G Load the AoE kernel module: cjac@san0:~$ sudo modprobe aoe Install the package containing the vblade block device export server: cjac@san0:~$ sudo apt-get install vblade Export the block device. Note that the ethernet bridge on which I export the device is called loc : cjac@san0:~$ sudo vbladed 0 1 loc /dev/vg0/e0.1 Install the AoE discovery tools on the client: cjac@dev0:~$ sudo apt-get install aoetools Load the AoE kernel module: cjac@dev0:~$ sudo modprobe aoe Probe for exported AoE devices: cjac@dev0:~$ sudo aoe-discover Verify that our exported device was discovered: cjac@dev0:~$ test -e /dev/etherd/e0.1 && echo "yep"
yep
You can now treat /dev/etherd/e0.1 as you would any other block device. You can format it directly, or partition it and format a partition, use it as a device in your software RAID array, use it as swap space (ha), or something completely different. Now to figure out this iSCSI stuff
<script>//<script language="javascript" src="http://reddit.com/button.js?t=1"></script>

Craig Small: Happy new RC

It's the late afternoon of the first day of 2010 here, though I suppose its still 2009 for someone for a little while.

After a lot of waiting, JFFNMS release candidate 1 for 0.8.5 got uploaded to sourceforge. This release is mainly about fixing some database release bugs 0.8.4 had and they're all caused by the fact that working with PHP and database to release code is plain awful.

The problem is tracking changes in your database. So version 1 has 3 tables and 60 rows, version 2 has 4 tables and 90 rows, but what changed? Everything I've seen so far is a bit of a hack or is real fiddly. JFFNMS release process is both which is why I'll go and release several versions of C code or Debian packages before trying to crack that nut again.

If you are wondering what JFFNMS is, its a Network Management System. It makes graphs and red/green icons depending on the status of your routers and servers. Written in PHP and web based and of course licensed under the GPL
<script defer="defer" src="http://static.zemanta.com/readside/loader.js" type="text/javascript"></script>

30 December 2009

David Welton: Detecting BlackBerry JDE Version

Recently, I went back and added some preprocessor code (it's pretty much necessary in the world of J2ME) to ensure that Hecl would compile with older versions of the BlackBerry JDE. However, I also faced a problem: how to figure out what version of the JDE we're using. It could be my latest cold clouding my mind, but I couldn't find a simple way to do this. It never seems to be simple with the BlackBerry platform, unfortunately. I did, however, finally find a nice way to obtain this information programmatically: the bin/rapc.jar file, which ships with the JDE, contains a file called app.version, which, indeed, contains the version of the JDE in use. I hacked up this code to read it and print it out: <script src="http://gist.github.com/266010.js"></script>

25 December 2009

Clint Adams: Distributed shell scripts for Christmas

When being told about the differences between Puppet and Chef by Puppet fans, a few common talking points emerged. They included the terms DSL , declarative , and shell script . Shell script was probably the most entertaining meme, and referred to how, without dependencies, Chef was just like writing shell scripts in Ruby. In actuality, a Chef cookbook appears to be roughly 17 files in 7 different directories. Most of them end in .rb, but there is also a metadata.json to keep the metadata.rb company, and perhaps an .rdoc if you like that sort of thing. I have no idea who writes shell scripts this way. I am also stupefied that people say this is easier than Puppet, because I doubt I would want to perform system administration by writing one of these things. So when I started writing my own thing, I contemplated the dependency issue, and for a few minutes decided to be lazy and leave them out entirely. Once it dawned on me, however, that I was then writing shell scripts in Lua, I became ashamed. If I were going to go that route, why not just set up a git repository with a bunch of shell scripts, and set up cronjobs on each of the clients to do a git pull and run these shell scripts as root? After all, nothing beats shell scripting for shell scripting, not Perl, not Python, not Ruby, and least of all Lua. So I wrote a dependency resolver. It was easier than I had anticipated and I have vague plans to make it smarter than Puppet's. Then I'm justified in sticking to my Lua DSL . Now Steve has a list of requirements that seem to be relatively mild. I think he could easily get away with a VCS-shellscript-cron setup. The VCS pull can be tunneled over ssh or stunnel or anything else which will provide authentication and encryption, so there is no need for a fancy custom server with its own certificate authority. In short, all the tools are there and you just need glue. I will continue to pretend that I'll need more complex logic and power that only Lua can give me, so I'll end up doing VCS-Lua-cron until I change my mind.

20 December 2009

Wouter Verhelst: Good, not evil

There is a bit of a fluff online currently about the following clause in the "jsmin" code (whatever that is):
The software shall be used for good, not evil
This seems to have started when Google rejected a project based on that code due to its license being not free or open source according to their standards, and therefore not welcome anymore. The arguments then quickly degenerated into things like 'when did google stop being against evil'. But those are all besides the point. One of the most important properties of free and open source software is that anyone can use it for any purpose; there are no restrictions to using them. The DFSG (and hence, the OSD which was derived from the DFSG) encode this as follows:
No Discrimination Against Fields of Endeavor
The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.
You may think "how is 'evil' a field of endeavor", but that is the wrong question. To "use software for evil" can mean any of a number of things, including "nuclear research", "weapons development", "abortion", or, heck "the cash register in a butchery shop", depending on the ethical and moral framework through which the person writing the license sees the world. The ability to give someone a CD or DVD with a bunch of software on there, being able to tell them that they can just use this in any which way they see fit, is a very strong and important feature of the free and open source community. Every time someone comes up with a clause like the above, however, this ability is diluted somewhat; and if it is readily accepted within the greater free and open source movement, then eventually everyone interested in using a piece of software must first check whether they're not trying to use software that forbids someone's pet evil, and we lose one of the greatest strenghts that does exist for free software, but not for proprietary software. The sad thing is, the jsmin author seems to agree. From a video/transcript on which he talks about his absurd license clause is the following quote:
Also about once a year, I get a letter from a lawyer, every year a different lawyer, at a company I don't want to embarrass the company by saying their name, so I'll just say their initials IBM
[laughter]
saying that they want to use something I wrote. Because I put this on everything I write, now. They want to use something that I wrote in something that they wrote, and they were pretty sure they weren't going to use it for evil, but they couldn't say for sure about their customers. So could I give them a special license for that? Of course. So I wrote back this happened literally two weeks ago "I give permission for IBM, its customers, partners, and minions, to use JSLint for evil."
Or, in other words, all you have to do if you want to use this software for evil is set up a second company, tell Douglas that this second company wants to sell software that uses his software to people who might use it for "evil", even though the first company won't, and you're in business. Because Douglas doesn't really oppose evildoers. So the question is, why is that clause in there in the first place? There are only two possibilities; either Douglas didn't really think about those issues, in which case I hope he will one day see the light and remove the clause; or he did, and decided to go ahead and put that clause there anyway. And that would be evil.

Next.

Previous.