Search Results: "ema"

14 June 2012

Russell Coker: The Financial Value of a University Degree

I ve read quite a few articles about the value of a degree. Most of them come from the US where the combination of increasing tuition fees and uncertain job market makes a degree seem like a risky investment. I think that most analysis of the value of a degree are missing some important points. The Value of Money at Different Times The value of money is different at various stages of your life. The impression that I get is that when a married couple have their house fully paid off and they either don t/won t have children or their children are old enough to leave home the amount of money that they earn seems to matter a lot less. Doing a university degree involves 3 or 4 years not earning money (or more if doing post-graduate studies), which is usually starting at the age of 18. Effectively getting a degree involves giving up some money while young for the opportunity to earn more when older. Any analysis based on directly comparing the money spent on the degree to the amount of financial return without considering when money is needed is not very useful. I think that a reasonable analysis would exclude income earned after the age of about 45. By that age most people have either achieved a solid financial position and learned to live within their means or messed up their finances so badly that they won t live long enough to recover. A Degree as a Signal The Wikipedia page on economic signalling gives education as an example of a signal. A signal in this case means something that doesn t inherently mean anything but which signifies something else. So completing a degree doesn t necessarily mean that you learned anything relevant to work, but if you are able to do it then it means that you can probably also do things which are economically useful for an employer. This raises the question of how else one might signal their ability to work. One obvious answer is by working, someone who has remained steadily employed for 3 or 4 years has demonstrated their ability to work reliably and get along with other people which should be at least as useful as a signal. It s Not Only the Degree Most analysis seem to compare average income of people with degrees with the average of income with people who didn t attend university. That is based on the assumption that the degree was the only difference. When I was young my parents spent a moderate amount of money on a full set of paper encyclopedias (about 2 meters of shelf space). I m sure that this gave me some educational benefit as they intended, and it was something that was apparently quite rare I don t recall seeing a full encyclopedia in anyone else s house before the Wikireader [1]. My parents also bought me quite a lot of computer gear (back when hardware was really expensive), were always available to drive me to computer users group meetings etc, and did everything else that seemed likely to have an educational benefit. The value of such learning opportunities is significant. I think that almost everyone who had similar learning opportunities to me when they were young will probably have experienced similar support and pressure to attend university. I also think that almost everyone who receives such opportunities will be able to earn more than the median income even if they don t attend university. To a large extent people who are going to be successful attend university. A university degree doesn t make anyone successful if they couldn t succeed without a degree. There are some careers that just aren t options if you don t have a relevant degree (such as medicine and law). But I believe that anyone who is capable of completing a difficult course such as medicine or law (or any other career that has legal requirements for a degree) is capable of being successful without a degree in many other fields. So comparing the wages of a doctor or a lawyer to an average person doesn t make sense, it makes more sense to try and compare their wages to someone of similar skill who didn t have such a qualification. Conclusion It seems to me that the question is, of the people who had great learning opportunities when they were young and who wanted to succeed, would they have earned much less if they hadn t attended university? The next question is, of the people who might earn significantly less without getting a degree, would that salary difference really have mattered, or would it just be a matter of earning some luxury money when they are too old to really need it? Related posts:
  1. Ideas for a Home University There seems to be a recent trend towards home-schooling. The...
  2. university degrees Recently someone asked me for advice on what they can...
  3. Microsoft Hires University Drop-Out for Recruiting Campaign news.com.au reports that MS has hired former Miss Australia Erin...

10 June 2012

Gregor Herrmann: RC bugs 2012/23

here's the list of RC bugs I've worked on during the last week:

C.J. Adams-Collier: Spinning up a telco with Asterisk

Back in 2010 our family moved to an island in an archipelago in the very top left of the US s continental 48. Before moving, we did some research regarding high speed network access. There are a number of ISPs in the county, and even one headquartered on our particular island. But we found that the location where we were moving was in a location that was only serviced by 1.5Mbit DSL. We had been spoiled in our previous two locations and had seen bittorrent swarms reach over 1 megabyte (8Mbit) per second. Our bill said 12 down and 2 up and I wouldn t be surprised if we achieved that. At any rate, we were disappointed by the speeds available and I wanted to do something to increase the availability of higher speed network access. Some of my friends from the North American Network Operators Group (NANOG) told me that one way to improve the infrastructure would be to get involved with telecommunications provisioning as a Competitive Local Exchange Carrier (CLEC). With the above in mind, I did some research and found that in my state, the authority which certifies and regulates telecommunication companies is the Washington Utilities and Transportation Commission (UTC). I found the application forms on their web site, filled them in, filed the appropriate documents with the appropriate departments and received a certificate in the mail shortly thereafter. We have since filed annual reports as required by regulations. And with the regulatory bits out of the way, we re now left with only the technical details. We have established collocation of some equipment near our state s central telecommunications facility at the Westin Building. Technically speaking, the equipment is not in the Westin, but at a facility connected to the Westin by multiple 10Gbit links. Close enough for me. On this equipment, we re running a Debian Squeeze Xen virtual machine and Digium s Asterisk packages: 
cjac@sip0:/etc/asterisk$ grep -B1 -A3 Digium /etc/apt/sources.list
#
#  Digium asterisk packages
#
deb http://packages.asterisk.org/deb squeeze main
deb-src http://packages.asterisk.org/deb squeeze main
cjac@sip0:/etc/asterisk$ dpkg -l   grep digium
ii  asterisk                           1:1.8.11.1-1digium1~squeeze  Open Source Private Branch Exchange (PBX)
ii  asterisk-config                    1:1.8.11.1-1digium1~squeeze  Configuration files for Asterisk
ii  asterisk-core-sounds-en-gsm        1.4.21-1digium1~squeeze      asterisk PBX sound files - English/gsm
ii  asterisk-voicemail                 1:1.8.11.1-1digium1~squeeze  simple voicemail support for the Asterisk PBX
We saved up our nickles and dimes to purchase a 24-port analog telephony adapter (ATA). At the recommendation of one of the denizens of the #asterisk IRC channel on Freenode, we decided on the GrandStream GXW4024. It is pretty simple to configure them to register with the Asterisk server using SIP. The only problem we had was getting the RTP streams, and specifically their dynamically negotiated service ports, to traverse the NATting routers. We currently have no public IPv4 address space at the facility which hosts the ATA and so the rest of the internet sees all traffic originating from this facility as sourced from a single IP address. In order for this to work, the edge NAT/router device (in our case, BIG-IP LTM) needs to map its external L4 ports (TCP and UDP) to the ports of internal hosts so that traffic can flow correctly. This works fine for traffic which originates from internal hosts and is destined to hosts on the outside, but does not work so well when hosts on the outside wish to establish connections to hosts on the inside. Call management sessions established with the SIP protocol fall in the former category, but call media (voice, video, txt) sessions established with the RTP protocol fall into the latter; the host and port used to carry the media data are negotiated during the SIP session and the media socket is established out of band. The problem occurs when the call server (on the outside) attempts to establish an RTP stream with the caller (on the inside). The call server attempts to connect to the caller using the IP address from which it connected. The problem is that this address belongs to the NATting router, not the caller, and the router is not expecting the incoming request. To get around the NAT problem, we temporarily configured an OpenVPN tunnel to extend our L2 network segment. Our long-term solution will be to provision an Asterisk server in the same facility as the ATA and then register with it instead of the server in the exterior facility. Calls will then be switched at the local facility and the upstream link will be established using the IAX2 protocol, which does not suffer from the same NAT problems. All of the above provides us with the ability to make voice calls between the 24 channels of the ATA and any other SIP client which happens to be registered with our call server. This is all well and good, but we want to allow our customers to call the rest of the world. Enter our upstream provider, Threshold Communications, Inc. These folks are experienced and professional and have helped us successfully establish both outbound and inbound calling for very reasonable monthly costs. The call quality is indistinguishable from the lines provisioned by the local bell company, and inter-operates flawlessly with the Asterisk call server stack. Because calls transit our Asterisk server, we can add additional features such voicemail, speech to text, DTMF-driven menus, or call forwarding, for instance. We could even theoretically set up autopatch service using amateur frequencies or launch perl scripts to automate arbitrary fancyness. Our next step is to finish configuration of our 12-channel ZyXEL AAM1212 DSLAM modules so that we can deliver ADSL service over the same copper pairs as we deliver voice.

9 June 2012

Mike Hommey: Attempting to close a LinkedIn account

Following the trend, I attempted to close my LinkedIn account. Closing a LinkedIn account involves confirming and confirming and confirming again. Once it s all done, you d expect to, well, be done with it. I m outraged at the result: The only upside is that after I login, I can only see a page saying Your LinkedIn account has been temporarily restricted . Contact our customer service team to get this resolved as soon as possible. Update: After contacting their customer service, the account was closed and the public profile is now unavailable.

8 June 2012

Alexander Reichle-Schmehl: Release Critical Bug report for Week 23

The bug webinterface of the Ultimate Debian Database currently knows about the following release critical bugs:
In Total:1422
Affecting Wheezy:978
Wheezy only:244
Remaining to be fixed in Wheezy:734
Of these 734 bugs, the following tags are set:
Pending in Wheezy:39
Patched in Wheezy:93
Duplicates in Wheezy:63
Can be fixed in a security Update:18
Contrib or non-free in Wheezy:11
Claimed in Wheezy:2
Delayed in Wheezy:7
Otherwise fixed in Wheezy:69
Ignoring all the above (multiple tags possible) 496 bugs need to be fixed by Debian Contributors to get Debian 7.0 Wheezy released. However, with the view of the Release Managers, 874 need to be dealt with for the release to happen. Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

Lars Wirzenius: Obnam 1.0 (backup software); a story in many words

tl;dr: Version 1.0 of Obnam, my snapshotting, de-duplicating, encrypting backup program is released. See the end of this announcement for the details. Where we see the hero in his formative years; parental influence From the very beginning, my computing life has involved backups. In 1984, when I was 14, my father was an independent telecommunications consultant, which meant he needed a personal computer for writing reports. He bought a Luxor ABC-802, a Swedish computer with a Z80 microprocessor and two floppy drives. My father also taught me how to use it. When I needed to save files, he gave me not one, but two floppies, and explained that I should store my files one one, and then copy them to the other one every now and then. Later on, over the years, I've made backups from a hard disk (30 megabytes!) to a stack of floppies, to a tape drive installed into a floppy interface (400 megabytes!), to a DAT drive, and various other media. It was always a bit tedious. The start of the quest; lengthy justification for NIH In 2004, I decided to do a full backup, by burning a copy of all my files onto CD-R disks. It took me most of the day. Afterwards, I sat admiring the large stack of disks, and realized that I would not ever do that again. I'm too lazy for that. That I had done it once was an aberration in the space-time continuum. Switching to DVD-Rs instead CD-Rs would reduce to the number of disks to burn, but not enough: it would still take a stack of them. I needed something much better. I had a little experience with tape drives, and that was enough to convince me that I didn't want them. Tape drives are expensive hardware, and the tapes also cost money. If the drive goes bad, you have to get a compatible one, or all your backups are toast. The price per gigabyte was coming down fast for hard drives, and it was clear that they were about to be very competitive with tapes for price. I looked for backup programs that I could use for disk based backups. rsync, of course, was the obvious choice, but there were others. I ended up doing what many geeks do: I wrote my own wrapper around rsync. There's hundred, possibly thousands, of such wrappers around the Internet. I also got the idea that doing a startup to provide online backup space would be a really cool thing. However, I didn't really do anything about that until 2007. More on that later. The rsync wrapper script I wrote used hardlinked directory trees to provide a backup history, though not in the smart way that backuppc does it. The hardlinks were wonderful, because they were cheap, and provided de-duplication. They were also quite cumbersome, when I needed to move my backups to a new disk the first time. It turned out that a lot of tools deal very badly with directory trees with large numbers of hardlinks. I also decided I wanted encrypted backups. This led me to find duplicity, which is a nice program that does encrypted backups, but I had issues with some of its limitations. To fix those limitations, I would have had to re-design and possibly re-implement the entire program. The biggest limitation was that it treated backups as full backup, plus a sequence of incremental backups, which were deltas against the previous backup. Delta based incrementals make sense for tape drives. You run a full backup once, then incremental deltas for every day. When enough time has passed since the full backup, you do a new full backup, and then future incrementals are based on that. Repeat forever. I decided that this makes no sense for disk based backups. If I already have backed up a file, there's no point in making me backup it again, since it's already there on the same hard disk. It makes even less sense for online backups, since doing a new full backup would require me to transmit all the data all over again, even though it's already on the server. The first battle I could not find a program that did what I wanted to do, and like every good NIHolic, I started writing my own. After various aborted attempts, I started for real in 2006. Here is the first commit message: 
revno: 1
committer: Lars Wirzenius <liw@iki.fi>
branch nick: wibbr
timestamp: Wed 2006-09-06 18:35:52 +0300
message:
  Initial commit.
wibbr was the placeholder name for Obnam until we came up with something better. We was myself and Richard Braakman, who was going to be doing the backup startup with me. We eventually founded the company near the end of 2006, and started doing business in 2007. However, we did not do very much business, and ran out of money in September 2007. We ended the backup startup experiment. That's when I took a job with Canonical, and Obnam became a hobby project of mine: I still wanted a good backup tool. In September 2007, Obnam was working, but it was not very good. For example, it was quite slow and wasteful of backup space. That version of Obnam used deltas, based on the rsync algorithm, to backup only changes. It did not require the user to do full and incremental backups manually, but essentially created an endless sequence of incrementals. It was possible to remove any generation, and Obnam would manage the deltas as necessary, keeping the ones needed for the remaining generations, and removing the rest. Obnam made it look as if each generation was independent of each other. The wasteful part was the way in which metadata about files was stored: each generation stored the full list of filenames and their permissions and other inode fields. This turned out to be bigger than my daily delta. The lost years; getting lost in the forest For the next two years, I did a little work on Obnam, but I did not make progress very fast. I changed the way metadata was stored, for example, but I picked another bad way of doing it: the new way was essentially building a tree of directory and file nodes, and any unchanged subtrees were shared between generations. This reduced the space overhead per generation, but made it quite slow to look up the metadata for any one file. The final battle; finding cows in the forest In 2009 I decided to leave Canonical and after that, my Obnam hobby picked up in speed again. Below is a table of the number of commits per year, from the very first commit (bzr log -n0 awk '/timestamp:/ print $3 ' sed 's/-.*//' uniq -c awk ' print $2, $1 ' tac): 
2006 466
2007 353
2008 402
2009 467
2010 616
2011 790
2012 282
During most of 2010 and 2011 I was unemployed, and happily hacking Obnam, while moving to another country twice. I don't recommend that as a way to hack on hobby projects, but it worked for me. After Canonical, I decided to tackle the way Obnam stores data from a new angle. Richard told me about the copy-on-write (or COW) B-trees that btrfs uses, originally designed by Ohad Rodeh (see his paper for details), and I started reading about that. It turned out that they're pretty ideal for backups: each B-tree stores data about one generation. To start a new generation, you clone the previous generation's B-tree, and make any modifications you need. I implemented the B-tree library myself, in Python. I wanted something that was flexible about how and where I stored data, which the btrfs implementation did not seem to give me. (Also, I worship at the altar of NIH.) With the B-trees, doing file deltas from the previous generation no longer made any sense. I realized that it was, in any case, a better idea to store file data in chunks, and re-use chunks in different generations as needed. This makes it much easier to manage changes to files: with deltas, you need to keep a long chain of deltas and apply many deltas to reconstruct a particular version. With lists of chunks, you just get the chunks you need. The spin-off franchise; lost in a maze of dependencies, all alike In the process of developing Obnam, I have split off a number of helper programs and libraries: I have found it convenient to keep these split off, since I've been able to use them in other projects as well. However, it turns out that those installing Obnam don't like this: it would probably make sense to have a fat release with Obnam and all dependencies, but I haven't bothered to do that yet. The blurb; readers advised about blatant marketing The strong points of Obnam are, I think: Backups may be stored on local hard disks (e.g., USB drives), any locally mounted network file shares (NFS, SMB, almost anything with remotely Posix-like semantics), or on any SFTP server you have access to. What's not so strong is backing up online over SFTP, particularly with long round trip times to the server, or many small files to back up. That performance is Obnam's weakest part. I hope to fix that in the future, but I don't want to delay 1.0 for it. The big news; readers sighing in relief I am now ready to release version 1.0 of Obnam. Finally. It's been a long project, much longer than I expected, and much longer than was really sensible. However, it's ready now. It's not bug free, and it's not as fast as I would like, but it's time to declare it ready for general use. If nothing else, this will get more people to use it, and they'll find the remaining problems faster than I can do on my own. I have packaged Obnam for Debian, and it is in unstable, and will hopefully get into wheezy before the Debian freeze. I provide packages built for squeeze on my own repository, see the download page. The changes in the 1.0 release compared to the previous one: The future; not including winning lottery numbers I expect to get a flurry of bug reports in the near future as new people try Obnam. It will take a bit of effort dealing with that. Help is, of course, welcome! After that, I expect to be mainly working on Obnam performance for the foreseeable future. There may also be a FUSE filesystem interface for restoring from backups, and a continous backup version of Obnam. Plus other features, too. I make no promises about how fast new features and optimizations will happen: Obnam is a hobby project for me, and I work on it only in my free time. Also, I have a bunch of things that are on hold until I get Obnam into shape, and I may decide to do one of those things before the next big Obnam push. Where; the trail of an errant hacker I've developed Obnam in a number of physical locations, and I thought it might be interesting to list them: Espoo, Helsinki, Vantaa, Kotka, Raahe, Oulu, Tampere, Cambridge, Boston, Plymouth, London, Los Angeles, Auckland, Wellington, Christchurch, Portland, New York, Edinburgh, Manchester, San Giorgio di Piano. I've also hacked on Obnam in trains, on planes, and once on a ship, but only for a few minutes on the ship before I got seasick. Thank you; sincerely SEE ALSO

6 June 2012

Craig Small: pam bugs hurt

I did some upgrades of what seemed like a million packages today on my debian sid computer. I was doing this remotely and when I tried to ssh back in I got kicked out after entering my password, hmmm. OK, so I waited until I could get in front of it and tried to login to the console and was greeted with the message Cannot make/remove an entry and kicked out, even as root; double hmmm. So it was boot into single user time which, thankfully worked. bash loaded ok so it wasn t a shell problem. However su failed which means we are into pam problem territory. I found a bug report that sounded like my problem, but with no clear solution except upgrade in Ubuntu. The solution was pretty simple for me. It s something funky with /etc/pam.d/common-sesssion and a simple /usr/sbin/pam-auth-update fixed it for me.

5 June 2012

Anand Kumria: Internet-o! Or why the Internode Business SLA is not worth a damn.

Last week (28th May), I had five Internet services across two countries. This week, I have seven (7), with two new ones in Australia. It all begin, when Internode decided that in their own best interest to migrate me from my existing DSLAM to one of their own infrastructure. They had sent an email indicating when it was likely to occur. And they were sending invoices to the same address. Naturally, they would send the date of the migration to the same address they had used for the past 9 months to communicate to me with, right? Wrong. Internode still do not understand why they never sent the notification email, as it turns out. Compare the response time, update times, and restore times they specify to what actually occurred.
2012-05-28 12:05 BST
I'm informed of an outage. In the past, the problem has been at my end, so I run through the standard diagnostic procedure. Everything seems to be normal, so I call Internode. Sure enough, our service dropped off the Internet. According to Timothy, that happened at 11:15 AEST because of a migration to their own DSLAM. After some diagnosis from Internode, they realise that whilst the migration has successfully disconnected the service, they have not reconnected anything. There is no sync. Ticket #454448 raised - which is when I'm informed that I'm on the business SLA. I've asked for updated to both an Australian and UK mobile. "Not a problem". Estimated time of resolution: 48 hours. Yes, 48 hours. There goes the first SLA milestone.
2012-05-30 07:28 BST
I ring and speak with Carl. He indicates that as best he knows, the service is to be 7pm. for those keeping score, no updates have occurred during this time. To either number. I query him about the SLA period. He says the response time is 2hrs and restore time is 10hrs. He also indicates that this can all be checked online and that we can apply for a refund.
2012-05-30 10:27 BST
I ring, again, and this time speak with Justin. I ask why I've not have an updated, since 7pm AEST has well and truly gone. He authenticates me and places me on hold to investigate. Advises that the issue is fixed, says he sees sync on the line. I indicate that I do not. He asks me to turn off the ADSL modem, we do, and he still sees sync. on the line. Believes there is another problem at the exchange, another issue raised with wholesaler. No ticket number issues to me. ETA to fix 7pm 2nd June. Note:This was a 1 hour International call
2012-06-01 04:00 BST
I happened to check, and notice I have connectivity to the external interfaces (and thus, obviously, line sync.) again. No notification though.
2012-06-01 14:31 BST
Amidst all this, I also have to to source two external connections - Telstra & Optus, get them setup, reconfigure routing, email delivery and other network services so that the businesses which are relying on the Internet, still work. Not fun. Especially when I have no access to the routers. I had to get multiple people on site.
2012-06-04 11:XX CEST
I'm told that the second site has gone done. Inexplicably. After some round-trip and diagnosis, we determine that the routing change has occurred. However it has been done incorrectly. And, of course, no notice / or update occurred either. Multiple tickets are raised by multiple people, amongst them: #4562559 from my mobile phone whilst I am in Zurich.
2012-06-04 16:56 CEST
I am informed, and it is confirmed, that the attempted routing change has been reverted and we are back to how things were on 2012-06-01.
2012-06-05 05:44 CEST
I wake up, and find out that of at least three hours ago, they appear to have called (finally!) and actually fixed the routing to both sites. That is a week of down time, multiple people (4 people) around in various countries around the World and hundreds of dollars in calls between us all to get things working. Now, to look into compensation.
2012-06-05 09:43 CEST
I receive a missed call, which I later learn is Internode actually calling me (for once!). At that moment, I'm actually giving a lightening talk on Django-PaaS
And that, dear reader, is why Internode Business SLA is a waste of money. The services were broken by a change of no net benefit to me, was over a week ago. Also, during this process. My UK home Internet went down. My UK Three post-paid service "ran out of money" and Vodafone has a service outage for half a day. A perfect storm of Internet dis-connectivity. So, if you have been wondering, that is why I've been offline for various parts of the last week.

Ond&#345;ej &#268;ert&iacute;k: How to convert scanned images to pdf

From time to time I need to convert scanned documents to a pdf format.


Usage scenario 1: I scan part of a book (i.e. some article) on a school's scanner that sends me 10 big separate color pdf files (one pdf per page). I want to get one nice, small (black and white) pdf file with all the pages.


Usage scenario 2: I download a web form, print it, fill it in, sign it, scan it on my own scanner using Gimp and now I want to convert the image into a nice pdf file (either color or black & white) to send back over email.

Solution: I save the original files (be it pdf or png) into a folder and use git to track it. Then create a simple reduce script to convert it to the final format (view it as a pipeline). Often I need to tweak one or two parameters in the pipeline.

Here is a script for scenario 1:
<script src="https://gist.github.com/2871625.js?file=reduce2.sh"></script>
And here for scenario 2:
<script src="https://gist.github.com/2871625.js?file=reduce1.sh"></script>
There can be several unexpected surprises along the way. From my experience:

In general, I am happy with my solution. So far I was always able to get what I needed using this "pipeline" method.

4 June 2012

MJ Ray: I m going to kohacon12

Probably by the time you read this, I ll be on the road to kohacon12. It s been a fun week or so with the final arrangements on top of our other planned work and some unexpected work too, but I think we ve done enough and it will all come together. (There are still a few places left if you d like to register, but not lots.) The most nerve-wracking bit has been the sponsorship. Kohacons are free to attend and funded entirely by sponsors, which is great in so many ways. It s a bit scary for the host organisation(s), but I think the community has helped so much that it ll break even. We won t know for sure until the final reckoning in a month or so. One unneeded worry was Paypal freezing our account for a couple of weeks and sending us contradictory and absurd demands for information. Why isn t there an easier way to get money out of the USA (and Australia and NZ, actually) that isn t either slow (cheques) or expensive (wire)? Once again, I m left feeling that banks are being a big problem for business. The best bit has been the spirit of the volunteers. Our co-op couldn t have done this without them. Some of them have gone a really long way to help metaphorically, bailing us out while Paypal was chewing up our time, and sometimes physically, flying around the world I ll be raising a glass to all of them, whether they re in Edinburgh or not. So, next update should be from Edinburgh!

Benjamin Mako Hill: Why Facebook's Network Effects are Overrated

A lot of people interested in free software, and user autonomy and network services are very worried about Facebook. Folks are worried for the same reason that so many investors are interested: the networks effects brought by hundreds of millions of folks signed up to use the service. Network effects -- the concept that a good or service increases in value as more people use it -- are not a new problem for free software. Software developers target Microsoft Windows because that is where the large majority of users are. Users with no love for Microsoft and who are otherwise sympathetic to free software use Windows because programs they need will only run there. Folks worried about Facebook are afraid for similar reasons. Sure, you can close down your Facebook account and move to Diaspora. But who will you talk to there? You can already hear people complaining about Facebook the same way they've been complaining about Windows or Office for years. People feel that their hands are tied and that their software, and their social network, will be determined by what everybody is doing. I'm worried about Facebook. But I'm not too intimidated by Facebook's network effects for two reasons. First, using Facebook doesn't preclude using anything else. Twitter has enormous overlapping functionality with Facebook. Sure, people use the systems very differently. But they both ask you to create lists of friends and followers and are designed around sending and receiving short status messages. Millions of people do both and both systems are thriving. For the millions of people who use both Facebook and Twitter, the two services have had to negotiate their marginal utility in a world they share with the other one. People decide that Twitter is for certain types of short messages and Facebook is for others. But these arrangements shift over time. And the relationships between services aren't always peaceful coexistence. Remember Friendster? Remember Orkut? Remember Tribe? Remember MySpace? MySpace, and all the others, are great examples of how social networks die. They very slowly fade away. MySpace users signed up for Facebook accounts and used both. They almost never just switched. Over time, as one platform became more attractive than the other, for many complicated reasons, attention and activity shifted. People logged in on MySpace less and Facebook more and, eventually, realized they were effectively no longer MySpace users. Anyone that has been on the Internet long enough to watch a few of these shifts from one platform to another knows that they're not abrupt -- even if they can be set in motion by a particular event or action. Users of social networking sites simply don't have to choose in the way that a person choosing to boot Windows and GNU/Linux does. I'm sure the vast majority of people with Diaspora accounts use Facebook actively. This is not a problem for Diaspora. It is how Diaspora -- or whatever else eventually achieves what many of us hoped Diaspora would -- could win. Second, Facebook is for the ephemeral. Facebook is primarily used for information that was produced very recently. This week if not today. If not this hour. Facebook has an enormous amount of data that users have fed it that may be hard to get out and move somewhere else. But most people don't care very much about having any regular access to the large majority of this information. What people care deeply about is having access to the data that they and their friends created today. And that data can just as easily be created somewhere else tomorrow. Or, with the right tools, created just as easily in both places. Compare this to something like Windows where moving away would require learning, converting, and perhaps even writing, new software. Perhaps even in new programming languages that most developers don't know yet. Compared to Windows, a migration away from Facebook will be easy. Facebook's photo galleries are an example of an important place where this holds less well. Social network information -- i.e., the list of who is friends with who -- is another example of something that is persistently valuable. That said, people really enjoy the act of finding and friending. Indeed, this process was part of the initial draw of Facebook and other social networks. None of this means that Facebook is over. It doesn't even mean that its ascendancy will be slowed. What it does mean is that Facebook is vulnerable to the next thing more than many technology firms that have benefited from network effects in the past. If users are given compelling reasons to switch to something else, they can with less trouble and they will. That compelling reason might be a new social network with better features or an awesome distributed architecture that allows freedom for users and the ability of those users to benefit from new and fantastic things that Facebook's overseers would never let them have and without the things Facebook's users suffer through today. Or it might be a sexier proprietary box to store users' private information. It doesn't mean that I'm not worried about Facebook. I remain deeply worried. It's just not very hard for me to imagine the end.

3 June 2012

Sandro Tosi: How to migrate your entire Google account to a new one

I've been using a Google account since years, but having a "more serious" one along. The time has come to make the switch and elect the other as main one.

Google has a lot of services, and I'm using several of them, so changing the main account requires to migrate data of those services to the new account, given it's not possible to merge two accounts. To know the full list of services your account is signed in, go to your Account Product page. Some migrations are easy, others hell no: so I'm writing this post to keep track of the migration as it's going on.

First of all, there's a really good post about this migration at LifeHacker: it contains a lot of info I'm using here, but not all the services I need. Additionally, Google has a list of services and methods to migrate from one account to another.

Just to be generic, I'll call AccountSRC the original account and AccountDST the one I want to migrate to.

Migrate GMail

GMail is probably the most important service I have, and also the most difficult one to migrate. I have a lot of filters, so the first step is to migrate them:
  1. Login into GMail for AccountSRC
  2. Gear icon > Settings > Labs
  3. Enable the "Filter import/export" plugin
  4. Reload GMail to enable the plugin (if not done automatically)
  5. Gear icon > Settings > Filters
  6. At the bottom of the page, Select all
  7. Export: this will download an XML file with the filters in it
  8. Login into GMail for AccountDST
  9. Enable "Filter import/export" plugin
  10. Load the file containing the saved filters
  11. Import all (or select which one) filters; this will also create automatically the labels defined in the filters.
I currently have a forward rule from AccountDST to AccountSRC: so let's revert the forward direction: from AccountSRC to AccountDST. This will allow the new account to receive the mails sent to the old one.
  1. Login into GMail for AccountDST
  2. Gear icon > Forwarding and POP/IMAP > Disable forwarding: this will stop redirecting mails from AccountDST to AccountSRC
  3. Login into GMail for AccountSRC
  4. Gear icon > Forwarding and POP/IMAP > add a forwarding address
  5. Enter AccountDST as the address to forward mails to
  6. A verification code is sent to AccountDST and you'll need to enter that code in AccountSRC to verify you have access to both mailboxes
  7. Once verified, select "Forward a copy of incoming mail to" to AccountDST and to delete the GMail copy on AccountSRC
There's no automatic way to migrate all your settings from the old GMail account to the new one, so you'll have to comparing settings pages to set what you had before to AccountDST. I suggest to first enable the labs you have on AccountSRC, and then go thru every page and report the configurations.

Did I forget something? yes, the hardest part: migrate mails! There are a lot of guides about migrating mails, the way I prefer is thru IMAP and Thunderbird (as described here): you'll move IMAP folders, and since they are equivalent to GMail labels, you'll automatically get your email with the correct labels (yes, the same mail appears in several folders, one for each label, and moving an email from one folder will not remove it from all the others).
  1. Enable IMAP access on both AccountSRC and AccountDST
  2. Open Thunderbird (or Icedove if you're on Debian like me) and register both accounts
  3. You can now start moving folders/emails from AccountSRC to AccountDST
    1. Copying a (missing) folder, will automatically create a label with the same name
    2. Moving a mail from the same folders on the two accounts will add the label called with the name of the folder to that email
  4. Be ready, it's a LOOOOONG process, but it's safe and it guarantees a perfect result
With the method above you have all the labels you had before, but not the label colors (and I have a lot :( ).

Another tool that's would be interesting to evaluate for this task is Gmvault.


Migrate Contacts

Contacts are available inside GMail:
  1. Login into GMail for AccountSRC
  2. On the combobox on top-left, select Contacts
  3. More > Export
  4. Export all the contacts in Google CSV, since it's specific to be re-imported
  5. Login into GMail for AccountDST
  6. Contacts > More > Import
  7. Select the previously saved file and import it

Migrate Reader

Migrating Reader subscriptions is easy:
  1. Login into Reader for AccountSRC
  2. Reader Settings -> Import/export
  3. Export your subscriptions in OPML format
  4. Now login into Reader for AccountDST
  5. Reader Settings -> Import/export
  6. And import the saved OPML file
Sadly, that doesn't migrate your starred items, but there's a (long & manual) solution here.


Migrate Blogger

Login to Blogger for AccountSRC; then, for each blog you have there do this dance:
  1. Select the blog
  2. Go into Settings > Permissions
  3. Add AccountDST as a new author: this will send an email to AccountDST GMail account
  4. Accept the invitation (this will ask you to create a new Blogger account for AccountDST if you don't already have one)
  5. Go back to to Blogger with AccountSRC
  6. Blog > Settings > Permissions
  7. Grand admin rights to AccountDST
At this point, both AccountSRC and AccountDST have admin rights; you can leave it as it is, or remove AccountSRC and so only AccountDST will be the admin (other shares should be kept unchanged). The same procedure is described in this Google support answer.

What can't be migrated as easily are all the profile settings you've done on AccountSRC. So profile information, blogs followed and so on have to be migrated by hand: boring but they are just few things, doable in a bunch of minutes.


Migrate Android phone

Android phones require a primary Google account to work, and of course my phone uses AccountSRC. Several sources say the only way to change primary account is to factory reset the phone. But since new market app supports multiple accounts and Android 4.0 (Ice Cream Sandwich) not longer needs a primary account, I just waited for the update to ICS to come.

What you really need to do is to add a new account, AccountDST, to Settings > Account & Sync, and all the Google apps on the phone will be able to use the new account (you might need to select what to sync, and the new account from inside each app).


Migrate Calendar

Even with Calendar the settings will have to be migrated by hand: they are just in one page, so just keep two windows opened and keep switching to sync them. The calendars are easy to move over: for the main calendar:
  1. Login into Calendars for AccountSRC
  2. Go into Settings > Calendars tab
  3. Select export calendars: this will download an archive of your calendars in a zip file; you will need to extract all the ics files from that zip archive so you'll have the main calendar readily available
  4. Login into Calendars for AccountDST
  5. Settings > Calendars tab
  6. Import the main calendar
for the other calendars you can do this way:
  1. Login into Calendars for AccountSRC
  2. Settings > Calendars tab
  3. Share the calendar with AccountDST and give it "Make changes AND manage sharing" right - the new account will have full control over that calendar

Migrate Docs

The procedure to migrate documents in Google Docs (now Google Drive) is described here, and it boils down to:
  1. Login into Docs for AccountSRC
  2. Select all the documents
  3. Share them with AccountDST
  4. In the same window select AccountDST as "is owner"
and you've got all the docs as owned by the new account.


Migrate Groups

There's no way to migrate all your Groups subscriptions from one account to another (if not playing with alternate email address, but it was too risky to try it), so you'll have to migrate all of them by hand.

On the other hand, if you've created some groups, that's possible to move them to the new account:
  1. Login into Groups for AccountSRC
  2. Enter in the "manage" area for the groups you've created
  3. Manage members > Invite members
  4. Invite AccountDST
  5. Accept the invitation, clicking on the email sent to AccountDST account
  6. Go back into AccountSRC manage area
  7. Select the AccountDST account
  8. Edit the subscription and set "Membership type" to "owner": now AccountDST is the owner of the list

Migrate Analytics

The process to migrate the account on Google Analytics is similar to Groups:
  1. Login into Analytics for AccountSRC
  2. Select Admin on top-right, then for each account you have
  3. Select the account, select "Users" tab, click on "+ New User"
  4. Enter AccountDST into the "Email Address" field and give that account the "Administrator" role
  5. Save
  6. Now AccountDST is able to access the same data on Analytics as AccountSRC

Migrate Picasa

To transfer your whole Picasa account follow these steps (as also described in this support post):
  1. Login into Picasa for AccountSRC
  2. Click on the gear on top-right > Photo Settings > Privacy and permissions
  3. On the "Migrate account" line click on "Migrate my photos"
  4. Insert AccountDST into the box for destination address
  5. In case you're using picasa to store blogger photos, also select the tick below
  6. At this point, AccountSRC will receive an email: copy the link in a session where you've logged in as AccountDST and then visit the page: that will start the process
  7. Wait until the transfer is completed: you'll receive an email to AccountDST when it's done

Migrate Google+

I'm not exactly a heavy user of G+ but I have some circles I'd like to move over (also because Reader sharing now works with G+ and it's a feature I user a lot), this is a process to move circles and contacts; as stated there, there's no way to move shared items and stuff (sad!).


Migrate AdSense

It's not possible to migrate the current AdSense account: you have to register a new one. When you do, it's possible that Google notice the new account is for the same person of the first one, and will ask to close the old account on AdSense in favor of the new one.


Migrate Alert

Alerts allows to export the list of alerts you've created but it's not possible to import that list in another Google account, so you'll have to recreate them from scratch (I had just a few so it was a matter of 1 minute or so).


Migrate Zoho Notebook

If you still miss Google Notebook (why big-G you let it die? couldn't you open-sourced it? it was so awesome! anyway...), you probably know that there are two main alternatives to it: Evernote and Zoho Notebook. I decided to go with Zoho (I'm not entirely happy with it, but it's running for now), and given it allows to login with Google account, it was currently using AccountSRC, so I need to migrate that too:
  1. Login into Zoho Notebook for AccountSRC
  2. My Account > Profile > Email address
  3. "Add new email" and add AccountDST, it will send a confirmation email
  4. Then "Make Primary" on AccountDST, so now the new account can login into Zoho notebook with the Google info and have all the notes there.

Migrate Bookmarks

Google provides a guide to migrate Bookmarks from one account to another that basically is:
  1. Login into Bookmarks for AccountSRC
  2. Click on Export bookmarks and it will save an HTML file with the bookmarks in it
  3. Import the file in a browser
  4. Use the Google toolbar to import in into AccountDST
That's kinda ugly way, but it seems to work.


Re-enable Web History

If you used History, then you need to re-enable it on the new account:
  1. Login into History for AccountDST
  2. Click on "Turn Web History on" to enable Web History on the new account
but it seems to be impossible to export/import or migrate the history from one account to another, that's a shame: years and years of stats lost.


Migrate Books

Books doesn't allow to migrate the contents between accounts, so you'll have to do it by hand:
  1. Select one book from AccountSRC, copying its link
  2. Go to AccountDST and navigate to the copied link
  3. Add the book to the right category, as it was before.

Migrate Webmaster Tools

Google released a document that describes how to migrate to another account; it's similar to Analytics and Groups migration.

In addition to that, you might also want to become an owner, that will give full control on the sites.


Did I forget some services? Let me know!

Everyone has a different experience with Google products, so in case you want me to let know how you migrated from one account to another for a product I didn't mentioned, just leave a comment and I'll edit the post (to be as complete as possible).

Johannes Schauer: port bootstrap build-ordering tool report 1

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de. Diary May 21 May 22 May 23 May 24 May 25 May 29 May 30 May 31 June 1 June 2 Results I learned a good chunk of ocaml and how to use dose3 and libcudf. I created a gitorious project and a git repository for all the sourcecode. 
git clone git://gitorious.org/debian-bootstrap/bootstrap.git
The git as of now contains 30 commits and 1197 lines of ocaml code. So far, 62 emails have been exchanged between me and Pietro and Wookey. I created a mailinglist for this project where all email exchange so far is publicly accessible in the archives. You can also download all of the email exchange in mbox format. Everybody is welcome to join and/or read the list. What seems to be finished: the program that finds the minimal amount of source packages that have to be cross compiled to end up with a minimal build system. What it does is:
  1. get all essential packages
  2. get their runtime dependencies
  3. get build-essential plus runtime dependencies
  4. get all source packages that are necessary to build 1.-3. those are the packages that have to be cross compiled
  5. get a list of all packages that are built by source packages from 4.
  6. add all packages from 1.,2.,3. and 5. plus all arch:all packages to a universe
  7. use Depsolver.trim on that universe to figure out which of those packages are actually installable
The result of 7. will then contain a list of packages that are available automatically on the foreign system due to cross compiled source packages and arch:all packages. For Debian Sid, the output of my program is: 
# (1) number of packages with priority:required: 62
# (2) plus, number of dependencies of priority:required packages: 20
# (3) plus, build-essential and dependencies: 31
# number of source packages to build the above: 71
# number of additional packages built from the above source packages: 292
# (4) number of packages of those plus arch:all packages that are installable: 6421
# total number of installable packages (1)+(2)+(3)+(4): 6534
For Ubuntu Natty it is: 
# (1) number of packages with priority:required: 96
# (2) plus, number of dependencies of priority:required packages: 7
# (3) plus, build-essential and dependencies: 31
# number of source packages to build the above: 87
# number of additional packages built from the above source packages: 217
# (4) number of packages of those plus arch:all packages that are installable: 2102
# total number of installable packages (1)+(2)+(3)+(4): 2236
So for Debian, 71 source packages definitely have to be made cross compilable while for Natty, the number is 87. The last two days I was toying around with these minimal systems to see how big the number of source packages is, that can be built on top of them without running into dependency cycles. After installing the binary packages that were built, I checked again until no new packages could be built. For Natty, I was only able to find 28 additional packages that can be built on top of the 2236 existing ones. This means that a number of dependency cycles prevent building anything else. In the coming two weeks I will focus on coming up with a tool that cleverly helps the user to identify packages that would be useful to have for building more packages (probably determined by how many packages depend on it - debhelper is an obvious candidate). The tool would then show why that crucial package is not available (in case of debhelper because some of its runtime dependencies are not available and require debhelper to be built) and how the situation can best be resolved. The possible methods to do so are to identify a package that is part of a cycle and either cross compile it or let it have staged build dependencies.

Jose Luis Rivas Contreras: Little CoffeeScript to resize images

I needed to resize about 300 images and couldn t find an easier way to do this than with help of node + CoffeeScript. So here s my script, it requires you to install (via npm) imagemagick and underscore: <script src="https://gist.github.com/2862235.js?file=resize.coffee"></script> Grab it if you want, is under Do Whatever You Want license. As you can see from the shebang (that s the first line of the script, with the #!) I ran it from my command-line, this was not used in a website, although it could without too much modifications (the magic of JavaScript).

1 June 2012

C.J. Adams-Collier: Vacation Tasks

So, I m taking vacation this week and next. Here s a (partial) list of tasks that are left to do: That s probably enough, if not too much. Here s some of the bits I ve done this week: And probably some other stuff ;-)

Alexander Reichle-Schmehl: Release Critical Bug report for Week 22

The bug webinterface of the Ultimate Debian Database currently knows about the following release critical bugs:
In Total:1412
Affecting Wheezy:977
Wheezy only:272
Remaining to be fixed in Wheezy:705
Of these 705 bugs, the following tags are set:
Pending in Wheezy:51
Patched in Wheezy:118
Duplicates in Wheezy:57
Can be fixed in a security Update:24
Contrib or non-free in Wheezy:9
Claimed in Wheezy:2
Delayed in Wheezy:4
Otherwise fixed in Wheezy:99
Ignoring all the above (multiple tags possible) 427 bugs need to be fixed by Debian Contributors to get Debian 7.0 Wheezy released. However, with the view of the Release Managers, 867 need to be dealt with for the release to happen. Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

Petter Reinholdtsen: SOAP based webservice from Dell to check server support status

A few years ago I wrote how to extract support status for your Dell and HP servers. Recently I have learned from colleges here at the University of Oslo that Dell have made this even easier, by providing a SOAP based web service. Given the service tag, one can now query the Dell servers and get machine readable information about the support status. This perl code demonstrate how to do it: 
use strict;
use warnings;
use SOAP::Lite;
use Data::Dumper;
my $GUID = '11111111-1111-1111-1111-111111111111';
my $App = 'test';
my $servicetag = $ARGV[0] or die "Please supply a servicetag. $!\n";
my ($deal, $latest, @dates);
my $s = SOAP::Lite
    -> uri('http://support.dell.com/WebServices/')
    -> on_action( sub   join '', @_   )
    -> proxy('http://xserv.dell.com/services/assetservice.asmx')
    ;
my $a = $s->GetAssetInformation(
    SOAP::Data->name('guid')->value($GUID)->type(''),
    SOAP::Data->name('applicationName')->value($App)->type(''),
    SOAP::Data->name('serviceTags')->value($servicetag)->type(''),
);
print Dumper($a -> result) ;
The output can look like this: 
$VAR1 =  
          'Asset' =>  
                     'Entitlements' =>  
                                       'EntitlementData' => [
                                                             
                                                              'EntitlementType' => 'Expired',
                                                              'EndDate' => '2009-07-29T00:00:00',
                                                              'Provider' => '',
                                                              'StartDate' => '2006-07-29T00:00:00',
                                                              'DaysLeft' => '0'
                                                             ,
                                                             
                                                              'EntitlementType' => 'Expired',
                                                              'EndDate' => '2009-07-29T00:00:00',
                                                              'Provider' => '',
                                                              'StartDate' => '2006-07-29T00:00:00',
                                                              'DaysLeft' => '0'
                                                             ,
                                                             
                                                              'EntitlementType' => 'Expired',
                                                              'EndDate' => '2007-07-29T00:00:00',
                                                              'Provider' => '',
                                                              'StartDate' => '2006-07-29T00:00:00',
                                                              'DaysLeft' => '0'
                                                             
                                                          ]
                                      ,
                     'AssetHeaderData' =>  
                                          'SystemModel' => 'GX620',
                                          'ServiceTag' => '8DSGD2J',
                                          'SystemShipDate' => '2006-07-29T19:00:00-05:00',
                                          'Buid' => '2323',
                                          'Region' => 'Europe',
                                          'SystemID' => 'PLX_GX620',
                                          'SystemType' => 'OptiPlex'
                                         
                    
         ;
I have not been able to find any documentation from Dell about this service outside the inline documentation, and according to one comment it can have stability issues, but it is a lot better than scraping HTML pages. :) Wonder if HP and other server vendors have a similar service. If you know of one, drop me an email. :)

Rapha&#235;l Hertzog: My Debian Activities in May 2012

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (338.26 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg Like last month, I did almost nothing concerning dpkg. This will probably change in June now that the book is out The only thing worth noting is that I have helped Carey Underwood who was trying to diagnose why btrfs was performing so badly when unpacking Debian packages (compared to ext4). Apparently this already resulted in some btrfs improvements. But not as much as what could be hoped. The sync_file_range() calls that dpkg are doing only force the writeback of the underlying data and not of the meta-data. So the numerous fsync() that follow still create many journal transactions that would be better handled as one big transaction. As a proof of this, replacing the fsync() with a sync() brings the performance on par with ext4. (Beware this is my own recollection of the discussion, while it should be close to the truth, it s probably not 100% accurate when speaking of the brtfs behaviour) Packaging I uploaded new versions of smarty-gettext and smarty-validate because they were uninstallable after the removal of smarty. The whole history of smarty in Debian/Ubuntu has been a big FAIL since the start. Once upon a time, there was a smarty package and some plugins. Everything was great except that the files were installed in a way that differs from the upstream recommendations. So Ubuntu changed the path in their version of the package and did not check whether it broke anything else (and it did break all the plugins). Despite the brokenness of the plugins, this divergence survived for years. So several packages that were using Smarty were modified to use dpkg-vendor to use the correct path depending on whether it was built on Debian or Ubuntu. In 2010, Smarty 3.0 has been released and instead of upgrading the smarty package to this version, one of the smarty co-maintainers introduced a smarty3 package that used yet another path (despite the fact that smarty 3 had a mode to be compatible with smarty 2).
At some point, I informed him that he had to handle the migration of users of smarty to smarty3 he acknowledged and then lost interest in smarty ( I m no longer using it ) and did nothing. After some more bitrot, smarty has been forcefully orphaned in August 2011 by a member of the security team. And in March this year, it has been removed from unstable despite the fact that it still had reverse dependencies (usually removals only happen when they impact no other packages, I don t know why this wasn t the case here). At least the brokenness attracted some attention to the situation and Mike Gabriel contacted me about it. I offered him to take over the various packages since they all needed a real maintainer and he accepted. I sponsored his uploads of all smarty related packages (bringing in the latest upstream versions at the same time). In the end, the situation is looking better now, except that there s no migration path from users who rely on smarty in Squeeze. They will discover that they need smarty3 in Wheezy and that the various paths have to be adjusted. It s probably acceptable since the new upstream versions are no longer backwards compatible with smarty 2 The Debian Administrator s Handbook At the start of the month, I was busy preparing the release of the book. I introduced the publican-debian package to unstable, it s a Publican brand (aka a set of CSS and XSL stylesheets to tailor the output of Publican) using the Debian colors and using the Debian logo. This brand is used by the book. I also created the debian-handbook package and setup the public Git repository on alioth.debian.org. I was ready or so I thought. A few hours after the announce, the website became unusable because the numerous visitors were exhausting the maximum number of client connections. And I could not increase the limit due to Apache s memory usage (with PHP and WordPress). We quickly off-loaded most of the static files traffic to another machine and we setup bittorrent. The problem was solved for the short term. Thousands of persons downloaded the ebook and to this date, 135 copies of the paperback have been sold. Then I took a one-week vacation. Even though I had no Internet at the place I was, I wandered in the street to find a Freewifi wifi network (customers of the Free ISP can use those freely) to stay on top of incoming email. We quickly received some bug reports and I dealt with the easy ones (typos and the like) on the fly. When I came back at home, I manually placed 54 lulu orders for the people who opted for the paperback as reward during the fundraising campaign. A bit tedious but it had to be done (if only Lulu supported a way to batch many orders at once ). I also wanted a long term solution to avoid the use of an external host to serve static files (should a new traffic spike arrive ). So I installed nginx as a front-end. It serves static files directly, as well as WordPress pages which have been cached by wp-super-cache. Apache is still here listening on a local port and responding to the remaining queries forwarded by nginx. Once I ll migrate to wheezy, I might completely ditch apache in favor of php5-fpm to handle the PHP pages. Last but not least, I wanted to bootstrap the various translations that people offered to contribute. I wrote some documentation for interested translators and blogged about it. It s shaping up nicely check it out if you re interested to help! Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Francois Marier: Proper indentation of Javascript files using js2-mode in emacs

If you use emacs for Javascript or Node.js development, you should have a look at js2-mode (apt-get install js2-mode in Debian / Ubuntu). In addition to providing syntax highlight, it will parse your Javascript and issue errors and warnings. (It's not as pedantic as jslint or jshint so you'll probably want to fix all of them before committing your files.) Unfortunately the default indentation style looks a bit crazy and adds excessive indentation to your code. Here's an example: 

    navigator.id.secret.generate(identity, function (plainKey, wrappedKey)  
                                     var data =  wrappedKey: wrappedKey ;
                                     $.post('/loggedin', data, function (res)  
                                                $("#message").text('All done');
                                             , 'json');
                                  , function (error)  
                                     $("#message").text('ERROR: ' + error);
                                  );
It turns out that indenting Javascript properly is really hard, but you can turn on a special flag which will cause the indentation to "bounce" between different strategies, starting with the most likely one. This is what I now have in my ~/.emacs: 
(custom-set-variables
 '(js2-basic-offset 2)
 '(js2-bounce-indent-p t)
)
You can find and configure all other options by issuing the following command:
M-x customize-group [RET] js2-mode [RET]
So yes, you can have both the reasonable indentation of the standard js-mode and the helpful warnings and errors of js2-mode!

31 May 2012

Julian Andres Klode: Memory allocators

In the past days, I looked at various memory allocators in a quest to improve performance in my multi-threaded test cases of a reference counting language runtime / object allocator (a fun project). It turns out the glibc s memory allocator is relatively slow, especially if you do loops that create one element and destroy one element at the same time (for example, map() on a list you no longer use after you passed it to map). To fix this problem, I considered various options. The first option was to add a thread-local cache around malloc(). So whenever we want to free() an object, we place it on a thread-local list instead, and if a malloc() request for an object of that size comes in, we reuse the old object. This fixes the problem with the lists, but experiments shown another problem with the glibc allocator: Allocating many objects without releasing others (let s say appending an element to a functional list). I started testing with tcmalloc instead, and noticed that it was several times faster (reducing run time from 6.75 seconds to 1.33 seconds (5 times faster)). As I do not want to depend on a C++ code base, I decided to write a simple allocator that allocates blocks of memory via mmap(), splits those into objects, and puts the objects on a local free list. That allocator performed faster than tcmalloc(), but was also just a simple test case, and not really useable, due to potential fragmentation problems (and because the code was statically linked in, causing thread-local storage to be considerably faster, as it does not need to call a function on every TLS access, but can rather go through a segment register). I then discovered jemalloc, and tried testing with this. It turned out that jemalloc was even faster than tcmalloc in all test cases, and also required about the same amount of memory as tcmalloc (and slightly more than the custom test code, as that combined reference counting with memory allocator metadata and thus had less meta data overhead). Using jemalloc, the list building test performs in 0.9X seconds now (instead of tcmalloc s 1.33 or glibc s 6.75 seconds), requires 0 major page faults (tcmalloc has 2), and uses 5361472 kb memory (tcmalloc uses 5290240, glibc requires 7866608). Given that jemalloc is written in C, I can only recommend it to everyone in need of a scalable memory allocator. Depending on your workload, it might require less memory than tcmalloc (at least that s the case at Facebook) and is most likely faster than tcmalloc. It also provides tcmalloc-compatible profiling facilities (as Facebook needed them). Furthermore, jemalloc is also the memory allocator used by FreeBSD, and is used by Mozilla projects, and on several Facebook projects, and should thus be relatively stable and useable. All tests were run with 4 threads, on a dual-core laptop with hyper-threading, using (e)glibc 2.13, tcmalloc 2.0, and jemalloc 3.0. The tests may not be representative of real world performance, and do not account for fragmentation. Should I try replacing Python s custom caching of malloc() calls with simply calling jemalloc, and run a few Python benchmarks? Anyone interested in that?
Filed under: General

Next.

Previous.