I have been pondering the idea behind this article for a while, and finally had a bit of time to implement it.
The basic idea is this: certain platforms have higher costs in terms of memory per concurrent connection. Those translate into increased costs in dollar terms.
Nota Bene: Having run
LangPop.com for some time, I'm used to people getting hot and bothered about this or that aspect of statistics that are rough in nature, so I'm going to try and address those issues from the start, with more detail below.
-
Constructive criticism is welcome. I expect to utilize it to revisit these results and improve them. Frothing at the mouth is not welcome.
-
There is something of a "comparing apples and oranges" problem inherent in doing these sorts of comparisons. As an example, Rails gives you a huge amount of functionality "out of the box", whereas Mochiweb does much less. More on that below.
-
I am not familiar with all of these systems: meaning that I may not have configured them as I should have. Helpful suggestions are, of course, welcome. Links to source code are provided below.
-
You can likely handle many more 'users' than concurrent connections, which means multiple browsers connecting to the site at the same time.
-
Programmer costs are probably higher than anything else, so more productive platforms can save a great deal of money, which more than makes up for the cost of extra memory. There's a reason that most people, outside of Google and Yahoo and sites like that, don't use much C for their web applications. Indeed, I use Rails myself, even though it uses a lot of memory and isn't terribly fast: I'd rather get sites out there, see how they do, and then worry about optimizing them (which is of course quite possible in Rails).
Methodology
All tests were run like so: my new laptop with two cores and four gigs of memory was used as a server, and my older laptop was used to run the ab (Apache Benchmark) program - they're connected via ethernet. I built up to successive levels of concurrency, running first 1 concurrent connection, 2, 10, and so on and so forth. The "server" computer is running Ubuntu 9.10, "karmic".
Platforms
The platforms I tested:
-
Apache 2.2, running the worker MPM, serving static files.
-
Nginx 0.7.62, serving static files.
-
Mochiweb from svn (revision 125), serving static files.
-
Jetty 6.1.20-2, serving static files.
-
Rails 2.3.5, serving up a simple template with the current date and time.
-
PHP 5.2.10.dfsg.1-2ubuntu6.3, serving up a single php file that prints the current date and time.
-
Django 1.1.1-1ubuntu1, serving up a template with the date and time.
-
Mochiweb, serving a simple template (erltl) with the date and time.
-
Jetty, serving a simple .war file containing a JSP file, with, as clever observers will have surmised, the date and time.
As stated above, it's pretty obvious that using Rails or Django for something so simple is overkill:
Better Tests for the Future
I would like to run similar tests with a more realistic application, but I simply don't have the time or expertise to sit down and write a blog, say, for all of the above platforms. If I can find a few volunteers, I'd be happy to discuss some rough ideas about what those tests ought to look like. Some ideas:
-
They should test the application framework with a realistic, real world type of example.
-
The data store should figure as little as possible - I want to concentrate on testing the application platform for the time being, rather than Postgres vs Sqlite vs Redis. Sqlite would probably be a good choice to utilize for the data store.
-
Since this first test is so minimalistic, I think a second one ought to be fairly inclusive, making use of a fair amount of what the larger systems like Rails, Django and PHP offer.
-
I'd also be interested in seeing other languages/platforms.
-
The Holy Grail would be to script all these tests so that they're very easy to run repeatably.
Results
With that out of the way, I do think the results are meaningful, and reflect something of what I've seen on various platforms in the real world.
First of all, here we look at the total "VSZ" (as ps puts it) or Virtual Size of the process(es) in memory. Much of this might be shared, between libraries, and "copy on write" where applicable.
The results are impressive: Rails, followed by Django and PHP eats up a lot of memory for each new concurrent connection. Rails, which I know fairly well, most likely suffers from several problems: 1) it includes a lot of code. That's actually a good thing if you're building a reasonably sized app that makes use of all it has to offer. 2) Its garbage collector doesn't play well with "copy on write". Which is what "
Enterprise Ruby" aims to fix. Django and PHP are also fairly large, capable platforms when compared to something small and light like mochiweb.
That said, excuses aside, Erlang and Mochiweb are very impressive in how little additional memory they utilize when additional concurrent connections are thrown at them. I was also impressed with Jetty. I don't have a lot of experience with Java on the web (I work more with J2ME for mobile phones), so I expected something a bit more "bloated", which is the reputation Java has. As we'll see below, Jetty does take up a lot of initial memory, but subsequent concurrent connections appear to not take up much. Of course, this is also likely another 'apples and oranges' comparison and it would be good to utilize a complete Java framework, rather than just a tiny web app with one JSP file.
So what's this mean in real world terms of dollars and cents? As your Rails application gets more popular, you're going to have to invest relatively more money to make it scale, in terms of memory.
For this comparison, I utilized the bytes/dollar that I'm getting for my
Linode, which works out to 18,889,040.85 ($79.95 for 1440 MB a month).
As we can see, to have a similar amount of concurrent users is essentially free for Mochiweb, whereas with Rails, it has a significant cost. This information is particularly relevant when deciding how to monetize a site: with Erlang and Jetty it would appear that scaling up to lots of users is relatively cheap, so even a small amount of revenue per user per month is going to be a profit, whereas with Rails, scaling up to huge numbers of users is going to be more expensive, so revenue streams such as advertising may not be as viable. It's worth noting that 37 signals, the company that created Rails, is a vocal supporter of charging money for products.
There's another interesting statistic that I wanted to include as well. The previous graph shows the average cost per additional concurrent user, but this one shows how much the platform costs (using when there is just one user, so it acts as a sort of baseline:
As we can see, Jetty is particularly expensive from this point of view. The default settings (on Ubuntu) seem to indicate that, for instance, the basic $20 a month Linode package would not be sufficient to run Jetty, plus a database, plus other software. I think that the Apache Worker number is off a bit, and may reflect settings made to handle a large number of connections, or perhaps a different MPM would make sense.
Source Code / Spreadsheet
The spreadsheet I put together is here:
http://spreadsheets.google.com/ccc?key=0An76R90VwaRodElEYjVYQXpFRmtreGV3MEtsaWYzbXc&hl=en
And the source code (admittedly not terribly well organized) is here:
http://github.com/davidw/marginalmemory/
I've talked before about the minimal way in which I've been using a lot of the available automation tools. I tend to use them to carry out only a few operations:
News is slow this month on the FusionForge development front. We're
all busy gathering all the things that we want to go into the next
release, but there's no big news from the code. However, there is
something of interest.
You may have heard about the Coclico project, which is an initiative
aiming at collaboration and convergence between several forge engines,
most notably FusionForge, Codendi and Novaforge. That project was
started last October, and it holds regular meetings with its members.
The next meeting is scheduled for the 2nd of February in Paris, and we
thought we could host an open meeting on the 3rd for non-Coclico
members, a bit like the forge meeting we had last year (which is when
FusionForge was officially born), but with an emphasis on what Coclico
did so far. Since most of the FusionForge hackers are in Western
Europe, and several are in Paris (especially if we add those who go to
Paris for the Coclico meeting), we thought it would also be a good
opportunity to gather for a technical and social meeting.
It seems the Coclico open session didn't generate much interest this
time (at least, it hasn't so far), so I proposed to hijack the room
for this FusionForge meeting, and I didn't hear any objections. I
have several themes I'd like to discuss with people, and possibly
start implementing during that day:
It has been quite some time since my last post. I have been involved in quite a few things and the last part of 2009 which left no time for anything else. At the begining of 2010, just to set things on a good tone from the new year, I went on a snowboarding vacation to Austria to Kitzb hel.
I just published three new security issues on phpMyAdmin. They are not exactly
something what you would call new as you can see from CVE ids and even from
phpMyAdmin version which does fix them (none of them existed in 3.x branch and
were fixed in 2.9.10 for 2.9 branch).
However they got assigned CVE ids recently what means that we should tell
about them to our users even when we don't think these are that important. So
don't be frightened of them :-).
.
However, I managed to get a reasonable first release out the door this evening. Grab it from 
Sarah hit the 20 week mark on Monday, and so we had the big anatomical
ultrasound. We also found out that it looks like we're having a girl. We're
now trying to come up with a name.
There was also a worrying piece of news: they couldn't see the cavum septum
pellucidum, while they were checking out her brain.
Apparently the cavum septum pellucidum is a particular marker they look for
when they're checking everything out.
Now what the absence of the cavum septum pellucidum actually means isn't
terribly clear. Apparently normally this is found along with other
abnormalities, but the rest of her brain structures look fine. The nuchal
translucency we had earlier in the pregnancy came back fine also.
So we got bustled off to have a chat with the genetic counsellor (during
which the 4.1 earthquake happened). We got booked in for an amniocentesis
for later that afternoon, as well as a fetal MRI at 22 weeks, and trundled
off home.
To cap things off, when we were trying to park back at the hospital for the
2pm appointment, I managed to scrape the car up against one of the poles in
the car park. (The lower level parking is notoriously tight). Note to self:
always use the free valet parking service from now on.
Sarah had some second thoughts about the amniocentesis, as it does carry
with it some risk of miscarriage, and after chatting with with a couple of
the obstetricians, we elected not to do it. The fetal MRI should
definitively determine if the cavum septum pellucidum is absent, and then we
can talk to a paediatric neurologist about what the ramifications of that
might be.
One of the reasons they wanted to do an amniocentesis now, rather than after
the fetal MRI, is the amniocentesis itself takes a couple of weeks for the
results to come back, and that would bring us up to the 24 week mark.
Apparently if we wanted to terminate the pregnancy, once it was at 24 weeks,
we'd have to go to LA to do it.
Whilst I consider myself an atheist these days, I was raised a Catholic, and
although I like to think of myself as pro-choice, I still have a lot of
problems with abortion, so I don't think I'd be comfortable terminating
unless it could be shown with reasonable certainty that our baby was going
to be in a really bad way. Based on how the rest of the brain looked, and
the nuchal translucency results, and the fact that the amniocentesis would
only identify chromosomal and not genetic neurological disorders, I don't
think we'll be aborting.
From my limited research, it sounds like the cavum septum pellucidum
disappears at about 3 months after birth anyway, so looking at it one way,
you could say our baby's brain is developing faster. Other research has
indicated a correlation with optic nerve development issues, so I don't know
if that means she might be blind. We really need to chat with a paediatric
neurologist. But hopefully the fetal MRI will find it, and this will turn
out to be nothing.
An anxious two weeks will now follow.
The Tool Factory closed its doors on 31st December 2009.
As you may remember, 
Somehow missed during the the end-of-year switchover was the fact that my
review of Guenther Sawitzki's Computational Statistics: An Introduction to R
(CRC / Chapman \& Hall, 2009) is now up
5 years ago I founded my own company