It's not too obvious how to test email receive handlers in Appengine. The important observation is that the handlers take HTTP POSTs with multipart/form-data encoded data. In python you can build an email to be handled with the following code:

from email.message import Message 

def test_email(self):
  body = Message()
  body.add_header('to', 'test-unknown@other-app.com')
  body.add_header('from', 'test@app.com')
  body.add_header('Content-Type', 'multipart/alternative', boundary=self.boundary)
  text = Message()
  text['content-type'] = 'text/plain'
  text.set_payload('I am I! Don Quixote!  The man of La Mancha!')
  body.attach(text)

  post(payload=body.as_string())

I like App Engine, my wedding website runs on App Engine as well as a couple recent commercial projects. However, there are some annoyances that make a lot of the useful things I learned at Google not applicable.

1. Chunked responses
   App Engine doesn't have a way of doing chunked responses.
   1.5. Processing after sending the responseMy annoyance here is that I can't send an early response and wait for some resource. Say I want to do 5 url fetches that I cache for a short period of time. If I want to reply in 250ms, I want to use any of the results I've gotten and reply to the user. Then once the rest of the fetches come in, I want to put them in the cache, so they will come back quickly the next time.
2. Counters
   App Engine has crappy support for counters. You can shard your counters into the datastore: https://developers.google.com/appengine/articles/sharding_counters#counter_python or you can use taskqueues: http://blog.appenginefan.com/2009/10/non-sharded-counters-part-2-using-task.html . Now, how do you graph these counters over time, well, that is left as an exercise for the reader. Google has special datastores for stores and retrieving counters.
3. Backends
   Backends in appengine are versioned separately from frontends. This sounds like a good idea, but to run the devserver with backends, the backends need to be in the same source tree. Thus backends share the same url space. Backends also share the same app.yaml, so again the url space is shared. Thus, if you make a change that you want to coordinate between the frontend and the backend, you need to change the application version in app.yaml, then rename all the backends in backends.yaml. If backends are supposed to be in the same source tree as frontends, then they should be prefixed with the application version. If backends should be versioned separately from frontends, then they shouldn't share url handlers from app.yaml.
   3.5 Uploading backends is slow
   Each backend is uploaded through appcfg.py separately. So if you are crazy and have one backend per backend level, then you need to upload 4 copies of the source code each time the backends get updated.
4. Logging
   I want structured logs. Instead what you have to do is use the python logging module to write out human readable logs, then parse them out into the structure you want.
   http://code.google.com/p/google-app-engine-samples/source/browse/trunk/logparser/logparser.py
   

As a preemptive counter, 1.5 could be dealt with using backends and a background thread, except backends don't scale to the number of queries, so my use case above isn't fixed.

Recently, I got to try out working in a couple different environments. Working in these environments helped me formalize some of the things that I like and some of the things I don't like. With these new environments I've come to realize my style of development. It's called penance driven development. It's called this because there are two important things:
1) It's better to ask forgiveness than for permissions
2) You need to be deserving of forgiveness.

So the concept is simple, whenever you break something, you first fix the symptom, then you improve the infrastructure so similar breaks should be harder to do. The cost of the penance doesn't have to be huge, but the penance has to be done, otherwise technical debt accumulates and people believe there is no downside to breaking things.

What needs to be improved depends on what is missing. For example, if an engineer broke a feature, but that feature didn't have a test, the penance should be to write a test. Even if the one that broke the feature isn't the owner or the one that wrote the feature. If someone breaks the build and no one noticed because the continuous build didn't email, the fix could be setting up the continuous build to email out build failures, or even better, run each change on a build bot before allowing commits.

I think I'll have to start a penance of the day blog/twitter stream.

I've been practicing this style of development for most of my career, and it's similar if less precise than the 5 whys . As mentioned in the The Lean Startup, the cost of the penance should be proportional to the cost of the break.

Dear Lazyweb,

I'd like a video of the 90s commercial "That's what daffodils do". If you do that, I'll release a basic library for the iPhone to use SPDY[1].

[1] I said basic!

I've been working with appengine for a few months now. I've managed to find out the hard way that appcfg.py rollback is useless. I've learned to create a git branch for each refactoring I start on. The git branch also includes a new app version for the branch. When push all my changes to my live site, I upload one last time to the new app. Merge my branch into the master then change the live version from the old one to the new one.

I haven't learned how this work flow translates to having multiple developers. Hopefully, it is simple enough that other people can follow.

I also use make since it's an easy way to automate tasks. I hear redo is good, but for python, I'm not really compiling anything, I'm simply writing shell scripts and make is a great shell script dispatcher.

To get appengine to send queued tasks to a backend, you need to set the host header when queuing the task. E.g.

    deferred.defer(
        batch.DoStuff, arg1, arg2, arg3,
        _headers= 'Host': backends.get_hostname(backend='backend_name') )


The following is a makefile fragment I seem to start each of my appengine projects with.

GAEPATH = $(HOME)/bin/google_appengine
PORT=8081

PYLINTS = $(wildcard *.py */*.py */*/*.py)
PYLINTFILES = $(patsubst %.py,.%.lint,$(notdir $(PYLINTS)))
PYLINT = $(join $(dir $(PYLINTS)),$(PYLINTFILES))

PYTHONPATH=$(GAEPATH):$(GAEPATH)/lib/yaml/lib:$(GAEPATH)/lib/webob:$(GAEPATH)/lib/django_0_96:.

APP=new-app

run:
$(GAEPATH)/dev_appserver.py ./ --port=$(PORT) --datastore_path=/tmp/$(APP).dev_appserver.datastore


.%.lint: %.py
@PYTHONPATH=$(PYTHONPATH) pychecker --only --no-miximport $?
@touch $@


lint: $(PYLINT)


For the few of you out there that are parsing PDFs manually in python, JPEG images (including CMYK images) can be extracted with the following code fragment.

  # reader is a PDFReader object from pyPdf, value is the operand to a Do operator. 
from PIL import Image, ImageChops

  xobject = reader.getObject(value)
  if xobject['/Filter'] == '/DCTDecode':
    raw_data = xobject.getRawData()
    if xobject['/ColorSpace'] == '/DeviceRGB':
      _CreateFile('image/jpeg', filename, raw_data))
    else:
      f = cStringIO.StringIO(raw_data)
      of = cStringIO.StringIO()
      i = Image.open(f)
      if xobject['/ColorSpace'] == '/DeviceCMYK':
        i = ImageChops.invert(i)
      i.convert('RGB').save(of, 'JPEG')
      _CreateFile('image/jpeg', filename, of.getvalue())

The CMYK images I found in the PDFs needed to inverted. PIL versions before 1.1.7 would do that for you, but version 1.1.7 removed in the ImageChops.invert() call.

I'm reading through http://scientopia.org/blogs/goodmath/2011/07/06/things-everyone-should-do-code-review/. So, I'll make my comments here since I expect they will be long. As a caveat, I still work at Google and have done a lot of code reviews. I've made a lot of mistakes while code reviewing, I've had difficult code reviews, but my biggest problems were non-code reviews. The caveat is for most of the code reviews, I know the code being changed better than the person changing the code.

I'll reply to sections as I come across them: "Given a problem, there are usually a dozen different ways to solve it. Andgiven a solution, there's a million ways to render it as code. As a reviewer, your job isn't to make sure that the code is what you would have written - because it won't be.". I disagree with this. There are many times where code is first written in a (1) Ignorant way (2) Sloppy way (to try to save trivial amounts of work). I agree that the code probably won't be the way the reviewer wrote it, but it is good to comment on how it could have been written. Many times the reviewee isn't familiar (or ignores) with the idiomatic style of a larger bit of code.

"The second major pitfall of review is that people feel obligated to say something.". It's fine to say something, but if that's what you say most of the time then you are wasting the reviewee's time. If you do a first pass and don't find anything, you should go back and read the code and be sure that you understand it. Most times I've seen people say nothing is that they don't want to know what the new (or changed) code does.

The last section is about speed. For speed there are two types of code reviews, one which is quick, small and fixes something important. Those should only take one round and typically only need the reviewer to ask for tests. Then there is everything else, which includes small and unimportant. For those, it's ok to take more time to start the review, but each iteration should get smaller and faster.

For anyone that's gotten this far, I guess I can say my comments about Mark's blog post are purely bike shedding, but that's something with code reviews, it's ok to say you think the bike shed should be neon pink as long as you don't force the reviewee to paint the bike shed neon pink.

I went to a borders today that was going out of business and got a couple new toys. I got a Kobo ebook reader and a Demy by Key Ingredients. First impression of the kobo is that it is fast enough to read off of, but not as fast as the 3rd generation Kindle. The Demy is a slick device with a couple flaws. One is that it requires desktop software to sync and two the "saved list" feature/button isn't a list, it's simply rotates through the items you have saved, so it is really painful to use the saved list to actually save recipes to cook later. I got the Demy cheap enough that I'm considering getting another one that I can alter in fun ways :)

A couple quick impressions from the trip Sonia and I took to Taiwan for her cousin Arthur's wedding. One is that sidewalks are for selling not for walking. The streets are good enough to walk on. Second is that there are a lot of scooters in Taiwan. It should be easy to meet green house gas goals**. 1) Get electric scooters down in price (it looks like they are close), 2) Finish off the nuclear power plan so that the new electric scooters are powered by something other than coal.


** Obviously there are many complications and emission sources I'm not thinking of.

I'll start with an apology that I didn't get any pictures this weekend. However, many, many other people did. This weekend started with a simple holiday supper for Sonia's work (ok, there was some Karaoke). Then it moved on to the crazy. Saturday was Santacon. I pre-partied with Sonia's crew (she even stopped in on her lunch break). Then had a delivery of 1/2 Cord of firewood that will be well used over the coming year. Then it was back to SantaCon still dressed as an Elf to meet up with Lina. Then back to Sonia's crew to bring Sonia her reindeer antlers that I'd forgotten to drop off at her place. (ok, there was some stops for karaoke, Lina was at the Mint, and Sonia was at Encore).

Sunday was as crazy as Saturday, but not as long. The Santa Skivvies run was a simple get dressed down to the bare essentials of red underwear and a santa hat, then run into the Castro from The Lookout, and back. Since any Santa event wouldn't be complete with only Santas, Sonia dressed and Santa and I was the red-nosed reindeer (complete with reins). (ok, there was no karaoke on sunday (the Mint was closed on the way home from The Lookout).

So I never had the chance to see Weeping Tile in Concert, but this year I have managed to see both Luther Wright in concert and Sarah Harmer. That and I found "The show 1995" channel on youtube which does have weeping tile in concert. And Uncle Remus.

I discovered another awesome monthly event. This time it was Sci-fi Burlesque. I got a couple pictures, but unfortunately, I didn't get a good picture of the storm trooper. Nothing says awesome like tribbles, storm troopers and burlesque!

The previous interesting event was Bawdy Storytelling.

I have a lot of papers lying around. I now try to keep them organized, but I haven't got through the back log of when I didn't organize them. Sometimes I come across stuff that I'm happy I had in the pile. The following was my attempt at a valedictorian speech for the 2005 Math class at the University of Waterloo. I was not the valedictorian, but I tried, and thus have a speech.

Ladies, Gentlemen, Doctors and children of all ages. Thank you for spending our last day at Waterloo, here with us. I am one of those people who won't be staying here after day, but somehow I was still chosen to be the valedictorian. So, when I was asked to be the valedictorian by some friends of mine, I was thrilled. I also though I could do a good job, heck, I got my degree in CS, therefore, I can do anything. So, I accepted the nomination. I looked at the nomination form and didn't find any hints on what to talk about. At this point I realized that this speech may be a little underspecified. In fact, there was no specification at all.

I gathered what I could from the form, i.e. a speech had to be written by, and performed on the 18th of June 2005. Skip forward a few weeks, I'd accomplished nothing more than vague thoughts on what to talk about, then I got an email. All potential speeches had to be presented on the 6th of April 2005 (the deadline had moved up, and still no specification). However, a performance objective was given in that email -- each speech must be completed in 5 minutes. Given that I don't have a spec, I can only hope no one else has one.

What follows is my implementation of this unknown specification.

Another cycle in our lives is over. We are finished our latest round of education. This time from the University of Waterloo, in some Math program. Many of us have changed significantly while we've been here, but we are still the same people. We know we are older, we can guess that we are wiser, and we are certainly more educated. So, here is a quick tour of my cycle through Waterloo.

When I got here back in September 2000, I was amazed at how many smart people could be put together in one small place. I am actually still amazed by all the smart people here. I simply happen to know a bit more about some of these people and the amazing things they have done. With all these smart people here, I even had to do homework to keep up.

Skip forward to second year. Second year is when the university decided there is no more room for me in residence. In fact, they do that to almost all second year students. So, I found a sleezy place to live off-campus and started discovering what life is like outside the university bubble. For example, there are stores not in the plaza. There are a couple malls, a bus station, and a train station.

By third year Waterloo had become home despite working in two other cities and growing up on the other side of Toronto. I no longer had to fear being kicked out of CS since I had enough courses with good marks racked up that my cumulative average wouldn't drop too low unless I started failing, which would have been a bad thing to do.

In fourth year, I could see the end in sight. Grad school applications needed to be done pronto and some interesting job prospects lured me away from yet more years of school. Co-op didn't help keep me in school, I enjoyed the working world. I know there are others that used co-op to find that the working world wasn't for them. I expect those people to be in some school for a good long time.

Now we are at the end, and the cycle is complete. I've been across the country, lived over in sunny California, and biked through Scotland, but no where has compared to Waterloo, or the great people that make it such a wonderful place to be. I'm sad to go, as I'm sure many of you are, but I hope you are excited about the next step at the same time.

That was my 5 years, in a non-chronologically accurate order. But it's been even better to help and see people do amazing things. Small things, like getting a group of friends together to go get pints of Bubbletea. To large things like running frosh week for thousands of first year students.

Now that we have looked at the past, what about the future? One of the things I didn't wan to say during this speech is that "We are the future". What does that even mean? Are we useless now? I don't know what it means, it could even be partially true, but we are not useless. In fact our time is now! Among this graduating class are those who are doing cutting edge research. There are those we are already running their own companies. There are those who are already half-way through there plan to rule the world. There are even those of us who are simply going to help others do those things. So, we are not "the future", we are "the now".

Being "the now", (don't I sound like Dubya?) is one thing, but we should always be "the now". We can always make now be our time. We should do this because if we aren't the ones living happily and making the world a better place, in our own ways, then who is going to do it?

If nothing else, I really hope everyone lives happily. For me, I found if I can keep my hat then I can be happy. Well, I can be happy promoting "No Pants Day"!

DOWN WITH PANTS!!

At the end, I finally figured out the specification for this speech. It's simple: Remind people of the awesome things they have done here, the amazing people they shouldn't forget. Lastly, to convince people to keep doing the awesome things despite what might happen or what obstacles appear.

Thank you, I enjoyed being your VD today.

I was riding home tonight along Howard street. I passed a cop on a bicycle, then started speeding up so I would make the next green. I made the green I was shooting for, but unfortunately was caught at the next couple lights. So the cop (who I think took the pedestrian signal as a means to go at 4th and howard) caught up to make at 6th street. At that point he said to me "Boy, you are in shape!". Ahh, it's good to be home :)

Is it me or does Apple leave out obvious features from v1 product launches? 3G on a phone, a camera on a tablet... It's like they don't want them to sell.

I've been running home from work on mondays for the last couple months. I started with the direct 3.5 mile route home. Now each week I vary my route a bit, last night I took Geary most of the way west. However, while running I was thinking I should find as many hills to take on the way home with only a small amount of back tracking. Well this 5.3 mile run looks to be a good approximation. Anyone up for a run next monday evening?

Sita sings the blues is playing at the red vic. I went to see it tonight, it was really cool. There are 4 styles of animation and the narration is a lot of fun. The narration style is 3 people sitting around chatting trying to remember a story. The really cool part is most of the music is recordings of Annette Hanshaw from the 20s.

My part in tonight was using 2 rolls of dimes to pay to get into the movie :) That's two rolls of dimes now back in circulation.

On thursday I went out to see Loquat, The Greycoats, and Head like a kite. Head like a kite was a lot of fun, they had a dancing panda and foam heads (the ones you put whigs on) to throw around the audience. The strange part of the night was that I met Mark Johnson again. Mark was someone who knew a bunch of the same people as me in secondary school, then we both did co-op Math/CS degrees at UWaterloo, and since leaving school have both lived in the bay area. I didn't know Mark was in SF until this week, well, now we've met again, four years later.

Random thought:
Why are Canadians more patriotic when they are away from Canada? I saw a guy with a flag on his ipod shuffle while running yesterday.