There have been some discussions about this already, notably a QCon presentation, but a redundant tutorial or two never hurt a soul. This is for the Python version of App Engine, but it shouldn’t be much different for Java.

Why?

App Engine datastore writes are really expensive, and prohibitively so for something like custom statistics. For example, in an early version of SearchEkko, where I read and wrote to an “Admin” Model object on most operations, it was costing over $6.20 CPM to serve SearchEkko search results pages! This when the marginal cost of web-delivered content is supposed to be zero. Obviously, App Engine wasn’t designed with this kind of usage in mind, but it was equally clear that I needed some type of custom statistics dashboard (how many displays, how many installs, etc). So I started poking around.

The Requirements (And Can-Do-Withouts)

To summarize, I needed some type of database solution that:

• Offered cheap (as in resources) read and write access
• Was cheap (as in money)
• Was easily accessible from App Engine (so probably REST)
• Tolerated frequent accesses and responded relatively quickly

Since all I needed was a way to track statistics, I could forego a lot of typical database niceties, like:

• Transactions and thread safety
• Guaranteed data integrity
• Bullet-proof security

Google Spreadsheets was what I eventually found, and so far it’s met my needs almost to a T.

Note: if you’re looking for something to use with real, critical data, look elsewhere. This is definitely a little hacky, and certainly isn’t perfect, but it’s a relatively easy way to track things you don’t care that much about in App Engine.

Setting Up Your Spreadsheet

Just go into Google Documents and create a new spreadsheet. The URL will be something like

http://spreadsheets.google.com/ccc?key=bNadyGyiH2Ma6Cx54NmiL2e__4g&hl=en

Note the value of the key argument — you’ll have to use it later.

When you’re accessing your spreadsheet from App Engine, it’s easiest to just set specific cells to the values you want. This means you can lay out the spreadsheet pretty intuitively. In my SearchEkko statistics spreadsheet, I simply made the first row labels for each column, with the corresponding data falling below each label, exactly as you’d set up a standard spreadsheet.

Accessing Your Spreadsheet

In order to access your spreadsheet you have to use the Google Spreadsheets API. There is a Python library you can use, but it’s apparently outdated now. Plus, for something pretty easy like this, I’d rather just use the base protocol, and that way I’ll be more likely to know what’s going on when something breaks.

In any case, you need to authenticate in order access your private spreadsheet data from App Engine. The best way to do that for our purposes is by using AuthSub. AuthSub is similar to OAuth, but isn’t an open standard, and is a bit better-suited to Google-specific tasks. More to the point, it’s what I used, so it’s what you’re learning.

One of the easiest ways to play around with AuthSub is by downloading RESTClient, an addon for Firefox. Play around with it to learn how to submit HTTP POST and GET requests to specific URLs, and with modified headers, or use another tool of your choice.

The next steps are also documented here, but I’ve tried to simplify and streamline them for your coding pleasure.

1. Direct your browser to

   
   https://www.google.com/accounts/AuthSubRequest?scope=http%3A%2F%2Fspreadsheets.google.com%2Ffeeds%2F&session=1&secure=0&next=http%3A%2F%2Fwww.example.com
   

   Naturally swap www.example.com with your own site. Grant access, and note the URL that it sends you to. Copy the the token parameter. That’s your single-use AuthSub token.

   Note: This should work fine as-is if your site is hosted on App Engine, but if it’s not (ignoring for the moment why you’re reading this), there may be some additional steps here.

2. Send a GET request (in RESTClient terms) to the URL

   
   https://www.google.com/accounts/AuthSubSessionToken
   

   Use headers:

   Content-Type: application/x-www-form-urlencoded
   Authorization: AuthSub token="yourAuthToken"
   

   Obviously replace "yourAuthToken" with the single-use token you received, but keep the quotes. You should get back a 200 OK status code, and the body will include your session token. Though some documentation shows this value having an expiration, it really is usable indefinitely. (You can revoke any tokens your Google Account has given out here).

Alright, now you have your indefinite-use AuthSub session token, and you’re one giant step closer to mingling Google Spreadsheets and App Engine.

Or at least it would have if I had a PR firm and a few thousand dollars to get some coverage. But the code has gotten to the point where there’s a real, useful product (in my opinion, anyway ;-). If you’re not in the know, here’s the blurb from the home page.

SearchEkko is like a “related posts” widget for your blog, but it’s only visible to readers who arrive from a search engine. It uses each visitor’s search query to find the best-matching pages on your site.

So basically it uses visitors’ intent to help drive page views and provide a better user experience. There are some good comments over at Hacker News.

What now?

The next step is to actually get some distribution. It’s installed on about fifteen sites at this point, and it’s displaying on the order of 1,000 times a day. Nipping at the heels of ShareThis, right?

Tim Westergren, CEO of Pandora and a man I would have taken a bullet for until all these new audio ads, pitched 347 VCs before he got his second round of funding. 347. That is almost a week of non-stop, twenty-four-hours-a-day, PowerPoint pitching. But of course it was much worse than that, instead spreading across the collapse of the internet bubble and three soul-crushing years.

Anyway, if Tim can do that, then I should be able to hang in there for a few months of direct sales. So my goal is to email the 1,000 top business/tech blogs on Technorati, and then see where I am.

Wish me luck.

Update: Paul Stamatiou just alerted me to WP Greet Box, which offers a very similar product. Their targeting seems to be worse than that provided by Yahoo BOSS, but they have more features (naturally including some I’d been planning on doing). In any case, I’ll have to muse on SearchEkko’s future more. In the meantime you can read about competition at Jessica Mah’s blog.

What we may be witnessing is not just the end of the Cold War … but the end of history as such … That is, the end point of mankind’s ideological evolution and the universalization of Western liberal democracy as the final form of human government.

Francis Fukuyama, The End of History and the Last Man

Fukuyama wrote those words in 1992, with the fall of the Soviet Union fresh and the bounds of Western achievement limitless. Since then there have been many missteps for democracy and its cousin the free market, with the current economic worries coming in at the top of the list. Yet few of us would seriously consider anything else – indeed, few could even conceive of an alternative.¹ After all, we always seem to muddle through. Why should the future be any different?

There are a million possible answers to that question, and it’s anyone’s guess which is right (if any). But muddling is hardly something to aim for. And, more importantly, our government hasn’t fundamentally changed in over two centuries. Wouldn’t it be surprising if something better wasn’t out there?²

An Alternative

I don’t much like paying taxes (who does?), but what I really hate is seeing that money misspent. So what if citizens got to choose where their tax dollars went? And I don’t mean how it is now, with this quasi-representational democracy. I mean each and every dollar earmarked for a specific purpose. You could pay a dollar of your taxes and have it be used only for scientific research and education, or only for infrastructure and immigration. The money would go directly to the agencies responsible for the tasks, and you could pay only for whatever you personally deemed most important.

Obviously some government domains are important and universal enough that they should have some guaranteed revenue. For example, maybe everyone would send at least 5% of their tax bill to defense. And some people might not care – they could just spend it in all domains equally, or according to some default arrangement. (But I bet most people would leap at the chance to better control where their dollars went.) And some people might want to zoom in past the broad categories, spending on scientific research but NOT stem cell research, for example.

You of course would also need some type of overarching administrative body, perhaps similar to a dramatically scaled back version of our current legislature. They would handle the mechanics of collecting tax money and other non-domain-specific issues (such as tax rates, creating new agencies, emergency revenue needs, etc.).

Pros and Cons

The biggest attraction is tightening the leash from tax payers’ wallets to their government. No end of pain has come from democratic governments misinterpreting and debasing the wills of their citizens. If a government agency is ineffective, it won’t have a budget. End of story. Maybe even people could create bids to replace agencies that are doing poorly.

The biggest drawback seems to be introducing an unhealthy degree of focus on marketing and salesmanship into the government. Each agency will be wooing the taxpayer directly, even more than they already do today. But is this even a true con? Aggressive marketing sharpens companies visions, and holds them accountable when they can’t keep promises. I think the same would hold true for government agencies.

Will This Ever Happen?

Not likely. But it seems like a good dream. And, according to Thomas Jefferson, we’re long overdue for a revolution.

Ideas don’t have much street cred in the tech world. Execution, not inspiration, is what creates great companies. And it’s true; execution beats the idea 99 times out of 100. Yet even seasonded Entrepreneurs in Residence are usually looking for a “good opportunity” — no small component of which is the idea. The idea is what drives the passion, and from the passion all else follows. But finding the right one can be hard.

This idea-less-ness is a stage I’ve been flitting in and out of for the last couple of years (my current project is here), and I think it’s particularly overwhelming for young people first getting interested in the tech world. However, I’ve discovered two classes of ideas to avoid during this period, and, having found them, I think I’m much further along the road to creating a real business.

Magic Carpets

The first class is “magic carpet ideas”. These are the ideas that have huge and proven demand, but, unfortunately, are impossible. At a larger scale, you might call them pipe dreams. Naturally the inherent infeasibility of these ideas tends to prevent them from reaching the light of day, but nevertheless they appear from time to time.

Even though most of these ideas get killed off before they hit the market, an ambitious young wantrepreneur can delude themself for quite some time (I know because I have). And that is time that could be spent working on a real idea — or at least engaging in a more satisfying fantasy. So what should you do to avoid this mess? Research. Of course it’s best to tackle ideas in areas that you know something about to begin with, but the bottom line is look into the details. Don’t waste time choosing names or making logos — figure out how exactly what your company sells, how it makes it, and (eventually) who buys it.

Pyramids

The second category of ideas to avoid is “pyramid ideas”. And I don’t mean the Madoff variety (though they also have some drawbacks), but the sort that took 20 years and 200,000 men to build. These are the ideas that are, in fact, possible, but are way too big for a young bootstrapper to manage. I, for example, at one time hoped to create a new web browser. Anyone who has ever attended a business plan competition will agree that other examples are anything but lacking.

This class of idea is especially alluring as most VCs want to invest in home runs, not singles, which makes you want to start slugging as hard as you can.¹ And, even worse, examples of successful pyramid builders dominate the news. Nevertheless, in my experience it is very easy to be “working” on a large, overwhelming project while accomplishing very little. Chipping away at a smaller, more manageable project is a far surer path to tangible results.

So Which Ideas Are Good?

I’ve given you two types of ideas to avoid, but of course there are infinitely many wrong ways to do something. The good beginner ideas are ones that you’re passionate about and that you can really, truly do (like, start working on it this afternoon). These two goals seem even more important to me than demand and profitability for the first couple of projects. After all, if you successfully complete a project and it turns out you’re only scratching your own itch, the worst case scenario is you learned something and you don’t have an itch anymore.²