Berkeley DB

I confess that I strayed. Perhaps its been spending too much time in fake grad school that had confused me, or perhaps I'm just not as saavy a developer as I thought I was.

I've been working on a certain 100,000,000 row dataset problem using hand built data structures and mmap-ing them out of the file system. This morning, on a lark, I loaded the data into a Berkeley BD instance, and I'm haven't looked back since. Even with a Btree index the database builds faster than my own code, or that of other good netflix libraries, and then you get the benefit of the index on retrieval.

I haven't evagelized for Berkeley DB here yet. And not everyone will agree with me that its a good solution for most of your data access needs (certainly my alma mater didn't. But if you have a highly performant problem and what a quick, simple and very low overhead mechanism for managing your data give Berkeley DB a look. It eve has ruby bindings. It's free as in beer and code. [Updated: See comment below for an actual description of the licensing requirements.]

So for all two of you that read this blog and know and care about these things: Why are we still using relational databases for our websites. I understand the need for flexible systems for our reporting tools and data mining, but when a website has maybe 30 queries that need to be executed, and that database is almost always the first piece of a architecture to fail, why not use a customer database without the overhead of query management? Why are we still parsing SQL for simple CRUD calls when it could be done 10 times faster?

Web 2.0 should be so much more.

Yeah. Second post promised about a month ago. Okay so this post is maybe two years after everyone stopped trying to figure out what's going on here and the models have even reached the point of complete mockability. But I'm still thinking about these things, primarily because I'm reasonably convinced that everyone's got the answer wrong. Web 2.0 by and large refers to a class of Ajaxy (completely irrelevent), community driven (tangentially relevent - but epinions is still Web 1.0) and tagged content (and this is the real story). I'm pretty happy to start with Web 2.0 as a system of tagging platforms. That doesn't include Facebook and MySpace.

So what's going on here then. I think Mark Zuckerberg get's it right when he takes about creating graphs of meaning (I'm extending his comments slightly). I'm confused as to why we've gotten this far into the evolution of Web 2.0 and Facebook is the first company to publically disclose itself as a content managment system with nuanced edges making navigation between content nodes trivial. What's del.icio.us other than a graph of url nodes with typed edges between this nodes -- there's no reason on consider just one set of edges, Facebook as two edge types (users, groups), plus an open API for creating new edge types.

It's frustrating to me that the conversation of Web 2.0 has been so focussed on social edges between active users because there's so much more meaning that can be categorized and made accessible using these tools. The original set of companies, Flickr and Del.icio.us weren't just platforms for communicating, but systems that enabled gross categorization of human knowledge (well the part on the web). That part's mostly gone, and now you just build a site for people to find a date on.

Couldn't we somehow crowdsource an asset of considerable social value? Can't someone create the Google of this world? The data stuctures, query languages and general approaches to these problems have even been solved already using Topic Maps. What's missing is the successful commercialization of these solutions in a slick (okay maybe Ajaxy) interface that caters to people's need to categorize the web.

It's probably way to late for these solutions though. Because next batch of startups are coming that categorize dynamicaly using learning machines rather than crowdsourcing. The technical challenges here are impressive but it's nice to see a few kids here and there getting hot and bothered about these solutions. Since these systems won't have people or tags -- does that make them Web 3.0?

freemium

There's nothing you haven't read before here. And it will probably change.

Freemium is everyone's new favorite business model and for good reason. With freemium you allow anyone to use your product for free, albeit with limitations, and you lift these restrictions as consumers agree to pay more than nothing for the service you are providing. It's interesting that this works, and what's more interesting about this is that freemium is the exact same business model we all made a ton of fun of during the .com days as completely non-functional. In interested in why is it that this time around freemium isn't just not suicidal pricing plan, but in fact, is the preferred way, in many cases, to deliver your services to your customers.

At its heart, the freemium plan is a simple price discrimination plan where you drastically lower the admission price for the service while charging substantially higher costs for other additional benefits. Think admissions to fun parks, and then additional charges for the tickets to rides. Some consumers (parents) would only be willing to pay the lower fees associated with getting into the park, whereas other consumers (kids) are willing to pay the much high cost for taking a lot of rides. Price discriminating plans like the freemium model allow the service vendor to collect money from both demographics in a natural way that doesn't rely on bizarre restrictions and satisifies each group (on the margin) exactly as much as they end up paying. It's a much more efficient way to run a business.

But what happens when the admissions price is free? That's where some of the problems leading to .com nonsense started. Becauase as Josh Kopelman from First Round reminds us in his pricing post, the linearity in the demand curve simply doesn't exist as you get closer to free. In fact, I'll take this further and say that as the price approaches zero you'll get near infinite demand (there's always some frictional costs that you would have to pay someone to avoid). Fantastic right? Well, lotsa sites thought so but you're left with the problem of figuring out how to afford delivering your fantastic service to the masses. That's where we start looking for even more fantastic features for our service that some people (the vast minority) might be ready to pay for.

People will pay for these services too. Converstion rates can be as high as 3%. This can add up to some serious money over time, with very little staffing. Build your product correctly and these incoming streams essentially become annuities that you can use to fund further growth. You might really be able to limit your need for external investments. Sure worked for 37signals.


So what went wrong then, and why can this model work now? I'm pretty sure that the solution to this has to do with network effects. In Web 1.0 (are we really calling it that), users' experiences were pretty much silos, so users that aren't paying just put a drag on your system, costing you tons of money. Now when you don't have commodity choices for delivering your webservice you can get into a lot of trouble and hosting fees can kill you. Certainaly the cheapening of bandwidth and server costs post .com crash helps with these issues, but that only gets you so far. Hit sites still pay tons of money to deliver their services to their users. So what else changed?

I'm guessing that the addition of social and community components to online services have raised the prices the paying are willing to pay in order to use these services (or increased the number of users willing to pay at a given rate -- same difference). In fact, with social components, each new user to the system creates additional value for the users already on board. That's not to say that all communal services can make a mint. If your network effects are restricted by grouping functions and other logical barriers that limit users ability to interact with your other users, then you've contructed mega-silos that contrain incremental value building into buckets.Salesforce.com doesn't really let you stay a user after the first month (free CRM doesn't really offer the functionality of the Salesforce.com platform).

So what types of services should look at this pricing model? Social networks are the obvious candidate. Tools that produce propriatary formats are great for this too, as it gets easier to use these tools are more people adopt them. But a lot of web sites get good traction, but won't reach phenomemal, or scalable, growth using this technique because they product simply doesn't map well to the pricing model.


Services likely do to well with this model: LinkedIn, Facebook, AIR

Services likely not to scale with this model: Pownce, 37Signals