I'm Hung Up on Main Memory Databases

by Brian Jepson

At a former job, I worked on a small part of a big system. Although I got to work with some extremely cool stuff, the part of the system that got me drooling was based on a main-memory database system. Even though I didn't get my hands dirty with that database, I got to sit in on product pitches from Polyhedra and TimesTen as well as a few high-level design meetings. Along the way, I learned that main-memory databases are popular with anyone who needs real-time database access. Telecommunications companies love this stuff. The main memory databases are also popular in places where there is no disk (embedded devices). And at the time I spoke with the sales reps,
e-commerce types had just begun sniffing around in that area, hoping to make their web sites faster and faster.

Ever since then, I've been hoping to find time to play around with a main-memory database. But, there are only so many hours in a day, and there is so much other fun stuff competing for my attention.

Main memory databases are very cool. Once you throw away the assumption that everything is stored on a disk, you get a lot of advantages. For example, you can use hashtables for indexes instead of B-plus trees. That makes everything a lot faster, but you're not allowed to perform partial matches in queries (perhaps you are in some implementations, but the queries would be very expensive).

Of course, there is a lot of weirdness, too. Since all your data would disappear if the power went out, you need a good replication architecture. Or, you could use the main memory DBMS as a super-cache for something like Oracle. In that configuration, you could send UPDATE and INSERT statements directly to Oracle, but perform all your SELECTs against the main memory database.

If you want to play around with an open source main memory database, you might already have one installed. Check out MySQL's HEAP tables. I haven't evaluated how they compare with Polyhedra and TimesTen, but MySQL's HEAP table implementation seems comprehensive.

Here are some main memory database links I picked up on a quick Google search (if you have more links, please suggest them):

Speaking of Google, this article suggests that Google uses main memory databases to make their searches so darn fast. The Slashdot discussion of this piece makes for interesting reading, too.

Have you worked with a main-memory database? How did it compare to traditional relational database offerings?