Migrating to Ruby on Rails and PostgreSQL: An Interview with CD Baby
by Robby Russell
Related link: http://www.cdbaby.com/
Derek Sivers started CDBaby.com back in 1998 as a personal project... a way to sell his band's album. He then started selling his friends albums. Now, CDBaby.com is the second largest online record store in the world ( second only to Amazon). Between 1998 and 2004, Derek was the sole developer on the CDBaby project. He alone managed all of the source code that powered the online store in addition to the web applications that are used by employees for handling orders, shipping and inventory. This was all done with PHP and MySQL. Not any more! In this interview Derek Sivers and Jeremy Kemper discuss the decision to move to Ruby on Rails and PostgreSQL as they rebuild the entire infrastructure that supports all of CDBaby, using best practices and design patterns. Jeremy is not only a CDBaby developer, but one of the core commiters for the Ruby on Rails framework.
Russell: How did you find your way to the Ruby programming language? I seem to remember hearing a story about you being stuck in a cabin. Perhaps that was someone else. In any event, would you care to share a short version of how you started to pickup Ruby?
Sivers: Christmas 2003. In Sweden for 3 weeks, stuck in a cabin in the woods with my wife and her mom, and no internet. Perfect time to learn a new language on my laptop.
Russell: What about you Jeremy?
Kemper: I heard of a light, expressive, pure-OO smalltalk-meets-perl that was popular in Japan. These were dangerously titillating language pheromones. I stumbled directly to Dave Thomas' Programming Ruby (the first one's free, they say..)
Russell: Derek, you were a self-taught PHP programmer for several years, what were your first thoughts of Ruby?
Sivers: I loved it right away. It made so much sense. After 7 years with PHP, I *still* have to look up the parameter order for array_search or strpos. Ruby felt intuitive right away.
I started using it for some shell scripts, and wanted it to use it more, but since almost everything I do is on the web, I had no practical way to use it until Rails came out.
Funny aside : I was subscribed to DHH's (David Heinemeier Hansson) blog since *before* Rails came out, and when he was dropping hints of it on his blog, I was nagging him every couple weeks saying, "Is it done! This is just what I need!"
CD Baby Meets Rails
Russell: Jeremy, what is your day job?
Kemper: I'm a CD Baby. Derek Sivers and I are bringing Rails to cdbaby.com and its satellites.
Sivers: That was around January 2005, so about a year after I learned Ruby and a few months after I saw Rails.
Really Rails didn't impress me that much at first. I didn't want to make a simple blog. I needed to make this massive ecommerce and intranet. It didn't seem up for the job.
I emailed DHH and told him I *wanted* to love Rails but didn't see how it could work for CD Baby. He hooked me up with Tobias Lütke who sat with me on the phone for an hour over VNC - (me watching his screen while we talked) - and once I saw how
./script/console and helpers worked, I was convinced. Console and helpers are what pushed me over the edge.
PostgreSQL was a no-brainer. Within an hour of learning it my very first time, I was hooked for life. As CD Baby has grown, I've had so so many problems with MySQL. PostgreSQL solves them all. (Mostly the forced
data-integrity : foreign keys, constraints, etc.)
Russell: Jeremy, at a recent Portland Ruby Brigade meeting, you discussed how you were previously attempting to implement your own framework prior to using Rails. How long after Rails was first released to the public did you finally make the decision to join the Rails camp and start helping out?
Kemper: I was struggling with some typical Not-Invented-Here.
Active Record was released a month before Rails; it complemented my work and was a great joy. But Action Pack wasn't /my way/. I hacked it up and stole its best ideas.
Then I saw this community precipitating out of nowhere. Rails had teased out a sweet spot and its users were already there cheering its arrival. I felt short-sighted and foolish having plucked its wings and prettiest bits to play with in my sandbox.
So I put that away and started sending patches.
Is Rails Mature Enough?
Russell: What were some concerns about moving to Ruby? Do you still have those concerns? Do you have new concerns?
Sivers: Ruby? Never. Rails? Yes.
I had to make sure it was up for something this size. CD Baby has grown pretty complex behind the scenes, as we're one of the main distributors of music to Apple iTunes, Napster, Rhapsody, and 50 other digital music services. Our database has 95 tables (optimized!), runs a huge backend
intranet, audio encoding farm, direct payment-management with a few banks, and more.
Most people are using Rails for cute little single-focus projects. But I had to switch my system over to all-or-nothing.
With the smarties at the helm of Rails, I'm convinced it'll scale. No concerns.
The Rails Community
Russell: What is it like to have seen the community grow so large since last year?
Kemper: I have a poor grasp of the community's size. At RubyConf I fleshed up with the other Rails committers for the first time. Following the mailing list and patch queue is impossible, much less the IRC channel and its spinoffs.
Russell: At what point did you realize that Rails was going to really take off?
Kemper: When its users started taking off. Some of these folks needed Rails bad.
Russell: How long ago did you actually start re-writing CD Baby in Ruby on Rails?
Sivers: January 2005.
Russell: If you could pick one component of Ruby on Rails that has consistently brought you a smile, what would it be?
Best Practices and Patterns
Russell: In previous discussions that I have had with you, even before you made the decision to move to Ruby on Rails, you made references to authors such as Martin Fowler and Dave Thomas. How much of their programming philosophy do you feel has influenced your decision to use Ruby?
Sivers: Before I was about to start the rewrite anyway, I decided I really wanted to do things according to the best practices in real object-oriented systems. Encapsulation was my goal. The CD Baby PHP code is 7 years of tangled mess. There are many things the system is unable to do without a total rewrite, because of its total lack of encapsulation.
I was already studying Martin Fowler's books like a schoolboy, ingesting the lessons while preparing for the rewrite. Then when Tobi really showed me Rails I got it : by sticking with Rails' conventions, I would be already
working by these best-practices I was aspiring towards.
Russell: Are there other authors that you enjoy reading?
Russell: What programming books do you keep at arms reach on a consistent basis?
Sivers: This year I hardly stop to read. Every minute is better-spent programming. So it's mostly reference manuals kept handy for quick lookups. "Agile Web Development with Rails" is awesome.
Russell: Are you a PDF book or a Hard copy book person?
Sivers: PDF! All the way! I get frustrated when a book is not in PDF. Usually I'll skip it altogether if not in PDF. Of course, I'm a big fan of O'Reilly's Safari service.
Does it Scale?
Russell: How many developers are currently working on your rewrite of CD Baby from PHP to Rails?
Sivers: Just me and Jeremy Kemper. Hell - I'm even "part time", really, since I'm still the president of the company that has to go to conferences, resolve conflicts, do business deals, hiring, firing, etc.
Russell: How did you and Derek meet?
Kemper: Derek offered me a job through a whois lookup.
Russell: How long have you been part of the Rails-commit team? What areas of Rails do you typically get delegated with to work on and oversee patches coming in?
Kemper: Since the beginning of the summer. Delegation is uncommon, though we each have things we 'own' and we 'always' fix things we break. I have a soft spot for good old core Rails -- Active Record and Action Pack -- because I use them most.
Russell: Many people are probably not familiar with the whole infrastructure of CD Baby, so when they hear that you are rewriting the website, they might not be thinking beyond what the public sees. If you were to put a percentage to how many lines of code were on the public interface to the number on the inside. What might you guess the percentage would be?
Sivers: What's on public view at cdbaby.com is probably only 20% of the code. The backend is much more complex. It's really four projects:
#1 - cdbaby.com
#2 - members.cdbaby.com (musicians' account management)
#3 - the intranet (EVERYTHING that runs the company, even payroll)
#4 - our digital distribution audio encoding and delivery system
Each one is a separate Rails project, though they all access the same database.
We're using Subversion's "svn:external" feature to share our model classes ("./app/models") between all four projects.
Migrating from MySQL to PostgreSQL
Sivers: Data integrity. Forcing foreign keys. Constraints. If a field is meant to hold a barcode, it should not accept "123456" because that's not a valid
EAN/UPC barcode. If it's meant to hold a country code, "xx" should fail. And "0000-00-00" is not a date!
You can try to prevent these things at the interface-level, but... y'know... "shit happens". I like a strict database that will prevent that data corruption, no matter what interface is used.
Russell: When is CD Baby planning on launching the new system? 3 months? 6 months?
Sivers: Three months ago. ;-)
Russell: What part of rails do you find the most lacking? How could it be better?
Kemper: I think the web site is too dense. The info you're seeking should find you. Questions like "how to multi-parameter form attributes get turned into objects" shouldn't be answered by a multi-paned window of class, method, and file names.
But more crucially, is Rails truly enterprise-ready? Does it scale?
Russell: Do you feel that the critics of Rails are missing something when they claim that it is nothing more than just a fad or hype?
Sivers: No. They're totally right. It'll all be gone next week. Nothing to see here. Move on, people.
Russell: What do you think the rails community needs to do to legitimize Rails to large, conservative enterprises?
Kemper: Build great apps. Don't get distracted by legitimacy.
Russell: Where do you see rails in a year from now?
Kemper: Rails 2 will be tickling your fancy.
foreign key constraints
Strict date checking is relatively new to MySQL (added in the 5.0 version), but foreign key checking has been there for the past four years.
foreign key constraints
To some degree (http://sql-info.de/mysql/referential-integrity.html)
Good Luck, Derek!
I'll be pulling for you guys, since the last CD I tried to buy, the database was lagging so hard it disuaded me from continuing on! Good luck!