Google hates XML

I assure you, Google does not hate XML.

On the contrary, we're one of the biggest supporters of XML-based APIs, with a huge variety of AtomPub services (http://code.google.com/apis/gdata/). Between AtomPub and our JSON-based Ajax services we're deeply committed to interop-friendly internet protocols.

Protocol Buffers are advantageous in other situations, though. Protobufs offer known endpoints the ability to serialize data with a high degree of time and space efficiency. The gain in performance and reduction in size is significant -- especially on the scale of data that Google processes! -- but we don't expose protobuf-based endpoints to external, unknown clients. For those we do indeed tend to favor XML-based protocols.

Cheers,

-DeWitt

To be fair, I've run into the same issues when storing large sets of small items for fast random access. Binary serialization had a huge (don't know about 200x, but definitely 10x) advantage, perhaps because it's more compact (which means less disk access and better use of caches) and is simpler to parse.

I've tested various "binary" XML formats, but didn't find any that offered a significant advantage over plain old gzip (especially when you take CPU use into consideration). Gzip is great for distributing medium-to-large documents, but less so for storing and transferring small pieces of data within a system.

I agree that the best optimization is to simplify the data model, but there is only so much you can do, and if you reach the point where engineering is removing stuff from the customer data model, then that's what I'd call having the inmates run the asylum!

But then I don't really have my life invested in any particular markup scheme, so I don't need to be personally insulted when somebody decides XML doesn't work for them ;)

XML is good for some things, but dealing with large amounts of data at wirespeed is not one of them.

I really think that what they have is a decent solution to a problem that XML isn't directly suited for.If you want the things XML provides when you are creating RPC then use XML, if you want raw throughput and low latency then google has _one_ solution.

CORBA, RMI, DBUS are other methods of doing the same thing.

If you're so good, why don't you just make xml run circles? Nobody is holding you back!

This post made me finally unsubscribe your blog. Happy days in your angle bracket world!

The shortest possible XML document, encoding a single-digit number, is by my count 29 bytes.

I don't think any super-clever XML parser implementation is going to be able to get around the overhead that a textual, markup-based representation has. It's a speed-of-light issue; you have to walk over and parse those characters.

For public APIs, XML has a lot of virtues (for example, highly corrupt XML documents are probably not going to cause sparks to fly in your application). But I wouldn't be surprised if Twitter was kind of wishing that they used Protocol Buffers for their internal message passing.

End point. Don't cry

Protocol Buffers:
person {
name: "John Doe"
email: "jdoe@example.com"
}

XML With Attributes:
name="John Doe"
email="jdoe@example.com"
/>

So where is the efficiency if I am using an XML appliance with with XPath in VLSI ASIC hardware?

<person
name="John Doe"
email="jdoe@example.com"
/>

(Internet Scale would imply # of concurrent users being measured in tens of millions)

XML is geared towards representation of data in a way where you really do not care all that much about performance. A key idea is to have a (presumably) self explanatory representation of the data than offered by pure, binary serialization formats. By virtue this means the data will be bulky and contain lots of information that is redundant in a representation that is extremely verbose.

Every data type, when expressed in valid XML, will use many times more bits to represent the same information than would be the case in a format where you take great care to only use as many bits as you have to (with some tradeoffs for performance).

This is fine. It is a design tradeoff for XML. It is why it makes sense to use XML for documents and in messaging architectures where performance is less of an issue than for the protocol to be possible to figure out by casual inspection.

Protobuffers have a completely different goal, and I think you missed that in your post. Protobuffers are about saving network bandwidth, memory AND CPU in situations where you have an abundance of data but a finite amount of money and space for hardware. Their intent is not as a format for representing documents in a way that makes recovering the information without any knowledge of the system that wrote the information. It is performance.

I thought it was rather obvious.

(How many XML parsers do you know that produce code for parsing specific schemas? Hey, perhaps they should.)

Here's a challenge for you. Implement a simple networked file system and use XML as your transport. Say the filesystem will be used by a multimedia-system for streaming multimedia content aboard a plane. You have 450 seats, each equipped with their own viewing device connected to your server by ethernet. Each user should be able to choose what he or she wants to see when, so you could end up serving 450 streams all at once. You can choose from the most common XML implementations that are published as open source, but you can not write your own specialized parser that only understands your schema.

In this scenario, electricity, networking, cooling, unit cost of the viewing device, weight of the system, latency, and availability will have an impact on feasability and usability.

Show me why XML would make sense in this scenario.

I believe you are confusing the .proto specification file (which is a text file that is input to the toolset, and is not part of what is sent over the wire) with the binary data that is sent over the wire.

Dan McCreary: You appear to have missed that messages can include other messages, thus creating an element-like relation.

Personally I don't see too much wrong with their approach. I wish they would have simply adopted JSON since they aren't too far from it and it is a well understood approach by more people - but its clear that they felt that they needed to re-engineer everything from the ground up for performance and I'm hopeful that they didn't just do a clean room approach for grins before looking at JSON and similar as solutions to the problem.

I wrote an article called the performance woe of binary XML

They should all be taken seriously, this one being the point in case:

3) Bandwidth is infinite.

Well - it really, really isn't.
Whoever tried to build a large scale server even by orders of magnitude simpler then what Google does would soon find that out for himself.