The Pet Store.... Again

by Ted Neward

I live in two worlds.

I've been writing Java code for over 6 years now, specifically in the back-end server space. (I was using the NetDynamics product as far back as late 1996, when JDK 1.0 as the norm.) I've been writing about Java since 1998, and teaching about Java since 1999. I've also been writing .NET code for over 2 years now, specifically in the back-end server space. I've been writing about .NET since 2001, and teaching about .NET for about the same amount of time. Living in both worlds gives me a really interesting perspective, both politically and technologically; I'm a member of the JSR-175 Expert Group, to bring metadata annotations into Java, and I'm writing a book on Rotor, the Shared Source .NET runtime implementation. I love Java, I love .NET. I want both to thrive, because the innovations produced as a result of competition between the two benefit us all.

Normally, I really really try to avoid the whole benchmarking thing--it's a black hole that ultimately does nobody any good. People ask me in my classes (J2EE or .NET, doesn't matter) which technology is faster, Java or .NET. I tell them that I've never run a benchmark, because I believe the variables between the two camps are too many and too diverse to accurately benchmark, and that ultimately it doesn't matter because the choice of which technology to use is usually made on a golf course or over a fancy dinner, not in a benchmarking lab. DevelopMentor, the company I teach for, went through this "should we do a benchmark to compare the two" discussion about two years ago, and we all almost unanimously voted against it, for many of the same reasons.



When Microsoft released their version of the Pet Store a year or so ago, I was moderately amused--they'd rewritten the sample to use what they believe to be .NET best practices (which at this point is realistically anybody's guess), and behold! Massive performance improvements. The Java community as a whole ridiculed the effort, and several J2EE app server vendors responded by writing their own versions of the Pet Store demo, and behold! Massive performance improvements. Nothing new here.



But now, The Middleware Company, a reputable Java and J2EE training and consulting company (and, I must, in the interests of full disclosure, point out, a rival to DevelopMentor) has taken the plunge and attempted to produce a benchmark that seriously compares J2EE and .NET on the subject of the Pet Store once again. They published their results last week, and, to put it politely, all Hell broke loose.



First and foremost, I want to point out that I deeply respect the pure cojones of The Middleware Company (TMC) in producing this--they knew, when they saw the results, what the reaction within the industry would be, and they published the results anyway. I have to believe that this was in the honest interests of trying to toe the careful line of honest reporting--either that, or TMC has decided to force themselves into bankruptcy. It simply makes no sense for them to do anything otherwise (unless they've suddenly decided to become a .NET training company, which I doubt). So I think it's important to give TMC some props on that score. I believe they did try to make it as fair of a benchmark as they could, and they did achieve a serious performance improvement (17 times better than Sun's original implementation, according to the report (p. 7). Whether they did everything they possibly could have done is another issue, addressed later.



In the meantime, however, it seems as if everybody within the industry, particularly the Java guys, have been leaping to their feet to decry the results of the benchmarks. In basketball, we'd be seeing a bench-clearing brawl right about now. More vitriolic poison has been slung between posters on message forums over this benchmark than in the last U.S. Presidential election--it's amazing.



As someone who lives and breathes in both camps, I feel compelled to offer my own spin on things, to anybody who's still listening. In particular, I want to address some comments made by Dion Almaer, in his
OReilly weblog, and Rickard Oberg,
in his online website review of the benchmark. I've never met either personally, but I have exchanged email with Rickard (a number of years ago). I deeply respect both.



Comments


Without further ado, let's get into this. As always, I welcome comments and criticisms of my criticism.

  • Dion points out that "What was the point of this report/benchmark? It seems unclear to me. Was it just about
    performance? Was it about ease of development (LOC)? It blurs." Absolutely. The LOC measurement in this benchmark definitely dilutes the results of a performance-oriented benchmark, and vice versa. More importantly, I'd like to see a benchmark that drills down even further--how would you optimize the J2EE/.NET application to optimize for performance as opposed to scalability? Or even the numbers when "optimizing" for design purity (which Dion touches on later)? The benchmark doesn't state clearly what it's trying to measure, and as such, leaves us with a sense of confusion regarding what we're trying to measure here.


  • Dion writes that "M$ could even tweak their .NET core! Do you think you can download that .NET runtime right now?" Dion, I hate to say this, but that's a spurious argument and a foolish one to rely on. Do you honestly believe that Microsoft wrote a specialized version of the runtime just for this benchmark? I can honestly say that I've met a number of the guys on the CLR team in Redmond, and let me reassure you, they have MUCH bigger things to deal with than this benchmark. More importantly, I challenge anybody out there to download the Rotor source and point out what optimizations could be made to the .NET runtime (or to the JVM, for that matter) that would seriously impact the performance of this benchmark. This is zealotry talking, and we don't need that here.


  • Dion says that the report "does not fully disclose who did the report, who was involved in the benchmarking, why certain "rules" were chosen, and who paid for it." This is a fair criticism; in addition, both Dion and Rickard point out that the way this thing played out, it *really* looks like it was skewed from the beginning to favor the Microsoft guys. I would think that, given the leak of the email from Microsoft showign that they knew about the benchmark results before it was posted, TMC at least owes an apology the community at large, *ASSUMING* they presented it to Microsoft before the results. In fairness to TMC, however, we must consider the possibility that they never actually presented the results to Microsoft--just as Rickard got the internal Microsoft email, it's fair to consider the possibility that somebody within TMC leaked the benchmark results to Microsoft, meaning TMC wasn't giving Microsoft a heads-up after all. Check out the exact verbase of the email: "The week of Monday Oct 21, a draft of the Middleware report was distributed to analsysts during private 1:1 analyst pre-brief meetings with...." NOWHERE does it say that TMC gave the report to Microsoft, or was even present at these meetings.


  • A number of people have criticized that the benchmark wasn't run on Solaris or Linux. The benchmark report states, quite clearly, that "Both application servers were tested on both RedHat Linux 7.2 and on Windows 2000 Advanced Server (SP2)", and that TMC "used the operating system that provided the best performance with that application server for the final, published test runs." Should the tests run on Solaris, as a comparison? Absolutely. Will it be hard to find "comparable" hardware to run the tests on? Not at all--just buy whatever equivalent hardware I can get for the same price as the Compaq hardware used to run the tests, at list prices.


  • Dion writes that certain optimizations weren't part of the app, and lists Local Interfaces and other JVMs as examples of what wasn't used. Again, however, TMC states that these were considered, and that Local Interfaces turned out to be no faster than remote ones, since any sane App Server would optimize the Remote Interface under the hood anyway, and that JRockit was, in fact, used, or at least attempted to: AppServer B couldn't support anything beyond JDK 1.3. This in turn makes me question whether AppServer B should even have been used, but that's another story for another day. (See the FAQ.) Honestly, the fact that AppServer B doesn't support 1.4 tells me that the major App Server vendors aren't all that concerned with the performance of their products, either. (And what the heck are they doing internally that prevents the app from running on 1.4, anyway?)


  • A number of people have criticized the J2EE PetStore implementation for being unoptimized. First of all, let's not forget that this served as Sun's "Blueprints" document for a number of years, offering advice on how to build scalable systems using EJB. If it's not optimized for performance, then Sun's been leading all of us down a dark and unoptimized path for four years now. Shame on them.

    However, I am really getting sick and tired of Java developers trying to justify EJB as being the way to
    scale up and perform well, then claiming that the PetStore, which uses EJB, isn't optimized to support scalability and
    performance. Folks, it's high time we took the blinders off, stopped drinking the Kool-Ade, and realize that EJB doesn't magically give you performance and scalability--it's your application implementation and design that gives you that.



    So, then, would somebody *PLEASE* step up, write the PetStore the way the community thinks it should be written (using stored procs, no EJB, whatever), so we can kill that original implementation and be done with it? If it's not written the way it should be, then Sun should remove all traces of the Pet Store from their site and release one that does. If they don't, then they're implicitly still backing the idea that the Pet Store is the blueprint to building EJB applications, and they deserve what they get when they get crucified in benchmarks.



  • Rickard writes that the .NET implementation is "seriously flawed", in that the .NET implementation mixes SQL logic into the middleware componentry to do the actual data-access layer. "Look at, for example, Product.cs which contains both object definition of the Product class as well as the SQL/code to access it as static methods. That is not very "object oriented", now is it?" Frankly, as I've been trying to say for years, EJB and other distributed technologies aren't about building distributed objects, but about building distributed components, also known as services. So the argument that this approach isn't very object-oriented is, to me, a red herring.

    I think it's also worth pointing out that having the SQL in the Product class really isn't much different from having the SQL hard-coded in the ProductEntity bean (if it's a BMP bean) or in the deployment descriptor (if it's a CMP bean, thinking specifically of finders and/or selectors). I will point out that layering business rules in a SessionBean and data-access code in an EntityBean yields better segregation of code, at the expense of violating good O-O encapsulation between the two objects--so which do we want? Segregation of business rules from data access, or good encapsulation boundaries?



  • Much of the rest of Rickard's assessment of the benchmark, particularly his dissection of the J2EE implementation, sheds some serious question on how the J2EE implementation was optimized. Rickard comments that Ed Roman has been calling various people, "trying to convince them that [Rickard's] review is basically written because [Rickard] is out to 'get' [TMC]." If this is true, then I'm seriously disappointed in TMC's response. I grant you, TMC is definitely in the hot seat, but I would hope that they would welcome an opportunity to retest the benchmark with some of the optimizations that the Java community is calling for. What I would *really* love to see, as I said before, though, is somebody to write a version of the Pet Store that we can all agree is the "right" one to use, and put this question of Pet Store's implementation to bed forever.



Summary


At the end of the day, however, I hope that this benchmark does a couple of things:

  1. Java developers, .NET is a serious contender and is here to stay. Simply calling Microsoft names and writing the derogatory "M$" everywhere doesn't erase the fact that Microsoft has produced a viable alternative to Java, particularly on the Win32 platform. Stop being silly and closing your eyes to this. Instead, do what Microsoft does so well, and steal from them: steal ideas, steal concepts, steal designs, then twist them around to make your Java apps faster. .NET doesn't provide any kind of support for
    entity beans? Ask yourself why that is. .NET doesn't provide any kind of support for object pooling? Ask yourself why that is. I'm not saying that everything Microsoft does is perfect--Windows DNA is a prime example of that--but don't lull yourself into complacency by simply chanting "Microsoft sucks" to yourself and ignoring what's going on over there. There's a lot of bright people there in Redmond, and they've been thinking about this problem a long time.


  2. .NET developers, don't be lulled into complacency, either. The various Java vendors are *not* going to take this benchmark laying down, and you can bet that the Java guys are going to come after it with both barrels blazing. In particular, there is a tremendous amount of good ideas that the J2EE world embraces that are somewhat awkward to do in .NET; one such case in point is the whole Model-View-Controller paradigm for building webapps. Pick up a Servlets/JSP book, and figure out how to bring that over to ASP.NET. Don't start thinking that you can just dribble SQL all over your .aspx pages and expect that the .NET runtime will somehow make your application "just run faster"--it isn't going to work.


  3. Java developers, this benchmark is essentially an indictment of EJB, not of Java. Demand to see a benchmark that doesn't use EJB in the middle, using stored procedures against the Oracle database, built in an intelligent fashion that doesn't include dead code or redundant transactional boundaries. Think you need EJB to work against simultaneous distributed databases? Think again--most servlet containers (the principal one I'm thinking of is Tomcat) support JTA, which means you can do distributed transactions against multiple databases directly from the servlet container. Think this means you're losing the benefits of connection pooling? Not at all--the servlet container naturally acts as a connection-pooling funnel, particularly when coupled with connection-pooling JDBC drivers. (The same, remarkably enough, can be said for .NET and COM+.)


  4. .NET developers, this benchmark is essentially an indictment of EJB, not of Java. Demand to see a benchmark that doesn't use EJB in the middle, so you can get a fair and unbiased comparison of how the Java bits work against the .NET bits. Be prepared to admit that there's really not a whole lot of difference between the two once we get down to that level.


  5. Everybody, step back for a second and realize that at the end of the day, IT DOESN'T MATTER. This benchmark only proves that .NET turned out to be faster for this application in this environment for this particular test. Is your application precisely similar, in both architecture and implementation, to the Pet Store? It's a hideous leap of logic to assume that "because the .NET implementation of the Pet Store runs faster in the lab, my online human resources system will run faster in our data center"; once the code diverges at all from the benchmark, entirely new variables come into question that can seriously affect the results.



To me, the verdict is clear: run your own benchmarks, form your own conclusions. It would be nice if we could find somebody who's completely neutral to run a benchmark, but what company exists today, working in both spaces (so as to provide equal degrees of expertise) that doesn't have ties to Microsoft or Sun/IBM/Oracle/etc? Let's forget the benchmarks, analyze the code to find out why they each performed the way they did, and apply the lessons to our own systems.




How could the J2EE implementation be better designed? How many of you in one camp have looked at what's recommended as best practices in the other?


19 Comments

jbond
2002-11-05 01:03:47
LAMP
I would love to see the Pet Store app written for a LAMP platform and benchmarked against .Net and Java. It might not be "Industrial strength", as easy to maintain, and do all sorts of nasty things like mix app and presentation layers but I have a sneaking suspicion it would run faster.
rwinston
2002-11-05 07:27:38
Good points
Hear hear. At the end of the day, all zealotry and pointless flaming aside, its only people like Ted, who have an extensive knowledge of the technologies 'on both sides of the fence', aho can make informed decisions. I think when it comes to FUD, bigotry, and downright erm, untruths, you need look no further than Sun for some shining examples. I'm a Java/J2EE fan, I'm also a fan of quite a few of Microsoft's technologies. I can't comment on .NET, as I know next to nothing about it at the moment, but I'm sure I will evaluate it at some point. The point I'm making is that as developers, we need to be informed, impartial, and willing to constructively recognize deficencies in our chosen technologies. To quote:


"....would somebody *PLEASE* step up, write the PetStore the way the community thinks it should be written (using stored procs, no EJB, whatever), so we can kill that original implementation and be done with it? If it's not written the way it should be, then Sun ....deserve what they get when they get crucified in benchmarks."


PS unlike Ted, I am a HUGE fan of Open Source. I however cannot understand the slavering zealotry which plagues sections of the Open Source community.


rwinston
2002-11-05 07:29:19
LAMP
Interesting! I have a sneaking desire to see it implemented on a mod_perl platform, with an emphasis on code cleanliness and reusability. Hmmm....future project maybe :-)
alexmoffat
2002-11-05 10:11:08
Take a look at JPetStore
See http://www.ibatis.com/jpetstore/jpetstore.html for something that's much more comparable to .NET version of PetStore. The paper on the site also has many interesting views on potential problems (can you say HTML embedded in the database) with .NET. Hopefully we'll soon see some performance figures for this implementation. Well worth looking at.
perrin
2002-11-05 12:17:00
LAMP
There's no need to do a sloppy unmaintainable version. The best thing would be to write a really great version in Perl using CPAN modules and solid OO design. It would probably still outperform the Java version.
anonymous2
2002-11-05 13:00:01
Except it is not a fair comparison...
Everything stated is true and has merit....but only if the comparison was truly equal.


It was not.


The latest beta version of .Net was compared to two outdated version of J2EE servers supporting an older version of the J2EE standards. So, .Net got to use all the latest "optimizations" while J2EE could not (no object caching, no CMP2.x etc).


So who do you think would win such a benchmark test?


This alone makes all your points completely moot....The test was not fair.


End of story.


We still do not know if .Net is a better performer than J2EE. We know that a new version of one is better than an old version of another....Duh!


It like saying a P4 2.2 is faster than an Anthon 800....duh!



Mike

anonymous2
2002-11-05 14:17:16
It does matter.
Hi Ted,


Thanks for a nice, objective weblog post. Java is my language of choice for now (though I'm not a fan of EJB) but I try not to get "religious" about it--I definitely recognize the merits of C#/.NET. I do have to disagree with one important statement, though.


]] Everybody, step back for a second and realize that at the end of the day, IT DOESN'T MATTER. This benchmark only proves that .NET turned out to be faster for this application in this environment for this particular test. [[


Smart engineers already realize these kinds of benchmarks don't matter. Non-technical managers and less-smart or less-careful engineers think they do matter. Plenty of good technology has failed in the marketplace because of FUD and marketing.

anonymous2
2002-11-05 17:46:20
do you believe?
Q: Do you honestly believe that Microsoft wrote a specialized version of the runtime just for this benchmark?


A: Yes. There was more than one new 1.1 build that addressed specific performance issues that were found in the course of the testing.

anonymous2
2002-11-05 19:02:46
Great comments from the Author
I thought is was a well written and balanced article.


I too design & code in both environments. I really like both .NET and Java/J2EE.


I quite like coarse grained Stateless session EJBs, but I definitely hate Entity EJBs.


email me on pb1@bigfoot.com to discuss further.


cheers
Peter

anonymous2
2002-11-05 22:52:50
TMC != idiots?
Old app servers + Old Petstore != beta.Not


It is not that TMC was so stupid to use BMP when
CMP src was available:
http://java.sun.com/blueprints/code/jps13/src/index.html
but there BMP JDBC was so awful.Did this CRUD come from the original Petstore?: http://dreambean.com/petstore.html#bugs


They didnt use PreparedStatements:
sb.append(insertLineItemInsert);
lineItem = (LineItem) it.next();
if (lineItem == null)
throw new OrderDAOAppException("null");
int qty = lineItem.getQty();
String itemNo =lineItem.getItemNo().trim();


sb.append(lineItem.getLineNo());
sb.append(",'");
sb.append(itemNo);
sb.append("',");
sb.append(qty);
sb.append(",");
sb.append(lineItem.getUnitPrice());
sb.append(")");
queryStr = sb.toString();
ordStmt.addBatch(queryStr);
sb.delete(0, sb.length());
if(invROHome==null) { invROHome=EJBUtil.getInventoryROHome();
}


sb.append(updateInventoryUpdate);
sb.append(qty);
sb.append(" WHERE itemid = '");
sb.append(itemNo);
sb.append("'");
queryStr = sb.toString();
invStmt.addBatch(queryStr);
InventoryRO roBean = invROHome.findByPrimaryKey(itemNo);
roBean.getMutableDetails().reduceQty(qty);
sb.delete(0, sb.length());



anonymous2
2002-11-05 23:08:43
TMC Magic
"EJB doesn't magically give you performance and scalability"


Magic + Old Pet Store + BMP + Old AppServers == .Net?


I wish the rest of your article could offer more
insight.

anonymous2
2002-11-06 04:44:33
Senhttp://www.oreillynet.comseless.
A benchmark is senseless for one reason:


.NET guys think using SQL tied to objects and stored procedures are good. This makes the application faster but less portable. The .NET guys say "hu, why should I write protable code?". J2EE guys think thats bad and prefer portable and maintainable code over raw performance.


I assume, because both technologies use the same basics (JIT etc.) they will have roughtly the same performance, especially with heavy HTML caching enabled. To run a performance benchmark, just rewrite the .NET PetStore in Java, which is pointless to the Java guys because they say: "I would never write such unmaintainable code." Can Java be faster (on reads) ? Perhaps with CMP strategies (read ahead, multi-fetch, deep-fetch). Perhaps not. Can .NET scale as J2EE ? Probably. Is it as easy to cluster a .NET application as in J2EE ?


What's funny is how TMC shot themselves in the foot. I laughed out loud when I read they have a LOC and have classes and methods they don't use. And .NET programmers call this benchmark realistic. Thats funny and with a laugh I might be more productive, this is the only thing that counts.

anonymous2
2002-11-06 06:22:17
Broken Link / Lost Respect
Ted Neward: "First and foremost, I want to point out that I deeply respect the pure cojones of The Middleware Company (TMC) in producing this"


But you loose (self?)respect for defendinding the
TMC/MS benchmark.


Rickard Oberg uses only facts like the age of the PetStores(): (http://dreambean.com/petstore.html)


"It should be noted that the J2EE PetStore used in this benchmark report is based on Suns Java Pet Store 1.1.2, which is over two years old. The most recent version (1.3.1) uses CMP, local interfaces, and caching. If the J2EE PetStore used in this report is supposedly "fully optimized for performance", why was it based on a very old version of the PetStore?"


Like everything else (but grammer)in your article
Your link to Rickard Oberg page is broken.
Rickard Oberg is found here:
http://dreambean.com/petstore.html

anonymous2
2002-11-06 10:56:52
When were you born Moron? - After Watergate I suppose
Ted Neward writes >Dion writes that "M$ could even tweak their .NET core! Do you think you can download that .NET runtime right now?" Dion, I hate to say this, but that's a spurious argument and a foolish one to rely on. Do you honestly believe that Microsoft wrote a specialized version of the runtime just for this benchmark?


I remember Sun had learned their lesson:
http://www.cis.strath.ac.uk/~dunc/cdrom/archives/ay1997/teaching/cad/related/sunjavabenchmark.html

jonogden
2002-11-07 05:20:01
Excellent Points, Gutsy Author
Too bad so many 'true believers' out there see fit to attack the motives and the intelligence of anyone who dares to question the perfection of Java. They are the language's own worst enemy (not MSFT) since they will refuse to believe they have been out-done until the work dries up.
jonogden
2002-11-07 06:00:28
Say What?
>It might not be "Industrial strength", as easy to maintain, and do all sorts of nasty things like mix app and presentation layers but I have a sneaking suspicion it would run faster. <


That attitude is wrong, regardless of the language.

anonymous2
2002-11-10 12:07:31
Disclosure? Can TMC be trusted?
Question1: Did TMC approach the J2EE vendors for participation in the production of the report?


November 8 TMC finally answers: "We did not approach the J2EE vendors."


2)Why wasnt this information in the report?
3)Why wasnt this information in the FAQ?


http://www.sys-con.com/java/article.cfm?id=1739


Question4: Can you be trusted?

jonogden
2002-11-13 08:59:46
do you believe?
>Q: Do you honestly believe that Microsoft wrote a specialized version of the runtime just for this benchmark?


A: Yes. There was more than one new 1.1 build that addressed specific performance issues that were found in the course of the testing.<


You forgot to start with "Once upon a time," or end with "and they lived happily Java after."

musnat
2004-02-18 01:10:00
Java seem to be losing
More and more I realize that zealots on the Java side are trying to hide the facts from people. Thanks to them it become easier for me to decide on which language to use. The article is extremely objective. So when an objective person says that on the java side people are stupid and idiot and they don't even look at the facts, you realize that java will not be as popular as asp.net. I know both technologies quite well, initially I knew a lot about java, I recently started to learn asp.net and it is definitely a wonderful technology. The only problem that I can cite with asp.net is that you probably have to buy some os from Microsoft, but that's not a big issue, since if you want support you are going to buy something from a company anyway. In that regard Microsoft's solution is much cheaper.