Is a Java macro language enough?

by Sam Griffith

Related link: http://blogs.tedneward.com/2006/03/12/My+Kingdom+For+A+Good+Macro+Language.aspx




First off let me say it's great to be on the O'Reilly site as a blogger and I hope this first entry is thought provoking. Welcome and I hope you'll subscribe as I plan on being prolific in this blog. I've been doing OO development for almost 20 years now and hopefully I can share an opinion that has value to you. And with introductions over, lets get on with our first entry!



Ted Neward yesterday mentioned in his blog entry My kingdom for a good macro language! some points around the idea of us needing more features for Java, specifically:


  1. A MOP

  2. A better language for generating byte-code for the platform

  3. Macros for Java - Not preprocessor macros, but fully language level aware macros



His points led me to these questions: What do you get if you add a MOP to Java? What will allowing Lisp style Macros do to Java? Is there a better language to generate byte-code?

What do you get if you add a MOP?


Today we have some level of meta capabilities thru the reflection abilities in Java and the several Aspect-Oriented tools out there. (AspectJ, JBoss AOP, Spring AOP and others) But those clearly aren't a MOP. A MOP is about fully exposing the languages core concepts and implementation thus allowing the language to be extended at well defined places and allowing us to provide new capabilities and extensions previously not thought of. All of that while still maintaining the same basic syntax of the language. Meta-objects and Meta-object Protocols. Aspect-Oriented programming tools give us some capabilities to reposition our behavior and to cause it to be called in the correct situations, but fall short of a true MOP. Java itself has class objects (they don't do much, but we have them), we have Dynamic Proxies and we have XDoclet and byte-code modifiers. All this to give us the ability to work with the code and meta-system.

What will allowing Lisp Style Macros allow?


Ted mentions OpenJava as a tool that allowed macros of similar power to Lisp macros to be added to Java. This is very interesting work and you should read the papers on that site for a good understanding of what they were working towards. There is a point in the paper OpenJava: A Class-Based Macro System for Java, that I disagree with concerning the power of the Lisp macros to be customized for specific types, but lets just say that you can get that same effect using a combination of macro system with and on the Lisp MOP. As for other macro systems for Java, I don't think we'll see one worth working with without a MOP of some kind. If you read this paper, they basically describes a simple MOP that is able to manipulate a program at compile time and then save that meta-data in a separate set of byte-code as the JVM doesn't need that extra info to run. But notice, that they have to separate that description and also that they save some of that data in little strings in the classes. Ted doesn't mention whether he wants the MOP data around at run-time. But for us suffice to say macros are about allowing us to have code that manipulates code that we have and either modifies it or creates new code.

Is there a better language to generate byte-code?


On the better language for generating byte-code he doesn't really reference anything besides the work Microsoft is doing on the CLR and CLI stuff for .NET. He does mention them as more language friendly but does mention that it mostly has the same drawbacks as the Java platform. I conjecture that the byte-codes lack of full power is part of the issue. This is why SUN is going to add a new byte-code to support scripting languages.
There was a push both internally and externally early on in Java's history before 1.0 was even out, to extend the byte-code in concert with Parc-Place Systems (an old Smalltalk vendor - some of who's employees went to SUN to work on Java) to support more of this dynamic capability including MOPs, but choose not too.


So what do we get really if we add all that?


I conjecture one step "Back to the Future!". Ted mentions one of the single most powerful languages ever invented right at the start. Lisp. He then mentions one of it's most powerful features. The code as data - data as code idea. Lisp almost from the start had this idea that you can manipulate the program. It's 40 years old in the Lisp incarnation. Lisp 1.5 was based on S-expressions (denoted in Lisp by those parentheses you may hate). The big idea being that the language itself was self describing. The compile parser and run-time parser deal with the same representation, as do you! Love the S-expression! That's part of it's power. The MOP of Lisp really is only for CLOS, not for the Lisp language, because the Lisp language already has a great meta-level capability in it's READER and it's macro system which Ted alludes too. But can we get as much power and simplicity and expressiveness by adding a MOP and macro system to Java?

I dare say we can not! But we may get closer...

Lets look at Ruby as he mentions it in his blog. He shows how Ruby using it's meta-level features allows what appear to be new keywords to be added to the language. These are attr_reader and attr_writter. But he then calls them terse. You can't have it both ways.

If you want full MOPs and macros, then calling the Ruby terse is negating your own request. All macros will be terse in that situation. This is akin to the argument we saw a few months ago about minimal API's versus humane API's. People have no problem creating all kinds of utility helper classes and methods, but doing it thru some meta-level operation is somehow terse. The point clearly defeats itself. I guess something that looks like a function call is "OK", but something that looks like a language extension is not. Well, put your minds at ease because the Ruby that is mentioned is exactly that! The Ruby is just some method calls that are standard, but are implemented via the meta-level layer in Ruby and provided via that meta layer.

Ruby is not typed, making code generation easier. The thing to notice though is that in the Ruby code (you'll have to follow the link to the Person class he mentions to see the full code), method calls can happen anywhere! Those attr_* methods are called right at the top. There's really no special place for them. In Java things are not so simple! Lisp has the same thing going for it. Function calls can be done anywhere! This has to do with the fact that the language definitions for Ruby and Lisp are simpler than the one for Java. The fact is they have less syntax and less syntax actually provides more power!

While Ruby and Lisp have less syntax they can both appear to add new capabilities to the language itself, not just a new library. Today people are doing that with the tools mentioned above and are now starting to do it with the new Annotations added in Java 5. Aren't those terse too? They certainly added new syntax and more special rules about when and how they work, etc. Compare that to just a simple method call. Seems the method call is easier to understand, debug, put in to the system in the first place, more pure, etc.

Why is that it seems more pure for Ruby and Lisp? Because the languages were built with the idea that a rich meta system was important. Java's concerns were different and coming back and adding something as intrinsic as a MOP and macros is going to be difficult to do. And adding more syntax creates more special cases and rules etc. that makes your MOP much harder and the macro system more complicated. Not only that, but the code manipulator will me more complicated too! But does that matter to us if we get the MOP and the macros? I think it does and here's why: If we expect to use that same MOP and macro system, then we want it to be nice and clean and easy to use and understand and without all kinds of exceptions and special cases. Think Java Generics. Lots os special syntax, special cases, etc. Java is now much harder to read. Terse if you will!

So the Java world is now going in lots of directions it seems to me. 1) We have new features being added to the language at a glacial speed. 2) We have new wants and desires being expressed for more power at the language level itself for programmers. 3) We have the want for simplicity, readability, etc. 4) We have the people who want only one way to do something and the people who want more than one way to do some things (minimal vs. humane).

Underlying all that is a language that was created for remote controls, set-top boxes and interactive television! A language with a syntax to appeal to C and C++ coders, networking and security being the prime concerns. A language whose least concern was a meta-level self description!

Instead of adding a MOP etc, we've seen Java add what the C++ community added. Java has taken a similar track to that community and it's not surprising. Most of the converts were old C++ people. Not people who used and programmed highly meta-level systems. But now we as programmers have been influenced by the more dynamic meta-level systems community and the power of those systems that we lust for.

We've used all the patterns to try and get close to what we really want, we've added Aspects, we've used refection and dynamic proxies and byte-code manipulation..... but what do we really want....? No really, what is it we really want?

What do we want?


To me its as clear as the nose on my face... we want a totally new language that we don't have to learn, that gives us all the power we already have, can use our current libraries and provides us all the power of yester-year and still is flexible enough to add new features at any time!!!

And no I'm not kidding! To add a MOP and macro system and a different code generator, we are basically talking about a new language. Maybe a familiar syntax will still be kept, but it will be a new language. Much as Java 5 really is a new language. Once you start using it's features, you program differently, you solve your problems differently.... it is a new language. It's changes are that significant. No set of Java features before I would say that about, but Java 5 was. And it's all due to the Annotations and Generics. Terse is being kind here.

So if we're going for a new language why don't we start with the meta-level stuff being in from the beginning along with security, networking, multiple language friendliness, a better byte-code set, a MOP, macros, and all the other stuff we're hearing people want now!

Or maybe we're not going to have a new language.... because we don't want to leave behind our frameworks!... How sad a choice that would be. A new language would empower us to make much better frameworks with out the limitations and the code bloat we see today to just so that we can simulate those powerful features we want.

I'd also suggest that we also add optional typing to the mix. We have a perfect example of how to do it. Again it's Lisp. Lisp itself lets you specify the type of a variable or object optionally, it also lets you specify a safety level and a optimization level allowing you full control over both type safety, runtime safety as well as how much it will try and optimize around that variable to gain speed/safety. Lets use that so that we can have a language that is both a fully generalize language good for full systems development as well as for scripting. And let that safety be linked in to the secure environment, etc.

We have as programmers and developers a great example of the things we seem to be clamoring for Java to add and it is Lisp. Code is Data/Data is code, full macro system, fully controllable typing system, customizable parser (the reader), a MOP, self aware system, highly efficient code generation, able to operate in compiled or interpreted mode, simple syntax, etc. (You'll find all these in Smalltalk as well)

Ruby is a nice partitioning of some of those features, and you can see how hot that community has gotten over it. Java caused the same stir in it's day with garbage collection, networking libraries standard, threads standard. But no language right now has all that we seem to want. Not even the .NET CLR/CLI set of tools and languages.

We need a new language..... or we need to "Go Back to the Future". I myself, think going back to the future by building on Lisp or Smalltalk is the shorter path, but I know that the syntax is a big issue. Lisp has always been a ball of mud in that it could take on anything and still look like Lisp even though it now had all kinds of new capabilities. CLOS being a perfect example. But because I'm enlightened and experienced and I've seen history over these almost 20 years I've been programming, I know that the majority is what makes a new language stick and for that we will need a "New Language"!

In closing, any time you want to dump on Lisp or Smalltalk or some of those "other" languages, look closely at them again. Then create "Hello, World" in Java and Swing (9 Meg just to print "hello") and then the other language. Then look at what all you need to run a J2EE app server vs. something similar in one of those other languages, look at there web frameworks, etc. Then compare how humane there libraries are verses Java's. Finally, think that some of them can do all the same stuff we do in Java in about 1/8th the RAM and code size and for the most part they are just as stable and scale just as well as the Java tools do. Then think why? Can we get to that point with a "New Language" You bet we can!

Having said all that, Java isn't really bad... We are as programmers growing to want more power than ever before and it's leading us the same place it lead those researchers all those years ago. Kinda nice to know that isn't it? It's also kinda nice to know that we have a community that is pushing the envelope and pushing the language vendor to add new power all the time.

Here's hoping I've stirred the pot a bit....

Till next time - Thanks for reading....

Please let me know what you thought and your viewpoint on MOPS, macros and whether you think we need a new language.


10 Comments

errorter
2006-03-16 18:28:25
cool stuff
how about groovy?
errorter
2006-03-16 18:32:26
Groovy: The Sleeping Giant
Richard Monson-Haefel's Groovy: The Sleeping Giant


"Within the next five years most organizations will use dynamic programming languages in some enterprise development.


The future seems obvious: Dynamic languages are growing in popularity and their productivity and broad applicability cannot be ignored. The Java platform is supported by a huge ecosystem of 4 million-plus developers and thousands of tools and APIs. If any dynamic language is to be successful it has to (a) be standardized (b) appeal to Java developers (c) be fairly easy to learn, and (d) leverage the existing Java ecosystem. There is only one dynamic language that meets those needs and that's Groovy.

"

staypufd
2006-03-16 21:21:06
what about Groovy
I think Groovy is a great thing for Java, but it still does not give Java itself a macro language or a MOP. Groovy I think can turn out to be a big winner especially if SUN makes the new byte-code powerful enough for Groovy to take advantage of to get some more performance....
albert bulawa
2007-01-06 15:13:18
Beware of the dragons, er, macros. And such.


To say it shortly: Java is the single most used programming language on earth exactly because it doesn't have macros, MOP and other stuff, like dynamic typing, which, while looking attractive kill the very hope of code maintenance in the long term.


Java didn't need annotations or AOP to gain the broadest acceptance a language ever has, and, in fact if it had them at 1.2, it wouldn't get so broadly accepted. It should be significant that many, if not most, real enterprise projects decide to develop with 1.4.2 (even those started lately) or even so-called Foundation Environment.


Unfortunately Sun has gone wild into competition with MS and started adding all those silly features just because MS has them. The only feature worth adding really was generics, but they should have restricted it so it can't do any damn template-like development, just adds more compile-time type safety. Guess they could not resist.


So instead of adding bells and whistles, they should have started deprecating some most dubious features of Java, like anonymous classes to name just one.


Strange they are going for acceptance of open source weenies, not serious corporate developers who are still using 1.4 anyway.

albert bulawa
2007-01-06 15:20:57
One thing more: why projects like JSE or OpenJava are quietly dying and even AOP gets little acceptance in the real enterprise world? I'll tell you why: because they are simply unwanted there. Serious people don't need or want a "powerful extensible language", they do not use Java for "rapid development" (read: "write and never touch again"), they want a stable and mature language with no bullshit to let them write their code in nine instead of ten weeks and then have a major PITA when they need to change a thing.


If they want rapid development, they will take Ruby or Python anyway, Java won't be any attractive for "rapid developers" (read: "cowboy coders") and still be Java. So just don't do it.


2007-01-06 17:58:34
what's a MOP?
Andrew
2007-01-07 18:25:35
The main reason why a MOP or macro system for Java will never become mainstream is that its syntax is too complicated and it is too static.


Take a look at lisp. The reason why macros became so popular is exactly as Sam said: code as data. But there's another reason---the concrete syntax of lisp is extremely simple.


The problem with the JSE is that those who write macros for it must also have some understanding of the underlying parser technology. This makes the act of writing so much more difficult.


Whereas a full-blown macro system like defmacros in common lisp is never going to be made in Java, I see annotations as a huge step forward. There is a lot of language extension that can be done through annotations and I think this is going to be a big bonus for Java developers going forward.

albert bulawa
2007-01-10 04:50:31
Language extensions are bad. In fact, they preclude ANY serious enterprise development. Of course, you can even create a nice success story using a dynamic language, something like Viaweb for expample. But this has a downside too - companies that buy out such successful startups more often than not end up spending even more money on rewriting newly bought code than on the startup itself. (Of course, this is not a problem of the startup founders anymore then :)).


You sure can make lisp look more like basic or cobol than lisp itself, or make ruby look smalltalkish, but this "power" is in fact the most significant reason of those languages' defeat. Java can't be really made look like anything but itself - there is no "power" to abuse, so people instead of spending hours on thinking how to define an operator to do what they think it should do, just write readable code using existing operators which is more than enough. This is one of main reasons of Java's success and it should stay so.


Annotations are bad. They let you do nasty things like modifying behavior of so-far well-understood code. In fact, all techniques using such "processing", or, even more atrocious byte-code instrumenting should be avoided at all cost. Too bad Sun seems to embrace them, instead of banning.


2007-02-17 07:01:42
debby-68 :)
SlowStrider
2007-11-25 15:44:37
I would really like to see some powerful macro language in Java. Like Albert Bulawa said, this can certainly be misused, but in my view having a lot of so-called boilerplate code is not good.


Currently it's not easy to minimize boilerplate code, even with annotations. For example with the new Java Persistence API, which can be used to load and save your objects in a relational database, you still have to write a constructor, getters, setters, equals() method, hashcode(), toString() while all of this can be automatically generated for most use cases. I would simply like to write something like


@Entity
public class Product {
@Id @Property Integer id;
@Property String name;
@Property BigDecimal price;
// Read only.
@Property(setter=false) int currentStock;


public int someBusinessMethod() {
....
}
}


for my entities. I tried to make a code generation processor for this with the new Java 6 API, but found out it's not going to work. I can write out the getters/setters, but I don't have access to the source code to write out any other user defined business methods.