Grok, Repoze, and WSGI beg the question, are webframeworks yesterday's news?

by Noah Gift

UPDATE: If you want see an "application", Plone 3.0, run under WSGI with a "framework", Grok, you can check out this demo on the Repoze Site. This is one of the more exciting things I have seen all year. So what is this called, a framework, a product, WSGI...you tell me

Our local Atlanta, Python User's Group, PyAtl, is collaborating with the local, Plone User's Group, to build a Plone 3.0 site. I have been attending the Atlanta Plone meetings recently at ifPeople, and we had a mini-sprint last weekend where we got a demo site running behind a, newish, forward proxy server called Varnish.



On Wed, I attended an Atlanta Plone meeting, where Brandon Rhodes, gave a presentation on Grok. Brandon has kindly agreed to post his presentation on his blog, which he did in KeyJnote, which is written in Python, of course.

In Brandon's talk, he mentioned that Grok, which is built on top of Zope 3, is less of a framework and more of a component based architecture. In looking at the Grok website, I liked two things in particular:

1. Grok offers a lot of building blocks for your web application.
2. Grok is informed by a lot of hard-earned wisdom.

One of the reasons why web application developers should look at Grok, and Zope 3, in particular, is the second point. Failure over a long period of time, leads to experience. When people learn from their failures, it then turns into wisdom. The Python world has much to learn from Zope, and that is why I am especially interested in Grok, and Zope 3. Remember Zope Corporation was founded in 1995, a lifetime in software development.

All of this background information brings us to the meat of the discussion, which is Repoze, WSGI, and the eventual death of webframeworks as they exist in their current state. I for one, welcome our new WSGI, overlords. WSGI, and products like Repoze, will make webapplication frameworks much less important, and component based architectures, much more important.

One of the massive, unaddressed until WSGI, problems of standalone, web frameworks, is that, ultimately, they could never win in the long term. If a web framework has a frenzied pace of development, then the API, and documentation will never be accurate. Additionally, web frameworks, like Django, can lock you into a specific set of "tightly coupled" components, in which your ability to use other components, like SQLAlchemy, is not possible.

With WSGI, these issues become much like relevant, as something like Repoze, can use the WSGI spec, to do things like, "allow for systems that participate in a WSGI pipeline to make use of the existing two-phase commit transaction management provided by the ZODB transaction package." WSGI is clearly the future for Python web development, and in hindsight, it even seems silly that such an emphasis has been placed on web frameworks as a one stop solution.

Clearly the better solution, is a loose set of components that work together in a unified fashion under one specification, much like how the standard library of Python itself works. I suppose, to further illustrate the point, can you imagine how irritating it would be if you used, say, the tarfile module, and that meant that you could not use the zip module? That would clearly be unpythonic, and this is why WSGI is the new buzzword. By using the WSGI spec, and optionally, something like Repoze, combined with deliverance,it is possible to run multiple web applications, running under one URL, all with the same "skinning", or look and feel. This smells like victory, for Python web application developers.

Note update: I want to make it clear that I am not slamming any web framework, including Django, my point is that ANY web framework can run independently inside of something like Repoze, coupled with Deliverence, making even a full framework an actual component and in fact eliminating problems like one architecture is tied to a specific ORM, or templating engine etc.. My apologizes if this point was not clear in the original post. What this does is take the emphasis, and pressure, off of a site being one specific component architecture, or framework. A site won't be a "Django site", or a "Turbogears" site, it will be a "website".

(Thanks to Derek Richardson, for being such a strong advocate of Repoze, WSGI, and deliverence, as much of this post is based on things I have heard for the first time from him. On that note, it is looking more and more likely that we have the authors of Repoze speak at PyAtl in December, 2007. If you are interesting in helping to sponsor this event, please contact me or Derek Richardson.)

[1] Update: Update May 17th, 2008: I would refer anyone who is angry about my opinion to first read this essay by Paul Graham: How To Disagree. Note, I privately attempted to contact Jacob Kaplan Moss to prevent him from leaving the drunk post, but I received no response.

[2] Update: A particularly juvenile response to this post seems oddly relevant to this article from the New York Times. I don't see any distinction between his behavior, and the protagonist in the Times article, "Weev". I attempted to contact the poster privately via email to resolve the issues, and was confronted with a similarly juvenile response. I have the correspondence available for interested parties.

13 Comments

Jacob Kaplan-Moss
2007-11-10 21:22:43
OK, I've had a bit to drink tonight, so apoligies up front for not being more diplomatic here. But this:


"""
[..] web frameworks, like Django, can lock you into a specific set of “tightly coupled” components, in which your ability to use other components, like SQLAlchemy, is not possible.
"""


is complete, 100% pure, bullshit. Smelly, nasty, disgusting bullshit.


Here's how to use SQLAlchemy in Django: "import sqlalchemy". How, exactly, could any library "forbid" the use of another, anyway? Is that even technically possible?


Next time you want to shit all over someone else's hard work, at leaqst get your fucking facts right.


Next time you decide to trash someone else's code

Jonathan
2007-11-10 22:40:49
The thing that is wrong with web frameworks is they are frameworks and not components; wsgi fixes that. I likes plug-able architectures, dat's why I likes unix. Then again, I'm drunk, too.
Noah Gift
2007-11-10 22:49:35
Jacob/Let me preface by using the quote,"I aim to be less wrong then yesterday." You know much more than I do about web development, so there may be things I don't understand, or am missing. I will gladly correct anything that is wrong in my post, if you point it out.


On the other hand, I have been frustrated that I can't use cutting edge libraries, like SQLAlchemy, and have it work perfectly with say the Django admin interface. I am also frustrated that Turbogears can't use the Django admin interface too. This is what I mean by a library "forbidding" me to use another. If it means I have to write the code to glue things together, it just ain't gonna happen, due to a combination of skill and time. That is why the standard library for Python is so great. You don't have to write things like shutil or tarfile, you just import them.


I do think WSGI and things like Repoze are a good thing, as they take the stack up one level and the web apps become a component. No offense was intended toward Django in particular, as I think all web frameworks should be broken into components or become parts of a bigger stack. I am glad this is becoming more of a reality with WSGI. I think I heard something about the last Turbogears sprint maybe heading that direction...but I am not sure. I know Pylons also seems to be heading that direction.

Jacob Kaplan-Moss
2007-11-11 09:14:26
Wow, I didn't realize I looked quite so incoherent last night. I can assure you it wasn't the liquor talking, though: I'm perfectly capable of making myself look like an ass without chemical assistance.


I appreciate what Noah's trying to do here: WSGI is indeed brilliant (and has done wonders for Python's status in the web stack); deliverance and Repoze are similarly worth paying attention to. But none of that makes this "lock-in" FUD any easier to swallow. Broad statements about how Django prevents use of other libraries doesn't do anything to advance interop; it simply makes people mad (c.f. my comment above :)


To take one example: there's a couple of bugs in Django's WSGI handling. They aren't serious, but they *are* going to be tricky to solve in way that doesn't inconvenience the average developer who thinks WSGI is something you drink. People bitten by these bugs can choose to respond in two ways: by offering solutions (or testing proposed ones), or by using them as evidence that Django is opposed to inter-op. Unfortunately, nearly everyone's chosen the later.


I find it very hard to stay motivated when my work's being trashed; those bugs are still unfixed. Good times.

anonymous coward
2007-11-11 09:32:52
I'm kinda new to Python and I've been advocating to be able to use Django at my job for a long time, and have met up with a lot of "but it's not PHP and everybody has to use PHP for the web" crap. When my manager finally agrees to check it out, she googles Python and Django and finds a bunch of silly arguments like this, and I have to do damage control on that.


Can you just calm down about it? I like Django but Zope and Plone don't do it for me, but they seem to work fine for lots of other people. Django doesn't cook your bacon, but you like Plone. That's great! Can't we all just get along? Sheesh!

Noah Gift
2007-11-11 10:04:15
Jacob/It is funny how poor electronic communication still is at communicating subtle points. I recently got really pissed, thankfully in a private email, when someone said he had doubts about the book we are writing on Python For Systems Administration, because we were going to talk about Django. My point was not to say Django locks you in, but that WSGI helps eliminate any FUD. If that didn't come out correctly, then I have failed.


Of course, I have to keep remembering that people actually read my O'Reilly blogs :) It has been a learning experience, and have much to still learn.


AC/I am sorry if my point didn't come across as well as I intended. With WSGI, what framework you use is irrelevant, you can use ALL of them, and pick and choose, what you like best, and make them run under one roof. I suppose I tried to tie too many concepts into one post: Grok/WSGI/DJango/SQLAlchemy/Repoze/Deliverence. Maybe next post, I will only cover 100 topics,not 101 :)



George Jempty
2007-11-12 08:22:30
I've come to the conclusion that *all* web frameworks amount to "premature exacerbation" if the MVC/separation-of-concerns piece isn't essentially as simple as applyInputToModel/applyModelToView; also, if the framework makes you implement any classes with "model" in the name that does not have to do strictly with transferring data across the model/view boundary, and if the framework specially integrates whatsoever with any database-abstraction/ORM system.


That is precisely what my framework that is linked to from this comment does (or doesn't do ;), which, by the way, I ported from PHP to mod_python/psp earlier this summer. I know mine is a radical stance but in support of the concept of "feature amputation" please see the following PDF on Kent Beck's site:
http://www.threeriversinstitute.org/Cutting%20with%20the%20Grain.pdf


A web framework should not be decided upon until after prototyping an application with the applyInputToModel/applyModelToRequest system that can be easily implemented on any OO/web platform.


Sorry I'll get off my soapbox now

Martijn Faassen
2007-11-12 19:23:56
Just found this entry. Thanks for the kind words about Grok!


Upset Django people: Zope has been catching flack about causing lock-in for like, um, like 2 times the period you even existed. We've been working on getting out of this position for many many years. The impression was of course, only partially true (and more true than it ever was for Django, as the original Zope pattern encourages storing code in an object database!)


I'm happy to see the hard work in the Zope community has finally started to pay off: for a change, our Zope work is actually being associated with flexible pluggable things instead of evil monolithic things. It's welcome news.


Django's been getting a lot of positive press. So did Zope back in '99 or so. Good thing we stayed motivated while our work was being trashed. :)

Noah Gift
2007-11-13 19:50:28
Martijn/Interesting to hear your comments. Some of the things that Phillip Eby says about Zope have peaked my curiosity. Maybe the death of web frameworks is what Zope is doing next. I have the Python Cheeseshop in my RSS Reader, and I see massive amounts of Zope components being updated at a furious pace. Wow, it would be cool if everything worked that way.


To quote and, and completely butcher, the lyrics by John Lennon to Imagine:


"Imagine there's no WEB 2.0
It isn't hard to do
Nothing to kill or die for
And no WEBFRAMEWORK too
Imagine all the people
USING COMPONENT BASED ARCHITECTURES AND UPLOADING EGGS TO THE CHEESESHOP
"
Well, I might need to work on the last sentence, not exactly rythmic...


Simon Willison
2007-11-17 06:13:16
Additionally, web frameworks, like Django, can lock you into a specific set of “tightly coupled” components, in which your ability to use other components, like SQLAlchemy, is not possible.


I have to disagree here. "Practise loose coupling" is one of the unofficial slogans for the Django project - it's highlighted in chapter 1 of the official book and it's even printed on some of our t-shirts. The framework was designed to support loose coupling from the very start, precisely because we didn't want to attach ourselves to any one component (what if we later found that our ORM or template system weren't the best option?). I've personally built a major Django project using SQLAlchemy instead of the Django ORM.


The only components of Django that you have to use in every project are the Django request and response objects and the URL mapping system - absolutely everything else can be swapped out at will.


Of course, sticking to the components that Django recommends does provide benefits in that you'll be able to integrate other pre-existing Django applications, such as the admin interface or the various other open source projects that have sprung up around the framework.

Noah Gift
2007-11-18 10:29:15
Simon,


Thank you for taking the time to respond. One thing I have really admired about Django has been the exceptionally good documentation. I think Django has set a gold standard in that respect. I know I would personally like to see official documentation, or books, about Django using other components in a "loosely coupled" way.


My personal opinion is that Django cannot be both, "loosely coupled" and "tightly integrated". Tightly integrated is not necessarily a bad thing either, but I don't think it is fair to try to claim both. I also don't see any activity in the cheeseshop for Django components like I do with the other Python web application frameworks. Again, this is just an observation, but it leads me to the conclusion that Django is a tightly integrated framework. Of course, I like using Django for certain things I have done, for exactly that reason too. The integration of the admin interface with the ORM and the template can make things go very quickly. I don't see why this is a bad thing to admit to. Of course, I am also selfish, and would love to use the admin interface in a Pylons or Turbogears project using SQLAlchemy too. I would also like to use a Toscawidget in Django, but there is another widget system in Django. I guess I want to use any component in any web framework, and I want it to be well documented and easy to do. I know, I demand a lot... :)


For example:


Routes
Toscawidgets
SQLAlchemy
Storm
Dejavu


It would be awesome to see documentation about using these things in Django in a "loosely coupled" way. To close, I like Django, and I like Turbogears, and I like Pylons, and I like Grok and I like Plone. I want it all...is that too much to ask :)

Mike Orr
2007-11-21 13:29:12
I'm still trying to get a handle on what Repoze really does for non-Zope users, but my impression is it's not really a revolution but part of a larger trend. Ian blogged in August that the number of active web frameworks actually shrank this year for the first time in Python history, and the remaining frameworks are becoming more mix-and-match with features. TurboGears2 is being built on top of Pylons in a way that allows hybrid apps, add-ons like ToscaWidgets are being spun off and interact well with FormEncode validators, and even a core feature like @expose has been partially ported to Pylons. Pylons-dev adpoted Paste's new webOb, a standard request/response API that I understand Django is interested in too. Beaker's sessions and caching are available for any framework that wants them. It seems like Repoze merely fits into this trend, making Zope components available to non-Zope applications, if I understand it right.


It looks like quite a breakthrough for Plone though. Plone has been stuck at Zope 2 due to the enormity of upgrading. It looks like they'll get Zope 3 and WSGI in one fell swoop, which makes me much more interested in using it or parts of it. But I have a feeling Plone has Zope dependencies throughout. That's fine if I just have to install some Zope packages and it "plays well" with my other code. But it would dissuade me from using small parts of Plone where the Zope dependencies are larger than the function I'd want to borrow. So that's still an issue.


The other issue is, WSGI is not the end-all of everything. The API is a pain to write to and get right, though WSGI2 promises to help this immensely if it ever gets off the ground. There's the debate over "true" middleware vs application component non-middleware, and whether the latter should use the WSGI API at all. I actually think the question should be reversed: call the latter "middleware" and the former "transparent middleware". Some people have recommended not using WSGI for anything the application depends on, but rather an importable module. While there is certainly overuse of WSGI -- middleware that puts things in the environment that would really be more straightforward as an importable library -- WSGI does allow one to write a generic middleware for diverse uses, which is useful even if each application depends on it.


The quote, “allow for systems that participate in a WSGI pipeline to make use of the existing two-phase commit transaction management provided by the ZODB transaction package.” is the only concrete advantage I've heard of for using Repoze. This assumes the application wants/needs two-phased commits and versioning and an object database. Most non-Zope apps don't need this, so I fail to understand how this is "revolutionary" or will sweep away frameworks as we know them.

Noah Gift
2007-11-21 13:55:42
Mike,


Thanks for the comments. I think part of the revolution is less technical and more psychological. The idea that you don't need to have one web framework, and thus choose, one over the other, is a revolution to me. There are so many people that are just sick of the whole, my framework does this stuff, that they have even left Python for other languages until is "settles".


I think WSGI can, and will change all of this, and reduce a lot of the "pulling my shirt off at high school football game" stuff :)


On a related note, there is going to be a Python Web Development conference this summer in Atlanta, and we would love it, if you showed up. We should have a date nailed by then end of December 07. A big WSGI specific sprint is a no brainer. We can talk more offline.


Also, my good friend Shannon, pointed out to me privately, the best way to learn about a framework is to actually use it :) So this holiday break, I am going to do a bunch web application development, so I can avoid being full of ****, when I am talking about something. We are also having a Repoze Sprint after the guys fly down in December, so that should give me a good idea of what is going on too.


-Noah