The Year in Perl 2005

by chromatic

Related link: http://www.perl.org/

A year is a surprisingly long and short time in the life of a software project. It may go by quickly, but when you review what actually happened, you may find that many things happened.

The theme for Perl in 2005 is one of rebirth. Several projects that looked dormant started again — not just technical, but social. While Perl may have entered the year a little slowly, it's leaving 2005 much stronger.

Stable Releases

Perl 5.8.x pumpking Nicholas Clark announced last year that he intended to change the quarterly stable release schedule to once every four months. Unfortunately, real life intervened. Fortunately, his excellent work in making the 5.8.x series stable and usable continues. He released Perl 5.8.7 at the end of May. Few people have tested release candidates for future releases with their code or their modules — feedback is very important.

Hopefully next year will see more regular releases.

Despite the fact that stable releases bring bugfixes, newer versions of the core modules, and, as always, ever-improving test coverage, some businesses still stick with Perl versions that have gone unmaintained this millennium. Perl 5.004 is ancient, Perl 5.005 is old, and Perl 5.6.1 is creaky. If you're deploying a new installation of Perl, please consider the latest stable 5.8.x release, or if you really need old software, at least Perl 5.6.2.

Upcoming Releases

Perl 5.10 pumpking Hugo van der Sanden, perhaps the kindest man in Perl, passed on the full pumpking to Rafael Garcia-Suarez, another candidate for the title of "the kindest man in Perl". Unfortunately, this means that Hugo is unlikely to tackle the proposed and much-anticipated rewrite of the regular expression engine.

Perl 5.10 will be impressive, though. Proposed new features include:

  • the defined-or operator, ported from Perl 6
  • a smaller memory footprint, with much slimmer internal data structures
  • an improved Switch module with smart matching and no source filtering
  • assertions
  • a lexical $_
  • variable names in uninitialized value warnings
  • a new pragma, feature, which allows you to use features from Perl 6
  • more tests
  • updated core modules
  • continual improvements to the internals, thanks to Ponie
  • regexp trie optimizations (especially thanks to Yves Orton)
  • a somewhat cleaner and safer API (thanks to Andy Lester and cleanup patches)

New features planned for inclusion are:

  • integrated Module::Build (and perhaps CPANPLUS)
  • proper lexical pragmas
  • state variables, as found in Perl 6

There's no timetable yet on the 5.10 release, but there's a sense that it's getting close to the time to think about such things. Pumpking Rafael Garcia-Suarez plans to release the 5.9.3 development version just after the start of the year.

Notable Modules

Adam Kennedy continued the Australian domination of the weird on the CPAN, releasing PPI 1.0 — an independent Perl parser that can parse Perl without running it. PPI makes a lot of things possible — better IDE integration, refactoring, automated style guides, et cetera. Still, only perl can still parse every Perl file especially when you get into areas where the other crazy Australian hacker has been.

Which other crazy Australian hacker? Damian Conway. He did not rest this year either. In conjunction with the release of his new book, Perl Best Practices, he released several modules to help you write code well. One of the most interesting is Class::Std, which itself touched off a few debates on Perl Monks about the use, abuse, value, and implementation of inside-out objects. Though Class::Std itself may not be the ultimate and final way to write clean, maintainable, and performant OO code in Perl 5, it does show why Damian's a master Perl hacker. Would you have considered inside-out objects if he hadn't written it?

The web world has seen some work too. Simon Cozens's Maypole has new maintainership and new evolution. The spinoff project Catalyst has a lot of activity, excitement, and open job requests. Of course, CGI::Application is still vital and valuable, and Randal Schwartz's CGI::Prototype takes a new approach.

There's no standout Ruby on Rails killer yet, as much as Catalyst would like to claim that title. Perhaps the Jifty project from Jesse Vincent and Best Practical will be the right blend of database magic, cleanliness, and intelligent defaults. Certainly its use of continuations is very compelling.

Curtis "Ovid" Poe's Class::Trait is also compelling, bringing the designish goodness of Perl 6 roles to Perl 5 in a very Perl 5 way.

The venerable object-relational mapper Class::DBI has some healthy competition in the form of DBIx::Class and Rose::DB::Object. The Class::DBI::Sweet extension is a worthwhile complement. Tangram is still around too... and Jifty::DBI might possibly make Rails's ActiveRecord look verbose and clunky, at least when it gets a little more documentation.

Perl 6

2005 was yet another year for Perl 6. "When will it be out?" you ask. "Sooner, if you help" the designers and implementers and true believers still answer. After struggling for a few months, sooner really is, well, sooner.

Allison Randal, president of the Perl Foundation and project manager of Perl 6, stepped back from both positions to devote more time to coding. Jesse Vincent replaced her as the day-to-day project manager of Perl 6 and now spends his time attempting to elicit progress reports from developers and asking both "What's blocking you?" and "Are you having fun?"

So far, it's working ever better.

Parrot Reborn

Parrot hit a low point late last year, with several unanswered design decisions, a lack of direction, and arguments on the list replacing actual working code. Citing a lack of motivation and time, longtime designer Dan Sugalski stepped down and Perl 5.004 pumpking emeritus Chip Salzenberg surprised many people by volunteering as a replacement.

Though Chip has spent several months fighting a persistent (and, in the opinion of your editor, unfair) legal battle (see GeeksUnite), he's also reviewed several pending design decisions and started to meet the mini-milestones on the way to completing Parrot's design and implementation. For example, calling conventions have improved (yet again, but this time they fixed the sticky continuation issue) and the lexical implementation works.

Thanks and well-wishes go to Dan for several years of tireless and often thankless work. Thanks and well-wishes also go to Chip and Parrot Pumpking Leo Toetsch for their existing service and in anticipation of even more good work.

In semi-related news, the Dutch foundation NLnet generously sponsored two milestones of Parrot development, enabling both Chip and Leo to devote paid time to the project. These two milestones include nine critical subsystems. The developers finished two in 2005, leaving seven to go. (Thanks to Mark Overmeer for corrections here.)

Pugs

No one could have known at this point last year that a working Perl 6 prototype implementation was only a couple of months away, for various "in one sense or another" definitions of the word "working". The Taiwanese world-traveler Audrey "Autrijus" Tang sparked a near-revolution in two ways. First, by making actual Perl 6 code run with a few weeks of hacking and second, by demonstrating a guided near-anarchy development style that brought new energy, new questions and ideas, and, more importantly, new contributors to the Perl 6 development process.

You may have thought that Perl 6 would only run atop Parrot, but would you believe that it may run on JavaScript, Java, and even Perl 5? Back in February 2005, who could have guessed that you can embed both Parrot and Perl 5 in Pugs, so as to run existing CPAN modules with Perl 6 code?

The project has been exceedingly liberal in granting commit bits (to the point of answering all bug reports and feature requests not with "patches welcome" but "if you check in a test, someone will add the feature"), even to the point of offering commit bits to Python designer Guido van Rossum (who kindly declined).

The roaring velocity of Pugs did slow somewhat after the normal conference season, but lately work has started again on the infrastructural features necessary not only to produce Parrot-compatible code but to support all of the amazingly powerful but conceptually complex features of the Perl 6 object system. Oh, and it looks very likely that the Pugs solution to connecting to Parrot will help the rest of the Parrot and Perl 6 compiler tools evolve and grow too.

Synopses

In the beginning, there were Perl 6 mailing lists and Requests For Comments. Then Larry began synthesizing and producing long, weighty Apocalypses. Damian followed, extracting the practical wisdom from the Apocalypses to produce practical code examples in Exegeses.

When it's too much work to build thirty pretty great pyramids to show how to build a Really Great Pyramid, pass around blueprints instead.

Now the Apocalypses and Exegeses have become historical curiosities (for various comedic misspellings of the word "hysterical"). The Perl 6 Synopses are the new Apocalypses, except shorter, more accurate, and more up to date. (Pity the poor Perl.com editor who tries to keep Larry's hundreds of thousands of words current. That's a hypothetical editor, by the way, because the one you have now just doesn't.)

Fortunately for mere mortals who lack Larry's and Damian's metaphysical ability to hoist great blocks of metaphorical sandstone into place during the Nile delta's dry season, the Synopses are also easier to read and digest.

PGE

Patrick Michaud, Perl 6 pumpking, released the Parrot Grammar Engine early this year. This is (now) a set of Parrot libraries implementing rules (as described in Apocalypse, Exegesis, and Synopsis 5). PGE not only provides regular expression support to all languages hosted on Parrot but it forms the basis of the grammar engine used to build Perl 6.

This has lead to small cleanups in Parrot, namely removing the experimental and long underused regular expression operations.

Patrick's first cut at the grammar engine was a prototype written in C. The current version is a port of that to PIR, the "native language" of Parrot as far as Parrot has a native language that you might actually use to write programs. Perhaps it's a compliment to the design of PIR that Patrick considers this version more maintainable and cleaner than his prototype.

Of course, PIR is an object oriented assembly language with higher-order functions and aggregate data structures.

The next step in the process is a shift-reduce parser. As of the last report, Patrick's code can parse Perl 6 expressions. That leaves the rest of the compiler tool suite to transform the parse tree into code that Parrot (or any other compatible backend) can execute.

A Compiler Roadmap

Speaking of the compiler tools, Allison Randal published her ideas on how to build the full compiler suite for Perl 6 while making the same tools available to any other language hosted on Parrot. In particular, the approach seems to be to perform a series of tree transformations (see "attribute grammars") to turn the output from PGE and the language parser into syntax trees which various tools can analyze, annotate, optimize, and translate into Parrot's AST input format.

Allison's Punie project (which bears some relation to a failed project your editor started a couple of years ago) has succeeded in running a subset of Perl 1 code (you read that correctly) through the entire compiler suite. Perl 1 now runs on Parrot and passes a few tests. At least, part of Perl 1 does, with more on the way.

Though that sounds frivolous, Perl 1 is a real language, if somewhat simpler than Perl 6. If Parrot and the compiler tools work to run Perl 1 programs (and its test suite), it validates the design decisions for the process. It's also a small project with no particular pressure to succeed on its own merits, the nice poetic joy of working aside.

Miscellaneous

Larry recently revealed that he has almost completed a Perl 5 to Perl 5 translator. That may seem flippant and silly until you realize that he's modified the Perl 5.9.2 parser and lexer to save all of the information — including whitespace and comments — it normally throws away. He can reconstruct the original Perl 5 program from an annotated parse tree.

If he can emit working Perl 5 code, he can emit working Perl 6 code.

It may not be perfect (the latest estimate is that it handles 97% of the core Perl code correctly), but even if it only works for 95% of your code, that's still 95% of your code you don't have to translate by hand.

Audrey Tang and Ingy döt Net (you thought your editor had an odd name) might have a similar project up their collective sleeves, too.

Nicholas Clark, wearer of many hats, officially retired as the Ponie pumpking. He and Jesse Vincent have put out a call for a new Ponie pumpking to work with Nicholas to continue porting Perl 5.10-to-be to Parrot.

Social

There's little point in participating in the Perl community without a community. There may be code out there, but unless people can work together, the code isn't as useful or as well-used as it could be. Fortunately, 2005 was a good year.

Revamping TPF

The Perl Foundation doesn't lead the Perl community. Its mission is to help things happen when it can, whether by organizing events and donations or to find the right people to fund for projects.

One common concern in recent years has been that TPF seems awfully quiet, apart from issuing a press release about Perl 6 once in a while. This illustrates a sort of truism in open source development. Ideas are cheap. Implementors are worth their weight in gold. After several years of overworking the same volunteers, TPF has recruited new volunteers and harnessed their energy and ideas to revamp the group.

In particular, Bill Odom (the man who knows practically everyone) is now the new TPF president. Jim Brandt, the primary organizer of 2004's successful YAPC::NA in Buffalo, is the new chairman of the YAPC committee. Andy Lester is in charge of PR and grant manager Curtis "Ovid" Poe is now the head of the Grants Committee. Richard Dice, who lead the organization of this year's YAPC::NA in Toronto, replaced Bill Odom as the chairman of the steering committee.

That doesn't mean that everything TPF could possibly do has someone ready, willing, and able to do it — if you'd like to make the Perl community a little better and have the time and energy and commitment to make it so, TPF could use your help!

TPF also launched The Perl Foundation group weblog for its volunteers to communicate with the broader Perl community about what the Foundation is doing, how it's doing it, and why.

Google's Summer of Code

Does anyone remember an Internet before Google? Even if you do, the 800-exabyte gorilla doled out a cool couple of million dollars to sponsor college and graduate students to increase the amount and quality of open source code in the world. Perl had eight projects participating this year.

Leon Brocard kept an eye on all of the projects and posted interviews with each Summer of Code Perl grant recipient.

TPF Grants

In addition to Adam Kennedy's PPI project mentioned earlier, TPF funded Ivan Tubert-Brohman to build AnnoCPAN. Ivan's site mirrors the documentation of CPAN modules while allowing users to annotate sections that are confusing or unclear, or to add additional notes where possible. Novices often compare Perl's documentation to that of PHP.net; this is one place to improve and surpass that example.

Ivan launched AnnoCPAN just before YAPC::NA in Toronto. With all of the spare time and goodwill he earned, he immediately set to work making the Perl documentation better indexable.

A Conference for All Seasons

Several YAPCs took place this year, including those in Toronto, Israel, Portugal, and somewhere in Asia. There were also hackathons all over the globe (wherever Audrey Tang is, there's a hackathon), workshops, weekends, and various Perl monger meetings.

Miscellaneous

The Perl 5 list summaries restarted.

The Perl 6 list summaries did not stop, so they did not restart.

Everything Else That Didn't Fit Into a Parallel Top-Level Heading

CPAN celebrated its 10th anniversary. Take that, pretty much every other language. See you on Parrot.

Did I miss something? Do you want to try prognosticating? What was significant to you? Let us all know right here.


20 Comments

merlyn
2005-12-31 14:10:02
Thanks! and slight correction
That's CGI::Prototype (not -ed), built on Class::Prototyped. Yeah, I've made that mistake a few times myself, so maybe you've picked it up from one of my quick comments somewhere.


Now, when I get prototype.js included (for AJAX-style calls), I'm not sure whether I should call that CGI::Prototype::Prototype, or perhaps CGI::PrototypeSquared, or maybe just Perl on Planks. :)

chromatic
2005-12-31 15:03:49
Thanks! and slight correction
Link fixed. Thanks for the correction, Randal.
autrijus
2006-01-01 00:44:24
Guido's commit bit
... is actually active, as he accepted the commit bit on OSCON when Larry invited him. We have seen several commits from Larry, but there's currently still zero commits from Guido. Maybe when we start interoperating with PyPy...
aristotle
2006-01-01 15:23:15
Re:
Furthermore, itís Class::Trait (not ::Traits).
chromatic
2006-01-01 21:14:24
More Corrections
Fixed also. Thank you, Aristotle.
Alkon
2006-01-02 04:33:42
TPF
The Perl Foundation doesn't lead the Perl community. It's mission is to help things happen when it can, whether by organizing events and donations or to find the right people to fund for projects.


Given its privately-held governance, the social and legal role of the organization called "The Perl Foundation" is controversial and remains major concern.

chromatic
2006-01-02 09:34:35
TPF
TPF is an organization composed entirely of volunteers. You could easily be a positive influence on the organization.
adrianh
2006-01-03 03:13:59
TPF

...privately-held ...


You mean volunteer.


...governance...


"governance" of what exactly?


... and remains major concern


If stuff concerns you then volunteer and help fix it. At the very least be specific so other people can help fix whatever problems you are perceiving.

pudge
2006-01-03 11:09:00
TPF
What I find controversial and of major concern is that you misquoted chromatic by adding an improper apstrophe to the word "its."
chromatic
2006-01-03 11:33:05
TPF
Actually, I saw the quote and the homophone error and corrected my text without marking it as a correction.
Alkon
2006-01-03 14:24:27
TPF

I mean TPF governance. TPF is a legal entity and existing TPF
governance is privately-held, because TPF bylaws establish no direct
mechanism for Perl community to participate in TPF governance (TPF bylaws are
actually completely unaware of the very existence of Perl community) and
instead give full powers (effectively ownership) to the self-elected board that
consist of only five members, a very narrow group that is in no way
substantially representative of the Perl community at large. From the legal
point, participation of Perl community in TPF governance is at mercy of the
board (with no obligation). Such privately-held governance model is inadequate,
and it is a major concern, because it happens that many Perl community affairs
are now TPF affairs, and TPF happens to hold copyrights for Perl. There are
also important social objections.


Those are not new things. In general, some form of reliable,
community-representative governance mechanism is fundamental for organizations
that rely on community's trust, and cannot be substituted with or traded
for something else.


Alkon
2006-01-03 14:34:09
TPF
Anyway, the quoting semantics is preserved by the very tag name. Don't mind, it's immaterial.
chromatic
2006-01-03 14:43:01
TPF
Why is copyright an issue?


The Perl "community" itself holds no copyrights. Only individuals hold copyrights. Those individuals who have chosen to grant TPF copyright over the compilation of code did so as individual decisions.


What's the problem?


How does community representation work, when of the, say, million people who have programmed Perl in the past few years, you can only find maybe ten thousand here and there to discuss things actively?


It's very difficult for me to believe that any other governance model (besides meritocratic volunteerism) would work at all, let alone any better.

adrianh
2006-01-04 01:45:29
TPF
Sorry. Still don't see what the problem is.


From the legal
point, participation of Perl community in TPF governance is at mercy of the
board (with no obligation). Such privately-held governance model is inadequate,
and it is a major concern,


What exactly is the concern? What problems have this governance caused? How do you go about fixing them? In my experience the TPF is exactly the way organisations like this are run. Volunteers step up to actually do stuff that they think needs to be done.


(disclaimer - I recently volunteered to be a TPM grant manager, but haven't actually managed any grants yet :-)


If you don't like the way it works then come up with actual proposals on how you would like it to work. Publish them. See if you can make the TPF change. If it doesn't want to, then work to setup an alternative.


TPF happens to hold copyrights for Perl


It does? As far as I am aware the TPF holds no copyrights on any Perl code. References?

Alkon
2006-01-04 17:33:13
TPF
"It does? As far as I am aware the TPF holds no copyrights on any Perl code. References?"

"...Holds the copyright on Perl 6 and Parrot..." - on the very first page at http://www.perlfoundation.org/.


Also, when you visit there, it is worth reading legal staff published - it
illustrates my point and explains what TPF governance exactly is (somewhat
beyond the "volunteers step in - step out" way of thinking). Actually, TPF
governance is defined by single document - bylaws, rest describes implications
of bylaws.


I'll answer other questions a just bit later, sorry...


PS: It seems that this site badly lacks preview for comments:)


Alkon
2006-01-04 17:53:58
TPF
I will answer a bit later, sorry...


By the way, those new license and contributor agreements you've mentioned some time ago, are they already available? If so, where I can see them?

Alkon
2006-01-05 12:47:39
TPF

"What exactly is the concern? What problems have this governance caused?
How do you go about fixing them? In my experience the TPF is exactly the way
organisations like this are run. Volunteers step up to actually do stuff that
they think needs to be done."


The common mistake people make is messing governance with
volunteerism
- those are absolutely different concepts.


The concern is that TPF, as it is now, is a privately-held legal entity, not
community-governed one. The major problem with it is that privately-held
governance kills trust, motivation and, as a result, community participation.
Schematically, the problem is: Privately-held governance >> Low trust >> Low
contribution and participation, disappointment >> Inactive community and slow
development. The solution is: Community-held governance >> High trust >> High
contributions and participations >> Vibrant community and productive
development. The fix is: Privately-held governance >> Community-held
governance. For example of excellent community-representative governance
model see governance model used by Apache Foundation, a very successful
community.


The TPF's governance model is actually scandalous. I was shocked when
by incident first discovered what it actually is - I could never have expected
this. For me it is absolute mystery how Perl community could have come out with
such poor governance model for TPF that is in such striking contrast with
fundamental values of that community. If someone can refer me to a discussion
board where it was designed, please let me know, I am very interested to see
myself arguments of a "designers".

chromatic
2006-01-05 12:58:17
TPF
I think you had better start by defining exactly what you mean by the "Perl community", because that seems like the place where our understanding starts to diverge.
chromatic
2006-01-05 13:03:11
TPF
I believe the agreements are still in the final stages of legal review. TPF's main counsel was on holiday at the end of 2005.
Alkon
2006-01-05 13:33:49
TPF

"Why is copyright an issue? The Perl "community" itself holds no
copyrights. Only individuals hold copyrights. Those individuals who have chosen
to grant TPF copyright over the compilation of code did so as individual
decisions."


How copyright could possibly not be the issue? Perl and its copyright
are highly valuable intellectual property being created by collaborative effort
of and contributions (financing, designing, coding, testing, educating,
conferencing, volunteers etc.) from entire community. Its placement is a very
sensitive issue. I think people can be much happier with their "individual
decisions" and much more willing to contribute, if that copyright is placed
with community-governed legal entity, not privately-held corporation owned by
five people (what TPF currently is). That's what the real problem is,
and TPF's "awful quietness" is only a symptom.


"It's very difficult for me to believe that any other governance model
(besides meritocratic volunteerism) would work at all, let alone any better."


The TPF's current governance model has nothing in common with "meritocratic
volunteerism" as we all would like to believe. Actually, the "meritocratic
volunteerism" simply could not work without community-representative
governance, later is absolutely necessary to correctly measure merits
and provide trust and motivation for contributions and volunteers.


"How does community representation work, when of the, say, million people
who have programmed Perl in the past few years, you can only find maybe ten
thousand here and there to discuss things actively?"


It is very simple. In our case there are two principal solutions for
community-representative governance. One such well designed and excellent
community-representative governance model is used by Apache Software Foundation, a very
successful community that also employees so sweet to our hearts "meritocratic
volunteerism". We can easily use this model. In this model Foundation is
governed by community-representative, wide constituency that includes
many reputable community members known for their principal
contribution
in the past - they constitute members of the Foundation
(currently ASF has about 150 members). This members' constituency elects its
new members and (!) every year elects new Board (for details see http://apache.org/foundation/bylaws.html).
In our case, everyone can easily name at least 50 such people, who made
principal contributions to make Perl what it is now. The seconds possible (less
preferable, however) solution is some mechanism that involves Perl grass-root
organizations to regularly reelect Board. Both are relatively easy to implement
and can be combined, if necessary. Moreover, Apache Software Foundation's governance
model is fully developed and needs minimal efforts for adoption.