OSCON Tutorial Review: MJD's Iterators and Generators with Perl

by Sarah Burcham

Mark-Jason Dominus

Iterators and Generators in Perl Tutorial

World Premiere Presentation

I was there. The dawn of an era. Where it all began. Iterator value 0. Generator value 'rand()'. The unveiling of Mark-Jason Dominus's "Iterators and Generators in Perl" tutorial was up to MJD's usual exacting standards, carrying the audience with both his dry humour and generally renowned technical aptitude. What has always amazed me about his presentation style is his ability to both move quickly through content and still present explanations carefully and methodically... in such a way that builds up to a crescendo that is both completely unsurprising, and totally cool.
This contrasts Damian Conway's mindbending presentation style, which is to start slowly and then suddenly accelerate beyond the speed of light. They both understand the
rule of pedagogy: only present one new concept at a time.
Speaking of concepts, let's move along.

So...what's a generator?

A subroutine that returns different values each time is a generator (e.g. rand).

So... what's an iterator?

An iterator is an object interface to a list. It keeps track of the 'current position' in the list. It's a method for advancing to the next item in the list. Things like
filehandles, dirhandles, perl's each hash function, the unix yes command. Mark-Jason takes especially twisted delight in using the yes command to iterate no responses with the "yes no command" format. The first half of the tutorial walks us through a case study building up to an iterator version of File::Find. This helps address the problem of a find call
taking a very very very very long time to finish. He shows us iterator implementations using perl object-oriented features, using closures, and a hybrid approach using both. Tricky things such as scope and duration with regard to closures (Why do the variables stay in memory?!), and anonymous subroutine pointers were addressed. MJD explains variable references and garbage collection! What a man. Here's a look at a simple closure implementation of an iterator creator upto that takes a low integer value $lo, a high integer value $hi, and returns an iterator that will iterate through integer values from $lo to $hi.

sub upto {
my ($lo, $hi) = @_;
return sub {
return if $lo > $hi;
return $lo++;

$iterator->() # NEXTVAL

General strategy: To convert a recursive function to an iterator, maintain a to-do list. What's a to-do list? A list of things I know I have to look at, but I know I haven't looked at them yet. MJD pauses and says, "Sometimes also called an agenda. (Latin for a to-do list)." The crowd goes wild with laughter. MJD pauses again, "I wonder how many people laughed because they knew I was telling the truth." Oooh, roots humour... how base.

When power was accidentally cut by an audience member, Mark-Jason comes up fists: "A saboteur... Damian sent you didn't he?" When the projection equipment recovers and begins displaying a countdown to power-up "30...
29... 28", he quips "It's just like New Year's Eve." Definitely in top form

Other in-depth topics covered include: iterators and exception handling (using eval to catch), depth-first vs. breadth first, Genome Sequence Generator example, Iterated database queries, imap (iterated map), igrep (iterated grep), a replacement for WWW:SimpleRobot.
Iterator termination and the semipredicate problem.

All these topics and more are covered extensively in his book Perl Advanced Techniques Handbook, to be published this year by Morgan Kauffman. Will also eventually be available for free at:


Also, if you sign up now, you can get a draft of chapter 4 covering Iterators for 90 pages of iterators iterators iterators. Send email to: mjd-book-subscribe@plover.com

In the tutorial summary, mjd sums up his talk when someone attempts a question, "One second, I'm ranting." I'd rather hear MJD rant than most other people talk.