Animation Pipelines in Python, and the Linux Kernel: Woot!

by Noah Gift

As Jeremy and I get more down the road of finishing our book on Python for *NIX systems administration, working title, we are both going to start dishing out some meatier nuggets of Python that we expose in our book. I have a background in Feature Film Animation Pipelines, and I thought I would share an interesting Python module that Python programmers in the Animation world might enjoy. The module is called pyinotify, and it "monitor's filesystem events with Python under Linux". Sound cools right, well, it is, so lets get a little background.

Inotify went into the linux kernel in release 2.6.13, and, according to Wikipedia, "Inotify uses an API that uses minimal file descriptors, allowing programmers to use the established select and poll interface", in plain english it notices changes to the filesystem and reports those changes to applications. In animation pipelines, generally, files need to make a round trip between CGI, and Editorial. The Animators are given a scene to create, and very large, raw image files, like 16 bit TIFFs, are created. The files generally need to be processed in many different ways depending on where those images need to go. They may need to be converted into a HD Quicktime Movie for viewing on a 2K or 4K, stands for thousands of pixels, digital projector, or they may need to be converted into a compressed format like MXF, or DNxHD, a variant on the MXF file format that Avid has developed. If the files are to get sent to the editorial department for editing, then they will need to be processed with a timecode, or keykode, value embedded into the metadata of the file.

Embedding either a timecode, or keykode, is mandatory, as it allows image files to be assembled automatically according to a shot list, or else they would need to be visually arranged by hand, which just wouldn't work in an animation pipeline. One of the reasons why it wouldn't work, is that the image files could be cut out of a scene, and it would be an incredible waste of money to animate a frame or many frames, that were cut out in editorial. This is just one of the reasons why embedded metadata in files is so important in animation. Of course things need to be kept track of, and that is why using something like SQLAlchemy could make a lot of sense. I am actually building a metadata management system into open source tool I am creating called Liten, and, when I get more time, it will use SQLAlchemy.

All of this rambling background material, leads me back to Python, and Pyinotify. Pyinotify, by talking the Linux Kernel API, can watch a directory or a whole filesystem, for the moment that say, a Maya Artist, has exported a sequence of frames to the "shot tree", or file server, in plain english. At that point, when Pyinotify notices these changes, it could begun to process these files and perhaps move them to a High Speed Fibre SAN like Avid Unity, or XSAN, that is built for playing back HD media files.

Lets take a look at how that might work:

I threw this "toy code", together in about an hour or so, and all it does is pretend to do things when a file is added to the /tmp directory and it closes. I might get around to actually making a useful tool this weekend and adding threading, etc.

If you would like to check this code out, I put up a Google Code Project here. These are some pictures of my "toy code" pretending to do things, when I create a file in a directory, I am watching. Thanks to author of pyinotify for making this so easy to work with!

import os
import sys
import optparse
from pyinotify import WatchManager, Notifier, ProcessEvent, EventsCodes

class PClose(ProcessEvent):
Processes on close event

def __init__(self, path):
self.path = path
self.file = file

def process_IN_CLOSE(self, event):
process 'IN_CLOSE_*' events
can be passed an action function
path = self.path
self.file = "%s" % os.path.join(event.path,
self.file = "%s" % event.path
print "%s Closed" % self.file
print "Performing pretend action on %s...." % self.file
import time
print "%s has been processed" % self.file

class Controller(object):

def __init__(self, path='/tmp'):
self.path = path

def run(self):
self.pclose = PClose(self.path)
PC = self.pclose
# only watch these events
mask = EventsCodes.IN_CLOSE_WRITE | EventsCodes.IN_CLOSE_NOWRITE

# watch manager instance
wm = WatchManager()
notifier = Notifier(wm, PC)

print 'monitoring of %s started' % self.path

added_flag = False
# read and process events
while True:
if not added_flag:
# on first iteration, add a watch on path:
# watch path for events handled by mask.
wm.add_watch(self.path, mask)
added_flag = True
if notifier.check_events():
except KeyboardInterrupt:
# ...until c^c signal
print 'stop monitoring...'
# stop monitoring
except Exception, err:
# otherwise keep on watching
print err

def main():
monitor = Controller()

if __name__ == '__main__':

Sound fun? if so, let me know.


2007-12-05 00:00:40
The funny thing is, XSan is not a SAN. XSan is a cluster file system suitable for running over a SAN. And since it is MacOSX only, so good look getting an XSan filesystem even mountable under Linux. I guess you could use an OSX Server to gateway an XSan filesystem to NFS or CIFS. GFS is the Linux equivalent to XSan.

Avid Unity maybe accessible from Linux (they say it only works with Windows and OSX though), as long it presents either block devices or CIFS access.

But does Inotify work with any network filesystems? Since it appears to be work at the VFS, it should be. But it does not explicitly say that.

2007-12-05 10:35:38
XSan MacOSX only? I guess it's possible all these press releases are marketing fiction -
2007-12-05 10:45:06
A lot of SANs nowadays support just about everyone, from a Flame, to a Final Cut Box, to a Windows Maya machin. XSan is really just licensed from STORNEXT, and rebranded for Apple. The nice thing about the XSAN is that it is a SAN for the masses, they are dirt cheap. When you get into software companies that license proprietary SANs, like Avid Unity, or Discreet Stone Disk Array's, it gets pricey, very quickly.
David Janes
2007-12-05 15:13:32
A suggestion for an Iterator simplification.
Michael Sparks
2007-12-12 02:42:46
Hmm. Useful. I'll have to integrate this into Kamaelia . I'd been tempted to write a pyFUSE filesystem to do some automatic wrangling of where files live on my machine, but this could work nicely too.

(cf (and /Cookbook &) for an idea of what Kamaelia is/can do)
2008-02-29 14:21:10
you are a legend

"Legend is distinguished from the genre of chronicle by the fact that legends apply structures that reveal a moral definition to events"

Indeed w00t is in order here ;)