Animation Pipelines in Python, and the Linux Kernel: Woot!
by Noah Gift
Inotify went into the linux kernel in release 2.6.13, and, according to Wikipedia, "Inotify uses an API that uses minimal file descriptors, allowing programmers to use the established select and poll interface", in plain english it notices changes to the filesystem and reports those changes to applications. In animation pipelines, generally, files need to make a round trip between CGI, and Editorial. The Animators are given a scene to create, and very large, raw image files, like 16 bit TIFFs, are created. The files generally need to be processed in many different ways depending on where those images need to go. They may need to be converted into a HD Quicktime Movie for viewing on a 2K or 4K, stands for thousands of pixels, digital projector, or they may need to be converted into a compressed format like MXF, or DNxHD, a variant on the MXF file format that Avid has developed. If the files are to get sent to the editorial department for editing, then they will need to be processed with a timecode, or keykode, value embedded into the metadata of the file.
Embedding either a timecode, or keykode, is mandatory, as it allows image files to be assembled automatically according to a shot list, or else they would need to be visually arranged by hand, which just wouldn't work in an animation pipeline. One of the reasons why it wouldn't work, is that the image files could be cut out of a scene, and it would be an incredible waste of money to animate a frame or many frames, that were cut out in editorial. This is just one of the reasons why embedded metadata in files is so important in animation. Of course things need to be kept track of, and that is why using something like SQLAlchemy could make a lot of sense. I am actually building a metadata management system into open source tool I am creating called Liten, and, when I get more time, it will use SQLAlchemy.
All of this rambling background material, leads me back to Python, and Pyinotify. Pyinotify, by talking the Linux Kernel API, can watch a directory or a whole filesystem, for the moment that say, a Maya Artist, has exported a sequence of frames to the "shot tree", or file server, in plain english. At that point, when Pyinotify notices these changes, it could begun to process these files and perhaps move them to a High Speed Fibre SAN like Avid Unity, or XSAN, that is built for playing back HD media files.
Lets take a look at how that might work:
I threw this "toy code", together in about an hour or so, and all it does is pretend to do things when a file is added to the /tmp directory and it closes. I might get around to actually making a useful tool this weekend and adding threading, etc.
If you would like to check this code out, I put up a Google Code Project here. These are some pictures of my "toy code" pretending to do things, when I create a file in a directory, I am watching. Thanks to author of pyinotify for making this so easy to work with!
from pyinotify import WatchManager, Notifier, ProcessEvent, EventsCodes
Processes on close event
def __init__(self, path):
self.path = path
self.file = file
def process_IN_CLOSE(self, event):
process 'IN_CLOSE_*' events
can be passed an action function
path = self.path
self.file = "%s" % os.path.join(event.path, event.name)
self.file = "%s" % event.path
print "%s Closed" % self.file
print "Performing pretend action on %s...." % self.file
print "%s has been processed" % self.file
def __init__(self, path='/tmp'):
self.path = path
self.pclose = PClose(self.path)
PC = self.pclose
# only watch these events
mask = EventsCodes.IN_CLOSE_WRITE | EventsCodes.IN_CLOSE_NOWRITE
# watch manager instance
wm = WatchManager()
notifier = Notifier(wm, PC)
print 'monitoring of %s started' % self.path
added_flag = False
# read and process events
if not added_flag:
# on first iteration, add a watch on path:
# watch path for events handled by mask.
added_flag = True
# ...until c^c signal
print 'stop monitoring...'
# stop monitoring
except Exception, err:
# otherwise keep on watching
monitor = Controller()
if __name__ == '__main__':
Sound fun? if so, let me know.
The funny thing is, XSan is not a SAN. XSan is a cluster file system suitable for running over a SAN. And since it is MacOSX only, so good look getting an XSan filesystem even mountable under Linux. I guess you could use an OSX Server to gateway an XSan filesystem to NFS or CIFS. GFS is the Linux equivalent to XSan.
|XSan MacOSX only? I guess it's possible all these press releases are marketing fiction - http://www.xsanity.com/article.php/2006051212230893|
|A lot of SANs nowadays support just about everyone, from a Flame, to a Final Cut Box, to a Windows Maya machin. XSan is really just licensed from STORNEXT, and rebranded for Apple. The nice thing about the XSAN is that it is a SAN for the masses, they are dirt cheap. When you get into software companies that license proprietary SANs, like Avid Unity, or Discreet Stone Disk Array's, it gets pricey, very quickly.|
A suggestion for an Iterator simplification.
Hmm. Useful. I'll have to integrate this into Kamaelia . I'd been tempted to write a pyFUSE filesystem to do some automatic wrangling of where files live on my machine, but this could work nicely too.
you are a legend