Cascading Python Module Reloads

by Chris DiBona

I've been exploring python quite a bit lately in the context of Damage. At Damage we use c++ and python quite a bit, and I'm writing interfaces that work via different protocols (irc, web, etc..) and had an occasion to really explore the python module reloading mechanism.


Python is a fascinating language in a lot of ways and like all new languages brings its share of adjustments to the table. One of the strongest aspects of python is its ability to reload modules on the fly. One of the trickiest parts with this mechanism is how it seems to relate to the scoping rules of python.


For instance...suppose you import a module at one level and want to call a function that reloads the modules that have changed since last checked. You can do that, but if you do it at the called function level, you are actually only changing it within that scope. So if you want to do a base reload of an entire system, you have to have one known good base and have the sub modules reload all of their submodules in turn within the proper scoping context (If I understand this correctly). The outrageously cool thing about this is that you can wrap these importation statements within try/except blocks and thus give your program an extra level of robustness.


The irony here is that repeated reloads (I tested it to around 1m reloads) presents the same amount of memory leakages as redefining the function would (via executing a file within an exec block for instance) , so you end up not making up any sort memory savings, which appear to be additive, but you do get a kind of correctness of scoping.


Anyhow, thought I'd plop that and the following code on the table and see what O'Reilly Networkingans thought of it. Note, indenting is very important in python and ignored mostly for readability sake here:


The code to reload a module would look something like this:


import dispatch


and then later in the code (and indented after a test for the reloading command event)


try:


    reload(dispatch)


except:


    print '-'*60

    traceback.print_exc(file=sys.stdout)

    print '-'*60



The cool, and slightly dangerous and somewhat hard to debug, thing is that if you aren't really concerned about allowing the old bot_dispatch code to run, if it fails the old code still exists and runs until you replace it with a good module that doesn't throw an exception. Of course this can lead to some very lazy programming due to its robustness, but its a nice feature, sort of like an emergency backup.


I'd love to hear how other python folks are doing this kind of reloading, and if my solution is the truly pythonic one or wildly off base and indicative of my ignorance with the language.

Module reloading in python.


2 Comments

anonymous2
2003-09-21 02:18:40
See the PyUnit GUI
The GUI for PyUnit, a.k.a. 'unittest', reloads modules automatically. A note about the problems presented by the task can be found at http://pyunit.sourceforge.net/notes/reloading.html, though the current implementation has evolved slightly from the sample in that page.
anonymous2
2003-09-24 08:34:44
reloading leaks?
I don't think repeatedly reloading a module should cause memory to leak, unless you're using some ancient version of Python that doesn't have the cycle collector...