[pygtk] Garbage collection prematurely clears cyclical objects referenced from GTK callbacks

Hrvoje Nikšić hrvoje.niksic at avl.com
Wed Aug 13 17:29:14 WST 2008


This is PyGTK version 2.12.1 with Python 2.5.

I noticed very strange behavior of a certain class of objects referenced
only from GTK.  Take this simple example:

import gtk, gc

class Buggy(object):
    def __init__(self):
        w = gtk.Window()
        w.add(gtk.Label('ignore this'))
        w.connect('delete-event', lambda *args: self.die())
        w.show_all()
        self.window = w      # remove this line and the bug goes away

    def die(self):
        gtk.main_quit()

if __name__ == '__main__':
    Buggy()
    gc.collect()
    gtk.main()

Running this example, I'd expect the program to show a window with a
label, and for the program to exit once the window is closed.  What I
get instead, when closing the window, is:

Traceback (most recent call last):
  File "bug.py", line 9, in <lambda>
    w.connect('delete-event', lambda *args: self.die())
NameError: free variable 'self' referenced before assignment in enclosing scope

The 'die' method is never called, it can as well be empty.  A closer
examination of the error message shows that it is not a regular
"accessing unbound variable prior to assignment" often showing up with
Python closures; after all, this use of closures is correct.  This is a
much stranger error message that indicates that the closure is in an
incoherent state.  In fact, it shouldn't even be possible to get this
message through regular Python programming (without messing with
function object internals, that is).

Note that, to repeat the bug, the example should be pasted as-is.  The
bug is quite easy to miss, for example it fails to show if either of
these modifications are made:

* if the instance of Buggy is referenced by something other than a GTK
callback, or
* if the instance of Buggy doesn't contain a reference leading back to
itself (see last line of the constructor), or
* if Buggy implements __del__, therefore disabling GC on its instances,
or
* if no garbage collection happens to occur between setup and exit from
the application

This might be why the bug has gone unnoticed so far, although some
google hits indicate that others have reported it before.  (For example,
googling for "pygtk collection beagle" [without quotes] reveals a
similar issue with "beagle" bindings, although it was written off as a
possible bug in the bindings).

I believe this a genuine bug, because it happens with pure PyGTK code
without other extensions loaded, and I can find no fault with the above
code.  The closure connected to 'delete-event' maintains a strong
reference to 'self', so the instance should live as long as the widget.

If you suspect something fishy is going on with closures (like I did),
one can try converting the closure to a regular object.  The same bug
presents, with only slightly different symptoms:

#!/usr/bin/python

import gtk, gc

class Runmeth(object):
    def __init__(self, obj, meth):
        self.obj = obj
        self.meth = meth

    def __call__(self, *args):
        meth = getattr(self.obj, self.meth)
        meth()

class Buggy(object):
    def __init__(self):
        w = gtk.Window()
        w.add(gtk.Label('ignore this'))
        w.connect('delete-event', Runmeth(self, 'die'))
        w.show_all()
        self.window = w

    def die(self):
        gtk.main_quit()

if __name__ == '__main__':
    Buggy()
    gc.collect()
    gtk.main()

Traceback (most recent call last):
  File "bug2.py", line 11, in __call__
    meth = getattr(self.obj, self.meth)
AttributeError: 'Runmeth' object has no attribute 'obj'

At a glance, this bug looks like a typo when naming the 'obj' attribute,
but again, when you remove the last line of Buggy.__init__, the bug goes
away.

What really happens in both cases is that the garbage collection treats
the "Buggy" object and the closure as unreachable during garbage
collection (and *before* delete-event ever fires).  In case of closure,
tp_clear clears its co_freevars tuple, and in case of a (new-style)
class, tp_clear clears the dict.  In both cases the tp_clear invocation
is quite wrong: garbage collection shouldn't clear cyclical objects that
are referenced from the outside, in this case from a GTK's callback.
Since the objects are in fact reachable from the outside, tp_clear is
not enough to kill them off (which is why it doesn't crash), it only
cripples them by clearing their innards, co_freelist and dict
respectively.

I don't quite understand how this bug occurs, and why it fails to appear
when the cyclical reference is removed.  After all, if the object is not
correctly increffed from GTK, it should die as soon as Buggy() finishes,
but it doesn't happen.

If someone understands the mechanism of PyGTK's memory management that
could provoke this bug, any insight would be appreciated.



More information about the pygtk mailing list