-
-
Notifications
You must be signed in to change notification settings - Fork 186
Description
My application sometimes pickles functions after the module they were imported from has been removed from sys.modules (for reasons we hopefully don't have to get into). Usually dill handles this just fine, but there seems to be a bug when the pickled function is used in a closure which also maintains a reference to the pickled function's __globals__ dictionary. The following code yields an assertion failure with dill 0.3.5.1 (and also the latest version on master; in each case I was using Python 3.9):
import dill
import sys
import other # another module; see below
def make():
func = other.blah
globs = func.__globals__
def inner():
assert globs is func.__globals__
return inner
f = make()
del sys.modules['other']
f() # assertion passes here
d = dill.dumps(f)
f2 = dill.loads(d)
f2() # assertion fails here
where other.py contains the code:
def blah():
return 42
The assertion fails because in the unpickled version of inner, the free variable func is bound to the unpickled version of blah but the free variable globs is bound to a dict which is not the globals dict of func (nor is it the globals dict of other.blah, in fact).
What seems to be happening is that when blah is unpickled, dill calls _create_function with the argument fglobals set to an empty dictionary, so that the line
func = FunctionType(fcode, fglobals or dict(), fname, fdefaults, fclosure)
creates a new dict to be the globals dict of the new function. Then when the "post-processing" updates the cells of inner, it sets the cell for globs to the old dictionary instead of this newly-created one. To confirm this diagnosis, I removed the or dict() from the line above, and changed the "F1" case of save_function to always set globs = globs_copy (i.e. removing the else branch where it sets globs = {'__name__': obj.__module__}), and indeed the problem went away. I'm not sure how to properly fix the issue, though.