some errors with dump_module and load_module

In trying out `dump_module` and `load_module` a bit more extensively after #507, and I ran across some interesting cases:

1) dump an imported builtin module that contains unpicklable objects
```
>>> import dill
>>> import math
>>> dill.dump_module('math.pkl', main=math)
>>> _math = dill.load_module('math.pkl')
>>> _math.sin(0)
0.0
>>> _math = dill.load_module('math.pkl', main='math')
>>> _math.sin(0)
0.0
>>> import abc
>>> dill.dump_module('abc.pkl', main=abc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 488, in dump_module
    pickler.dump(main)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 880, in dump
    StockPickler.dump(self, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 2099, in save_module
    state=_main_dict)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 1667, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 2172, in save_type
    )), obj=obj, postproc_list=postproc_list)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 1551, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 1667, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _abc_data objects
```
2) dump an imported module by name that fails otherwise, then fails to load
```
>>> import dill
>>> dill.dump_module('abc.pkl', main='abc')
>>> abc = dill.load_module('abc.pkl')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 640, in load_module
    pickle_main = _identify_module(file, main)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 551, in _identify_module
    raise UnpicklingError("reached STOP without finding main module")
_pickle.UnpicklingError: reached STOP without finding main module
>>> abc = dill.load_module('abc.pkl', main='abc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 640, in load_module
    pickle_main = _identify_module(file, main)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 551, in _identify_module
    raise UnpicklingError("reached STOP without finding main module")
_pickle.UnpicklingError: reached STOP without finding main module
>>> import abc
>>> dill.load_module('abc.pkl', main=abc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 640, in load_module
    pickle_main = _identify_module(file, main)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 551, in _identify_module
    raise UnpicklingError("reached STOP without finding main module")
_pickle.UnpicklingError: reached STOP without finding main module
```
3) interestingly, `math` also fails to load when saved by name
```
>>> import dill
>>> dill.dump_module('math.pkl', main='math')
>>> dill.load_module('math.pkl')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 640, in load_module
    pickle_main = _identify_module(file, main)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 551, in _identify_module
    raise UnpicklingError("reached STOP without finding main module")
_pickle.UnpicklingError: reached STOP without finding main module
>>> dill.load_module('math.pkl', main='math')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 640, in load_module
    pickle_main = _identify_module(file, main)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 551, in _identify_module
    raise UnpicklingError("reached STOP without finding main module")
_pickle.UnpicklingError: reached STOP without finding main module
>>> import math
>>> dill.load_module('math.pkl', main=math)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 640, in load_module
    pickle_main = _identify_module(file, main)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 551, in _identify_module
    raise UnpicklingError("reached STOP without finding main module")
_pickle.UnpicklingError: reached STOP without finding main module
>>> 
>>> dill.dump_module('math.pkl', main=math)
>>> dill.load_module('math.pkl')
<module 'math' from '/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload/math.cpython-37m-darwin.so'>
>>> dill.load_module('math.pkl', main='math')
<module 'math' from '/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload/math.cpython-37m-darwin.so'>
```
4) same-named module-type object and imported module; apparently correct behavior, but not obvious
```
>>> import dill
>>> import types
>>> fail = types.ModuleType('fail')
>>> fail.x = 1
>>> dill.dump_module('fail.pkl', main=fail)
```
~ new session ~
```
>>> import dill
>>> fail = dill.load_module('fail.pkl')
>>> fail.x
1
>>> dill.load_module('fail.pkl', main='fail')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 667, in load_module
    % pickle_main
ValueError: can't restore non-imported module 'fail' into an imported one
>>> import fail  # module contains ```y = 0```
>>> dill.load_module('fail.pkl', main=fail)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 667, in load_module
    % pickle_main
ValueError: can't restore non-imported module 'fail' into an imported one
>>> 
>>> dill.dump_module('fail.pkl', main=fail)
>>> 
>>> _fail = dill.load_module('fail.pkl')
>>> _fail.y
0
>>> import types
>>> fail = types.ModuleType('fail')
>>> _fail = dill.load_module('fail.pkl', main=fail)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 672, in load_module
    % pickle_main
ValueError: can't restore imported module 'fail' into a non-imported one
>>> _fail = dill.load_module('fail.pkl', main='fail')
>>> _fail.y
0
```
5) Note that a .pkl file is left even upon failure of `dump_module` (artifact from case 1 above).


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

some errors with dump_module and load_module #525

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

some errors with dump_module and load_module #525

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions