recursion issues for member functions with inheritance

[mmckerns]

From #234, submitted by @aleneum.

Member functions and inheritance cause some recursion issues. This works for Python 2.7.11 but not for Python 3.3.5:

import dill as pickle
from functools import partial


class Machine(object):

    def __init__(self):
        self.go = partial(self.member, self)

    def member(self, model):
        pass


class SubMachine(Machine):

    def __init__(self):
        super(SubMachine, self).__init__()


m = SubMachine()
dump = pickle.dumps(m)

[mmckerns]

Checking various versions of python...

Python	Status
2.6.9	✔
2.7.13	✔
pypy(2.7.13)	✔
3.1.5	✘
3.2.6	✘
3.3.6	✘
3.4.6	✔
3.5.3	✔
3.6.1	✔

Here's the modified test, checking different pickle settings:

import dill
from functools import partial
from dill.dill import PY3, OLDER


class Machine(object):
    def __init__(self):
        self.go = partial(self.member, self)
    def member(self, model):
        pass


class SubMachine(Machine):
    def __init__(self):
        super(SubMachine, self).__init__()


def test_partials():
    assert dill.copy(SubMachine(), byref=True)
    assert dill.copy(SubMachine(), byref=True, recurse=True)
    if not OLDER:
        assert dill.copy(SubMachine(), recurse=True)
    assert dill.copy(SubMachine())



if __name__ == '__main__':
    test_partials()

With traceback:

dude@borel>$ python Test_partials.py 
Traceback (most recent call last):
  File "Test_partials.py", line 28, in <module>
    test_partials()
  File "Test_partials.py", line 23, in test_partials
    assert dill.copy(SubMachine())
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.8.dev0-py3.3.egg/dill/dill.py", line 227, in copy
    return loads(dumps(obj, *args, **kwds))
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.8.dev0-py3.3.egg/dill/dill.py", line 281, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.8.dev0-py3.3.egg/dill/dill.py", line 274, in dump
    pik.dump(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 235, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 342, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 430, in save_reduce
    save(state)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.8.dev0-py3.3.egg/dill/dill.py", line 871, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 633, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 671, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.8.dev0-py3.3.egg/dill/dill.py", line 1008, in save_functor
    obj.keywords), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 412, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 553, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.8.dev0-py3.3.egg/dill/dill.py", line 1047, in save_instancemethod0
    pickler.save_reduce(MethodType, (obj.__func__, obj.__self__), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 412, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 539, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 342, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 430, in save_reduce
    save(state)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.8.dev0-py3.3.egg/dill/dill.py", line 871, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 633, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 671, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.8.dev0-py3.3.egg/dill/dill.py", line 1008, in save_functor
    obj.keywords), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 412, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 553, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self

... [repeats] ...

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 342, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 407, in save_reduce
    save(cls)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.8.dev0-py3.3.egg/dill/dill.py", line 1284, in save_type
    obj.__bases__, _dict), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 412, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 553, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 539, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.8.dev0-py3.3.egg/dill/dill.py", line 1284, in save_type
    obj.__bases__, _dict), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 412, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 553, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 539, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.8.dev0-py3.3.egg/dill/dill.py", line 1252, in save_type
    pickler.save_reduce(_load_type, (_typemap[obj],), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 412, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 539, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 295, in save
    f = self.dispatch.get(t)
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.8.dev0-py3.3.egg/dill/dill.py", line 398, in get
    return self[key]
RuntimeError: maximum recursion depth exceeded while calling a Python object

[mmckerns]

Given the versions of python that are affected, and the workaround that is suggested in test_recursive in 84e3511, I'm closing this as a corner case that probably won't get handled before 3.3 is obsoleted. However if anyone finds a solution, a PR is welcome.

The long and short of it is that for 3.1-3.3, pickling a class method instance using super in a subclass is not supported -- but can be worked around by pointing to a module-level _super = super.

[aleneum]

thanks for the investigation!

[mmckerns]

@aleneum: I don't like to have any small loss of existing functionality, and this was one of those rare cases that it regrettably happened as python's handling of super changed.

[aleneum]

tbh I actually wanted to suggest dropping 3.3 since -- as you mentioned -- Python 3.3 EOL approaches and pip has already dropped support for 3.3. I know that steps like these usually cause an uproar in the community. But a) it seems the amount of people still using 3.3 is rather low (see the pip issue) which b) may not justify the increased maintenance effort necessary caused by covering all the corner cases and increased code clutter. So, we might drop 3.3. support with the next major release as well. Long story short: I agree that this is the right call.

[mmckerns]

@aleneum: That's totally my train of thought -- except that dill is not dropping support for 3.3. I however am not going to spend excess time chasing down fixes that only affect 3.3 and earlier. This issue is a corner case, and the code in dill that works-around the super issue/bug in python is a bit of a hack. If better code comes along for a super work-around, and it supports 3.3, I'll use it. I think that since pip and other major players in the python community have dropped support for 3.3, letting this corner case slide is the right call.

[aleneum]

except that dill is not dropping support for 3.3. If better code comes along [...] and it supports 3.3, I'll use it.

I see. I jumped to conclusions there. Thanks for clarifying.