|
|
 |
 |
 |
 |
Python Programming Language
|
 |
 |
 |
 |
|
|
|
|
Automatic reloading, metaclasses, and pickle
Hello all,
I'm using the metaclass trick for automatic reloading of class member
functions, found at: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164
My problem is that if I
1) pickle an object that inherits from "AutoReloader"
2) unpickle the object
3) modify one of the pickled' object's derived class methods
4) reload the module holding the class
... then the changes don't affect the unpickled object. If I unpickle
the object again, of course the changes take effect.
My friend that loves Smalltalk is laughing at me. I thought I had the
upperhand when I discovered the metaclasses but now I am not sure what
to do. I really don't want to have to unpickle again, I'm processing
video and it can take a long time.
By the way, I used to avoid all of these problems by never making
classes, and always building complex structures of lists,
dictionaries, and tuples with global functions. It's going to take me
a while to kick those horrible habits (during my transition, I'm
deriving from list, dict, etc. hehe), perhaps a link to the metaclass
trick is in order in the tutorial's comments on reload?
Any help that avoids having to unpickle again is appreciated!
Thanks,
Andrew Felch
Andrew Felch wrote: > Hello all, > I'm using the metaclass trick for automatic reloading of class member
> functions, found at:http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164
> My problem is that if I
> 1) pickle an object that inherits from "AutoReloader"
> 2) unpickle the object
> 3) modify one of the pickled' object's derived class methods
> 4) reload the module holding the class
> ... then the changes don't affect the unpickled object. If I unpickle
> the object again, of course the changes take effect.
> My friend that loves Smalltalk is laughing at me. I thought I had the
> upperhand when I discovered the metaclasses but now I am not sure what
> to do. I really don't want to have to unpickle again, I'm processing
> video and it can take a long time.
> By the way, I used to avoid all of these problems by never making
> classes, and always building complex structures of lists,
> dictionaries, and tuples with global functions. It's going to take me
> a while to kick those horrible habits (during my transition, I'm
> deriving from list, dict, etc. hehe), perhaps a link to the metaclass
> trick is in order in the tutorial's comments on reload?
> Any help that avoids having to unpickle again is appreciated!
> Thanks,
> Andrew Felch
calling its type. MetaInstanceTracker registers an instance only
when it is created by calling a class.
You can solve this by moving the instance registration to
AutoReloader.__new__ and using pickle protocol version 2, but the
best solution is to avoid both pickle (old pickles break if you
change your code) and autoreloading (it's meant to be used in
interactive console and entertaining ircbots, not in normal code).
Ziga
On Feb 27, 3:50 pm, "Ziga Seilnacht" <ziga.seilna@gmail.com> wrote:
> Andrew Felch wrote: > > Hello all, > > I'm using the metaclass trick for automatic reloading of class member
> > functions, found at:http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164
> > My problem is that if I
> > 1) pickle an object that inherits from "AutoReloader"
> > 2) unpickle the object
> > 3) modify one of the pickled' object's derived class methods
> > 4) reload the module holding the class
> > ... then the changes don't affect the unpickled object. If I unpickle
> > the object again, of course the changes take effect.
> > My friend that loves Smalltalk is laughing at me. I thought I had the
> > upperhand when I discovered the metaclasses but now I am not sure what
> > to do. I really don't want to have to unpickle again, I'm processing
> > video and it can take a long time.
> > By the way, I used to avoid all of these problems by never making
> > classes, and always building complex structures of lists,
> > dictionaries, and tuples with global functions. It's going to take me
> > a while to kick those horrible habits (during my transition, I'm
> > deriving from list, dict, etc. hehe), perhaps a link to the metaclass
> > trick is in order in the tutorial's comments on reload?
> > Any help that avoids having to unpickle again is appreciated!
> > Thanks,
> > Andrew Felch
> This happens because unpickling doesn't recreate your object by
> calling its type. MetaInstanceTracker registers an instance only
> when it is created by calling a class.
> You can solve this by moving the instance registration to
> AutoReloader.__new__ and using pickle protocol version 2, but the
> best solution is to avoid both pickle (old pickles break if you
> change your code) and autoreloading (it's meant to be used in
> interactive console and entertaining ircbots, not in normal code).
> Ziga- Hide quoted text -
>
command: "cPickle.dump(obj, file, 2)"
I did your suggestion, i commented out the "__call__" function of
MetaInstanceTracker and copied the text to the __new__ function of
AutoReloader (code appended). I got a crazy recursive error message
(also appended below). In my code, I am creating a new instance,
rather than using the pickled object (it needs to work in both modes).
Thanks very much for helping me get through this. With my development
approach, finding a solution to this problem is really important to
me.
...
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
RuntimeError: maximum recursion depth exceeded
<code>
import weakref, inspect
class MetaInstanceTracker(type):
def __new__(cls, name, bases, ns):
t = super(MetaInstanceTracker, cls).__new__(cls, name, bases,
ns)
t.__instance_refs__ = []
return t
def __instances__(self):
instances = [(r, r()) for r in self.__instance_refs__]
instances = filter(lambda (x,y): y is not None, instances)
self.__instance_refs__ = [r for (r, o) in instances]
return [o for (r, o) in instances]
## def __call__(self, *args, **kw):
## instance = super(MetaInstanceTracker, self).__call__(*args,
**kw)
## self.__instance_refs__.append(weakref.ref(instance))
## return instance
class InstanceTracker:
__metaclass__ = MetaInstanceTracker
class MetaAutoReloader(MetaInstanceTracker):
def __new__(cls, name, bases, ns):
new_class = super(MetaAutoReloader, cls).__new__(
cls, name, bases, ns)
f = inspect.currentframe().f_back
for d in [f.f_locals, f.f_globals]:
if d.has_key(name):
old_class = d[name]
for instance in old_class.__instances__():
instance.change_class(new_class)
new_class.__instance_refs__.append(
weakref.ref(instance))
# this section only works in 2.3
for subcls in old_class.__subclasses__():
newbases = ()
for base in subcls.__bases__:
if base is old_class:
newbases += (new_class,)
else:
newbases += (base,)
subcls.__bases__ = newbases
break
return new_class
class AutoReloader:
__metaclass__ = MetaAutoReloader
def change_class(self, new_class):
self.__class__ = new_class
def __new__(self, *args, **kw):
instance = super(MetaInstanceTracker, self).__call__(*args,
**kw)
self.__instance_refs__.append(weakref.ref(instance))
return instance
Thanks,
Andrew Felch
Andrew Felch wrote: > Thanks Ziga. I use pickle protocol 2 and binary file types with the
> command: "cPickle.dump(obj, file, 2)"
> I did your suggestion, i commented out the "__call__" function of
> MetaInstanceTracker and copied the text to the __new__ function of
> AutoReloader (code appended). I got a crazy recursive error message
> (also appended below). In my code, I am creating a new instance,
> rather than using the pickled object (it needs to work in both modes).
> Thanks very much for helping me get through this. With my development
> approach, finding a solution to this problem is really important to
> me.
see InstanceTracker.__reduce_ex__. Note that all subclasses of
InstanceTracker and AutoReloader should be cautious when overriding
the
__new__ method. They must call their base class' __new__ method,
preferably by using super(), or the tracking won't work.
# adapted from http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164
import weakref, inspect
class MetaInstanceTracker(type):
def __init__(cls, name, bases, ns):
super(MetaInstanceTracker, cls).__init__(name, bases, ns)
cls.__instance_refs__ = []
def __instances__(cls):
instances = []
validrefs = []
for ref in cls.__instance_refs__:
instance = ref()
if instance is not None:
instances.append(instance)
validrefs.append(ref)
cls.__instance_refs__ = validrefs
return instances
class InstanceTracker(object):
__metaclass__ = MetaInstanceTracker
def __new__(*args, **kwargs):
cls = args[0]
self = super(InstanceTracker, cls).__new__(*args, **kwargs)
cls.__instance_refs__.append(weakref.ref(self))
return self
def __reduce_ex__(self, proto):
return super(InstanceTracker, self).__reduce_ex__(2)
class MetaAutoReloader(MetaInstanceTracker):
def __init__(cls, name, bases, ns):
super(MetaAutoReloader, cls).__init__(name, bases, ns)
f = inspect.currentframe().f_back
for d in [f.f_locals, f.f_globals]:
if name in d:
old_class = d[name]
for instance in old_class.__instances__():
instance.change_class(cls)
cls.__instance_refs__.append(weakref.ref(instance))
for subcls in old_class.__subclasses__():
newbases = []
for base in subcls.__bases__:
if base is old_class:
newbases.append(cls)
else:
newbases.append(base)
subcls.__bases__ = tuple(newbases)
break
class AutoReloader(InstanceTracker):
__metaclass__ = MetaAutoReloader
def change_class(self, new_class):
self.__class__ = new_class
Ziga
On Feb 27, 5:30 pm, "Ziga Seilnacht" <ziga.seilna@gmail.com> wrote:
> Andrew Felch wrote: > > Thanks Ziga. I use pickle protocol 2 and binary file types with the
> > command: "cPickle.dump(obj, file, 2)"
> > I did your suggestion, i commented out the "__call__" function of
> > MetaInstanceTracker and copied the text to the __new__ function of
> > AutoReloader (code appended). I got a crazy recursive error message
> > (also appended below). In my code, I am creating a new instance,
> > rather than using the pickled object (it needs to work in both modes).
> > Thanks very much for helping me get through this. With my development
> > approach, finding a solution to this problem is really important to
> > me.
> Here is a version that should work. It should work with all protocols,
> see InstanceTracker.__reduce_ex__. Note that all subclasses of
> InstanceTracker and AutoReloader should be cautious when overriding
> the
> __new__ method. They must call their base class' __new__ method,
> preferably by using super(), or the tracking won't work.
> # adapted fromhttp://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164
> import weakref, inspect
> class MetaInstanceTracker(type):
> def __init__(cls, name, bases, ns):
> super(MetaInstanceTracker, cls).__init__(name, bases, ns)
> cls.__instance_refs__ = []
> def __instances__(cls):
> instances = []
> validrefs = []
> for ref in cls.__instance_refs__:
> instance = ref()
> if instance is not None:
> instances.append(instance)
> validrefs.append(ref)
> cls.__instance_refs__ = validrefs
> return instances
> class InstanceTracker(object):
> __metaclass__ = MetaInstanceTracker
> def __new__(*args, **kwargs):
> cls = args[0]
> self = super(InstanceTracker, cls).__new__(*args, **kwargs)
> cls.__instance_refs__.append(weakref.ref(self))
> return self
> def __reduce_ex__(self, proto):
> return super(InstanceTracker, self).__reduce_ex__(2)
> class MetaAutoReloader(MetaInstanceTracker):
> def __init__(cls, name, bases, ns):
> super(MetaAutoReloader, cls).__init__(name, bases, ns)
> f = inspect.currentframe().f_back
> for d in [f.f_locals, f.f_globals]:
> if name in d:
> old_class = d[name]
> for instance in old_class.__instances__():
> instance.change_class(cls)
> cls.__instance_refs__.append(weakref.ref(instance))
> for subcls in old_class.__subclasses__():
> newbases = []
> for base in subcls.__bases__:
> if base is old_class:
> newbases.append(cls)
> else:
> newbases.append(base)
> subcls.__bases__ = tuple(newbases)
> break
> class AutoReloader(InstanceTracker):
> __metaclass__ = MetaAutoReloader
> def change_class(self, new_class):
> self.__class__ = new_class
> Ziga
work for either unpickled objects or new objects. I add methods to a
class that inherits from AutoReloader and reload the module, but the
new methods are not callable on the old objects. Man! It seems we're
so close, it will be huge if this ends up working. This stuff is so
over my head, I wish I could help in some additional way.
-Andrew
Andrew Felch wrote: > I pasted the code into mine and replaced the old. It seems not to > work for either unpickled objects or new objects. I add methods to a > class that inherits from AutoReloader and reload the module, but the > new methods are not callable on the old objects. Man! It seems we're > so close, it will be huge if this ends up working. This stuff is so > over my head, I wish I could help in some additional way. > -Andrew
of
the original code. I did some base testing and it worked for me. Could
you post the traceback? Note that Google Groups messed up indentation;
in MetaAutoReloader.__init__, the line starting with
cls.__instance_refs__ should be at the same level as previous line.
Did you restart Python? InstanceTracker, MetaInstanceTracker and
MetaAutoReloader are not auto reloaded :).
Ziga
On Feb 27, 3:23 pm, "Ziga Seilnacht" <ziga.seilna@gmail.com> wrote:
> Andrew Felch wrote: > > I pasted the code into mine and replaced the old. It seems not to > > work for either unpickled objects or new objects. I add methods to a > > class that inherits from AutoReloader and reload the module, but the > > new methods are not callable on the old objects. Man! It seems we're > > so close, it will be huge if this ends up working. This stuff is so > > over my head, I wish I could help in some additional way. > > -Andrew
> Did you copy and paste the entire module? I fiddled almost all parts
> of
> the original code. I did some base testing and it worked for me. Could
> you post the traceback? Note that Google Groups messed up indentation;
> in MetaAutoReloader.__init__, the line starting with
> cls.__instance_refs__ should be at the same level as previous line.
> Did you restart Python? InstanceTracker, MetaInstanceTracker and
> MetaAutoReloader are not auto reloaded :).
> Ziga
inheritance. I inherit from list or some other container first:
class PointList( list, AutoReloader ):
def PrintHi1(self):
print "Hi2"
class MyPrintingClass( AutoReloader ):
def PrintHi2(self):
print "Hi2v2"
Automatic reloading works for MyPrintingClass but not for PointList.
Any ideas?
-Andrew
Andrew Felch wrote: > Thanks for checking. I think I narrowed the problem down to > inheritance. I inherit from list or some other container first: > class PointList( list, AutoReloader ):
> def PrintHi1(self):
> print "Hi2"
> class MyPrintingClass( AutoReloader ):
> def PrintHi2(self):
> print "Hi2v2"
> Automatic reloading works for MyPrintingClass but not for PointList.
> Any ideas?
> -Andrew
class in MRO. Try to switch the bases, so that AutoReloader's
__new__ method will be called first.
Ziga
On Feb 27, 3:47 pm, "Ziga Seilnacht" <ziga.seilna@gmail.com> wrote:
> Andrew Felch wrote: > > Thanks for checking. I think I narrowed the problem down to > > inheritance. I inherit from list or some other container first: > > class PointList( list, AutoReloader ):
> > def PrintHi1(self):
> > print "Hi2"
> > class MyPrintingClass( AutoReloader ):
> > def PrintHi2(self):
> > print "Hi2v2"
> > Automatic reloading works for MyPrintingClass but not for PointList.
> > Any ideas?
> > -Andrew
> Ah yes, this is the problem of list.__new__ not calling the next
> class in MRO. Try to switch the bases, so that AutoReloader's
> __new__ method will be called first.
> Ziga
This really seems to extend Python to be a much better rapid
prototyping langauge. It certainly allows me to now use classes where
I avoided them like the plague before. Perhaps I'll be able to make
programs > 1500 lines now! :-)
-Andrew
On Feb 27, 6:47 pm, "Ziga Seilnacht" <ziga.seilna@gmail.com> wrote:
> Andrew Felch wrote: > > Thanks for checking. I think I narrowed the problem down to > > inheritance. I inherit from list or some other container first: > > class PointList( list, AutoReloader ):
> > def PrintHi1(self):
> > print "Hi2"
> > class MyPrintingClass( AutoReloader ):
> > def PrintHi2(self):
> > print "Hi2v2"
> > Automatic reloading works for MyPrintingClass but not for PointList.
> > Any ideas?
> > -Andrew
> Ah yes, this is the problem of list.__new__ not calling the next
> class in MRO. Try to switch the bases, so that AutoReloader's
> __new__ method will be called first.
> Ziga- Hide quoted text -
>
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=628925&g...
). Pickle and cPickle get confused after reloading, and won't let you
dump the objects to file using protocol 2. The bug occurred for me
when an instance of an AutoReloader class had a member not derived
from AutoReloader (error text: "it's not the same object" ). The
workaround is to use text mode, which I assume means not using
protocol 2. After that, you can reload them in text mode, and then
dump them using protocol 2. This is not so bad since you only do it
when you were going to dump objects to file anyway, so it really only
takes 3x as long on those occassions. The metaclass adaptation is
still incredibly useful because it avoids having to do pickle dump-
loads when you wouldn't otherwise have had to (approaching an infinite
speedup in those cases, thanks Ziga! :-).
Another caveat is that pickle dumping doesn't work if there is a
pointer to a global function (that changes) that is a member of an
instance of a class derived from AutoLoader. The error message has
subtext: "it's not the same object ". That doesn't seem so bad to me,
since you don't have to do that type of thing (function pointers) so
much now that you can use classes!
The best solution would be to make pickle protocol 2 have the same
behavior as the text protocol (i.e. actually work in this case).
Regarding the function pointers, anybody (Ziga :-) know why it is that
pickle chokes so hard on this? Is there some adaptation of the
metaclass or a less beautiful hack that could fix it?
Thanks to those readers following me down the rabbit hole. Ziga's
metaclass adaptation has saved Python in my mind, and these nits seem
to be worth discussing.
- Andrew Felch
|
|
|
|
|
|
Add to del.icio.us |
Digg this |
Stumble it |
Powered by Megasolutions Inc