Python Programming Language :: New-style classes and special methods


	New-style classes and special methods Hi My question is about how special methods are stored internally in Python objects. Consider a new-style class which implements special methods such as __call__ and __new__ class C(type): def __call__(...): <body> class B: __metaclass__ = C <stuff> b= B() The type of C is 'type', that of B is 'C'. When B is instantiated, the __call__ method of C is first invoked, since C is the metaclass for B. Internally, when a Python callable object 'obj' is called, the actual function called seems to be 'obj->ob_type->tp_call'. Does this that somehow the '__call__' method defined in C above is assigned to the 'tp_call' slot in the object representing the class C, instead of it just being stored in the dictionary like a normal attribute? Where and how does this magic happen exactly? I'd appreciate any level of detail. Thanks! Raj - Hide quoted text - Raj B <r@rice.edu> wrote: > Hi > My question is about how special methods are stored internally in > Python objects. > Consider a new-style class which implements special methods such as > __call__ and __new__ > class C(type): > def __call__(...): > <body> > class B: > __metaclass__ = C > <stuff> > b= B() > The type of C is 'type', that of B is 'C'. When B is instantiated, > the __call__ method of C is first invoked, since C is the metaclass > for B. > Internally, when a Python callable object 'obj' is called, the actual > function called seems to be > 'obj->ob_type->tp_call'. > Does this that somehow the '__call__' method defined in C above is > assigned to the 'tp_call' slot in the object representing the class > C, instead of it just being stored in the dictionary like a normal > attribute? Where and how does this magic happen exactly? I'd > appreciate any level of detail. Yes, special methods populate the slots in the structures which Python uses to represent types. Objects/typeobject.c in the Python source distribution does the hard work, particularly in function type_new (line 1722 in my current SVN checkout). If you're not comfortable reading C code you may want to try looking at the "Python implemented in Python" project, pypy, or perhaps alternatives such as Jython (in Java) or better IronPython (in C#), but I am not familiar in detail with how they deal with the issue. Alex > Yes, special methods populate the slots in the structures which Python > uses to represent types. Objects/typeobject.c in the Python source > distribution does the hard work, particularly in function type_new Thanks for that quick response. I am quite comfortable with C code and am trying to understand exactly what happens when a new-style class is created, and then instantiated. I have been reading typeobject.c and type_new() inside it in detail, and there are a few issues I am trying to figure out. I can see a lot of SLOT() macros in the file that seem to set the slots to appropriate values. What I am having trouble figuring out is the connection i.e. at what point during construction of the class object in type_new() are those slots allotted? Is it the tp_alloc() function which does this? Is there some kind of descriptor or other mechanism connecting special method names with their slots in the object representation? (e.g. "__call__" with type->tp_call) Also, what happens when a class inherits from multiple classes with their own __call__ methods? Where and how is it decided which __call__ goes into the tp_call slot? I'm sure I'll eventually figure it out if I stare at the code hard enough, but would totally appreciate any help I can get :) Thanks again! Raj - Hide quoted text - Raj B <r@rice.edu> wrote: > > Yes, special methods populate the slots in the structures which > Python > > uses to represent types. Objects/typeobject.c in the Python source > > distribution does the hard work, particularly in function type_new > Thanks for that quick response. I am quite comfortable with C code > and am trying to understand exactly what happens when a new-style > class is created, and then instantiated. > I have been reading typeobject.c and type_new() inside it in detail, > and there are a few issues I am trying to figure out. > I can see a lot of SLOT() macros in the file that seem to set the > slots to appropriate values. What I am having trouble figuring out is > the connection i.e. at what point during construction of the class > object in type_new() are those slots allotted? Is it the tp_alloc() > function which does this? I believe there are different times -- one for the fundamental struct that represents all types, others for secondary structs that represent e.g. numerical/arithmetic methods, sequence methods, etc, each of which is present only if necessary for a given type. > Is there some kind of descriptor or other mechanism connecting > special method names with their slots in the object representation? > (e.g. "__call__" with type->tp_call) The online docs for the C/API interface do some effort at explaining the type-struct and auxiliary ones, and I spent a couple of pages on that same subject in "Python in a Nutshell", though nowhere close to doing it justice (such docs normally address people who want to write C-coded extensions, rather than ones whose goal is to understand the existing Python runtime -- though many of the issues are identical). > Also, what happens when a class inherits from multiple classes with > their own __call__ methods? Where and how is it decided which > __call__ goes into the tp_call slot? The MRO is used (that stands for Method Resolution Order, and is also implemented in the same C file, and documented there AND in a strong essay by M. Simionato whose URL is, I believe, also in a comment in that file) to find the relevant implementation. > I'm sure I'll eventually figure it out if I stare at the code hard > enough, but would totally appreciate any help I can get :) > Thanks again! You're welcome, but I hope somebody else will also step up to offer useful comments, because (what between my talk at Google Developer's Day tomorrow, and traveling next week to speak in Krakow and then at the first Italian conference on Python in Florence) I'm rather overwhelmed these days;-). Alex - Hide quoted text - Raj B wrote: > > Yes, special methods populate the slots in the structures which Python > > uses to represent types. Objects/typeobject.c in the Python source > > distribution does the hard work, particularly in function type_new > Thanks for that quick response. I am quite comfortable with C code and > am trying to understand exactly what happens when a new-style class is > created, and then instantiated. > I have been reading typeobject.c and type_new() inside it in detail, and > there are a few issues I am trying to figure out. > I can see a lot of *SLOT() macros in the file that seem to set the slots > to appropriate values. What I am having trouble figuring out is the > connection i.e. at what point during construction of the class object in > type_new() are those slots allotted? Is it the tp_alloc() function which > does this? The place to start is the PyType_Type tp_new slot function type_new(). The second to last statement is a call to fixup_slot_dispatchers(). This function goes through the dictionary looking for special methods and adds the appropriate slot functions. This gets very involved. I had to use the Visual Studio debugger to follow what was happening when trying to figure out what happens when assigning a special method to a class after it is declared. > Is there some kind of descriptor or other mechanism connecting special > method names with their slots in the object representation? (e.g. > "__call__" with type->tp_call) There is a special tp_call slot function, slot_tp_call(), that calls the user defined __call__. The same goes for other special methods. Descriptors only come into play with extension types. In this case if a slot function is found a descriptor is added to make the slot function accessible from Python as a special method. > Also, what happens when a class inherits from multiple classes with > their own __call__ methods? Where and how is it decided which __call__ > goes into the tp_call slot? As Alex Martelli mentioned, __call__ is found using the method resolution order. The tp_call slot function slot_tp_call() uses lookup_method(), a variation of PyObject_GetAttribute(), to finding the appropriate Python method. Its all documented in the C file. > I'm sure I'll eventually figure it out if I stare at the code hard > enough, but would totally appreciate any help I can get :) Just ask. -- Lenard Lindstrom <l@telus.net>