backToExample.html | intro.html | compilation.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
1.3 Back to the (backToExample.html)
Up:
1. Extending Python with (intro.html)
Next:
1.5 Compilation and Linkage (compilation.html)
---
# 1.4 The Module's Method Table and Initialization Function
I promised to show how spam_system() is called from Python
programs. First, we need to list its name and address in a ``method
table'':
```text

static PyMethodDef SpamMethods[] = {
...
{"system", spam_system, METH_VARARGS,
"Execute a shell command."},
...
{NULL, NULL, 0, NULL} /* Sentinel */
};
```
Note the third entry ("METH_VARARGS"). This is a flag telling
the interpreter the calling convention to be used for the C
function. It should normally always be "METH_VARARGS" or
"METH_VARARGS | METH_KEYWORDS"; a value of `0` means that an
obsolete variant of PyArg_ParseTuple() is used.
When using only "METH_VARARGS", the function should expect
the Python-level parameters to be passed in as a tuple acceptable for
parsing via PyArg_ParseTuple(); more information on this
function is provided below.
The METH_KEYWORDS bit may be set in the third field if
keyword arguments should be passed to the function. In this case, the
C function should accept a third "PyObject *" parameter which
will be a dictionary of keywords. Use
PyArg_ParseTupleAndKeywords() to parse the arguments to
such a function.
The method table must be passed to the interpreter in the module's
initialization function. The initialization function must be named
initname(), where name is the name of the
module, and should be the only non-static item defined in
the module file:
```text

PyMODINIT_FUNC
initspam(void)
{
(void) Py_InitModule("spam", SpamMethods);
}
```
Note that PyMODINIT_FUNC declares the function as `void` return type,
declares any special linkage declarations required by the platform, and for
C++declares the function as `extern "C"`.
When the Python program imports module spam for the first
time, initspam() is called. (See below for comments about
embedding Python.) It calls
Py_InitModule(), which creates a ``module object'' (which
is inserted in the dictionary `sys.modules` under the key
`"spam"`), and inserts built-in function objects into the newly
created module based upon the table (an array of PyMethodDef
structures) that was passed as its second argument.
Py_InitModule() returns a pointer to the module object
that it creates (which is unused here). It aborts with a fatal error
if the module could not be initialized satisfactorily, so the caller
doesn't need to check for errors.
When embedding Python, the initspam() function is not
called automatically unless there's an entry in the
_PyImport_Inittab table. The easiest way to handle this is to
statically initialize your statically-linked modules by directly
calling initspam() after the call to
Py_Initialize() or PyMac_Initialize():
```text

int
main(int argc, char *argv[])
{
/* Pass argv[0] to the Python interpreter */
Py_SetProgramName(argv[0]);

/* Initialize the Python interpreter. Required. */
Py_Initialize();

/* Add a static module */
initspam();
```
An example may be found in the file Demo/embed/demo.c in the
Python source distribution.
Note:
Removing entries from `sys.modules` or importing
compiled modules into multiple interpreters within a process (or
following a fork() without an intervening
exec()) can create problems for some extension modules.
Extension module authors should exercise caution when initializing
internal data structures.
Note also that the reload() function can be used with
extension modules, and will call the module initialization function
(initspam() in the example), but will not load the module
again if it was loaded from a dynamically loadable object file
(.so on Unix, .dll on Windows).
A more substantial example module is included in the Python source
distribution as Modules/xxmodule.c. This file may be used as a
template or simply read as an example. The modulator.py
script included in the source distribution or Windows install provides
a simple graphical user interface for declaring the functions and
objects which a module should implement, and can generate a template
which can be filled in. The script lives in the
Tools/modulator/ directory; see the README file there
for more information.


dnt-basics.html | dnt-basics.html | node23.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
2.1 The Basics (dnt-basics.html)
Up:
2.1 The Basics (dnt-basics.html)
Next:
2.1.2 Providing finer control (node23.html)
---
## 2.1.1 Adding data and methods to the Basic example
Let's expend the basic example to add some data and methods. Let's
also make the type usable as a base class. We'll create
a new module, noddy2 that adds these capabilities:
```text
#include <Python.h>
#include "structmember.h"

typedef struct {
PyObject_HEAD
PyObject *first;
PyObject *last;
int number;
} Noddy;

static void
Noddy_dealloc(Noddy* self)
{
Py_XDECREF(self->first);
Py_XDECREF(self->last);
self->ob_type->tp_free((PyObject*)self);
}

static PyObject *
Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
Noddy *self;

self = (Noddy *)type->tp_alloc(type, 0);
if (self != NULL) {
self->first = PyString_FromString("");
if (self->first == NULL)
{
Py_DECREF(self);
return NULL;
}

self->last = PyString_FromString("");
if (self->last == NULL)
{
Py_DECREF(self);
return NULL;
}

self->number = 0;
}

return (PyObject *)self;
}

static int
Noddy_init(Noddy *self, PyObject *args, PyObject *kwds)
{
PyObject *first=NULL, *last=NULL;

static char *kwlist[] = {"first", "last", "number", NULL};

if (! PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist,
&first, &last,
&self->number))
return -1;

if (first) {
Py_XDECREF(self->first);
Py_INCREF(first);
self->first = first;
}

if (last) {
Py_XDECREF(self->last);
Py_INCREF(last);
self->last = last;
}

return 0;
}

static PyMemberDef Noddy_members[] = {
{"first", T_OBJECT_EX, offsetof(Noddy, first), 0,
"first name"},
{"last", T_OBJECT_EX, offsetof(Noddy, last), 0,
"last name"},
{"number", T_INT, offsetof(Noddy, number), 0,
"noddy number"},
{NULL} /* Sentinel */
};

static PyObject *
Noddy_name(Noddy* self)
{
static PyObject *format = NULL;
PyObject *args, *result;

if (format == NULL) {
format = PyString_FromString("%s %s");
if (format == NULL)
return NULL;
}

if (self->first == NULL) {
PyErr_SetString(PyExc_AttributeError, "first");
return NULL;
}

if (self->last == NULL) {
PyErr_SetString(PyExc_AttributeError, "last");
return NULL;
}

args = Py_BuildValue("OO", self->first, self->last);
if (args == NULL)
return NULL;

result = PyString_Format(format, args);
Py_DECREF(args);

return result;
}

static PyMethodDef Noddy_methods[] = {
{"name", (PyCFunction)Noddy_name, METH_NOARGS,
"Return the name, combining the first and last name"
},
{NULL} /* Sentinel */
};

static PyTypeObject NoddyType = {
PyObject_HEAD_INIT(NULL)
0, /*ob_size*/
"noddy.Noddy", /*tp_name*/
sizeof(Noddy), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor)Noddy_dealloc, /*tp_dealloc*/
0, /*tp_print*/
0, /*tp_getattr*/
0, /*tp_setattr*/
0, /*tp_compare*/
0, /*tp_repr*/
0, /*tp_as_number*/
0, /*tp_as_sequence*/
0, /*tp_as_mapping*/
0, /*tp_hash */
0, /*tp_call*/
0, /*tp_str*/
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
"Noddy objects", /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
Noddy_methods, /* tp_methods */
Noddy_members, /* tp_members */
0, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
(initproc)Noddy_init, /* tp_init */
0, /* tp_alloc */
Noddy_new, /* tp_new */
};

static PyMethodDef module_methods[] = {
{NULL} /* Sentinel */
};

#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif
PyMODINIT_FUNC
initnoddy2(void)
{
PyObject* m;

if (PyType_Ready(&NoddyType) < 0)
return;

m = Py_InitModule3("noddy2", module_methods,
"Example module that creates an extension type.");

if (m == NULL)
return;

Py_INCREF(&NoddyType);
PyModule_AddObject(m, "Noddy", (PyObject *)&NoddyType);
}
```
Download as text (original file name: noddy2.c). (noddy2.txt)
This version of the module has a number of changes.
We've added an extra include:
```text

#include "structmember.h"
```
This include provides declarations that we use to handle attributes,
as described a bit later.
The name of the Noddy object structure has been shortened to
Noddy. The type object name has been shortened to
NoddyType.
The Noddy type now has three data attributes, first,
last, and number. The first and last
variables are Python strings containing first and last names. The
number attribute is an integer.
The object structure is updated accordingly:
```text

typedef struct {
PyObject_HEAD
PyObject *first;
PyObject *last;
int number;
} Noddy;
```
Because we now have data to manage, we have to be more careful about
object allocation and deallocation. At a minimum, we need a
deallocation method:
```text

static void
Noddy_dealloc(Noddy* self)
{
Py_XDECREF(self->first);
Py_XDECREF(self->last);
self->ob_type->tp_free((PyObject*)self);
}
```
which is assigned to the tp_dealloc member:
```text

(destructor)Noddy_dealloc, /*tp_dealloc*/
```
This method decrements the reference counts of the two Python
attributes. We use Py_XDECREF() here because the
first and last members could be NULL. It then
calls the tp_free member of the object's type to free the
object's memory. Note that the object's type might not be
NoddyType, because the object may be an instance of a
subclass.
We want to make sure that the first and last names are initialized to
empty strings, so we provide a new method:
```text

static PyObject *
Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
Noddy *self;

self = (Noddy *)type->tp_alloc(type, 0);
if (self != NULL) {
self->first = PyString_FromString("");
if (self->first == NULL)
{
Py_DECREF(self);
return NULL;
}

self->last = PyString_FromString("");
if (self->last == NULL)
{
Py_DECREF(self);
return NULL;
}

self->number = 0;
}

return (PyObject *)self;
}
```
and install it in the tp_new member:
```text

Noddy_new, /* tp_new */
```
The new member is responsible for creating (as opposed to
initializing) objects of the type. It is exposed in Python as the
__new__() method. See the paper titled ``Unifying types and
classes in Python'' for a detailed discussion of the __new__()
method. One reason to implement a new method is to assure the initial
values of instance variables. In this case, we use the new method to
make sure that the initial values of the members first and
last are not NULL. If we didn't care whether the initial
values were NULL, we could have used PyType_GenericNew() as
our new method, as we did before. PyType_GenericNew()
initializes all of the instance variable members to NULLs.
The new method is a static method that is passed the type being
instantiated and any arguments passed when the type was called,
and that returns the new object created. New methods always accept
positional and keyword arguments, but they often ignore the arguments,
leaving the argument handling to initializer methods. Note that if the
type supports subclassing, the type passed may not be the type being
defined. The new method calls the tp_alloc slot to allocate memory.
We don't fill the tp_alloc slot ourselves. Rather
PyType_Ready() fills it for us by inheriting it from our
base class, which is object by default. Most types use the
default allocation.
We provide an initialization function:
```text

static int
Noddy_init(Noddy *self, PyObject *args, PyObject *kwds)
{
PyObject *first=NULL, *last=NULL;

static char *kwlist[] = {"first", "last", "number", NULL};

if (! PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist,
&first, &last,
&self->number))
return -1;

if (first) {
Py_XDECREF(self->first);
Py_INCREF(first);
self->first = first;
}

if (last) {
Py_XDECREF(self->last);
Py_INCREF(last);
self->last = last;
}

return 0;
}
```
by filling the tp_init slot.
```text

(initproc)Noddy_init, /* tp_init */
```
The tp_init slot is exposed in Python as the
__init__() method. It is used to initialize an object after
it's created. Unlike the new method, we can't guarantee that the
initializer is called. The initializer isn't called when unpickling
objects and it can be overridden. Our initializer accepts arguments
to provide initial values for our instance. Initializers always accept
positional and keyword arguments.
We want to want to expose our instance variables as attributes. There
are a number of ways to do that. The simplest way is to define member
definitions:
```text

static PyMemberDef Noddy_members[] = {
{"first", T_OBJECT_EX, offsetof(Noddy, first), 0,
"first name"},
{"last", T_OBJECT_EX, offsetof(Noddy, last), 0,
"last name"},
{"number", T_INT, offsetof(Noddy, number), 0,
"noddy number"},
{NULL} /* Sentinel */
};
```
and put the definitions in the tp_members slot:
```text

Noddy_members, /* tp_members */
```
Each member definition has a member name, type, offset, access flags
and documentation string. See the ``Generic Attribute Management''
section below for details.
A disadvantage of this approach is that it doesn't provide a way to
restrict the types of objects that can be assigned to the Python
attributes. We expect the first and last names to be strings, but any
Python objects can be assigned. Further, the attributes can be
deleted, setting the C pointers to NULL. Even though we can make
sure the members are initialized to non-NULLvalues, the members can
be set to NULLif the attributes are deleted.
We define a single method, name, that outputs the objects
name as the concatenation of the first and last names.
```text

static PyObject *
Noddy_name(Noddy* self)
{
static PyObject *format = NULL;
PyObject *args, *result;

if (format == NULL) {
format = PyString_FromString("%s %s");
if (format == NULL)
return NULL;
}

if (self->first == NULL) {
PyErr_SetString(PyExc_AttributeError, "first");
return NULL;
}

if (self->last == NULL) {
PyErr_SetString(PyExc_AttributeError, "last");
return NULL;
}

args = Py_BuildValue("OO", self->first, self->last);
if (args == NULL)
return NULL;

result = PyString_Format(format, args);
Py_DECREF(args);

return result;
}
```
The method is implemented as a C function that takes a Noddy (or
Noddy subclass) instance as the first argument. Methods
always take an instance as the first argument. Methods often take
positional and keyword arguments as well, but in this cased we don't
take any and don't need to accept a positional argument tuple or
keyword argument dictionary. This method is equivalent to the Python
method:
```text

def name(self):
return "%s %s" % (self.first, self.last)
```
Note that we have to check for the possibility that our first
and last members are NULL. This is because they can be
deleted, in which case they are set to NULL. It would be better to
prevent deletion of these attributes and to restrict the attribute
values to be strings. We'll see how to do that in the next section.
Now that we've defined the method, we need to create an array of
method definitions:
```text

static PyMethodDef Noddy_methods[] = {
{"name", (PyCFunction)Noddy_name, METH_NOARGS,
"Return the name, combining the first and last name"
},
{NULL} /* Sentinel */
};
```
and assign them to the tp_methods slot:
```text

Noddy_methods, /* tp_methods */
```
Note that used the METH_NOARGS flag to indicate that the
method is passed no arguments.
Finally, we'll make our type usable as a base class. We've written
our methods carefully so far so that they don't make any assumptions
about the type of the object being created or used, so all we need to
do is to add the Py_TPFLAGS_BASETYPE to our class flag
definition:
```text

Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
```
We rename initnoddy() to initnoddy2()
and update the module name passed to Py_InitModule3().
Finally, we update our setup.py file to build the new module:
```text

from distutils.core import setup, Extension
setup(name="noddy", version="1.0",
ext_modules=[
Extension("noddy", ["noddy.c"]),
Extension("noddy2", ["noddy2.c"]),
])
```


node22.html | dnt-basics.html | node24.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
2.1.1 Adding data and (node22.html)
Up:
2.1 The Basics (dnt-basics.html)
Next:
2.1.3 Supporting cyclic garbage (node24.html)
---
## 2.1.2 Providing finer control over data attributes
In this section, we'll provide finer control over how the
first and last attributes are set in the
Noddy example. In the previous version of our module, the
instance variables first and last could be set to
non-string values or even deleted. We want to make sure that these
attributes always contain strings.
```text
#include <Python.h>
#include "structmember.h"

typedef struct {
PyObject_HEAD
PyObject *first;
PyObject *last;
int number;
} Noddy;

static void
Noddy_dealloc(Noddy* self)
{
Py_XDECREF(self->first);
Py_XDECREF(self->last);
self->ob_type->tp_free((PyObject*)self);
}

static PyObject *
Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
Noddy *self;

self = (Noddy *)type->tp_alloc(type, 0);
if (self != NULL) {
self->first = PyString_FromString("");
if (self->first == NULL)
{
Py_DECREF(self);
return NULL;
}

self->last = PyString_FromString("");
if (self->last == NULL)
{
Py_DECREF(self);
return NULL;
}

self->number = 0;
}

return (PyObject *)self;
}

static int
Noddy_init(Noddy *self, PyObject *args, PyObject *kwds)
{
PyObject *first=NULL, *last=NULL;

static char *kwlist[] = {"first", "last", "number", NULL};

if (! PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist,
&first, &last,
&self->number))
return -1;

if (first) {
Py_DECREF(self->first);
Py_INCREF(first);
self->first = first;
}

if (last) {
Py_DECREF(self->last);
Py_INCREF(last);
self->last = last;
}

return 0;
}

static PyMemberDef Noddy_members[] = {
{"number", T_INT, offsetof(Noddy, number), 0,
"noddy number"},
{NULL} /* Sentinel */
};

static PyObject *
Noddy_getfirst(Noddy *self, void *closure)
{
Py_INCREF(self->first);
return self->first;
}

static int
Noddy_setfirst(Noddy *self, PyObject *value, void *closure)
{
if (value == NULL) {
PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute");
return -1;
}

if (! PyString_Check(value)) {
PyErr_SetString(PyExc_TypeError,
"The first attribute value must be a string");
return -1;
}

Py_DECREF(self->first);
Py_INCREF(value);
self->first = value;

return 0;
}

static PyObject *
Noddy_getlast(Noddy *self, void *closure)
{
Py_INCREF(self->last);
return self->last;
}

static int
Noddy_setlast(Noddy *self, PyObject *value, void *closure)
{
if (value == NULL) {
PyErr_SetString(PyExc_TypeError, "Cannot delete the last attribute");
return -1;
}

if (! PyString_Check(value)) {
PyErr_SetString(PyExc_TypeError,
"The last attribute value must be a string");
return -1;
}

Py_DECREF(self->last);
Py_INCREF(value);
self->last = value;

return 0;
}

static PyGetSetDef Noddy_getseters[] = {
{"first",
(getter)Noddy_getfirst, (setter)Noddy_setfirst,
"first name",
NULL},
{"last",
(getter)Noddy_getlast, (setter)Noddy_setlast,
"last name",
NULL},
{NULL} /* Sentinel */
};

static PyObject *
Noddy_name(Noddy* self)
{
static PyObject *format = NULL;
PyObject *args, *result;

if (format == NULL) {
format = PyString_FromString("%s %s");
if (format == NULL)
return NULL;
}

args = Py_BuildValue("OO", self->first, self->last);
if (args == NULL)
return NULL;

result = PyString_Format(format, args);
Py_DECREF(args);

return result;
}

static PyMethodDef Noddy_methods[] = {
{"name", (PyCFunction)Noddy_name, METH_NOARGS,
"Return the name, combining the first and last name"
},
{NULL} /* Sentinel */
};

static PyTypeObject NoddyType = {
PyObject_HEAD_INIT(NULL)
0, /*ob_size*/
"noddy.Noddy", /*tp_name*/
sizeof(Noddy), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor)Noddy_dealloc, /*tp_dealloc*/
0, /*tp_print*/
0, /*tp_getattr*/
0, /*tp_setattr*/
0, /*tp_compare*/
0, /*tp_repr*/
0, /*tp_as_number*/
0, /*tp_as_sequence*/
0, /*tp_as_mapping*/
0, /*tp_hash */
0, /*tp_call*/
0, /*tp_str*/
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
"Noddy objects", /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
Noddy_methods, /* tp_methods */
Noddy_members, /* tp_members */
Noddy_getseters, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
(initproc)Noddy_init, /* tp_init */
0, /* tp_alloc */
Noddy_new, /* tp_new */
};

static PyMethodDef module_methods[] = {
{NULL} /* Sentinel */
};

#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif
PyMODINIT_FUNC
initnoddy3(void)
{
PyObject* m;

if (PyType_Ready(&NoddyType) < 0)
return;

m = Py_InitModule3("noddy3", module_methods,
"Example module that creates an extension type.");

if (m == NULL)
return;

Py_INCREF(&NoddyType);
PyModule_AddObject(m, "Noddy", (PyObject *)&NoddyType);
}
```
Download as text (original file name: noddy3.c). (noddy3.txt)
To provide greater control, over the first and last
attributes, we'll use custom getter and setter functions. Here are
the functions for getting and setting the first attribute:
```text

Noddy_getfirst(Noddy *self, void *closure)
{
Py_INCREF(self->first);
return self->first;
}

static int
Noddy_setfirst(Noddy *self, PyObject *value, void *closure)
{
if (value == NULL) {
PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute");
return -1;
}

if (! PyString_Check(value)) {
PyErr_SetString(PyExc_TypeError,
"The first attribute value must be a string");
return -1;
}

Py_DECREF(self->first);
Py_INCREF(value);
self->first = value;

return 0;
}
```
The getter function is passed a Noddy object and a
``closure'', which is void pointer. In this case, the closure is
ignored. (The closure supports an advanced usage in which definition
data is passed to the getter and setter. This could, for example, be
used to allow a single set of getter and setter functions that decide
the attribute to get or set based on data in the closure.)
The setter function is passed the Noddy object, the new value,
and the closure. The new value may be NULL, in which case the
attribute is being deleted. In our setter, we raise an error if the
attribute is deleted or if the attribute value is not a string.
We create an array of PyGetSetDef structures:
```text

static PyGetSetDef Noddy_getseters[] = {
{"first",
(getter)Noddy_getfirst, (setter)Noddy_setfirst,
"first name",
NULL},
{"last",
(getter)Noddy_getlast, (setter)Noddy_setlast,
"last name",
NULL},
{NULL} /* Sentinel */
};
```
and register it in the tp_getset slot:
```text

Noddy_getseters, /* tp_getset */
```
to register out attribute getters and setters.
The last item in a PyGetSetDef structure is the closure
mentioned above. In this case, we aren't using the closure, so we just
pass NULL.
We also remove the member definitions for these attributes:
```text

static PyMemberDef Noddy_members[] = {
{"number", T_INT, offsetof(Noddy, number), 0,
"noddy number"},
{NULL} /* Sentinel */
};
```
With these changes, we can assure that the first and
last members are never NULL so we can remove checks for NULL
values in almost all cases. This means that most of the
Py_XDECREF() calls can be converted to Py_DECREF()
calls. The only place we can't change these calls is in the
deallocator, where there is the possibility that the initialization of
these members failed in the constructor.
We also rename the module initialization function and module name in
the initialization function, as we did before, and we add an extra
definition to the setup.py file.


node23.html | dnt-basics.html | dnt-type-methods.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
2.1.2 Providing finer control (node23.html)
Up:
2.1 The Basics (dnt-basics.html)
Next:
2.2 Type Methods (dnt-type-methods.html)
---
## 2.1.3 Supporting cyclic garbage collection
Python has a cyclic-garbage collector that can identify unneeded
objects even when their reference counts are not zero. This can happen
when objects are involved in cycles. For example, consider:
```text

>>> l = []
>>> l.append(l)
>>> del l
```
In this example, we create a list that contains itself. When we delete
it, it still has a reference from itself. It's reference count doesn't
drop to zero. Fortunately, Python's cyclic-garbage collector will
eventually figure out that that the list is garbage and free it.
In the second version of the Noddy example, we allowed any
kind of object to be stored in the first or last
attributes. This means that Noddy objects can participate in
cycles:
```text

>>> import noddy2
>>> n = noddy2.Noddy()
>>> l = [n]
>>> n.first = l
```
This is pretty silly, but it gives us an excuse to add support for the
cyclic-garbage collector to the Noddy example. To support
cyclic garbage collection, types need to fill two slots and set a
class flag that enables these slots:
```text
#include <Python.h>
#include "structmember.h"

typedef struct {
PyObject_HEAD
PyObject *first;
PyObject *last;
int number;
} Noddy;

static int
Noddy_traverse(Noddy *self, visitproc visit, void *arg)
{
if (self->first && visit(self->first, arg) < 0)
return -1;
if (self->last && visit(self->last, arg) < 0)
return -1;

return 0;
}

static int
Noddy_clear(Noddy *self)
{
Py_XDECREF(self->first);
self->first = NULL;
Py_XDECREF(self->last);
self->last = NULL;

return 0;
}

static void
Noddy_dealloc(Noddy* self)
{
Noddy_clear(self);
self->ob_type->tp_free((PyObject*)self);
}

static PyObject *
Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
Noddy *self;

self = (Noddy *)type->tp_alloc(type, 0);
if (self != NULL) {
self->first = PyString_FromString("");
if (self->first == NULL)
{
Py_DECREF(self);
return NULL;
}

self->last = PyString_FromString("");
if (self->last == NULL)
{
Py_DECREF(self);
return NULL;
}

self->number = 0;
}

return (PyObject *)self;
}

static int
Noddy_init(Noddy *self, PyObject *args, PyObject *kwds)
{
PyObject *first=NULL, *last=NULL;

static char *kwlist[] = {"first", "last", "number", NULL};

if (! PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist,
&first, &last,
&self->number))
return -1;

if (first) {
Py_XDECREF(self->first);
Py_INCREF(first);
self->first = first;
}

if (last) {
Py_XDECREF(self->last);
Py_INCREF(last);
self->last = last;
}

return 0;
}

static PyMemberDef Noddy_members[] = {
{"first", T_OBJECT_EX, offsetof(Noddy, first), 0,
"first name"},
{"last", T_OBJECT_EX, offsetof(Noddy, last), 0,
"last name"},
{"number", T_INT, offsetof(Noddy, number), 0,
"noddy number"},
{NULL} /* Sentinel */
};

static PyObject *
Noddy_name(Noddy* self)
{
static PyObject *format = NULL;
PyObject *args, *result;

if (format == NULL) {
format = PyString_FromString("%s %s");
if (format == NULL)
return NULL;
}

if (self->first == NULL) {
PyErr_SetString(PyExc_AttributeError, "first");
return NULL;
}

if (self->last == NULL) {
PyErr_SetString(PyExc_AttributeError, "last");
return NULL;
}

args = Py_BuildValue("OO", self->first, self->last);
if (args == NULL)
return NULL;

result = PyString_Format(format, args);
Py_DECREF(args);

return result;
}

static PyMethodDef Noddy_methods[] = {
{"name", (PyCFunction)Noddy_name, METH_NOARGS,
"Return the name, combining the first and last name"
},
{NULL} /* Sentinel */
};

static PyTypeObject NoddyType = {
PyObject_HEAD_INIT(NULL)
0, /*ob_size*/
"noddy.Noddy", /*tp_name*/
sizeof(Noddy), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor)Noddy_dealloc, /*tp_dealloc*/
0, /*tp_print*/
0, /*tp_getattr*/
0, /*tp_setattr*/
0, /*tp_compare*/
0, /*tp_repr*/
0, /*tp_as_number*/
0, /*tp_as_sequence*/
0, /*tp_as_mapping*/
0, /*tp_hash */
0, /*tp_call*/
0, /*tp_str*/
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /*tp_flags*/
"Noddy objects", /* tp_doc */
(traverseproc)Noddy_traverse, /* tp_traverse */
(inquiry)Noddy_clear, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
Noddy_methods, /* tp_methods */
Noddy_members, /* tp_members */
0, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
(initproc)Noddy_init, /* tp_init */
0, /* tp_alloc */
Noddy_new, /* tp_new */
};

static PyMethodDef module_methods[] = {
{NULL} /* Sentinel */
};

#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif
PyMODINIT_FUNC
initnoddy4(void)
{
PyObject* m;

if (PyType_Ready(&NoddyType) < 0)
return;

m = Py_InitModule3("noddy4", module_methods,
"Example module that creates an extension type.");

if (m == NULL)
return;

Py_INCREF(&NoddyType);
PyModule_AddObject(m, "Noddy", (PyObject *)&NoddyType);
}
```
Download as text (original file name: noddy4.c). (noddy4.txt)
The traversal method provides access to subobjects that
could participate in cycles:
```text

static int
Noddy_traverse(Noddy *self, visitproc visit, void *arg)
{
if (self->first && visit(self->first, arg) < 0)
return -1;
if (self->last && visit(self->last, arg) < 0)
return -1;

return 0;
}
```
For each subobject that can participate in cycles, we need to call the
visit() function, which is passed to the traversal method.
The visit() function takes as arguments the subobject and
the extra argument arg passed to the traversal method.
We also need to provide a method for clearing any subobjects that can
participate in cycles. We implement the method and reimplement the
deallocator to use it:
```text

static int
Noddy_clear(Noddy *self)
{
Py_XDECREF(self->first);
self->first = NULL;
Py_XDECREF(self->last);
self->last = NULL;

return 0;
}

static void
Noddy_dealloc(Noddy* self)
{
Noddy_clear(self);
self->ob_type->tp_free((PyObject*)self);
}
```
Finally, we add the Py_TPFLAGS_HAVE_GC flag to the class
flags:
```text

Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /*tp_flags*/
```
That's pretty much it. If we had written custom tp_alloc or
tp_free slots, we'd need to modify them for cyclic-garbage
collection. Most extensions will use the versions automatically
provided.


2.2.1 Finalization and De-allocation

dnt-type-methods.html | dnt-type-methods.html | node27.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
2.2 Type Methods (dnt-type-methods.html)
Up:
2.2 Type Methods (dnt-type-methods.html)
Next:
2.2.2 Object Presentation (node27.html)
---
## 2.2.1 Finalization and De-allocation
```text

destructor tp_dealloc;
```
This function is called when the reference count of the instance of
your type is reduced to zero and the Python interpreter wants to
reclaim it. If your type has memory to free or other clean-up to
perform, put it here. The object itself needs to be freed here as
well. Here is an example of this function:
```text

static void
newdatatype_dealloc(newdatatypeobject * obj)
{
free(obj->obj_UnderlyingDatatypePtr);
obj->ob_type->tp_free(obj);
}
```
One important requirement of the deallocator function is that it
leaves any pending exceptions alone. This is important since
deallocators are frequently called as the interpreter unwinds the
Python stack; when the stack is unwound due to an exception (rather
than normal returns), nothing is done to protect the deallocators from
seeing that an exception has already been set. Any actions which a
deallocator performs which may cause additional Python code to be
executed may detect that an exception has been set. This can lead to
misleading errors from the interpreter. The proper way to protect
against this is to save a pending exception before performing the
unsafe action, and restoring it when done. This can be done using the
PyErr_Fetch()and
PyErr_Restore()functions:
```text

static void
my_dealloc(PyObject *obj)
{
MyObject *self = (MyObject *) obj;
PyObject *cbresult;

if (self->my_callback != NULL) {
PyObject *err_type, *err_value, *err_traceback;
int have_error = PyErr_Occurred() ? 1 : 0;

if (have_error)
PyErr_Fetch(&err_type, &err_value, &err_traceback);

cbresult = PyObject_CallObject(self->my_callback, NULL);
if (cbresult == NULL)
PyErr_WriteUnraisable();
else
Py_DECREF(cbresult);

if (have_error)
PyErr_Restore(err_type, err_value, err_traceback);

Py_DECREF(self->my_callback);
}
obj->ob_type->tp_free((PyObject*)self);
}
```


2.2.2 Object Presentation

node26.html | dnt-type-methods.html | node28.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
2.2.1 Finalization and De-allocation (node26.html)
Up:
2.2 Type Methods (dnt-type-methods.html)
Next:
2.2.3 Attribute Management (node28.html)
---
## 2.2.2 Object Presentation
In Python, there are three ways to generate a textual representation
of an object: the repr() function (or
equivalent backtick syntax), the str()function, and the print statement. For most objects, the
print statement is equivalent to the str()
function, but it is possible to special-case printing to a
FILE* if necessary; this should only be done if efficiency is
identified as a problem and profiling suggests that creating a
temporary string object to be written to a file is too expensive.
These handlers are all optional, and most types at most need to
implement the tp_str and tp_repr handlers.
```text

reprfunc tp_repr;
reprfunc tp_str;
printfunc tp_print;
```
The tp_repr handler should return a string object containing
a representation of the instance for which it is called. Here is a
simple example:
```text

static PyObject *
newdatatype_repr(newdatatypeobject * obj)
{
return PyString_FromFormat("Repr-ified_newdatatype{{size:\%d}}",
obj->obj_UnderlyingDatatypePtr->size);
}
```
If no tp_repr handler is specified, the interpreter will
supply a representation that uses the type's tp_name and a
uniquely-identifying value for the object.
The tp_str handler is to str() what the
tp_repr handler described above is to repr(); that
is, it is called when Python code calls str() on an
instance of your object. Its implementation is very similar to the
tp_repr function, but the resulting string is intended for
human consumption. If tp_str is not specified, the
tp_repr handler is used instead.
Here is a simple example:
```text

static PyObject *
newdatatype_str(newdatatypeobject * obj)
{
return PyString_FromFormat("Stringified_newdatatype{{size:\%d}}",
obj->obj_UnderlyingDatatypePtr->size);
}
```
The print function will be called whenever Python needs to "print" an
instance of the type. For example, if 'node' is an instance of type
TreeNode, then the print function is called when Python code calls:
```text

print node
```
There is a flags argument and one flag, Py_PRINT_RAW, and
it suggests that you print without string quotes and possibly without
interpreting escape sequences.
The print function receives a file object as an argument. You will
likely want to write to that file object.
Here is a sampe print function:
```text

static int
newdatatype_print(newdatatypeobject *obj, FILE *fp, int flags)
{
if (flags & Py_PRINT_RAW) {
fprintf(fp, "<{newdatatype object--size: %d}>",
obj->obj_UnderlyingDatatypePtr->size);
}
else {
fprintf(fp, "\"<{newdatatype object--size: %d}>\"",
obj->obj_UnderlyingDatatypePtr->size);
}
return 0;
}
```


2.2.3 Attribute Management

node27.html | dnt-type-methods.html | node29.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
2.2.2 Object Presentation (node27.html)
Up:
2.2 Type Methods (dnt-type-methods.html)
Next:
2.2.3.1 Generic Attribute Management (node29.html)
---
## 2.2.3 Attribute Management
For every object which can support attributes, the corresponding type
must provide the functions that control how the attributes are
resolved. There needs to be a function which can retrieve attributes
(if any are defined), and another to set attributes (if setting
attributes is allowed). Removing an attribute is a special case, for
which the new value passed to the handler is NULL.
Python supports two pairs of attribute handlers; a type that supports
attributes only needs to implement the functions for one pair. The
difference is that one pair takes the name of the attribute as a
char*, while the other accepts a PyObject*. Each type
can use whichever pair makes more sense for the implementation's
convenience.
```text

getattrfunc tp_getattr; /* char * version */
setattrfunc tp_setattr;
/* ... */
getattrofunc tp_getattrofunc; /* PyObject * version */
setattrofunc tp_setattrofunc;
```
If accessing attributes of an object is always a simple operation
(this will be explained shortly), there are generic implementations
which can be used to provide the PyObject* version of the
attribute management functions. The actual need for type-specific
attribute handlers almost completely disappeared starting with Python
2.2, though there are many examples which have not been updated to use
some of the new generic mechanism that is available.


2.2.3.1 Generic Attribute Management

node28.html | node28.html | node30.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
2.2.3 Attribute Management (node28.html)
Up:
2.2.3 Attribute Management (node28.html)
Next:
2.2.3.2 Type-specific Attribute Management (node30.html)
---
### 2.2.3.1 Generic Attribute Management
New in version 2.2.
Most extension types only use simple attributes. So, what
makes the attributes simple? There are only a couple of conditions
that must be met:
1. The name of the attributes must be known when
PyType_Ready() is called.
2. No special processing is needed to record that an attribute
was looked up or set, nor do actions need to be taken based
on the value.
Note that this list does not place any restrictions on the values of
the attributes, when the values are computed, or how relevant data is
stored.
When PyType_Ready() is called, it uses three tables
referenced by the type object to create descriptors which are
placed in the dictionary of the type object. Each descriptor controls
access to one attribute of the instance object. Each of the tables is
optional; if all three are NULL, instances of the type will only have
attributes that are inherited from their base type, and should leave
the tp_getattro and tp_setattro fields NULL as
well, allowing the base type to handle attributes.
The tables are declared as three fields of the type object:
```text

struct PyMethodDef *tp_methods;
struct PyMemberDef *tp_members;
struct PyGetSetDef *tp_getset;
```
If tp_methods is not NULL, it must refer to an array of
PyMethodDef structures. Each entry in the table is an
instance of this structure:
```text

typedef struct PyMethodDef {
char *ml_name; /* method name */
PyCFunction ml_meth; /* implementation function */
int ml_flags; /* flags */
char *ml_doc; /* docstring */
} PyMethodDef;
```
One entry should be defined for each method provided by the type; no
entries are needed for methods inherited from a base type. One
additional entry is needed at the end; it is a sentinel that marks the
end of the array. The ml_name field of the sentinel must be
NULL.
XXX Need to refer to some unified discussion of the structure fields,
shared with the next section.
The second table is used to define attributes which map directly to
data stored in the instance. A variety of primitive C types are
supported, and access may be read-only or read-write. The structures
in the table are defined as:
```text

typedef struct PyMemberDef {
char *name;
int type;
int offset;
int flags;
char *doc;
} PyMemberDef;
```
For each entry in the table, a descriptor will be constructed and
added to the type which will be able to extract a value from the
instance structure. The type field should contain one of the
type codes defined in the structmember.h header; the value will
be used to determine how to convert Python values to and from C
values. The flags field is used to store flags which control
how the attribute can be accessed.
XXX Need to move some of this to a shared section!
The following flag constants are defined in structmember.h;
they may be combined using bitwise-OR.
An interesting advantage of using the tp_members table to
build descriptors that are used at runtime is that any attribute
defined this way can have an associated docstring simply by providing
the text in the table. An application can use the introspection API
to retrieve the descriptor from the class object, and get the
docstring using its __doc__ attribute.
As with the tp_methods table, a sentinel entry with a
name value of NULL is required.


node29.html | node28.html | node31.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
2.2.3.1 Generic Attribute Management (node29.html)
Up:
2.2.3 Attribute Management (node28.html)
Next:
2.2.4 Object Comparison (node31.html)
---
### 2.2.3.2 Type-specific Attribute Management
For simplicity, only the char* version will be demonstrated
here; the type of the name parameter is the only difference between
the char* and PyObject* flavors of the interface.
This example effectively does the same thing as the generic example
above, but does not use the generic support added in Python 2.2. The
value in showing this is two-fold: it demonstrates how basic attribute
management can be done in a way that is portable to older versions of
Python, and explains how the handler functions are called, so that if
you do need to extend their functionality, you'll understand what
needs to be done.
The tp_getattr handler is called when the object requires an
attribute look-up. It is called in the same situations where the
__getattr__() method of a class would be called.
A likely way to handle this is (1) to implement a set of functions
(such as newdatatype_getSize() and
newdatatype_setSize() in the example below), (2) provide a
method table listing these functions, and (3) provide a getattr
function that returns the result of a lookup in that table. The
method table uses the same structure as the tp_methods field
of the type object.
Here is an example:
```text

static PyMethodDef newdatatype_methods[] = {
{"getSize", (PyCFunction)newdatatype_getSize, METH_VARARGS,
"Return the current size."},
{"setSize", (PyCFunction)newdatatype_setSize, METH_VARARGS,
"Set the size."},
{NULL, NULL, 0, NULL} /* sentinel */
};

static PyObject *
newdatatype_getattr(newdatatypeobject *obj, char *name)
{
return Py_FindMethod(newdatatype_methods, (PyObject *)obj, name);
}
```
The tp_setattr handler is called when the
__setattr__() or __delattr__() method of a class
instance would be called. When an attribute should be deleted, the
third parameter will be NULL. Here is an example that simply raises
an exception; if this were really all you wanted, the
tp_setattr handler should be set to NULL.
```text

static int
newdatatype_setattr(newdatatypeobject *obj, char *name, PyObject *v)
{
(void)PyErr_Format(PyExc_RuntimeError, "Read-only attribute: \%s", name);
return -1;
}
```


2.2.4 Object Comparison

node30.html | dnt-type-methods.html | node32.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
2.2.3.2 Type-specific Attribute Management (node30.html)
Up:
2.2 Type Methods (dnt-type-methods.html)
Next:
2.2.5 Abstract Protocol Support (node32.html)
---
## 2.2.4 Object Comparison
```text

cmpfunc tp_compare;
```
The tp_compare handler is called when comparisons are needed
and the object does not implement the specific rich comparison method
which matches the requested comparison. (It is always used if defined
and the PyObject_Compare() or PyObject_Cmp()
functions are used, or if cmp() is used from Python.)
It is analogous to the __cmp__() method. This function
should return `-1` if obj1 is less than
obj2, `0` if they are equal, and `1` if
obj1 is greater than
obj2.
(It was previously allowed to return arbitrary negative or positive
integers for less than and greater than, respectively; as of Python
2.2, this is no longer allowed. In the future, other return values
may be assigned a different meaning.)
A tp_compare handler may raise an exception. In this case it
should return a negative value. The caller has to test for the
exception using PyErr_Occurred().
Here is a sample implementation:
```text

static int
newdatatype_compare(newdatatypeobject * obj1, newdatatypeobject * obj2)
{
long result;

if (obj1->obj_UnderlyingDatatypePtr->size <
obj2->obj_UnderlyingDatatypePtr->size) {
result = -1;
}
else if (obj1->obj_UnderlyingDatatypePtr->size >
obj2->obj_UnderlyingDatatypePtr->size) {
result = 1;
}
else {
result = 0;
}
return result;
}
```


2.2.5 Abstract Protocol Support

node31.html | dnt-type-methods.html | node33.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
2.2.4 Object Comparison (node31.html)
Up:
2.2 Type Methods (dnt-type-methods.html)
Next:
2.2.6 More Suggestions (node33.html)
---
## 2.2.5 Abstract Protocol Support
Python supports a variety of abstract `protocols;' the specific
interfaces provided to use these interfaces are documented in the
Python/C API Reference Manual (../api/api.html) in the
chapter ``Abstract Objects Layer (../api/abstract.html).''
A number of these abstract interfaces were defined early in the
development of the Python implementation. In particular, the number,
mapping, and sequence protocols have been part of Python since the
beginning. Other protocols have been added over time. For protocols
which depend on several handler routines from the type implementation,
the older protocols have been defined as optional blocks of handlers
referenced by the type object. For newer protocols there are
additional slots in the main type object, with a flag bit being set to
indicate that the slots are present and should be checked by the
interpreter. (The flag bit does not indicate that the slot values are
non-NULL. The flag may be set to indicate the presense of a slot,
but a slot may still be unfilled.)
```text

PyNumberMethods tp_as_number;
PySequenceMethods tp_as_sequence;
PyMappingMethods tp_as_mapping;
```
If you wish your object to be able to act like a number, a sequence,
or a mapping object, then you place the address of a structure that
implements the C type PyNumberMethods,
PySequenceMethods, or PyMappingMethods, respectively.
It is up to you to fill in this structure with appropriate values. You
can find examples of the use of each of these in the Objects
directory of the Python source distribution.
```text

hashfunc tp_hash;
```
This function, if you choose to provide it, should return a hash
number for an instance of your datatype. Here is a moderately
pointless example:
```text

static long
newdatatype_hash(newdatatypeobject *obj)
{
long result;
result = obj->obj_UnderlyingDatatypePtr->size;
result = result * 3;
return result;
}
```
```text

ternaryfunc tp_call;
```
This function is called when an instance of your datatype is "called",
for example, if `obj1` is an instance of your datatype and the Python
script contains `obj1('hello')`, the tp_call handler is
invoked.
This function takes three arguments:
1. arg1 is the instance of the datatype which is the subject of
the call. If the call is `obj1('hello')`, then arg1 is
`obj1`.
2. arg2 is a tuple containing the arguments to the call. You
can use PyArg_ParseTuple() to extract the arguments.
3. arg3 is a dictionary of keyword arguments that were passed.
If this is non-NULL and you support keyword arguments, use
PyArg_ParseTupleAndKeywords() to extract the
arguments. If you do not want to support keyword arguments and
this is non-NULL, raise a TypeError with a message
saying that keyword arguments are not supported.
Here is a desultory example of the implementation of the call function.
```text

/* Implement the call function.
* obj1 is the instance receiving the call.
* obj2 is a tuple containing the arguments to the call, in this
* case 3 strings.
*/
static PyObject *
newdatatype_call(newdatatypeobject *obj, PyObject *args, PyObject *other)
{
PyObject *result;
char *arg1;
char *arg2;
char *arg3;

if (!PyArg_ParseTuple(args, "sss:call", &arg1, &arg2, &arg3)) {
return NULL;
}
result = PyString_FromFormat(
"Returning -- value: [\%d] arg1: [\%s] arg2: [\%s] arg3: [\%s]\n",
obj->obj_UnderlyingDatatypePtr->size,
arg1, arg2, arg3);
printf("\%s", PyString_AS_STRING(result));
return result;
}
```
XXX some fields need to be added here...
```text

/* Added in release 2.2 */
/* Iterators */
getiterfunc tp_iter;
iternextfunc tp_iternext;
```
These functions provide support for the iterator protocol. Any object
which wishes to support iteration over its contents (which may be
generated during iteration) must implement the `tp_iter`
handler. Objects which are returned by a `tp_iter` handler must
implement both the `tp_iter` and `tp_iternext` handlers.
Both handlers take exactly one parameter, the instance for which they
are being called, and return a new reference. In the case of an
error, they should set an exception and return NULL.
For an object which represents an iterable collection, the
`tp_iter` handler must return an iterator object. The iterator
object is responsible for maintaining the state of the iteration. For
collections which can support multiple iterators which do not
interfere with each other (as lists and tuples do), a new iterator
should be created and returned. Objects which can only be iterated
over once (usually due to side effects of iteration) should implement
this handler by returning a new reference to themselves, and should
also implement the `tp_iternext` handler. File objects are an
example of such an iterator.
Iterator objects should implement both handlers. The `tp_iter`
handler should return a new reference to the iterator (this is the
same as the `tp_iter` handler for objects which can only be
iterated over destructively). The `tp_iternext` handler should
return a new reference to the next object in the iteration if there is
one. If the iteration has reached the end, it may return NULL
without setting an exception or it may set StopIteration;
avoiding the exception can yield slightly better performance. If an
actual error occurs, it should set an exception and return NULL.


2.2.6 More Suggestions

node32.html | dnt-type-methods.html | building.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
2.2.5 Abstract Protocol Support (node32.html)
Up:
2.2 Type Methods (dnt-type-methods.html)
Next:
3. Building C and (building.html)
---
## 2.2.6 More Suggestions
Remember that you can omit most of these functions, in which case you
provide `0` as a value. There are type definitions for each of
the functions you must provide. They are in object.h in the
Python include directory that comes with the source distribution of
Python.
In order to learn how to implement any specific method for your new
datatype, do the following: Download and unpack the Python source
distribution. Go the the Objects directory, then search the
C source files for `tp_` plus the function you want (for
example, `tp_print` or `tp_compare`). You will find
examples of the function you want to implement.
When you need to verify that an object is an instance of the type
you are implementing, use the PyObject_TypeCheck function.
A sample of its use might be something like the following:


B. History and License

reporting-bugs.html | ext.html | node49.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
A. Reporting Bugs (reporting-bugs.html)
Up:
Extending and Embedding the (ext.html)
Next:
B.1 History of the (node49.html)
---
# B. History and License


B.1 History of the software

node48.html | node48.html | node50.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
B. History and License (node48.html)
Up:
B. History and License (node48.html)
Next:
B.2 Terms and conditions (node50.html)
---
# B.1 History of the software
Python was created in the early 1990s by Guido van Rossum at Stichting
Mathematisch Centrum (CWI, see http://www.cwi.nl/) in the Netherlands
as a successor of a language called ABC. Guido remains Python's
principal author, although it includes many contributions from others.
In 1995, Guido continued his work on Python at the Corporation for
National Research Initiatives (CNRI, see http://www.cnri.reston.va.us/)
in Reston, Virginia where he released several versions of the
software.
In May 2000, Guido and the Python core development team moved to
BeOpen.com to form the BeOpen PythonLabs team. In October of the same
year, the PythonLabs team moved to Digital Creations (now Zope
Corporation; see http://www.zope.com/). In 2001, the Python
Software Foundation (PSF, see http://www.python.org/psf/) was
formed, a non-profit organization created specifically to own
Python-related Intellectual Property. Zope Corporation is a
sponsoring member of the PSF.
All Python releases are Open Source (see
http://www.opensource.org/ for the Open Source Definition).
Historically, most, but not all, Python releases have also been
GPL-compatible; the table below summarizes the various releases.
Note:
GPL-compatible doesn't mean that we're distributing
Python under the GPL. All Python licenses, unlike the GPL, let you
distribute a modified version without making your changes open source.
The GPL-compatible licenses make it possible to combine Python with
other software that is released under the GPL; the others don't.
Thanks to the many outside volunteers who have worked under Guido's
direction to make these releases possible.


node49.html | node48.html | about.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
B.1 History of the (node49.html)
Up:
B. History and License (node48.html)
Next:
About this document ... (about.html)
---
# B.2 Terms and conditions for accessing or otherwise using Python
PSF LICENSE AGREEMENT FOR PYTHON 2.3
1. This LICENSE AGREEMENT is between the Python Software Foundation
(``PSF''), and the Individual or Organization (``Licensee'') accessing
and otherwise using Python 2.3 software in source or binary
form and its associated documentation.
2. Subject to the terms and conditions of this License Agreement, PSF
hereby grants Licensee a nonexclusive, royalty-free, world-wide
license to reproduce, analyze, test, perform and/or display publicly,
prepare derivative works, distribute, and otherwise use Python
2.3 alone or in any derivative version, provided, however, that
PSF's License Agreement and PSF's notice of copyright, i.e.,
``Copyright © 2001-2003 Python Software Foundation; All
Rights Reserved'' are retained in Python 2.3 alone or in any
derivative version prepared by Licensee.
3. In the event Licensee prepares a derivative work that is based on
or incorporates Python 2.3 or any part thereof, and wants to
make the derivative work available to others as provided herein, then
Licensee hereby agrees to include in any such work a brief summary of
the changes made to Python 2.3.
4. PSF is making Python 2.3 available to Licensee on an ``AS IS''
basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND
DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 2.3 WILL
NOT INFRINGE ANY THIRD PARTY RIGHTS.
5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
2.3 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR
LOSS AS A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON
2.3, OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE
POSSIBILITY THEREOF.
6. This License Agreement will automatically terminate upon a material
breach of its terms and conditions.
7. Nothing in this License Agreement shall be deemed to create any
relationship of agency, partnership, or joint venture between PSF and
Licensee. This License Agreement does not grant permission to use PSF
trademarks or trade name in a trademark sense to endorse or promote
products or services of Licensee, or any third party.
8. By copying, installing or otherwise using Python 2.3, Licensee
agrees to be bound by the terms and conditions of this License
Agreement.
BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0
BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1
1. This LICENSE AGREEMENT is between BeOpen.com (``BeOpen''), having an
office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the
Individual or Organization (``Licensee'') accessing and otherwise
using this software in source or binary form and its associated
documentation (``the Software'').
2. Subject to the terms and conditions of this BeOpen Python License
Agreement, BeOpen hereby grants Licensee a non-exclusive,
royalty-free, world-wide license to reproduce, analyze, test, perform
and/or display publicly, prepare derivative works, distribute, and
otherwise use the Software alone or in any derivative version,
provided, however, that the BeOpen Python License is retained in the
Software, alone or in any derivative version prepared by Licensee.
3. BeOpen is making the Software available to Licensee on an ``AS IS''
basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND
DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT
INFRINGE ANY THIRD PARTY RIGHTS.
4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE
SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS
AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY
DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
5. This License Agreement will automatically terminate upon a material
breach of its terms and conditions.
6. This License Agreement shall be governed by and interpreted in all
respects by the law of the State of California, excluding conflict of
law provisions. Nothing in this License Agreement shall be deemed to
create any relationship of agency, partnership, or joint venture
between BeOpen and Licensee. This License Agreement does not grant
permission to use BeOpen trademarks or trade names in a trademark
sense to endorse or promote products or services of Licensee, or any
third party. As an exception, the ``BeOpen Python'' logos available
at http://www.pythonlabs.com/logos.html may be used according to the
permissions granted on that web page.
7. By copying, installing or otherwise using the software, Licensee
agrees to be bound by the terms and conditions of this License
Agreement.
CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1
1. This LICENSE AGREEMENT is between the Corporation for National
Research Initiatives, having an office at 1895 Preston White Drive,
Reston, VA 20191 (``CNRI''), and the Individual or Organization
(``Licensee'') accessing and otherwise using Python 1.6.1 software in
source or binary form and its associated documentation.
2. Subject to the terms and conditions of this License Agreement, CNRI
hereby grants Licensee a nonexclusive, royalty-free, world-wide
license to reproduce, analyze, test, perform and/or display publicly,
prepare derivative works, distribute, and otherwise use Python 1.6.1
alone or in any derivative version, provided, however, that CNRI's
License Agreement and CNRI's notice of copyright, i.e., ``Copyright
© 1995-2001 Corporation for National Research Initiatives;
All Rights Reserved'' are retained in Python 1.6.1 alone or in any
derivative version prepared by Licensee. Alternately, in lieu of
CNRI's License Agreement, Licensee may substitute the following text
(omitting the quotes): ``Python 1.6.1 is made available subject to the
terms and conditions in CNRI's License Agreement. This Agreement
together with Python 1.6.1 may be located on the Internet using the
following unique, persistent identifier (known as a handle):
1895.22/1013. This Agreement may also be obtained from a proxy server
on the Internet using the following URL:
http://hdl.handle.net/1895.22/1013.''
3. In the event Licensee prepares a derivative work that is based on
or incorporates Python 1.6.1 or any part thereof, and wants to make
the derivative work available to others as provided herein, then
Licensee hereby agrees to include in any such work a brief summary of
the changes made to Python 1.6.1.
4. CNRI is making Python 1.6.1 available to Licensee on an ``AS IS''
basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND
DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT
INFRINGE ANY THIRD PARTY RIGHTS.
5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1,
OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
6. This License Agreement will automatically terminate upon a material
breach of its terms and conditions.
7. This License Agreement shall be governed by the federal
intellectual property law of the United States, including without
limitation the federal copyright law, and, to the extent such
U.S. federal law does not apply, by the law of the Commonwealth of
Virginia, excluding Virginia's conflict of law provisions.
Notwithstanding the foregoing, with regard to derivative works based
on Python 1.6.1 that incorporate non-separable material that was
previously distributed under the GNU General Public License (GPL), the
law of the Commonwealth of Virginia shall govern this License
Agreement only as to issues arising under or with respect to
Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this
License Agreement shall be deemed to create any relationship of
agency, partnership, or joint venture between CNRI and Licensee. This
License Agreement does not grant permission to use CNRI trademarks or
trade name in a trademark sense to endorse or promote products or
services of Licensee, or any third party.
8. By clicking on the ``ACCEPT'' button where indicated, or by copying,
installing or otherwise using Python 1.6.1, Licensee agrees to be
bound by the terms and conditions of this License Agreement.
ACCEPT
CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2
Copyright © 1991 - 1995, Stichting Mathematisch Centrum
Amsterdam, The Netherlands. All rights reserved.
Permission to use, copy, modify, and distribute this software and its
documentation for any purpose and without fee is hereby granted,
provided that the above copyright notice appear in all copies and that
both that copyright notice and this permission notice appear in
supporting documentation, and that the name of Stichting Mathematisch
Centrum or CWI not be used in advertising or publicity pertaining to
distribution of the software without specific, written prior
permission.
STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO
THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE
FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT
OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.


1.10.4 NULL Pointers

thinIce.html | refcounts.html | cplusplus.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
1.10.3 Thin Ice (thinIce.html)
Up:
1.10 Reference Counts (refcounts.html)
Next:
1.11 Writing Extensions in (cplusplus.html)
---
## 1.10.4 NULL Pointers
In general, functions that take object references as arguments do not
expect you to pass them NULL pointers, and will dump core (or
cause later core dumps) if you do so. Functions that return object
references generally return NULL only to indicate that an
exception occurred. The reason for not testing for NULL
arguments is that functions often pass the objects they receive on to
other function -- if each function were to test for NULL,
there would be a lot of redundant tests and the code would run more
slowly.
It is better to test for NULL only at the ``source:'' when a
pointer that may be NULL is received, for example, from
malloc() or from a function that may raise an exception.
The macros Py_INCREF() and Py_DECREF()
do not check for NULL pointers -- however, their variants
Py_XINCREF() and Py_XDECREF() do.
The macros for checking for a particular object type
(`Py type _Check()`) don't check for NULL pointers --
again, there is much code that calls several of these in a row to test
an object against various different expected types, and this would
generate redundant tests. There are no variants with NULL
checking.
The C function calling mechanism guarantees that the argument list
passed to C functions (`args` in the examples) is never
NULL -- in fact it guarantees that it is always a tuple.1.4 (#foot456)
It is a severe error to ever let a NULL pointer ``escape'' to
the Python user.


1.10.2 Ownership Rules

refcountsInPython.html | refcounts.html | thinIce.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
1.10.1 Reference Counting in (refcountsInPython.html)
Up:
1.10 Reference Counts (refcounts.html)
Next:
1.10.3 Thin Ice (thinIce.html)
---
## 1.10.2 Ownership Rules
Whenever an object reference is passed into or out of a function, it
is part of the function's interface specification whether ownership is
transferred with the reference or not.
Most functions that return a reference to an object pass on ownership
with the reference. In particular, all functions whose function it is
to create a new object, such as PyInt_FromLong() and
Py_BuildValue(), pass ownership to the receiver. Even if
the object is not actually new, you still receive ownership of a new
reference to that object. For instance, PyInt_FromLong()
maintains a cache of popular values and can return a reference to a
cached item.
Many functions that extract objects from other objects also transfer
ownership with the reference, for instance
PyObject_GetAttrString(). The picture is less clear, here,
however, since a few common routines are exceptions:
PyTuple_GetItem(), PyList_GetItem(),
PyDict_GetItem(), and PyDict_GetItemString()
all return references that you borrow from the tuple, list or
dictionary.
The function PyImport_AddModule() also returns a borrowed
reference, even though it may actually create the object it returns:
this is possible because an owned reference to the object is stored in
`sys.modules`.
When you pass an object reference into another function, in general,
the function borrows the reference from you -- if it needs to store
it, it will use Py_INCREF() to become an independent
owner. There are exactly two important exceptions to this rule:
PyTuple_SetItem() and PyList_SetItem(). These
functions take over ownership of the item passed to them -- even if
they fail! (Note that PyDict_SetItem() and friends don't
take over ownership -- they are ``normal.'')
When a C function is called from Python, it borrows references to its
arguments from the caller. The caller owns a reference to the object,
so the borrowed reference's lifetime is guaranteed until the function
returns. Only when such a borrowed reference must be stored or passed
on, it must be turned into an owned reference by calling
Py_INCREF().
The object reference returned from a C function that is called from
Python must be an owned reference -- ownership is tranferred from the
function to its caller.


callingPython.html | intro.html | parseTupleAndKeywords.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
1.6 Calling Python Functions (callingPython.html)
Up:
1. Extending Python with (intro.html)
Next:
1.8 Keyword Parameters for (parseTupleAndKeywords.html)
---
# 1.7 Extracting Parameters in Extension Functions
The PyArg_ParseTuple() function is declared as follows:
```text

int PyArg_ParseTuple(PyObject *arg, char *format, ...);
```
The arg argument must be a tuple object containing an argument
list passed from Python to a C function. The format argument
must be a format string, whose syntax is explained in
``Parsing arguments and building
values (../api/arg-parsing.html)'' in the
Python/C API Reference Manual (../api/api.html). The
remaining arguments must be addresses of variables whose type is
determined by the format string.
Note that while PyArg_ParseTuple() checks that the Python
arguments have the required types, it cannot check the validity of the
addresses of C variables passed to the call: if you make mistakes
there, your code will probably crash or at least overwrite random bits
in memory. So be careful!
Note that any Python object references which are provided to the
caller are borrowed references; do not decrement their
reference count!
Some example calls:
```text

int ok;
int i, j;
long k, l;
char *s;
int size;

ok = PyArg_ParseTuple(args, ""); /* No arguments */
/* Python call: f() */
```
```text

ok = PyArg_ParseTuple(args, "s", &s); /* A string */
/* Possible Python call: f('whoops!') */
```
```text

ok = PyArg_ParseTuple(args, "lls", &k, &l, &s); /* Two longs and a string */
/* Possible Python call: f(1, 2, 'three') */
```
```text

ok = PyArg_ParseTuple(args, "(ii)s#", &i, &j, &s, &size);
/* A pair of ints and a string, whose size is also returned */
/* Possible Python call: f((1, 2), 'three') */
```
```text

{
char *file;
char *mode = "r";
int bufsize = 0;
ok = PyArg_ParseTuple(args, "s|si", &file, &mode, &bufsize);
/* A string, and optionally another string and an integer */
/* Possible Python calls:
f('spam')
f('spam', 'w')
f('spam', 'wb', 100000) */
}
```
```text

{
int left, top, right, bottom, h, v;
ok = PyArg_ParseTuple(args, "((ii)(ii))(ii)",
&left, &top, &right, &bottom, &h, &v);
/* A rectangle and a point */
/* Possible Python call:
f(((0, 0), (400, 300)), (10, 10)) */
}
```
```text

{
Py_complex c;
ok = PyArg_ParseTuple(args, "D:myfunction", &c);
/* a complex, also providing a function name for errors */
/* Possible Python call: myfunction(1+2j) */
}
```


parseTuple.html | intro.html | buildValue.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
1.7 Extracting Parameters in (parseTuple.html)
Up:
1. Extending Python with (intro.html)
Next:
1.9 Building Arbitrary Values (buildValue.html)
---
# 1.8 Keyword Parameters for Extension Functions
The PyArg_ParseTupleAndKeywords() function is declared as
follows:
```text

int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict,
char *format, char *kwlist[], ...);
```
The arg and format parameters are identical to those of the
PyArg_ParseTuple() function. The kwdict parameter
is the dictionary of keywords received as the third parameter from the
Python runtime. The kwlist parameter is a NULL-terminated
list of strings which identify the parameters; the names are matched
with the type information from format from left to right. On
success, PyArg_ParseTupleAndKeywords() returns true,
otherwise it returns false and raises an appropriate exception.
Note:
Nested tuples cannot be parsed when using keyword
arguments! Keyword parameters passed in which are not present in the
kwlist will cause TypeError to be raised.
Here is an example module which uses keywords, based on an example by
Geoff Philbrick (philbrick@hks.com):
```text

#include "Python.h"

static PyObject *
keywdarg_parrot(PyObject *self, PyObject *args, PyObject *keywds)
{
int voltage;
char *state = "a stiff";
char *action = "voom";
char *type = "Norwegian Blue";

static char *kwlist[] = {"voltage", "state", "action", "type", NULL};

if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist,
&voltage, &state, &action, &type))
return NULL;

printf("-- This parrot wouldn't %s if you put %i Volts through it.\n",
action, voltage);
printf("-- Lovely plumage, the %s -- It's %s!\n", type, state);

Py_INCREF(Py_None);

return Py_None;
}

static PyMethodDef keywdarg_methods[] = {
/* The cast of the function is necessary since PyCFunction values
* only take two PyObject* parameters, and keywdarg_parrot() takes
* three.
*/
{"parrot", (PyCFunction)keywdarg_parrot, METH_VARARGS | METH_KEYWORDS,
"Print a lovely skit to standard output."},
{NULL, NULL, 0, NULL} /* sentinel */
};
```
```text

void
initkeywdarg(void)
{
/* Create the module and add the functions */
Py_InitModule("keywdarg", keywdarg_methods);
}
```


5.3 Pure Embedding

lower-level-embedding.html | embedding.html | extending-with-embedding.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
5.2 Beyond Very High (lower-level-embedding.html)
Up:
5. Embedding Python in (embedding.html)
Next:
5.4 Extending Embedded Python (extending-with-embedding.html)
---
# 5.3 Pure Embedding
The first program aims to execute a function in a Python
script. Like in the section about the very high level interface,
the Python interpreter does not directly interact with the
application (but that will change in th next section).
The code to run a function defined in a Python script is:
```text
#include <Python.h>

int
main(int argc, char *argv[])
{
PyObject *pName, *pModule, *pDict, *pFunc;
PyObject *pArgs, *pValue;
int i;

if (argc < 3) {
fprintf(stderr,"Usage: call pythonfile funcname [args]\n");
return 1;
}

Py_Initialize();
pName = PyString_FromString(argv[1]);
/* Error checking of pName left out */

pModule = PyImport_Import(pName);
Py_DECREF(pName);

if (pModule != NULL) {
pDict = PyModule_GetDict(pModule);
/* pDict is a borrowed reference */

pFunc = PyDict_GetItemString(pDict, argv[2]);
/* pFun: Borrowed reference */

if (pFunc && PyCallable_Check(pFunc)) {
pArgs = PyTuple_New(argc - 3);
for (i = 0; i < argc - 3; ++i) {
pValue = PyInt_FromLong(atoi(argv[i + 3]));
if (!pValue) {
Py_DECREF(pArgs);
Py_DECREF(pModule);
fprintf(stderr, "Cannot convert argument\n");
return 1;
}
/* pValue reference stolen here: */
PyTuple_SetItem(pArgs, i, pValue);
}
pValue = PyObject_CallObject(pFunc, pArgs);
Py_DECREF(pArgs);
if (pValue != NULL) {
printf("Result of call: %ld\n", PyInt_AsLong(pValue));
Py_DECREF(pValue);
}
else {
Py_DECREF(pModule);
PyErr_Print();
fprintf(stderr,"Call failed\n");
return 1;
}
/* pDict and pFunc are borrowed and must not be Py_DECREF-ed */
}
else {
if (PyErr_Occurred())
PyErr_Print();
fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
}
Py_DECREF(pModule);
}
else {
PyErr_Print();
fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
return 1;
}
Py_Finalize();
return 0;
}
```
Download as text (original file name: run-func.c). (run-func.txt)
This code loads a Python script using `argv[1]`, and calls the
function named in `argv[2]`. Its integer arguments are the other
values of the `argv` array. If you compile and link this
program (let's call the finished executable call), and use
it to execute a Python script, such as:
```text

def multiply(a,b):
print "Will compute", a, "times", b
c = 0
for i in range(0, a):
c = c + b
return c
```
then the result should be:
```text

$ call multiply multiply 3 2
Will compute 3 times 2
Result of call: 6
```
Although the program is quite large for its functionality, most of the
code is for data conversion between Python and C, and for error
reporting. The interesting part with respect to embedding Python
starts with
```text

Py_Initialize();
pName = PyString_FromString(argv[1]);
/* Error checking of pName left out */
pModule = PyImport_Import(pName);
```
After initializing the interpreter, the script is loaded using
PyImport_Import(). This routine needs a Python string
as its argument, which is constructed using the
PyString_FromString() data conversion routine.
```text

pFunc = PyObject_GetAttrString(pModule, argv[2]);
/* pFunc is a new reference */

if (pFunc && PyCallable_Check(pFunc)) {
...
}
Py_XDECREF(pFunc);
```
Once the script is loaded, the name we're looking for is retrieved
using PyObject_GetAttrString(). If the name exists, and
the object returned is callable, you can safely assume that it is a
function. The program then proceeds by constructing a tuple of
arguments as normal. The call to the Python function is then made
with:
```text

pValue = PyObject_CallObject(pFunc, pArgs);
```
Upon return of the function, `pValue` is either NULL or it
contains a reference to the return value of the function. Be sure to
release the reference after examining the value.


1.10 Reference Counts

buildValue.html | intro.html | refcountsInPython.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
1.9 Building Arbitrary Values (buildValue.html)
Up:
1. Extending Python with (intro.html)
Next:
1.10.1 Reference Counting in (refcountsInPython.html)
---
# 1.10 Reference Counts
In languages like C or C++, the programmer is responsible for
dynamic allocation and deallocation of memory on the heap. In C,
this is done using the functions malloc() and
free(). In C++, the operators new and
delete are used with essentially the same meaning and
we'll restrict the following discussion to the latter.
Every block of memory allocated with malloc() should
eventually be returned to the pool of available memory by exactly one
call to free(). It is important to call
free() at the right time. If a block's address is
forgotten but free() is not called for it, the memory it
occupies cannot be reused until the program terminates. This is
called a memory leak. On the other hand, if a program calls
free() for a block and then continues to use the block, it
creates a conflict with re-use of the block through another
malloc() call. This is called using freed memory.
It has the same bad consequences as referencing uninitialized data --
core dumps, wrong results, mysterious crashes.
Common causes of memory leaks are unusual paths through the code. For
instance, a function may allocate a block of memory, do some
calculation, and then free the block again. Now a change in the
requirements for the function may add a test to the calculation that
detects an error condition and can return prematurely from the
function. It's easy to forget to free the allocated memory block when
taking this premature exit, especially when it is added later to the
code. Such leaks, once introduced, often go undetected for a long
time: the error exit is taken only in a small fraction of all calls,
and most modern machines have plenty of virtual memory, so the leak
only becomes apparent in a long-running process that uses the leaking
function frequently. Therefore, it's important to prevent leaks from
happening by having a coding convention or strategy that minimizes
this kind of errors.
Since Python makes heavy use of malloc() and
free(), it needs a strategy to avoid memory leaks as well
as the use of freed memory. The chosen method is called
reference counting. The principle is simple: every object
contains a counter, which is incremented when a reference to the
object is stored somewhere, and which is decremented when a reference
to it is deleted. When the counter reaches zero, the last reference
to the object has been deleted and the object is freed.
An alternative strategy is called automatic garbage collection.
(Sometimes, reference counting is also referred to as a garbage
collection strategy, hence my use of ``automatic'' to distinguish the
two.) The big advantage of automatic garbage collection is that the
user doesn't need to call free() explicitly. (Another claimed
advantage is an improvement in speed or memory usage -- this is no
hard fact however.) The disadvantage is that for C, there is no
truly portable automatic garbage collector, while reference counting
can be implemented portably (as long as the functions malloc()
and free() are available -- which the C Standard guarantees).
Maybe some day a sufficiently portable automatic garbage collector
will be available for C. Until then, we'll have to live with
reference counts.
While Python uses the traditional reference counting implementation,
it also offers a cycle detector that works to detect reference
cycles. This allows applications to not worry about creating direct
or indirect circular references; these are the weakness of garbage
collection implemented using only reference counting. Reference
cycles consist of objects which contain (possibly indirect) references
to themselves, so that each object in the cycle has a reference count
which is non-zero. Typical reference counting implementations are not
able to reclaim the memory belonging to any objects in a reference
cycle, or referenced from the objects in the cycle, even though there
are no further references to the cycle itself.
The cycle detector is able to detect garbage cycles and can reclaim
them so long as there are no finalizers implemented in Python
(__del__() methods). When there are such finalizers, the
detector exposes the cycles through the gc
module (../lib/module-gc.html) (specifically, the `garbage`
variable in that module). The gc module also exposes a way
to run the detector (the collect() function), as well as
configuration interfaces and the ability to disable the detector at
runtime. The cycle detector is considered an optional component;
though it is included by default, it can be disabled at build time
using the --without-cycle-gc option to the
configure script on Unix platforms (including Mac OS X)
or by removing the definition of `WITH_CYCLE_GC` in the
pyconfig.h header on other platforms. If the cycle detector is
disabled in this way, the gc module will not be available.


1.10.1 Reference Counting in Python

refcounts.html | refcounts.html | ownershipRules.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
1.10 Reference Counts (refcounts.html)
Up:
1.10 Reference Counts (refcounts.html)
Next:
1.10.2 Ownership Rules (ownershipRules.html)
---
## 1.10.1 Reference Counting in Python
There are two macros, `Py_INCREF(x)` and `Py_DECREF(x)`,
which handle the incrementing and decrementing of the reference count.
Py_DECREF() also frees the object when the count reaches zero.
For flexibility, it doesn't call free() directly -- rather, it
makes a call through a function pointer in the object's type
object. For this purpose (and others), every object also contains a
pointer to its type object.
The big question now remains: when to use `Py_INCREF(x)` and
`Py_DECREF(x)`? Let's first introduce some terms. Nobody
``owns'' an object; however, you can own a reference to an
object. An object's reference count is now defined as the number of
owned references to it. The owner of a reference is responsible for
calling Py_DECREF() when the reference is no longer
needed. Ownership of a reference can be transferred. There are three
ways to dispose of an owned reference: pass it on, store it, or call
Py_DECREF(). Forgetting to dispose of an owned reference
creates a memory leak.
It is also possible to borrow1.2 (#foot395) a reference to an object. The borrower
of a reference should not call Py_DECREF(). The borrower must
not hold on to the object longer than the owner from which it was
borrowed. Using a borrowed reference after the owner has disposed of
it risks using freed memory and should be avoided
completely.1.3 (#foot527)
The advantage of borrowing over owning a reference is that you don't
need to take care of disposing of the reference on all possible paths
through the code -- in other words, with a borrowed reference you
don't run the risk of leaking when a premature exit is taken. The
disadvantage of borrowing over leaking is that there are some subtle
situations where in seemingly correct code a borrowed reference can be
used after the owner from which it was borrowed has in fact disposed
of it.
A borrowed reference can be changed into an owned reference by calling
Py_INCREF(). This does not affect the status of the owner from
which the reference was borrowed -- it creates a new owned reference,
and gives full owner responsibilities (the new owner must
dispose of the reference properly, as well as the previous owner).


A. Reporting Bugs

link-reqs.html | ext.html | node48.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
5.6 Linking Requirements (link-reqs.html)
Up:
Extending and Embedding the (ext.html)
Next:
B. History and License (node48.html)
---
# A. Reporting Bugs
Python is a mature programming language which has established a
reputation for stability. In order to maintain this reputation, the
developers would like to know of any deficiencies you find in Python
or its documentation.
Before submitting a report, you will be required to log into SourceForge;
this will make it possible for the developers to contact you
for additional information if needed. It is not possible to submit a
bug report anonymously.
All bug reports should be submitted via the Python Bug Tracker on
SourceForge (http://sourceforge.net/bugs/?group_id=5470). The
bug tracker offers a Web form which allows pertinent information to be
entered and submitted to the developers.
The first step in filing a report is to determine whether the problem
has already been reported. The advantage in doing so, aside from
saving the developers time, is that you learn what has been done to
fix it; it may be that the problem has already been fixed for the next
release, or additional information is needed (in which case you are
welcome to provide it if you can!). To do this, search the bug
database using the search box near the bottom of the page.
If the problem you're reporting is not already in the bug tracker, go
back to the Python Bug Tracker
(http://sourceforge.net/bugs/?group_id=5470). Select the
``Submit a Bug'' link at the top of the page to open the bug reporting
form.
The submission form has a number of fields. The only fields that are
required are the ``Summary'' and ``Details'' fields. For the summary,
enter a very short description of the problem; less than ten
words is good. In the Details field, describe the problem in detail,
including what you expected to happen and what did happen. Be sure to
include the version of Python you used, whether any extension modules
were involved, and what hardware and software platform you were using
(including version information as appropriate).
The only other field that you may want to set is the ``Category''
field, which allows you to place the bug report into a broad category
(such as ``Documentation'' or ``Library'').
Each bug report will be assigned to a developer who will determine
what needs to be done to correct the problem. You will
receive an update each time action is taken on the bug.
See Also:


1.1 A Simple Example

intro.html | intro.html | errors.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
1. Extending Python with (intro.html)
Up:
1. Extending Python with (intro.html)
Next:
1.2 Intermezzo: Errors and (errors.html)
---
# 1.1 A Simple Example
Let's create an extension module called "spam" (the favorite food
of Monty Python fans...) and let's say we want to create a Python
interface to the C library function system().1.1 (#foot511)This function takes a null-terminated character string as argument and
returns an integer. We want this function to be callable from Python
as follows:
```text

>>> import spam
>>> status = spam.system("ls -l")
```
Begin by creating a file spammodule.c. (Historically, if a
module is called "spam", the C file containing its implementation
is called spammodule.c; if the module name is very long, like
"spammify", the module name can be just spammify.c.)
The first line of our file can be:
```text

#include <Python.h>
```
which pulls in the Python API (you can add a comment describing the
purpose of the module and a copyright notice if you like).
Since Python may define some pre-processor definitions which affect
the standard headers on some systems, you must include Python.h
before any standard headers are included.
All user-visible symbols defined by Python.h have a prefix of
"Py" or "PY", except those defined in standard header files.
For convenience, and since they are used extensively by the Python
interpreter, `"Python.h"` includes a few standard header files:
`<stdio.h>`, `<string.h>`, `<errno.h>`, and
`<stdlib.h>`. If the latter header file does not exist on your
system, it declares the functions malloc(),
free() and realloc() directly.
The next thing we add to our module file is the C function that will
be called when the Python expression "spam.system(string)"is evaluated (we'll see shortly how it ends up being called):
```text

static PyObject *
spam_system(PyObject *self, PyObject *args)
{
char *command;
int sts;

if (!PyArg_ParseTuple(args, "s", &command))
return NULL;
sts = system(command);
return Py_BuildValue("i", sts);
}
```
There is a straightforward translation from the argument list in
Python (for example, the single expression `"ls -l"`) to the
arguments passed to the C function. The C function always has two
arguments, conventionally named self and args.
The self argument is only used when the C function implements a
built-in method, not a function. In the example, self will
always be a NULL pointer, since we are defining a function, not a
method. (This is done so that the interpreter doesn't have to
understand two different types of C functions.)
The args argument will be a pointer to a Python tuple object
containing the arguments. Each item of the tuple corresponds to an
argument in the call's argument list. The arguments are Python
objects -- in order to do anything with them in our C function we have
to convert them to C values. The function PyArg_ParseTuple()
in the Python API checks the argument types and converts them to C
values. It uses a template string to determine the required types of
the arguments as well as the types of the C variables into which to
store the converted values. More about this later.
PyArg_ParseTuple() returns true (nonzero) if all arguments have
the right type and its components have been stored in the variables
whose addresses are passed. It returns false (zero) if an invalid
argument list was passed. In the latter case it also raises an
appropriate exception so the calling function can return
NULL immediately (as we saw in the example).


1.10.3 Thin Ice

ownershipRules.html | refcounts.html | nullPointers.html | Extending and Embedding the Python Interpreter | contents.html
Previous:
1.10.2 Ownership Rules (ownershipRules.html)
Up:
1.10 Reference Counts (refcounts.html)
Next:
1.10.4 NULL Pointers (nullPointers.html)
---
## 1.10.3 Thin Ice
There are a few situations where seemingly harmless use of a borrowed
reference can lead to problems. These all have to do with implicit
invocations of the interpreter, which can cause the owner of a
reference to dispose of it.
The first and most important case to know about is using
Py_DECREF() on an unrelated object while borrowing a
reference to a list item. For instance:
```text

void
bug(PyObject *list)
{
PyObject *item = PyList_GetItem(list, 0);

PyList_SetItem(list, 1, PyInt_FromLong(0L));
PyObject_Print(item, stdout, 0); /* BUG! */
}
```
This function first borrows a reference to `list[0]`, then
replaces `list[1]` with the value `0`, and finally prints
the borrowed reference. Looks harmless, right? But it's not!
Let's follow the control flow into PyList_SetItem(). The list
owns references to all its items, so when item 1 is replaced, it has
to dispose of the original item 1. Now let's suppose the original
item 1 was an instance of a user-defined class, and let's further
suppose that the class defined a __del__() method. If this
class instance has a reference count of 1, disposing of it will call
its __del__() method.
Since it is written in Python, the __del__() method can execute
arbitrary Python code. Could it perhaps do something to invalidate
the reference to `item` in bug()? You bet! Assuming
that the list passed into bug() is accessible to the
__del__() method, it could execute a statement to the effect of
"del list[0]", and assuming this was the last reference to that
object, it would free the memory associated with it, thereby
invalidating `item`.
The solution, once you know the source of the problem, is easy:
temporarily increment the reference count. The correct version of the
function reads:
```text

void
no_bug(PyObject *list)
{
PyObject *item = PyList_GetItem(list, 0);

Py_INCREF(item);
PyList_SetItem(list, 1, PyInt_FromLong(0L));
PyObject_Print(item, stdout, 0);
Py_DECREF(item);
}
```
This is a true story. An older version of Python contained variants
of this bug and someone spent a considerable amount of time in a C
debugger to figure out why his __del__() methods would fail...
The second case of problems with a borrowed reference is a variant
involving threads. Normally, multiple threads in the Python
interpreter can't get in each other's way, because there is a global
lock protecting Python's entire object space. However, it is possible
to temporarily release this lock using the macro
Py_BEGIN_ALLOW_THREADS, and to re-acquire it using
Py_END_ALLOW_THREADS. This is common around blocking
I/O calls, to let other threads use the processor while waiting for
the I/O to complete. Obviously, the following function has the same
problem as the previous one:
```text

void
bug(PyObject *list)
{
PyObject *item = PyList_GetItem(list, 0);
Py_BEGIN_ALLOW_THREADS
...some blocking I/O call...
Py_END_ALLOW_THREADS
PyObject_Print(item, stdout, 0); /* BUG! */
}
```