3 Writing the Setup Script

simple-example.html | dist.html | setup-config.html | Distributing Python Modules
---
- 3.1 Listing whole packages (setup-script.html#SECTION000310000000000000000)
  3.2 Listing individual modules (setup-script.html#SECTION000320000000000000000)
  3.3 Describing extension modules (setup-script.html#SECTION000330000000000000000)
  - 3.3.1 Extension names and packages (setup-script.html#SECTION000331000000000000000)
  3.3.2 Extension source files (setup-script.html#SECTION000332000000000000000)
  3.3.3 Preprocessor options (setup-script.html#SECTION000333000000000000000)
  3.3.4 Library options (setup-script.html#SECTION000334000000000000000)
  3.3.5 Other options (setup-script.html#SECTION000335000000000000000)
  3.4 Listing scripts (setup-script.html#SECTION000340000000000000000)
  3.5 Listing additional files (setup-script.html#SECTION000350000000000000000)
---
# 3 Writing the Setup Script
The setup script is the centre of all activity in building,
distributing, and installing modules using the Distutils. The main
purpose of the setup script is to describe your module distribution to
the Distutils, so that the various commands that operate on your modules
do the right thing. As we saw in section 2.1 (simple-example.html#simple-example) above,
the setup script consists mainly of a call to setup(), and
most information supplied to the Distutils by the module developer is
supplied as keyword arguments to setup().
Here's a slightly more involved example, which we'll follow for the next
couple of sections: the Distutils' own setup script. (Keep in mind that
although the Distutils are included with Python 1.6 and later, they also
have an independent existence so that Python 1.5.2 users can use them to
install other module distributions. The Distutils' own setup script,
shown here, is used to install the package into Python 1.5.2.)
```text

#!/usr/bin/env python

from distutils.core import setup

setup(name="Distutils",
version="1.0",
description="Python Distribution Utilities",
author="Greg Ward",
author_email="gward@python.net",
url="http://www.python.org/sigs/distutils-sig/",
packages=['distutils', 'distutils.command'],
)
```
There are only two differences between this and the trivial one-file
distribution presented in section 2.1 (simple-example.html#simple-example): more
meta-data, and the specification of pure Python modules by package,
rather than by module. This is important since the Distutils consist of
a couple of dozen modules split into (so far) two packages; an explicit
list of every module would be tedious to generate and difficult to
maintain.
Note that any pathnames (files or directories) supplied in the setup
script should be written using the Unix convention, i.e.
slash-separated. The Distutils will take care of converting this
platform-neutral representation into whatever is appropriate on your
current platform before actually using the pathname. This makes your
setup script portable across operating systems, which of course is one
of the major goals of the Distutils. In this spirit, all pathnames in
this document are slash-separated (MacOS programmers should keep in
mind that the absence of a leading slash indicates a relative
path, the opposite of the MacOS convention with colons).
This, of course, only applies to pathnames given to Distutils functions.
If you, for example, use standard python functions such as glob.glob
or os.listdir to specify files, you should be careful to write portable
code instead of hardcoding path separators:
```text

glob.glob(os.path.join('mydir', 'subdir', '*.html'))
os.listdir(os.path.join('mydir', 'subdir'))
```
## 3.1 Listing whole packages
The packages option tells the Distutils to process (build,
distribute, install, etc.) all pure Python modules found in each package
mentioned in the packages list. In order to do this, of
course, there has to be a correspondence between package names and
directories in the filesystem. The default correspondence is the most
obvious one, i.e. package distutils is found in the directory
distutils relative to the distribution root. Thus, when you say
`packages = ['foo']` in your setup script, you are promising that
the Distutils will find a file foo/__init__.py (which might
be spelled differently on your system, but you get the idea) relative to
the directory where your setup script lives. (If you break this
promise, the Distutils will issue a warning but process the broken
package anyways.)
If you use a different convention to lay out your source directory,
that's no problem: you just have to supply the package_dir
option to tell the Distutils about your convention. For example, say
you keep all Python source under lib, so that modules in the
``root package'' (i.e., not in any package at all) are right in
lib, modules in the foo package are in lib/foo,
and so forth. Then you would put
```text

package_dir = {'': 'lib'}
```
in your setup script. (The keys to this dictionary are package names,
and an empty package name stands for the root package. The values are
directory names relative to your distribution root.) In this case, when
you say `packages = ['foo']`, you are promising that the file
lib/foo/__init__.py exists.
Another possible convention is to put the foo package right in
lib, the foo.bar package in lib/bar, etc. This
would be written in the setup script as
```text

package_dir = {'foo': 'lib'}
```
A `package : dir` entry in the package_dir
dictionary implicitly applies to all packages below package, so
the foo.bar case is automatically handled here. In this
example, having `packages = ['foo', 'foo.bar']` tells the Distutils
to look for lib/__init__.py and
lib/bar/__init__.py. (Keep in mind that although
package_dir applies recursively, you must explicitly list all
packages in packages: the Distutils will not recursively
scan your source tree looking for any directory with an
__init__.py file.)
## 3.2 Listing individual modules
For a small module distribution, you might prefer to list all modules
rather than listing packages--especially the case of a single module
that goes in the ``root package'' (i.e., no package at all). This
simplest case was shown in section 2.1 (simple-example.html#simple-example); here is a
slightly more involved example:
```text

py_modules = ['mod1', 'pkg.mod2']
```
This describes two modules, one of them in the ``root'' package, the
other in the pkg package. Again, the default package/directory
layout implies that these two modules can be found in mod1.py and
pkg/mod2.py, and that pkg/__init__.py exists as well.
And again, you can override the package/directory correspondence using
the package_dir option.
## 3.3 Describing extension modules
Just as writing Python extension modules is a bit more complicated than
writing pure Python modules, describing them to the Distutils is a bit
more complicated. Unlike pure modules, it's not enough just to list
modules or packages and expect the Distutils to go out and find the
right files; you have to specify the extension name, source file(s), and
any compile/link requirements (include directories, libraries to link
with, etc.).
All of this is done through another keyword argument to
setup(), the extensions option. extensions
is just a list of Extension instances, each of which describes a
single extension module. Suppose your distribution includes a single
extension, called foo and implemented by foo.c. If no
additional instructions to the compiler/linker are needed, describing
this extension is quite simple:
```text

Extension("foo", ["foo.c"])
```
The Extension class can be imported from
distutils.core, along with setup(). Thus, the setup
script for a module distribution that contains only this one extension
and nothing else might be:
```text

from distutils.core import setup, Extension
setup(name="foo", version="1.0",
ext_modules=[Extension("foo", ["foo.c"])])
```
The Extension class (actually, the underlying extension-building
machinery implemented by the `build_ext` command) supports a
great deal of flexibility in describing Python extensions, which is
explained in the following sections.
### 3.3.1 Extension names and packages
The first argument to the Extension constructor is always the
name of the extension, including any package names. For example,
```text

Extension("foo", ["src/foo1.c", "src/foo2.c"])
```
describes an extension that lives in the root package, while
```text

Extension("pkg.foo", ["src/foo1.c", "src/foo2.c"])
```
describes the same extension in the pkg package. The source
files and resulting object code are identical in both cases; the only
difference is where in the filesystem (and therefore where in Python's
namespace hierarchy) the resulting extension lives.
If you have a number of extensions all in the same package (or all under
the same base package), use the ext_package keyword argument
to setup(). For example,
```text

setup(...
ext_package="pkg",
ext_modules=[Extension("foo", ["foo.c"]),
Extension("subpkg.bar", ["bar.c"])]
)
```
will compile foo.c to the extension pkg.foo, and
bar.c to pkg.subpkg.bar.
### 3.3.2 Extension source files
The second argument to the Extension constructor is a list of
source files. Since the Distutils currently only support C/C++
extensions, these are normally C/C++ source files. (Be sure to use
appropriate extensions to distinguish C++ source files: .cc and
.cpp seem to be recognized by both Unix and Windows compilers.)
However, you can also include SWIG interface (.i) files in the
list; the `build_ext` command knows how to deal with SWIG
extensions: it will run SWIG on the interface file and compile the
resulting C/C++ file into your extension.
** SWIG support is rough around the edges and largely untested;
especially SWIG support of C++ extensions! Explain in more detail
here when the interface firms up. **
On some platforms, you can include non-source files that are processed
by the compiler and included in your extension. Currently, this just
means Windows message text (.mc) files and resource definition
(.rc) files for Visual C++. These will be compiled to binary resource
(.res) files and linked into the executable.
### 3.3.3 Preprocessor options
Three optional arguments to Extension will help if you need to
specify include directories to search or preprocessor macros to
define/undefine: `include_dirs`, `define_macros`, and
`undef_macros`.
For example, if your extension requires header files in the
include directory under your distribution root, use the
`include_dirs` option:
```text

Extension("foo", ["foo.c"], include_dirs=["include"])
```
You can specify absolute directories there; if you know that your
extension will only be built on Unix systems with X11R6 installed to
/usr, you can get away with
```text

Extension("foo", ["foo.c"], include_dirs=["/usr/include/X11"])
```
You should avoid this sort of non-portable usage if you plan to
distribute your code: it's probably better to write your code to include
(e.g.) `<X11/Xlib.h>`.
If you need to include header files from some other Python extension,
you can take advantage of the fact that the Distutils install extension
header files in a consistent way. For example, the Numerical Python
header files are installed (on a standard Unix installation) to
/usr/local/include/python1.5/Numerical. (The exact location will
differ according to your platform and Python installation.) Since the
Python include directory--/usr/local/include/python1.5 in this
case--is always included in the search path when building Python
extensions, the best approach is to include (e.g.)
`<Numerical/arrayobject.h>`. If you insist on putting the
Numerical include directory right into your header search path,
though, you can find that directory using the Distutils
sysconfig module:
```text

from distutils.sysconfig import get_python_inc
incdir = os.path.join(get_python_inc(plat_specific=1), "Numerical")
setup(...,
Extension(..., include_dirs=[incdir]))
```
Even though this is quite portable--it will work on any Python
installation, regardless of platform--it's probably easier to just
write your C code in the sensible way.
You can define and undefine pre-processor macros with the
`define_macros` and `undef_macros` options.
`define_macros` takes a list of `(name, value)` tuples, where
`name` is the name of the macro to define (a string) and
`value` is its value: either a string or `None`. (Defining a
macro `FOO` to `None` is the equivalent of a bare
`#define FOO` in your C source: with most compilers, this sets
`FOO` to the string `1`.) `undef_macros` is just
a list of macros to undefine.
For example:
```text

Extension(...,
define_macros=[('NDEBUG', '1')],
('HAVE_STRFTIME', None),
undef_macros=['HAVE_FOO', 'HAVE_BAR'])
```
is the equivalent of having this at the top of every C source file:
```text

#define NDEBUG 1
#define HAVE_STRFTIME
#undef HAVE_FOO
#undef HAVE_BAR
```
### 3.3.4 Library options
You can also specify the libraries to link against when building your
extension, and the directories to search for those libraries. The
`libraries` option is a list of libraries to link against,
`library_dirs` is a list of directories to search for libraries at
link-time, and `runtime_library_dirs` is a list of directories to
search for shared (dynamically loaded) libraries at run-time.
For example, if you need to link against libraries known to be in the
standard library search path on target systems
```text

Extension(...,
libraries=["gdbm", "readline"])
```
If you need to link with libraries in a non-standard location, you'll
have to include the location in `library_dirs`:
```text

Extension(...,
library_dirs=["/usr/X11R6/lib"],
libraries=["X11", "Xt"])
```
(Again, this sort of non-portable construct should be avoided if you
intend to distribute your code.)
** Should mention clib libraries here or somewhere else! **
### 3.3.5 Other options
There are still some other options which can be used to handle special
cases.
The extra_objects option is a list of object files to be passed
to the linker. These files must not have extensions, as the default
extension for the compiler is used.
extra_compile_args and extra_link_args can be used
to specify additional command line options for the compiler resp.
the linker command line.
export_symbols is only useful on windows, it can contain a list
of symbols (functions or variables) to be exported. This option
is not needed when building compiled extensions: the `initmodule`
function will automatically be added to the exported symbols list
by Distutils.
## 3.4 Listing scripts
So far we have been dealing with pure and non-pure Python modules,
which are usually not run by themselves but imported by scripts.
Scripts are files containing Python source code, indended to be started
from the command line.
Distutils doesn't provide much functionality for the scripts: the only
support Distutils gives is to adjust the first line of the script
if it starts with `#!` and contains the word ``python'' to refer
to the current interpreter location.
The scripts option simply is a list of files to be handled
in this way.
## 3.5 Listing additional files
The data_files option can be used to specify additional
files needed by the module distribution: configuration files,
data files, anything which does not fit in the previous categories.
data_files specify a sequence of `(directory, files)`
pairs in the following way:
```text

setup(...
data_files=[('bitmaps', ['bm/b1.gif', 'bm/b2.gif']),
('config', ['cfg/data.cfg'])])
```
Note that you can specify the directory names where the data files
will be installed, but you cannot rename the data files themselves.
You can specify the data_files options as a simple sequence
of files without specifying a target directory, but this is not recommended,
and the `install` command will print a warning in this case.
To install data files directly in the target directory, an empty
string should be given as the directory.


2 Concepts & Terminology

intro.html | dist.html | setup-script.html | Distributing Python Modules
---
- 2.1 A simple example (simple-example.html#SECTION000210000000000000000)
  2.2 General Python terminology (simple-example.html#SECTION000220000000000000000)
  2.3 Distutils-specific terminology (simple-example.html#SECTION000230000000000000000)
---
# 2 Concepts & Terminology
Using the Distutils is quite simple, both for module developers and for
users/administrators installing third-party modules. As a developer,
your responsibilities (apart from writing solid, well-documented and
well-tested code, of course!) are:
- write a setup script (setup.py by convention)
- (optional) write a setup configuration file
- create a source distribution
- (optional) create one or more built (binary) distributions
Each of these tasks is covered in this document.
Not all module developers have access to a multitude of platforms, so
it's not always feasible to expect them to create a multitude of built
distributions. It is hoped that a class of intermediaries, called
packagers, will arise to address this need. Packagers will take
source distributions released by module developers, build them on one or
more platforms, and release the resulting built distributions. Thus,
users on the most popular platforms will be able to install most popular
Python module distributions in the most natural way for their platform,
without having to run a single setup script or compile a line of code.
## 2.1 A simple example
The setup script is usually quite simple, although since it's written in
Python, there are no arbitrary limits to what you can do with
it.1 (#foot685) If
all you want to do is distribute a module called foo, contained
in a file foo.py, then your setup script can be as little as
this:
```text

from distutils.core import setup
setup(name="foo",
version="1.0",
py_modules=["foo"])
```
Some observations:
- most information that you supply to the Distutils is supplied as
keyword arguments to the setup() function
- those keyword arguments fall into two categories: package
meta-data (name, version number) and information about what's in the
package (a list of pure Python modules, in this case)
- modules are specified by module name, not filename (the same will
hold true for packages and extensions)
- it's recommended that you supply a little more meta-data, in
particular your name, email address and a URL for the project
(see section 3 (setup-script.html#setup-script) for an example)
To create a source distribution for this module, you would create a
setup script, setup.py, containing the above code, and run:
```text

python setup.py sdist
```
which will create an archive file (e.g., tarball on Unix, ZIP file on
Windows) containing your setup script, setup.py, and your module,
foo.py. The archive file will be named Foo-1.0.tar.gz (or
.zip), and will unpack into a directory Foo-1.0.
If an end-user wishes to install your foo module, all she has
to do is download Foo-1.0.tar.gz (or .zip), unpack it,
and--from the Foo-1.0 directory--run
```text

python setup.py install
```
which will ultimately copy foo.py to the appropriate directory
for third-party modules in their Python installation.
This simple example demonstrates some fundamental concepts of the
Distutils: first, both developers and installers have the same basic
user interface, i.e. the setup script. The difference is which
Distutils commands they use: the `sdist` command is
almost exclusively for module developers, while `install` is
more often for installers (although most developers will want to install
their own code occasionally).
If you want to make things really easy for your users, you can create
one or more built distributions for them. For instance, if you are
running on a Windows machine, and want to make things easy for other
Windows users, you can create an executable installer (the most
appropriate type of built distribution for this platform) with the
`bdist_wininst` command. For example:
```text

python setup.py bdist_wininst
```
will create an executable installer, Foo-1.0.win32.exe, in the
current directory.
Currently (Distutils 0.9.2), the only other useful built
distribution format is RPM, implemented by the `bdist_rpm`
command. For example, the following command will create an RPM file
called Foo-1.0.noarch.rpm:
```text

python setup.py bdist_rpm
```
(This uses the `rpm` command, so has to be run on an RPM-based
system such as Red Hat Linux, SuSE Linux, or Mandrake Linux.)
You can find out what distribution formats are available at any time by
running
```text

python setup.py bdist --help-formats
```
## 2.2 General Python terminology
If you're reading this document, you probably have a good idea of what
modules, extensions, and so forth are. Nevertheless, just to be sure
that everyone is operating from a common starting point, we offer the
following glossary of common Python terms:
module: the basic unit of code reusability in Python: a block of
code imported by some other code. Three types of modules concern us
here: pure Python modules, extension modules, and packages.
pure Python module: a module written in Python and contained in a
single .py file (and possibly associated .pyc and/or
.pyo files). Sometimes referred to as a ``pure module.''
extension module: a module written in the low-level language of
the Python implementation: C/C++ for Python, Java for JPython.
Typically contained in a single dynamically loadable pre-compiled
file, e.g. a shared object (.so) file for Python extensions on
Unix, a DLL (given the .pyd extension) for Python extensions
on Windows, or a Java class file for JPython extensions. (Note that
currently, the Distutils only handles C/C++ extensions for Python.)
package: a module that contains other modules; typically contained
in a directory in the filesystem and distinguished from other
directories by the presence of a file __init__.py.
root package: the root of the hierarchy of packages. (This isn't
really a package, since it doesn't have an __init__.py
file. But we have to call it something.) The vast majority of the
standard library is in the root package, as are many small, standalone
third-party modules that don't belong to a larger module collection.
Unlike regular packages, modules in the root package can be found in
many directories: in fact, every directory listed in `sys.path`
can contribute modules to the root package.
## 2.3 Distutils-specific terminology
The following terms apply more specifically to the domain of
distributing Python modules using the Distutils:
module distribution: a collection of Python modules distributed
together as a single downloadable resource and meant to be installed
en masse. Examples of some well-known module distributions are
Numeric Python, PyXML, PIL (the Python Imaging Library), or
mxDateTime. (This would be called a package, except that term
is already taken in the Python context: a single module distribution
may contain zero, one, or many Python packages.)
pure module distribution: a module distribution that contains only
pure Python modules and packages. Sometimes referred to as a ``pure
distribution.''
non-pure module distribution: a module distribution that contains
at least one extension module. Sometimes referred to as a ``non-pure
distribution.''
distribution root: the top-level directory of your source tree (or
source distribution); the directory where setup.py exists and
is run from


5 Creating a Source Distribution

setup-config.html | dist.html | creating-wininst.html | Distributing Python Modules
---
- 5.1 Specifying the files to distribute (source-dist.html#SECTION000510000000000000000)
  5.2 Manifest-related options (source-dist.html#SECTION000520000000000000000)
---
# 5 Creating a Source Distribution
As shown in section 2.1 (simple-example.html#simple-example), you use the
`sdist` command to create a source distribution. In the
simplest case,
```text

python setup.py sdist
```
(assuming you haven't specified any `sdist` options in the setup
script or config file), `sdist` creates the archive of the
default format for the current platform. The default format is gzip'ed
tar file (.tar.gz) on Unix, and ZIP file on Windows.
** no MacOS support here **
You can specify as many formats as you like using the
--formats option, for example:
```text

python setup.py sdist --formats=gztar,zip
```
to create a gzipped tarball and a zip file. The available formats are:
Notes:
(1): default on Windows
(2): default on Unix
(3): requires either external zip utility or
zipfile module (not part of the standard Python library)
(4): requires external utilities: tar and possibly one
of gzip, bzip2, or compress
## 5.1 Specifying the files to distribute
If you don't supply an explicit list of files (or instructions on how to
generate one), the `sdist` command puts a minimal default set
into the source distribution:
- all Python source files implied by the py_modules and
packages options
- all C source files mentioned in the ext_modules or
libraries options (** getting C library sources currently
broken - no get_source_files() method in build_clib.py! **)
- anything that looks like a test script: test/test*.py
(currently, the Distutils don't do anything with test scripts except
include them in source distributions, but in the future there will be
a standard for testing Python module distributions)
- README.txt (or README), setup.py (or whatever
you called your setup script), and setup.cfg
Sometimes this is enough, but usually you will want to specify
additional files to distribute. The typical way to do this is to write
a manifest template, called MANIFEST.in by default. The
manifest template is just a list of instructions for how to generate
your manifest file, MANIFEST, which is the exact list of files to
include in your source distribution. The `sdist` command
processes this template and generates a manifest based on its
instructions and what it finds in the filesystem.
If you prefer to roll your own manifest file, the format is simple: one
filename per line, regular files (or symlinks to them) only. If you do
supply your own MANIFEST, you must specify everything: the
default set of files described above does not apply in this case.
The manifest template has one command per line, where each command
specifies a set of files to include or exclude from the source
distribution. For an example, again we turn to the Distutils' own
manifest template:
```text

include *.txt
recursive-include examples *.txt *.py
prune examples/sample?/build
```
The meanings should be fairly clear: include all files in the
distribution root matching `*.txt`, all files anywhere under the
examples directory matching `*.txt` or `*.py`, and
exclude all directories matching `examples/sample?/build`. All of
this is done after the standard include set, so you can exclude
files from the standard set with explicit instructions in the manifest
template. (Or, you can use the --no-defaults option to
disable the standard set entirely.) There are several other commands
available in the manifest template mini-language; see
section 7.2 (sdist-cmd.html#sdist-cmd).
The order of commands in the manifest template matters: initially, we
have the list of default files as described above, and each command in
the template adds to or removes from that list of files. Once we have
fully processed the manifest template, we remove files that should not
be included in the source distribution:
- all files in the Distutils ``build'' tree (default build/)
- all files in directories named RCS or CVS
Now we have our complete list of files, which is written to the manifest
for future reference, and then used to build the source distribution
archive(s).
You can disable the default set of included files with the
--no-defaults option, and you can disable the standard
exclude set with --no-prune.
Following the Distutils' own manifest template, let's trace how the
`sdist` command builds the list of files to include in the
Distutils source distribution:
1. include all Python source files in the distutils and
distutils/command subdirectories (because packages
corresponding to those two directories were mentioned in the
packages option in the setup script--see
section 3 (setup-script.html#setup-script))
2. include README.txt, setup.py, and setup.cfg
(standard files)
3. include test/test*.py (standard files)
4. include *.txt in the distribution root (this will find
README.txt a second time, but such redundancies are weeded out
later)
5. include anything matching *.txt or *.py in the
sub-tree under examples,
6. exclude all files in the sub-trees starting at directories
matching examples/sample?/build--this may exclude files
included by the previous two steps, so it's important that the
`prune` command in the manifest template comes after the
`recursive-include` command
7. exclude the entire build tree, and any RCS or
CVS directories
Just like in the setup script, file and directory names in the manifest
template should always be slash-separated; the Distutils will take care
of converting them to the standard representation on your platform.
That way, the manifest template is portable across operating systems.
## 5.2 Manifest-related options
The normal course of operations for the `sdist` command is as
follows:
- if the manifest file, MANIFEST doesn't exist, read
MANIFEST.in and create the manifest
- if neither MANIFEST nor MANIFEST.in exist, create a
manifest with just the default file set3 (#foot692)
- if either MANIFEST.in or the setup script (setup.py)
are more recent than MANIFEST, recreate MANIFEST by
reading MANIFEST.in
- use the list of files now in MANIFEST (either just
generated or read in) to create the source distribution archive(s)
There are a couple of options that modify this behaviour. First, use
the --no-defaults and --no-prune to
disable the standard ``include'' and ``exclude'' sets.4 (#foot693)
Second, you might want to force the manifest to be regenerated--for
example, if you have added or removed files or directories that match an
existing pattern in the manifest template, you should regenerate the
manifest:
```text

python setup.py sdist --force-manifest
```
Or, you might just want to (re)generate the manifest, but not create a
source distribution:
```text

python setup.py sdist --manifest-only
```
--manifest-only implies --force-manifest.
-o is a shortcut for --manifest-only, and
-f for --force-manifest.


About this document ...

discussion.html | doc.html | Documenting Python | contents.html
---
# About this document ...
Documenting Python,
April 15, 2001, Release 2.1
This document was generated using the LaTeX2HTML (http://saftsack.fs.uni-bayreuth.de/~latex2ht/) translator.
LaTeX2HTML (http://saftsack.fs.uni-bayreuth.de/~latex2ht/) is Copyright ©
1993, 1994, 1995, 1996, 1997, Nikos
Drakos (http://cbl.leeds.ac.uk/nikos/personal.html), Computer Based Learning Unit, University of
Leeds, and Copyright © 1997, 1998, Ross
Moore (http://www.maths.mq.edu.au/~ross/), Mathematics Department, Macquarie University,
Sydney.
The application of LaTeX2HTML (http://saftsack.fs.uni-bayreuth.de/~latex2ht/) to the Python
documentation has been heavily tailored by Fred L. Drake,
Jr. Original navigation icons were contributed by Christopher
Petrilli.
---
## Comments and Questions
General comments and questions regarding this document should
be sent by email to python-docs@python.org (mailto:python-docs@python.org). If you find specific errors in
this document, please report the bug at the Python Bug
Tracker (http://sourceforge.net/bugs/?group_id=5470) at SourceForge (http://sourceforge.net/).
Questions regarding how to use the information in this
document should be sent to the Python news group, comp.lang.python (news:comp.lang.python), or the Python mailing list (http://www.python.org/mailman/listinfo/python-list) (which is gated to the newsgroup and
carries the same content).
For any of these channels, please be sure not to send HTML email.
Thanks.
---
discussion.html | doc.html | Documenting Python | contents.html
---
Release 2.1, documentation updated on April 15, 2001.


4 Document Classes

node6.html | doc.html | node8.html | Documenting Python | contents.html
---
# 4 Document Classes
Two LATEX document classes are defined specifically for use with
the Python documentation. The `manual` class is for large
documents which are sectioned into chapters, and the `howto`
class is for smaller documents.
The `manual` documents are larger and are used for most of the
standard documents. This document class is based on the standard
LATEX `report` class and is formatted very much like a long
technical report. The Python Reference
Manual (../ref/ref.html) is a good example of a `manual` document, and the
Python Library Reference (../lib/lib.html) is a large
example.
The `howto` documents are shorter, and don't have the large
structure of the `manual` documents. This class is based on
the standard LATEX `article` class and is formatted somewhat
like the Linux Documentation Project's ``HOWTO'' series as done
originally using the LinuxDoc software. The original intent for the
document class was that it serve a similar role as the LDP's HOWTO
series, but the applicability of the class turns out to be somewhat
broader. This class is used for ``how-to'' documents (this
document is an example) and for shorter reference manuals for small,
fairly cohesive module libraries. Examples of the later use include
the standard Macintosh Library Modules (../mac/mac.html)
and
Using
Kerberos from Python (http://starship.python.net/crew/fdrake/manuals/krb5py/krb5py.html), which contains reference material for an
extension package. These documents are roughly equivalent to a
single chapter from a larger work.


Contents

doc.html | doc.html | node2.html | Documenting Python
---
## Contents
Table of Contents
- Contents (contents.html)
  1 Introduction (node2.html)
  2 Directory Structure (node3.html)
  3 LATEX Primer (latex-primer.html)
  - 3.1 Syntax (node5.html)
  3.2 Hierarchical Structure (node6.html)
  4 Document Classes (classes.html)
  5 Special Markup Constructs (node8.html)
  - 5.1 Markup for the Preamble (preamble-info.html)
  5.2 Meta-information Markup (meta-info.html)
  5.3 Information Units (info-units.html)
  5.4 Showing Code Examples (node12.html)
  5.5 Inline Markup (node13.html)
  5.6 Module-specific Markup (node14.html)
  5.7 Library-level Markup (node15.html)
  5.8 Table Markup (node16.html)
  5.9 Reference List Markup (references.html)
  5.10 Index-generating Markup (indexing.html)
  6 Special Names (node19.html)
  7 Processing Tools (node20.html)
  - 7.1 External Tools (node21.html)
  7.2 Internal Tools (node22.html)
  8 Future Directions (futures.html)
  - 8.1 Structured Documentation (structured.html)
  8.2 Discussion Forums (discussion.html)
  About this document ... (about.html)
End of Table of Contents


8.2 Discussion Forums

structured.html | futures.html | about.html | Documenting Python | contents.html
---
## 8.2 Discussion Forums
Discussion of the future of the Python documentation and related
topics takes place in the Documentation Special Interest Group, or
``Doc-SIG.'' Information on the group, including mailing list
archives and subscription information, is available at
http://www.python.org/sigs/doc-sig/. The SIG is open to all
interested parties.
Comments and bug reports on the standard documents should be sent
to python-docs@python.org. This may include comments
about formatting, content, grammatical and spelling errors, or
this document. You can also send comments on this document
directly to the author at fdrake@acm.org.


Documenting Python

../index.html | contents.html | Documenting Python | contents.html
---
# Documenting Python
Fred L. Drake, Jr.
PythonLabs
E-mail: fdrake@acm.org
Release 2.1
April 15, 2001
### Abstract:
The Python language has a substantial body of
documentation, much of it contributed by various authors. The markup
used for the Python documentation is based on LATEX and requires a
significant set of macros written specifically for documenting Python.
This document describes the macros introduced to support Python
documentation and how they should be used to support a wide range of
output formats.
This document describes the document classes and special markup used
in the Python documentation. Authors may use this guide, in
conjunction with the template files provided with the
distribution, to create or maintain whole documents or sections.


8 Future Directions

node22.html | doc.html | structured.html | Documenting Python | contents.html
---
# 8 Future Directions
The history of the Python documentation is full of changes, most of
which have been fairly small and evolutionary. There has been a
great deal of discussion about making large changes in the markup
languages and tools used to process the documentation. This section
deals with the nature of the changes and what appears to be the most
likely path of future development.


Documenting Python


5.10 Index-generating Markup

references.html | node8.html | node19.html | Documenting Python | contents.html
---
## 5.10 Index-generating Markup
Effective index generation for technical documents can be very
difficult, especially for someone familiar with the topic but not
the creation of indexes. Much of the difficulty arises in the
area of terminology: including the terms an expert would use for a
concept is not sufficient. Coming up with the terms that a novice
would look up is fairly difficult for an author who, typically, is
an expert in the area she is writing on.
The truly difficult aspects of index generation are not areas with
which the documentation tools can help. However, ease
of producing the index once content decisions are made is within
the scope of the tools. Markup is provided which the processing
software is able to use to generate a variety of kinds of index
entry with minimal effort. Additionally, many of the environments
described in section 5.3 (info-units.html#info-units), ``Information Units,'' will
generate appropriate entries into the general and module indexes.
The following macro can be used to control the generation of index
data, and should be used in the document preamble:
There are a number of macros that are useful for adding index
entries for particular concepts, many of which are specific to
programming languages or even Python.
Additional macros are provided which are useful for conveniently
creating general index entries which should appear at many places
in the index by rotating a list of words. These are simple macros
that simply use \index to build some number of index
entries. Index entries build using these macros contain both
primary and secondary text.


5.3 Information Units

meta-info.html | node8.html | node12.html | Documenting Python | contents.html
---
## 5.3 Information Units
XXX Explain terminology, or come up with something more ``lay.''
There are a number of environments used to describe specific
features provided by modules. Each environment requires
parameters needed to provide basic information about what is being
described, and the environment content should be the description.
Most of these environments make entries in the general index (if
one is being produced for the document); if no index entry is
desired, non-indexing variants are available for many of these
environments. The environments have names of the form
`feature desc`, and the non-indexing variants are named
`feature descni`. The available variants are explicitly
included in the list below.
For each of these environments, the first parameter, name,
provides the name by which the feature is accessed.
Environments which describe features of objects within a module,
such as object methods or data attributes, allow an optional
type name parameter. When the feature is an attribute of
class instances, type name only needs to be given if the
class was not the most recently described class in the module; the
name value from the most recent \classdesc is implied.
For features of built-in or extension types, the type name
value should always be provided. Another special case includes
methods and members of general ``protocols,'' such as the
formatter and writer protocols described for the
formatter module: these may be documented without any
specific implementation classes, and will always require the
type name parameter to be provided.


3 LATEX Primer

node3.html | doc.html | node5.html | Documenting Python | contents.html
---
# 3 LATEX Primer
This section is a brief introduction to LATEX concepts and
syntax, to provide authors enough information to author documents
productively without having to become ``TEXnicians.''
Perhaps the most important concept to keep in mind while marking up
Python documentation is that while TEX is unstructured, LATEX was
designed as a layer on top of TEX which specifically supports
structured markup. The Python-specific markup is intended to extend
the structure provided by standard LATEX document classes to
support additional information specific to Python.
LATEX documents contain two parts: the preamble and the body.
The preamble is used to specify certain metadata about the document
itself, such as the title, the list of authors, the date, and the
class the document belongs to. Additional information used
to control index generation and the use of bibliographic databases
can also be placed in the preamble. For most authors, the preamble
can be most easily created by copying it from an existing document
and modifying a few key pieces of information.
The class of a document is used to place a document within a
broad category of documents and set some fundamental formatting
properties. For Python documentation, two classes are used: the
`manual` class and the `howto` class. These classes also
define the additional markup used to document Python concepts and
structures. Specific information about these classes is provided in
section 4 (classes.html#classes), ``Document Classes,'' below. The first thing
in the preamble is the declaration of the document's class.
After the class declaration, a number of macros are used to
provide further information about the document and setup any
additional markup that is needed. No output is generated from the
preamble; it is an error to include free text in the preamble
because it would cause output.
The document body follows the preamble. This contains all the
printed components of the document marked up structurally. Generic
LATEX structures include hierarchical sections


5.2 Meta-information Markup

preamble-info.html | node8.html | info-units.html | Documenting Python | contents.html
---
## 5.2 Meta-information Markup


5.4 Showing Code Examples

info-units.html | node8.html | node13.html | Documenting Python | contents.html
---
## 5.4 Showing Code Examples
Examples of Python source code or interactive sessions are
represented as \verbatim environments. This environment
is a standard part of LATEX. It is important to only use
spaces for indentation in code examples since TEX drops tabs
instead of converting them to spaces.
Representing an interactive session requires including the prompts
and output along with the Python code. No special markup is
required for interactive sessions. After the last line of input
or output presented, there should not be an ``unused'' primary
prompt; this is an example of what not to do:
```text

>>> 1 + 1
2
>>>
```
Within the \verbatim environment, characters special to
LATEX do not need to be specially marked in any way. The entire
example will be presented in a monospaced font; no attempt at
``pretty-printing'' is made, as the environment must work for
non-Python code and non-code displays. There should be no blank
lines at the top or bottom of any \verbatim display.
The Python Documentation Special Interest Group has discussed a
number of approaches to creating pretty-printed code displays and
interactive sessions; see the Doc-SIG area on the Python Web site
for more information on this topic.


5.5 Inline Markup

node12.html | node8.html | node14.html | Documenting Python | contents.html
---
## 5.5 Inline Markup
The macros described in this section are used to mark just about
anything interesting in the document text. They may be used in
headings (though anything involving hyperlinks should be avoided
there) as well as in the body text.


5.6 Module-specific Markup

node13.html | node8.html | node15.html | Documenting Python | contents.html
---
## 5.6 Module-specific Markup
The markup described in this section is used to provide information
about a module being documented. A typical use of this markup
appears at the top of the section used to document a module. A
typical example might look like this:
```text

\section{\module{spam} ---
Access to the SPAM facility}

\declaremodule{extension}{spam}
\platform{Unix}
\modulesynopsis{Access to the SPAM facility of \UNIX{}.}
\moduleauthor{Jane Doe}{jane.doe@frobnitz.org}
```
Python packages-- collections of modules that can
be described as a unit -- are documented using the same markup as
modules. The name for a module in a package should be typed in
``fully qualified'' form (i.e., it should include the package name).
For example, a module ``foo'' in package ``bar'' should be marked as
"\module{bar.foo}", and the beginning of the reference
section would appear as:
```text

\section{\module{bar.foo} ---
Module from the \module{bar} package}

\declaremodule{extension}{bar.foo}
\modulesynopsis{Nifty module from the \module{bar} package.}
\moduleauthor{Jane Doe}{jane.doe@frobnitz.org}
```
Note that the name of a package is also marked using
\module.


5.7 Library-level Markup

node14.html | node8.html | node16.html | Documenting Python | contents.html
---
## 5.7 Library-level Markup
This markup is used when describing a selection of modules. For
example, the Macintosh Library
Modules (../mac/mac.html) document uses this to help provide an overview of the
modules in the collection, and many chapters in the
Python Library Reference (../lib/lib.html) use it for
the same purpose.


5.8 Table Markup

node15.html | node8.html | references.html | Documenting Python | contents.html
---
## 5.8 Table Markup
There are three general-purpose table environments defined which
should be used whenever possible. These environments are defined
to provide tables of specific widths and some convenience for
formatting. These environments are not meant to be general
replacements for the standard LATEX table environments, but can
be used for an advantage when the documents are processed using
the tools for Python documentation processing. In particular, the
generated HTML looks good! There is also an advantage for the
eventual conversion of the documentation to SGML (see section
8 (futures.html#futures), ``Future Directions'').
Each environment is named \tablecols, where cols
is the number of columns in the table specified in lower-case
Roman numerals. Within each of these environments, an additional
macro, \linecols, is defined, where cols
matches the cols value of the corresponding table
environment. These are supported for cols values of
`ii`, `iii`, and `iv`. These environments are all
built on top of the \tabular environment. Variants based on
the \longtable environment are also provided.
Note that all tables in the standard Python documentation use
vertical lines between columns, and this must be specified in the
markup for each table. A general border around the outside of the
table is not used, but would be the responsibility of the
processor; the document markup should not include an exterior
border.
The \longtable-based variants of the table environments are
formatted with extra space before and after, so should only be
used on tables which are long enough that splitting over multiple
pages is reasonable; tables with fewer than twenty rows should
never by marked using the long flavors of the table environments.
The header row is repeated across the top of each part of the
table.
An additional table-like environment is \synopsistable. The
table generated by this environment contains two columns, and each
row is defined by an alternate definition of
\modulesynopsis. This environment is not normally used by
authors, but is created by the \localmoduletable macro.


6 Special Names

indexing.html | doc.html | node20.html | Documenting Python | contents.html
---
# 6 Special Names
Many special names are used in the Python documentation, including
the names of operating systems, programming languages, standards
bodies, and the like. Many of these were assigned LATEX macros
at some point in the distant past, and these macros lived on long
past their usefulness. In the current markup, these entities are
not assigned any special markup, but the preferred spellings are
given here to aid authors in maintaining the consistency of
presentation in the Python documentation.
POSIX: The name assigned to a particular group of standards. This is
always uppercase.
Python: The name of our favorite programming language is always
capitalized.
Unicode: The name of a character set and matching encoding. This is
always written capitalized.


1 Introduction

contents.html | doc.html | node3.html | Documenting Python | contents.html
---
# 1 Introduction
Python's documentation has long been considered to be good for a
free programming language. There are a number of reasons for this,
the most important being the early commitment of Python's creator,
Guido van Rossum, to providing documentation on the language and its
libraries, and the continuing involvement of the user community in
providing assistance for creating and maintaining documentation.
The involvement of the community takes many forms, from authoring to
bug reports to just plain complaining when the documentation could
be more complete or easier to use. All of these forms of input from
the community have proved useful during the time I've been involved
in maintaining the documentation.
This document is aimed at authors and potential authors of
documentation for Python. More specifically, it is for people
contributing to the standard documentation and developing additional
documents using the same tools as the standard documents. This
guide will be less useful for authors using the Python documentation
tools for topics other than Python, and less useful still for
authors not using the tools at all.
The material in this guide is intended to assist authors using the
Python documentation tools. It includes information on the source
distribution of the standard documentation, a discussion of the
document types, reference material on the markup defined in the
document classes, a list of the external tools needed for processing
documents, and reference material on the tools provided with the
documentation resources. At the end, there is also a section
discussing future directions for the Python documentation and where
to turn for more information.


7 Processing Tools

node19.html | doc.html | node21.html | Documenting Python | contents.html
---
# 7 Processing Tools


7.1 External Tools

node20.html | node20.html | node22.html | Documenting Python | contents.html
---
## 7.1 External Tools
Many tools are needed to be able to process the Python
documentation if all supported formats are required. This
section lists the tools used and when each is required. Consult
the Doc/README file to see if there are specific version
requirements for any of these.
dvips: This program is a typical part of TEX installations. It is
used to generate PostScript from the ``device independent''
.dvi files. It is needed for the conversion to
PostScript.
emacs: Emacs is the kitchen sink of programmers' editors, and a damn
fine kitchen sink it is. It also comes with some of the
processing needed to support the proper menu structures for
Texinfo documents when an info conversion is desired. This is
needed for the info conversion. Using xemacs
instead of FSF emacs may lead to instability in the
conversion, but that's because nobody seems to maintain the
Emacs Texinfo code in a portable manner.
latex: This is a world-class typesetter by Donald Knuth. It is used
for the conversion to PostScript, and is needed for the HTML
conversion as well (LATEX2HTML requires one of the
intermediate files it creates).
latex2html: Probably the longest Perl script anyone ever attempted to
maintain. This converts LATEX documents to HTML documents,
and does a pretty reasonable job. It is required for the
conversions to HTML and GNU info.
lynx: This is a text-mode Web browser which includes an
HTML-to-plain text conversion. This is used to convert
`howto` documents to text.
make: Just about any version should work for the standard documents,
but GNU make is required for the experimental
processes in Doc/tools/sgmlconv/, at least while
they're experimental.
makeindex: This is a standard program for converting LATEX index data
to a formatted index; it should be included with all LATEX
installations. It is needed for the PDF and PostScript
conversions.
makeinfo: GNU makeinfo is used to convert Texinfo documents to
GNU info files. Since Texinfo is used as an intermediate
format in the info conversion, this program is needed in that
conversion.
pdflatex: pdfTEX is a relatively new variant of TEX, and is used to
generate the PDF version of the manuals. It is typically
installed as part of most of the large TEX distributions.
pdflatex is pdfTEX using the LATEX format.
perl: Perl is required for LATEX2HTML and one of the scripts used
to post-process LATEX2HTML output, as well as the
HTML-to-Texinfo conversion. This is required for
the HTML and GNU info conversions.
python: Python is used for many of the scripts in the
Doc/tools/ directory; it is required for all
conversions. This shouldn't be a problem if you're interested
in writing documentation for Python!


7.2 Internal Tools

node21.html | node20.html | futures.html | Documenting Python | contents.html
---
## 7.2 Internal Tools
This section describes the various scripts that are used to
implement various stages of document processing or to orchestrate
entire build sequences. Most of these tools are only useful
in the context of building the standard documentation, but some
are more general.
mkhowto: This is the primary script used to format third-party
documents. It contains all the logic needed to ``get it
right.'' The proper way to use this script is to make a
symbolic link to it or run it in place; the actual script file
must be stored as part of the documentation source tree,
though it may be used to format documents outside the
tree. Use mkhowto --help
for a list of
command line options.
mkhowto can be used for both `howto` and
`manual` class documents. (For the later, be sure to get
the latest version from the Python CVS repository rather than
the version distributed in the latex-1.5.2.tgz source
archive.)
XXX Need more here.


2 Directory Structure

node2.html | doc.html | latex-primer.html | Documenting Python | contents.html
---
# 2 Directory Structure
The source distribution for the standard Python documentation
contains a large number of directories. While third-party documents
do not need to be placed into this structure or need to be placed
within a similar structure, it can be helpful to know where to look
for examples and tools when developing new documents using the
Python documentation tools. This section describes this directory
structure.
The documentation sources are usually placed within the Python
source distribution as the top-level directory Doc/, but
are not dependent on the Python source distribution in any way.
The Doc/ directory contains a few files and several
subdirectories. The files are mostly self-explanatory, including a
README and a Makefile. The directories fall into
three categories:


3.1 Syntax

latex-primer.html | latex-primer.html | node6.html | Documenting Python | contents.html
---
## 3.1 Syntax
There are a things that an author of Python documentation needs to
know about LATEX syntax.
A comment is started by the ``percent'' character
("%") and continues through the end of the line and all
leading whitespace on the following line. This is a little
different from any programming language I know of, so an example
is in order:
```text

This is text.% comment
This is more text. % another comment
Still more text.
```
The first non-comment character following the first comment is the
letter "T" on the second line; the leading whitespace on
that line is consumed as part of the first comment. This means
that there is no space between the first and second sentences, so
the period and letter "T" will be directly adjacent in
the typeset document.
Note also that though the first non-comment character after the
second comment is the letter "S", there is whitespace
preceding the comment, so the two sentences are separated as
expected.
A group is an enclosure for a collection of text and
commands which encloses the formatting context and constrains the
scope of any changes to that context made by commands within the
group. Groups can be nested hierarchically. The formatting
context includes the font and the definition of additional macros
(or overrides of macros defined in outer groups). Syntactically,
groups are enclosed in braces:
```text

{text in a group}
```
An alternate syntax for a group using brackets (`[...]`) is
used by macros and environment constructors which take optional
parameters; brackets do not normally hold syntactic significance.
A degenerate group, containing only one atomic bit of content,
does not need to have an explicit group, unless it is required to
avoid ambiguity. Since Python tends toward the explicit, groups
are also made explicit in the documentation markup.
Groups are used only sparingly in the Python documentation, except
for their use in marking parameters to macros and environments.
A macro is usually simple construct which is identified by
name and can take some number of parameters. In normal LATEX
usage, one of these can be optional. The markup is introduced
using the backslash character ("\"), and the name is
given by alphabetic characters (no digits, hyphens, or
underscores). Required parameters should be marked as a group,
and optional parameters should be marked using the alternate
syntax for a group.
For example, a macro named ``foo'' which takes a single parameter
would appear like this:
```text

\name{parameter}
```
A macro which takes an optional parameter would be typed like this
when the optional paramter is given:
```text

\name[optional]
```
If both optional and required parameters are to be required, it
looks like this:
```text

\name[optional]{required}
```
A macro name may be followed by a space or newline; a space
between the macro name and any parameters will be consumed, but
this usage is not practiced in the Python documentation. Such a
space is still consumed if there are no parameters to the marco,
in which case inserting an empty group (`{}`) or explicit
word space ("\ ") immediately after the macro name helps to
avoid running the expansion of the macro into the following text.
Macros which take no parameters but which should not be followed
by a word space do not need special treatment if the following
character in the document source if not a name character (such as
puctuation).
Each line of this example shows an appropriate way to write text
which includes a macro which takes no parameters:
```text

This \UNIX{} is followed by a space.
This \UNIX\ is also followed by a space.
\UNIX, followed by a comma, needs no additional markup.
```
An environment is a larger construct than a macro, and can
be used for things with more content that would conveniently fit
in a macro parameter. They are primarily used when formatting
parameters need to be changed before and after a large chunk of
content, but the content itself needs to be highly flexible. Code
samples are presented using an environment, and descriptions of
functions, methods, and classes are also marked using envionments.
Since the content of an environment is free-form and can consist
of several paragraphs, they are actually marked using a pair of
macros: \begin and \end. These macros both take the
name of the environment as a parameter. An example is the
environment used to mark the abstract of a document:
```text

\begin{abstract}
This is the text of the abstract. It concisely explains what
information is found in the document.

It can consist of multiple paragraphs.
\end{abstract}
```
An environment can also have required and optional parameters of
its own. These follow the parameter of the \begin macro.
This example shows an environment which takes a single required
parameter:
```text

\begin{datadesc}{controlnames}
A 33-element string array that contains the \ASCII{} mnemonics for
the thirty-two \ASCII{} control characters from 0 (NUL) to 0x1f
(US), in order, plus the mnemonic \samp{SP} for the space character.
\end{datadesc}
```
There are a number of less-used marks in LATEX are used to
enter non-ASCII characters, especially those used in European
names. Given that these are often used adjacent to other
characters, the markup required to produce the proper character
may need to be followed by a space or an empty group, or the the
markup can be enclosed in a group. Some which are found in Python
documentation are:


3.2 Hierarchical Structure

node5.html | latex-primer.html | classes.html | Documenting Python | contents.html
---
## 3.2 Hierarchical Structure
LATEX expects documents to be arranged in a conventional,
hierarchical way, with chapters, sections, sub-sections,
appendixes, and the like. These are marked using macros rather
than environments, probably because the end of a section can be
safely inferred when a section of equal or higher level starts.
There are six ``levels'' of sectioning in the document classes
used for Python documentation, and the lowest two levels are not
used. The levels are:
Notes:
(1): Only used for the `manual` documents, as described in
section 4 (classes.html#classes), ``Document Classes.''
(2): Not the same as a paragraph of text; nobody seems to use this.


5 Special Markup Constructs

classes.html | doc.html | preamble-info.html | Documenting Python | contents.html
---
# 5 Special Markup Constructs
The Python document classes define a lot of new environments and
macros. This section contains the reference material for these
facilities.


5.1 Markup for the Preamble

node8.html | node8.html | meta-info.html | Documenting Python | contents.html
---
## 5.1 Markup for the Preamble


5.9 Reference List Markup

node16.html | node8.html | indexing.html | Documenting Python | contents.html
---
## 5.9 Reference List Markup
Many sections include a list of references to module documentation
or external documents. These lists are created using the
\seealso environment. This environment defines some
additional macros to support creating reference entries in a
reasonable manner.
The \seealso environment is typically placed in a section
just before any sub-sections. This is done to ensure that
reference links related to the section are not hidden in a
subsection in the hypertext renditions of the documentation.
For each of the following macros, why should be one or more
complete sentences, starting with a capital letter (unless it
starts with an identifier, which should not be modified), and
ending with the apropriate punctuation.
These macros are only defined within the content of the
\seealso environment.


8.1 Structured Documentation

futures.html | futures.html | discussion.html | Documenting Python | contents.html
---
## 8.1 Structured Documentation
Most of the small changes to the LATEX markup have been made
with an eye to divorcing the markup from the presentation, making
both a bit more maintainable. Over the course of 1998, a large
number of changes were made with exactly this in mind; previously,
changes had been made but in a less systematic manner and with
more concern for not needing to update the existing content. The
result has been a highly structured and semantically loaded markup
language implemented in LATEX. With almost no basic TEX or
LATEX markup in use, however, the markup syntax is about the
only evidence of LATEX in the actual document sources.
One side effect of this is that while we've been able to use
standard ``engines'' for manipulating the documents, such as
LATEX and LATEX2HTML, most of the actual transformations have
been created specifically for Python. The LATEX document
classes and LATEX2HTML support are both complete implementations
of the specific markup designed for these documents.
Combining highly customized markup with the somewhat esoteric
systems used to process the documents leads us to ask some
questions: Can we do this more easily? and, Can we do this
better? After a great deal of discussion with the community, we
have determined that actively pursuing modern structured
documentation systems is worth some investment of time.
There appear to be two real contenders in this arena: the Standard
General Markup Language (SGML), and the Extensible Markup Language
(XML). Both of these standards have advantages and disadvantages,
and many advantages are shared.
SGML offers advantages which may appeal most to authors,
especially those using ordinary text editors. There are also
additional abilities to define content models. A number of
high-quality tools with demonstrated maturity is available, but
most are not free; for those which are, portability issues remain
a problem.
The advantages of XML include the availability of a large number
of evolving tools. Unfortunately, many of the associated
standards are still evolving, and the tools will have to follow
along. This means that developing a robust tool set that uses
more than the basic XML 1.0 recommendation is not possible in the
short term. The promised availability of a wide variety of
high-quality tools which support some of the most important
related standards is not immediate. Many tools are likely to be
free.
XXX Eventual migration to SGML/XML.


About this document ...

reporting-bugs.html | ext.html | Extending and Embedding the Python Interpreter | contents.html
---
# About this document ...
Extending and Embedding the Python Interpreter,
April 15, 2001, Release 2.1
This document was generated using the LaTeX2HTML (http://saftsack.fs.uni-bayreuth.de/~latex2ht/) translator.
LaTeX2HTML (http://saftsack.fs.uni-bayreuth.de/~latex2ht/) is Copyright ©
1993, 1994, 1995, 1996, 1997, Nikos
Drakos (http://cbl.leeds.ac.uk/nikos/personal.html), Computer Based Learning Unit, University of
Leeds, and Copyright © 1997, 1998, Ross
Moore (http://www.maths.mq.edu.au/~ross/), Mathematics Department, Macquarie University,
Sydney.
The application of LaTeX2HTML (http://saftsack.fs.uni-bayreuth.de/~latex2ht/) to the Python
documentation has been heavily tailored by Fred L. Drake,
Jr. Original navigation icons were contributed by Christopher
Petrilli.
---
## Comments and Questions
General comments and questions regarding this document should
be sent by email to python-docs@python.org (mailto:python-docs@python.org). If you find specific errors in
this document, please report the bug at the Python Bug
Tracker (http://sourceforge.net/bugs/?group_id=5470) at SourceForge (http://sourceforge.net/).
Questions regarding how to use the information in this
document should be sent to the Python news group, comp.lang.python (news:comp.lang.python), or the Python mailing list (http://www.python.org/mailman/listinfo/python-list) (which is gated to the newsgroup and
carries the same content).
For any of these channels, please be sure not to send HTML email.
Thanks.
---
reporting-bugs.html | ext.html | Extending and Embedding the Python Interpreter | contents.html
---
Release 2.1, documentation updated on April 15, 2001.


1.3 Back to the Example

errors.html | intro.html | methodTable.html | Extending and Embedding the Python Interpreter | contents.html
---
# 1.3 Back to the Example
Going back to our example function, you should now be able to
understand this statement:
```text

if (!PyArg_ParseTuple(args, "s", &command))
return NULL;
```
It returns NULL (the error indicator for functions returning
object pointers) if an error is detected in the argument list, relying
on the exception set by PyArg_ParseTuple(). Otherwise the
string value of the argument has been copied to the local variable
command. This is a pointer assignment and you are not supposed
to modify the string to which it points (so in Standard C, the variable
command should properly be declared as "const char
*command").
The next statement is a call to the Unix function
system(), passing it the string we just got from
PyArg_ParseTuple():
```text

sts = system(command);
```
Our spam.system() function must return the value of
sts as a Python object. This is done using the function
Py_BuildValue(), which is something like the inverse of
PyArg_ParseTuple(): it takes a format string and an
arbitrary number of C values, and returns a new Python object.
More info on Py_BuildValue() is given later.
```text

return Py_BuildValue("i", sts);
```
In this case, it will return an integer object. (Yes, even integers
are objects on the heap in Python!)
If you have a C function that returns no useful argument (a function
returning void), the corresponding Python function must return
`None`. You need this idiom to do so:
```text

Py_INCREF(Py_None);
return Py_None;
```
Py_None is the C name for the special Python object
`None`. It is a genuine Python object rather than a NULL
pointer, which means ``error'' in most contexts, as we have seen.


dnt-type-methods.html | ext.html | custom-interps.html | Extending and Embedding the Python Interpreter | contents.html
---
# 3. Building C and C++ Extensions on Unix
Starting in Python 1.4, Python provides a special make file for
building make files for building dynamically-linked extensions and
custom interpreters. The make file make file builds a make file
that reflects various system variables determined by configure when
the Python interpreter was built, so people building module's don't
have to resupply these settings. This vastly simplifies the process
of building extensions and custom interpreters on Unix systems.
The make file make file is distributed as the file
Misc/Makefile.pre.in in the Python source distribution. The
first step in building extensions or custom interpreters is to copy
this make file to a development directory containing extension module
source.
The make file make file, Makefile.pre.in uses metadata
provided in a file named Setup. The format of the Setup
file is the same as the Setup (or Setup.dist) file
provided in the Modules/ directory of the Python source
distribution. The Setup file contains variable definitions:
```text

EC=/projects/ExtensionClass
```
and module description lines. It can also contain blank lines and
comment lines that start with "#".
A module description line includes a module name, source files,
options, variable references, and other input files, such
as libraries or object files. Consider a simple example:
```text

ExtensionClass ExtensionClass.c
```
This is the simplest form of a module definition line. It defines a
module, ExtensionClass, which has a single source file,
ExtensionClass.c.
This slightly more complex example uses an -I option to
specify an include directory:
```text

EC=/projects/ExtensionClass
cPersistence cPersistence.c -I$(EC)
```
This example also illustrates the format for variable references.
For systems that support dynamic linking, the Setup file should
begin:
```text

*shared*
```
to indicate that the modules defined in Setup are to be built
as dynamically linked modules. A line containing only "*static*"can be used to indicate the subsequently listed modules should be
statically linked.
Here is a complete Setup file for building a
cPersistent module:
```text

# Set-up file to build the cPersistence module.
# Note that the text should begin in the first column.
*shared*

# We need the path to the directory containing the ExtensionClass
# include file.
EC=/projects/ExtensionClass
cPersistence cPersistence.c -I$(EC)
```
After the Setup file has been created, Makefile.pre.in
is run with the "boot" target to create a make file:
```text

make -f Makefile.pre.in boot
```
This creates the file, Makefile. To build the extensions, simply
run the created make file:
```text

make
```
It's not necessary to re-run Makefile.pre.in if the
Setup file is changed. The make file automatically rebuilds
itself if the Setup file changes.


distributing.html | ext.html | win-cookbook.html | Extending and Embedding the Python Interpreter | contents.html
---
# 4. Building C and C++ Extensions on Windows
This chapter briefly explains how to create a Windows extension module
for Python using Microsoft Visual C++, and follows with more
detailed background information on how it works. The explanatory
material is useful for both the Windows programmer learning to build
Python extensions and the Unix programmer interested in producing
software which can be successfully built on both Unix and Windows.


1.9 Building Arbitrary Values

parseTupleAndKeywords.html | intro.html | refcounts.html | Extending and Embedding the Python Interpreter | contents.html
---
# 1.9 Building Arbitrary Values
This function is the counterpart to PyArg_ParseTuple(). It is
declared as follows:
```text

PyObject *Py_BuildValue(char *format, ...);
```
It recognizes a set of format units similar to the ones recognized by
PyArg_ParseTuple(), but the arguments (which are input to the
function, not output) must not be pointers, just values. It returns a
new Python object, suitable for returning from a C function called
from Python.
One difference with PyArg_ParseTuple(): while the latter
requires its first argument to be a tuple (since Python argument lists
are always represented as tuples internally),
Py_BuildValue() does not always build a tuple. It builds
a tuple only if its format string contains two or more format units.
If the format string is empty, it returns `None`; if it contains
exactly one format unit, it returns whatever object is described by
that format unit. To force it to return a tuple of size 0 or one,
parenthesize the format string.
When memory buffers are passed as parameters to supply data to build
objects, as for the "s" and "s#" formats, the required data
is copied. Buffers provided by the caller are never referenced by the
objects created by Py_BuildValue(). In other words, if
your code invokes malloc() and passes the allocated memory
to Py_BuildValue(), your code is responsible for
calling free() for that memory once
Py_BuildValue() returns.
In the following description, the quoted form is the format unit; the
entry in (round) parentheses is the Python object type that the format
unit will return; and the entry in [square] brackets is the type of
the C value(s) to be passed.
The characters space, tab, colon and comma are ignored in format
strings (but not within format units such as "s#"). This can be
used to make long format strings a tad more readable.
"s" (string) [char *]: Convert a null-terminated C string to a Python object. If the C
string pointer is NULL, `None` is used.
"s#" (string) [char *, int]: Convert a C string and its length to a Python object. If the C string
pointer is NULL, the length is ignored and `None` is
returned.
"z" (string or `None`) [char *]: Same as "s".
"z#" (string or `None`) [char *, int]: Same as "s#".
"u" (Unicode string) [Py_UNICODE *]: Convert a null-terminated buffer of Unicode (UCS-2) data to a Python
Unicode object. If the Unicode buffer pointer is NULL,
`None` is returned.
"u#" (Unicode string) [Py_UNICODE *, int]: Convert a Unicode (UCS-2) data buffer and its length to a Python
Unicode object. If the Unicode buffer pointer is NULL, the length
is ignored and `None` is returned.
"i" (integer) [int]: Convert a plain C int to a Python integer object.
"b" (integer) [char]: Same as "i".
"h" (integer) [short int]: Same as "i".
"l" (integer) [long int]: Convert a C long int to a Python integer object.
"c" (string of length 1) [char]: Convert a C int representing a character to a Python string of
length 1.
"d" (float) [double]: Convert a C double to a Python floating point number.
"f" (float) [float]: Same as "d".
"D" (complex) [Py_complex *]: Convert a C Py_complex structure to a Python complex number.
"O" (object) [PyObject *]: Pass a Python object untouched (except for its reference count, which
is incremented by one). If the object passed in is a NULL
pointer, it is assumed that this was caused because the call producing
the argument found an error and set an exception. Therefore,
Py_BuildValue() will return NULL but won't raise an
exception. If no exception has been raised yet,
PyExc_SystemError is set.
"S" (object) [PyObject *]: Same as "O".
"U" (object) [PyObject *]: Same as "O".
"N" (object) [PyObject *]: Same as "O", except it doesn't increment the reference count on
the object. Useful when the object is created by a call to an object
constructor in the argument list.
"O&" (object) [converter, anything]: Convert anything to a Python object through a converter
function. The function is called with anything (which should be
compatible with void *) as its argument and should return a
``new'' Python object, or NULL if an error occurred.
"(items)" (tuple) [matching-items]: Convert a sequence of C values to a Python tuple with the same number
of items.
"[items]" (list) [matching-items]: Convert a sequence of C values to a Python list with the same number
of items.
"{items}" (dictionary) [matching-items]: Convert a sequence of C values to a Python dictionary. Each pair of
consecutive C values adds one item to the dictionary, serving as key
and value, respectively.
If there is an error in the format string, the
PyExc_SystemError exception is raised and NULL returned.
Examples (to the left the call, to the right the resulting Python value):
```text

Py_BuildValue("") None
Py_BuildValue("i", 123) 123
Py_BuildValue("iii", 123, 456, 789) (123, 456, 789)
Py_BuildValue("s", "hello") 'hello'
Py_BuildValue("ss", "hello", "world") ('hello', 'world')
Py_BuildValue("s#", "hello", 4) 'hell'
Py_BuildValue("()") ()
Py_BuildValue("(i)", 123) (123,)
Py_BuildValue("(ii)", 123, 456) (123, 456)
Py_BuildValue("(i,i)", 123, 456) (123, 456)
Py_BuildValue("[i,i]", 123, 456) [123, 456]
Py_BuildValue("{s:i,s:i}",
"abc", 123, "def", 456) {'abc': 123, 'def': 456}
Py_BuildValue("((ii)(ii)) (ii)",
1, 2, 3, 4, 5, 6) (((1, 2), (3, 4)), (5, 6))
```


1.6 Calling Python Functions from C

compilation.html | intro.html | parseTuple.html | Extending and Embedding the Python Interpreter | contents.html
---
# 1.6 Calling Python Functions from C
So far we have concentrated on making C functions callable from
Python. The reverse is also useful: calling Python functions from C.
This is especially the case for libraries that support so-called
``callback'' functions. If a C interface makes use of callbacks, the
equivalent Python often needs to provide a callback mechanism to the
Python programmer; the implementation will require calling the Python
callback functions from a C callback. Other uses are also imaginable.
Fortunately, the Python interpreter is easily called recursively, and
there is a standard interface to call a Python function. (I won't
dwell on how to call the Python parser with a particular string as
input -- if you're interested, have a look at the implementation of
the -c command line option in Python/pythonmain.c
from the Python source code.)
Calling a Python function is easy. First, the Python program must
somehow pass you the Python function object. You should provide a
function (or some other interface) to do this. When this function is
called, save a pointer to the Python function object (be careful to
Py_INCREF() it!) in a global variable -- or wherever you
see fit. For example, the following function might be part of a module
definition:
```text

static PyObject *my_callback = NULL;

static PyObject *
my_set_callback(dummy, args)
PyObject *dummy, *args;
{
PyObject *result = NULL;
PyObject *temp;

if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {
if (!PyCallable_Check(temp)) {
PyErr_SetString(PyExc_TypeError, "parameter must be callable");
return NULL;
}
Py_XINCREF(temp); /* Add a reference to new callback */
Py_XDECREF(my_callback); /* Dispose of previous callback */
my_callback = temp; /* Remember new callback */
/* Boilerplate to return "None" */
Py_INCREF(Py_None);
result = Py_None;
}
return result;
}
```
This function must be registered with the interpreter using the
METH_VARARGS flag; this is described in section
1.4 (methodTable.html#methodTable), ``The Module's Method Table and Initialization
Function.'' The PyArg_ParseTuple() function and its
arguments are documented in section 1.7 (parseTuple.html#parseTuple), ``Format Strings
for PyArg_ParseTuple().''
The macros Py_XINCREF() and Py_XDECREF()
increment/decrement the reference count of an object and are safe in
the presence of NULL pointers (but note that temp will not be
NULL in this context). More info on them in section
1.10 (refcounts.html#refcounts), ``Reference Counts.''
Later, when it is time to call the function, you call the C function
PyEval_CallObject(). This function has two arguments, both
pointers to arbitrary Python objects: the Python function, and the
argument list. The argument list must always be a tuple object, whose
length is the number of arguments. To call the Python function with
no arguments, pass an empty tuple; to call it with one argument, pass
a singleton tuple. Py_BuildValue() returns a tuple when its
format string consists of zero or more format codes between
parentheses. For example:
```text

int arg;
PyObject *arglist;
PyObject *result;
...
arg = 123;
...
/* Time to call the callback */
arglist = Py_BuildValue("(i)", arg);
result = PyEval_CallObject(my_callback, arglist);
Py_DECREF(arglist);
```
PyEval_CallObject() returns a Python object pointer: this is
the return value of the Python function. PyEval_CallObject() is
``reference-count-neutral'' with respect to its arguments. In the
example a new tuple was created to serve as the argument list, which
is Py_DECREF()-ed immediately after the call.
The return value of PyEval_CallObject() is ``new'': either it
is a brand new object, or it is an existing object whose reference
count has been incremented. So, unless you want to save it in a
global variable, you should somehow Py_DECREF() the result,
even (especially!) if you are not interested in its value.
Before you do this, however, it is important to check that the return
value isn't NULL. If it is, the Python function terminated by
raising an exception. If the C code that called
PyEval_CallObject() is called from Python, it should now
return an error indication to its Python caller, so the interpreter
can print a stack trace, or the calling Python code can handle the
exception. If this is not possible or desirable, the exception should
be cleared by calling PyErr_Clear(). For example:
```text

if (result == NULL)
return NULL; /* Pass error back */
...use result...
Py_DECREF(result);
```
Depending on the desired interface to the Python callback function,
you may also have to provide an argument list to
PyEval_CallObject(). In some cases the argument list is
also provided by the Python program, through the same interface that
specified the callback function. It can then be saved and used in the
same manner as the function object. In other cases, you may have to
construct a new tuple to pass as the argument list. The simplest way
to do this is to call Py_BuildValue(). For example, if
you want to pass an integral event code, you might use the following
code:
```text

PyObject *arglist;
...
arglist = Py_BuildValue("(l)", eventcode);
result = PyEval_CallObject(my_callback, arglist);
Py_DECREF(arglist);
if (result == NULL)
return NULL; /* Pass error back */
/* Here maybe use the result */
Py_DECREF(result);
```
Note the placement of "Py_DECREF(arglist)" immediately after the
call, before the error check! Also note that strictly spoken this
code is not complete: Py_BuildValue() may run out of
memory, and this should be checked.


1.5 Compilation and Linkage

methodTable.html | intro.html | callingPython.html | Extending and Embedding the Python Interpreter | contents.html
---
# 1.5 Compilation and Linkage
There are two more things to do before you can use your new extension:
compiling and linking it with the Python system. If you use dynamic
loading, the details depend on the style of dynamic loading your
system uses; see the chapters about building extension modules on
Unix (chapter 3 (building-on-unix.html#building-on-unix)) and Windows (chapter
4 (building-on-windows.html#building-on-windows)) for more information about this.
If you can't use dynamic loading, or if you want to make your module a
permanent part of the Python interpreter, you will have to change the
configuration setup and rebuild the interpreter. Luckily, this is
very simple: just place your file (spammodule.c for example) in
the Modules/ directory of an unpacked source distribution, add
a line to the file Modules/Setup.local describing your file:
```text

spam spammodule.o
```
and rebuild the interpreter by running make in the toplevel
directory. You can also run make in the Modules/
subdirectory, but then you must first rebuild Makefile
there by running `make Makefile'. (This is necessary each
time you change the Setup file.)
If your module requires additional libraries to link with, these can
be listed on the line in the configuration file as well, for instance:
```text

spam spammodule.o -lX11
```


Contents

front.html | ext.html | intro.html | Extending and Embedding the Python Interpreter
---
## Contents
Table of Contents
- Front Matter (front.html)
  1. Extending Python with C or C++ (intro.html)
  - 1.1 A Simple Example (simpleExample.html)
  1.2 Intermezzo: Errors and Exceptions (errors.html)
  1.3 Back to the Example (backToExample.html)
  1.4 The Module's Method Table and Initialization Function (methodTable.html)
  1.5 Compilation and Linkage (compilation.html)
  1.6 Calling Python Functions from C (callingPython.html)
  1.7 Extracting Parameters in Extension Functions (parseTuple.html)
  1.8 Keyword Parameters for Extension Functions (parseTupleAndKeywords.html)
  1.9 Building Arbitrary Values (buildValue.html)
  1.10 Reference Counts (refcounts.html)
  - 1.10.1 Reference Counting in Python (refcountsInPython.html)
  1.10.2 Ownership Rules (ownershipRules.html)
  1.10.3 Thin Ice (thinIce.html)
  1.10.4 NULL Pointers (nullPointers.html)
  1.11 Writing Extensions in C++ (cplusplus.html)
  1.12 Providing a C API for an Extension Module (using-cobjects.html)
  2. Defining New Types (defining-new-types.html)
  - 2.1 The Basics (dnt-basics.html)
  2.2 Type Methods (dnt-type-methods.html)
  3. Building C and C++ Extensions on Unix (building-on-unix.html)
  - 3.1 Building Custom Interpreters (custom-interps.html)
  3.2 Module Definition Options (module-defn-options.html)
  3.3 Example (module-defn-example.html)
  3.4 Distributing your extension modules (distributing.html)
  4. Building C and C++ Extensions on Windows (building-on-windows.html)
  - 4.1 A Cookbook Approach (win-cookbook.html)
  4.2 Differences Between Unix and Windows (dynamic-linking.html)
  4.3 Using DLLs in Practice (win-dlls.html)
  5. Embedding Python in Another Application (embedding.html)
  - 5.1 Embedding Python in C++ (embeddingInCplusplus.html)
  5.2 Linking Requirements (link-reqs.html)
  A. Reporting Bugs (reporting-bugs.html)
  About this document ... (about.html)
End of Table of Contents


1.11 Writing Extensions in C++

nullPointers.html | intro.html | using-cobjects.html | Extending and Embedding the Python Interpreter | contents.html
---
# 1.11 Writing Extensions in C++
It is possible to write extension modules in C++. Some restrictions
apply. If the main program (the Python interpreter) is compiled and
linked by the C compiler, global or static objects with constructors
cannot be used. This is not a problem if the main program is linked
by the C++ compiler. Functions that will be called by the
Python interpreter (in particular, module initalization functions)
have to be declared using `extern "C"`.
It is unnecessary to enclose the Python header files in
`extern "C" {...}` -- they use this form already if the symbol
"__cplusplus" is defined (all recent C++ compilers define this
symbol).


3.1 Building Custom Interpreters

building-on-unix.html | building-on-unix.html | module-defn-options.html | Extending and Embedding the Python Interpreter | contents.html
---
# 3.1 Building Custom Interpreters
The make file built by Makefile.pre.in can be run with the
"static" target to build an interpreter:
```text

make static
```
Any modules defined in the Setup file before the
"*shared*" line will be statically linked into the interpreter.
Typically, a "*shared*" line is omitted from the
Setup file when a custom interpreter is desired.


2. Defining New Types

using-cobjects.html | ext.html | dnt-basics.html | Extending and Embedding the Python Interpreter | contents.html
---
# 2. Defining New Types
As mentioned in the last chapter, Python allows the writer of an
extension module to define new types that can be manipulated from
Python code, much like strings and lists in core Python.
This is not hard; the code for all extension types follows a pattern,
but there are some details that you need to understand before you can
get started.


3.4 Distributing your extension modules

module-defn-example.html | building-on-unix.html | building-on-windows.html | Extending and Embedding the Python Interpreter | contents.html
---
# 3.4 Distributing your extension modules
There are two ways to distribute extension modules for others to use.
The way that allows the easiest cross-platform support is to use the
distutilspackage. The manual
Distributing Python Modules (../dist/dist.html) contains
information on this approach. It is recommended that all new
extensions be distributed using this approach to allow easy building
and installation across platforms. Older extensions should migrate to
this approach as well.
What follows describes the older approach; there are still many
extensions which use this.
When distributing your extension modules in source form, make sure to
include a Setup file. The Setup file should be named
Setup.in in the distribution. The make file make file,
Makefile.pre.in, will copy Setup.in to Setup if
the person installing the extension doesn't do so manually.
Distributing a Setup.in file makes it easy for people to
customize the Setup file while keeping the original in
Setup.in.
It is a good idea to include a copy of Makefile.pre.in for
people who do not have a source distribution of Python.
Do not distribute a make file. People building your modules
should use Makefile.pre.in to build their own make file. A
README file included in the package should provide simple
instructions to perform the build.