7 PEP 279: enumerate()

node7.html | whatsnew23.html | node9.html | What's New in Python 2.3 | contents.html
Previous:
6 PEP 278: Universal (node7.html)
Up:
What's New in Python (whatsnew23.html)
Next:
8 PEP 282: The (node9.html)
---
# 7 PEP 279: enumerate()
A new built-in function, enumerate(), will make
certain loops a bit clearer. `enumerate(thing)`, where
thing is either an iterator or a sequence, returns a iterator
that will return `(0, thing [0])`, `(1, thing [1])`, `(2, thing [2])`, and so forth.
A common idiom to change every element of a list looks like this:
```text

for i in range(len(L)):
item = L[i]
# ... compute some result based on item ...
L[i] = result
```
This can be rewritten using enumerate() as:
```text

for i, item in enumerate(L):
# ... compute some result based on item ...
L[i] = result
```
See Also:


2 PEP 255: Simple Generators

node2.html | whatsnew23.html | section-encodings.html | What's New in Python 2.3 | contents.html
Previous:
1 PEP 218: A (node2.html)
Up:
What's New in Python (whatsnew23.html)
Next:
3 PEP 263: Source (section-encodings.html)
---
# 2 PEP 255: Simple Generators
In Python 2.2, generators were added as an optional feature, to be
enabled by a `from __future__ import generators` directive. In
2.3 generators no longer need to be specially enabled, and are now
always present; this means that yield is now always a
keyword. The rest of this section is a copy of the description of
generators from the ``What's New in Python 2.2'' document; if you read
it back when Python 2.2 came out, you can skip the rest of this section.
You're doubtless familiar with how function calls work in Python or C.
When you call a function, it gets a private namespace where its local
variables are created. When the function reaches a return
statement, the local variables are destroyed and the resulting value
is returned to the caller. A later call to the same function will get
a fresh new set of local variables. But, what if the local variables
weren't thrown away on exiting a function? What if you could later
resume the function where it left off? This is what generators
provide; they can be thought of as resumable functions.
Here's the simplest example of a generator function:
```text

def generate_ints(N):
for i in range(N):
yield i
```
A new keyword, yield, was introduced for generators. Any
function containing a yield statement is a generator
function; this is detected by Python's bytecode compiler which
compiles the function specially as a result.
When you call a generator function, it doesn't return a single value;
instead it returns a generator object that supports the iterator
protocol. On executing the yield statement, the generator
outputs the value of `i`, similar to a return
statement. The big difference between yield and a
return statement is that on reaching a yield the
generator's state of execution is suspended and local variables are
preserved. On the next call to the generator's `.next()` method,
the function will resume executing immediately after the
yield statement. (For complicated reasons, the
yield statement isn't allowed inside the try block
of a try...finally statement; read PEP 255 (http://www.python.org/peps/pep-0255.html) for a full
explanation of the interaction between yield and
exceptions.)
Here's a sample usage of the generate_ints() generator:
```text

>>> gen = generate_ints(3)
>>> gen
<generator object at 0x8117f90>
>>> gen.next()
0
>>> gen.next()
1
>>> gen.next()
2
>>> gen.next()
Traceback (most recent call last):
File "stdin", line 1, in ?
File "stdin", line 2, in generate_ints
StopIteration
```
You could equally write `for i in generate_ints(5)`, or
`a,b,c = generate_ints(3)`.
Inside a generator function, the return statement can only
be used without a value, and signals the end of the procession of
values; afterwards the generator cannot return any further values.
return with a value, such as `return 5`, is a syntax
error inside a generator function. The end of the generator's results
can also be indicated by raising StopIteration manually,
or by just letting the flow of execution fall off the bottom of the
function.
You could achieve the effect of generators manually by writing your
own class and storing all the local variables of the generator as
instance variables. For example, returning a list of integers could
be done by setting `self.count` to 0, and having the
next() method increment `self.count` and return it.
However, for a moderately complicated generator, writing a
corresponding class would be much messier.
Lib/test/test_generators.py contains a number of more
interesting examples. The simplest one implements an in-order
traversal of a tree using generators recursively.
```text

# A recursive generator that generates Tree leaves in in-order.
def inorder(t):
if t:
for x in inorder(t.left):
yield x
yield t.label
for x in inorder(t.right):
yield x
```
Two other examples in Lib/test/test_generators.py produce
solutions for the N-Queens problem (placing queens on an
chess board so that no queen threatens another) and the Knight's Tour
(a route that takes a knight to every square of an chessboard
without visiting any square twice).
The idea of generators comes from other programming languages,
especially Icon (http://www.cs.arizona.edu/icon/), where the
idea of generators is central. In Icon, every
expression and function call behaves like a generator. One example
from ``An Overview of the Icon Programming Language'' at
http://www.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of
what this looks like:
```text

sentence := "Store it in the neighboring harbor"
if (i := find("or", sentence)) > 5 then write(i)
```
In Icon the find() function returns the indexes at which the
substring ``or'' is found: 3, 23, 33. In the if statement,
`i` is first assigned a value of 3, but 3 is less than 5, so the
comparison fails, and Icon retries it with the second value of 23. 23
is greater than 5, so the comparison now succeeds, and the code prints
the value 23 to the screen.
Python doesn't go nearly as far as Icon in adopting generators as a
central concept. Generators are considered part of the core
Python language, but learning or using them isn't compulsory; if they
don't solve any problems that you have, feel free to ignore them.
One novel feature of Python's interface as compared to
Icon's is that a generator's state is represented as a concrete object
(the iterator) that can be passed around to other functions or stored
in a data structure.
See Also:


20 Other Changes and Fixes

node20.html | whatsnew23.html | node22.html | What's New in Python 2.3 | contents.html
Previous:
19 Build and C (node20.html)
Up:
What's New in Python (whatsnew23.html)
Next:
21 Porting to Python (node22.html)
---
# 20 Other Changes and Fixes
As usual, there were a bunch of other improvements and bugfixes
scattered throughout the source tree. A search through the CVS change
logs finds there were 523 patches applied and 514 bugs fixed between
Python 2.2 and 2.3. Both figures are likely to be underestimates.
Some of the more notable changes are:
- If the PYTHONINSPECT environment variable is set, the
Python interpreter will enter the interactive prompt after running a
Python program, as if Python had been invoked with the -i
option. The environment variable can be set before running the Python
interpreter, or it can be set by the Python program as part of its
execution.
- The regrtest.py script now provides a way to allow ``all
resources except foo.'' A resource name passed to the
-u option can now be prefixed with a hyphen
("-") to mean ``remove this resource.'' For example, the
option ``-u all,-bsddb`' could be used to enable the
use of all resources except `bsddb`.
- The tools used to build the documentation now work under Cygwin
as well as Unix.
- The `SET_LINENO` opcode has been removed. Back in the
mists of time, this opcode was needed to produce line numbers in
tracebacks and support trace functions (for, e.g., pdb).
Since Python 1.5, the line numbers in tracebacks have been computed
using a different mechanism that works with ``python -O''. For Python
2.3 Michael Hudson implemented a similar scheme to determine when to
call the trace function, removing the need for `SET_LINENO`
entirely.
It would be difficult to detect any resulting difference from Python
code, apart from a slight speed up when Python is run without
-O.
C extensions that access the f_lineno field of frame objects
should instead call `PyCode_Addr2Line(f->f_code, f->f_lasti)`.
This will have the added effect of making the code work as desired
under ``python -O'' in earlier versions of Python.
A nifty new feature is that trace functions can now assign to the
f_lineno attribute of frame objects, changing the line that
will be executed next. A "jump" command has been added to the
pdb debugger taking advantage of this new feature.
(Implemented by Richie Hindle.)


node11.html | whatsnew23.html | section-pep302.html | What's New in Python 2.3 | contents.html
Previous:
10 PEP 293: Codec (node11.html)
Up:
What's New in Python (whatsnew23.html)
Next:
12 PEP 302: New (section-pep302.html)
---
# 11 PEP 301: Package Index and Metadata for
Distutils
Support for the long-requested Python catalog makes its first
appearance in 2.3.
The heart of the catalog is the new Distutils `register` command.
Running `python setup.py register` will collect the metadata
describing a package, such as its name, version, maintainer,
description, &c., and send it to a central catalog server. The
resulting catalog is available from http://www.python.org/pypi.
To make the catalog a bit more useful, a new optional
classifiers keyword argument has been added to the Distutils
setup() function. A list of
Trove (http://catb.org/~esr/trove/)-style
strings can be supplied to help classify the software.
Here's an example setup.py with classifiers, written to be compatible
with older versions of the Distutils:
```text

from distutils import core
kw = {'name': "Quixote",
'version': "0.5.1",
'description': "A highly Pythonic Web application framework",
# ...
}

if (hasattr(core, 'setup_keywords') and
'classifiers' in core.setup_keywords):
kw['classifiers'] = \
['Topic :: Internet :: WWW/HTTP :: Dynamic Content',
'Environment :: No Input/Output (Daemon)',
'Intended Audience :: Developers'],

core.setup(**kw)
```
The full list of classifiers can be obtained by running
`python setup.py register --list-classifiers`.
See Also:


12 PEP 302: New Import Hooks

section-pep301.html | whatsnew23.html | node14.html | What's New in Python 2.3 | contents.html
Previous:
11 PEP 301: Package (section-pep301.html)
Up:
What's New in Python (whatsnew23.html)
Next:
13 PEP 305: Comma-separated (node14.html)
---
# 12 PEP 302: New Import Hooks
While it's been possible to write custom import hooks ever since the
ihooks module was introduced in Python 1.3, no one has ever
been really happy with it because writing new import hooks is
difficult and messy. There have been various proposed alternatives
such as the imputil and iu modules, but none of them
has ever gained much acceptance, and none of them were easily usable
from C code.
PEP 302 (http://www.python.org/peps/pep-0302.html) borrows ideas from its predecessors, especially from
Gordon McMillan's iu module. Three new items
are added to the sys module:
- `sys.path_hooks` is a list of callable objects; most
often they'll be classes. Each callable takes a string containing a
path and either returns an importer object that will handle imports
from this path or raises an ImportError exception if it
can't handle this path.
- `sys.path_importer_cache` caches importer objects for
each path, so `sys.path_hooks` will only need to be traversed
once for each path.
- `sys.meta_path` is a list of importer objects that will
be traversed before `sys.path` is checked. This list is
initially empty, but user code can add objects to it. Additional
built-in and frozen modules can be imported by an object added to
this list.
Importer objects must have a single method,
find_module(fullname, path=None). fullname
will be a module or package name, e.g. "string" or
"distutils.core". find_module() must return a loader object
that has a single method, load_module(fullname), that
creates and returns the corresponding module object.
Pseudo-code for Python's new import logic, therefore, looks something
like this (simplified a bit; see PEP 302 (http://www.python.org/peps/pep-0302.html) for the full details):
```text

for mp in sys.meta_path:
loader = mp(fullname)
if loader is not None:
<module> = loader.load_module(fullname)

for path in sys.path:
for hook in sys.path_hooks:
try:
importer = hook(path)
except ImportError:
# ImportError, so try the other path hooks
pass
else:
loader = importer.find_module(fullname)
<module> = loader.load_module(fullname)

# Not found!
raise ImportError
```
See Also:


14 PEP 307: Pickle Enhancements

node14.html | whatsnew23.html | section-slices.html | What's New in Python 2.3 | contents.html
Previous:
13 PEP 305: Comma-separated (node14.html)
Up:
What's New in Python (whatsnew23.html)
Next:
15 Extended Slices (section-slices.html)
---
# 14 PEP 307: Pickle Enhancements
The pickle and cPickle modules received some
attention during the 2.3 development cycle. In 2.2, new-style classes
could be pickled without difficulty, but they weren't pickled very
compactly; PEP 307 (http://www.python.org/peps/pep-0307.html) quotes a trivial example where a new-style class
results in a pickled string three times longer than that for a classic
class.
The solution was to invent a new pickle protocol. The
pickle.dumps() function has supported a text-or-binary flag
for a long time. In 2.3, this flag is redefined from a Boolean to an
integer: 0 is the old text-mode pickle format, 1 is the old binary
format, and now 2 is a new 2.3-specific format. A new constant,
pickle.HIGHEST_PROTOCOL, can be used to select the fanciest
protocol available.
Unpickling is no longer considered a safe operation. 2.2's
pickle provided hooks for trying to prevent unsafe classes
from being unpickled (specifically, a
__safe_for_unpickling__ attribute), but none of this code
was ever audited and therefore it's all been ripped out in 2.3. You
should not unpickle untrusted data in any version of Python.
To reduce the pickling overhead for new-style classes, a new interface
for customizing pickling was added using three special methods:
__getstate__, __setstate__, and
__getnewargs__. Consult PEP 307 (http://www.python.org/peps/pep-0307.html) for the full semantics
of these methods.
As a way to compress pickles yet further, it's now possible to use
integer codes instead of long strings to identify pickled classes.
The Python Software Foundation will maintain a list of standardized
codes; there's also a range of codes for private use. Currently no
codes have been specified.
See Also:


node18.html | whatsnew23.html | node20.html | What's New in Python 2.3 | contents.html
Previous:
17 New, Improved, and (node18.html)
Up:
What's New in Python (whatsnew23.html)
Next:
19 Build and C (node20.html)
---
# 18 Pymalloc: A Specialized Object Allocator
Pymalloc, a specialized object allocator written by Vladimir
Marangozov, was a feature added to Python 2.1. Pymalloc is intended
to be faster than the system malloc() and to have less
memory overhead for allocation patterns typical of Python programs.
The allocator uses C's malloc() function to get large
pools of memory and then fulfills smaller memory requests from these
pools.
In 2.1 and 2.2, pymalloc was an experimental feature and wasn't
enabled by default; you had to explicitly enable it when compiling
Python by providing the
--with-pymalloc option to the configure
script. In 2.3, pymalloc has had further enhancements and is now
enabled by default; you'll have to supply
--without-pymalloc to disable it.
This change is transparent to code written in Python; however,
pymalloc may expose bugs in C extensions. Authors of C extension
modules should test their code with pymalloc enabled,
because some incorrect code may cause core dumps at runtime.
There's one particularly common error that causes problems. There are
a number of memory allocation functions in Python's C API that have
previously just been aliases for the C library's malloc()
and free(), meaning that if you accidentally called
mismatched functions the error wouldn't be noticeable. When the
object allocator is enabled, these functions aren't aliases of
malloc() and free() any more, and calling the
wrong function to free memory may get you a core dump. For example,
if memory was allocated using PyObject_Malloc(), it has to
be freed using PyObject_Free(), not free(). A
few modules included with Python fell afoul of this and had to be
fixed; doubtless there are more third-party modules that will have the
same problem.
As part of this change, the confusing multiple interfaces for
allocating memory have been consolidated down into two API families.
Memory allocated with one family must not be manipulated with
functions from the other family. There is one family for allocating
chunks of memory and another family of functions specifically for
allocating Python objects.
- To allocate and free an undistinguished chunk of memory use
the ``raw memory'' family: PyMem_Malloc(),
PyMem_Realloc(), and PyMem_Free().
- The ``object memory'' family is the interface to the pymalloc
facility described above and is biased towards a large number of
``small'' allocations: PyObject_Malloc,
PyObject_Realloc, and PyObject_Free.
- To allocate and free Python objects, use the ``object'' family
PyObject_New(), PyObject_NewVar(), and
PyObject_Del().
Thanks to lots of work by Tim Peters, pymalloc in 2.3 also provides
debugging features to catch memory overwrites and doubled frees in
both extension modules and in the interpreter itself. To enable this
support, compile a debugging version of the Python interpreter by
running configure with --with-pydebug.
To aid extension writers, a header file Misc/pymemcompat.h is
distributed with the source to Python 2.3 that allows Python
extensions to use the 2.3 interfaces to memory allocation while
compiling against any version of Python since 1.5.2. You would copy
the file from Python's source distribution and bundle it with the
source of your extension.
See Also:


15 Extended Slices

section-pep305.html | whatsnew23.html | node17.html | What's New in Python 2.3 | contents.html
Previous:
14 PEP 307: Pickle (section-pep305.html)
Up:
What's New in Python (whatsnew23.html)
Next:
16 Other Language Changes (node17.html)
---
# 15 Extended Slices
Ever since Python 1.4, the slicing syntax has supported an optional
third ``step'' or ``stride'' argument. For example, these are all
legal Python syntax: `L[1:10:2]`, `L[:-1:1]`,
`L[::-1]`. This was added to Python at the request of
the developers of Numerical Python, which uses the third argument
extensively. However, Python's built-in list, tuple, and string
sequence types have never supported this feature, raising a
TypeError if you tried it. Michael Hudson contributed a
patch to fix this shortcoming.
For example, you can now easily extract the elements of a list that
have even indexes:
```text

>>> L = range(10)
>>> L[::2]
[0, 2, 4, 6, 8]
```
Negative values also work to make a copy of the same list in reverse
order:
```text

>>> L[::-1]
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
```
This also works for tuples, arrays, and strings:
```text

>>> s='abcd'
>>> s[::2]
'ac'
>>> s[::-1]
'dcba'
```
If you have a mutable sequence such as a list or an array you can
assign to or delete an extended slice, but there are some differences
between assignment to extended and regular slices. Assignment to a
regular slice can be used to change the length of the sequence:
```text

>>> a = range(3)
>>> a
[0, 1, 2]
>>> a[1:3] = [4, 5, 6]
>>> a
[0, 4, 5, 6]
```
Extended slices aren't this flexible. When assigning to an extended
slice, the list on the right hand side of the statement must contain
the same number of items as the slice it is replacing:
```text

>>> a = range(4)
>>> a
[0, 1, 2, 3]
>>> a[::2]
[0, 2]
>>> a[::2] = [0, -1]
>>> a
[0, 1, -1, 3]
>>> a[::2] = [0,1,2]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ValueError: attempt to assign sequence of size 3 to extended slice of size 2
```
Deletion is more straightforward:
```text

>>> a = range(4)
>>> a
[0, 1, 2, 3]
>>> a[::2]
[0, 2]
>>> del a[::2]
>>> a
[1, 3]
```
One can also now pass slice objects to the
__getitem__ methods of the built-in sequences:
```text

>>> range(10).__getitem__(slice(0, 5, 2))
[0, 2, 4]
```
Or use slice objects directly in subscripts:
```text

>>> range(10)[slice(0, 5, 2)]
[0, 2, 4]
```
To simplify implementing sequences that support extended slicing,
slice objects now have a method indices(length) which,
given the length of a sequence, returns a `( start , stop , step )` tuple that can be passed directly to
range().
indices() handles omitted and out-of-bounds indices in a
manner consistent with regular slices (and this innocuous phrase hides
a welter of confusing details!). The method is intended to be used
like this:
```text

class FakeSeq:
...
def calc_item(self, i):
...
def __getitem__(self, item):
if isinstance(item, slice):
indices = item.indices(len(self))
return FakeSeq([self.calc_item(i) in range(*indices)])
else:
return self.calc_item(i)
```
From this example you can also see that the built-in slice
object is now the type object for the slice type, and is no longer a
function. This is consistent with Python 2.2, where int,
str, etc., underwent the same change.


What's New in Python 2.3

../index.html | contents.html | What's New in Python 2.3 | contents.html
Up:
Python Documentation Index (../index.html)
Next:
---
# What's New in Python 2.3
A.M. Kuchling
amk@amk.ca