6. Modules

node7.html | tut.html | node9.html | Python Tutorial | node2.html
Previous:
5. Data Structures (node7.html)
Up:
Python Tutorial (tut.html)
Next:
7. Input and Output (node9.html)
---
- 6.1 More on Modules (node8.html#SECTION008100000000000000000)
  - 6.1.1 The Module Search Path (node8.html#SECTION008110000000000000000)
  6.1.2 ``Compiled'' Python files (node8.html#SECTION008120000000000000000)
  6.2 Standard Modules (node8.html#SECTION008200000000000000000)
  6.3 The dir() Function (node8.html#SECTION008300000000000000000)
  6.4 Packages (node8.html#SECTION008400000000000000000)
  - 6.4.1 Importing * From a Package (node8.html#SECTION008410000000000000000)
  6.4.2 Intra-package References (node8.html#SECTION008420000000000000000)
  6.4.3 Packages in Multiple Directories (node8.html#SECTION008430000000000000000)
---
# 6. Modules
If you quit from the Python interpreter and enter it again, the
definitions you have made (functions and variables) are lost.
Therefore, if you want to write a somewhat longer program, you are
better off using a text editor to prepare the input for the interpreter
and running it with that file as input instead. This is known as creating a
script. As your program gets longer, you may want to split it
into several files for easier maintenance. You may also want to use a
handy function that you've written in several programs without copying
its definition into each program.
To support this, Python has a way to put definitions in a file and use
them in a script or in an interactive instance of the interpreter.
Such a file is called a module; definitions from a module can be
imported into other modules or into the main module (the
collection of variables that you have access to in a script
executed at the top level
and in calculator mode).
A module is a file containing Python definitions and statements. The
file name is the module name with the suffix .py appended. Within
a module, the module's name (as a string) is available as the value of
the global variable `__name__`. For instance, use your favorite text
editor to create a file called fibo.py in the current directory
with the following contents:
```text

# Fibonacci numbers module

def fib(n): # write Fibonacci series up to n
a, b = 0, 1
while b < n:
print b,
a, b = b, a+b

def fib2(n): # return Fibonacci series up to n
result = []
a, b = 0, 1
while b < n:
result.append(b)
a, b = b, a+b
return result
```
Now enter the Python interpreter and import this module with the
following command:
```text

>>> import fibo
```
This does not enter the names of the functions defined in `fibo`
directly in the current symbol table; it only enters the module name
`fibo` there.
Using the module name you can access the functions:
```text

>>> fibo.fib(1000)
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987
>>> fibo.fib2(100)
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
>>> fibo.__name__
'fibo'
```
If you intend to use a function often you can assign it to a local name:
```text

>>> fib = fibo.fib
>>> fib(500)
1 1 2 3 5 8 13 21 34 55 89 144 233 377
```
# 6.1 More on Modules
A module can contain executable statements as well as function
definitions.
These statements are intended to initialize the module.
They are executed only the
first time the module is imported somewhere.6.1 (#foot698)
Each module has its own private symbol table, which is used as the
global symbol table by all functions defined in the module.
Thus, the author of a module can use global variables in the module
without worrying about accidental clashes with a user's global
variables.
On the other hand, if you know what you are doing you can touch a
module's global variables with the same notation used to refer to its
functions,
`modname.itemname`.
Modules can import other modules. It is customary but not required to
place all import statements at the beginning of a module (or
script, for that matter). The imported module names are placed in the
importing module's global symbol table.
There is a variant of the import statement that imports
names from a module directly into the importing module's symbol
table. For example:
```text

>>> from fibo import fib, fib2
>>> fib(500)
1 1 2 3 5 8 13 21 34 55 89 144 233 377
```
This does not introduce the module name from which the imports are taken
in the local symbol table (so in the example, `fibo` is not
defined).
There is even a variant to import all names that a module defines:
```text

>>> from fibo import *
>>> fib(500)
1 1 2 3 5 8 13 21 34 55 89 144 233 377
```
This imports all names except those beginning with an underscore
(`_`).
## 6.1.1 The Module Search Path
When a module named spam is imported, the interpreter searches
for a file named spam.py in the current directory,
and then in the list of directories specified by
the environment variable PYTHONPATH. This has the same syntax as
the shell variable PATH, that is, a list of
directory names. When PYTHONPATH is not set, or when the file
is not found there, the search continues in an installation-dependent
default path; on Unix, this is usually .:/usr/local/lib/python.
Actually, modules are searched in the list of directories given by the
variable `sys.path` which is initialized from the directory
containing the input script (or the current directory),
PYTHONPATH and the installation-dependent default. This allows
Python programs that know what they're doing to modify or replace the
module search path. Note that because the directory containing the
script being run is on the search path, it is important that the
script not have the same name as a standard module, or Python will
attempt to load the script as a module when that module is imported.
This will generally be an error. See section 6.2 (node8.html#standardModules),
``Standard Modules.'' for more information.
## 6.1.2 ``Compiled'' Python files
As an important speed-up of the start-up time for short programs that
use a lot of standard modules, if a file called spam.pyc exists
in the directory where spam.py is found, this is assumed to
contain an already-``byte-compiled'' version of the module spam.
The modification time of the version of spam.py used to create
spam.pyc is recorded in spam.pyc, and the
.pyc file is ignored if these don't match.
Normally, you don't need to do anything to create the
spam.pyc file. Whenever spam.py is successfully
compiled, an attempt is made to write the compiled version to
spam.pyc. It is not an error if this attempt fails; if for any
reason the file is not written completely, the resulting
spam.pyc file will be recognized as invalid and thus ignored
later. The contents of the spam.pyc file are platform
independent, so a Python module directory can be shared by machines of
different architectures.
Some tips for experts:
- When the Python interpreter is invoked with the -O flag,
optimized code is generated and stored in .pyo files. The
optimizer currently doesn't help much; it only removes
assert statements. When -O is used, all
bytecode is optimized; `.pyc` files are ignored and `.py`
files are compiled to optimized bytecode.
- Passing two -O flags to the Python interpreter
(-OO) will cause the bytecode compiler to perform
optimizations that could in some rare cases result in malfunctioning
programs. Currently only `__doc__` strings are removed from the
bytecode, resulting in more compact .pyo files. Since some
programs may rely on having these available, you should only use this
option if you know what you're doing.
- A program doesn't run any faster when it is read from a .pyc or
.pyo file than when it is read from a .py file; the only
thing that's faster about .pyc or .pyo files is the
speed with which they are loaded.
- When a script is run by giving its name on the command line, the
bytecode for the script is never written to a .pyc or
.pyo file. Thus, the startup time of a script may be reduced
by moving most of its code to a module and having a small bootstrap
script that imports that module. It is also possible to name a
.pyc or .pyo file directly on the command line.
- It is possible to have a file called spam.pyc (or
spam.pyo when -O is used) without a file
spam.py for the same module. This can be used to distribute a
library of Python code in a form that is moderately hard to reverse
engineer.
- The module compileallcan create
.pyc files (or .pyo files when -O is used) for
all modules in a directory.
# 6.2 Standard Modules
Python comes with a library of standard modules, described in a separate
document, the Python Library Reference (../lib/lib.html)
(``Library Reference'' hereafter). Some modules are built into the
interpreter; these provide access to operations that are not part of
the core of the language but are nevertheless built in, either for
efficiency or to provide access to operating system primitives such as
system calls. The set of such modules is a configuration option which
also dependson the underlying platform For example,
the amoeba module is only provided on systems that somehow
support Amoeba primitives. One particular module deserves some
attention: sys, which is built into every
Python interpreter. The variables `sys.ps1` and
`sys.ps2` define the strings used as primary and secondary
prompts:
```text

>>> import sys
>>> sys.ps1
'>>> '
>>> sys.ps2
'... '
>>> sys.ps1 = 'C> '
C> print 'Yuck!'
Yuck!
C>
```
These two variables are only defined if the interpreter is in
interactive mode.
The variable `sys.path` is a list of strings that determine the
interpreter's search path for modules. It is initialized to a default
path taken from the environment variable PYTHONPATH, or from
a built-in default if PYTHONPATH is not set. You can modify
it using standard list operations:
```text

>>> import sys
>>> sys.path.append('/ufs/guido/lib/python')
```
# 6.3 The dir() Function
The built-in function dir() is used to find out which names
a module defines. It returns a sorted list of strings:
```text

>>> import fibo, sys
>>> dir(fibo)
['__name__', 'fib', 'fib2']
>>> dir(sys)
['__displayhook__', '__doc__', '__excepthook__', '__name__', '__stderr__',
'__stdin__', '__stdout__', '_getframe', 'api_version', 'argv',
'builtin_module_names', 'byteorder', 'callstats', 'copyright',
'displayhook', 'exc_clear', 'exc_info', 'exc_type', 'excepthook',
'exec_prefix', 'executable', 'exit', 'getdefaultencoding', 'getdlopenflags',
'getrecursionlimit', 'getrefcount', 'hexversion', 'maxint', 'maxunicode',
'meta_path', 'modules', 'path', 'path_hooks', 'path_importer_cache',
'platform', 'prefix', 'ps1', 'ps2', 'setcheckinterval', 'setdlopenflags',
'setprofile', 'setrecursionlimit', 'settrace', 'stderr', 'stdin', 'stdout',
'version', 'version_info', 'warnoptions']
```
Without arguments, dir() lists the names you have defined
currently:
```text

>>> a = [1, 2, 3, 4, 5]
>>> import fibo, sys
>>> fib = fibo.fib
>>> dir()
['__name__', 'a', 'fib', 'fibo', 'sys']
```
Note that it lists all types of names: variables, modules, functions, etc.
dir() does not list the names of built-in functions and
variables. If you want a list of those, they are defined in the
standard module __builtin__:
```text

>>> import __builtin__
>>> dir(__builtin__)
['ArithmeticError', 'AssertionError', 'AttributeError',
'DeprecationWarning', 'EOFError', 'Ellipsis', 'EnvironmentError',
'Exception', 'False', 'FloatingPointError', 'IOError', 'ImportError',
'IndentationError', 'IndexError', 'KeyError', 'KeyboardInterrupt',
'LookupError', 'MemoryError', 'NameError', 'None', 'NotImplemented',
'NotImplementedError', 'OSError', 'OverflowError', 'OverflowWarning',
'PendingDeprecationWarning', 'ReferenceError',
'RuntimeError', 'RuntimeWarning', 'StandardError', 'StopIteration',
'SyntaxError', 'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError',
'True', 'TypeError', 'UnboundLocalError', 'UnicodeError', 'UserWarning',
'ValueError', 'Warning', 'ZeroDivisionError', '__debug__', '__doc__',
'__import__', '__name__', 'abs', 'apply', 'bool', 'buffer',
'callable', 'chr', 'classmethod', 'cmp', 'coerce', 'compile', 'complex',
'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod',
'enumerate', 'eval', 'execfile', 'exit', 'file', 'filter', 'float',
'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id',
'input', 'int', 'intern', 'isinstance', 'issubclass', 'iter',
'len', 'license', 'list', 'locals', 'long', 'map', 'max', 'min',
'object', 'oct', 'open', 'ord', 'pow', 'property', 'quit',
'range', 'raw_input', 'reduce', 'reload', 'repr', 'round',
'setattr', 'slice', 'staticmethod', 'str', 'string', 'sum', 'super',
'tuple', 'type', 'unichr', 'unicode', 'vars', 'xrange', 'zip']
```
# 6.4 Packages
Packages are a way of structuring Python's module namespace
by using ``dotted module names''. For example, the module name
A.B designates a submodule named "B" in a package named
"A". Just like the use of modules saves the authors of different
modules from having to worry about each other's global variable names,
the use of dotted module names saves the authors of multi-module
packages like NumPy or the Python Imaging Library from having to worry
about each other's module names.
Suppose you want to design a collection of modules (a ``package'') for
the uniform handling of sound files and sound data. There are many
different sound file formats (usually recognized by their extension,
for example: .wav, .aiff, .au), so you may need
to create and maintain a growing collection of modules for the
conversion between the various file formats. There are also many
different operations you might want to perform on sound data (such as
mixing, adding echo, applying an equalizer function, creating an
artificial stereo effect), so in addition you will be writing a
never-ending stream of modules to perform these operations. Here's a
possible structure for your package (expressed in terms of a
hierarchical filesystem):
```text

Sound/ Top-level package
__init__.py Initialize the sound package
Formats/ Subpackage for file format conversions
__init__.py
wavread.py
wavwrite.py
aiffread.py
aiffwrite.py
auread.py
auwrite.py
...
Effects/ Subpackage for sound effects
__init__.py
echo.py
surround.py
reverse.py
...
Filters/ Subpackage for filters
__init__.py
equalizer.py
vocoder.py
karaoke.py
...
```
When importing the package, Python searchs through the directories
on `sys.path` looking for the package subdirectory.
The __init__.py files are required to make Python treat the
directories as containing packages; this is done to prevent
directories with a common name, such as "string", from
unintentionally hiding valid modules that occur later on the module
search path. In the simplest case, __init__.py can just be an
empty file, but it can also execute initialization code for the
package or set the `__all__` variable, described later.
Users of the package can import individual modules from the
package, for example:
```text

import Sound.Effects.echo
```
This loads the submodule Sound.Effects.echo. It must be referenced
with its full name.
```text

Sound.Effects.echo.echofilter(input, output, delay=0.7, atten=4)
```
An alternative way of importing the submodule is:
```text

from Sound.Effects import echo
```
This also loads the submodule echo, and makes it available without
its package prefix, so it can be used as follows:
```text

echo.echofilter(input, output, delay=0.7, atten=4)
```
Yet another variation is to import the desired function or variable directly:
```text

from Sound.Effects.echo import echofilter
```
Again, this loads the submodule echo, but this makes its function
echofilter() directly available:
```text

echofilter(input, output, delay=0.7, atten=4)
```
Note that when using `from package import item`, the
item can be either a submodule (or subpackage) of the package, or some
other name defined in the package, like a function, class or
variable. The `import` statement first tests whether the item is
defined in the package; if not, it assumes it is a module and attempts
to load it. If it fails to find it, an
ImportError exception is raised.
Contrarily, when using syntax like `import item.subitem.subsubitem`, each item except for the last must be
a package; the last item can be a module or a package but can't be a
class or function or variable defined in the previous item.
## 6.4.1 Importing * From a Package
Now what happens when the user writes `from Sound.Effects import
*`? Ideally, one would hope that this somehow goes out to the
filesystem, finds which submodules are present in the package, and
imports them all. Unfortunately, this operation does not work very
well on Mac and Windows platforms, where the filesystem does not
always have accurate information about the case of a filename! On
these platforms, there is no guaranteed way to know whether a file
ECHO.PY should be imported as a module echo,
Echo or ECHO. (For example, Windows 95 has the
annoying practice of showing all file names with a capitalized first
letter.) The DOS 8+3 filename restriction adds another interesting
problem for long module names.
The only solution is for the package author to provide an explicit
index of the package. The import statement uses the following
convention: if a package's __init__.py code defines a list
named `__all__`, it is taken to be the list of module names that
should be imported when `from package import *` is
encountered. It is up to the package author to keep this list
up-to-date when a new version of the package is released. Package
authors may also decide not to support it, if they don't see a use for
importing * from their package. For example, the file
Sounds/Effects/__init__.py could contain the following code:
```text

__all__ = ["echo", "surround", "reverse"]
```
This would mean that `from Sound.Effects import *` would
import the three named submodules of the Sound package.
If `__all__` is not defined, the statement `from Sound.Effects
import *` does not import all submodules from the package
Sound.Effects into the current namespace; it only ensures that the
package Sound.Effects has been imported (possibly running its
initialization code, __init__.py) and then imports whatever names are
defined in the package. This includes any names defined (and
submodules explicitly loaded) by __init__.py. It also includes any
submodules of the package that were explicitly loaded by previous
import statements. Consider this code:
```text

import Sound.Effects.echo
import Sound.Effects.surround
from Sound.Effects import *
```
In this example, the echo and surround modules are imported in the
current namespace because they are defined in the
Sound.Effects package when the `from...import` statement
is executed. (This also works when `__all__` is defined.)
Note that in general the practice of importing `*` from a module or
package is frowned upon, since it often causes poorly readable code.
However, it is okay to use it to save typing in interactive sessions,
and certain modules are designed to export only names that follow
certain patterns.
Remember, there is nothing wrong with using `from Package
import specific_submodule`! In fact, this is the
recommended notation unless the importing module needs to use
submodules with the same name from different packages.
## 6.4.2 Intra-package References
The submodules often need to refer to each other. For example, the
surround module might use the echo module. In fact, such references
are so common that the `import` statement first looks in the
containing package before looking in the standard module search path.
Thus, the surround module can simply use `import echo` or
`from echo import echofilter`. If the imported module is not
found in the current package (the package of which the current module
is a submodule), the `import` statement looks for a top-level module
with the given name.
When packages are structured into subpackages (as with the
Sound package in the example), there's no shortcut to refer
to submodules of sibling packages - the full name of the subpackage
must be used. For example, if the module
Sound.Filters.vocoder needs to use the echo module
in the Sound.Effects package, it can use `from
Sound.Effects import echo`.
## 6.4.3 Packages in Multiple Directories
Packages support one more special attribute, __path__. This
is initialized to be a list containing the name of the directory
holding the package's __init__.py before the code in that file
is executed. This variable can be modified; doing so affects future
searches for modules and subpackages contained in the package.
While this feature is not often needed, it can be used to extend the
set of modules found in a package.


7. Input and Output

node8.html | tut.html | node10.html | Python Tutorial | node2.html
Previous:
6. Modules (node8.html)
Up:
Python Tutorial (tut.html)
Next:
8. Errors and Exceptions (node10.html)
---
- 7.1 Fancier Output Formatting (node9.html#SECTION009100000000000000000)
  7.2 Reading and Writing Files (node9.html#SECTION009200000000000000000)
  - 7.2.1 Methods of File Objects (node9.html#SECTION009210000000000000000)
  7.2.2 The pickle Module (node9.html#SECTION009220000000000000000)
---
# 7. Input and Output
There are several ways to present the output of a program; data can be
printed in a human-readable form, or written to a file for future use.
This chapter will discuss some of the possibilities.
# 7.1 Fancier Output Formatting
So far we've encountered two ways of writing values: expression
statements and the print statement. (A third way is using
the write() method of file objects; the standard output file
can be referenced as `sys.stdout`. See the Library Reference for
more information on this.)
Often you'll want more control over the formatting of your output than
simply printing space-separated values. There are two ways to format
your output; the first way is to do all the string handling yourself;
using string slicing and concatenation operations you can create any
lay-out you can imagine. The standard module
stringcontains some useful operations
for padding strings to a given column width; these will be discussed
shortly. The second way is to use the `%` operator with a
string as the left argument. The `%` operator interprets the
left argument much like a sprintf()-style format
string to be applied to the right argument, and returns the string
resulting from this formatting operation.
One question remains, of course: how do you convert values to strings?
Luckily, Python has ways to convert any value to a string: pass it to
the repr() or str() functions. Reverse quotes
(````) are equivalent to repr(), but their use is
discouraged.
The str() function is meant to return representations of
values which are fairly human-readable, while repr() is
meant to generate representations which can be read by the interpreter
(or will force a SyntaxError if there is not equivalent
syntax). For objects which don't have a particular representation for
human consumption, str() will return the same value as
repr(). Many values, such as numbers or structures like
lists and dictionaries, have the same representation using either
function. Strings and floating point numbers, in particular, have two
distinct representations.
Some examples:
```text

>>> s = 'Hello, world.'
>>> str(s)
'Hello, world.'
>>> repr(s)
"'Hello, world.'"
>>> str(0.1)
'0.1'
>>> repr(0.1)
'0.10000000000000001'
>>> x = 10 * 3.25
>>> y = 200 * 200
>>> s = 'The value of x is ' + repr(x) + ', and y is ' + repr(y) + '...'
>>> print s
The value of x is 32.5, and y is 40000...
>>> # The repr() of a string adds string quotes and backslashes:
... hello = 'hello, world\n'
>>> hellos = repr(hello)
>>> print hellos
'hello, world\n'
>>> # The argument to repr() may be any Python object:
... repr((x, y, ('spam', 'eggs')))
"(32.5, 40000, ('spam', 'eggs'))"
>>> # reverse quotes are convenient in interactive sessions:
... `x, y, ('spam', 'eggs')`
"(32.5, 40000, ('spam', 'eggs'))"
```
Here are two ways to write a table of squares and cubes:
```text

>>> import string
>>> for x in range(1, 11):
... print string.rjust(repr(x), 2), string.rjust(repr(x*x), 3),
... # Note trailing comma on previous line
... print string.rjust(repr(x*x*x), 4)
...
1 1 1
2 4 8
3 9 27
4 16 64
5 25 125
6 36 216
7 49 343
8 64 512
9 81 729
10 100 1000
>>> for x in range(1,11):
... print '%2d %3d %4d' % (x, x*x, x*x*x)
...
1 1 1
2 4 8
3 9 27
4 16 64
5 25 125
6 36 216
7 49 343
8 64 512
9 81 729
10 100 1000
```
(Note that one space between each column was added by the way
print works: it always adds spaces between its arguments.)
This example demonstrates the function string.rjust(),
which right-justifies a string in a field of a given width by padding
it with spaces on the left. There are similar functions
string.ljust() and string.center(). These
functions do not write anything, they just return a new string. If
the input string is too long, they don't truncate it, but return it
unchanged; this will mess up your column lay-out but that's usually
better than the alternative, which would be lying about a value. (If
you really want truncation you can always add a slice operation, as in
"string.ljust(x, n)[0:n]".)
There is another function, string.zfill(), which pads a
numeric string on the left with zeros. It understands about plus and
minus signs:
```text

>>> import string
>>> string.zfill('12', 5)
'00012'
>>> string.zfill('-3.14', 7)
'-003.14'
>>> string.zfill('3.14159265359', 5)
'3.14159265359'
```
Using the `%` operator looks like this:
```text

>>> import math
>>> print 'The value of PI is approximately %5.3f.' % math.pi
The value of PI is approximately 3.142.
```
If there is more than one format in the string, you need to pass a
tuple as right operand, as in this example:
```text

>>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 7678}
>>> for name, phone in table.items():
... print '%-10s ==> %10d' % (name, phone)
...
Jack ==> 4098
Dcab ==> 7678
Sjoerd ==> 4127
```
Most formats work exactly as in C and require that you pass the proper
type; however, if you don't you get an exception, not a core dump.
The `%s` format is more relaxed: if the corresponding argument is
not a string object, it is converted to string using the
str() built-in function. Using `*` to pass the width
or precision in as a separate (integer) argument is supported. The
C formats `%n` and `%p` are not supported.
If you have a really long format string that you don't want to split
up, it would be nice if you could reference the variables to be
formatted by name instead of by position. This can be done by using
form `%(name)format`, as shown here:
```text

>>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 8637678}
>>> print 'Jack: %(Jack)d; Sjoerd: %(Sjoerd)d; Dcab: %(Dcab)d' % table
Jack: 4098; Sjoerd: 4127; Dcab: 8637678
```
This is particularly useful in combination with the new built-in
vars() function, which returns a dictionary containing all
local variables.
# 7.2 Reading and Writing Files
open() returns a file
object, and is most commonly used with two arguments:
"open(filename, mode)".
```text

>>> f=open('/tmp/workfile', 'w')
>>> print f
<open file '/tmp/workfile', mode 'w' at 80a0960>
```
The first argument is a string containing the filename. The second
argument is another string containing a few characters describing the
way in which the file will be used. mode can be `'r'` when
the file will only be read, `'w'` for only writing (an existing
file with the same name will be erased), and `'a'` opens the file
for appending; any data written to the file is automatically added to
the end. `'r+'` opens the file for both reading and writing.
The mode argument is optional; `'r'` will be assumed if
it's omitted.
On Windows and the Macintosh, `'b'` appended to the
mode opens the file in binary mode, so there are also modes like
`'rb'`, `'wb'`, and `'r+b'`. Windows makes a
distinction between text and binary files; the end-of-line characters
in text files are automatically altered slightly when data is read or
written. This behind-the-scenes modification to file data is fine for
ASCII text files, but it'll corrupt binary data like that in JPEGs or
.EXE files. Be very careful to use binary mode when reading and
writing such files. (Note that the precise semantics of text mode on
the Macintosh depends on the underlying C library being used.)
## 7.2.1 Methods of File Objects
The rest of the examples in this section will assume that a file
object called `f` has already been created.
To read a file's contents, call `f.read( size )`, which reads
some quantity of data and returns it as a string. size is an
optional numeric argument. When size is omitted or negative,
the entire contents of the file will be read and returned; it's your
problem if the file is twice as large as your machine's memory.
Otherwise, at most size bytes are read and returned. If the end
of the file has been reached, `f.read()` will return an empty
string (`""`).
```text

>>> f.read()
'This is the entire file.\n'
>>> f.read()
''
```
`f.readline()` reads a single line from the file; a newline
character (`\n`) is left at the end of the string, and is only
omitted on the last line of the file if the file doesn't end in a
newline. This makes the return value unambiguous; if
`f.readline()` returns an empty string, the end of the file has
been reached, while a blank line is represented by `'\n'`, a
string containing only a single newline.
```text

>>> f.readline()
'This is the first line of the file.\n'
>>> f.readline()
'Second line of the file\n'
>>> f.readline()
''
```
`f.readlines()` returns a list containing all the lines of data
in the file. If given an optional parameter sizehint, it reads
that many bytes from the file and enough more to complete a line, and
returns the lines from that. This is often used to allow efficient
reading of a large file by lines, but without having to load the
entire file in memory. Only complete lines will be returned.
```text

>>> f.readlines()
['This is the first line of the file.\n', 'Second line of the file\n']
```
`f.write( string )` writes the contents of string to
the file, returning `None`.
```text

>>> f.write('This is a test\n')
```
`f.tell()` returns an integer giving the file object's current
position in the file, measured in bytes from the beginning of the
file. To change the file object's position, use
"f.seek(offset, from_what)". The position is
computed from adding offset to a reference point; the reference
point is selected by the from_what argument. A
from_what value of 0 measures from the beginning of the file, 1
uses the current file position, and 2 uses the end of the file as the
reference point. from_what can be omitted and defaults to 0,
using the beginning of the file as the reference point.
```text

>>> f=open('/tmp/workfile', 'r+')
>>> f.write('0123456789abcdef')
>>> f.seek(5) # Go to the 6th byte in the file
>>> f.read(1)
'5'
>>> f.seek(-3, 2) # Go to the 3rd byte before the end
>>> f.read(1)
'd'
```
When you're done with a file, call `f.close()` to close it and
free up any system resources taken up by the open file. After calling
`f.close()`, attempts to use the file object will automatically fail.
```text

>>> f.close()
>>> f.read()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ValueError: I/O operation on closed file
```
File objects have some additional methods, such as
isatty() and truncate() which are less frequently
used; consult the Library Reference for a complete guide to file
objects.
## 7.2.2 The pickle Module
Strings can easily be written to and read from a file. Numbers take a
bit more effort, since the read() method only returns
strings, which will have to be passed to a function like
string.atoi(), which takes a string like `'123'` and
returns its numeric value 123. However, when you want to save more
complex data types like lists, dictionaries, or class instances,
things get a lot more complicated.
Rather than have users be constantly writing and debugging code to
save complicated data types, Python provides a standard module called
pickle. This is an amazing module that can take almost
any Python object (even some forms of Python code!), and convert it to
a string representation; this process is called pickling.
Reconstructing the object from the string representation is called
unpickling. Between pickling and unpickling, the string
representing the object may have been stored in a file or data, or
sent over a network connection to some distant machine.
If you have an object `x`, and a file object `f` that's been
opened for writing, the simplest way to pickle the object takes only
one line of code:
```text

pickle.dump(x, f)
```
To unpickle the object again, if `f` is a file object which has
been opened for reading:
```text

x = pickle.load(f)
```
(There are other variants of this, used when pickling many objects or
when you don't want to write the pickled data to a file; consult the
complete documentation for pickle in the Library Reference.)
pickle is the standard way to make Python objects which can
be stored and reused by other programs or by a future invocation of
the same program; the technical term for this is a
persistent object. Because pickle is so widely used,
many authors who write Python extensions take care to ensure that new
data types such as matrices can be properly pickled and unpickled.


Python Tutorial

../index.html | node1.html | Python Tutorial | node2.html
Up:
Python Documentation Index (../index.html)
Next:
Front Matter (node1.html)
---
# Python Tutorial
Guido van Rossum
Fred L. Drake, Jr., editor
PythonLabs
Email: python-docs@python.org
Release 2.3
July 29, 2003


About this document ...

acks.html | whatsnew23.html | What's New in Python 2.3 | contents.html
Previous:
22 Acknowledgements (acks.html)
Up:
What's New in Python (whatsnew23.html)
---
# About this document ...
What's New in Python 2.3
This document was generated using the LaTeX2HTML (http://saftsack.fs.uni-bayreuth.de/~latex2ht/) translator.
LaTeX2HTML (http://saftsack.fs.uni-bayreuth.de/~latex2ht/) is Copyright ©
1993, 1994, 1995, 1996, 1997, Nikos
Drakos (http://cbl.leeds.ac.uk/nikos/personal.html), Computer Based Learning Unit, University of
Leeds, and Copyright © 1997, 1998, Ross
Moore (http://www.maths.mq.edu.au/~ross/), Mathematics Department, Macquarie University,
Sydney.
The application of LaTeX2HTML (http://saftsack.fs.uni-bayreuth.de/~latex2ht/) to the Python
documentation has been heavily tailored by Fred L. Drake,
Jr. Original navigation icons were contributed by Christopher
Petrilli.
---
## Comments and Questions
General comments and questions regarding this document should
be sent by email to python-docs@python.org (mailto:python-docs@python.org). If you find specific errors in
this document, either in the content or the presentation, please
report the bug at the Python Bug
Tracker (http://sourceforge.net/bugs/?group_id=5470) at SourceForge (http://sourceforge.net/).
Questions regarding how to use the information in this
document should be sent to the Python news group, comp.lang.python (news:comp.lang.python), or the Python mailing list (http://www.python.org/mailman/listinfo/python-list) (which is gated to the newsgroup and
carries the same content).
For any of these channels, please be sure not to send HTML email.
Thanks.
---
acks.html | whatsnew23.html | What's New in Python 2.3 | contents.html
Previous:
22 Acknowledgements (acks.html)
Up:
What's New in Python (whatsnew23.html)
---
Release 1.00.


22 Acknowledgements

node22.html | whatsnew23.html | about.html | What's New in Python 2.3 | contents.html
Previous:
21 Porting to Python (node22.html)
Up:
What's New in Python (whatsnew23.html)
Next:
About this document ... (about.html)
---
# 22 Acknowledgements
The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Jeff Bauer, Simon Brunning, Brett Cannon, Michael Chermside,
Andrew Dalke, Scott David Daniels, Fred L. Drake, Jr., Kelly Gerber,
Raymond Hettinger, Michael Hudson, Chris Lambert, Detlef Lannert,
Martin von Löwis, Andrew MacIntyre, Lalo Martins, Chad Netzer,
Gustavo Niemeyer, Neal Norwitz, Hans Nowak, Chris Reedy, Francesco
Ricciardi, Vinay Sajip, Neil Schemenauer, Roman Suzi, Jason Tishler,
Just van Rossum.


Contents

whatsnew23.html | whatsnew23.html | node2.html | What's New in Python 2.3
Previous:
What's New in Python (whatsnew23.html)
Up:
What's New in Python (whatsnew23.html)
Next:
1 PEP 218: A (node2.html)
---
## Contents
Table of Contents
- 1 PEP 218: A Standard Set Datatype (node2.html)
  2 PEP 255: Simple Generators (section-generators.html)
  3 PEP 263: Source Code Encodings (section-encodings.html)
  4 PEP 273: Importing Modules from Zip Archives (node5.html)
  5 PEP 277: Unicode file name support for Windows NT (node6.html)
  6 PEP 278: Universal Newline Support (node7.html)
  7 PEP 279: enumerate() (section-enumerate.html)
  8 PEP 282: The logging Package (node9.html)
  9 PEP 285: A Boolean Type (section-bool.html)
  10 PEP 293: Codec Error Handling Callbacks (node11.html)
  11 PEP 301: Package Index and Metadata for
  Distutils (section-pep301.html)
  12 PEP 302: New Import Hooks (section-pep302.html)
  13 PEP 305: Comma-separated Files (node14.html)
  14 PEP 307: Pickle Enhancements (section-pep305.html)
  15 Extended Slices (section-slices.html)
  16 Other Language Changes (node17.html)
  - 16.1 String Changes (node17.html#SECTION0001710000000000000000)
  16.2 Optimizations (node17.html#SECTION0001720000000000000000)
  17 New, Improved, and Deprecated Modules (node18.html)
  - 17.1 Date/Time Type (node18.html#SECTION0001810000000000000000)
  17.2 The optparse Module (node18.html#SECTION0001820000000000000000)
  18 Pymalloc: A Specialized Object Allocator (section-pymalloc.html)
  19 Build and C API Changes (node20.html)
  - 19.1 Port-Specific Changes (node20.html#SECTION0002010000000000000000)
  20 Other Changes and Fixes (section-other.html)
  21 Porting to Python 2.3 (node22.html)
  22 Acknowledgements (acks.html)
  About this document ... (about.html)
End of Table of Contents
This article explains the new features in Python 2.3. Python 2.3 was
released on July 29, 2003.
The main themes for Python 2.3 are polishing some of the features
added in 2.2, adding various small but useful enhancements to the core
language, and expanding the standard library. The new object model
introduced in the previous version has benefited from 18 months of
bugfixes and from optimization efforts that have improved the
performance of new-style classes. A few new built-in functions have
been added such as sum() and enumerate(). The
in operator can now be used for substring searches (e.g.
`"ab" in "abc"` returns True).
Some of the many new library features include Boolean, set, heap, and
date/time data types, the ability to import modules from ZIP-format
archives, metadata support for the long-awaited Python catalog, an
updated version of IDLE, and modules for logging messages, wrapping
text, parsing CSV files, processing command-line options, using BerkeleyDB
databases... the list of new and enhanced modules is lengthy.
This article doesn't attempt to provide a complete specification of
the new features, but instead provides a convenient overview. For
full details, you should refer to the documentation for Python 2.3,
such as the Python Library Reference (../lib/lib.html) and
the Python Reference Manual (../ref/ref.html). If you want
to understand the complete implementation and design rationale,
refer to the PEP for a particular new feature.


What's New in Python 2.3

../index.html | contents.html | What's New in Python 2.3 | contents.html
Up:
Python Documentation Index (../index.html)
Next:
---
# What's New in Python 2.3
A.M. Kuchling
amk@amk.ca


section-bool.html | whatsnew23.html | section-pep301.html | What's New in Python 2.3 | contents.html
Previous:
9 PEP 285: A (section-bool.html)
Up:
What's New in Python (whatsnew23.html)
Next:
11 PEP 301: Package (section-pep301.html)
---
# 10 PEP 293: Codec Error Handling Callbacks
When encoding a Unicode string into a byte string, unencodable
characters may be encountered. So far, Python has allowed specifying
the error processing as either ``strict'' (raising
UnicodeError), ``ignore'' (skipping the character), or
``replace'' (using a question mark in the output string), with
``strict'' being the default behavior. It may be desirable to specify
alternative processing of such errors, such as inserting an XML
character reference or HTML entity reference into the converted
string.
Python now has a flexible framework to add different processing
strategies. New error handlers can be added with
codecs.register_error, and codecs then can access the error
handler with codecs.lookup_error. An equivalent C API has
been added for codecs written in C. The error handler gets the
necessary state information such as the string being converted, the
position in the string where the error was detected, and the target
encoding. The handler can then either raise an exception or return a
replacement string.
Two additional error handlers have been implemented using this
framework: ``backslashreplace'' uses Python backslash quoting to
represent unencodable characters and ``xmlcharrefreplace'' emits
XML character references.
See Also:


13 PEP 305: Comma-separated Files

section-pep302.html | whatsnew23.html | section-pep305.html | What's New in Python 2.3 | contents.html
Previous:
12 PEP 302: New (section-pep302.html)
Up:
What's New in Python (whatsnew23.html)
Next:
14 PEP 307: Pickle (section-pep305.html)
---
# 13 PEP 305: Comma-separated Files
Comma-separated files are a format frequently used for exporting data
from databases and spreadsheets. Python 2.3 adds a parser for
comma-separated files.
Comma-separated format is deceptively simple at first glance:
```text

Costs,150,200,3.95
```
Read a line and call `line.split(',')`: what could be simpler?
But toss in string data that can contain commas, and things get more
complicated:
```text

"Costs",150,200,3.95,"Includes taxes, shipping, and sundry items"
```
A big ugly regular expression can parse this, but using the new
csv package is much simpler:
```text

import csv

input = open('datafile', 'rb')
reader = csv.reader(input)
for line in reader:
print line
```
The reader function takes a number of different options.
The field separator isn't limited to the comma and can be changed to
any character, and so can the quoting and line-ending characters.
Different dialects of comma-separated files can be defined and
registered; currently there are two dialects, both used by Microsoft Excel.
A separate csv.writer class will generate comma-separated files
from a succession of tuples or lists, quoting strings that contain the
delimiter.
See Also:


16 Other Language Changes

section-slices.html | whatsnew23.html | node18.html | What's New in Python 2.3 | contents.html
Previous:
15 Extended Slices (section-slices.html)
Up:
What's New in Python (whatsnew23.html)
Next:
17 New, Improved, and (node18.html)
---
- 16.1 String Changes (node17.html#SECTION0001710000000000000000)
  16.2 Optimizations (node17.html#SECTION0001720000000000000000)
---
# 16 Other Language Changes
Here are all of the changes that Python 2.3 makes to the core Python
language.
- The yield statement is now always a keyword, as
described in section 2 (section-generators.html#section-generators) of this document.
- A new built-in function enumerate()
was added, as described in section 7 (section-enumerate.html#section-enumerate) of this
document.
- Two new constants, True and False were
added along with the built-in bool type, as described in
section 9 (section-bool.html#section-bool) of this document.
- The int() type constructor will now return a long
integer instead of raising an OverflowError when a string
or floating-point number is too large to fit into an integer. This
can lead to the paradoxical result that
`isinstance(int( expression ), int)` is false, but that seems
unlikely to cause problems in practice.
- Built-in types now support the extended slicing syntax,
as described in section 15 (section-slices.html#section-slices) of this document.
- A new built-in function, sum(iterable, start=0),
adds up the numeric items in the iterable object and returns their sum.
sum() only accepts numbers, meaning that you can't use it
to concatenate a bunch of strings. (Contributed by Alex
Martelli.)
- `list.insert( pos , value )` used to
insert value at the front of the list when pos was
negative. The behaviour has now been changed to be consistent with
slice indexing, so when pos is -1 the value will be inserted
before the last element, and so forth.
- `list.index( value )`, which searches for value
within the list and returns its index, now takes optional
start and stop arguments to limit the search to
only part of the list.
- Dictionaries have a new method, pop(key[,
default]), that returns the value corresponding to key
and removes that key/value pair from the dictionary. If the requested
key isn't present in the dictionary, default is returned if it's
specified and KeyError raised if it isn't.
```text

>>> d = {1:2}
>>> d
{1: 2}
>>> d.pop(4)
Traceback (most recent call last):
File "stdin", line 1, in ?
KeyError: 4
>>> d.pop(1)
2
>>> d.pop(1)
Traceback (most recent call last):
File "stdin", line 1, in ?
KeyError: 'pop(): dictionary is empty'
>>> d
{}
>>>
```
There's also a new class method,
dict.fromkeys(iterable, value), that
creates a dictionary with keys taken from the supplied iterator
iterable and all values set to value, defaulting to
`None`.
(Patches contributed by Raymond Hettinger.)
Also, the dict() constructor now accepts keyword arguments to
simplify creating small dictionaries:
```text

>>> dict(red=1, blue=2, green=3, black=4)
{'blue': 2, 'black': 4, 'green': 3, 'red': 1}
```
(Contributed by Just van Rossum.)
- The assert statement no longer checks the `__debug__`
flag, so you can no longer disable assertions by assigning to `__debug__`.
Running Python with the -O switch will still generate
code that doesn't execute any assertions.
- Most type objects are now callable, so you can use them
to create new objects such as functions, classes, and modules. (This
means that the new module can be deprecated in a future
Python version, because you can now use the type objects available in
the types module.)
For example, you can create a new module object with the following code:
```text

>>> import types
>>> m = types.ModuleType('abc','docstring')
>>> m
<module 'abc' (built-in)>
>>> m.__doc__
'docstring'
```
- A new warning, PendingDeprecationWarning was added to
indicate features which are in the process of being
deprecated. The warning will not be printed by default. To
check for use of features that will be deprecated in the future,
supply -Walways::PendingDeprecationWarning:: on the
command line or use warnings.filterwarnings().
- The process of deprecating string-based exceptions, as
in `raise "Error occurred"`, has begun. Raising a string will
now trigger PendingDeprecationWarning.
- Using `None` as a variable name will now result in a
SyntaxWarning warning. In a future version of Python,
`None` may finally become a keyword.
- The xreadlines() method of file objects, introduced in
Python 2.1, is no longer necessary because files now behave as their
own iterator. xreadlines() was originally introduced as a
faster way to loop over all the lines in a file, but now you can
simply write `for line in file_obj`. File objects also have a
new read-only encoding attribute that gives the encoding used
by the file; Unicode strings written to the file will be automatically
converted to bytes using the given encoding.
- The method resolution order used by new-style classes has
changed, though you'll only notice the difference if you have a really
complicated inheritance hierarchy. Classic classes are unaffected by
this change. Python 2.2 originally used a topological sort of a
class's ancestors, but 2.3 now uses the C3 algorithm as described in
the paper ``A Monotonic Superclass Linearization for
Dylan'' (http://www.webcom.com/haahr/dylan/linearization-oopsla96.html).
To understand the motivation for this change,
read Michele Simionato's article
``Python 2.3 Method Resolution Order'' (http://www.python.org/2.3/mro.html), or
read the thread on python-dev starting with the message at
http://mail.python.org/pipermail/python-dev/2002-October/029035.html.
Samuele Pedroni first pointed out the problem and also implemented the
fix by coding the C3 algorithm.
- Python runs multithreaded programs by switching between threads
after executing N bytecodes. The default value for N has been
increased from 10 to 100 bytecodes, speeding up single-threaded
applications by reducing the switching overhead. Some multithreaded
applications may suffer slower response time, but that's easily fixed
by setting the limit back to a lower number using
sys.setcheckinterval(N).
The limit can be retrieved with the new
sys.getcheckinterval() function.
- One minor but far-reaching change is that the names of extension
types defined by the modules included with Python now contain the
module and a "." in front of the type name. For example, in
Python 2.2, if you created a socket and printed its
__class__, you'd get this output:
```text

>>> s = socket.socket()
>>> s.__class__
<type 'socket'>
```
In 2.3, you get this:
```text

>>> s.__class__
<type '_socket.socket'>
```
- One of the noted incompatibilities between old- and new-style
classes has been removed: you can now assign to the
__name__ and __bases__ attributes of new-style
classes. There are some restrictions on what can be assigned to
__bases__ along the lines of those relating to assigning to
an instance's __class__ attribute.
## 16.1 String Changes
- The in operator now works differently for strings.
Previously, when evaluating `X in Y` where X
and Y are strings, X could only be a single character.
That's now changed; X can be a string of any length, and
`X in Y` will return True if X is a
substring of Y. If X is the empty string, the result is
always True.
```text

>>> 'ab' in 'abcd'
True
>>> 'ad' in 'abcd'
False
>>> '' in 'abcd'
True
```
Note that this doesn't tell you where the substring starts; if you
need that information, use the find() string method.
- The strip(), lstrip(), and rstrip()
string methods now have an optional argument for specifying the
characters to strip. The default is still to remove all whitespace
characters:
```text

>>> ' abc '.strip()
'abc'
>>> '><><abc<><><>'.strip('<>')
'abc'
>>> '><><abc<><><>\n'.strip('<>')
'abc<><><>\n'
>>> u'\u4000\u4001abc\u4000'.strip(u'\u4000')
u'\u4001abc'
>>>
```
(Suggested by Simon Brunning and implemented by Walter Dörwald.)
- The startswith() and endswith()
string methods now accept negative numbers for the start and end
parameters.
- Another new string method is zfill(), originally a
function in the string module. zfill() pads a
numeric string with zeros on the left until it's the specified width.
Note that the `%` operator is still more flexible and powerful
than zfill().
```text

>>> '45'.zfill(4)
'0045'
>>> '12345'.zfill(4)
'12345'
>>> 'goofy'.zfill(6)
'0goofy'
```
(Contributed by Walter Dörwald.)
- A new type object, basestring, has been added.
Both 8-bit strings and Unicode strings inherit from this type, so
`isinstance(obj, basestring)` will return True for
either kind of string. It's a completely abstract type, so you
can't create basestring instances.
- Interned strings are no longer immortal and will now be
garbage-collected in the usual way when the only reference to them is
from the internal dictionary of interned strings. (Implemented by
Oren Tirosh.)
## 16.2 Optimizations
- The creation of new-style class instances has been made much
faster; they're now faster than classic classes!
- The sort() method of list objects has been extensively
rewritten by Tim Peters, and the implementation is significantly
faster.
- Multiplication of large long integers is now much faster thanks
to an implementation of Karatsuba multiplication, an algorithm that
scales better than the O(n*n) required for the grade-school
multiplication algorithm. (Original patch by Christopher A. Craig,
and significantly reworked by Tim Peters.)
- The `SET_LINENO` opcode is now gone. This may provide a
small speed increase, depending on your compiler's idiosyncrasies.
See section 20 (section-other.html#section-other) for a longer explanation.
(Removed by Michael Hudson.)
- xrange() objects now have their own iterator, making
`for i in xrange(n)` slightly faster than
`for i in range(n)`. (Patch by Raymond Hettinger.)
- A number of small rearrangements have been made in various
hotspots to improve performance, such as inlining a function or removing
some code. (Implemented mostly by GvR, but lots of people have
contributed single changes.)
The net result of the 2.3 optimizations is that Python 2.3 runs the
pystone benchmark around 25% faster than Python 2.2.


node17.html | whatsnew23.html | section-pymalloc.html | What's New in Python 2.3 | contents.html
Previous:
16 Other Language Changes (node17.html)
Up:
What's New in Python (whatsnew23.html)
Next:
18 Pymalloc: A Specialized (section-pymalloc.html)
---
- 17.1 Date/Time Type (node18.html#SECTION0001810000000000000000)
  17.2 The optparse Module (node18.html#SECTION0001820000000000000000)
---
# 17 New, Improved, and Deprecated Modules
As usual, Python's standard library received a number of enhancements and
bug fixes. Here's a partial list of the most notable changes, sorted
alphabetically by module name. Consult the
Misc/NEWS file in the source tree for a more
complete list of changes, or look through the CVS logs for all the
details.
- The array module now supports arrays of Unicode
characters using the "u" format character. Arrays also now
support using the `+=` assignment operator to add another array's
contents, and the `*=` assignment operator to repeat an array.
(Contributed by Jason Orendorff.)
- The bsddb module has been replaced by version 4.1.1
of the PyBSDDB (http://pybsddb.sourceforge.net) package,
providing a more complete interface to the transactional features of
the BerkeleyDB library.
The old version of the module has been renamed to
bsddb185 and is no longer built automatically; you'll
have to edit Modules/Setup to enable it. Note that the new
bsddb package is intended to be compatible with the
old module, so be sure to file bugs if you discover any
incompatibilities. When upgrading to Python 2.3, if the new interpreter is compiled
with a new version of
the underlying BerkeleyDB library, you will almost certainly have to
convert your database files to the new version. You can do this
fairly easily with the new scripts db2pickle.py and
pickle2db.py which you will find in the distribution's
Tools/scripts directory. If you've already been using the PyBSDDB
package and importing it as bsddb3, you will have to change your
`import` statements to import it as bsddb.
- The new bz2 module is an interface to the bz2 data
compression library. bz2-compressed data is usually smaller than
corresponding zlib-compressed data. (Contributed by Gustavo Niemeyer.)
- A set of standard date/type types has been added in the new datetime
module. See the following section for more details.
- The Distutils Extension class now supports
an extra constructor argument named depends for listing
additional source files that an extension depends on. This lets
Distutils recompile the module if any of the dependency files are
modified. For example, if sampmodule.c includes the header
file sample.h, you would create the Extension object like
this:
```text

ext = Extension("samp",
sources=["sampmodule.c"],
depends=["sample.h"])
```
Modifying sample.h would then cause the module to be recompiled.
(Contributed by Jeremy Hylton.)
- Other minor changes to Distutils:
it now checks for the CC, CFLAGS, CPP,
LDFLAGS, and CPPFLAGS environment variables, using
them to override the settings in Python's configuration (contributed
by Robert Weber).
- Previously the doctest module would only search the
docstrings of public methods and functions for test cases, but it now
also examines private ones as well. The DocTestSuite(
function creates a unittest.TestSuite object from a set of
doctest tests.
- The new gc.get_referents(object) function returns a
list of all the objects referenced by object.
- The getopt module gained a new function,
gnu_getopt(), that supports the same arguments as the existing
getopt() function but uses GNU-style scanning mode.
The existing getopt() stops processing options as soon as a
non-option argument is encountered, but in GNU-style mode processing
continues, meaning that options and arguments can be mixed. For
example:
```text

>>> getopt.getopt(['-f', 'filename', 'output', '-v'], 'f:v')
([('-f', 'filename')], ['output', '-v'])
>>> getopt.gnu_getopt(['-f', 'filename', 'output', '-v'], 'f:v')
([('-f', 'filename'), ('-v', '')], ['output'])
```
(Contributed by Peter Åstrand.)
- The grp, pwd, and resource modules
now return enhanced tuples:
```text

>>> import grp
>>> g = grp.getgrnam('amk')
>>> g.gr_name, g.gr_gid
('amk', 500)
```
- The gzip module can now handle files exceeding 2 Gb.
- The new heapq module contains an implementation of a
heap queue algorithm. A heap is an array-like data structure that
keeps items in a partially sorted order such that, for every index
k, `heap[ k ] <= heap[2* k +1]` and
`heap[ k ] <= heap[2* k +2]`. This makes it quick to
remove the smallest item, and inserting a new item while maintaining
the heap property is O(lg n). (See
http://www.nist.gov/dads/HTML/priorityque.html for more
information about the priority queue data structure.)
The heapq module provides heappush() and
heappop() functions for adding and removing items while
maintaining the heap property on top of some other mutable Python
sequence type. Here's an example that uses a Python list:
```text

>>> import heapq
>>> heap = []
>>> for item in [3, 7, 5, 11, 1]:
... heapq.heappush(heap, item)
...
>>> heap
[1, 3, 5, 11, 7]
>>> heapq.heappop(heap)
1
>>> heapq.heappop(heap)
3
>>> heap
[5, 7, 11]
```
(Contributed by Kevin O'Connor.)
- The IDLE integrated development environment has been updated
using the code from the IDLEfork project
(http://idlefork.sf.net). The most notable feature is that the
code being developed is now executed in a subprocess, meaning that
there's no longer any need for manual `reload()` operations.
IDLE's core code has been incorporated into the standard library as the
idlelib package.
- The imaplib module now supports IMAP over SSL.
(Contributed by Piers Lauder and Tino Lange.)
- The itertools contains a number of useful functions for
use with iterators, inspired by various functions provided by the ML
and Haskell languages. For example,
`itertools.ifilter(predicate, iterator)` returns all elements in
the iterator for which the function predicate() returns
True, and `itertools.repeat(obj, N )` returns
`obj` N times. There are a number of other functions in
the module; see the package's reference
documentation (../lib/module-itertools.html) for details.
(Contributed by Raymond Hettinger.)
- Two new functions in the math module,
degrees(rads) and radians(degs),
convert between radians and degrees. Other functions in the
math module such as math.sin() and
math.cos() have always required input values measured in
radians. Also, an optional base argument was added to
math.log() to make it easier to compute logarithms for
bases other than `e` and `10`. (Contributed by Raymond
Hettinger.)
- Several new POSIX functions (getpgid(), killpg(),
lchown(), loadavg(), major(), makedev(),
minor(), and mknod()) were added to the
posix module that underlies the os module.
(Contributed by Gustavo Niemeyer, Geert Jansen, and Denis S. Otkidach.)
- In the os module, the *stat() family of
functions can now report fractions of a second in a timestamp. Such
time stamps are represented as floats, similar to
the value returned by time.time().
During testing, it was found that some applications will break if time
stamps are floats. For compatibility, when using the tuple interface
of the stat_result time stamps will be represented as integers.
When using named fields (a feature first introduced in Python 2.2),
time stamps are still represented as integers, unless
os.stat_float_times() is invoked to enable float return
values:
```text

>>> os.stat("/tmp").st_mtime
1034791200
>>> os.stat_float_times(True)
>>> os.stat("/tmp").st_mtime
1034791200.6335014
```
In Python 2.4, the default will change to always returning floats.
Application developers should enable this feature only if all their
libraries work properly when confronted with floating point time
stamps, or if they use the tuple API. If used, the feature should be
activated on an application level instead of trying to enable it on a
per-use basis.
- The optparse module contains a new parser for command-line arguments
that can convert option values to a particular Python type
and will automatically generate a usage message. See the following section for
more details.
- The old and never-documented linuxaudiodev module has
been deprecated, and a new version named ossaudiodev has been
added. The module was renamed because the OSS sound drivers can be
used on platforms other than Linux, and the interface has also been
tidied and brought up to date in various ways. (Contributed by Greg
Ward and Nicholas FitzRoy-Dale.)
- The new platform module contains a number of functions
that try to determine various properties of the platform you're
running on. There are functions for getting the architecture, CPU
type, the Windows OS version, and even the Linux distribution version.
(Contributed by Marc-André Lemburg.)
- The parser objects provided by the pyexpat module
can now optionally buffer character data, resulting in fewer calls to
your character data handler and therefore faster performance. Setting
the parser object's buffer_text attribute to True
will enable buffering.
- The sample(population, k) function was
added to the random module. population is a sequence or
xrange object containing the elements of a population, and
sample() chooses k elements from the population without
replacing chosen elements. k can be any value up to
`len( population )`. For example:
```text

>>> days = ['Mo', 'Tu', 'We', 'Th', 'Fr', 'St', 'Sn']
>>> random.sample(days, 3) # Choose 3 elements
['St', 'Sn', 'Th']
>>> random.sample(days, 7) # Choose 7 elements
['Tu', 'Th', 'Mo', 'We', 'St', 'Fr', 'Sn']
>>> random.sample(days, 7) # Choose 7 again
['We', 'Mo', 'Sn', 'Fr', 'Tu', 'St', 'Th']
>>> random.sample(days, 8) # Can't choose eight
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "random.py", line 414, in sample
raise ValueError, "sample larger than population"
ValueError: sample larger than population
>>> random.sample(xrange(1,10000,2), 10) # Choose ten odd nos. under 10000
[3407, 3805, 1505, 7023, 2401, 2267, 9733, 3151, 8083, 9195]
```
The random module now uses a new algorithm, the Mersenne
Twister, implemented in C. It's faster and more extensively studied
than the previous algorithm.
(All changes contributed by Raymond Hettinger.)
- The readline module also gained a number of new
functions: get_history_item(),
get_current_history_length(), and redisplay().
- The rexec and Bastion modules have been
declared dead, and attempts to import them will fail with a
RuntimeError. New-style classes provide new ways to break
out of the restricted execution environment provided by
rexec, and no one has interest in fixing them or time to do
so. If you have applications using rexec, rewrite them to
use something else.
(Sticking with Python 2.2 or 2.1 will not make your applications any
safer because there are known bugs in the rexec module in
those versions. To repeat: if you're using rexec, stop using
it immediately.)
- The rotor module has been deprecated because the
algorithm it uses for encryption is not believed to be secure. If
you need encryption, use one of the several AES Python modules
that are available separately.
- The shutil module gained a move(src,
dest) function that recursively moves a file or directory to a new
location.
- Support for more advanced POSIX signal handling was added
to the signal but then removed again as it proved impossible
to make it work reliably across platforms.
- The socket module now supports timeouts. You
can call the settimeout(t) method on a socket object to
set a timeout of t seconds. Subsequent socket operations that
take longer than t seconds to complete will abort and raise a
socket.timeout exception.
The original timeout implementation was by Tim O'Malley. Michael
Gilfix integrated it into the Python socket module and
shepherded it through a lengthy review. After the code was checked
in, Guido van Rossum rewrote parts of it. (This is a good example of
a collaborative development process in action.)
- On Windows, the socket module now ships with Secure
Sockets Layer (SSL) support.
- The value of the C PYTHON_API_VERSION macro is now
exposed at the Python level as `sys.api_version`. The current
exception can be cleared by calling the new sys.exc_clear()
function.
- The new tarfile module
allows reading from and writing to tar-format archive files.
(Contributed by Lars Gustäbel.)
- The new textwrap module contains functions for wrapping
strings containing paragraphs of text. The wrap(text,
width) function takes a string and returns a list containing
the text split into lines of no more than the chosen width. The
fill(text, width) function returns a single
string, reformatted to fit into lines no longer than the chosen width.
(As you can guess, fill() is built on top of
wrap(). For example:
```text

>>> import textwrap
>>> paragraph = "Not a whit, we defy augury: ... more text ..."
>>> textwrap.wrap(paragraph, 60)
["Not a whit, we defy augury: there's a special providence in",
"the fall of a sparrow. If it be now, 'tis not to come; if it",
...]
>>> print textwrap.fill(paragraph, 35)
Not a whit, we defy augury: there's
a special providence in the fall of
a sparrow. If it be now, 'tis not
to come; if it be not to come, it
will be now; if it be not now, yet
it will come: the readiness is all.
>>>
```
The module also contains a TextWrapper class that actually
implements the text wrapping strategy. Both the
TextWrapper class and the wrap() and
fill() functions support a number of additional keyword
arguments for fine-tuning the formatting; consult the module's
documentation (../lib/module-textwrap.html) for details.
(Contributed by Greg Ward.)
- The thread and threading modules now have
companion modules, dummy_thread and dummy_threading,
that provide a do-nothing implementation of the thread
module's interface for platforms where threads are not supported. The
intention is to simplify thread-aware modules (ones that don't
rely on threads to run) by putting the following code at the top:
```text

try:
import threading as _threading
except ImportError:
import dummy_threading as _threading
```
In this example, _threading is used as the module name to make
it clear that the module being used is not necessarily the actual
threading module. Code can call functions and use classes in
_threading whether or not threads are supported, avoiding an
if statement and making the code slightly clearer. This
module will not magically make multithreaded code run without threads;
code that waits for another thread to return or to do something will
simply hang forever.
- The time module's strptime() function has
long been an annoyance because it uses the platform C library's
strptime() implementation, and different platforms
sometimes have odd bugs. Brett Cannon contributed a portable
implementation that's written in pure Python and should behave
identically on all platforms.
- The new timeit module helps measure how long snippets
of Python code take to execute. The timeit.py file can be run
directly from the command line, or the module's Timer class
can be imported and used directly. Here's a short example that
figures out whether it's faster to convert an 8-bit string to Unicode
by appending an empty Unicode string to it or by using the
unicode() function:
```text

import timeit

timer1 = timeit.Timer('unicode("abc")')
timer2 = timeit.Timer('"abc" + u""')

# Run three trials
print timer1.repeat(repeat=3, number=100000)
print timer2.repeat(repeat=3, number=100000)

# On my laptop this outputs:
# [0.36831796169281006, 0.37441694736480713, 0.35304892063140869]
# [0.17574405670166016, 0.18193507194519043, 0.17565798759460449]
```
- The Tix module has received various bug fixes and
updates for the current version of the Tix package.
- The Tkinter module now works with a thread-enabled
version of Tcl. Tcl's threading model requires that widgets only be
accessed from the thread in which they're created; accesses from
another thread can cause Tcl to panic. For certain Tcl interfaces,
Tkinter will now automatically avoid this
when a widget is accessed from a different thread by marshalling a
command, passing it to the correct thread, and waiting for the
results. Other interfaces can't be handled automatically but
Tkinter will now raise an exception on such an access so that
you can at least find out about the problem. See
http://mail.python.org/pipermail/python-dev/2002-December/031107.html for a more detailed explanation of this change. (Implemented by
Martin von Löwis.)
- Calling Tcl methods through _tkinter no longer
returns only strings. Instead, if Tcl returns other objects those
objects are converted to their Python equivalent, if one exists, or
wrapped with a _tkinter.Tcl_Obj object if no Python equivalent
exists. This behavior can be controlled through the
wantobjects() method of tkapp objects.
When using _tkinter through the Tkinter module (as
most Tkinter applications will), this feature is always activated. It
should not cause compatibility problems, since Tkinter would always
convert string results to Python types where possible.
If any incompatibilities are found, the old behavior can be restored
by setting the wantobjects variable in the Tkinter
module to false before creating the first tkapp object.
```text

import Tkinter
Tkinter.wantobjects = 0
```
Any breakage caused by this change should be reported as a bug.
- The UserDict module has a new DictMixin class which
defines all dictionary methods for classes that already have a minimum
mapping interface. This greatly simplifies writing classes that need
to be substitutable for dictionaries, such as the classes in
the shelve module.
Adding the mix-in as a superclass provides the full dictionary
interface whenever the class defines __getitem__,
__setitem__, __delitem__, and keys.
For example:
```text

>>> import UserDict
>>> class SeqDict(UserDict.DictMixin):
... """Dictionary lookalike implemented with lists."""
... def __init__(self):
... self.keylist = []
... self.valuelist = []
... def __getitem__(self, key):
... try:
... i = self.keylist.index(key)
... except ValueError:
... raise KeyError
... return self.valuelist[i]
... def __setitem__(self, key, value):
... try:
... i = self.keylist.index(key)
... self.valuelist[i] = value
... except ValueError:
... self.keylist.append(key)
... self.valuelist.append(value)
... def __delitem__(self, key):
... try:
... i = self.keylist.index(key)
... except ValueError:
... raise KeyError
... self.keylist.pop(i)
... self.valuelist.pop(i)
... def keys(self):
... return list(self.keylist)
...
>>> s = SeqDict()
>>> dir(s) # See that other dictionary methods are implemented
['__cmp__', '__contains__', '__delitem__', '__doc__', '__getitem__',
'__init__', '__iter__', '__len__', '__module__', '__repr__',
'__setitem__', 'clear', 'get', 'has_key', 'items', 'iteritems',
'iterkeys', 'itervalues', 'keylist', 'keys', 'pop', 'popitem',
'setdefault', 'update', 'valuelist', 'values']
```
(Contributed by Raymond Hettinger.)
- The DOM implementation
in xml.dom.minidom can now generate XML output in a
particular encoding by providing an optional encoding argument to
the toxml() and toprettyxml() methods of DOM nodes.
- The xmlrpclib module now supports an XML-RPC extension
for handling nil data values such as Python's `None`. Nil values
are always supported on unmarshalling an XML-RPC response. To
generate requests containing `None`, you must supply a true value
for the allow_none parameter when creating a Marshaller
instance.
- The new DocXMLRPCServer module allows writing
self-documenting XML-RPC servers. Run it in demo mode (as a program)
to see it in action. Pointing the Web browser to the RPC server
produces pydoc-style documentation; pointing xmlrpclib to the
server allows invoking the actual methods.
(Contributed by Brian Quinlan.)
- Support for internationalized domain names (RFCs 3454, 3490,
3491, and 3492) has been added. The ``idna'' encoding can be used
to convert between a Unicode domain name and the ASCII-compatible
encoding (ACE) of that name.
```text

>>> u"www.Alliancefran¸ caise.nu".encode("idna")

'www.xn-alliancefranaise-npb.nu'
```
The socket module has also been extended to transparently
convert Unicode hostnames to the ACE version before passing them to
the C library. Modules that deal with hostnames such as
httplib and ftplib) also support Unicode host names;
httplib also sends HTTP "Host" headers using the ACE
version of the domain name. urllib supports Unicode URLs
with non-ASCII host names as long as the `path` part of the URL
is ASCII only.
To implement this change, the stringprep module, the
`mkstringprep` tool and the `punycode` encoding have been added.
## 17.1 Date/Time Type
Date and time types suitable for expressing timestamps were added as
the datetime module. The types don't support different
calendars or many fancy features, and just stick to the basics of
representing time.
The three primary types are: date, representing a day, month,
and year; time, consisting of hour, minute, and second; and
datetime, which contains all the attributes of both
date and time. There's also a
timedelta class representing differences between two points
in time, and time zone logic is implemented by classes inheriting from
the abstract tzinfo class.
You can create instances of date and time by either
supplying keyword arguments to the appropriate constructor,
e.g. `datetime.date(year=1972, month=10, day=15)`, or by using
one of a number of class methods. For example, the date.today()
class method returns the current local date.
Once created, instances of the date/time classes are all immutable.
There are a number of methods for producing formatted strings from
objects:
```text

>>> import datetime
>>> now = datetime.datetime.now()
>>> now.isoformat()
'2002-12-30T21:27:03.994956'
>>> now.ctime() # Only available on date, datetime
'Mon Dec 30 21:27:03 2002'
>>> now.strftime('%Y %d %b')
'2002 30 Dec'
```
The replace() method allows modifying one or more fields
of a date or datetime instance, returning a new instance:
```text

>>> d = datetime.datetime.now()
>>> d
datetime.datetime(2002, 12, 30, 22, 15, 38, 827738)
>>> d.replace(year=2001, hour = 12)
datetime.datetime(2001, 12, 30, 12, 15, 38, 827738)
>>>
```
Instances can be compared, hashed, and converted to strings (the
result is the same as that of isoformat()). date and
datetime instances can be subtracted from each other, and
added to timedelta instances. The largest missing feature is
that there's no standard library support for parsing strings and getting back a
date or datetime.
For more information, refer to the module's reference
documentation (../lib/module-datetime.html).
(Contributed by Tim Peters.)
## 17.2 The optparse Module
The getopt module provides simple parsing of command-line
arguments. The new optparse module (originally named Optik)
provides more elaborate command-line parsing that follows the Unix
conventions, automatically creates the output for --help,
and can perform different actions for different options.
You start by creating an instance of OptionParser and telling
it what your program's options are.
```text

import sys
from optparse import OptionParser

op = OptionParser()
op.add_option('-i', '--input',
action='store', type='string', dest='input',
help='set input filename')
op.add_option('-l', '--length',
action='store', type='int', dest='length',
help='set maximum length of output')
```
Parsing a command line is then done by calling the parse_args()
method.
```text

options, args = op.parse_args(sys.argv[1:])
print options
print args
```
This returns an object containing all of the option values,
and a list of strings containing the remaining arguments.
Invoking the script with the various arguments now works as you'd
expect it to. Note that the length argument is automatically
converted to an integer.
```text

$ ./python opt.py -i data arg1
<Values at 0x400cad4c: {'input': 'data', 'length': None}>
['arg1']
$ ./python opt.py --input=data --length=4
<Values at 0x400cad2c: {'input': 'data', 'length': 4}>
[]
$
```
The help message is automatically generated for you:
```text

$ ./python opt.py --help
usage: opt.py [options]

options:
-h, --help show this help message and exit
-iINPUT, --input=INPUT
set input filename
-lLENGTH, --length=LENGTH
set maximum length of output
$
```
See the module's documentation (../lib/module-optparse.html)
for more details.
Optik was written by Greg Ward, with suggestions from the readers of
the Getopt SIG.


1 PEP 218: A Standard Set Datatype

contents.html | whatsnew23.html | section-generators.html | What's New in Python 2.3 | contents.html
Previous:
Up:
What's New in Python (whatsnew23.html)
Next:
2 PEP 255: Simple (section-generators.html)
---
# 1 PEP 218: A Standard Set Datatype
The new sets module contains an implementation of a set
datatype. The Set class is for mutable sets, sets that can
have members added and removed. The ImmutableSet class is for
sets that can't be modified, and instances of ImmutableSet can
therefore be used as dictionary keys. Sets are built on top of
dictionaries, so the elements within a set must be hashable.
Here's a simple example:
```text

>>> import sets
>>> S = sets.Set([1,2,3])
>>> S
Set([1, 2, 3])
>>> 1 in S
True
>>> 0 in S
False
>>> S.add(5)
>>> S.remove(3)
>>> S
Set([1, 2, 5])
>>>
```
The union and intersection of sets can be computed with the
union() and intersection() methods; an alternative
notation uses the bitwise operators `&` and `|`.
Mutable sets also have in-place versions of these methods,
union_update() and intersection_update().
```text

>>> S1 = sets.Set([1,2,3])
>>> S2 = sets.Set([4,5,6])
>>> S1.union(S2)
Set([1, 2, 3, 4, 5, 6])
>>> S1 | S2 # Alternative notation
Set([1, 2, 3, 4, 5, 6])
>>> S1.intersection(S2)
Set([])
>>> S1 & S2 # Alternative notation
Set([])
>>> S1.union_update(S2)
>>> S1
Set([1, 2, 3, 4, 5, 6])
>>>
```
It's also possible to take the symmetric difference of two sets. This
is the set of all elements in the union that aren't in the
intersection. Another way of putting it is that the symmetric
difference contains all elements that are in exactly one
set. Again, there's an alternative notation (`^`), and an
in-place version with the ungainly name
symmetric_difference_update().
```text

>>> S1 = sets.Set([1,2,3,4])
>>> S2 = sets.Set([3,4,5,6])
>>> S1.symmetric_difference(S2)
Set([1, 2, 5, 6])
>>> S1 ^ S2
Set([1, 2, 5, 6])
>>>
```
There are also issubset() and issuperset() methods
for checking whether one set is a subset or superset of another:
```text

>>> S1 = sets.Set([1,2,3])
>>> S2 = sets.Set([2,3])
>>> S2.issubset(S1)
True
>>> S1.issubset(S2)
False
>>> S1.issuperset(S2)
True
>>>
```
See Also:


19 Build and C API Changes

section-pymalloc.html | whatsnew23.html | section-other.html | What's New in Python 2.3 | contents.html
Previous:
18 Pymalloc: A Specialized (section-pymalloc.html)
Up:
What's New in Python (whatsnew23.html)
Next:
20 Other Changes and (section-other.html)
---
- 19.1 Port-Specific Changes (node20.html#SECTION0002010000000000000000)
---
# 19 Build and C API Changes
Changes to Python's build process and to the C API include:
- The C-level interface to the garbage collector has been changed
to make it easier to write extension types that support garbage
collection and to debug misuses of the functions.
Various functions have slightly different semantics, so a bunch of
functions had to be renamed. Extensions that use the old API will
still compile but will not participate in garbage collection,
so updating them for 2.3 should be considered fairly high priority.
To upgrade an extension module to the new API, perform the following
steps:
- Rename Py_TPFLAGS_GC to PyTPFLAGS_HAVE_GC.
- Use PyObject_GC_New or PyObject_GC_NewVar to
allocate objects, and PyObject_GC_Del to deallocate them.
- Rename PyObject_GC_Init to PyObject_GC_Track and
PyObject_GC_Fini to PyObject_GC_UnTrack.
- Remove PyGC_HEAD_SIZE from object size calculations.
- Remove calls to PyObject_AS_GC and PyObject_FROM_GC.
- The cycle detection implementation used by the garbage collection
has proven to be stable, so it's now been made mandatory. You can no
longer compile Python without it, and the
--with-cycle-gc switch to configure has been removed.
- Python can now optionally be built as a shared library
(libpython2.3.so) by supplying --enable-shared
when running Python's configure script. (Contributed by Ondrej
Palkovsky.)
- The DL_EXPORT and DL_IMPORT macros
are now deprecated. Initialization functions for Python extension
modules should now be declared using the new macro
PyMODINIT_FUNC, while the Python core will generally
use the PyAPI_FUNC and PyAPI_DATA
macros.
- The interpreter can be compiled without any docstrings for
the built-in functions and modules by supplying
--without-doc-strings to the configure script.
This makes the Python executable about 10% smaller, but will also
mean that you can't get help for Python's built-ins. (Contributed by
Gustavo Niemeyer.)
- The PyArg_NoArgs() macro is now deprecated, and code
that uses it should be changed. For Python 2.2 and later, the method
definition table can specify the
METH_NOARGS flag, signalling that there are no arguments, and
the argument checking can then be removed. If compatibility with
pre-2.2 versions of Python is important, the code could use
`PyArg_ParseTuple( args , "")` instead, but this will be slower
than using METH_NOARGS.
- A new function, PyObject_DelItemString(mapping,
char *key) was added as shorthand for
`PyObject_DelItem( mapping , PyString_New( key ))`.
- File objects now manage their internal string buffer
differently, increasing it exponentially when needed. This results in
the benchmark tests in Lib/test/test_bufio.py speeding up
considerably (from 57 seconds to 1.7 seconds, according to one
measurement).
- It's now possible to define class and static methods for a C
extension type by setting either the METH_CLASS or
METH_STATIC flags in a method's PyMethodDef
structure.
- Python now includes a copy of the Expat XML parser's source code,
removing any dependence on a system version or local installation of
Expat.
- If you dynamically allocate type objects in your extension, you
should be aware of a change in the rules relating to the
__module__ and __name__ attributes. In summary,
you will want to ensure the type's dictionary contains a
`'__module__'` key; making the module name the part of the type
name leading up to the final period will no longer have the desired
effect. For more detail, read the API reference documentation or the
source.
## 19.1 Port-Specific Changes
Support for a port to IBM's OS/2 using the EMX runtime environment was
merged into the main Python source tree. EMX is a POSIX emulation
layer over the OS/2 system APIs. The Python port for EMX tries to
support all the POSIX-like capability exposed by the EMX runtime, and
mostly succeeds; fork() and fcntl() are
restricted by the limitations of the underlying emulation layer. The
standard OS/2 port, which uses IBM's Visual Age compiler, also gained
support for case-sensitive import semantics as part of the integration
of the EMX port into CVS. (Contributed by Andrew MacIntyre.)
On MacOS, most toolbox modules have been weaklinked to improve
backward compatibility. This means that modules will no longer fail
to load if a single routine is missing on the curent OS version.
Instead calling the missing routine will raise an exception.
(Contributed by Jack Jansen.)
The RPM spec files, found in the Misc/RPM/ directory in the
Python source distribution, were updated for 2.3. (Contributed by
Sean Reifschneider.)
Other new platforms now supported by Python include AtheOS
(http://www.atheos.cx/), GNU/Hurd, and OpenVMS.


21 Porting to Python 2.3

section-other.html | whatsnew23.html | acks.html | What's New in Python 2.3 | contents.html
Previous:
20 Other Changes and (section-other.html)
Up:
What's New in Python (whatsnew23.html)
Next:
22 Acknowledgements (acks.html)
---
# 21 Porting to Python 2.3
This section lists previously described changes that may require
changes to your code:
- yield is now always a keyword; if it's used as a
variable name in your code, a different name must be chosen.
- For strings X and Y, `X in Y` now works
if X is more than one character long.
- The int() type constructor will now return a long
integer instead of raising an OverflowError when a string
or floating-point number is too large to fit into an integer.
- If you have Unicode strings that contain 8-bit characters, you
must declare the file's encoding (UTF-8, Latin-1, or whatever) by
adding a comment to the top of the file. See
section 3 (section-encodings.html#section-encodings) for more information.
- Calling Tcl methods through _tkinter no longer
returns only strings. Instead, if Tcl returns other objects those
objects are converted to their Python equivalent, if one exists, or
wrapped with a _tkinter.Tcl_Obj object if no Python equivalent
exists.
- Large octal and hex literals such as
`0xffffffff` now trigger a FutureWarning. Currently
they're stored as 32-bit numbers and result in a negative value, but
in Python 2.4 they'll become positive long integers.
There are a few ways to fix this warning. If you really need a
positive number, just add an "L" to the end of the literal. If
you're trying to get a 32-bit integer with low bits set and have
previously used an expression such as `(1 « 31)`, it's probably
clearest to start with all bits set and clear the desired upper bits.
For example, to clear just the top bit (bit 31), you could write
`0xffffffffL &~(1L«31)`.
- You can no longer disable assertions by assigning to `__debug__`.
- The Distutils setup() function has gained various new
keyword arguments such as depends. Old versions of the
Distutils will abort if passed unknown keywords. A solution is to check
for the presence of the new get_distutil_options() function
in your setup.py and only uses the new keywords
with a version of the Distutils that supports them:
```text

from distutils import core

kw = {'sources': 'foo.c', ...}
if hasattr(core, 'get_distutil_options'):
kw['depends'] = ['foo.h']
ext = Extension(**kw)
```
- Using `None` as a variable name will now result in a
SyntaxWarning warning.
- Names of extension types defined by the modules included with
Python now contain the module and a "." in front of the type
name.


section-encodings.html | whatsnew23.html | node6.html | What's New in Python 2.3 | contents.html
Previous:
3 PEP 263: Source (section-encodings.html)
Up:
What's New in Python (whatsnew23.html)
Next:
5 PEP 277: Unicode (node6.html)
---
# 4 PEP 273: Importing Modules from Zip Archives
The new zipimport module adds support for importing
modules from a ZIP-format archive. You don't need to import the
module explicitly; it will be automatically imported if a ZIP
archive's filename is added to `sys.path`. For example:
```text

amk@nyman:~/src/python$ unzip -l /tmp/example.zip
Archive: /tmp/example.zip
Length Date Time Name
-------- ---- ---- ----
8467 11-26-02 22:30 jwzthreading.py
-------- -------
8467 1 file
amk@nyman:~/src/python$ ./python
Python 2.3 (#1, Aug 1 2003, 19:54:32)
>>> import sys
>>> sys.path.insert(0, '/tmp/example.zip') # Add .zip file to front of path
>>> import jwzthreading
>>> jwzthreading.__file__
'/tmp/example.zip/jwzthreading.py'
>>>
```
An entry in `sys.path` can now be the filename of a ZIP archive.
The ZIP archive can contain any kind of files, but only files named
*.py, *.pyc, or *.pyo can be imported. If an
archive only contains *.py files, Python will not attempt to
modify the archive by adding the corresponding *.pyc file, meaning
that if a ZIP archive doesn't contain *.pyc files, importing may be
rather slow.
A path within the archive can also be specified to only import from a
subdirectory; for example, the path /tmp/example.zip/lib/
would only import from the lib/ subdirectory within the
archive.
See Also:


node5.html | whatsnew23.html | node7.html | What's New in Python 2.3 | contents.html
Previous:
4 PEP 273: Importing (node5.html)
Up:
What's New in Python (whatsnew23.html)
Next:
6 PEP 278: Universal (node7.html)
---
# 5 PEP 277: Unicode file name support for Windows NT
On Windows NT, 2000, and XP, the system stores file names as Unicode
strings. Traditionally, Python has represented file names as byte
strings, which is inadequate because it renders some file names
inaccessible.
Python now allows using arbitrary Unicode strings (within the
limitations of the file system) for all functions that expect file
names, most notably the open() built-in function. If a Unicode
string is passed to os.listdir(), Python now returns a list
of Unicode strings. A new function, os.getcwdu(), returns
the current directory as a Unicode string.
Byte strings still work as file names, and on Windows Python will
transparently convert them to Unicode using the `mbcs` encoding.
Other systems also allow Unicode strings as file names but convert
them to byte strings before passing them to the system, which can
cause a UnicodeError to be raised. Applications can test
whether arbitrary Unicode strings are supported as file names by
checking os.path.supports_unicode_filenames, a Boolean value.
Under MacOS, os.listdir() may now return Unicode filenames.
See Also:


6 PEP 278: Universal Newline Support

node6.html | whatsnew23.html | section-enumerate.html | What's New in Python 2.3 | contents.html
Previous:
5 PEP 277: Unicode (node6.html)
Up:
What's New in Python (whatsnew23.html)
Next:
7 PEP 279: enumerate() (section-enumerate.html)
---
# 6 PEP 278: Universal Newline Support
The three major operating systems used today are Microsoft Windows,
Apple's Macintosh OS, and the various Unix derivatives. A minor
irritation of cross-platform work
is that these three platforms all use different characters
to mark the ends of lines in text files. Unix uses the linefeed
(ASCII character 10), MacOS uses the carriage return (ASCII
character 13), and Windows uses a two-character sequence of a
carriage return plus a newline.
Python's file objects can now support end of line conventions other
than the one followed by the platform on which Python is running.
Opening a file with the mode `'U'` or `'rU'` will open a file
for reading in universal newline mode. All three line ending
conventions will be translated to a "\n" in the strings
returned by the various file methods such as read() and
readline().
Universal newline support is also used when importing modules and when
executing a file with the execfile() function. This means
that Python modules can be shared between all three operating systems
without needing to convert the line-endings.
This feature can be disabled when compiling Python by specifying
the --without-universal-newlines switch when running Python's
configure script.
See Also:


8 PEP 282: The logging Package

section-enumerate.html | whatsnew23.html | section-bool.html | What's New in Python 2.3 | contents.html
Previous:
7 PEP 279: enumerate() (section-enumerate.html)
Up:
What's New in Python (whatsnew23.html)
Next:
9 PEP 285: A (section-bool.html)
---
# 8 PEP 282: The logging Package
A standard package for writing logs, logging, has been added
to Python 2.3. It provides a powerful and flexible mechanism for
generating logging output which can then be filtered and processed in
various ways. A configuration file written in a standard format can
be used to control the logging behavior of a program. Python
includes handlers that will write log records to
standard error or to a file or socket, send them to the system log, or
even e-mail them to a particular address; of course, it's also
possible to write your own handler classes.
The Logger class is the primary class.
Most application code will deal with one or more Logger
objects, each one used by a particular subsystem of the application.
Each Logger is identified by a name, and names are organized
into a hierarchy using "." as the component separator. For
example, you might have Logger instances named "server",
"server.auth" and "server.network". The latter two
instances are below "server" in the hierarchy. This means that
if you turn up the verbosity for "server" or direct "server"messages to a different handler, the changes will also apply to
records logged to "server.auth" and "server.network".
There's also a root Logger that's the parent of all other
loggers.
For simple uses, the logging package contains some
convenience functions that always use the root log:
```text

import logging

logging.debug('Debugging information')
logging.info('Informational message')
logging.warning('Warning:config file %s not found', 'server.conf')
logging.error('Error occurred')
logging.critical('Critical error -- shutting down')
```
This produces the following output:
```text

WARNING:root:Warning:config file server.conf not found
ERROR:root:Error occurred
CRITICAL:root:Critical error -- shutting down
```
In the default configuration, informational and debugging messages are
suppressed and the output is sent to standard error. You can enable
the display of informational and debugging messages by calling the
setLevel() method on the root logger.
Notice the warning() call's use of string formatting
operators; all of the functions for logging messages take the
arguments `( msg , arg1 , arg2 , ...)` and log the
string resulting from `msg % ( arg1 , arg2 ,
...)`.
There's also an exception() function that records the most
recent traceback. Any of the other functions will also record the
traceback if you specify a true value for the keyword argument
exc_info.
```text

def f():
try: 1/0
except: logging.exception('Problem recorded')

f()
```
This produces the following output:
```text

ERROR:root:Problem recorded
Traceback (most recent call last):
File "t.py", line 6, in f
1/0
ZeroDivisionError: integer division or modulo by zero
```
Slightly more advanced programs will use a logger other than the root
logger. The getLogger(name) function is used to get
a particular log, creating it if it doesn't exist yet.
getLogger(None) returns the root logger.
```text

log = logging.getLogger('server')
...
log.info('Listening on port %i', port)
...
log.critical('Disk full')
...
```
Log records are usually propagated up the hierarchy, so a message
logged to "server.auth" is also seen by "server" and
"root", but a Logger can prevent this by setting its
propagate attribute to False.
There are more classes provided by the logging package that
can be customized. When a Logger instance is told to log a
message, it creates a LogRecord instance that is sent to any
number of different Handler instances. Loggers and handlers
can also have an attached list of filters, and each filter can cause
the LogRecord to be ignored or can modify the record before
passing it along. When they're finally output, LogRecord
instances are converted to text by a Formatter class. All of
these classes can be replaced by your own specially-written classes.
With all of these features the logging package should provide
enough flexibility for even the most complicated applications. This
is only an incomplete overview of its features, so please see the
package's reference documentation (../lib/module-logging.html)
for all of the details. Reading PEP 282 (http://www.python.org/peps/pep-0282.html) will also be helpful.
See Also:


9 PEP 285: A Boolean Type

node9.html | whatsnew23.html | node11.html | What's New in Python 2.3 | contents.html
Previous:
8 PEP 282: The (node9.html)
Up:
What's New in Python (whatsnew23.html)
Next:
10 PEP 293: Codec (node11.html)
---
# 9 PEP 285: A Boolean Type
A Boolean type was added to Python 2.3. Two new constants were added
to the __builtin__ module, True and
False. (True and
False constants were added to the built-ins
in Python 2.2.1, but the 2.2.1 versions are simply set to integer values of
1 and 0 and aren't a different type.)
The type object for this new type is named
bool; the constructor for it takes any Python value and
converts it to True or False.
```text

>>> bool(1)
True
>>> bool(0)
False
>>> bool([])
False
>>> bool( (1,) )
True
```
Most of the standard library modules and built-in functions have been
changed to return Booleans.
```text

>>> obj = []
>>> hasattr(obj, 'append')
True
>>> isinstance(obj, list)
True
>>> isinstance(obj, tuple)
False
```
Python's Booleans were added with the primary goal of making code
clearer. For example, if you're reading a function and encounter the
statement `return 1`, you might wonder whether the `1`
represents a Boolean truth value, an index, or a
coefficient that multiplies some other quantity. If the statement is
`return True`, however, the meaning of the return value is quite
clear.
Python's Booleans were not added for the sake of strict
type-checking. A very strict language such as Pascal would also
prevent you performing arithmetic with Booleans, and would require
that the expression in an if statement always evaluate to a
Boolean result. Python is not this strict and never will be, as
PEP 285 (http://www.python.org/peps/pep-0285.html) explicitly says. This means you can still use any
expression in an if statement, even ones that evaluate to a
list or tuple or some random object. The Boolean type is a
subclass of the int class so that arithmetic using a Boolean
still works.
```text

>>> True + 1
2
>>> False + 1
1
>>> False * 75
0
>>> True * 75
75
```
To sum up True and False in a sentence: they're
alternative ways to spell the integer values 1 and 0, with the single
difference that str() and repr() return the
strings `'True'` and `'False'` instead of `'1'` and
`'0'`.
See Also:


3 PEP 263: Source Code Encodings

section-generators.html | whatsnew23.html | node5.html | What's New in Python 2.3 | contents.html
Previous:
2 PEP 255: Simple (section-generators.html)
Up:
What's New in Python (whatsnew23.html)
Next:
4 PEP 273: Importing (node5.html)
---
# 3 PEP 263: Source Code Encodings
Python source files can now be declared as being in different
character set encodings. Encodings are declared by including a
specially formatted comment in the first or second line of the source
file. For example, a UTF-8 file can be declared with:
```text

#!/usr/bin/env python
# -*- coding: UTF-8 -*-
```
Without such an encoding declaration, the default encoding used is
7-bit ASCII. Executing or importing modules that contain string
literals with 8-bit characters and have no encoding declaration will result
in a DeprecationWarning being signalled by Python 2.3; in
2.4 this will be a syntax error.
The encoding declaration only affects Unicode string literals, which
will be converted to Unicode using the specified encoding. Note that
Python identifiers are still restricted to ASCII characters, so you
can't have variable names that use characters outside of the usual
alphanumerics.
See Also: