Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 342 – Coroutines via Enhanced Generators

Author:
Guido van Rossum, Phillip J. Eby
Status:
Final
Type:
Standards Track
Created:
10-May-2005
Python-Version:
2.5
Post-History:

Table of Contents

Introduction

This PEP proposes some enhancements to the API and syntax of generators, to make them usable as simple coroutines. It is basically a combination of ideas from these two PEPs, which may be considered redundant if this PEP is accepted:

  • PEP 288, Generators Attributes and Exceptions. The current PEP covers its second half, generator exceptions (in fact the throw() method name was taken from PEP 288). PEP 342 replaces generator attributes, however, with a concept from an earlier revision of PEP 288, the yield expression.
  • PEP 325, Resource-Release Support for Generators. PEP 342 ties up a few loose ends in the PEP 325 spec, to make it suitable for actual implementation.

Motivation

Coroutines are a natural way of expressing many algorithms, such as simulations, games, asynchronous I/O, and other forms of event-driven programming or co-operative multitasking. Python’s generator functions are almost coroutines – but not quite – in that they allow pausing execution to produce a value, but do not provide for values or exceptions to be passed in when execution resumes. They also do not allow execution to be paused within the try portion of try/finally blocks, and therefore make it difficult for an aborted coroutine to clean up after itself.

Also, generators cannot yield control while other functions are executing, unless those functions are themselves expressed as generators, and the outer generator is written to yield in response to values yielded by the inner generator. This complicates the implementation of even relatively simple use cases like asynchronous communications, because calling any functions either requires the generator to block (i.e. be unable to yield control), or else a lot of boilerplate looping code must be added around every needed function call.

However, if it were possible to pass values or exceptions into a generator at the point where it was suspended, a simple co-routine scheduler or trampoline function would let coroutines call each other without blocking – a tremendous boon for asynchronous applications. Such applications could then write co-routines to do non-blocking socket I/O by yielding control to an I/O scheduler until data has been sent or becomes available. Meanwhile, code that performs the I/O would simply do something like this:

data = (yield nonblocking_read(my_socket, nbytes))

in order to pause execution until the nonblocking_read() coroutine produced a value.

In other words, with a few relatively minor enhancements to the language and to the implementation of the generator-iterator type, Python will be able to support performing asynchronous operations without needing to write the entire application as a series of callbacks, and without requiring the use of resource-intensive threads for programs that need hundreds or even thousands of co-operatively multitasking pseudothreads. Thus, these enhancements will give standard Python many of the benefits of the Stackless Python fork, without requiring any significant modification to the CPython core or its APIs. In addition, these enhancements should be readily implementable by any Python implementation (such as Jython) that already supports generators.

Specification Summary

By adding a few simple methods to the generator-iterator type, and with two minor syntax adjustments, Python developers will be able to use generator functions to implement co-routines and other forms of co-operative multitasking. These methods and adjustments are:

  1. Redefine yield to be an expression, rather than a statement. The current yield statement would become a yield expression whose value is thrown away. A yield expression’s value is None whenever the generator is resumed by a normal next() call.
  2. Add a new send() method for generator-iterators, which resumes the generator and sends a value that becomes the result of the current yield-expression. The send() method returns the next value yielded by the generator, or raises StopIteration if the generator exits without yielding another value.
  3. Add a new throw() method for generator-iterators, which raises an exception at the point where the generator was paused, and which returns the next value yielded by the generator, raising StopIteration if the generator exits without yielding another value. (If the generator does not catch the passed-in exception, or raises a different exception, then that exception propagates to the caller.)
  4. Add a close() method for generator-iterators, which raises GeneratorExit at the point where the generator was paused. If the generator then raises StopIteration (by exiting normally, or due to already being closed) or GeneratorExit (by not catching the exception), close() returns to its caller. If the generator yields a value, a RuntimeError is raised. If the generator raises any other exception, it is propagated to the caller. close() does nothing if the generator has already exited due to an exception or normal exit.
  5. Add support to ensure that close() is called when a generator iterator is garbage-collected.
  6. Allow yield to be used in try/finally blocks, since garbage collection or an explicit close() call would now allow the finally clause to execute.

A prototype patch implementing all of these changes against the current Python CVS HEAD is available as SourceForge patch #1223381 (https://bugs.python.org/issue1223381).

Specification: Sending Values into Generators

New generator method: send(value)

A new method for generator-iterators is proposed, called send(). It takes exactly one argument, which is the value that should be sent in to the generator. Calling send(None) is exactly equivalent to calling a generator’s next() method. Calling send() with any other value is the same, except that the value produced by the generator’s current yield expression will be different.

Because generator-iterators begin execution at the top of the generator’s function body, there is no yield expression to receive a value when the generator has just been created. Therefore, calling send() with a non-None argument is prohibited when the generator iterator has just started, and a TypeError is raised if this occurs (presumably due to a logic error of some kind). Thus, before you can communicate with a coroutine you must first call next() or send(None) to advance its execution to the first yield expression.

As with the next() method, the send() method returns the next value yielded by the generator-iterator, or raises StopIteration if the generator exits normally, or has already exited. If the generator raises an uncaught exception, it is propagated to send()’s caller.

New syntax: Yield Expressions

The yield-statement will be allowed to be used on the right-hand side of an assignment; in that case it is referred to as yield-expression. The value of this yield-expression is None unless send() was called with a non-None argument; see below.

A yield-expression must always be parenthesized except when it occurs at the top-level expression on the right-hand side of an assignment. So

x = yield 42
x = yield
x = 12 + (yield 42)
x = 12 + (yield)
foo(yield 42)
foo(yield)

are all legal, but

x = 12 + yield 42
x = 12 + yield
foo(yield 42, 12)
foo(yield, 12)

are all illegal. (Some of the edge cases are motivated by the current legality of yield 12, 42.)

Note that a yield-statement or yield-expression without an expression is now legal. This makes sense: when the information flow in the next() call is reversed, it should be possible to yield without passing an explicit value (yield is of course equivalent to yield None).

When send(value) is called, the yield-expression that it resumes will return the passed-in value. When next() is called, the resumed yield-expression will return None. If the yield-expression is a yield-statement, this returned value is ignored, similar to ignoring the value returned by a function call used as a statement.

In effect, a yield-expression is like an inverted function call; the argument to yield is in fact returned (yielded) from the currently executing function, and the return value of yield is the argument passed in via send().

Note: the syntactic extensions to yield make its use very similar to that in Ruby. This is intentional. Do note that in Python the block passes a value to the generator using send(EXPR) rather than return EXPR, and the underlying mechanism whereby control is passed between the generator and the block is completely different. Blocks in Python are not compiled into thunks; rather, yield suspends execution of the generator’s frame. Some edge cases work differently; in Python, you cannot save the block for later use, and you cannot test whether there is a block or not. (XXX - this stuff about blocks seems out of place now, perhaps Guido can edit to clarify.)

Specification: Exceptions and Cleanup