Skip to main content

Python Iterators and Generators Tutorial

Explore the difference between Python Iterators and Generators and learn which are the best to use in various situations.
· 10 min read

Explore with AI

Iterators are objects that can be iterated upon. They serve as a common feature of the Python programming language, neatly tucked away for looping and list comprehensions. Any object that can derive an iterator is known as an iterable. 

There is a lot of work that goes into constructing an iterator. For instance, the implementation of each iterator object must consist of an __iter__() and __next__() method. In addition to the prerequisite above, the implementation must also have a way to track the object's internal state and raise a StopIteration exception once no more values can be returned. These rules are known as the iterator protocol. 

Implementing your own iterator is a drawn-out process, and it is only sometimes necessary. A simpler alternative is to use a generator object. Generators are a special type of function that use the yield keyword to return an iterator that may be iterated over, one value at a time. 

The ability to discern the appropriate scenarios to implement an iterator or use a generator will improve your skills as a Python programmer. In the remainder of this tutorial, we will emphasize the distinctions between the two objects, which will help you decide the best one to use for various situations. 

Glossary

Term

Definition

Iterable 

A Python object which can be looped over or iterated over in a loop. Examples of iterables include lists, sets, tuples, dictionaries, strings, etc. 

Iterator

An iterator is an object that can be iterated upon. Thus, iterators contain a countable number of values. 

Generator

A special type of function which does not return a single value: it returns an iterator object with a sequence of values.

Lazy Evaluation 

An evaluation strategy whereby certain objects are only produced when required. Consequently, certain developer circles also refer to lazy evaluation as “call-by-need.”

Iterator Protocol 

A set of rules that must be followed to define an iterator in Python. 

next()

A built-in function used to return the next item in an iterator. 

iter()

A built-in function used to convert an iterable to an iterator. 

yield()

A python keyword similar to the return keyword, except yield returns a generator object instead of a value. 

Python Iterators & Iterables

Iterables are objects capable of returning their members one at a time – they can be iterated over. Popular built-in Python data structures such as lists, tuples, and sets qualify as iterables. Other data structures like strings and dictionaries are also considered iterables: a string can produce iteration of its characters, and the keys of a dictionary can be iterated upon. As a rule of thumb, consider any object that can be iterated over in a for-loop as an iterable. 

Exploring Python iterables with examples

Given the definitions, we may conclude that all iterators are also iterable. However, every iterable is not necessarily an iterator. An iterable produces an iterator only once it is iterated on.

To demonstrate this functionality, we will instantiate a list, which is an iterable, and produce an iterator by calling the iter() built-in function on the list. 

list_instance = [1, 2, 3, 4]
print(iter(list_instance))

"""
<list_iterator object at 0x7fd946309e90>
"""

Although the list by itself is not an iterator, calling the iter() function converts it to an iterator and returns the iterator object.

To demonstrate that not all iterables are iterators, we will instantiate the same list object and attempt to call the next() function, which is used to return the next item in an iterator.  

list_instance = [1, 2, 3, 4]
print(next(list_instance))
"""
--------------------------------------------------------------------
TypeError                         Traceback (most recent call last)
<ipython-input-2-0cb076ed2d65> in <module>()
    3 print(iter(list_instance))
    4
----> 5 print(next(list_instance))
TypeError: 'list' object is not an iterator
"""

In the code above, you can see that attempting to call the next() function on the list raised a TypeError – learn more about Exception and Error Handling in Python. This behavior occurred for the simple fact that a list object is an iterable and not an iterator. 

Exploring Python iterators with examples

Thus, if the goal is to iterate on a list, then an iterator object must first be produced. Only then can we manage the iteration through the values of the list.

# instantiate a list object
list_instance = [1, 2, 3, 4]

# convert the list to an iterator
iterator = iter(list_instance)

# return items one at a time
print(next(iterator))
print(next(iterator))
print(next(iterator))
print(next(iterator))
"""
1
2
3
4
"""

Python automatically produces an iterator object whenever you attempt to loop through an iterable object. 

# instantiate a list object
list_instance = [1, 2, 3, 4]

# loop through the list
for item in list_instance:
  print(item)
"""
1
2
3
4
"""

When the StopIteration exception is caught, then the loop ends.

The values obtained from an iterator can only be retrieved from left to right. Python does not have a previous() function to enable developers to move backward through an iterator. 

The lazy nature of iterators

It is possible to define multiple iterators based on the same iterable object. Each iterator will maintain its own state of progress. Thus, by defining multiple iterator instances of an iterable object, it is possible to iterate to the end of one instance while the other instance remains at the beginning.

list_instance = [1, 2, 3, 4]
iterator_a = iter(list_instance)
iterator_b = iter(list_instance)
print(f"A: {next(iterator_a)}")
print(f"A: {next(iterator_a)}")
print(f"A: {next(iterator_a)}")
print(f"A: {next(iterator_a)}")
print(f"B: {next(iterator_b)}")
"""
A: 1
A: 2
A: 3
A: 4
B: 1
"""

Notice iterator_b prints the first element of the series.

Thus, we can say iterators have a lazy nature: when an iterator is created, the elements are not yielded until they are requested. In other words, the elements of our list instance would only be returned once we explicitly ask them to be with next(iter(list_instance)). 

However, all of the values from an iterator may be extracted at once by calling a built-in iterable data structure container (i.e., list(), set(), tuple()) on the iterator object to force the iterator to generate all its elements at once.