Using the Python defaultdict Type for Handling Missing Keys

Using the Python defaultdict Type for Handling Missing Keys

by Leodanis Pozo Ramos basics data-structures python
Share Share Email
Remove ads

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Handling Missing Keys With the Python defaultdict Type

A common problem that you can face when working with Python dictionaries is to try to access or modify keys that don’t exist in the dictionary. This will raise a KeyError and break up your code execution. To handle these kinds of situations, the standard library provides the Python defaultdict type, a dictionary-like class that’s available for you in collections.

The Python defaultdict type behaves almost exactly like a regular Python dictionary, but if you try to access or modify a missing key, then defaultdict will automatically create the key and generate a default value for it. This makes defaultdict a valuable option for handling missing keys in dictionaries.

In this tutorial, you’ll learn:

  • How to use the Python defaultdict type for handling missing keys in a dictionary
  • When and why to use a Python defaultdict rather than a regular dict
  • How to use a defaultdict for grouping, counting, and accumulating operations

With this knowledge under your belt, you’ll be in a better condition to effectively use the Python defaultdict type in your day-to-day programming challenges.

To get the most out of this tutorial, you should have some previous understanding of what Python dictionaries are and how to work with them. If you need to freshen up, then check out the following resources:

Handling Missing Keys in Dictionaries

A common issue that you can face when working with Python dictionaries is how to handle missing keys. If your code is heavily based on dictionaries, or if you’re creating dictionaries on the fly all the time, then you’ll soon notice that dealing with frequent KeyError exceptions can be quite annoying and can add extra complexity to your code. With Python dictionaries, you have at least four available ways to handle missing keys:

  1. Use .setdefault()
  2. Use .get()
  3. Use the key in dict idiom
  4. Use a try and except block

The Python docs explain .setdefault() and .get() as follows:

setdefault(key[, default])

If key is in the dictionary, return its value. If not, insert key with a value of default and return default. default defaults to None.

get(key[, default])

Return the value for key if key is in the dictionary, else default. If default is not given, it defaults to None, so that this method never raises a KeyError.

(Source)

Here’s an example of how you can use .setdefault() to handle missing keys in a dictionary:

Python 
>>> a_dict = {}
>>> a_dict['missing_key']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    a_dict['missing_key']
KeyError: 'missing_key'
>>> a_dict.setdefault('missing_key', 'default value')
'default value'
>>> a_dict['missing_key']
'default value'
>>> a_dict.setdefault('missing_key', 'another default value')
'default value'
>>> a_dict
{'missing_key': 'default value'}

In the above code, you use .setdefault() to generate a default value for missing_key. Notice that your dictionary, a_dict, now has a new key called missing_key whose value is 'default value'. This key didn’t exist before you called .setdefault(). Finally, if you call .setdefault() on an existing key, then the call won’t have any effect on the dictionary. Your key will hold the original value instead of the new default value.

On the other hand, if you use .get(), then you can code something like this:

Python 
>>> a_dict = {}
>>> a_dict.get('missing_key', 'default value')
'default value'
>>> a_dict
{}

Here, you use .get() to generate a default value for missing_key, but this time, your dictionary stays empty. This is because .get() returns the default value, but this value isn’t added to the underlying dictionary. For example, if you have a dictionary called D, then you can assume that .get() works something like this:

Text 
D.get(key, default) -> D[key] if key in D, else default

With this pseudo-code, you can understand how .get() works internally. If the key exists, then .get() returns the value mapped to that key. Otherwise, the default value is returned. Your code never creates or assigns a value to key. In this example, default defaults to None.

You can also use conditional statements to handle missing keys in dictionaries. Take a look at the following example, which uses the key in dict idiom:

Python 
>>> a_dict = {}
>>> if 'key' in a_dict:
...     # Do something with 'key'...
...     a_dict['key']
... else:
...     a_dict['key'] = 'default value'
...
>>> a_dict
{'key': 'default value'}

In this code, you use an if statement along with the in operator to check if key is present in a_dict. If so, then you can perform any action with key or with its value. Otherwise, you create the new key, key, and assign it a 'default value'. Note that the above code works similar to .setdefault() but takes four lines of code, while .setdefault() would only take one line (in addition to being more readable).

You can also walk around the KeyError by using a try and except block to handle the exception. Consider the following piece of code:

Python 
>>> a_dict = {}
>>> try:
...     # Do something with 'key'...
...     a_dict['key']
... except KeyError:
...     a_dict['key'] = 'default value'
...
>>> a_dict
{'key': 'default value'}

The try and except block in the above example catches the KeyError whenever you try to get access to a missing key. In the except clause, you create the key and assign it a 'default value'.

So far, you’ve learned how to handle missing keys using the tools that dict and Python offer you. However, the examples you saw here are quite verbose and hard to read. They might not be as straightforward as you might want. That’s why the Python standard library provides a more elegant, Pythonic, and efficient solution. That solution is collections.defaultdict, and that’s what you’ll be covering from now on.

Understanding the Python defaultdict Type

The Python standard library provides collections, which is a module that implements specialized container types. One of those is the Python defaultdict type, which is an alternative to dict that’s specifically designed to help you out with missing keys. defaultdict is a Python type that inherits from dict:

Python 
>>> from collections import defaultdict
>>> issubclass(defaultdict, dict)
True

The above code shows that the Python defaultdict type is a subclass of dict. This means that defaultdict inherits most of the behavior of dict. So, you can say that defaultdict is much like an ordinary dictionary.

The main difference between defaultdict and dict is that when you try to access or modify a key that’s not present in the dictionary, a default value is automatically given to that key. In order to provide this functionality, the Python defaultdict type does two things:

  1. It overrides .__missing__().
  2. It adds .default_factory, a writable instance variable that needs to be provided at the time of instantiation.

The instance variable .default_factory will hold the first argument passed into defaultdict.__init__(). This argument can take a valid Python callable or None. If a callable is provided, then it’ll automatically be called by defaultdict whenever you try to access or modify the value associated with a missing key.

Take a look at how you can create and properly initialize a defaultdict:

Python 
>>> # Correct instantiation
>>> def_dict = defaultdict(list)  # Pass list to .default_factory
>>> def_dict['one'] = 1  # Add a key-value pair
>>> def_dict['missing']  # Access a missing key returns an empty list
[]
>>> def_dict['another_missing'].append(4)  # Modify a missing key
>>> def_dict
defaultdict(<class 'list'>, {'one': 1, 'missing': [], 'another_missing': [4]})

Here, you pass list to .default_factory when you create the dictionary. Then, you use def_dict just like a regular dictionary. Note that when you try to access or modify the value mapped to a non-existent key, the dictionary assigns it the default value that results from calling list().

Keep in mind that you must pass a valid Python callable object to .default_factory, so remember not to call it using the parentheses at initialization time. This can be a common issue when you start using the Python defaultdict type. Take a look at the following code:

Python 
>>> # Wrong instantiation
>>> def_dict = defaultdict(list())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    def_dict = defaultdict(list())
TypeError: first argument must be callable or None

Here, you try to create a defaultdict by passing list() to .default_factory. The call to list() raises a TypeError, which tells you that the first argument must be callable or None.

With this introduction to the Python defaultdict type, you can get start coding with practical examples. The next few sections will walk you through some common use cases where you can rely on a defaultdict to provide an elegant, efficient, and Pythonic solution.

Using the Python defaultdict Type

Sometimes, you’ll use a mutable built-in collection (a list, dict, or set) as values in your Python dictionaries. In these cases, you’ll need to initialize the keys before first use, or you’ll get a KeyError. You can either do this process manually or automate it using a Python defaultdict. In this section, you’ll learn how to use the Python defaultdict type for solving some common programming problems:

  • Grouping the items in a collection
  • Counting the items in a collection
  • Accumulating the values in a collection

You’ll be covering some examples that use list, set, int, and float to perform grouping, counting, and accumulating operations in a user-friendly and efficient way.

Grouping Items

A typical use of the Python defaultdict type is to set .default_factory to list and then build a dictionary that maps keys to lists of values. With this defaultdict, if you try to get access to any missing key, then the dictionary runs the following steps:

  1. Call list() to create a new empty list
  2. Insert the empty list into the dictionary using the missing key as key
  3. Return a reference to that list

This allows you to write code like this:

Python 
>>> from collections import defaultdict
>>> dd = defaultdict(list)
>>> dd['key'].append(1)
>>> dd
defaultdict(<class 'list'>, {'key': [1]})
>>> dd['key'].append(2)
>>> dd
defaultdict(<class 'list'>, {'key': [1, 2]})
>>> dd['key'].append(3)
>>> dd
defaultdict(<class 'list'>, {'key': [1, 2, 3]})

Here, you create a Python defaultdict called dd and pass list to .default_factory. Notice that even when key isn’t defined, you can append values to it without getting a KeyError. That’s because dd automatically calls .default_factory to generate a default value for the missing key.

You can use defaultdict along with list to group the items in a sequence or a collection. Suppose that you’ve retrieved the following data from your company’s database:

Department Employee Name
Sales John Doe
Sales Martin Smith
Accounting Jane Doe
Marketing Elizabeth Smith
Marketing Adam Doe
… …

With this data, you create an initial list of tuple objects like the following:

Python 
dep = [('Sales', 'John Doe'),
       ('Sales', 'Martin Smith'),
       ('Accounting', 'Jane Doe'),
       ('Marketing', 'Elizabeth Smith'),
       ('Marketing', 'Adam Doe')]

Now, you need to create a dictionary that groups the employees by department. To do this, you can use a defaultdict as follows:

Python 
from collections import defaultdict

dep_dd = defaultdict(list)
for department, employee in dep:
    dep_dd[department].append(employee)

Here, you create a defaultdict called dep_dd and use a for loop to iterate through your dep list. The statement dep_dd[department].append(employee) creates the keys for the departments, initializes them to an empty list, and then appends the employees to each department. Once you run this code, your dep_dd will look something like this:

Python 
defaultdict(<class 'list'>, {'Sales': ['John Doe', 'Martin Smith'],
                             'Accounting' : ['Jane Doe'],
                             'Marketing': ['Elizabeth Smith', 'Adam Doe']})

In this example, you group the employees by their department using a defaultdict with .default_factory set to list. To do this with a regular dictionary, you can use dict.setdefault() as follows:

Python 
dep_d = dict()
for department, employee in dep:
    dep_d.setdefault(department, []).append(employee)

This code is straightforward, and you’ll find similar code quite often in your work as a Python coder. However, the defaultdict version is arguably more readable, and for large datasets, it can also be a lot faster and more efficient. So, if speed is a concern for you, then you should consider using a defaultdict instead of a standard dict.

Grouping Unique Items

Continue working with the data of departments and employees from the previous section. After some processing, you realize that a few employees have been duplicated in the database by mistake. You need to clean up the data and remove the duplicated employees from your dep_dd dictionary. To do this, you can use a set as the .default_factory and rewrite your code as follows:

Python 
dep = [('Sales', 'John Doe'),
       ('Sales', 'Martin Smith'),
       ('Accounting', 'Jane Doe'),
       ('Marketing', 'Elizabeth Smith'),
       ('Marketing', 'Elizabeth Smith'),
       ('Marketing', 'Adam Doe'),
       ('Marketing', 'Adam Doe'),
       ('Marketing', 'Adam Doe')]

dep_dd = defaultdict(set)
for department, employee in dep:
    dep_dd[department].add(employee)

In this example, you set .default_factory to set. Sets are collections of unique objects, which means that you can’t create a set with repeated items. This is a really interesting feature of sets, which guarantees that you won’t have repeated items in your final dictionary.

Counting Items

If you set .default_factory to int, then your defaultdict will be useful for counting the items in a sequence or collection. When you call int() with no arguments, the function returns 0, which is the typical value you’d use to initialize a counter.

To continue with the example of the company database, suppose you want to build a dictionary that counts the number of employees per department. In this case, you can code something like this:

Python 
>>> from collections import defaultdict
>>> dep = [('Sales', 'John Doe'),
...        ('Sales', 'Martin Smith'),
...        ('Accounting', 'Jane Doe'),
...        ('Marketing', 'Elizabeth Smith'),
...        ('Marketing', 'Adam Doe')]
>>> dd = defaultdict(int)
>>> for department, _ in dep:
...     dd[department] += 1
>>> dd
defaultdict(<class 'int'>, {'Sales': 2, 'Accounting': 1, 'Marketing': 2})

Here, you set .default_factory to int. When you call int() with no argument, the returned value is 0. You can use this default value to start counting the employees that work in each department. For this code to work correctly, you need a clean dataset. There must be no repeated data. Otherwise, you’ll need to filter out the repeated employees.

Another example of counting items is the mississippi example, where you count the number of times each letter in a word is repeated. Take a look at the following code:

Python 
>>> from collections import defaultdict
>>> s = 'mississippi'
>>> dd = defaultdict(int)
>>> for letter in s:
...     dd[letter] += 1
...
>>> dd
defaultdict(<class 'int'>, {'m': 1, 'i': 4, 's': 4, 'p': 2})

In the above code, you create a defaultdict with .default_factory set to int. This sets the default value for any given key to 0. Then, you use a for loop to traverse the string s and use an augmented assignment operation to add 1 to the counter in every iteration. The keys of dd will be the letters in mississippi.

As counting is a relatively common task in programming, the Python dictionary-like class collections.Counter is specially designed for counting items in a sequence. With Counter, you can write the mississippi example as follows:

Python 
>>> from collections import Counter
>>> counter = Counter('mississippi')
>>> counter
Counter({'i': 4, 's': 4, 'p': 2, 'm': 1})

In this case, Counter does all the work for you! You only need to pass in a sequence, and the dictionary will count its items, storing them as keys and the counts as values. Note that this example works because Python strings are also a sequence type.

Accumulating Values

Sometimes you’ll need to calculate the total sum of the values in a sequence or collection. Let’s say you have the following Excel sheet with data about the sales of your Python website:

Products July August September
Books 1250.00 1300.00 1420.00
Tutorials 560.00 630.00 750.00
Courses 2500.00 2430.00 2750.00

Next, you process the data using Python and get the following list of tuple objects:

Python 
incomes = [('Books', 1250.00),
           ('Books', 1300.00),
           ('Books', 1420.00),
           ('Tutorials', 560.00),
           ('Tutorials', 630.00),
           ('Tutorials', 750.00),
           ('Courses', 2500.00),
           ('Courses', 2430.00),
           ('Courses', 2750.00),]

With this data, you want to calculate the total income per product. To do that, you can use a Python defaultdict with float as .default_factory and then code something like this:

Python 
 1from collections import defaultdict
 2
 3dd = defaultdict(float)
 4for product, income in incomes:
 5    dd[product] += income
 6
 7for product, income in dd.items():
 8    print(f'Total income for {product}: ${income:,.2f}')

Here’s what this code does:

  • In line 1, you import the Python defaultdict type.
  • In line 3, you create a defaultdict object with .default_factory set to float.
  • In line 4, you define a for loop to iterate through the items of incomes.
  • In line 5, you use an augmented assignment operation (+=) to accumulate the incomes per product in the dictionary.

The second loop iterates through the items of dd and prints the incomes to your screen.

If you put all this code into a file called incomes.py and run it from your command line, then you’ll get the following output:

Shell 
$ python3 incomes.py
Total income for Books: $3,970.00
Total income for Tutorials: $1,940.00
Total income for Courses: $7,680.00

You now have a summary of incomes per product, so you can make decisions on which strategy to follow for increasing the total income of your site.

Diving Deeper Into defaultdict

So far, you’ve learned how to use the Python defaultdict type by coding some practical examples. At this point, you can dive deeper into type implementation and other working details. That’s what you’ll be covering in the next few sections.

defaultdict vs dict

For you to better understand the Python defaultdict type, a good exercise would be to compare it with its superclass, dict. If you want to know the methods and attributes that are specific to the Python defaultdict type, then you can run the following line of code:

Python 
>>> set(dir(defaultdict)) - set(dir(dict))
{'__copy__', 'default_factory', '__missing__'}

In the above code, you use dir() to get the list of valid attributes for dict and defaultdict. Then, you use a set difference to get the set of methods and attributes that you can only find in defaultdict. As you can see, the differences between these two classes are. You have two methods and one instance attribute. The following table shows what the methods and the attribute are for:

Method or Attribute Description
.__copy__() Provides support for copy.copy()
.default_factory Holds the callable invoked by .__missing__() to automatically provide default values for missing keys
.__missing__(key) Gets called when .__getitem__() can’t find key

In the above table, you can see the methods and the attribute that make a defaultdict different from a regular dict. The rest of the methods are the same in both classes.

Additionally, you might notice that a defaultdict is equal to a dict with the same items:

Python 
>>> std_dict = dict(numbers=[1, 2, 3], letters=['a', 'b', 'c'])
>>> std_dict
{'numbers': [1, 2, 3], 'letters': ['a', 'b', 'c']}
>>> def_dict = defaultdict(list, numbers=[1, 2, 3], letters=['a', 'b', 'c'])
>>> def_dict
defaultdict(<class 'list'>, {'numbers': [1, 2, 3], 'letters': ['a', 'b', 'c']})
>>> std_dict == def_dict
True

Here, you create a regular dictionary std_dict with some arbitrary items. Then, you create a defaultdict with the same items. If you test both dictionaries for content equality, then you’ll see that they’re equal.

defaultdict.default_factory

The first argument to the Python defaultdict type must be a callable that takes no arguments and returns a value. This argument is assigned to the instance attribute, .default_factory. For this, you can use any callable, including functions, methods, classes, type objects, or any other valid callable. The default value of .default_factory is None.

If you instantiate defaultdict without passing a value to .default_factory, then the dictionary will behave like a regular dict and the usual KeyError will be raised for missing key lookup or modification attempts:

Python 
>>> from collections import defaultdict
>>> dd = defaultdict()
>>> dd['missing_key']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    dd['missing_key']
KeyError: 'missing_key'

Here, you instantiate the Python defaultdict type with no arguments. In this case, the instance behaves like a standard dictionary. So, if you try to access or modify a missing key, then you’ll get the usual KeyError. From this point on, you can use dd as a normal Python dictionary and, unless you assign a new callable to .default_factory, you won’t be able to use the ability of defaultdict to handle missing keys automatically.

If you pass None to the first argument of defaultdict, then the instance will behave the same way you saw in the above example. That’s because .default_factory defaults to None, so both initializations are equivalent. On the other hand, if you pass a valid callable object to .default_factory, then you can use it to handle missing keys in a user-friendly way. Here’s an example where you pass list to .default_factory:

Python 
>>> dd = defaultdict(list, letters=['a', 'b', 'c'])
>>> dd.default_factory
<class 'list'>
>>> dd
defaultdict(<class 'list'>, {'letters': ['a', 'b', 'c']})
>>> dd['numbers']
[]
>>> dd
defaultdict(<class 'list'>, {'letters': ['a', 'b', 'c'], 'numbers': []})
>>> dd['numbers'].append(1)
>>> dd
defaultdict(<class 'list'>, {'letters': ['a', 'b', 'c'], 'numbers': [1]})
>>> dd['numbers'] += [2, 3]
>>> dd
defaultdict(<class 'list'>, {'letters': ['a', 'b', 'c'], 'numbers': [1, 2, 3]})

In this example, you create a Python defaultdict called dd, then you use list for its first argument. The second argument is called letters and holds a list of letters. You see that .default_factory now holds a list object that will be called when you need to supply a default value for any missing key.

Notice that when you try to access numbers, dd tests if numbers is in the dictionary. If it’s not, then it calls .default_factory(). Since .default_factory holds a list object, the returned value is an empty list ([]).

Now that dd['numbers'] is initialized with an empty list, you can use .append() to add elements to the list. You can also use an augmented assignment operator (+=) to concatenate the lists [1] and [2, 3]. This way, you can handle missing keys in a more Pythonic and more efficient way.

On the other hand, if you pass a non-callable object to the initializer of the Python defaultdict type, then you’ll get a TypeError like in the following code:

Python 
>>> defaultdict(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    defaultdict(0)
TypeError: first argument must be callable or None

Here, you pass 0 to .default_factory. Since 0 is not a callable object, you get a TypeError telling you that the first argument must be callable or None. Otherwise, defaultdict doesn’t work.

Keep in mind that .default_factory is only called from .__getitem__() and not from other methods. This means that if dd is a defaultdict and key is a missing key, then dd[key] will call .default_factory to provide a default value, but dd.get(key) still returns None instead of the value that .default_factory would provide. That’s because .get() doesn’t call .__getitem__() to retrieve the key.

Take a look at the following code:

Python 
>>> dd = defaultdict(list)
>>> # Calls dd.__getitem__('missing')
>>> dd['missing']
[]
>>> # Don't call dd.__getitem__('another_missing')
>>> print(dd.get('another_missing'))
None
>>> dd
defaultdict(<class 'list'>, {'missing': []})

In this code fragment, you can see that dd.get() returns None rather than the default value that .default_factory would provide. That’s because .default_factory is only called from .__missing__(), which is not called by .get().

Notice that you can also add arbitrary values to a Python defaultdict. This means that you’re not limited to values with the same type as the values generated by .default_factory. Here’s an example:

Python 
>>> dd = defaultdict(list)
>>> dd
defaultdict(<class 'list'>, {})
>>> dd['string'] = 'some string'
>>> dd
defaultdict(<class 'list'>, {'string': 'some string'})
>>> dd['list']
[]
>>> dd
defaultdict(<class 'list'>, {'string': 'some string', 'list': []})

Here, you create a defaultdict and pass in a list object to .default_factory. This sets your default values to be empty lists. However, you can freely add a new key that holds values of a different type. That’s the case with the key string, which holds a str object instead of a list object.