4 Techniques for Testing Python Command-Line (CLI) Apps

You’ve just finished building your first Python command-line app. Or maybe your second or third. You’ve been learning Python for a while, and now you’re ready to build something bigger and more complex, but still runnable on a command-line. Or you are used to building and testing web applications or desktop apps with a GUI, but now are starting to build CLI applications.

In all these situations and more, you will need to learn and get comfortable with the various methods for testing a Python CLI application.

While the tooling choices can be intimidating, the main thing to keep in mind is that you’re just comparing the outputs your code generates to the outputs you expect. Everything follows from that.

In this tutorial you’ll learn four hands-on techniques for testing Python command-line apps:

“Lo-Fi” debugging with print()
Using a visual Python debugger
Unit testing with pytest and mocks
Integration testing

Free Bonus: Click here to get our Python Testing Cheat Sheet that summarizes the techniques demonstrated in this tutorial.

Everything will be structured around a basic Python CLI app that passes data in the form of a multi-level dictionary to two functions that transform it in some way, then prints it to the user.

We will use the code below to examine a few of the different methods that will aid you in testing. And while certainly not exhaustive, I hope this tutorial will give you enough breadth to get you confident in creating effective tests in the major testing domains.

I’ve sprinkled in a few bugs in this initial code, which we will expose with our testing methods.

Note: For simplicity’s sake, this code does not include some basic best practices, such as verifying the existence of keys in a dictionary.

As a first step, let’s think about our objects at every stage of this application. We start with a structure that describes John Q. Public:

JOHN_DATA = {
    'name': 'John Q. Public',
    'street': '123 Main St.',
    'city': 'Anytown',
    'state': 'FL',
    'zip': 99999,
    'relationships': {
        'siblings': ['Michael R. Public', 'Suzy Q. Public'],
        'parents': ['John Q. Public Sr.', 'Mary S. Public'],
    }
}

We then flatten the other dictionaries, expecting this after calling our first transform function, initial_transform:

JOHN_DATA = {
    'name': 'John Q. Public',
    'street': '123 Main St.',
    'city': 'Anytown',
    'state': 'FL',
    'zip': 99999,
    'siblings': ['Michael R. Public', 'Suzy Q. Public'],
    'parents': ['John Q. Public Sr.', 'Mary S. Public'],
}

Then we build all the address information into a single address entry with the function final_transform:

JOHN_DATA = {
    'name': 'John Q. Public',
    'address': '123 Main St. \nAnytown, FL 99999'
    'siblings': ['Michael R. Public', 'Suzy Q. Public'],
    'parents': ['John Q. Public Sr.', 'Mary S. Public'],
}

And the call to print_person will write this to the console:

Hello, my name is John Q. Public, my siblings are Michael R. Public 
and Suzy Q. Public, my parents are John Q. Public Sr. and Mary S. Public, 
and my mailing address is:
123 Main St. 
Anytown, FL 99999

testapp.py:

def initial_transform(data):
    """
    Flatten nested dicts
    """
    for item in list(data):
        if type(item) is dict:
            for key in item:
                data[key] = item[key]

    return data


def final_transform(transformed_data):
    """
    Transform address structures into a single structure
    """
    transformed_data['address'] = str.format(
        "{0}\n{1}, {2} {3}", transformed_data['street'], 
        transformed_data['state'], transformed_data['city'], 
        transformed_data['zip'])

    return transformed_data


def print_person(person_data):
    parents = "and".join(person_data['parents'])
    siblings = "and".join(person_data['siblings'])
    person_string = str.format(
        "Hello, my name is {0}, my siblings are {1}, "
        "my parents are {2}, and my mailing"
        "address is: \n{3}", person_data['name'], 
        parents, siblings, person_data['address'])
    print(person_string)


john_data = {
    'name': 'John Q. Public',
    'street': '123 Main St.',
    'city': 'Anytown',
    'state': 'FL',
    'zip': 99999,
    'relationships': {
        'siblings': ['Michael R. Public', 'Suzy Q. Public'],
        'parents': ['John Q. Public Sr.', 'Mary S. Public'],
    }
}

suzy_data = {
    'name': 'Suzy Q. Public',
    'street': '456 Broadway',
    'apt': '333',
    'city': 'Miami',
    'state': 'FL',
    'zip': 33333,
    'relationships': {
        'siblings': ['John Q. Public', 'Michael R. Public', 
                    'Thomas Z. Public'],
        'parents': ['John Q. Public Sr.', 'Mary S. Public'],
    }
}

inputs = [john_data, suzy_data]

for input_structure in inputs:
    initial_transformed = initial_transform(input_structure)
    final_transformed = final_transform(initial_transformed)
    print_person(final_transformed)

Right now, the code doesn’t actually meet those expectations, so we will investigate using the four techniques while we are learning about them. By doing this, you will get practical experience in using these techniques, expand your comfort zone to them, and begin to learn for which problems they are most suited.

“Lo-Fi” Debugging With Print

This is one of the simplest ways to test. All you have to do here is print a variable or object that you’re interested in–before a function call, after a function call, or within a function.

Respectively, these allow you to verify a function’s input, a function’s output, and a function’s logic.

If you save the code above as testapp.py and try to run it with python testapp.py you’ll see an error like so:

Traceback (most recent call last):
  File "testapp.py", line 60, in <module>
    print_person(final_transformed)
  File "testapp.py", line 23, in print_person
    parents = "and".join(person_data['parents'])
KeyError: 'parents'

There is a missing key in person_data that is passed into print_person. The first step would be to check the input to print_person and see why our expected output (a printed message) isn’t being generated. We will just add a print function call before the call to print_person:

final_transformed = final_transform(initial_transformed)
print(final_transformed)
print_person(final_transformed)

The print function does the job here, showing in its output that we don’t have the top-level parents key—nor the siblings key—but in the interest of our sanity, I’ll show you pprint, which prints multi-level objects in a more readable manner. To use it, add from pprint import pprint to the top of your script.

Instead of print(final_transformed), we call pprint(final_transformed) to inspect our object:

{'address': '123 Main St.\nFL, Anytown 99999',
 'city': 'Anytown',
 'name': 'John Q. Public',
 'relationships': {'parents': ['John Q. Public Sr.', 'Mary S. Public'],
                   'siblings': ['Michael R. Public', 'Suzy Q. Public']},
 'state': 'FL',
 'street': '123 Main St.',
 'zip': 99999}

Compare this with the expected final form above.

Because we know final_transform doesn’t touch the relationships dictionary, it’s time to see what is going on in initial_transform. Normally, I’d use a traditional debugger to step through this, but I want to show you another use of print debugging.

We can print the state of objects in code, but we aren’t limited to that. We can print whatever we want, so we can also print markers to see which logic branches are executed and when.

Because initial_transform is primarily a few loops, and because internal dictionaries are supposed to be handled by the inner for loop, we should check out what’s happening in there, if anything:

def initial_transform(data):
    """
    Flatten nested dicts
    """