Python performance optimization is the practice of making your code run faster. You find the bottleneck first, then fix only what’s slow, starting with algorithms and data structures before reaching for concurrency or C extensions.
You’ve written a Python script that works perfectly until you run it on real data and it crawls, or quietly consumes far more memory than it should. The results are correct, and the tests pass, so the code isn’t broken. It’s just inefficient.
The trap most developers fall into here is guessing. You stare at your code, decide which part looks slow, and rewrite it. Sometimes you get lucky. Often, you burn an afternoon speeding up a function that was never the problem. In this tutorial, you’ll learn to measure first and choose the right fix for a slowdown so you can stop guessing and start diagnosing.
By the end of this tutorial, you’ll understand that:
- Measuring with
timeitconfirms a bottleneck you already suspect, while a profiler finds one in an unfamiliar codebase. - Switching a membership test from a
listto asetturns an O(n) scan into a lookup that’s constant time on average. - Python’s built-ins and standard library often push per-item work down into C, so
Countercounts faster than a handwritten loop. functools.cachestores results by argument, so it’s only safe when a function’s output depends solely on its arguments.- The fixes here have a natural order, starting with the cheap, safe ones and escalating to more invasive ones only when the simpler fixes aren’t enough.
One thing before you start: this is a conceptual guide. The mindset comes first, and the tools follow. Once you understand the order in which to tackle a performance problem, the specific techniques become much easier to apply.
Before you dig in, these two related tutorials are worth bookmarking:
With those on hand, you’ll start by pinning down what optimization actually means.
Get Your Code: Click here to download the free sample code you’ll use to find and fix the performance bottlenecks in your Python code.
Take the Quiz: Test your knowledge with our interactive “Python Performance Optimization: A Practical Guide” quiz. You’ll receive a score upon completion to help you track your learning progress:
Interactive Quiz
Python Performance Optimization: A Practical GuideTest your grasp of Python performance optimization, from confirming a bottleneck with timeit to picking fixes like sets, built-ins, and caching.
What Is Python Performance Optimization?
Before you can fix a performance problem, you need to understand what you’re actually trying to do. This section starts with a clear definition and a mental model to guide you through the rest of the tutorial.
The Goal of Optimization
Here’s the part that trips people up: optimization isn’t the same as fixing bugs. Your code already works, so the question isn’t whether it produces the right answer but how efficiently it gets there. You pursue that efficiency deliberately, not by tinkering at random.
The Doctor Analogy
Think about how a good doctor works. You walk in feeling unwell, and they don’t immediately hand you a prescription. They run tests first, then treat the specific problem the results reveal. Prescribing medication before a diagnosis would be reckless.
When you optimize without measuring, you’re doing exactly that: prescribing a cure for a disease you haven’t diagnosed. The bottleneck—the specific part of your program that consumes the most time or memory—is the disease.
Keep this picture in mind because it anchors everything that follows: diagnose, confirm, then treat. Profilers do the diagnosing, and Real Python has tutorials devoted to them. This tutorial picks up at the confirmation step and focuses on treatment.
Now that you have a clear definition and the right mental model, you’re ready to put them to work. In the next section, you’ll first confirm your suspicion, then choose the fix that matches what you find.
Measure First With timeit
Before you fix anything, you need to know whether it’s actually slow and by how much. Locating a bottleneck in a large, unfamiliar program is a job for a profiler like cProfile in the standard library or the third-party line_profiler. To go deeper on both, read Profiling in Python: How to Find Performance Bottlenecks and Python Timer Functions: Three Ways to Monitor Your Code.
This tutorial isn’t about that process. It picks up one step later, once you already suspect a specific function is the problem and want to confirm it. This is where timeit comes in. It doesn’t hunt for bottlenecks. It measures one so you can confirm your suspicion with a real number instead of a guess.
Suppose you’re processing a batch of customer orders, and you suspect that the function for calculating each order’s total is holding up the job. Timing it over half a million calls gives you the two numbers you need:
import timeit
def calculate_order_total(items):
total = 0
for item in items:
total = total + item["price"] * item["quantity"]
return total
order = [
{"price": 19.99, "quantity": 3},
{"price": 5.50, "quantity": 10},
{"price": 42.00, "quantity": 1},
] * 100
runs = 500_000
print(f"{calculate_order_total(order):.2f}")
order_time = timeit.timeit(
lambda: calculate_order_total(order), number=runs
)
print(f"total for {runs:,} runs: {order_time:.4f} seconds")
print(f"per call: {order_time / runs * 1_000_000:.1f} microseconds")
$ python step-1.py
15697.00
total for 500,000 runs: 4.4478 seconds
per call: 8.9 microseconds
Read those two numbers carefully because conflating them is one of the easiest mistakes to make with timeit. The 4.4 seconds is the total time across half a million calls. A single call takes just 8.9 microseconds, shown on the line underneath. That cost adds up to something worth your attention only because the batch runs the function so many times.
Your own numbers will differ, and that’s expected because the machine matters. Focus on the ratio between measurements rather than any single figure.
Neither number tells you why the function takes that much time, but both provide useful baselines before you optimize anything. The general workflow is to measure, change one thing, then rerun timeit to see whether the numbers have changed. The examples in the next sections walk through that cycle.
With the per-call cost already this low, any real win has to come from doing the work fewer times or from changing what happens inside the loop, not from shaving microseconds off a single call. The fixes ahead each target a different kind of bottleneck.
Choose Your Fix
Once you know where the time is going, choose a fix based on what you’ve found. The sections below start with the fixes that pay off most for the least effort, so work through them only as far as you need to.
Choosing Better Algorithms and Data Structures
Choosing the right algorithm or data structure often yields the biggest performance gains. Even on a fast machine, a slow algorithm still produces a slow program. A common example is membership checking: searching for an item in a list is O(n), meaning Python may have to scan every element. With a set, Python performs the same check in constant time on average, O(1).
numbers_list holds 100,000 numbers, and the target is the very last one. That’s the worst case for a list because Python has to scan every element before it finds a match:
import timeit
numbers_list = list(range(100_000))
target = 99_999
list_time = timeit.timeit(lambda: target in numbers_list, number=1000)
print(target in numbers_list)
print(f"list membership: {list_time:.8f} seconds")
$ python step-2-slow.py
True
list membership: 0.39360604 seconds
numbers_set holds the same 100,000 numbers, stored in a hash table. Python jumps directly to the right location regardless of how many items the set holds:
import timeit
numbers_list = list(range(100_000))
target = 99_999
numbers_set = set(numbers_list)
set_time = timeit.timeit(lambda: target in numbers_set, number=1000)
print(target in numbers_set)
print(f"set membership: {set_time:.8f} seconds")
$ python step-2-fast.py
True
set membership: 0.00003825 seconds
The same caveat applies to every comparison that follows. What matters is the ratio between the times rather than the raw measurements.
Building the set isn’t free, though. set(numbers_list) walks the whole list once, but hashing and inserting each item is several times as expensive as a single comparison, so building the set takes roughly as long as two to three full scans. That’s a bargain when you’re testing membership over and over, which is the usual reason to reach for a set. Converting a list you’ll only search once will cost you more than it saves.
Real Python’s guide to Common Python Data Structures compares lists, sets, dictionaries, and the specialized containers in collections, so you can match the container to the access pattern your code actually needs.
Reaching for Built-Ins and the Standard Library
Python’s built-in functions and many standard library tools are already highly optimized, whether through an underlying C implementation or carefully tuned Python code. collections.Counter tallies items in a single call and beats a manual counting loop, not because it’s magic, but because someone has already done the optimization work for you.
manual_count() loops through the list and updates a dictionary one item at a time, with a conditional check on every iteration:
import timeit
words = ["apple", "pear", "apple", "cherry", "pear", "apple"] * 1000
def manual_count(items):
counts = {}
for item in items:
if item in counts:
counts[item] += 1
else:
counts[item] = 1
return counts
print(manual_count(words))
manual_time = timeit.timeit(lambda: manual_count(words), number=1000)
print(f"manual loop: {manual_time:.4f} seconds")
$ python step-3-slow.py
{'apple': 3000, 'pear': 2000, 'cherry': 1000}
manual loop: 0.2081 seconds
counter_count() hands the entire list to Counter, which does the same job without running a loop in Python. Counter is a dict subclass, so you get the same counts wrapped in a richer type.
Counter itself is written in Python, but when you give it an iterable, it delegates the counting to a helper that CPython implements in C, so the per-item work happens at C speed. The word list contains 6,000 items (6 entries multiplied by 1,000), and timeit runs each function 1,000 times, making the speed difference visible in the output:
from collections import Counter
import timeit
words