Efficient String Concatenation in Python

Efficient String Concatenation in Python

by Leodanis Pozo Ramos Updated Reading time estimate 19m basics best-practices python

Python string concatenation is a fundamental operation that combines multiple strings into a single string. In Python, you can concatenate strings using the + operator or the += operator for appending. For more efficient concatenation of multiple strings, the .join() method is recommended, especially when working with strings in a list. Other techniques include using StringIO for large datasets or the print() function for quick screen outputs.

By the end of this tutorial, you’ll understand that:

  • You can concatenate strings in Python using the + and += operators.
  • You can use += to append a string to an existing string.
  • The .join() method is used to combine strings in a list in Python.
  • You can handle a stream of strings efficiently by using StringIO as a container with a file-like interface.

To get the most out of this tutorial, you should have a basic understanding of Python, especially its built-in string data type.

Doing String Concatenation With Python’s Plus Operator (+)

String concatenation is a pretty common operation consisting of joining two or more strings together end to end to build a final string. Perhaps the quickest way to achieve concatenation is to take two separate strings and combine them with the plus operator (+), which is known as the concatenation operator in this context:

Language: Python
>>> "Hello, " + "Pythonista!"
'Hello, Pythonista!'

>>> head = "String Concatenation "
>>> tail = "is Fun in Python!"
>>> head + tail
'String Concatenation is Fun in Python!'

Using the concatenation operator to join two strings provides a quick solution for concatenating only a few strings.

For a more realistic example, say you have an output line that will print an informative message based on specific criteria. The beginning of the message might always be the same. However, the end of the message will vary depending on different criteria. In this situation, you can take advantage of the concatenation operator:

Language: Python
>>> def age_group(age):
...     if 0 <= age <= 9:
...         result = "a Child!"
...     elif 9 < age <= 18:
...         result = "an Adolescent!"
...     elif 19 < age <= 65:
...         result = "an Adult!"
...     else:
...         result = "in your Golden Years!"
...     print("You are " + result)
...

>>> age_group(29)
You are an Adult!
>>> age_group(14)
You are an Adolescent!
>>> age_group(68)
You are in your Golden Years!

In the above example, age_group() prints a final message constructed with a common prefix and the string resulting from the conditional statement. In this type of use case, the plus operator is your best option for quick string concatenation in Python.

The concatenation operator has an augmented version that provides a shortcut for concatenating two strings together. The augmented concatenation operator (+=) has the following syntax:

Language: Python
string += other_string

This expression will concatenate the content of string with the content of other_string. It’s equivalent to saying string = string + other_string.

Here’s a short example of how the augmented concatenation operator works in practice:

Language: Python
>>> word = "Py"
>>> word += "tho"
>>> word += "nis"
>>> word += "ta"
>>> word
'Pythonista'

In this example, every augmented assignment adds a new syllable to the final word using the += operator. This concatenation technique can be useful when you have several strings in a list or any other iterable and want to concatenate them in a for loop:

Language: Python
>>> def concatenate(iterable, sep=" "):
...     sentence = iterable[0]
...     for word in iterable[1:]:
...         sentence += (sep + word)
...     return sentence
...

>>> concatenate(["Hello,", "World!", "I", "am", "a", "Pythonista!"])
'Hello, World! I am a Pythonista!'

Inside the loop, you use the augmented concatenation operator to quickly concatenate several strings in a loop. Later you’ll learn about .join(), which is an even better way to concatenate a list of strings.

Python’s concatenation operators can only concatenate string objects. If you use them with a different data type, then you get a TypeError:

Language: Python
>>> "The result is: " + 42
Traceback (most recent call last):
    ...
TypeError: can only concatenate str (not "int") to str

>>> "Your favorite fruits are: " + ["apple", "grape"]
Traceback (most recent call last):
    ...
TypeError: can only concatenate str (not "list") to str

The concatenation operators don’t accept operands of different types. They only concatenate strings. A work-around to this issue is to explicitly use the built-in str() function to convert the target object into its string representation before running the actual concatenation:

Language: Python
>>> "The result is: " + str(42)
'The result is: 42'

By calling str() with your integer number as an argument, you’re retrieving the string representation of 42, which you can then concatenate to the initial string because both are now string objects.

String concatenation using + and its augmented variation, +=, can be handy when you only need to concatenate a few strings. However, these operators aren’t an efficient choice for joining many strings into a single one. Why? Python strings are immutable, so you can’t change their value in place. Therefore, every time you use a concatenation operator, you’re creating a new string object.

This behavior implies extra memory consumption and processing time because creating a new string uses both resources. So, the concatenation will be costly in two dimensions: memory consumption and execution time. Fortunately, Python has an efficient tool for you to deal with concatenating multiple strings. That tool is the .join() method from the str class.

Efficiently Concatenating Many Strings With .join() in Python

You can call the .join() method on a string object that will work as a separator in the string concatenation process. Given an iterable of strings, .join() efficiently concatenates all the contained strings together into one:

Language: Python
>>> " ".join(["Hello,", "World!", "I", "am", "a", "Pythonista!"])
'Hello, World! I am a Pythonista!'

In this example, you call .join() on a whitespace character, which is a concrete object of the built-in str class expressed as a literal. This character is inserted between the strings in the input list, generating a single string.

The .join() method is cleaner, more Pythonic, and more readable than concatenating many strings together in a loop using the augmented concatenation operator (+=), as you saw before. An explicit loop is way more complex and harder to understand than the equivalent call to .join(). The .join() method is also faster and more efficient regarding memory usage.

Unlike the concatenation operators, Python’s .join() doesn’t create new intermediate strings in each iteration. Instead, it creates a single new string object by joining the elements from the input iterable with the selected separator string. This behavior is more efficient than using the regular concatenation operators in a loop.

It’s important to note that .join() doesn’t allow you to concatenate non-string objects directly:

Language: Python
>>> "; ".join([1, 2, 3, 4, 5])
Traceback (most recent call last):
    ...
TypeError: sequence item 0: expected str instance, int found

When you try to join non-string objects using .join(), you get a TypeError, which is consistent with the behavior of concatenation operators. Again, to work around this behavior, you can take advantage of str() and a generator expression:

Language: Python
>>> numbers = [1, 2, 3, 4, 5]

>>> "; ".join(str(number) for number in numbers)
'1; 2; 3; 4; 5'

The generator expression in the call to .join() converts every number into a string object before running the concatenation and producing the final string. With this technique, you can concatenate objects of different types into a string.

Doing Repeated Concatenation With the Star Operator (*)

You can also use the star operator (*) to concatenate strings in Python. In this context, this operator is known as the repeated concatenation operator. It works by repeating a string a certain number of times. Its syntax is shown below, along with its augmented variation:

Language: Python
string = string * n

string *= n

The repetition operator takes two operands. The first operand is the string that you want to repeat in the concatenation, while the second operand is an integer number representing how many times you want to repeat the target string. The augmented syntax is equivalent to the regular one but shorter.

A common example of using this concatenation tool is when you need to generate a separator string to use in tabular outputs. For example, say you’ve read a CSV file with information about people into a list of lists:

Language: Python
>>> data = [
...    ["Name", "Age", "Hometown"],
...    ["Alice", "25", "New York"],
...    ["Bob", "30", "Los Angeles"],
...    ["Charlie", "35", "Chicago"]
... ]

The first row of your list contains the table headers. Now you want to display this info in a table. You can do something like the following:

Language: Python
>>> def display_table(data):
...     max_len = max(len(header) for header in data[0])
...     sep = "-" * max_len
...     for row in data:
...         print("|".join(header.ljust(max_len) for header in row))
...         if row == data[0]:
...             print("|".join(sep for _ in row))
...

>>> display_table(data)
Name    |Age     |Hometown
--------|--------|--------
Alice   |25      |New York
Bob     |30      |Los Angeles
Charlie |35      |Chicago

In this example, you use the * operator to repeat the string "-" as many times as defined in max_len, which holds the number of characters in the longest table header. The loop prints the data in a tabular format. Note how the conditional statement prints a separation line between the headers and the actual data.

Exploring Other Tools for String Concatenation

Python is a highly flexible and versatile programming language. Even though the Zen of Python states that there should be one—and preferably only one—obvious way to do it, you’ll often find several options for running a given computation or performing a certain action in Python. String concatenation is no exception to this behavior.

In the following sections, you’ll learn about a few additional techniques and tools that you can use for string concatenation.

Taking Advantage of String Literal Concatenation

Another quick way to concatenate multiple strings in Python is to write the string literals consecutively:

Language: Python
>>> "Hello," " " "World!"
'Hello, World!'

>>> message = "To: " "The Python Community" " -> " "Welcome Folks!"
>>> message
'To: The Python Community -> Welcome Folks!'

In these examples, you can see that Python automatically merges multiple strings into a single one when you place them side by side. This feature is known as string literal concatenation, and it’s documented as an intentional behavior of Python. Note that you can even define variables using this feature.

Some common use cases for string literal concatenation include:

  • Using different quoting styles without having to escape quote symbols
  • Splitting long strings conveniently across multiple lines
  • Adding comments to parts of strings

For example, say that you need to use double quotes and apostrophes in the same string:

Language: Python
>>> phrase = (
...     "Guido's talk wasn't named "  # String with apostrophes
...     '"Feeding CPython to ChatGPT"'  # String with double quotes
... )

>>> print(phrase)
Guido's talk wasn't named "Feeding CPython to ChatGPT"

In this example, you take advantage of string literal concatenation to add comments to different parts of your string and to escape double and single quotes within your final string.

This feature may seem neat at first glance. However, it can be a good way to shoot yourself in the foot. For example, say that you’re writing a list of strings and accidentally forget an intermediate comma:

Language: Python
>>> hobbies = [
...     "Reading",
...     "Writing",
...     "Painting",
...     "Drawing",
...     "Sculpting" # Accidentally missing comma
...     "Gardening",
...     "Cooking",
...     "Baking",
... ]

>>> hobbies
[
    'Reading',
    'Writing',
    'Painting',
    'Drawing',
    'SculptingGardening',
    'Cooking',
    'Baking'
]

When typing your list, you accidentally missed the comma that separates "Sculpting" from "Gardening". This mistake introduces a subtle bug into your programs. Because Python doesn’t find the separating comma, it triggers automatic string literal concatenation, joining both items in a single string, 'SculptingGardening'. This type of error may pass unnoticed, causing hard-to-debug issues in your code.

In general, to avoid surprises in your code, you should use explicit string concatenation with + or += whenever you’re working with long strings or strings that contain escape sequences.

Concatenating Strings With StringIO

If you’re working with many strings in a data stream, then StringIO may be a good option for string concatenation. This class provides a native in-memory Unicode container with great speed and performance.

To concatenate strings with StringIO, you first need to import the class from the io module. Then you can use the .write() method to append individual strings to the in-memory buffer:

Language: Python
>>> from io import StringIO

>>> words = ["Hello,", "World!", "I", "am", "a", "Pythonista!"]

>>> sentence = StringIO()
>>> sentence.write(words[0])
6
>>> for word in words[1:]:
...     sentence.write(" " + word)
...
7
2
3
2
12
>>> sentence.getvalue()
'Hello, World! I am a Pythonista!'

Here, you’ll quickly note that a bunch of numbers appear on your screen between operations on the StringIO object. These numbers represent the bytes written or retrieved from the object in each writing or reading operation.

In the above example, you’ve used a finite data stream represented by a list of strings. If you need to work with a potentially infinite data stream, then you should use a while loop instead. For example, here’s a small script that takes words from the user and concatenates them into a sentence:

Language: Python
# sentence.py

from io import StringIO

sentence = StringIO()
while True:
    word = input("Enter a word (or './!/?' to end the sentence): ")
    if word in ".!?":
        sentence.write(word)
        break
    if sentence.tell() == 0:
        sentence.write(word)
    else:
        sentence.write(" " + word)

print("The concatenated sentence is:", sentence.getvalue())

This script grabs the user’s input using the built-in input() function. If the input is a period, an exclamation point, or a question mark, then the loop breaks, terminating the input. Then you check if the buffer is empty by using the .tell() method. Depending on this check, the statement adds the current word only or the word with a leading whitespace.

Here’s how this script works in practice:

Language: Shell
$ python sentence.py
Enter a word (or './!/?' to end the sentence): Hello,
Enter a word (or './!/?' to end the sentence): welcome
Enter a word (or './!/?' to end the sentence): to
Enter a word (or './!/?' to end the sentence): Real
Enter a word (or './!/?' to end the sentence): Python
Enter a word (or './!/?' to end the sentence): !
The concatenated sentence is: Hello, welcome to Real Python!

Cool! Your script works nicely! It takes words at the command line and builds a sentence using StringIO for string concatenation.

Using StringIO to concatenate strings can be an excellent alternative to using the concatenation operators. This tool is handy when you need to deal with a large or unknown number of strings. StringIO can be pretty efficient because it avoids creating intermediate strings. Instead, it appends them directly to the in-memory buffer, which can give you great performance.

StringIO also provides a consistent interface with other file-like Python objects, such as those that open() returns. This means that you can use the same methods for reading and writing data with StringIO as you would with a regular file object.

Using print() to Concatenate Strings

You can also use the built-in print() function to perform some string concatenations, especially to concatenate strings for on-screen messages. This function can help you connect strings with a specific separator:

Language: Python
>>> words = ["Hello,", "World!", "I", "am", "a", "Pythonista!"]

>>> print(*words)
Hello, World! I am a Pythonista!

>>>