Python Text Processing

This chapter will discuss str methods and introduce a few examples with the string and re modules.

join

The join() method is similar to what the print() function does with the sep option, except that you get a str object as the result. The iterable you pass to join() can only have string elements. On the other hand, print() uses an object's __str__() method to get its string representation (__repr__() method is used as a fallback).

>>> print(1, 2)
1 2
>>> ' '.join((1, 2))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: sequence item 0: expected str instance, int found
>>> ' '.join(('1', '2'))
'1 2'

>>> c = ' :: '
>>> c.join(['This', 'is', 'a', 'sample', 'string'])
'This :: is :: a :: sample :: string'

As an exercise, check what happens if you pass multiple string values separated by comma to join() instead of an iterable.

Transliteration

The translate() method accepts a table of codepoints (numerical value of a character) mapped to another character/codepoint or None (if the character has to be deleted). You can use the ord() built-in function to get the codepoint of characters. Or, you can use the str.maketrans() method to generate the mapping for you.

>>> ord('a')
97
>>> ord('A')
65

>>> str.maketrans('aeiou', 'AEIOU')
{97: 65, 101: 69, 105: 73, 111: 79, 117: 85}

>>> greeting = 'have a nice day'
>>> greeting.translate(str.maketrans('aeiou', 'AEIOU'))
'hAvE A nIcE dAy'

The string module has a collection of constants that are often useful in text processing. Here's an example of deleting punctuation characters.

>>> import string
>>> string.punctuation
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

>>> para = '"Hi", there! How *are* you? All fine here.'
>>> para.translate(str.maketrans('', '', string.punctuation))
'Hi there How are you All fine here'

>>> chars_to_delete = ''.join(set(string.punctuation) - set('.!?'))
>>> para.translate(str.maketrans('', '', chars_to_delete))
'Hi there! How are you? All fine here.'

As an exercise, read the documentation for features covered in this section. See also stackoverflow: character translation examples.

Removing leading/trailing characters

The strip() method removes consecutive characters from the start/end of the given string. By default it removes whitespace characters, which you can change by passing a str argument. You can use lstrip() and rstrip() methods to work only on the leading and trailing characters respectively.

>>> greeting = '  \t\r\n have    a  nice \t day  \f\v\r\t\n '
>>> greeting.strip()
'have    a  nice \t day'
>>> greeting.lstrip()
'have    a  nice \t day  \x0c\x0b\r\t\n '
>>> greeting.rstrip()
'  \t\r\n have    a  nice \t day'

>>> '"Hi",'.strip(string.punctuation)
'Hi'

The removeprefix() and removesuffix() methods will delete a substring from the start/end of the input string.

>>> 'spare'.removeprefix('sp')
'are'
>>> 'free'.removesuffix('e')
'fre'

# difference between remove and strip
>>> 'cared'.removesuffix('de')
'cared'
>>> 'cared'.rstrip('de')
'car'

Dealing with case

Here's five different methods for changing the case of characters. Word level transformation is determined by consecutive occurrences of alphabets, not limited to separation by whitespace characters.

>>> sentence = 'thIs iS a saMple StrIng'

>>> sentence.capitalize()
'This is a sample string'

>>> sentence.title()
'This Is A Sample String'

>>> sentence.lower()
'this is a sample string'

>>> sentence.upper()
'THIS IS A SAMPLE STRING'

>>> sentence.swapcase()
'THiS Is A SAmPLE sTRiNG'

The string.capwords() method is similar to title() but also allows a specific word separator (whose default is whitespace).

>>> phrase = 'this-IS-a:colon:separated,PHRASE'

>>> phrase.title()
'This-Is-A:Colon:Separated,Phrase'
>>> string.capwords(phrase, ':')
'This-is-a:Colon:Separated,phrase'

is methods

The islower(), isupper() and istitle() methods check if the given string conforms to the specific case pattern. Characters other than alphabets do not influence the result, but at least one alphabet needs to be present for a True output.

>>> 'αλεπού'.islower()
True

>>> '123'.isupper()
False
>>> 'ABC123'.isupper()
True

>>> 'Today is Sunny'.istitle()
False

Here's some examples with isnumeric() and isascii() methods. As an exercise, read the documentation for the rest of the is methods.

# checks if string has numeric characters only, at least one
>>> '153'.isnumeric()
True
>>> ''.isnumeric()
False
>>> '1.2'.isnumeric()
False
>>> '-1'