Text Mining Python: Analyze Documents

To analyze documents in Python, convert their text into structured features software can inspect. Text mining creates features such as token counts and TF-IDF weights, with TF-IDF giving less weight to terms found in many documents.

I’ll show how to set up a small corpus and represent its documents with counts or TF-IDF.

TL;DR

Text mining in Python turns documents into countable features so you can find repeated terms, compare documents, or prepare text for a later model.

• Use token counts to inspect a small corpus.
• Use TF-IDF when terms should be weighted by how rare they are across documents.
• Keep preprocessing choices tied to the question, since filtering can erase useful signals.

What Is Text Mining in Python?

Text mining converts written documents into structured features that software can inspect. A corpus is the collection of documents being analyzed.

A corpus is the collection of documents being analyzed. In a document-term matrix, rows are documents, columns are terms, and cells hold counts or weights.

Match document boundaries to the question: a review and a report passage are different units.

How to Mine Text in Python Step by Step

This small example keeps two records separate, lowercases text and removes three selected common words.

Step 1: Set up a small corpus

Read files with an explicit encoding such as UTF-8. Keep originals while testing normalization on sample documents.

from collections import Counter
import re

texts = {
    "guide_a": "Python makes text analysis practical. Python counts words in documents.",
    "guide_b": "Text analysis compares documents. Python can compare text with counts.",
}
stop_words = {"a", "in", "with"}

for name, text in texts.items():
    tokens = re.findall(r"[a-z]+", text.lower())
    words = [token for token in tokens if token not in stop_words]
    print(f"{name}: {Counter(words).most_common(3)}")

The first document has two Python tokens, while the second has two text tokens. This inspection uses a stop-word set chosen for these sample sentences.

guide_a: [('python', 2), ('makes', 1), ('text', 1)]
guide_b: [('text', 2), ('analysis', 1), ('compares', 1)]

The sample is too small to measure meaning, and its stop-word set is specific to these sentences.

Step 2: Choose a representation

Choose counts to inspect occurrences and TF-IDF to weight terms by how widely they appear across the corpus.

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

documents = ["python text mining", "python compares documents"]
vectorizer = CountVectorizer()
counts = vectorizer.fit_transform(documents)
tfidf = TfidfVectorizer().fit_transform(documents)
print("shape:", counts.shape, tfidf.shape)
print("terms:", vectorizer.get_feature_names_out())
print("counts:", counts.toarray())

I ran the vectorizer file from the activated environment, and it printed:

shape: (2, 5) (2, 5)
terms: ['compares' 'documents' 'mining' 'python' 'text']
counts: [[0 0 1 1 1]
 [1 1 0 1 0]]

There are two document rows and five vocabulary columns, in the displayed term order.

Scikit-learn stores matrices sparsely. The example densifies a small result for printing, but a large corpus should stay sparse.

Vectorizers can treat adjacent words, such as “text mining”, as one feature. Check the learned vocabulary after setting n-gram and frequency limits.

Raw counts answer how often a token appears, but they do not account for document length. TF-IDF changes the weight using document frequency, so a term found in many documents carries less weight than one concentrated in a few. The choice depends on whether frequency itself or relative distinctiveness answers the question.

A word tokenizer may split punctuation, while a character analyzer can preserve useful fragments in misspelled or short text. For phrase analysis, n-grams retain local word order, but they expand the vocabulary and make the matrix wider.

Before modeling, split labeled data into training and test sets. Fit preprocessing and vectorization on the training set, then transform held-out text with those fitted steps.

Common Errors With Text Mining in Python

Preprocessing and representation choices can change what the analysis preserves. Use the table to diagnose a surprising output before drawing a conclusion.

How Should You Choose and Interpret Text Features?

Check these features before interpreting a result:

• Inspect whether punctuation and accents survive tokenization.
• Check whether negation remains in sentiment text.
• Review sample tokens for domain-specific terms.

Term counts alone do not classify sentiment. A classifier needs labeled examples or a validated lexicon, checked against the domain because technical and movie reviews use words differently.

Named entity recognition marks names and organizations. Part-of-speech tagging labels grammatical roles.

Cosine similarity compares nonnegative feature rows by angle rather than length. A score near one means similar direction. Inspect sample pairs to see which terms drive the match.

For counts, compare raw values when document sizes are similar and proportions when they differ.

A rare product name may look distinctive but be irrelevant. Check top terms against sample documents for source-specific meaning or repeated artifacts.

When comparing TF-IDF scores over time, keep the fitted vectorizer and corpus version fixed so added documents do not change the document frequencies.

A term outside the fitted vocabulary receives no feature. Inspect vocabulary and refit on representative text when terminology changes.

Conclusion: Match the Method to the Question

Choose a representation for the decision it supports, then check sample documents. Keep preprocessing tied to the question.

For the vectorizer API, see the scikit-learn text feature extraction guide and the NLTK text processing chapter. Related AskPython reading: Jaccard similarity and NLP and NLTK Punkt tokenization.

FAQ

These answers cover common choices when starting a text-mining project in Python.