Machine Learning Model Evaluation: How Do We Know Our Model Is Good?

Building a Machine Learning model is only half of the work.
The next question is much more important:
How do we know whether our model is actually performing well?

For example, suppose we build a model to predict whether a customer will leave a company.

Our model gives:

Customer → ML Model → Will Leave? → Yes / No

But simply saying that the model is 90% accurate does not always mean that it is a good model.
Imagine we have 10,000 customers:

• 9,500 customers will stay
• 500 customers will leave

If our model predicts “Stay” for everyone, it will achieve:

Accuracy = 9500 / 10000 = 95%

95% sounds excellent.
But the model has completely failed to identify the customers who are actually leaving.
This is why model evaluation metrics are important.
Different problems require different metrics.

What Are Machine Learning Metrics?

A metric is simply a way to measure how well our model is performing.

Different ML problems need different evaluation approaches.

We generally divide them into:

But before understanding these metrics, we need to understand one very important concept:

Confusion Matrix

For binary classification, predictions can fall into four categories.
Suppose we are predicting:
Will a customer churn?

There are two actual possibilities:

• Customer churns
• Customer does not churn

And our model can predict:

• Churn
• No churn

This gives us a Confusion Matrix.

Let’s understand these in simple words.
True Positive (TP)
Model predicted positive and it was actually positive.
Example:

Actual: Churn
Prediction: Churn

→ TP

True Negative (TN)
Model predicted negative and it was actually negative.

Actual: No Churn
Prediction: No Churn

→ TN

False Positive (FP)
Model predicted positive, but the actual value was negative.

Actual: No Churn
Prediction: Churn

→ FP

This is also called a Type I Error.

False Negative (FN)
Model predicted negative, but the actual value was positive.

Actual: Churn
Prediction: No Churn

→ FN

This is also called a Type II Error.
The entire family of classification metrics is built around these four values.

Accuracy
Accuracy answers a very simple question:
Out of all predictions, how many did we get correct?
Formula:

Accuracy = (TP + TN) / (TP + TN + FP + FN)
Suppose:
        TP = 80
        TN = 90
        FP = 10
        FN = 20
Then:
       Accuracy = (80 + 90) / (80 + 90 + 10 + 20)
                = 170 / 200
                = 85%
So our model correctly predicted 85% of the observations.

When should we use Accuracy?
Accuracy works well when:

• Classes are reasonably balanced
• False positives and false negatives have similar importance
  For example:

Class A = 50%
Class B = 50%

But accuracy can be misleading when the dataset is highly imbalanced.

Precision

Precision answers:
When my model predicts positive, how often is it actually positive?
Formula:

Precision = TP / (TP + FP)
For example:
           TP = 80
           FP = 20
Then:
      Precision = 80 / (80 + 20)
                = 80%

So when the model says:
“This customer will churn.”
It is correct 80% of the time.

When is Precision important?
Precision is important when false positives are expensive.
For example:
Spam detection
If our model marks legitimate emails as spam, that can be a problem.
We want:

Predicted Spam → Actually Spam

So we care about precision.

Recall
Recall answers:
Out of all actual positive cases, how many did my model successfully identify?Formula:

Recall = TP / (TP + FN)
Suppose:
       TP = 80
       FN = 20
Then:
     Recall = 80 / (80 + 20)
            = 80%
The model found 80% of all actual positive cases.

When is Recall important?
Recall is especially important when missing a positive case is costly.
For example:

Disease Detection
Fraud Detection
Security Threat Detection

Suppose 100 patients actually have a disease.
If our model identifies only 60:

Recall = 60 / 100 = 60%

Even if accuracy is high, missing 40 patients could be a serious problem.

Precision vs Recall
This is one of the most important concepts in classification.
Think about a medical screening system.

Precision
“When I say someone has the disease, how often am I correct?”
Recall
“Of all people who actually have the disease, how many did I find?”
So:

Precision → Quality of positive predictions
Recall → Coverage of actual positive cases

There is often a trade-off between them.
Increasing the classification threshold can reduce false positives but may also increase false negatives.

F1-Score
Sometimes we don’t want to look at precision or recall separately.
We want one metric that balances both.
That’s where F1-score comes in.
Formula:

F1 = 2 × (Precision × Recall)
        -------------------------
        (Precision + Recall)

For example:
           Precision = 0.80
           Recall    = 0.60
Then:
    F1 ≈ 0.686

F1-score is the harmonic mean of precision and recall.
Why harmonic mean?
Because it penalizes situations where one value is very high and the other is very low.
For example:

Precision = 0.99
Recall    = 0.10

The F1-score will still be low.

When should we use F1?
F1 is useful when:

• Classes are imbalanced
• Both precision and recall matter
• We want a single metric balancing both

Specificity
Specificity answers:

Out of all actual negative cases, how many did our model correctly identify as negative?
Formula:

Specificity = TN / (TN + FP)

It is also called: True Negative Rate (TNR)

Recall is:
         TP / (TP + FN)

Specificity is:
              TN / (TN + FP)

A useful way to remember:

Recall      → How many positives did we find?
Specificity → How many negatives did we correctly reject?

Specificity is particularly useful in medical testing and binary classification problems where false positives matter.

Classification Report

In Python, we often use:

from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))

It typically gives us:

What is Support?
Support tells us: How many actual samples belong to that class?
For example:

Class 0 → 1000 samples
Class 1 → 300 samples

Macro Average
Calculate the metric independently for each class and then take the simple average.
Every class gets equal importance.

Weighted Average
Calculate the metric for each class and weight it according to the number of samples in that class.

ROC Curve
ROC stands for: Receiver Operating Characteristic

A ROC curve shows how the model behaves when we change the classification threshold.

It plots:

True Positive Rate
       vs
False Positive Rate

Where:
 
TPR = TP / (TP + FN)

and:
 
FPR = FP / (FP + TN)

The model does not always have to use:
Probability > 0.5 → Positive

We can change the threshold.
For example:

Threshold = 0.9
Threshold = 0.8
Threshold = 0.7
Threshold = 0.6
...
Threshold = 0.1

Each threshold produces different TP, FP, TN and FN values.
Plotting these values gives us the ROC curve.

ROC-AUC
AUC stands for: Area Under the Curve
ROC-AUC measures how well the model separates positive and negative classes across different thresholds.
Conceptually:

AUC ≈ Probability that the model
      ranks a random positive example
      higher than a random negative example

A rough interpretation is:

AUC = 1.0 → Perfect separation
AUC = 0.5 → Random-like ranking
AUC < 0.5 → Worse than random ranking

However, we should not blindly use AUC as the only metric.
For highly imbalanced datasets, Precision-Recall AUC can sometimes be more informative.

Precision-Recall Curve and PR-AUC
A Precision-Recall curve plots:

Precision vs Recall

It is particularly useful when the positive class is rare.

For example:

Fraud transactions = 1%
Normal transactions = 99%

ROC-AUC can sometimes look strong even when the model’s ability to identify the rare positive class is not very useful.
In such situations, we should examine:

Precision
Recall
F1
PR-AUC

together.

Log Loss
Another classification metric is Log Loss, also called Cross-Entropy Loss.
Instead of only checking whether the final prediction is correct, log loss also looks at the probability assigned by the model.
Imagine two models:

Model A:
Actual class = 1
Predicted probability = 0.51

Model B:
Actual class = 1
Predicted probability = 0.99

Both may classify the example correctly.
But Model B is much more confident.
Log loss captures this difference.
For binary classification:

Log Loss = - [y log(p) + (1-y) log(1-p)]

Lower log loss is better.
It is useful when probability quality matters, not just the final class.

Regression Metrics

Now let’s move from classification to regression.
Suppose our model predicts house prices.

Actual Price → 50 lakh
Predicted    → 47 lakh

The difference is called an error or residual.

Error = Actual - Predicted

Regression metrics measure how large these errors are.
The most common metrics are:

MAE
MSE
RMSE
R²
Adjusted R²
MAPE
RMSLE

MAE — Mean Absolute Error
MAE stands for:Mean Absolute Error

Formula:

MAE = (1/n) Σ |yᵢ - ŷᵢ|

Suppose actual and predicted values are:

Actual:     100   200   300
Predicted:   90   220   280

Errors:

10
20
20

MAE:

MAE = (10 + 20 + 20) / 3

    = 16.67

So, on average, the prediction is about 16.67 units away from the actual value.

Why is MAE easy to understand?
Because it is in the same unit as the target.
If we predict house prices in lakh rupees, MAE is also in lakh rupees.

MSE — Mean Squared Error
MSE squares the errors before taking the average.
Formula:

MSE = (1/n) Σ(yᵢ - ŷᵢ)²

Why square the errors?
Because we want larger errors to receive more penalty.
Example:

Error = 2

Squared error = 4

But: 

Error = 10

Squared error = 100

So MSE heavily penalizes large mistakes.

When should we use MSE?
When large errors should be penalized more strongly.

RMSE — Root Mean Squared Error
RMSE is simply the square root of MSE.
Formula:

RMSE = √[(1/n) Σ(yᵢ - ŷᵢ)²]

Why take the square root?
Because MSE is in squared units.
If our target is:

House Price = ₹ lakh

MSE is: 
      ₹ lakh²

RMSE brings it back to:
      ₹ lakh

Therefore RMSE is easier to interpret.

MAE vs RMSE

Suppose we have:

MAE  = 5
RMSE = 12

The difference tells us that some larger errors are affecting the model.
A useful rule:

MAE  → Treats errors more equally

RMSE → Penalizes large errors more strongly

R² — R-Squared
R² is called the: Coefficient of Determination

It tries to answer:

How much of the variation in the target variable is explained by the model?
Formula:

R² = 1 - SS_res / SS_tot

Where:
     SS_res = Σ(yᵢ - ŷᵢ)²

     SS_tot = Σ(yᵢ - ȳ)²
   
For example:
   R² = 0.80

We can say that the model explains about 80% of the variation in the target relative to the baseline represented by the mean, under the usual R² interpretation.

But don’t interpret R² as: “The model is 80% accurate.”

That’s incorrect.

R² and accuracy measure different things.

Adjusted R²
There is one problem with R².
If we keep adding features to a regression model, R² generally does not decrease, even if the new feature is not useful.
For example:

Model 1:
Age + Income

R² = 0.75

Then we add:

Age + Income + Random_Number

R² might increase slightly.
But the random feature doesn’t actually help.
Adjusted R² addresses this by penalizing unnecessary predictors.
A common formula is:

Adjusted R² =  1 - [(1 - R²)(n - 1) / (n - p - 1)]

where:
     n = number of observations
     p = number of predictors

R² vs Adjusted R²

R²
→ How much variation is explained?

Adjusted R²
→ How much variation is explained while accounting
  for the number of predictors?

MAPE
MAPE stands for: Mean Absolute Percentage Error

Formula:

MAPE = (100/n) Σ |(yᵢ - ŷᵢ) / yᵢ|

It expresses error as a percentage.
For example:

Actual sales = 100
Prediction   = 90

Error = 10%

Problem with MAPE
MAPE can behave badly when actual values are zero or very close to zero.
Therefore, don’t blindly use MAPE for every regression problem.

Which Regression Metric Should We Use?
A simple decision guide:

Need easy interpretation?
        ↓
       MAE

Large errors should be heavily punished?
        ↓
       RMSE
Want squared-error based optimization?
        ↓
       MSE
Want variance explained?
        ↓
        R²
Comparing regression models with different
numbers of predictors?
        ↓
   Adjusted R²
Need percentage error?
        ↓
       MAPE

There is no universal “best” metric. The correct metric depends on the business problem.

Get Dhirendra Jha’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Training Error vs Validation Error
Now we move from metrics into one of the most important concepts in Machine Learning: Is our model actually learning, or is it simply memorizing the training data?

This brings us to:

Overfitting
Underfitting
Bias
Variance

Overfitting
Suppose we give a student 100 questions before an exam.The student memorizes the exact answers.
On those 100 questions:

Score = 100%

But when we give new questions:

Score = 55%

The student memorized instead of learning the underlying concept.This is similar to overfitting.
In Machine Learning:

Training Performance → Very good

Validation/Test Performance → Poor

The model has learned the training data too specifically.
Example:

Training Accuracy = 99%
Validation Accuracy = 72%

This can be a sign of overfitting.

Underfitting
Underfitting is almost the opposite.The model is too simple to capture the underlying relationship.
For example:

Training Accuracy   = 65%
Validation Accuracy = 63%

Both are poor.
The model hasn’t learned enough.
Common causes include:

• Model too simple
• Insufficient features
• Excessive regularization
• Poor feature engineering
• Insufficient training

Bias-Variance Tradeoff
This is one of the classic concepts in Machine Learning. Think of the model as trying to learn the underlying pattern.

High Bias
The model makes strong simplifying assumptions.

Model too simple
        ↓
Misses important patterns
        ↓
Underfitting

High Variance
The model is highly sensitive to the training data.

Model too complex
        ↓
Learns noise
        ↓
Overfitting

So we want a balance:

High Bias                    High Variance
    │                            │
    ↓                            ↓
Underfitting                Overfitting

              ↓
       Good Generalization

The goal is not to make training performance perfect. The goal is to generalize well to unseen data.

Train, Validation and Test Dataset
A common ML workflow is:

Original Dataset
       │
       ├───────────────┐
       │               │
    Training       Test Set
       │
       ├───────────────┐
       │               │
   Training       Validation

For example:

70% → Training
15% → Validation
15% → Test

Training Set: Used to learn model parameters.
Validation Set
Used to:

• Select models
• Tune hyperparameters
• Choose thresholds
• Compare configurations

Test Set
Used at the end to estimate performance on unseen data.
The test set should not become another tuning dataset.

Cross-Validation
Sometimes we don’t want to rely on just one train-validation split.We can use K-Fold Cross-Validation.

Suppose:

K = 5

We divide our dataset into five parts.

Fold 1
Fold 2
Fold 3
Fold 4
Fold 5

Then:

Round 1:
Validation = Fold 1
Training   = Fold 2,3,4,5

Round 2:
Validation = Fold 2
Training   = Fold 1,3,4,5
...
Round 5:
Validation = Fold 5
Training   = Fold 1,2,3,4

Finally, we calculate the average score. This gives us a more stable estimate of model performance.

Data Leakage
One of the most dangerous problems in Machine Learning is: Data Leakage

Data leakage happens when information that should not be available during training accidentally enters the model-building process.

For example, suppose we’re predicting: Whether a customer will default on a loan.
And we accidentally include:

Loan_Recovery_Status

which is only known after the customer defaults. The model may achieve extremely high performance. But in the real world, that information is not available when we need to make the prediction.
That’s leakage. A model with leakage can look excellent during evaluation and still fail badly in production.

Feature Scaling
Suppose we have two features:

Age       → 20 to 70
Salary    → 20,000 to 500,000

The features have very different numerical scales. Some ML algorithms can be affected by this.
This brings us to:

Standardization
Normalization

Standardization
Standardization transforms data so that it is centered around zero with a standard deviation of approximately one.
The common Z-score formula is:

z = (x - μ) / σ

where:

x = original value
μ = mean
σ = standard deviation

For example:

Mean = 50
Standard deviation = 10
Value = 70

Then

z = (70 - 50) / 10 = 2

The value is two standard deviations above the mean.

When is standardization useful?
It is particularly important for algorithms that depend on:

• Distance
• Gradient optimization
• Feature magnitude

Examples include:

Logistic Regression
Linear Regression with regularization
SVM
KNN
K-Means
Neural Networks
PCA

Tree-based algorithms such as:

Decision Tree
Random Forest
XGBoost

generally do not require feature scaling in the same way because their split decisions are based on feature thresholds.

Normalization
Normalization commonly refers to scaling values into a fixed range, often:

0 to 1

A common Min-Max formula is:

x' = (x - min(x)) / (max(x) - min(x))

Suppose:

Minimum = 10
Maximum = 100
Value = 55

Then:
x' = (55 - 10) / (100 - 10)

= 45 / 90 = 0.5

So 55 becomes:

0.5

Standardization vs Normalization

Standardization
→ Mean ≈ 0
→ Standard deviation ≈ 1

Normalization
→ Often scales values to a fixed range
→ Commonly 0 to 1

The exact preprocessing choice should depend on the algorithm and data distribution.

Hyperparameters vs Parameters

This is another concept that often confuses beginners.

Parameters
Parameters are learned by the model during training.
For linear regression:

y = b₀ + b₁x

The model learns:

These are parameters.

b₀
b₁

Hyperparameters
Hyperparameters are settings we choose before or around training.
For Random Forest:

n_estimators
max_depth
min_samples_split
max_features

For KNN:

n_neighbors

For SVM:

C
gamma
kernel

The model doesn’t automatically learn these in ordinary training.We tune them.

Hyperparameter Tuning
Suppose our Random Forest model has:

max_depth = 5

Maybe:

max_depth = 10

works better. We can test different combinations. Common approaches include:

Grid Search
Try every combination from a predefined grid.

max_depth = [5, 10, 20]
n_estimators = [100, 200]

Total combinations = 3 × 2 = 6

Random Search

Randomly sample combinations from a defined search space. This can be much more efficient when there are many hyperparameters.

Bayesian Optimization
Uses previous evaluation results to decide which configurations are worth testing next.

Regularization
Regularization is a technique used to reduce overfitting.

The basic idea is: Don’t allow the model to become unnecessarily complex.

Two famous types are:

L1 Regularization
L2 Regularization

L1 — Lasso

Adds the absolute value of coefficients to the loss.

Loss + λ Σ|w|

L1 can push some coefficients exactly toward zero. Therefore, it can also perform a kind of feature selection.

L2 — Ridge

Adds squared coefficients:

Loss + λ Σw²

L2 generally shrinks coefficients toward zero without typically making them exactly zero.

Class Imbalance

Suppose we have:

Normal transactions = 99,000
Fraud transactions  = 1,000

Fraud is only: 1%

This is an imbalanced dataset. A model that predicts:

Everything = Normal

gets:

Accuracy = 99%

But:

Fraud Reacll  = 0%

So accuracy alone is misleading. For imbalanced classification, we should examine metrics such as:

Precision
Recall
F1-score
PR-AUC
Specificity
Balanced Accuracy
MCC

depending on the business objective.

Balanced Accuracy
Balanced accuracy is useful when classes are imbalanced. For binary classification, it can be expressed as:

Balanced Accuracy = (Recall + Specificity) / 2

This gives equal importance to the positive and negative classes. It can therefore be more informative than ordinary accuracy for imbalanced datasets.

Curse of Dimensionality

Now let’s talk about another famous Machine Learning concept: Curse of Dimensionality

Imagine we have:

2 features

We can visualize them easily.

X → Width
Y → Height

Now imagine: 100 features

We can’t visualize the data easily anymore. But the problem is deeper than visualization. As the number of dimensions increases, the data becomes increasingly sparse.
For example:

2D
→ Data points can be relatively close

10D
→ Space becomes much larger
100D
→ Data can become extremely sparse

This can negatively affect distance-based algorithms.
Examples:

KNN
K-Means
Some similarity/distance-based methods

It can also increase computational requirements and make it harder to identify useful signals.

How Do We Handle High Dimensionality?
Common techniques include:

Feature Selection
        ↓
Remove unnecessary features

Dimensionality Reduction
        ↓
PCA
        ↓
Reduce dimensions while retaining important variation
Regularization
        ↓
Control model complexity
Domain Knowledge
        ↓
Keep meaningful features

For example:

1000 Features
      ↓
Feature Selection
      ↓
200 Features
      ↓
PCA
      ↓
50 Components
      ↓
Model

PCA — Principal Component Analysis

PCA is a dimensionality-reduction technique. Instead of keeping all original features, PCA creates new variables called: Principal Components

These components capture directions of high variance in the data.
Very roughly:

Original Features
       ↓
      PCA
       ↓
Principal Components
       ↓
Reduced Representation

The first principal component captures the largest amount of variance possible, subject to the PCA constraints. The second captures the next largest amount while being orthogonal to the first, and so on.

Instead of keeping all original features, PCA creates new features called Principal Components (PCs).

How PCA Works

1. Standardize the data — especially when features have different scales.
2. Calculate the covariance matrix.
3. Calculate its eigenvectors and eigenvalues.
4. Eigenvectors give the directions of the principal components.
5. Eigenvalues tell us how much variance each component captures.
6. Select the top components and project the data onto them.

Remember: PCA doesn’t simply remove random features. It creates new dimensions that capture the maximum possible variance from the original data.

Feature Selection vs Feature Extraction
These concepts are related but different.

Feature Selection
Choose a subset of the original features.

100 Features
      ↓
Select 20
      ↓
Model

The original features remain.

Feature Extraction
Create new features from the original features.
PCA is an example.

100 Original Features
        ↓
       PCA
        ↓
20 New Components

Don’t Depend on One Metric

This is probably the most important lesson from this entire article. Suppose we build a fraud detection model.
We get:

Accuracy = 99%

Should we celebrate?
Not yet.
We should ask:

What is Precision?
What is Recall?
What is F1?
What is PR-AUC?
How many fraud cases were missed?
How many normal transactions were incorrectly flagged?

Similarly, for regression:

R² = 0.90

sounds good.
But we should also ask:

What is MAE?
What is RMSE?
Are there large outliers?
How does the model perform on unseen data?

Metrics should be selected based on the problem, data distribution, cost of errors, and business objective.

A Complete Machine Learning Evaluation Workflow
A practical workflow can look like this:

             Dataset
                │
                ↓
        Train / Validation / Test
                │
                ↓
         Data Preprocessing
                │
       ┌────────┴─────────┐
       │                  │
 Scaling              Feature Engineering
       │                  │
       └────────┬─────────┘
                ↓
          Model Training
                │
                ↓
       Cross Validation
                │
                ↓
      Hyperparameter Tuning
                │
                ↓
        Validation Metrics
                │
                ↓
       Threshold Selection
       (Classification)
                │
                ↓
          Final Model
                │
                ↓
           Test Set
                │
                ↓
       Final Evaluation
                │
                ↓
          Deployment
                │
                ↓
       Monitor Performance

And after deployment, the job is still not finished.
We should monitor:

Prediction Performance
Data Drift
Concept Drift
Latency
Error Rates
Business KPIs

because real-world data changes over time. The most important thing is not memorizing every formula.
It is understanding what question each metric answers. Once you know the question, choosing the metric becomes much easier.

“A good ML model is not the model with the highest score on one metric. It is a model whose evaluation matches the real problem it is solving.”

Happy Learning! 🚀