The sigmoid activation function in Python

If you’re learning about neural networks, chances are high that you have come across the term activation function. In neural networks, an activation function decides whether a particular neuron will be activated or not. Activation functions take the weighted summation of the nodes as input and perform some mathematical computation, depending on the activation function, and output a value that decides whether a neuron will be activated or not.

They are many activation functions like

• ReLu
• Softmax
• tanh
• Linear
• Leaky ReLu
• Sigmoid

In this tutorial, we will be learning about the sigmoid activation function. So let’s begin!

---

What is the sigmoid function – The math behind it

Sigmoid is a non-linear activation function. It is mostly used in models where we need to predict the probability of something. As probability exists in the value range of 0 to 1, hence the range of sigmoid is also from 0 to 1, both inclusive.
Let’s have a look at the equation of the sigmoid function.

Sigmoid is usually denoted using the Greek symbol sigma. So, we can also write

In the above equation, ‘e‘ is Euler’s number. Its value is approximately 2.718. Similarly,

In fact, we can derive a relation between the above two equations as

We can also prove this relation as shown below:

LHS:

It can also be written as

RHS:

Therefore, LHS=RHS

Hence, we have proved the relation.

Another property of the sigmoid activation function is that it is differentiable. Let us see how we can differentiate it.

Differentiating sigmoid equation 1 we get

So, from