ReLU — Stopping the negative values

Step by step implementation with its derivative

3 min readDec 1, 2021

In this post, we will talk about the ReLU activation function and the Leaky ReLU activation function. ReLU stands for Rectified Linear Unit. In this function, we don’t use negative values. But sometimes we have to use them, so we use Leaky ReLU to use the negative values in a restricted manner.

You can download the Jupyter Notebook from here.

Back to the previous post

Back to the first post

3.4 What is the ReLU activation function and its derivative? And also Leaky ReLU.

This is the definition of the ReLU function.

And it is very easy to find the derivative of the ReLU function.

This is the definition of the Leaky ReLU function.

And it is very easy to find the derivative of the Leaky ReLU function.

This is the graph for the ReLU and Leaky ReLU functions and their derivatives.

ReLU and Leaky ReLU function and their derivatives graph

Note — Lines for ReLU and Leaky ReLU are overlapping for x > 0 in both graphs

We can easily implement the ReLU and Leaky ReLU functions in Python.

Note — We are implementing ReLU and Leaky ReLU in the same function because when leak = 0, Leaky ReLU is simple ReLU

import numpy as np                             # importing NumPy
np.random.seed(42)def relu(x, leak = 0):                         # ReLU
    return np.where(x <= 0, leak * x, x)def relu_dash(x, leak = 0):                    # ReLU derivative
    return np.where(x <= 0, leak, 1)

Defining ReLU and Leaky ReLU functions and their derivatives

How np.where() works?

in np.where(), we have 3 arguments
First, condition
Second, Do this if the condition is satisfied by a scalar in the array
Third, Do this if the condition is not satisfied by a scalar in the array

Let us have a look at an example with no leak

x = np.array([[0.11], [-2.2], [0], [50.2], [33.5], [-0.6]])
xrelu(x)relu_dash(x)

Another example with leak = 0.1

x = np.array([[-0.5], [0], [1], [2], [3], [4.5]])
xrelu(x, leak = 0.1)relu_dash(x, leak = 0.1)

I hope now you understand how to implement the ReLU and Leaky ReLU functions and their derivatives.

Watch the video on youtube and subscribe to the channel for videos and posts like this.
Every slide is 3 seconds long and without sound. You may pause the video whenever you like.
You may put on some music too if you like.

The video is basically everything in the post only in slides.