Adagrad — Storing squares of gradients

Step by step implementation with animation for better understanding.

6 min readNov 26, 2021

2.4 How does Adagrad works?

Adagrad stands for Adaptive Gradient. The idea is to store the square of gradients in an accumulator. The value of the accumulator is generally initialized as 0.1

And we calculate update as follow:

You can download the Jupyter Notebook from here.

Note — It is recommended that you have a look at the first post in this chapter.

This post is divided into 3 sections.

Adagrad in 1 variable
Adagrad animation for 1 variable
Adagrad in multi-variable function

Adagrad in 1 variable

In this method, we store the square of gradients in an accumulator which is initialized as 0.1

Adagrad algorithm in simple language is as follows:

Step 1 - Set starting point and learning rate
Step 2 - Initiate accumulator = 0.1 and set epsilon = 10**-8
Step 3 - Initiate loop
         Step 3.1 - add square of gradients to accumulator
         Step 3.2 - calculate update as stated above
         Step 3.3 - add update to point

First, let us define the function and its derivative and we start from x = -1

import numpy as np
np.random.seed(42)def f(x):                           # function definition                        
    return x - x**3def fdash(x):                       # function derivative definition
    return 1 - 3*(x**2)

Defining the function and its derivative

And now Adagrad

point = -1                                   # step 1
learning_rate = 0.05accumulator = 0.1                            # step 2
epsilon = 10**-8for i in range(1000):                        # step 3
    accumulator += fdash(point)**2           # step 3.1
    update = - learning_rate * fdash(point) / (accumulator**0.5 +
                                                       epsilon)
                                             # step 3.2
    point += update                          # step 3.3
    
point                                        # Minima

Note — Because we are storing the square of gradients in the accumulator, it gets big and big with time which slows the learning. So, the learning rate is slightly high.

And, we have successfully implemented Adagrad in Python

Adagrad animation for better understanding

Everything thing is the same as what we did earlier for the animation of the previous 3 optimizers. We will create a list to store starting point and updated points in it and will use the iᵗʰ index value for iᵗʰ frame of the animation.

import matplotlib.pyplot as plt 
import matplotlib.animation as animation 
from matplotlib.animation import PillowWriterpoint_adagrad = [-1]                      # initiating list with
                                          # starting point in itpoint = -1                                # step 1
learning_rate = 0.05accumulator = 0.1                         # step 2
epsilon = 10**-8for i in range(1000):                     # step 3
    accumulator += fdash(point)**2        # step 3.1
    update = - learning_rate * fdash(point) / (accumulator**0.5 +
                                                     epsilon)
                                          # step 3.2
    point += update                       # step 3.3
    
    point_adagrad.append(point)           # adding updated point to
                                          # the list
    
point                                     # Minima

Importing libraries and creating a list that has the starting point and updated points in the list

We will do some settings for our graph for the animation. You can change them if you want something different.

plt.rcParams.update({'font.size': 22})fig = plt.figure(dpi = 100)fig.set_figheight(10.80)
fig.set_figwidth(19.20)x_ = np.linspace(-5, 5, 10000)
y_ = f(x_)ax = plt.axes()
ax.plot(x_, y_)
ax.grid(alpha = 0.5)
ax.set_xlim(-5, 5)
ax.set_ylim(-5, 5)
ax.set_xlabel('x')
ax.set_ylabel('y', rotation = 0)
ax.scatter(-1, f(-1), color = 'red')
ax.hlines(f(-0.5773502691896256), -5, 5, linestyles = 'dashed', alpha = 0.5)ax.set_title('Adagrad, learning_rate = 0.05')

Few settings for our graph in the animation

Now we will animate the Adagrad optimizer.

def animate(i):
    ax.clear()
    ax.plot(x_, y_)
    ax.grid(alpha = 0.5)
    ax.set_xlim(-5, 5)
    ax.set_ylim(-5, 5)
    ax.set_xlabel('x')
    ax.set_ylabel('y', rotation = 0)
    ax.hlines(f(-0.5773502691896256), -5, 5, linestyles = 'dashed', alpha = 0.5)
    ax.set_title('Adagrad, learning_rate = 0.05')
    
    ax.scatter(point_adagrad[i], f(point_adagrad[i]), color = 'red')

The last line in the code snippet above is using the iᵗʰ index value from the list for iᵗʰ frame in the animation.

anim = animation.FuncAnimation(fig, animate, frames = 200, interval = 20)anim.save('2.4 Adagrad.gif')

We are creating an animation that only has 200 frames and the gif is at 50 fps or frame interval is 20 ms.

It is to be noted that in less than 200 iterations we have reached the minima.

Adagrad in multi-variable function (2 variables right now)

Everything is the same, we only have to initialize point (1, 0) and accumulator = 0.1 but with shape (2, 1) and replace fdash(point) with gradient(point).

But first, let us define the function, its partial derivatives and, gradient array

We know that Minima for this function is at (2, -1)
and we will start from (1, 0)

The partial derivatives are

def f(x, y):                                    # function
    return 2*(x**2) + 2*x*y + 2*(y**2) - 6*x    # definitiondef fdash_x(x, y):                              # partial derivative
    return 4*x + 2*y - 6                        # w.r.t xdef fdash_y(x, y):                              # partial derivative
    return 2*x + 4*y                            # w.r.t ydef gradient(point):
    return np.array([[     fdash_x(point[0][0], point[1][0])     ],
                     [     fdash_y(point[0][0], point[1][0])     ]], dtype = np.float64)                             # gradients

Defining the function, its partial derivatives, and gradient array

Now the steps for Adagrad in 2 variables are

point = np.array([[   1   ],                      # step 1
                  [   0   ]], dtype = np.float64)learning_rate = 0.05accumulator = np.array([[   0.1   ],              # step 2
                        [   0.1   ]], dtype = np.float64)
epsilon = 10**-8for i in range(1000):                             # step 3
    accumulator += gradient(point)**2             # step 3.1
    update = - learning_rate * gradient(point) / (accumulator**0.5 +
                                                         epsilon)
                                                  # step 3.2
   point += update                                # step 3.3
    
point                                             # Minima

Steps involved in Adagrad for 2 variable function

I hope now you understand Adagrad.

Watch the video on youtube and subscribe to the channel for videos and posts like this.
Every slide is 3 seconds long and without sound. You may pause the video whenever you like.
You may put on some music too if you like.

The video is basically everything in the post only in slides.