Tensors, Tensors, Tensors. Building blocks of a Neural Network.

Understanding how they are built.

neuralthreads

9 min readNov 23, 2021

Back to the previous post.

1.1 What are Tensors? Tensors as NumPy arrays.

You can download the Jupyter Notebook from here.

Let us start the course with the very first question, what are Tensors?

This post is divided into 4sections.

Understanding the built of Tensors
How to access a scalar and smaller dimension Tensor
Understanding the built of Tensors from right to left
Concept of Axis

Understanding the built of Tensors

In a very simple language ‘Tensors’ are collections. But, to understand their built, let us first see ‘Scalars’. First, let us import NumPy in Python.

import numpy as np
np.random.seed(42)

importing NumPy library

Scalars
Scalars are simply numbers. For example, 5 is a scalar and so are 0.3 and -0.2
We can easily create a scalar in Python like this:

x = 5
print(x)type(x)

The type of variable ‘x’ will either be int, float, or complex depending on the number assigned to it.

Moving Forward.

1-D Tensor or a Vector
1-D Tensor or a vector is a collection of Scalars.

We can easily create a 1-D Tensor or a Vector in Python like this:

x = np.array([5, 0.3, -0.2])   
print(x)type(x)

To create a NumPy array, we store np.array(elements) in the variable ‘x’, and type ‘x’ is showing us that ‘x’ is NumPy array or a Tensor for us.

len(x.shape)x.shape

The length of the shape tuple is telling us that ‘x’ is a 1-D Tensor.

And, the shape tuple is telling us that in this 1-D Tensor, we have 3 scalar elements in it.

We can see that 3 scalar elements are packed together between square brackets.

And one important thing to notice is that these scalar elements are in a horizontal manner.

Now with the same reasoning,

2-D Tensor or a Matrix
2-D Tensor or a Matrix is a collection of 1-D Tensors.

We can easily create a 2-D Tensor or a Matrix in Python like this:

x = np.array([[5, 0.3, -0.2], [4, 3, 5], [0.1, 2, -0.56], [1, 1.2,
                                                             1.23]])
print(x)type(x)

len(x.shape)x.shape

The length of the shape tuple is telling us that ‘x’ is a 2-D Tensor or a Matrix.

And, in this 2-D Tensor, we have 4 1-D Tensors, and in every 1-D Tensor, there are 3 scalar elements.

We can see that 4 1-D Tensors are packed together in between square brackets and each one has 3 scalar elements in it.

One important thing to notice is that every element of this 2-D Tensor, that is, every 1-D Tensor is in a vertical manner.

Moving Forward,

3-D Tensor
3-d Tensor is a collection of 2-D Tensors.

We can easily create a 3-D Tensor in Python like this:

x = np.array([[[5, 0.3, -0.2], [4, 3, 5], [0.1, 2, -0.56], [1, 1.2, 1.23]], [[1, 2, 3], [0.1, 0.2, 0.3], [4, 5, 6], [0.4, 0.5, 0.6]]])print(x)

len(x.shape)x.shape

The length of the shape tuple is telling us that ‘x’ is a 3-D Tensor.

And the shape tuple is telling us that in this 3-D Tensor we have 2 2-D Tensors, each 2-D Tensor has 4 1-D Tensors and each 1-D Tensor has 3 scalar elements in it.

We can see that in this 3-D Tensor, 2 2-D Tensors are packed together in between square brackets.

And each 2-D Tensor has 4 1-D Tensors and each 1-D Tensor has 3 scalar elements in it.

We can also see that the elements of this 3-D Tensor, that is, the 2-D Tensors are in a vertical manner.

Similarly,

4-D Tensor
4-D Tensor is a collection of 3-D Tensors.

We can easily create a 4-D Tensor in Python like this:

x = np.array([[[[5, 0.3, -0.2], [4, 3, 5], [0.1, 2, -0.56], [1, 1.2, 1.23]], [[1, 2, 3], [0.1, 0.2, 0.3], [4, 5, 6], [0.4, 0.5, 0.6]]], [[[5, 0.3, -0.2], [4, 3, 5], [0.1, 2, -0.56], [1, 1.2, 1.23]], [[1, 2, 3], [0.1, 0.2, 0.3], [4, 5, 6], [0.4, 0.5, 0.6]]]])
print(x)type(x)

len(x.shape)x.shape

The length of the shape tuple is telling us that ‘x’ is a 4-D Tensor.

And, the shape tuple is telling us that in this 4-D Tensor, there are 2 3-D Tensors, in each 3-D Tensor, there are 2 2-D Tensors, in each 2-D Tensor, there are 4 1-D Tensor and in each 1-D Tensor, there are 3 scalar elements.

We can see that 2 3-D Tensors are packed together in between square brackets.

And each 3-D Tensor has 2 2-D Tensors, each 2-D Tensor has 4 1-D Tensors and each 1-D Tensor has 3 scalar elements.

We can see that the elements of this 4-D Tensor, that is, the 3-D Tensors are in a vertical manner.

Going on like this,

5-D Tensor is a collection of 4-D Tensors
10-D Tensor is a collection of 9-D Tensors
and so on…

How to access a scalar element or a smaller dimension tensor?

First, let us create a 4-D Tensor for demonstration.

x = np.array([[[[5, 0.3, -0.2], [4, 3, 5], [0.1, 2, -0.56], [1, 1.2, 1.23]], [[1, 2, 3], [0.1, 0.2, 0.3], [4, 5, 6], [0.4, 0.5, 0.6]]], [[[5, 0.3, -0.2], [4, 3, 5], [0.1, 2, -0.56], [1, 1.2, 1.23]], [[1, 2, 3], [0.1, 0.2, 0.3], [4, 5, 6], [0.4, 0.5, 0.6]]]])
print(x)type(x)len(x.shape)x.shape

To access a scalar element of a Tensor, you have to provide indexes equal to the number of dimensions of the Tensor.

In this case, it is 4.

x[0][0][2][1]type(x[0][0][2][1])

x[0][0][2][1] means the second scalar element from the third vector in the second matrix of the first 3-D Tensor.

If the indexes are less than the number of dimensions, then we will be selecting a smaller dimension tensor.

x[0][1]len(x[0][1].shape)x[0][1].shapetype(x[0][1])

Here, we are selecting the second matrix from the first 3-D Tensor.

Note — The two 3-D Tensors are identical, don’t get confused.

Let us take one final example.

x = np.random.randint(0, 2, (2, 3, 4, 5, 6, 5, 6)) 
print(x)type(x)len(x.shape)x.shape

We are generating a NumPy array of a given shape of integers from 0 to 2(excluding).

The length of the shape tuple is telling ‘x’ is a 7-D Tensor.
And in this tensor, there are 2 6-D tensors, in each 6-D tensors, there are 3 5-D tensors, in each 5-D tensor, there are 4 4-D tensors, in each 4-D tensors, there are 5 3-D tensors, in each 3-D tensor, there are 6 2-D tensors, in each 2-D tensor, there are 5 1-D tensors, and in each 1-D tensor, there are 6 scalar elements.

Understanding the built of Tensors from right to left

We have understood the NumPy array or Tensor built from left to right. Now, let us understand from right to left.

First, we create a 3-D tensor.

x = np.array([[[1, 2], [3, 4], [5, 6]], [[7, 8], [9, 10], [11, 12]], [[13, 14], [15, 16], [17, 18]], [[19, 20], [21, 22], [23, 24]]])
xlen(x.shape)x.shape

At first, we have a scalar 1

Then we pack 2 scalars together to get this

Then we pack 3 such vectors to get this

Then we pack 4 such matrices to get our tensor.

I hope now you understand the Tensor building from left to right and right to left.

Concept of Axis

We will use the same 3-D tensor created above.

The number of Axis we have is equal to the dimension of the Tensor. In this case, we have a total of 3 Axis(es). Axis starts from 0.

Axis 0

Axis 0 refers to the first element in the shape tuple, which is actually all the N-1 D tensors in the N-D tensor.

Axis 1

Axis 1 refers to the second element in the shape tuple, which is actually all the N-2 D tensors in all N-1 D tensors.

Axis 2

Axis 2 refers to the third element in the shape tuple, which is actually all the N-3 D tensors in all N-2 D tensors.
In this case, it is all the scalars in all of the vectors.

Note — Axis = -1 can also be used for this case because -1 represents the last element in the tuple. In this case, it happens to be the third entry.

A small comparison

This is a small comparison between three tensors.

First, (4, 1)

x1 = np.array([[1], [2], [3], [4]])
print(x1)x1.shape

Second, (4,)

x2 = np.array([1, 2, 3, 4])
print(x2)x2.shape

Third, (1, 4)

x3 = np.array([[1, 2, 3, 4]])
print(x3)x3.shape

I hope now you understand the difference between (4, 1), (4, ) and, (1, 4) and how Tensors are built.

If you are looking for Images as NumPy Tensors, then you have to wait till Chapter 6 — CNNs.

The goal of this chapter is to help you understand how we will build the Neural Network.

Watch the video on youtube and subscribe to the channel for videos and posts like this.
Every slide is 3 seconds long and without sound. You may pause the video whenever you like.
You may put on some music too if you like.

The video is basically everything in the post only in slides.