Tensors, Tensors, Tensors. Building blocks of a Neural Network.
Understanding how they are built.
1.1 What are Tensors? Tensors as NumPy arrays.
You can download the Jupyter Notebook from here.
Let us start the course with the very first question, what are Tensors?
This post is divided into 4sections.
- Understanding the built of Tensors
- How to access a scalar and smaller dimension Tensor
- Understanding the built of Tensors from right to left
- Concept of Axis
Understanding the built of Tensors
In a very simple language ‘Tensors’ are collections. But, to understand their built, let us first see ‘Scalars’. First, let us import NumPy in Python.
import numpy as np
np.random.seed(42)
Scalars
Scalars are simply numbers. For example, 5 is a scalar and so are 0.3 and -0.2
We can easily create a scalar in Python like this:
x = 5
print(x)type(x)
The type of variable ‘x’ will either be int, float, or complex depending on the number assigned to it.
Moving Forward.
1-D Tensor or a Vector
1-D Tensor or a vector is a collection of Scalars.
We can easily create a 1-D Tensor or a Vector in Python like this:
x = np.array([5, 0.3, -0.2])
print(x)type(x)
To create a NumPy array, we store np.array(elements) in the variable ‘x’, and type ‘x’ is showing us that ‘x’ is NumPy array or a Tensor for us.
len(x.shape)x.shape
The length of the shape tuple is telling us that ‘x’ is a 1-D Tensor.
And, the shape tuple is telling us that in this 1-D Tensor, we have 3 scalar elements in it.
We can see that 3 scalar elements are packed together between square brackets.
And one important thing to notice is that these scalar elements are in a horizontal manner.
Now with the same reasoning,
2-D Tensor or a Matrix
2-D Tensor or a Matrix is a collection of 1-D Tensors.
We can easily create a 2-D Tensor or a Matrix in Python like this:
x = np.array([[5, 0.3, -0.2], [4, 3, 5], [0.1, 2, -0.56], [1, 1.2,
1.23]])
print(x)type(x)
len(x.shape)x.shape
The length of the shape tuple is telling us that ‘x’ is a 2-D Tensor or a Matrix.
And, in this 2-D Tensor, we have 4 1-D Tensors, and in every 1-D Tensor, there are 3 scalar elements.
We can see that 4 1-D Tensors are packed together in between square brackets and each one has 3 scalar elements in it.
One important thing to notice is that every element of this 2-D Tensor, that is, every 1-D Tensor is in a vertical manner.
Moving Forward,
3-D Tensor
3-d Tensor is a collection of 2-D Tensors.
We can easily create a 3-D Tensor in Python like this:
x = np.array([[[5, 0.3, -0.2], [4, 3, 5], [0.1, 2, -0.56], [1, 1.2, 1.23]], [[1, 2, 3], [0.1, 0.2, 0.3], [4, 5, 6], [0.4, 0.5, 0.6]]])print(x)
len(x.shape)x.shape
The length of the shape tuple is telling us that ‘x’ is a 3-D Tensor.
And the shape tuple is telling us that in this 3-D Tensor we have 2 2-D Tensors, each 2-D Tensor has 4 1-D Tensors and each 1-D Tensor has 3 scalar elements in it.
We can see that in this 3-D Tensor, 2 2-D Tensors are packed together in between square brackets.
And each 2-D Tensor has 4 1-D Tensors and each 1-D Tensor has 3 scalar elements in it.
We can also see that the elements of this 3-D Tensor, that is, the 2-D Tensors are in a vertical manner.
Similarly,
4-D Tensor
4-D Tensor is a collection of 3-D Tensors.
We can easily create a 4-D Tensor in Python like this:
x = np.array([[[[5, 0.3, -0.2], [4, 3, 5], [0.1, 2, -0.56], [1, 1.2, 1.23]], [[1, 2, 3], [0.1, 0.2, 0.3], [4, 5, 6], [0.4, 0.5, 0.6]]], [[[5, 0.3, -0.2], [4, 3, 5], [0.1, 2, -0.56], [1, 1.2, 1.23]], [[1, 2, 3], [0.1, 0.2, 0.3], [4, 5, 6], [0.4, 0.5, 0.6]]]])
print(x)type(x)
len(x.shape)x.shape
The length of the shape tuple is telling us that ‘x’ is a 4-D Tensor.
And, the shape tuple is telling us that in this 4-D Tensor, there are 2 3-D Tensors, in each 3-D Tensor, there are 2 2-D Tensors, in each 2-D Tensor, there are 4 1-D Tensor and in each 1-D Tensor, there are 3 scalar elements.
We can see that 2 3-D Tensors are packed together in between square brackets.
And each 3-D Tensor has 2 2-D Tensors, each 2-D Tensor has 4 1-D Tensors and each 1-D Tensor has 3 scalar elements.
We can see that the elements of this 4-D Tensor, that is, the 3-D Tensors are in a vertical manner.
Going on like this,
5-D Tensor is a collection of 4-D Tensors
10-D Tensor is a collection of 9-D Tensors
and so on…
How to access a scalar element or a smaller dimension tensor?
First, let us create a 4-D Tensor for demonstration.
x = np.array([[[[5, 0.3, -0.2], [4, 3, 5], [0.1, 2, -0.56], [1, 1.2, 1.23]], [[1, 2, 3], [0.1, 0.2, 0.3], [4, 5, 6], [0.4, 0.5, 0.6]]], [[[5, 0.3, -0.2], [4, 3, 5], [0.1, 2, -0.56], [1, 1.2, 1.23]], [[1, 2, 3], [0.1, 0.2, 0.3], [4, 5, 6], [0.4, 0.5, 0.6]]]])
print(x)type(x)len(x.shape)x.shape
To access a scalar element of a Tensor, you have to provide indexes equal to the number of dimensions of the Tensor.
In this case, it is 4.
x[0][0][2][1]type(x[0][0][2][1])
x[0][0][2][1] means the second scalar element from the third vector in the second matrix of the first 3-D Tensor.
If the indexes are less than the number of dimensions, then we will be selecting a smaller dimension tensor.
x[0][1]len(x[0][1].shape)x[0][1].shapetype(x[0][1])
Here, we are selecting the second matrix from the first 3-D Tensor.
Note — The two 3-D Tensors are identical, don’t get confused.
Let us take one final example.
x = np.random.randint(0, 2, (2, 3, 4, 5, 6, 5, 6))
print(x)type(x)len(x.shape)x.shape
We are generating a NumPy array of a given shape of integers from 0 to 2(excluding).
The length of the shape tuple is telling ‘x’ is a 7-D Tensor.
And in this tensor, there are 2 6-D tensors, in each 6-D tensors, there are 3 5-D tensors, in each 5-D tensor, there are 4 4-D tensors, in each 4-D tensors, there are 5 3-D tensors, in each 3-D tensor, there are 6 2-D tensors, in each 2-D tensor, there are 5 1-D tensors, and in each 1-D tensor, there are 6 scalar elements.
Understanding the built of Tensors from right to left
We have understood the NumPy array or Tensor built from left to right. Now, let us understand from right to left.
First, we create a 3-D tensor.
x = np.array([[[1, 2], [3, 4], [5, 6]], [[7, 8], [9, 10], [11, 12]], [[13, 14], [15, 16], [17, 18]], [[19, 20], [21, 22], [23, 24]]])
xlen(x.shape)x.shape
At first, we have a scalar 1
Then we pack 2 scalars together to get this
Then we pack 3 such vectors to get this
Then we pack 4 such matrices to get our tensor.
I hope now you understand the Tensor building from left to right and right to left.
Concept of Axis
We will use the same 3-D tensor created above.
The number of Axis we have is equal to the dimension of the Tensor. In this case, we have a total of 3 Axis(es). Axis starts from 0.
Axis 0
Axis 0 refers to the first element in the shape tuple, which is actually all the N-1 D tensors in the N-D tensor.
Axis 1
Axis 1 refers to the second element in the shape tuple, which is actually all the N-2 D tensors in all N-1 D tensors.
Axis 2
Axis 2 refers to the third element in the shape tuple, which is actually all the N-3 D tensors in all N-2 D tensors.
In this case, it is all the scalars in all of the vectors.
Note — Axis = -1 can also be used for this case because -1 represents the last element in the tuple. In this case, it happens to be the third entry.
A small comparison
This is a small comparison between three tensors.
First, (4, 1)
x1 = np.array([[1], [2], [3], [4]])
print(x1)x1.shape
Second, (4,)
x2 = np.array([1, 2, 3, 4])
print(x2)x2.shape
Third, (1, 4)
x3 = np.array([[1, 2, 3, 4]])
print(x3)x3.shape
I hope now you understand the difference between (4, 1), (4, ) and, (1, 4) and how Tensors are built.
If you are looking for Images as NumPy Tensors, then you have to wait till Chapter 6 — CNNs.
The goal of this chapter is to help you understand how we will build the Neural Network.
Watch the video on youtube and subscribe to the channel for videos and posts like this.
Every slide is 3 seconds long and without sound. You may pause the video whenever you like.
You may put on some music too if you like.
The video is basically everything in the post only in slides.
Many thanks for your support and feedback.
If you like this course, then you can support me at
It would mean a lot to me. Many thanks.
