Multiple Inputs & Multiple Outputs in a Neural Network

Step by step implementation in Python

In this post, we will see how to apply Backpropagaton to train the Neural Network which has Multiple Inputs and Multiple Outputs. This is equivalent to the functional API of Keras.

You can download the Jupyter Notebook from here.

Note — This post uses many things from the previous chapters. It is recommended that you have a look at the previous posts.

Back to the previous post

Back to the first post

5.8 Multiple Inputs & Multiple Outputs

We will use a different NN architecture for this post. Let us have a look at the architecture.

We can see that first, we have the first input layer. Then 2 parallel hidden layers. Then we concatenated the outputs of 2 hidden layers with the second input layer, which we will call ‘Concatenated Layer’. Then the third hidden layer. And finally 2 output layers.

A few things,

First, the activation function for the hidden layers is the ReLU function
Second, the activation function for the first output layer is the Sigmoid function and for the second output layer is the Softmax function
Third, we will use Binary Cross-entropy Error, BCE as loss function for the first output layer and Mean Absolute Error, MAE as loss function for the second output layer
Fourth, we will use SGD Optimizer with a learning rate = 0.01

Note — You can use some different architecture if you want. All the steps are same. It is the type of problem which will make you think what architecture should I use.

Now, let us take a look at the steps

Step 1 - A forward feed to calculate loss
Step 2 - Initializing SGD Optimizer
Step 3 - Entering training loop
         Step 3.1 - Forward feed to calculate losses
         Step 3.2 - Calculate gradients using Backpropagation
         Step 3.3 - Using SGD Optimizer to update weights and biases
Step 4 - A forward feed to verify that loss has been reduced and to 
         see how close predicted values are to true values

Step 1 - A forward feed to calculate loss

import numpy as np                          # importing NumPy
np.random.seed(42)
input_1_nodes = 5                             # nodes in each layer
input_2_nodes = 3
hidden_1_nodes = 4
hidden_2_nodes = 3
concatenated_nodes = input_2_nodes + hidden_1_nodes + hidden_2_nodes
hidden_3_nodes = 4
output_1_nodes = 5
output_2_nodes = 4

x1 = np.random.randint(1, 100, size = (input_1_nodes, 1)) / 100
x1                                       # Input 1
x2 = np.random.randint(1, 100, size = (input_2_nodes, 1)) / 100
x2                                       # Input 2
y1 = np.array([[1], [1], [0], [0], [1]])
y1                                       # Output 1
y2 = np.array([[0], [1], [0], [0]])
y2                                       # Output 2

def relu(x, leak = 0):                          # ReLU
    return np.where(x <= 0, leak * x, x)
def relu_dash(x, leak = 0):                     # ReLU derivative
    return np.where(x <= 0, leak, 1)
def sig(x):                                     # Sigmoid
    return 1/(1 + np.exp(-x))
def sig_dash(x):                                # Sogmoid derivative
    return sig(x) * (1 - sig(x))
def softmax(x):                                 # Softmax
    return np.exp(x) / np.sum(np.exp(x))
def softmax_dash(x):                            # Softmax derivative
    
    I = np.eye(x.shape[0])
    
    return softmax(x) * (I - softmax(x).T)
def B_cross_E(y_true, y_pred):                  # BCE
    return -np.mean(y_true * np.log(y_pred + 10**-100) + (1 -
                      y_true) * np.log(1 - y_pred + 10**-100))
def B_cross_E_grad(y_true, y_pred):             # BCE derivative
    
    N = y_true.shape[0]
    
    return -(y_true/(y_pred + 10**-100) - (1 - y_true)/(1 - y_pred +
                                                        10**-100))/N
def mae(y_true, y_pred):                        # MAE
    return np.mean(abs(y_true - y_pred))
def mae_grad(y_true, y_pred):                   # MAE derivative
    
    N = y_true.shape[0]
    
    return -((y_true - y_pred) / (abs(y_true - y_pred) +
                                            10**-100))/N

w1 = np.random.random(size = (hidden_1_nodes, input_1_nodes))  # w1
b1 = np.zeros(shape = (hidden_1_nodes, 1))                     # b1
w2 = np.random.random(size = (hidden_2_nodes, input_1_nodes))  # w2
b2 = np.zeros(shape = (hidden_2_nodes, 1))                     # b2
w3 = np.random.random(size = (hidden_3_nodes, concatenated_nodes))
b3 = np.zeros(shape = (hidden_3_nodes, 1))                  # w3, b3
w4 = np.random.random(size = (output_1_nodes, hidden_3_nodes)) # w4
b4 = np.zeros(shape = (output_1_nodes, 1))                     # b4
w5 = np.random.random(size = (output_2_nodes, hidden_3_nodes)) # w5
b5 = np.zeros(shape = (output_2_nodes, 1))                     # b5

Now let us take a look at the outputs of each layer

in_hidden_1 = w1.dot(x1) + b1
out_hidden_1 = relu(in_hidden_1)
print('out_hidden_1')
print(out_hidden_1, out_hidden_1.shape)
print('\n')

in_hidden_2 = w2.dot(x1) + b2
out_hidden_2 = relu(in_hidden_2)
print('out_hidden_2')
print(out_hidden_2, out_hidden_2.shape)
print('\n')

concatenated_layer = np.concatenate((x2, out_hidden_1, out_hidden_2))
print('concatenated_layer')
print(concatenated_layer, concatenated_layer.shape)
print('\n')

in_hidden_3 = w3.dot(concatenated_layer) + b3
out_hidden_3 = relu(in_hidden_3)
print('out_hidden_3')
print(out_hidden_3, out_hidden_3.shape)
print('\n')

in_output_layer_1 = w4.dot(out_hidden_3) + b4
y1_hat = sig(in_output_layer_1)
print('y1_hat')
print(y1_hat, y1_hat.shape)
print('\n')

in_output_layer_2 = w5.dot(out_hidden_3) + b5
y2_hat = softmax(in_output_layer_2)
print('y2_hat')
print(y2_hat, y2_hat.shape)
print('\n')

y1                                             # y_hat 1
y2                                             # y_hat 2
B_cross_E(y1, y1_hat) + mae(y2, y2_hat)        # total loss
B_cross_E(y1, y1_hat)                          # loss 1
mae(y2, y2_hat)                                # loss 2

Step 2 — Initializing SGD Optimizer

learning_rate = 0.01

Step 3 — Entering the training loop

epochs = 1000

Step 3.1 — Forward feed to calculate loss before training

for epoch in range(epochs):
    
    #------------------------Forward Propagation--------------------
    
    in_hidden_1 = w1.dot(x1) + b1
    out_hidden_1 = relu(in_hidden_1)
    in_hidden_2 = w2.dot(x1) + b2
    out_hidden_2 = relu(in_hidden_2)
    concatenated_layer = np.concatenate((x2, out_hidden_1,
                                            out_hidden_2))
    in_hidden_3 = w3.dot(concatenated_layer) + b3
    out_hidden_3 = relu(in_hidden_3)
    in_output_layer_1 = w4.dot(out_hidden_3) + b4
    y1_hat = sig(in_output_layer_1)
    in_output_layer_2 = w5.dot(out_hidden_3) + b5
    y2_hat = softmax(in_output_layer_2)
    
    loss = B_cross_E(y1, y1_hat) + mae(y2, y2_hat)
    print(f'loss before training is {loss}')
    print(f'output_1 loss is {B_cross_E(y1, y1_hat)}')
    print(f'output_2 loss is {mae(y2, y2_hat)}-- epoch number {epoch
                                                              + 1}')
    print('\n')

Step 3.2 — Calculate gradients using Backpropagation

We will use the rules of ‘Jumping Back’ to calculate the gradients. But this time we will do additional steps.

Let us begin with the losses.

We are calculating gradients for (w4, b4) and (w5, b5) separately because (w4, b4) and (w5, b5) are independent of each other.

    #-----------Gradient Calculations via Back Propagation----------
    grad_w4 = B_cross_E_grad(y1, y1_hat) *                # grad_w4
                  sig_dash(in_output_layer_1) .dot( out_hidden_3.T )
    
    grad_b4 = B_cross_E_grad(y1, y1_hat) *                # grad_b4
                sig_dash(in_output_layer_1)
    
    #-------------------------------------------
    
    error_upto_softmax = np.sum(mae_grad(y2, y2_hat) *
         softmax_dash(in_output_layer_2), axis = 0).reshape((-1, 1))
    
    grad_w5 = error_upto_softmax .dot( out_hidden_3.T )   # grad_w5
    
    grad_b5 = error_upto_softmax                          # grad_b5

Now, when we jump back across w4 and w5, we will add gradients after reshaping them to (-1, 1) and will proceed further

    error_grad_H3 = ( np.sum(B_cross_E_grad(y1, y1_hat) *
        sig_dash(in_output_layer_1) * w4, axis = 0).reshape((-1, 1))
                               +
        np.sum(error_upto_softmax * w5, axis = 0).reshape((-1, 1)) )
    
    grad_w3 = error_grad_H3 * relu_dash(in_hidden_3) .dot(
                                  concatenated_layer.T )   # grad_w3
    
    grad_b3 = error_grad_H3 * relu_dash(in_hidden_3)       # grad_b3

Now how to calculate gradients for (w1, b1) and (w2, b2).

If you have noticed, now we have nothing to do with ‘x2’. Why? Because ‘x2’ is connected to hidden layer 3 via w3 and b3 between concatenated layer and hidden layer 3.

The shape of w3 is (hidden_3_nodes, concatenated_nodes), i.e., (4, 10)

The weights matrix w3 is

In these weights, there are connections between components of concatenated layer and input of the hidden 3 layer.

The weights in the blue box above connect input 2, i.e., ‘x2’ with hidden 3 ‘I_H3’. Shape (hidden_3_nodes, input_2_nodes) or (4, 3)

The weights in the blue box above connect the output of hidden 1, i.e., ‘O_H1’ with hidden 3 ‘I_H3’. Shape (hidden_3_nodes, hidden_1_nodes) or (4, 4)

The weights in the blue box above connect the output of hidden 2, i.e., ‘O_H2’ with hidden 3 ‘I_H3’. Shape (hidden_3_nodes, hidden_2_nodes) or (4, 3)

To calculate gradients for (w1, b1) and (w2, b2), we will use only the second and third blue box sub-matrices.

    error_grad_H1 = np.sum(error_grad_H3 * relu_dash(in_hidden_3) *
              w3[:, input_2_nodes: input_2_nodes + hidden_1_nodes], 
                           axis = 0).reshape((-1, 1))
    
    grad_w1 = error_grad_H1 * relu_dash(in_hidden_1) .dot( x1.T )
                                                          # grad_w1
    grad_b1 = error_grad_H1 * relu_dash(in_hidden_1)      # grad_b1
    
    #-------------------------------------------
    
    error_grad_H2 = np.sum(error_grad_H3 * relu_dash(in_hidden_3) *
                              w3[:, input_2_nodes + hidden_1_nodes:                                             
                   input_2_nodes + hidden_1_nodes + hidden_2_nodes], 
                                          axis = 0).reshape((-1, 1))
    
    grad_w2 = error_grad_H2 * relu_dash(in_hidden_2) .dot( x1.T )
                                                          # grad_w2
    grad_b2 = error_grad_H2 * relu_dash(in_hidden_2)      # grad_b2

Step 3.3 — Using SGD Optimizer to update weights and biases

    #-------------Updating weights and biases with SGD--------------
    update_w1 = - learning_rate * grad_w1
    w1 += update_w1                                             # w1
    
    update_b1 = - learning_rate * grad_b1
    b1 += update_b1                                             # b1
    
    update_w2 = - learning_rate * grad_w2
    w2 += update_w2                                             # w2
    
    update_b2 = - learning_rate * grad_b2
    b2 += update_b2                                             # b2
    
    update_w3 = - learning_rate * grad_w3
    w3 += update_w3                                             # w3
    
    update_b3 = - learning_rate * grad_b3
    b3 += update_b3                                             # b3
    
    update_w4 = - learning_rate * grad_w4
    w4 += update_w4                                             # w4
    
    update_b4 = - learning_rate * grad_b4
    b4 += update_b4                                             # b4
    
    update_w5 = - learning_rate * grad_w5
    w5 += update_w5                                             # w5
    
    update_b5 = - learning_rate * grad_b5
    b5 += update_b5                                             # b5

The training loop will run 1,000 times.

This is a small screenshot after the training.

Step 4 — A forward feed to verify that loss has been reduced and to see how close predicted values are to true values

in_hidden_1 = w1.dot(x1) + b1                     # forward feed
out_hidden_1 = relu(in_hidden_1) 
in_hidden_2 = w2.dot(x1) + b2
out_hidden_2 = relu(in_hidden_2)
concatenated_layer = np.concatenate((x2, out_hidden_1, out_hidden_2))
in_hidden_3 = w3.dot(concatenated_layer) + b3
out_hidden_3 = relu(in_hidden_3)
in_ouput_layer_1 = w4.dot(out_hidden_3) + b4
y1_hat = sig(in_ouput_layer_1)
in_ouput_layer_2 = w5.dot(out_hidden_3) + b5
y2_hat = softmax(in_ouput_layer_2)
y1_hat                                   # first predicted output
y1                                       # first true output
y2_hat                                   # second predicted output
y2                                       # second true output
B_cross_E(y1, y1_hat) + mae(y2, y2_hat)  # total loss
B_cross_E(y1, y1_hat)                    # BCE loss
mae(y2, y2_hat)                          # MAE loss

I hope now you understand how such a Neural Network works.

In the upcoming post, which will be the last of this Chapter and of simple ANNs, we will fit a Neural Network on the UCI White Wine Quality dataset. And after that, we will begin Chapter 6 — CNNs.

If you like this post then please subscribe to my youtube channel neuralthreads and join me on Reddit.

I will be uploading new interactive videos soon on the youtube channel. And I will be happy to help you with any doubt on Reddit.

Many thanks for your support and feedback.

If you like this course, then you can support me at

It would mean a lot to me.

Continue to the next post — 5.9 UCI White Wine Quality dataset, Accuracy, and Confusion Matrix.