full backpropagation calculation for neural networks

Question 1

Forward Pass

Hidden Layer

$Z_1^{[1]} = W_{10}^{[1]} + W_{11}^{[1]}a_1^{[0]} + W_{12}^{[1]}a_2^{[0]}$

$Z_2^{[1]} = W_{20}^{[1]} + W_{21}^{[1]}a_1^{[0]} + W_{22}^{[1]}a_2^{[0]}$

$a_1^{[1]} = \sigma(Z_1^{[1]})$

$a_2^{[1]} = \sigma(Z_2^{[1]})$

Output layer

$Z^{[2]} = W_0^{[2]} + W_1^{[2]}a_1^{[1]} + W_2^{[2]}a_2^{[1]}$

$a^{[2]} = \sigma(Z^{[2]})$

Backpropagation

Binary Cross Entropy Loss Derivative

We assume binary cross entropy loss:

$L(a^{[2]}, y) = -[y \cdot \log(a^{[2]}) + (1 - y) \cdot \log(1 - a^{[2]})]$

$\frac{d}{dx} \log(x) = \frac{1}{x}$

$\frac{d}{dx} \log(1-x) = -\frac{1}{1-x}$

$\frac{\partial L(a^{[2]}, y)}{\partial a^{[2]}} = -\left[ y \cdot \frac{1}{a^{[2]}} + (1-y) \cdot \left( -\frac{1}{1-a^{[2]}} \right) \right]$

$\frac{\partial L(a^{[2]}, y)}{\partial a^{[2]}} = -\frac{y}{a^{[2]}} + \frac{1-y}{1-a^{[2]}}$

Derivative of Sigmoid Activation

$\frac{da}{dz} = \frac{d}{dz}(1 + e^{-z})^{-1}$

$\frac{da}{dz} = -1(1 + e^{-z})^{-2} \cdot (-e^{-z}) = \frac{e^{-z}}{(1 + e^{-z})^2}$

$\frac{da}{dz} = \left( \frac{1}{1 + e^{-z}} \right) \cdot \left( \frac{1 + e^{-z} - 1}{1 + e^{-z}} \right)$

$\frac{da}{dz} = \sigma(z) \cdot (1 - \sigma(z))$

Output Layer Weights

We calculate the gradient for each weight by multiplying the loss derivative, the activation derivative, and the logit derivative.

Weight $W_0^{[2]}$

$\frac{\partial L(a^{[2]}, y)}{\partial W_0^{[2]}} = \left( \frac{\partial L(a^{[2]}, y)}{\partial a^{[2]}} \right) \cdot \left( \frac{\partial a^{[2]}}{\partial Z^{[2]}} \right) \cdot \left( \frac{\partial Z^{[2]}}{\partial W_0^{[2]}} \right)$

$\frac{\partial L(a^{[2]}, y)}{\partial W_0^{[2]}} = \left( -\frac{y}{a^{[2]}} + \frac{1-y}{1-a^{[2]}} \right) \cdot \left( a^{[2]}(1 - a^{[2]}) \right) \cdot (1)$

$\frac{\partial L(a^{[2]}, y)}{\partial W_0^{[2]}} = a^{[2]} - y$

Weight $W_1^{[2]}$

$\frac{\partial L(a^{[2]}, y)}{\partial W_1^{[2]}} = \left( \frac{\partial L(a^{[2]}, y)}{\partial a^{[2]}} \right) \cdot \left( \frac{\partial a^{[2]}}{\partial Z^{[2]}} \right) \cdot \left( \frac{\partial Z^{[2]}}{\partial W_1^{[2]}} \right)$

$\frac{\partial L(a^{[2]}, y)}{\partial W_1^{[2]}} = \left( -\frac{y}{a^{[2]}} + \frac{1-y}{1-a^{[2]}} \right) \cdot \left( a^{[2]}(1 - a^{[2]}) \right) \cdot (a_1^{[1]})$

$\frac{\partial L(a^{[2]}, y)}{\partial W_1^{[2]}} = (a^{[2]} - y)a_1^{[1]}$

Weight $W_2^{[2]}$

$\frac{\partial L(a^{[2]}, y)}{\partial W_2^{[2]}} = \left( \frac{\partial L(a^{[2]}, y)}{\partial a^{[2]}} \right) \cdot \left( \frac{\partial a^{[2]}}{\partial Z^{[2]}} \right) \cdot \left( \frac{\partial Z^{[2]}}{\partial W_2^{[2]}} \right)$

$\frac{\partial L(a^{[2]}, y)}{\partial W_2^{[2]}} = \left( -\frac{y}{a^{[2]}} + \frac{1-y}{1-a^{[2]}} \right) \cdot \left( a^{[2]}(1 - a^{[2]}) \right) \cdot (a_2^{[1]})$

$\frac{\partial L(a^{[2]}, y)}{\partial W_2^{[2]}} = (a^{[2]} - y)a_2^{[1]}$

Hidden Layer

For the hidden layer, we must first find how the loss changes with respect to the hidden activations $a_i^{[1]}$.

Hidden Node 1 Weights

First, we find the gradient of the loss with respect to the activation $a_1^{[1]}$:

$\frac{\partial L(a^{[2]}, y)}{\partial a_1^{[1]}} = \left( \frac{\partial L(a^{[2]}, y)}{\partial a^{[2]}} \right) \cdot \left( \frac{\partial a^{[2]}}{\partial Z^{[2]}} \right) \cdot \left( \frac{\partial Z^{[2]}}{\partial a_1^{[1]}} \right)$

$\frac{\partial L(a^{[2]}, y)}{\partial a_1^{[1]}} = \left( -\frac{y}{a^{[2]}} + \frac{1-y}{1-a^{[2]}} \right) \cdot \left( a^{[2]}(1 - a^{[2]}) \right) \cdot (W_1^{[2]})$

Now we apply the chain rule for the weights in Node 1:

$\frac{\partial L(a^{[2]}, y)}{\partial W_{11}^{[1]}} = \left( \frac{\partial L(a^{[2]}, y)}{\partial a_1^{[1]}} \right) \cdot \left( \frac{\partial a_1^{[1]}}{\partial Z_1^{[1]}} \right) \cdot \left( \frac{\partial Z_1^{[1]}}{\partial W_{11}^{[1]}} \right)$

$\frac{\partial L(a^{[2]}, y)}{\partial W_{11}^{[1]}} = \left( \frac{\partial L(a^{[2]}, y)}{\partial a_1^{[1]}} \right) \cdot \left( a_1^{[1]}(1 - a_1^{[1]}) \right) \cdot (a_1^{[0]})$

$\frac{\partial L(a^{[2]}, y)}{\partial W_{11}^{[1]}} = (a^{[2]} - y)W_1^{[2]} \cdot a_1^{[1]}(1 - a_1^{[1]}) \cdot a_1^{[0]}$

Hidden Node 2

First, we find the gradient of the loss with respect to the activation $a_2^{[1]}$:

$\frac{\partial L(a^{[2]}, y)}{\partial a_2^{[1]}} = \left( \frac{\partial L(a^{[2]}, y)}{\partial a^{[2]}} \right) \cdot \left( \frac{\partial a^{[2]}}{\partial Z^{[2]}} \right) \cdot \left( \frac{\partial Z^{[2]}}{\partial a_2^{[1]}} \right)$

$\frac{\partial L(a^{[2]}, y)}{\partial a_2^{[1]}} = \left( -\frac{y}{a^{[2]}} + \frac{1-y}{1-a^{[2]}} \right) \cdot \left( a^{[2]}(1 - a^{[2]}) \right) \cdot (W_2^{[2]})$

Now we apply the chain rule for the weights in Node 2:

$\frac{\partial L(a^{[2]}, y)}{\partial W_{21}^{[1]}} = \left( \frac{\partial L(a^{[2]}, y)}{\partial a_2^{[1]}} \right) \cdot \left( \frac{\partial a_2^{[1]}}{\partial Z_2^{[1]}} \right) \cdot \left( \frac{\partial Z_2^{[1]}}{\partial W_{21}^{[1]}} \right)$

$\frac{\partial L(a^{[2]}, y)}{\partial W_{21}^{[1]}} = \left( \frac{\partial L(a^{[2]}, y)}{\partial a_2^{[1]}} \right) \cdot \left( a_2^{[1]}(1 - a_2^{[1]}) \right) \cdot (a_1^{[0]})$

$\frac{\partial L(a^{[2]}, y)}{\partial W_{21}^{[1]}} = (a^{[2]} - y)W_2^{[2]} \cdot a_2^{[1]}(1 - a_2^{[1]}) \cdot a_1^{[0]}$

Final Weight Derivations

$\frac{\partial L(a^{[2]}, y)}{\partial W_0^{[2]}} = a^{[2]} - y$

$\frac{\partial L(a^{[2]}, y)}{\partial W_1^{[2]}} = (a^{[2]} - y) a_1^{[1]}$

$\frac{\partial L(a^{[2]}, y)}{\partial W_2^{[2]}} = (a^{[2]} - y) a_2^{[1]}$

$\frac{\partial L(a^{[2]}, y)}{\partial W_{10}^{[1]}} = (a^{[2]} - y) W_1^{[2]} \cdot a_1^{[1]}(1 - a_1^{[1]})$

$\frac{\partial L(a^{[2]}, y)}{\partial W_{11}^{[1]}} = (a^{[2]} - y) W_1^{[2]} \cdot a_1^{[1]}(1 - a_1^{[1]}) \cdot a_1^{[0]}$

$\frac{\partial L(a^{[2]}, y)}{\partial W_{12}^{[1]}} = (a^{[2]} - y) W_1^{[2]} \cdot a_1^{[1]}(1 - a_1^{[1]}) \cdot a_2^{[0]}$

$\frac{\partial L(a^{[2]}, y)}{\partial W_{20}^{[1]}} = (a^{[2]} - y) W_2^{[2]} \cdot a_2^{[1]}(1 - a_2^{[1]})$

$\frac{\partial L(a^{[2]}, y)}{\partial W_{21}^{[1]}} = (a^{[2]} - y) W_2^{[2]} \cdot a_2^{[1]}(1 - a_2^{[1]}) \cdot a_1^{[0]}$

$\frac{\partial L(a^{[2]}, y)}{\partial W_{22}^{[1]}} = (a^{[2]} - y) W_2^{[2]} \cdot a_2^{[1]}(1 - a_2^{[1]}) \cdot a_2^{[0]}$

Question 2

Forward Pass

Hidden Layer

$Z_1^{[1]} = W_{10}^{[1]} + W_{11}^{[1]}a_1^{[0]} + W_{12}^{[1]}a_2^{[0]}$

$Z_2^{[1]} = W_{20}^{[1]} + W_{21}^{[1]}a_1^{[0]} + W_{22}^{[1]}a_2^{[0]}$

$a_1^{[1]} = \sigma(Z_1^{[1]})$

$a_2^{[1]} = \sigma(Z_2^{[1]})$

Output layer

$Z_1^{[2]} = W_{10}^{[2]} + W_{11}^{[2]}a_1^{[1]} + W_{12}^{[2]}a_2^{[1]}$

$Z_2^{[2]} = W_{20}^{[2]} + W_{21}^{[2]}a_1^{[1]} + W_{22}^{[2]}a_2^{[1]}$

$a_1^{[2]} = \frac{\exp(Z_1^{[2]})}{\exp(Z_1^{[2]}) + \exp(Z_2^{[2]})}$

$a_2^{[2]} = \frac{\exp(Z_2^{[2]})}{\exp(Z_1^{[2]}) + \exp(Z_2^{[2]})}$

Backpropagation

Categorical Cross Entropy Loss Derivative

We assume categorical cross entropy loss for the multiple output classes:

$L(a^{[2]}, y) = -\sum_{k=1}^{2} y_k \cdot \log(a_k^{[2]})$

$\frac{\partial L(a^{[2]}, y)}{\partial a_k^{[2]}} = -\frac{y_k}{a_k^{[2]}}$

Derivative of Softmax Activation

$a_i^{[2]} = \frac{\exp(Z_1^{[2]})}{\sum_{j=1}^{K} \exp(Z_j^{[2]})}$

When $i = j$: $\frac{\partial a_i^{[2]}}{\partial Z_i^{[2]}} = a_i^{[2]}(1 - a_i^{[2]})$

When $i \neq j$: $\frac{\partial a_i^{[2]}}{\partial Z_j^{[2]}} = -a_i^{[2]}a_j^{[2]}$

Applying the chain rule to find the gradient of the loss with respect to the logit $Z_i^{[2]}$:

$\frac{\partial L(a^{[2]}, y)}{\partial Z_i^{[2]}} = \sum_{k=1}^{2} \left( \frac{\partial L(a^{[2]}, y)}{\partial a_k^{[2]}} \right) \cdot \left( \frac{\partial a_k^{[2]}}{\partial Z_i^{[2]}} \right)$

For a specific node $i$ (and letting the other node be $j$):

$\frac{\partial L(a^{[2]}, y)}{\partial Z_i^{[2]}} = -y_i(1 - a_i^{[2]}) + y_j a_i^{[2]}$

$\frac{\partial L(a^{[2]}, y)}{\partial Z_i^{[2]}} = -y_i + y_i a_i^{[2]} + y_j a_i^{[2]}$

$\frac{\partial L(a^{[2]}, y)}{\partial Z_i^{[2]}} = a_i^{[2]}(y_i + y_j) - y_i$

We let $y_i + y_j = 1$:

$\frac{\partial L(a^{[2]}, y)}{\partial Z_i^{[2]}} = a_i^{[2]} - y_i$

Output Layer Weights

We calculate the gradient for each weight by multiplying the loss derivative with respect to the logit ($\frac{\partial L(a^{[2]}, y)}{\partial Z_i^{[2]}}$) and the logit derivative with respect to the weight.

Weights for Output Node 1

Weights for Output Node 2

Hidden Layer

For the hidden layer, we must first find how the loss changes with respect to the hidden activations $a_i^{[1]}$. Because each hidden node connects to multiple output nodes, we sum the gradients propagating backwards from all paths.

Hidden Node 1 Weights

First, we find the gradient of the loss with respect to the activation $a_1^{[1]}$:

Now we apply the chain rule for the weights in Hidden Node 1:

Hidden Node 2

First, we find the gradient of the loss with respect to the activation $a_2^{[1]}$:

Now we apply the chain rule for the weights in Hidden Node 2:

Final Weight Derivations

$\frac{\partial L(a^{[2]}, y)}{\partial W_{10}^{[2]}} = a_1^{[2]} - y_1$ $\frac{\partial L(a^{[2]}, y)}{\partial W_{11}^{[2]}} = (a_1^{[2]} - y_1) a_1^{[1]}$ $\frac{\partial L(a^{[2]}, y)}{\partial W_{12}^{[2]}} = (a_1^{[2]} - y_1) a_2^{[1]}$

$\frac{\partial L(a^{[2]}, y)}{\partial W_{20}^{[2]}} = a_2^{[2]} - y_2$ $\frac{\partial L(a^{[2]}, y)}{\partial W_{21}^{[2]}} = (a_2^{[2]} - y_2) a_1^{[1]}$ $\frac{\partial L(a^{[2]}, y)}{\partial W_{22}^{[2]}} = (a_2^{[2]} - y_2) a_2^{[1]}$