Correct answer: YES

The new $W_1$$W_1$ will be the same as the previous $W_1$$W_1$ after one iteration of gradient descent.

Let's understand why.

1. Understand the network

We have:

$Input\to HiddenLayer\to OutputLayer$$Input \rightarrow Hidden\ Layer \rightarrow Output\ Layer$

• Hidden layer activation = Sigmoid

• Output layer activation = Softmax

• $W_1$$W_1$ = Randomly initialized

• $W_2$$W_2$ = All zeros

• Bias = 0

2. How does gradient descent update weights?

The update rule is:

$W_1^{new}=W_1^{old}-\eta \frac{\partial L}{\partial W_1}$$W_1^{new}=W_1^{old}-\eta\frac{\partial L}{\partial W_1}$

Here:

• $\eta$$\eta$ = Learning rate

• $\frac{\partial L}{\partial W_1}$$\frac{\partial L}{\partial W_1}$ = Gradient of loss with respect to hidden layer weights.

For $W_1$$W_1$ to change, its gradient must be non-zero.

3. Apply the chain rule

The gradient is:

$\frac{\partial L}{\partial W_1}=\frac{\partial L}{\partial h}\cdot \frac{\partial h}{\partial W_1}$$\frac{\partial L}{\partial W_1} = \frac{\partial L}{\partial h} \cdot \frac{\partial h}{\partial W_1}$

But the gradient reaching the hidden layer depends on the output weights $W_2$$W_2$:

$\frac{\partial L}{\partial h}=W_2^T\frac{\partial L}{\partial z}$$\frac{\partial L}{\partial h} = W_2^T\frac{\partial L}{\partial z}$

Since all output weights are initialized to zero:

$W_2=0$$W_2=0$

Therefore:

$\frac{\partial L}{\partial h}=0$$\frac{\partial L}{\partial h}=0$

And consequently:

$\boxed{{\displaystyle \frac{\partial L}{\partial W_1}=0}}$$\boxed{\frac{\partial L}{\partial W_1}=0}$

4. Apply gradient descent

$W_1^{new}=W_1^{old}-\eta (0)$$W_1^{new}=W_1^{old}-\eta(0)$

$\boxed{{\displaystyle W_1^{new}=W_1^{old}}}$$\boxed{W_1^{new}=W_1^{old}}$

W₁ does not change

5. Important concept for exams

This is an example of the zero gradient problem caused by zero initialization of output weights.

Remember:

• $W_2=0$$W_2=0$ → Gradient reaching hidden layer = 0.

• Gradient of $W_1=0$$W_1=0$ → $W_1$$W_1$ does not update.

• $W_2$$W_2$, however, can update because its own gradient need not be zero.

Exam trick: If output layer weights are initialized to zero, the first backward pass cannot propagate a gradient to the hidden layer weights.

Correct answer: NO

This is the opposite of the previous question. Let's understand why.

1. Given information

• $W_1$$W_1$ = Random values

• $W_2$$W_2$ = All zeros

• Hidden activation = Sigmoid

• Output activation = Softmax

• Bias = 0

2. Gradient descent update rule

$W_2^{new}=W_2^{old}-\eta \frac{\partial L}{\partial W_2}$$W_2^{new}=W_2^{old}-\eta\frac{\partial L}{\partial W_2}$

For $W_2$$W_2$ to change, its gradient must be non-zero.

3. Calculate the gradient

Using the chain rule:

$\frac{\partial L}{\partial W_2}=\frac{\partial L}{\partial \hat{y}}\cdot \frac{\partial \hat{y}}{\partial z}\cdot \frac{\partial z}{\partial W_2}$$\frac{\partial L}{\partial W_2} = \frac{\partial L}{\partial \hat y} \cdot \frac{\partial \hat y}{\partial z} \cdot \frac{\partial z}{\partial W_2}$

The important point is:

$\frac{\partial z}{\partial W_2}=h$$\frac{\partial z}{\partial W_2}=h$

Here, $h$$h$ is the hidden layer output.

Since:

$h=\mathrm{sigmoid}(W_{1}x)$$h=\operatorname{sigmoid}(W_1x)$

And $W_1$$W_1$ is randomly initialized, the hidden layer produces non-zero values.

Therefore:

$\boxed{{\displaystyle \frac{\partial L}{\partial W_2}\ne 0}}$$\boxed{\frac{\partial L}{\partial W_2}\neq 0}$

Generally, the gradient is non-zero.

4. Apply gradient descent

$W_2^{new}=W_2^{old}-\eta (\text{non-zero gradient})$$W_2^{new}=W_2^{old}-\eta(\text{non-zero gradient})$

Therefore:

$\boxed{{\displaystyle W_2^{new}\ne W_2^{old}}}$$\boxed{W_2^{new}\neq W_2^{old}}$

5. Compare both questions

Remember this exam trick:

Zero initialization of weights does NOT mean those weights can never update.

• $W_1$$W_1$ cannot update in the first iteration because its gradient is blocked by $W_2=0$$W_2=0$.

• $W_2$$W_2$ can update because it receives a gradient directly from the loss through the hidden layer output.