Distill
Prep
Python
GenAI
GenAI Frameworks
NLP
Deep Learning
Machine Learning
ML Libraries
Statistics
SQL
MLOps
Cloud
System Design
Pricing
Blog
Learn
Practice
Test
Live Engine
Select Topic
Introduction To Neural Networks
(15)
Neurons And Perceptrons
(15)
Activation Functions
(15)
Forward Propagation
(15)
Loss And Cost Functions
(15)
Backpropagation
(15)
Optimizers
(15)
Ann Architectures
(15)
Regularization And Normalization
(15)
Weight Initialization
(15)
Cnn Architectures
(15)
Rnn Lstm Gru
(15)
Attention And Transformers Dl
(15)
Self Supervised And Contrastive Learning
(15)
Graph Neural Networks
(15)
Transfer Learning
(15)
easy
Weight Initialization
A new engineer initializes all weights in a 5-layer fully connected network to 0. After 100 training epochs, the model achieves exactly random chance (10% for a 10-class problem). What went wrong?
A
Zero initialization causes NaN during forward propagation
B
Zero initialization causes the symmetry problem: all neurons in each layer compute the same output and receive the same gradient. All neurons in a layer update identically on every step. The model effectively has only 1 neuron per layer regardless of the layer width. With all weights zero, every hidden layer outputs 0, and the gradient for every neuron in a layer is identical, so they all update to the same non-zero value — they remain symmetric forever
C
Zero initialization prevents the optimizer from computing gradients
D
Zero initialization only fails for layers with more than 100 neurons
Previous
Back
Next