Why's Xavier initialization important? In short, it helps signals reach deep into the network. If the weights in a network start too small, then the signal shrinks as it passes through each layer until it's too tiny to be useful.

How does Xavier initialization work?

The goal of Xavier Initialization is to initialize the weights such that the variance of the activations are the same across every layer. This constant variance helps prevent the gradient from exploding or vanishing.

What is the purpose of he initialization?

Kaiming Initialization, or He Initialization, is an initialization method for neural networks that takes into account the non-linearity of activation functions, such as ReLU activations. That is, a zero-centered Gaussian with standard deviation of 2 / n l (variance shown in equation above). Biases are initialized at .