Warmup is a method of warming up learning rate mentioned in ResNet paper. ... At the beginning of training, it uses a small learning rate to train some epoches or steps (for example, 4 epochs, 10000 steps), and then modifies it to the preset learning for training.

What is warm-up learning rate?

a) Warm-up: A phase in the beginning of your neural network training where you start with a learning rate much smaller than your "initial" learning rate and then increase it over a few iterations or epochs until it reaches that "initial" learning rate.

What is warmup steps in deep learning?

13. As the other answers already state: Warmup steps are just a few updates with low learning rate before / at the beginning of training. After this warmup, you use the regular learning rate (schedule) to train your model to convergence.