Learning Rate Decay

In short

• •An epoch is defined as completing one full pass through the entire training dataset.
• •The learning rate dictates the magnitude of parameter updates during model training.
• •Decaying the learning rate over time helps balance convergence speed with stability.

Key concepts

• Epoch0:09
• Learning Rate Influence0:43
• High Learning Rate Effects0:58
• Low Learning Rate Effects1:13
• Validation-based Decay2:07

Check yourself

1. 1. What happens if the learning rate alpha is set to a very large value?

   Answer: Parameter values vary rapidly and may not settle in a local minimum.

   If alpha is close to 1, parameter values change by large amounts and fail to settle into a local minimum. 0:58

2. 2. What is the primary risk of using a very low learning rate?

   Answer: The parameters may not change enough and could get stuck in false minima.

   Low learning rates lead to slow learning where parameters might not change enough to avoid getting stuck in false minima. 1:13

3. 3. Which of the following is a described method for decaying the learning rate?

   Answer: Decreasing the rate by a constant factor every epoch or set of epochs.

   One common technique is to reduce the learning rate by a constant factor periodically throughout the training. 1:34

4. 4. What defines one epoch in the context of gradient descent?

   Answer: Completing one full pass through the entire dataset.

   An epoch is defined as having gone through the entire data set once. 0:09

5. 5. When is the learning rate decreased based on validation data performance?

   Answer: Whenever the performance on the validation data improves.

   The speaker suggests decreasing the learning rate by a fixed factor whenever performance on validation data improves. 2:07