Can we change the learning rate half way through training?

What I notice is that some learning rate is great in the beginning but after some time it jiggles around instead of strictly decreasing, but if I decrease that learning rate by an order of 10 it learns too slow.

Can we change the learning rate halfway through training? This way in the beginning it trains fast and after some time we turn down the learning rate so the training becomes more stable.

If we can, how can we determine when to change the learning rate besides trial and error?

Thanks

You can use a scheduler that will change the LR for you during the training.

The ones I find most useful are CyclicLR and ReduceLROnPlateau

3 Likes