How Can We Help?
Nesterov Solver
NesterovSolver – this solver is similar to SGD, but the error gradient is computed on weights with added momentum where the momentum term is “pointed in the right direction”. For more information on the Netsterov momentum and how it differs from SGD + momentum, see the article “Understanding Nesterov Momentum (NAG)” by Dominik Schmidt.
Parameters
The SolverParameter specifies all parameters that are used to configure the solver used. This section describes the NESTEROV specific parameters.
type: the SolverType.NESTEROV type creates a new NesterovSolver.
momentum: specifies the amount of the previous weight to use in the weight update. For example, when momentum = 0.9, 90% of the previous weight is added to 10% of the gradient update. For more discussion on the impact of momentum, see the article “Stochastic Gradient Descent with Momentum” by Vitaly Bushaev.
When to Use
According to “SGD with Nesterov acceleration – How it reduces the oscillation in SGD with Momentum” by neuralthreads, suggests that Nesterov can reduce the optimization oscillation and thus “get faster convergence.”