MyCaffe  1.12.2.41
Deep learning software for Windows C# programmers.
MyCaffe.solvers.SGDSolver< T > Class Template Reference

Stochastic Gradient Descent solver with momentum updates weights by a linear combination of the negative gradient and the previous weight update. More...

Inheritance diagram for MyCaffe.solvers.SGDSolver< T >:
MyCaffe.solvers.Solver< T > MyCaffe.solvers.AdaDeltaSolver< T > MyCaffe.solvers.AdaGradSolver< T > MyCaffe.solvers.AdamSolver< T > MyCaffe.solvers.AdamWSolver< T > MyCaffe.solvers.NesterovSolver< T > MyCaffe.solvers.RmsPropSolver< T >

Public Member Functions

 SGDSolver (CudaDnn< T > cuda, Log log, SolverParameter p, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
 The SGDSolver constructor. More...
 
void PreSolve ()
 Runs the pre-solve which prepares the Solver to start Solving. More...
 
double GetLearningRate (int nIterationOverride=-1)
 Return the current learning rate. More...
 
override double ApplyUpdate (int nIterationOverride=-1)
 Compute the update values and apply them to the training Net. More...
 
virtual void Normalize (int param_id)
 Normalize a learnable Blob of the training Net. More...
 
virtual void Regularize (int param_id)
 Regularize a learnable Blob of the training net. More...
 
virtual void ComputeUpdateValue (int param_id, double dfRate, int nIterationOverride=-1)
 Compute the SGD update value that will be applied to a learnable blobs in the training Net. More...
 
virtual void ClipGradients ()
 Clip the gradients of all learnable blobs in the training Net. More...
 
- Public Member Functions inherited from MyCaffe.solvers.Solver< T >
 Solver (CudaDnn< T > cuda, Log log, SolverParameter p, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
 The Solver constructor. More...
 
void Dispose ()
 Discards the resources (GPU and Host) used by this Solver. More...
 
bool ForceOnTrainingIterationEvent ()
 Force an OnTrainingIterationEvent to fire. More...
 
void Init (SolverParameter p, Net< T > shareNet=null)
 Initializes the Solver. More...
 
void Reset ()
 Reset the iterations of the net. More...
 
virtual void Solve (int nIterationOverride=-1, byte[] rgWeights=null, byte[] rgState=null, TRAIN_STEP step=TRAIN_STEP.NONE)
 The main entry of the solver function. In default, iter will be zero. Pass in a non-zero iter number to resume training for a pre-trained net. More...
 
bool Step (int nIters, TRAIN_STEP step=TRAIN_STEP.NONE, bool bZeroDiffs=true, bool bApplyUpdates=true, bool bDisableOutput=false, bool bDisableProgress=false, double? dfLossOverride=null, bool? bAllowSnapshot=null)
 Steps a set of iterations through a training cycle. More...
 
void Restore (byte[] rgWeights, byte[] rgState, string strSkipBlobTypes=null)
 The restore method simply calls the RestoreSolverState method of the inherited class. More...
 
void Snapshot (bool bForced, bool bScheduled, bool bUpdateDatabase=true)
 The snapshot function implements the basic snapshotting utility that stores the learned net. This method calls the SnapshotSolverState method of the inherited class. More...
 
SnapshotArgs GetSnapshotArgs (byte[] rgState, byte[] rgWeights, double dfAccuracy, double dfError, int nIteration, SNAPSHOT_WEIGHT_UPDATE_METHOD wtUpdt)
 The GetSnapshotArgs method fills out a snapshot args structure. More...
 
double TestAll (int nIterationOverride=-1)
 Run a TestAll by running all test Nets. More...
 
double TestDetection (int nIterationOverride=-1, int nTestNetId=0)
 Run an SSD detection test on a given test Net by running it through its iterations. More...
 
double TestClassification (int nIterationOverride=-1, int nTestNetId=0)
 Run a test on a given test Net by running it through its iterations. More...
 
void UpdateSmoothedLoss (double dfLoss, int nStartIter, int nAverageLoss=0)
 Update the avaraged loss value. More...
 

Protected Member Functions

override void dispose ()
 Releases all resources (GPU and Host) used by the Solver. More...
 
override void RestoreSolverState (byte[] rgState)
 Restore the state of the Solver. More...
 
override byte[] SnapshotSolverState ()
 Take a snapshot of the Solver state. More...
 
- Protected Member Functions inherited from MyCaffe.solvers.Solver< T >
void InitTrainNet (Net< T > shareNet=null)
 Initializes the Net used by the solver for training. More...
 
void InitTestNets ()
 Initializes the Net used by the Solver for testing. More...
 

Protected Attributes

BlobCollection< T > m_colHistory = new BlobCollection<T>()
 History maintains the historical momentum data. More...
 
BlobCollection< T > m_colTemp = new BlobCollection<T>()
 Update maintains update related data and is not needed in snapshots. More...
 
- Protected Attributes inherited from MyCaffe.solvers.Solver< T >
CudaDnn< T > m_cuda
 Specifies the instance of CudaDnn used by the Solver that provides a connection to Cuda. More...
 
Log m_log
 Specifies the Log for output. More...
 
SolverParameter m_param
 Specifies the SolverParameter that defines how the Solver operates. More...
 
Net< T > m_net
 Specifies the training Net. More...
 
List< Net< T > > m_rgTestNets = new List<Net<T>>()
 Specifies the testing Nets. More...
 
int m_nIter
 Specifies the current iteration. More...
 
int m_nCurrentStep
 Specifies the current step. More...
 
List< double > m_rgLosses = new List<double>()
 Specifies the Losses used to calculate the smoothed Loss. More...
 
double m_dfSmoothedLoss = 0
 Specifies the smoothed loss protected for derived classes to use. More...
 
double? m_dfIterAccuracy = null
 Specifies the iteration accuracy calculated when a blob exists with the name 'accuracy'. More...
 
int m_nSolverCount = 1
 Specifies the Solver count in a multi-GPU training session. More...
 
int m_nSolverRank = 0
 Specifies the Solver rank of this solver, where rank == 0 is the root Solver. More...
 
IXPersist< T > m_persist
 Specifies the persistance object used to save weight and solver states. More...
 
double m_dfLearningRateOverride = 0
 Optionally, specifies a learning rate override (default = 0, which ignores this setting). More...
 

Properties

BlobCollection< T > history [get]
 Returns the history BlobCollection containing historical momentum data. More...
 
- Properties inherited from MyCaffe.solvers.Solver< T >
double LearningRateOverride [getset]
 Get/set the learning rate override. When 0, this setting is ignored. More...
 
int TrainingTimeLimitInMinutes [getset]
 Get/set the training time limit in minutes. When set to 0, no time limit is imposed on training. More...
 
SNAPSHOT_WEIGHT_UPDATE_METHOD SnapshotWeightUpdateMethod [getset]
 Get/set the snapshot weight update method. More...
 
IXDatabaseBase Database [get]
 Returns the in-memory MyCaffeDatabase used. More...
 
bool EnableTesting [getset]
 When enabled, the training cycle calls TestAll periodically based on the SolverParameter. Otherwise testing is not performed. More...
 
bool EnableBlobDebugging [getset]
 When enabled, the OnTrainingIteration event is set extra debugging information describing the state of each Blob used by the Solver. More...
 
bool EnableLayerDebugging [getset]
 Enable/disable layer debugging which causes each layer to check for NAN/INF on each forward/backward pass and throw an exception when found. More...
 
bool EnableBreakOnFirstNaN [getset]
 When enabled (requires EnableBlobDebugging = true), the Solver immediately stop training upon detecting the first NaN withing the training Net. More...
 
bool EnableDetailedNanDetection [getset]
 When enabled (requires EnableBlobDebugging = true), the detailed Nan (and Infinity) detection is perofmed on each blob when training Net. More...
 
bool EnableSingleStep [getset]
 When enabled (requires EnableBlobDebugging = true), the Solver only runs one training cycle. More...
 
bool WeightsUpdated [getset]
 Get/set when the weights have been updated. More...
 
object Tag [getset]
 Returns a generic tag associated with the Solver. More...
 
Net< T > TestingNet [get]
 Returns the testing Net used by the solver. More...
 
Net< T > TrainingNet [get]
 Returns the training Net used by the solver. More...
 
CudaDnn< T > Cuda [get]
 Returns the CudaDnn instance used by the Solver. More...
 
string ActiveLabelCounts [get]
 Returns a string describing the labels detected in the training along with the % that each label has participated in the training. More...
 
string LabelQueryHitPercents [get]
 Return the label query hit percentages for the active datasource. More...
 
string LabelQueryEpochs [get]
 Return the label query epochs for the active datasource. More...
 
int CurrentIteration [get]
 Returns the current training iteration. More...
 
int MaximumIteration [get]
 Returns the maximum training iterations. More...
 
int TrainingIterations [get]
 Returns the current training iterations remaining. More...
 
int? TestingIterations [get]
 Returns the current testing iterations remaining. More...
 
int TrainingIterationOverride [getset]
 Get/set the training iteration override. More...
 
int TestingIterationOverride [getset]
 Get/set the testing iteration override. More...
 
AutoResetEvent CompletedEvent [get]
 Returns an auto reset event that is set upon training completion. More...
 
CancelEvent CancelEvent [get]
 Returns the cancel event which when set cancels the current operation run by the Solver. More...
 
double smoothed_loss [get]
 Returns the smoothed loss. More...
 
SolverParameter parameter [get]
 Returns the SolverParameter used. More...
 
Net< T > net [get]
 Returns the main training Net. More...
 
List< Net< T > > test_nets [get]
 Returns the testing Nets. More...
 
int iter [get]
 Returns the current training iteration. More...
 
SolverParameter.SolverType type [get]
 Returns the type of solver. More...
 
bool forceSnapshot [get]
 Returns whether or not a snapshot has been forced. More...
 
bool forceTest [get]
 Returns whether or not a test has been forced. More...
 
int solver_count [get]
 Returns the solver count in a multi-GPU session. More...
 
int solver_rank [get]
 Returns this Solver's rank in a multi-GPU session. More...
 
bool? is_root_solver [get]
 Returns whether or not this is the root solver. More...
 

Additional Inherited Members

- Static Public Member Functions inherited from MyCaffe.solvers.Solver< T >
static SGDSolver< T > Create (CudaDnn< T > cuda, Log log, ProjectEx p, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
 Create a new Solver based on the project containing the SolverParameter. More...
 
static SGDSolver< T > Create (CudaDnn< T > cuda, Log log, SolverParameter solverParam, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
 Create a new Solver based on the project containing the SolverParameter. More...
 
- Events inherited from MyCaffe.solvers.Solver< T >
EventHandler OnStart
 The OnStart event fires at the start of each training iteration. More...
 
EventHandler OnAborted
 The OnAborted event fires after aborting a training cycle. More...
 
EventHandler< GradientsReadyArgsOnGradientsReady
 The OnGradientsReady event fires after the gradients of a Solver are ready for distribution to other Solvers in a multi-GPU training session. More...
 
EventHandler< SnapshotArgsOnSnapshot
 The OnSnapshot event fires when the Solver detects that a snapshot is needed. More...
 
EventHandler< TrainingIterationArgs< T > > OnTrainingIteration
 The OnTrainingIteration event fires at the end of each training iteration. More...
 
EventHandler< TestingIterationArgs< T > > OnTestingIteration
 The OnTestingIteration event fires at the end of each testing iteration. More...
 
EventHandler< TestResultArgs< T > > OnTestResults
 When specified, the OnTestResults event fires after each single test run. The recipient is responsible for setting the accuracy. More...
 
EventHandler< TestArgsOnTest
 When specified, the OnTest event fires during a TestAll and overrides the call to Test. More...
 
EventHandler OnTestStart
 The OnTestStart event fires at the start of each testing iteration. More...
 
EventHandler< CustomForwardBackArgs< T > > OnCustomForwardBack
 The OnCustomForwardBack allows for overriding the forward/backward operations within the solver. More...
 
EventHandler< WorkspaceArgsOnGetWorkspace
 Specifies the OnGetWorkspace event that fires when the getWorkspace() function is called by a layer to get a shareable workspace to conserve GPU memory. More...
 
EventHandler< WorkspaceArgsOnSetWorkspace
 Specifies the OnSetWorkspace event that fires when the setWorkspace() function is called by a layer to get a shareable workspace to conserve GPU memory. More...
 

Detailed Description

Stochastic Gradient Descent solver with momentum updates weights by a linear combination of the negative gradient and the previous weight update.

See also
Stochastic Gradient Descent Wikipedia.
Template Parameters
TSpecifies the base type float or double. Using float is recommended to conserve GPU memory.

Definition at line 21 of file SGDSolver.cs.

Constructor & Destructor Documentation

◆ SGDSolver()

MyCaffe.solvers.SGDSolver< T >.SGDSolver ( CudaDnn< T >  cuda,
Log  log,
SolverParameter  p,
CancelEvent  evtCancel,
AutoResetEvent  evtForceSnapshot,
AutoResetEvent  evtForceTest,
IXDatabaseBase  db,
IXPersist< T >  persist,
int  nSolverCount = 1,
int  nSolverRank = 0,
Net< T >  shareNet = null,
onGetWorkspace  getws = null,
onSetWorkspace  setws = null 
)

The SGDSolver constructor.

Parameters
cudaSpecifies the instance of CudaDnn to use.
logSpecifies the Log for output.
pSpecifies teh SolverParameter.
evtCancelSpecifies a CancelEvent used to cancel the current operation (e.g. training, testing) for which the Solver is performing.
evtForceSnapshotSpecifies an automatic reset event that causes the Solver to perform a Snapshot when set.
evtForceTestSpecifies an automatic reset event that causes teh Solver to run a testing cycle when set.
dbSpecifies the in-memory MyCaffeDatabase.
persistSpecifies the peristence used for loading and saving weights.
nSolverCountSpecifies the number of Solvers participating in a multi-GPU session.
nSolverRankSpecifies the rank of this Solver in a multi-GPU session.
shareNetOptionally, specifies the net to share when creating the training network (default = null, meaning no share net is used).
getwsOptionally, specifies the handler for getting the workspace.
setwsOptionally, specifies the handler for setting the workspace.

Definition at line 55 of file SGDSolver.cs.

Member Function Documentation

◆ ApplyUpdate()

override double MyCaffe.solvers.SGDSolver< T >.ApplyUpdate ( int  nIterationOverride = -1)
virtual

Compute the update values and apply them to the training Net.

Parameters
nIterationOverrideOptionally, specifies an iteration override, or -1 which is ignored.
Returns
The learning rate used is returned.

Implements MyCaffe.solvers.Solver< T >.

Definition at line 192 of file SGDSolver.cs.

◆ ClipGradients()

virtual void MyCaffe.solvers.SGDSolver< T >.ClipGradients ( )
virtual

Clip the gradients of all learnable blobs in the training Net.

Definition at line 340 of file SGDSolver.cs.

◆ ComputeUpdateValue()

virtual void MyCaffe.solvers.SGDSolver< T >.ComputeUpdateValue ( int  param_id,
double  dfRate,
int  nIterationOverride = -1 
)
virtual

Compute the SGD update value that will be applied to a learnable blobs in the training Net.

Parameters
param_idSpecifies the id of the Blob.
dfRateSpecifies the learning rate.
nIterationOverrideOptionally, specifies an iteration override, or -1 which is ignored.

Reimplemented in MyCaffe.solvers.AdaDeltaSolver< T >, MyCaffe.solvers.AdaGradSolver< T >, MyCaffe.solvers.AdamSolver< T >, MyCaffe.solvers.AdamWSolver< T >, MyCaffe.solvers.NesterovSolver< T >, and MyCaffe.solvers.RmsPropSolver< T >.

Definition at line 321 of file SGDSolver.cs.

◆ dispose()

override void MyCaffe.solvers.SGDSolver< T >.dispose ( )
protectedvirtual

Releases all resources (GPU and Host) used by the Solver.

Reimplemented from MyCaffe.solvers.Solver< T >.

Definition at line 64 of file SGDSolver.cs.

◆ GetLearningRate()

double MyCaffe.solvers.SGDSolver< T >.GetLearningRate ( int  nIterationOverride = -1)

Return the current learning rate.

The currently implemented learning rate policies are as follows:

  • fixed: always return $ base_lr $.
  • step: return $ base_lr * gamma ^ {floor{iter / step}} $
  • exp: return $ base_lr * gamma ^ iter $
  • inv: return $ base_lr * {1 + gamma * iter} ^ {-power} $
  • multistep: similar to step but it allows non-uniform steps defined by stepvalue.
  • poly: the effective learning rate follows a polynomial decay, to be zero by the max_iter. return $ base_lr * {1 - iter/max_iter} ^ {power} $
  • sigmoid: the effective learning rate follows a sigmoid decay. return $ base_lr * {1/{1 + exp{-gamma * {iter - stepsize}}}} $

where base_lr, max_iter, gamma, step, stepvalue and power are defined int the solver protocol buffer, and iter is the current iteration.

Parameters
nIterationOverrideOptionally, specifies an iteration override, or -1 which is ignored.
Returns
The learning rate value.

Definition at line 129 of file SGDSolver.cs.

◆ Normalize()

virtual void MyCaffe.solvers.SGDSolver< T >.Normalize ( int  param_id)
virtual

Normalize a learnable Blob of the training Net.

Parameters
param_idSpecifies the id of the Blob.

Definition at line 268 of file SGDSolver.cs.

◆ PreSolve()

void MyCaffe.solvers.SGDSolver< T >.PreSolve ( )

Runs the pre-solve which prepares the Solver to start Solving.

Definition at line 92 of file SGDSolver.cs.

◆ Regularize()

virtual void MyCaffe.solvers.SGDSolver< T >.Regularize ( int  param_id)
virtual

Regularize a learnable Blob of the training net.

Parameters
param_idSpecifies the id of the Blob.

Definition at line 287 of file SGDSolver.cs.

◆ RestoreSolverState()

override void MyCaffe.solvers.SGDSolver< T >.RestoreSolverState ( byte[]  rgState)
protectedvirtual

Restore the state of the Solver.

Parameters
rgStateSpecifies the state of the Solver.

Implements MyCaffe.solvers.Solver< T >.

Definition at line 230 of file SGDSolver.cs.

◆ SnapshotSolverState()

override byte[] MyCaffe.solvers.SGDSolver< T >.SnapshotSolverState ( )
protectedvirtual

Take a snapshot of the Solver state.

Returns
The Solver state snapshot is returned.

Implements MyCaffe.solvers.Solver< T >.

Definition at line 250 of file SGDSolver.cs.

Member Data Documentation

◆ m_colHistory

BlobCollection<T> MyCaffe.solvers.SGDSolver< T >.m_colHistory = new BlobCollection<T>()
protected

History maintains the historical momentum data.

Definition at line 26 of file SGDSolver.cs.

◆ m_colTemp

BlobCollection<T> MyCaffe.solvers.SGDSolver< T >.m_colTemp = new BlobCollection<T>()
protected

Update maintains update related data and is not needed in snapshots.

Temp maintains other information that might be needed in computation of gradients/updates and is not needed in snapshots.

Definition at line 37 of file SGDSolver.cs.

Property Documentation

◆ history

Returns the history BlobCollection containing historical momentum data.

Definition at line 84 of file SGDSolver.cs.


The documentation for this class was generated from the following file: