MyCaffe  1.12.2.41
Deep learning software for Windows C# programmers.
MyCaffe.solvers.AdamSolver< T > Class Template Reference

Use Adam Solver which uses gradient based optimization like SGD that includes 'adaptive momentum estimation' and can be thought of as a generalization of AdaGrad. More...

Inheritance diagram for MyCaffe.solvers.AdamSolver< T >:
MyCaffe.solvers.SGDSolver< T > MyCaffe.solvers.Solver< T >

Public Member Functions

 AdamSolver (CudaDnn< T > cuda, Log log, SolverParameter p, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
 The AdamSolver constructor. More...
 
virtual void AdamPreSolve ()
 Runs the AdamSolver pre-solve which parpares the Solver to start Solving. More...
 
override void ComputeUpdateValue (int param_id, double dfRate, int nIterationOverride=-1)
 Compute the AdamSolver update value that will be applied to a learnable blobs in the training Net. More...
 
- Public Member Functions inherited from MyCaffe.solvers.SGDSolver< T >
 SGDSolver (CudaDnn< T > cuda, Log log, SolverParameter p, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
 The SGDSolver constructor. More...
 
void PreSolve ()
 Runs the pre-solve which prepares the Solver to start Solving. More...
 
double GetLearningRate (int nIterationOverride=-1)
 Return the current learning rate. More...
 
override double ApplyUpdate (int nIterationOverride=-1)
 Compute the update values and apply them to the training Net. More...
 
virtual void Normalize (int param_id)
 Normalize a learnable Blob of the training Net. More...
 
virtual void Regularize (int param_id)
 Regularize a learnable Blob of the training net. More...
 
virtual void ClipGradients ()
 Clip the gradients of all learnable blobs in the training Net. More...
 
- Public Member Functions inherited from MyCaffe.solvers.Solver< T >
 Solver (CudaDnn< T > cuda, Log log, SolverParameter p, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
 The Solver constructor. More...
 
void Dispose ()
 Discards the resources (GPU and Host) used by this Solver. More...
 
bool ForceOnTrainingIterationEvent ()
 Force an OnTrainingIterationEvent to fire. More...
 
void Init (SolverParameter p, Net< T > shareNet=null)
 Initializes the Solver. More...
 
void Reset ()
 Reset the iterations of the net. More...
 
virtual void Solve (int nIterationOverride=-1, byte[] rgWeights=null, byte[] rgState=null, TRAIN_STEP step=TRAIN_STEP.NONE)
 The main entry of the solver function. In default, iter will be zero. Pass in a non-zero iter number to resume training for a pre-trained net. More...
 
bool Step (int nIters, TRAIN_STEP step=TRAIN_STEP.NONE, bool bZeroDiffs=true, bool bApplyUpdates=true, bool bDisableOutput=false, bool bDisableProgress=false, double? dfLossOverride=null, bool? bAllowSnapshot=null)
 Steps a set of iterations through a training cycle. More...
 
void Restore (byte[] rgWeights, byte[] rgState, string strSkipBlobTypes=null)
 The restore method simply calls the RestoreSolverState method of the inherited class. More...
 
void Snapshot (bool bForced, bool bScheduled, bool bUpdateDatabase=true)
 The snapshot function implements the basic snapshotting utility that stores the learned net. This method calls the SnapshotSolverState method of the inherited class. More...
 
SnapshotArgs GetSnapshotArgs (byte[] rgState, byte[] rgWeights, double dfAccuracy, double dfError, int nIteration, SNAPSHOT_WEIGHT_UPDATE_METHOD wtUpdt)
 The GetSnapshotArgs method fills out a snapshot args structure. More...
 
double TestAll (int nIterationOverride=-1)
 Run a TestAll by running all test Nets. More...
 
double TestDetection (int nIterationOverride=-1, int nTestNetId=0)
 Run an SSD detection test on a given test Net by running it through its iterations. More...
 
double TestClassification (int nIterationOverride=-1, int nTestNetId=0)
 Run a test on a given test Net by running it through its iterations. More...
 
void UpdateSmoothedLoss (double dfLoss, int nStartIter, int nAverageLoss=0)
 Update the avaraged loss value. More...
 

Additional Inherited Members

- Static Public Member Functions inherited from MyCaffe.solvers.Solver< T >
static SGDSolver< T > Create (CudaDnn< T > cuda, Log log, ProjectEx p, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
 Create a new Solver based on the project containing the SolverParameter. More...
 
static SGDSolver< T > Create (CudaDnn< T > cuda, Log log, SolverParameter solverParam, CancelEvent evtCancel, AutoResetEvent evtForceSnapshot, AutoResetEvent evtForceTest, IXDatabaseBase db, IXPersist< T > persist, int nSolverCount=1, int nSolverRank=0, Net< T > shareNet=null, onGetWorkspace getws=null, onSetWorkspace setws=null)
 Create a new Solver based on the project containing the SolverParameter. More...
 
- Protected Member Functions inherited from MyCaffe.solvers.SGDSolver< T >
override void dispose ()
 Releases all resources (GPU and Host) used by the Solver. More...
 
override void RestoreSolverState (byte[] rgState)
 Restore the state of the Solver. More...
 
override byte[] SnapshotSolverState ()
 Take a snapshot of the Solver state. More...
 
- Protected Member Functions inherited from MyCaffe.solvers.Solver< T >
void InitTrainNet (Net< T > shareNet=null)
 Initializes the Net used by the solver for training. More...
 
void InitTestNets ()
 Initializes the Net used by the Solver for testing. More...
 
- Protected Attributes inherited from MyCaffe.solvers.SGDSolver< T >
BlobCollection< T > m_colHistory = new BlobCollection<T>()
 History maintains the historical momentum data. More...
 
BlobCollection< T > m_colTemp = new BlobCollection<T>()
 Update maintains update related data and is not needed in snapshots. More...
 
- Protected Attributes inherited from MyCaffe.solvers.Solver< T >
CudaDnn< T > m_cuda
 Specifies the instance of CudaDnn used by the Solver that provides a connection to Cuda. More...
 
Log m_log
 Specifies the Log for output. More...
 
SolverParameter m_param
 Specifies the SolverParameter that defines how the Solver operates. More...
 
Net< T > m_net
 Specifies the training Net. More...
 
List< Net< T > > m_rgTestNets = new List<Net<T>>()
 Specifies the testing Nets. More...
 
int m_nIter
 Specifies the current iteration. More...
 
int m_nCurrentStep
 Specifies the current step. More...
 
List< double > m_rgLosses = new List<double>()
 Specifies the Losses used to calculate the smoothed Loss. More...
 
double m_dfSmoothedLoss = 0
 Specifies the smoothed loss protected for derived classes to use. More...
 
double? m_dfIterAccuracy = null
 Specifies the iteration accuracy calculated when a blob exists with the name 'accuracy'. More...
 
int m_nSolverCount = 1
 Specifies the Solver count in a multi-GPU training session. More...
 
int m_nSolverRank = 0
 Specifies the Solver rank of this solver, where rank == 0 is the root Solver. More...
 
IXPersist< T > m_persist
 Specifies the persistance object used to save weight and solver states. More...
 
double m_dfLearningRateOverride = 0
 Optionally, specifies a learning rate override (default = 0, which ignores this setting). More...
 
- Properties inherited from MyCaffe.solvers.SGDSolver< T >
BlobCollection< T > history [get]
 Returns the history BlobCollection containing historical momentum data. More...
 
- Properties inherited from MyCaffe.solvers.Solver< T >
double LearningRateOverride [getset]
 Get/set the learning rate override. When 0, this setting is ignored. More...
 
int TrainingTimeLimitInMinutes [getset]
 Get/set the training time limit in minutes. When set to 0, no time limit is imposed on training. More...
 
SNAPSHOT_WEIGHT_UPDATE_METHOD SnapshotWeightUpdateMethod [getset]
 Get/set the snapshot weight update method. More...
 
IXDatabaseBase Database [get]
 Returns the in-memory MyCaffeDatabase used. More...
 
bool EnableTesting [getset]
 When enabled, the training cycle calls TestAll periodically based on the SolverParameter. Otherwise testing is not performed. More...
 
bool EnableBlobDebugging [getset]
 When enabled, the OnTrainingIteration event is set extra debugging information describing the state of each Blob used by the Solver. More...
 
bool EnableLayerDebugging [getset]
 Enable/disable layer debugging which causes each layer to check for NAN/INF on each forward/backward pass and throw an exception when found. More...
 
bool EnableBreakOnFirstNaN [getset]
 When enabled (requires EnableBlobDebugging = true), the Solver immediately stop training upon detecting the first NaN withing the training Net. More...
 
bool EnableDetailedNanDetection [getset]
 When enabled (requires EnableBlobDebugging = true), the detailed Nan (and Infinity) detection is perofmed on each blob when training Net. More...
 
bool EnableSingleStep [getset]
 When enabled (requires EnableBlobDebugging = true), the Solver only runs one training cycle. More...
 
bool WeightsUpdated [getset]
 Get/set when the weights have been updated. More...
 
object Tag [getset]
 Returns a generic tag associated with the Solver. More...
 
Net< T > TestingNet [get]
 Returns the testing Net used by the solver. More...
 
Net< T > TrainingNet [get]
 Returns the training Net used by the solver. More...
 
CudaDnn< T > Cuda [get]
 Returns the CudaDnn instance used by the Solver. More...
 
string ActiveLabelCounts [get]
 Returns a string describing the labels detected in the training along with the % that each label has participated in the training. More...
 
string LabelQueryHitPercents [get]
 Return the label query hit percentages for the active datasource. More...
 
string LabelQueryEpochs [get]
 Return the label query epochs for the active datasource. More...
 
int CurrentIteration [get]
 Returns the current training iteration. More...
 
int MaximumIteration [get]
 Returns the maximum training iterations. More...
 
int TrainingIterations [get]
 Returns the current training iterations remaining. More...
 
int? TestingIterations [get]
 Returns the current testing iterations remaining. More...
 
int TrainingIterationOverride [getset]
 Get/set the training iteration override. More...
 
int TestingIterationOverride [getset]
 Get/set the testing iteration override. More...
 
AutoResetEvent CompletedEvent [get]
 Returns an auto reset event that is set upon training completion. More...
 
CancelEvent CancelEvent [get]
 Returns the cancel event which when set cancels the current operation run by the Solver. More...
 
double smoothed_loss [get]
 Returns the smoothed loss. More...
 
SolverParameter parameter [get]
 Returns the SolverParameter used. More...
 
Net< T > net [get]
 Returns the main training Net. More...
 
List< Net< T > > test_nets [get]
 Returns the testing Nets. More...
 
int iter [get]
 Returns the current training iteration. More...
 
SolverParameter.SolverType type [get]
 Returns the type of solver. More...
 
bool forceSnapshot [get]
 Returns whether or not a snapshot has been forced. More...
 
bool forceTest [get]
 Returns whether or not a test has been forced. More...
 
int solver_count [get]
 Returns the solver count in a multi-GPU session. More...
 
int solver_rank [get]
 Returns this Solver's rank in a multi-GPU session. More...
 
bool? is_root_solver [get]
 Returns whether or not this is the root solver. More...
 
- Events inherited from MyCaffe.solvers.Solver< T >
EventHandler OnStart
 The OnStart event fires at the start of each training iteration. More...
 
EventHandler OnAborted
 The OnAborted event fires after aborting a training cycle. More...
 
EventHandler< GradientsReadyArgsOnGradientsReady
 The OnGradientsReady event fires after the gradients of a Solver are ready for distribution to other Solvers in a multi-GPU training session. More...
 
EventHandler< SnapshotArgsOnSnapshot
 The OnSnapshot event fires when the Solver detects that a snapshot is needed. More...
 
EventHandler< TrainingIterationArgs< T > > OnTrainingIteration
 The OnTrainingIteration event fires at the end of each training iteration. More...
 
EventHandler< TestingIterationArgs< T > > OnTestingIteration
 The OnTestingIteration event fires at the end of each testing iteration. More...
 
EventHandler< TestResultArgs< T > > OnTestResults
 When specified, the OnTestResults event fires after each single test run. The recipient is responsible for setting the accuracy. More...
 
EventHandler< TestArgsOnTest
 When specified, the OnTest event fires during a TestAll and overrides the call to Test. More...
 
EventHandler OnTestStart
 The OnTestStart event fires at the start of each testing iteration. More...
 
EventHandler< CustomForwardBackArgs< T > > OnCustomForwardBack
 The OnCustomForwardBack allows for overriding the forward/backward operations within the solver. More...
 
EventHandler< WorkspaceArgsOnGetWorkspace
 Specifies the OnGetWorkspace event that fires when the getWorkspace() function is called by a layer to get a shareable workspace to conserve GPU memory. More...
 
EventHandler< WorkspaceArgsOnSetWorkspace
 Specifies the OnSetWorkspace event that fires when the setWorkspace() function is called by a layer to get a shareable workspace to conserve GPU memory. More...
 

Detailed Description

Use Adam Solver which uses gradient based optimization like SGD that includes 'adaptive momentum estimation' and can be thought of as a generalization of AdaGrad.

See also
Adam: A Method for Stochastic Optimization by Diederik P. Kingma, and Jimmy Ba, 2014.
Template Parameters
TSpecifies the base type float or double. Using float is recommended to conserve GPU memory.

Definition at line 21 of file AdamSolver.cs.

Constructor & Destructor Documentation

◆ AdamSolver()

MyCaffe.solvers.AdamSolver< T >.AdamSolver ( CudaDnn< T >  cuda,
Log  log,
SolverParameter  p,
CancelEvent  evtCancel,
AutoResetEvent  evtForceSnapshot,
AutoResetEvent  evtForceTest,
IXDatabaseBase  db,
IXPersist< T >  persist,
int  nSolverCount = 1,
int  nSolverRank = 0,
Net< T >  shareNet = null,
onGetWorkspace  getws = null,
onSetWorkspace  setws = null 
)

The AdamSolver constructor.

Parameters
cudaSpecifies the instance of CudaDnn to use.
logSpecifies the Log for output.
pSpecifies teh SolverParameter.
evtCancelSpecifies a CancelEvent used to cancel the current operation (e.g. training, testing) for which the Solver is performing.
evtForceSnapshotSpecifies an automatic reset event that causes the Solver to perform a Snapshot when set.
evtForceTestSpecifies an automatic reset event that causes teh Solver to run a testing cycle when set.
dbSpecifies the in-memory MyCaffeDatabase.
persistSpecifies the peristence used for loading and saving weights.
nSolverCountSpecifies the number of Solvers participating in a multi-GPU session.
nSolverRankSpecifies the rank of this Solver in a multi-GPU session.
shareNetOptionally, specifies the net to share when creating the training network (default = null, meaning no share net is used).
getwsOptionally, specifies the handler for getting the workspace.
setwsOptionally, specifies the handler for setting the workspace.

Definition at line 39 of file AdamSolver.cs.

Member Function Documentation

◆ AdamPreSolve()

virtual void MyCaffe.solvers.AdamSolver< T >.AdamPreSolve ( )
virtual

Runs the AdamSolver pre-solve which parpares the Solver to start Solving.

Definition at line 48 of file AdamSolver.cs.

◆ ComputeUpdateValue()

override void MyCaffe.solvers.AdamSolver< T >.ComputeUpdateValue ( int  param_id,
double  dfRate,
int  nIterationOverride = -1 
)
virtual

Compute the AdamSolver update value that will be applied to a learnable blobs in the training Net.

Parameters
param_idSpecifies the id of the Blob.
dfRateSpecifies the learning rate.
nIterationOverrideOptionally, specifies an iteration override, or -1 which is ignored.

Reimplemented from MyCaffe.solvers.SGDSolver< T >.

Definition at line 68 of file AdamSolver.cs.


The documentation for this class was generated from the following file: