MyCaffe  0.11.2.9-beta1
Deep learning software for Windows C# programmers.
MyCaffe.layers.DataLayer< T > Class Template Reference

The DataLayer loads data from the IXImageDatabase database. This layer is initialized with the MyCaffe.param.DataParameter. More...

Inheritance diagram for MyCaffe.layers.DataLayer< T >:
MyCaffe.layers.BasePrefetchingDataLayer< T > MyCaffe.layers.BaseDataLayer< T > MyCaffe.layers.Layer< T >

Public Member Functions

 DataLayer (CudaDnn< T > cuda, Log log, LayerParameter p, IXImageDatabaseBase db, CancelEvent evtCancel)
 The DataLayer constructor. More...
 
void Connect (DataLayer< T > src)
 The Connect method connects one Data Layer to another so that they can synchronize. More...
 
void Disconnect ()
 Disconnect any previously connected Data Layers. More...
 
- Public Member Functions inherited from MyCaffe.layers.BasePrefetchingDataLayer< T >
 BasePrefetchingDataLayer (CudaDnn< T > cuda, Log log, LayerParameter p, IXImageDatabaseBase db, CancelEvent evtCancel)
 The BaseDataLayer constructor. More...
 
override void LayerSetUp (BlobCollection< T > colBottom, BlobCollection< T > colTop)
 LayerSetUp implements common data layer setup functonality, and calls DatLayerSetUp to do special data layer setup for individual layer types. More...
 
- Public Member Functions inherited from MyCaffe.layers.BaseDataLayer< T >
 BaseDataLayer (CudaDnn< T > cuda, Log log, LayerParameter p, IXImageDatabaseBase db)
 The BaseDataLayer constructor. More...
 
override void Reshape (BlobCollection< T > colBottom, BlobCollection< T > colTop)
 Data layers have no bottoms, so reshaping is trivial. More...
 
- Public Member Functions inherited from MyCaffe.layers.Layer< T >
 Layer (CudaDnn< T > cuda, Log log, LayerParameter p)
 The Layer constructor. More...
 
void Dispose ()
 Releases all GPU and host resources used by the Layer. More...
 
virtual bool ReInitializeParameters (WEIGHT_TARGET target)
 Re-initialize the parameters of the layer. More...
 
void SetPhase (Phase phase)
 Changes the layer's Phase to the one specified. More...
 
void Setup (BlobCollection< T > colBottom, BlobCollection< T > colTop)
 Implements common Layer setup functionality. More...
 
virtual void SetNetParameterUsed (NetParameter np)
 This function allows other layers to gather needed information from the NetParameters if any, and is called when initialzing the Net. More...
 
void ConvertToBase (BlobCollection< T > col)
 ConvertToBase converts any blobs in a collection that are in half size to the base size. More...
 
double Forward (BlobCollection< T > colBottom, BlobCollection< T > colTop)
 Given the bottom (input) Blobs, this function computes the top (output) Blobs and the loss. More...
 
void Backward (BlobCollection< T > colTop, List< bool > rgbPropagateDown, BlobCollection< T > colBottom)
 Given the top Blob error gradients, compute the bottom Blob error gradients. More...
 
double loss (int nTopIdx)
 Returns the scalar loss associated with the top Blob at a given index. More...
 
void set_loss (int nTopIdx, double dfLoss)
 Sets the loss associated with a top Blob at a given index. More...
 
virtual bool AllowForceBackward (int nBottomIdx)
 Return whether to allow More...
 
bool param_propagate_down (int nParamIdx)
 Returns whether or not the Layer should compute gradients w.r.t. a parameter at a particular index given by a parameter index. More...
 
void set_param_propagate_down (int nParamIdx, bool bPropagate)
 Sets whether or not the Layer should compute gradients w.r.t. a parameter at a particular index given by a parameter index. More...
 
void SetEnablePassthrough (bool bEnable)
 Enables/disables the pass-through mode. More...
 

Protected Member Functions

override void preStop ()
 The preStop override is called just before stopping the internal thread managed by the base class. More...
 
override void dispose ()
 Releases all GPU and host resources used by the Layer. More...
 
override void DataLayerSetUp (BlobCollection< T > colBottom, BlobCollection< T > colTop)
 Setup the DataLayer by starting up the pre-fetching. More...
 
void Next ()
 Retrieves the next item from the database and rolls the cursor over once the end of the dataset is reached. More...
 
bool Skip ()
 Skip to the next value - used when training in a multi-GPU scenario. More...
 
override void final_process (Blob< T > blobTop)
 Provides a final processing step that takes place at the end of the base class forward = this is where we apply the mask if one exists and is enabled. More...
 
override void load_batch (Batch< T > batch)
 Load a batch of data in the background (this is run on an internal thread within the BasePrefetchingDataLayer class). More...
 
- Protected Member Functions inherited from MyCaffe.layers.BasePrefetchingDataLayer< T >
void statupPrefetch ()
 Starts the prefetch thread. More...
 
override void forward (BlobCollection< T > colBottom, BlobCollection< T > colTop)
 The forward override implements the functionality to load pre-fetch data and feed it into the top (output) Blobs. More...
 
- Protected Member Functions inherited from MyCaffe.layers.BaseDataLayer< T >
virtual bool setupSourceDescriptor ()
 Allows any derivative classes to pre-initialize the m_src which is used in LayerSetup before the DataLayerSetup. More...
 
override void backward (BlobCollection< T > colTop, List< bool > rgbPropagateDown, BlobCollection< T > colBottom)
 Not implemented - data Layers do not perform backward.
 
- Protected Member Functions inherited from MyCaffe.layers.Layer< T >
GetIterationArgs getCurrentIteration ()
 Fires the OnGetIteration event to query the current iteration. More...
 
long convert_to_full (int nCount, long hMem)
 Convert half memory to full memory. More...
 
void convert (BlobCollection< T > col)
 Convert a collection of blobs from / to half size. More...
 
bool reshapeNeeded (BlobCollection< T > colBottom, BlobCollection< T > colTop, bool bReset=true)
 Tests the shapes of both the bottom and top blobs and if they are the same as the previous sizing, returns false indicating that no reshape is needed. More...
 
bool compareShapes (BlobCollection< T > colBottom, BlobCollection< T > colTop)
 Compare the shapes of the top and bottom and if the same, return true, otherwise false. More...
 
void CheckBlobCounts (BlobCollection< T > colBottom, BlobCollection< T > colTop)
 Called by the Layer::Setup function to check the number of bottom (input) and top (output) Blobs provided match the expected number of blobs expected via the {EactNum,Min,Max}{Bottom,Top}Blobs functions. More...
 
void SetLossWeights (BlobCollection< T > colTop)
 Called by Layer::Setup to initialize the weights associated with any top (output) Blobs in the loss function ans store non-zero loss weights in the diff Blob. More...
 
bool shareParameter (Blob< T > b, List< int > rgMinShape)
 Attempts to share a parameter Blob if another parameter Blob with the same name and accpetable size is found. More...
 
bool shareLayerBlob (Blob< T > b, List< int > rgMinShape)
 Attempts to share a Layer Blob if another parameter Blob with the same name and acceptable size is found. More...
 
bool shareLayerBlobs (Layer< T > layer)
 Attempts to share the Layer blobs and internal_blobs with matching names and sizes with those in another matching layer. More...
 
virtual WorkspaceArgs getWorkspace ()
 Returns the WorkspaceArgs used to share a workspace between Layers. More...
 
virtual bool setWorkspace (ulong lSize)
 Sets the workspace size (in items) and returns true if set, false otherwise. More...
 
void check_nan (Blob< T > b)
 Checks a Blob for NaNs and throws an exception if found. More...
 
convert (double df)
 Converts a double to a generic. More...
 
double convertD (T df)
 Converts a generic to a double value. More...
 
float convertF (T df)
 Converts a generic to a float value. More...
 
double [] convertD (T[] rg)
 Converts an array of generic values into an array of double values. More...
 
T [] convert (double[] rg)
 Converts an array of double values into an array of generic values. More...
 
float [] convertF (T[] rg)
 Converts an array of float values into an array of generic values. More...
 
T [] convert (float[] rg)
 Converts an array of float values into an array of generic values. More...
 
int val_at (T[] rg, int nIdx)
 Returns the integer value at a given index in a generic array. More...
 
Size size_at (Blob< T > b)
 Returns the Size of a given two element Blob, such as one that stores Blob size information. More...
 

Protected Attributes

DB< T > m_db
 Specifies the database. More...
 
Cursor< T > m_cursor
 Specifies the database used to traverse through the database. More...
 
Stopwatch m_swTimerBatch
 Specifies a first timer used to calcualte the batch time. More...
 
Stopwatch m_swTimerTransaction
 Specfies a second timer used to calculate the transaction time. More...
 
double m_dfReadTime
 Specifies the read time. More...
 
double m_dfTransTime
 Specifies the transaction time. More...
 
- Protected Attributes inherited from MyCaffe.layers.BasePrefetchingDataLayer< T >
Batch< T > [] m_rgPrefetch
 Specifies the pre-fetch cache. More...
 
CancelEvent m_evtCancel
 Specifies the cancellation event for the internal thread. More...
 
- Protected Attributes inherited from MyCaffe.layers.BaseDataLayer< T >
DataTransformer< T > m_transformer
 Specifies the DataTransformer used to transform each data item as it loaded. More...
 
bool m_bOutputLabels
 Specifies whether or not the Layer should output labels. More...
 
IXImageDatabaseBase m_imgdb
 Specifies the MyCaffeImageDatabase. More...
 
SourceDescriptor m_src
 Specifies the SourceDescriptor of the data source. More...
 
SimpleDatum m_imgMean = null
 Specifies the SimpleDatum that optionally contains the image Mean for data centering. More...
 
- Protected Attributes inherited from MyCaffe.layers.Layer< T >
LayerParameter.LayerType m_type = LayerParameter.LayerType._MAX
 Specifies the Layer type. More...
 
CudaDnn< T > m_cuda
 Specifies the CudaDnn connection to Cuda. More...
 
Log m_log
 Specifies the Log for output. More...
 
LayerParameter m_param
 Specifies the LayerParameter describing the Layer. More...
 
Phase m_phase
 Specifies the Phase under which the Layer is run. More...
 
BlobCollection< T > m_colBlobs
 Specifies the learnable parameter Blobs of the Layer. More...
 
DictionaryMap< bool > m_rgbParamPropagateDown
 Specifies whether or not to compute the learnable diff of each parameter Blob. More...
 
DictionaryMap< double > m_rgLoss
 Specifies the loss values that indeicate whether each top (output) Blob has a non-zero weight in the objective function.. More...
 
m_tOne
 Specifies a generic type equal to 1.0. More...
 
m_tZero
 Specifies a generic type equal to 0.0. More...
 
bool m_bEnablePassthrough = false
 Enables/disables the pass-through mode for the layer. Default = false. More...
 
bool m_bUseHalfSize = false
 Specifies that the half size of the top (if any) should be converted to the base size. More...
 
bool m_bConvertTopOnFwd = false
 Specifies whether or not the layer should convert the top on the forward pass when using half sized memory (typically only done with input data). More...
 
bool m_bConvertTopOnBwd = true
 Specifies whether or not to convert the top on the backward pass when using half sized memory (typically not done on loss layers). More...
 
bool m_bConvertBottom = true
 Specifies whether or not the layer should convert the bottom when using half sized memory. More...
 
bool m_bReshapeOnForwardNeeded = true
 Specifies whether or not the reshape on forward is needed or not. More...
 

Properties

override bool delayPrefetch [get]
 Specifies to delay the prefetch when using a synchronized Data Layer. More...
 
override int ExactNumBottomBlobs [get]
 No bottom blobs are used by this layer. More...
 
override int MinTopBlobs [get]
 Specifies the minimum number of required top (output) Blobs: data More...
 
override int MaxTopBlobs [get]
 Specifies the maximum number of required top (output) Blobs: data, label More...
 
- Properties inherited from MyCaffe.layers.BasePrefetchingDataLayer< T >
virtual bool delayPrefetch [get]
 Specifies whether or not to delay the prefetch. More...
 
- Properties inherited from MyCaffe.layers.BaseDataLayer< T >
SimpleDatum ImageMean [get, set]
 Get/set the image mean. More...
 
DataTransformer< T > Transformer [get]
 Returns the data transformer used. More...
 
- Properties inherited from MyCaffe.layers.Layer< T >
BlobCollection< T > blobs [get]
 Returns the collection of learnable parameter Blobs for the Layer. More...
 
virtual BlobCollection< T > internal_blobs [get]
 Returns the collection of internal Blobs used by the Layer. More...
 
LayerParameter layer_param [get]
 Returns the LayerParameter for this Layer. More...
 
LayerParameter.LayerType type [get]
 Returns the LayerType of this Layer. More...
 
virtual int ExactNumBottomBlobs [get]
 Returns the exact number of bottom (input) Blobs required by the Layer, or -1 if no exact number is required. More...
 
virtual int MinBottomBlobs [get]
 Returns the minimum number of bottom (input) Blobs required by the Layer, or -1 if no minimum number is required. More...
 
virtual int MaxBottomBlobs [get]
 Returns the maximum number of bottom (input) Blobs required by the Layer, or -1 if no maximum number is required. More...
 
virtual int ExactNumTopBlobs [get]
 Returns the exact number of top (output) Blobs required by the Layer, or -1 if no exact number is required. More...
 
virtual int MinTopBlobs [get]
 Returns the minimum number of top (output) Blobs required by the Layer, or -1 if no minimum number is required. More...
 
virtual int MaxTopBlobs [get]
 Returns the maximum number of top (output) Blobs required by the Layer, or -1 if no maximum number is required. More...
 
virtual bool EqualNumBottomTopBlobs [get]
 Returns true if the Layer requires and equal number of bottom (input) and top (output) Blobs. More...
 
virtual bool AutoTopBlobs [get]
 Return whether "anonymous" top (output) Blobs are created automatically by the Layer. More...
 
double forward_timing [get]
 Returns the timing of the last forward pass in milliseconds. More...
 
double forward_timing_average [get]
 Returns the average timing of the forward passes in milliseconds. More...
 
double backward_timing [get]
 Returns the timing of the last backward pass in milliseconds. More...
 
double backward_timing_average [get]
 Returns the average timing of the backward passes in milliseconds. More...
 

Events

EventHandler< LastBatchLoadedArgsOnBatchLoad
 This event fires (only when set) each time a batch is loaded form this dataset. More...
 
- Events inherited from MyCaffe.layers.Layer< T >
EventHandler< WorkspaceArgsOnGetWorkspace
 Specifies the OnGetWorkspace event that fires when the getWorkspace() function is called by a layer to get a shareable workspace to conserve GPU memory. More...
 
EventHandler< WorkspaceArgsOnSetWorkspace
 Specifies the OnSetWorkspace event that fires when the setWorkspace() function is called by a layer to get a shareable workspace to conserve GPU memory. More...
 
EventHandler< GetIterationArgsOnGetIteration
 Specifies the OnGetIteration event that fires when a layer needs to get the current iteration from the solver. More...
 
EventHandler< GetWorkBlobArgs< T > > OnDebug
 Specifies the OnGetWorkBlob event that is only supported when debugging to get a work blob from the primary Net holding this layer. More...
 

Additional Inherited Members

- Static Public Member Functions inherited from MyCaffe.layers.Layer< T >
static Layer< T > Create (CudaDnn< T > cuda, Log log, LayerParameter p, CancelEvent evtCancel, IXImageDatabaseBase imgDb=null, TransferInput trxinput=null)
 Create a new Layer based on the LayerParameter. More...
 

Detailed Description

The DataLayer loads data from the IXImageDatabase database. This layer is initialized with the MyCaffe.param.DataParameter.

Template Parameters
TSpecifies the base type float or double. Using float is recommended to conserve GPU memory.

Definition at line 23 of file DataLayer.cs.

Constructor & Destructor Documentation

◆ DataLayer()

MyCaffe.layers.DataLayer< T >.DataLayer ( CudaDnn< T >  cuda,
Log  log,
LayerParameter  p,
IXImageDatabaseBase  db,
CancelEvent  evtCancel 
)

The DataLayer constructor.

Parameters
cudaSpecifies the CudaDnn connection to Cuda.
logSpecifies the Log for output.
pSpecifies the LayerParameter data_param
dbSpecifies the external database to use.
evtCancelSpecifies the CancelEvent used to cancel any pre-fetching operations.

Definition at line 76 of file DataLayer.cs.

Member Function Documentation

◆ Connect()

void MyCaffe.layers.DataLayer< T >.Connect ( DataLayer< T >  src)

The Connect method connects one Data Layer to another so that they can synchronize.

Parameters
srcSpecifies the source Data Layer whos OnBatchLoad event fires and is handled by this Data Layer.

Definition at line 185 of file DataLayer.cs.

◆ DataLayerSetUp()

override void MyCaffe.layers.DataLayer< T >.DataLayerSetUp ( BlobCollection< T >  colBottom,
BlobCollection< T >  colTop 
)
protectedvirtual

Setup the DataLayer by starting up the pre-fetching.

Parameters
colBottomNot used.
colTopSpecifies the collection of top (output) Blobs.

Implements MyCaffe.layers.BaseDataLayer< T >.

Definition at line 216 of file DataLayer.cs.

◆ Disconnect()

void MyCaffe.layers.DataLayer< T >.Disconnect ( )

Disconnect any previously connected Data Layers.

Definition at line 197 of file DataLayer.cs.

◆ dispose()

override void MyCaffe.layers.DataLayer< T >.dispose ( )
protectedvirtual

Releases all GPU and host resources used by the Layer.

Reimplemented from MyCaffe.layers.BasePrefetchingDataLayer< T >.

Definition at line 136 of file DataLayer.cs.

◆ final_process()

override void MyCaffe.layers.DataLayer< T >.final_process ( Blob< T >  blobTop)
protectedvirtual

Provides a final processing step that takes place at the end of the base class forward = this is where we apply the mask if one exists and is enabled.

Parameters
blobTopSpecifies the top blob just about to be set out the forward operation as the Top[0] blob.

Reimplemented from MyCaffe.layers.BasePrefetchingDataLayer< T >.

Definition at line 513 of file DataLayer.cs.

◆ load_batch()

override void MyCaffe.layers.DataLayer< T >.load_batch ( Batch< T >  batch)
protectedvirtual

Load a batch of data in the background (this is run on an internal thread within the BasePrefetchingDataLayer class).

Parameters
batchSpecifies the Batch of data to load.

Implements MyCaffe.layers.BasePrefetchingDataLayer< T >.

Definition at line 582 of file DataLayer.cs.

◆ Next()

void MyCaffe.layers.DataLayer< T >.Next ( )
protected

Retrieves the next item from the database and rolls the cursor over once the end of the dataset is reached.

Definition at line 482 of file DataLayer.cs.

◆ preStop()

override void MyCaffe.layers.DataLayer< T >.preStop ( )
protectedvirtual

The preStop override is called just before stopping the internal thread managed by the base class.

Reimplemented from MyCaffe.layers.BasePrefetchingDataLayer< T >.

Definition at line 127 of file DataLayer.cs.

◆ Skip()

bool MyCaffe.layers.DataLayer< T >.Skip ( )
protected

Skip to the next value - used when training in a multi-GPU scenario.

Returns

Definition at line 499 of file DataLayer.cs.

Member Data Documentation

◆ m_cursor

Cursor<T> MyCaffe.layers.DataLayer< T >.m_cursor
protected

Specifies the database used to traverse through the database.

Definition at line 32 of file DataLayer.cs.

◆ m_db

DB<T> MyCaffe.layers.DataLayer< T >.m_db
protected

Specifies the database.

Definition at line 28 of file DataLayer.cs.

◆ m_dfReadTime

double MyCaffe.layers.DataLayer< T >.m_dfReadTime
protected

Specifies the read time.

Definition at line 45 of file DataLayer.cs.

◆ m_dfTransTime

double MyCaffe.layers.DataLayer< T >.m_dfTransTime
protected

Specifies the transaction time.

Definition at line 49 of file DataLayer.cs.

◆ m_swTimerBatch

Stopwatch MyCaffe.layers.DataLayer< T >.m_swTimerBatch
protected

Specifies a first timer used to calcualte the batch time.

Definition at line 37 of file DataLayer.cs.

◆ m_swTimerTransaction

Stopwatch MyCaffe.layers.DataLayer< T >.m_swTimerTransaction
protected

Specfies a second timer used to calculate the transaction time.

Definition at line 41 of file DataLayer.cs.

Property Documentation

◆ delayPrefetch

override bool MyCaffe.layers.DataLayer< T >.delayPrefetch
getprotected

Specifies to delay the prefetch when using a synchronized Data Layer.

Definition at line 169 of file DataLayer.cs.

◆ ExactNumBottomBlobs

override int MyCaffe.layers.DataLayer< T >.ExactNumBottomBlobs
get

No bottom blobs are used by this layer.

Definition at line 458 of file DataLayer.cs.

◆ MaxTopBlobs

override int MyCaffe.layers.DataLayer< T >.MaxTopBlobs
get

Specifies the maximum number of required top (output) Blobs: data, label

Definition at line 474 of file DataLayer.cs.

◆ MinTopBlobs

override int MyCaffe.layers.DataLayer< T >.MinTopBlobs
get

Specifies the minimum number of required top (output) Blobs: data

Definition at line 466 of file DataLayer.cs.

Event Documentation

◆ OnBatchLoad

EventHandler<LastBatchLoadedArgs> MyCaffe.layers.DataLayer< T >.OnBatchLoad

This event fires (only when set) each time a batch is loaded form this dataset.

Definition at line 66 of file DataLayer.cs.


The documentation for this class was generated from the following file: