Specifies the parameters for the TokenizedDataLayer. More...

Inheritance diagram for MyCaffe.param.gpt.TokenizedDataParameter:

Public Types
enum	VOCABULARY_TYPE { CHARACTER , WORD , SENTENCEPIECE , CUSTOM }
	Defines the vocabulary type to use. More...

enum	SAMPLE_METHOD { ARGMAX , PROBABILITY }
	Defines the sampling method used. More...

enum	INPUT_TYPE { TEXT_FILE , CUSTOM }
	Defines the input type used. More...

Public Types inherited from MyCaffe.param.LayerParameterBase
enum	LABEL_TYPE { NONE , SINGLE , MULTIPLE , ONLY_ONE }
	Defines the label type. More...

Public Member Functions
	TokenizedDataParameter ()
	Constructor for the parameter. More...

override object	Load (System.IO.BinaryReader br, bool bNewInstance=true)
	Load the parameter from a binary reader. More...

override void	Copy (LayerParameterBase src)
	Copy on parameter to another. More...

override LayerParameterBase	Clone ()
	Creates a new copy of this instance of the parameter. More...

override RawProto	ToProto (string strName)
	Convert the parameter into a RawProto. More...

Public Member Functions inherited from MyCaffe.param.LayerParameterBase
	LayerParameterBase ()
	Constructor for the parameter. More...

virtual string	PrepareRunModelInputs ()
	This method gives derivative classes a chance specify model inputs required by the run model. More...

virtual void	PrepareRunModel (LayerParameter p)
	This method gives derivative classes a chance to prepare the layer for a run-model. More...

void	Save (BinaryWriter bw)
	Save this parameter to a binary writer. More...

abstract object	Load (BinaryReader br, bool bNewInstance=true)
	Load the parameter from a binary reader. More...

Public Member Functions inherited from MyCaffe.basecode.BaseParameter
	BaseParameter ()
	Constructor for the parameter. More...

virtual bool	Compare (BaseParameter p)
	Compare this parameter to another parameter. More...

Static Public Member Functions
static TokenizedDataParameter	FromProto (RawProto rp)
	Parses the parameter from a RawProto. More...

Static Public Member Functions inherited from MyCaffe.basecode.BaseParameter
static double	ParseDouble (string strVal)
	Parse double values using the US culture if the decimal separator = '.', then using the native culture, and if then lastly trying the US culture to handle prototypes containing '.' as the separator, yet parsed in a culture that does not use '.' as a decimal. More...

static bool	TryParse (string strVal, out double df)
	Parse double values using the US culture if the decimal separator = '.', then using the native culture, and if then lastly trying the US culture to handle prototypes containing '.' as the separator, yet parsed in a culture that does not use '.' as a decimal. More...

static float	ParseFloat (string strVal)
	Parse float values using the US culture if the decimal separator = '.', then using the native culture, and if then lastly trying the US culture to handle prototypes containing '.' as the separator, yet parsed in a culture that does not use '.' as a decimal. More...

static bool	TryParse (string strVal, out float f)
	Parse doufloatble values using the US culture if the decimal separator = '.', then using the native culture, and if then lastly trying the US culture to handle prototypes containing '.' as the separator, yet parsed in a culture that does not use '.' as a decimal. More...

Protected Attributes
PythonParameter	m_pythonParam = new PythonParameter()
	Python layer implementations use this parameter for Python specific settings such as the location of the runtime. More...

Properties
PythonParameter	python_param `[getset]`
	Specifies the PythonParameter used by the python implementation of the TokenizedDataPairsLayer, otherwise this is null. More...

int?	seed `[getset]`
	Specifies the seed used to initialize the random number generator (normally only for testing). More...

INPUT_TYPE	input_type `[getset]`
	Specifies data source input type. More...

VOCABULARY_TYPE	vocabulary_type `[getset]`
	Specifies the vocabulary type to use. More...

SAMPLE_METHOD	sample_method `[getset]`
	Specifies the sampling method used when post processing logits (default = ARGMAX). More...

string	source `[getset]`
	Specifies the data source based on the INPUT_TYPE used. Each dataset has both a training and testing data source. More...

string	debug_index_file `[getset]`
	Specifies an optional data index file used for debugging only. More...

uint	batch_size `[getset]`
	The number of heads used. More...

uint	block_size `[getset]`
	Specifies size of the block. More...

Detailed Description

Specifies the parameters for the TokenizedDataLayer.

Definition at line 16 of file TokenizedDataParameter.cs.

Member Enumeration Documentation

◆ INPUT_TYPE

enum MyCaffe.param.gpt.TokenizedDataParameter.INPUT_TYPE

Defines the input type used.

Enumerator
TEXT_FILE	Specifies that the source is a text filename.
CUSTOM	Specifies that the source is the filename of an assembly DLL supporting the IXCustomTokenInput interface used for custom token input.

Definition at line 74 of file TokenizedDataParameter.cs.

◆ SAMPLE_METHOD

enum MyCaffe.param.gpt.TokenizedDataParameter.SAMPLE_METHOD

Defines the sampling method used.

Enumerator
ARGMAX	Specifies to use the argmax method.
PROBABILITY	Specifies to use the probability sampling method where the probabilities are totaled until the sum >= a random number.

Definition at line 58 of file TokenizedDataParameter.cs.

◆ VOCABULARY_TYPE

enum MyCaffe.param.gpt.TokenizedDataParameter.VOCABULARY_TYPE

Defines the vocabulary type to use.

Enumerator
CHARACTER	Specifies to use a character based vocabulary.
WORD	Specifies to use a word based vocabulary.
SENTENCEPIECE	Specifies to use pre-generated SentencePiece vocabulary.
CUSTOM	Specifies to use a custom vocabulary.

Definition at line 35 of file TokenizedDataParameter.cs.

Constructor & Destructor Documentation

◆ TokenizedDataParameter()

MyCaffe.param.gpt.TokenizedDataParameter.TokenizedDataParameter ( )

Constructor for the parameter.

Definition at line 87 of file TokenizedDataParameter.cs.

Member Function Documentation

◆ Clone()

override LayerParameterBase MyCaffe.param.gpt.TokenizedDataParameter.Clone ( )

virtual

Creates a new copy of this instance of the parameter.

Returns: A new instance of this parameter is returned.

Implements MyCaffe.param.LayerParameterBase.

Definition at line 207 of file TokenizedDataParameter.cs.

◆ Copy()

override void MyCaffe.param.gpt.TokenizedDataParameter.Copy ( LayerParameterBase src )

virtual

Copy on parameter to another.

Parameters

src	Specifies the parameter to copy.

Implements MyCaffe.param.LayerParameterBase.

Definition at line 191 of file TokenizedDataParameter.cs.

◆ FromProto()

static TokenizedDataParameter MyCaffe.param.gpt.TokenizedDataParameter.FromProto ( RawProto rp )

static

Parses the parameter from a RawProto.

Parameters

rp	Specifies the RawProto to parse.

Returns: A new instance of the parameter is returned.

Definition at line 247 of file TokenizedDataParameter.cs.

◆ Load()

override object MyCaffe.param.gpt.TokenizedDataParameter.Load	(	System.IO.BinaryReader	br,
		bool	bNewInstance = `true`
	)

Load the parameter from a binary reader.

Parameters

br	Specifies the binary reader.
bNewInstance	When true a new instance is created (the default), otherwise the existing instance is loaded from the binary reader.

Returns: Returns an instance of the parameter.

Definition at line 179 of file TokenizedDataParameter.cs.

◆ ToProto()

override RawProto MyCaffe.param.gpt.TokenizedDataParameter.ToProto ( string strName )

virtual

Convert the parameter into a RawProto.

Parameters

strName Specifies the name to associate with the RawProto.

Returns: The new RawProto is returned.

Implements MyCaffe.basecode.BaseParameter.

Definition at line 219 of file TokenizedDataParameter.cs.

Member Data Documentation

◆ m_pythonParam

PythonParameter MyCaffe.param.gpt.TokenizedDataParameter.m_pythonParam = new PythonParameter()

protected

Python layer implementations use this parameter for Python specific settings such as the location of the runtime.

Definition at line 21 of file TokenizedDataParameter.cs.

Property Documentation

◆ batch_size

uint MyCaffe.param.gpt.TokenizedDataParameter.batch_size

getset

The number of heads used.

Definition at line 163 of file TokenizedDataParameter.cs.

◆ block_size

uint MyCaffe.param.gpt.TokenizedDataParameter.block_size

getset

Specifies size of the block.

Definition at line 172 of file TokenizedDataParameter.cs.

◆ debug_index_file

string MyCaffe.param.gpt.TokenizedDataParameter.debug_index_file

getset

Specifies an optional data index file used for debugging only.

Definition at line 153 of file TokenizedDataParameter.cs.

◆ input_type

INPUT_TYPE MyCaffe.param.gpt.TokenizedDataParameter.input_type

getset

Specifies data source input type.

Definition at line 113 of file TokenizedDataParameter.cs.

◆ python_param

PythonParameter MyCaffe.param.gpt.TokenizedDataParameter.python_param

getset

Specifies the PythonParameter used by the python implementation of the TokenizedDataPairsLayer, otherwise this is null.

Definition at line 94 of file TokenizedDataParameter.cs.

◆ sample_method

SAMPLE_METHOD MyCaffe.param.gpt.TokenizedDataParameter.sample_method

getset

Specifies the sampling method used when post processing logits (default = ARGMAX).

Definition at line 133 of file TokenizedDataParameter.cs.

◆ seed

int? MyCaffe.param.gpt.TokenizedDataParameter.seed

getset

Specifies the seed used to initialize the random number generator (normally only for testing).

Definition at line 103 of file TokenizedDataParameter.cs.

◆ source

string MyCaffe.param.gpt.TokenizedDataParameter.source

getset

Specifies the data source based on the INPUT_TYPE used. Each dataset has both a training and testing data source.

Definition at line 143 of file TokenizedDataParameter.cs.

◆ vocabulary_type

VOCABULARY_TYPE MyCaffe.param.gpt.TokenizedDataParameter.vocabulary_type

getset

Specifies the vocabulary type to use.

Definition at line 123 of file TokenizedDataParameter.cs.

The documentation for this class was generated from the following file:

C:/Data/Data/SS_Projects/Intelligence/GitHub/MyCaffe/MyCaffe/param.gpt/TokenizedDataParameter.cs

Public Types

Public Member Functions

Static Public Member Functions

Protected Attributes

Properties

Detailed Description

Member Enumeration Documentation

◆ INPUT_TYPE

◆ SAMPLE_METHOD

◆ VOCABULARY_TYPE

Constructor & Destructor Documentation

◆ TokenizedDataParameter()

Member Function Documentation

◆ Clone()

◆ Copy()

◆ FromProto()

◆ Load()

◆ ToProto()

Member Data Documentation

◆ m_pythonParam

Property Documentation

◆ batch_size

◆ block_size

◆ debug_index_file

◆ input_type

◆ python_param

◆ sample_method

◆ seed

◆ source

◆ vocabulary_type