This tutorial will guide you through the steps to create and train a Liquid Neural Net, first introduced by [1], to follow a Sine curve using the new Curve Gym.
Step 1 – Create the Project
In the first step we need to create the LiquidNN project that uses a Curve Gym based dataset. The Curve Gym iteratively generates the Sine curve, sending the current state to the model at each step along the way.
First select the Add Project (
) button at the bottom of the Solutions window, which will display the New Project dialog.
To select the Curve Gym dataset, click on the ‘…’ button to the right of the Dataset field and select the Gyms radio button at the bottom of the dialog and then select the Curve Gym listed.
Upon selecting OK on the New Project dialog, the new LiquidNN project will be displayed in the Solutions window.
Step 2 – Review the Model
Now that you have created the project, let’s open the model to see how it is organized. To review the model, double click on the LiquidNetCfc model within the new LiquidNN project, which will open the Model Editor window.
By clicking on the CFC Layer named cfc1 you can see the layer’s properties in the Properties window.
Step 3 – Training
During training, the MyCaffeTrainerDual loads and trains the LiquidNetCfc model and feeds inputs from the Curve Gym at each step along the Sine curve, thus allowing the model to learn to follow the curve. This simple Liquid Neural Net uses the CFC Layer to implement a closed-form continuous-time Liquid Neural Network.
The custom trainer settings of the ADAMW solver specify to use the RNN.SUPER_SIMPLE trainer which, as shown below, directs the MyCaffeTrainerDual to use the TrainerRNNSimple trainer during training.
During training, the following steps take place.
- The MyCaffeTrainerDual‘s Train method is called to train the model.
- Internally, the MyCaffeTrainerDual calls the TrainerRNNSimple trainer as specified in the custom trainer settings of the Solver.
- The TrainerRNNSimple calls a MyCaffeTrainerDual callback to get data from the configured gym, which…
- … in this case is the Curve Gym.
- At each step in the creation of the Sine curve, the Sine curve data and its history are sent to the TrainerRNNSimple via the callback called. The curve data is used to fill the x, tt, mask, and target blobs. The x blob is filled with the curve history data, the tt is filled with the timing data, the mask is filled with the masking data and the target is filled with the target (future curve) data to learn.
- Next, the Net is run through a forward and backward pass. The output of the Net forward pass is placed in the prediction xhat blob.
- And the Solver is directed to update the weights with one step.
The training step uses an AdamWSolver for optimization where weight decays are applied via the adamw_decay rate of 0.0 and the solver is run with no regularization. The base learning rate is set to 0.01, and the optimization uses a fixed learning rate policy. The Adam momentums are set to B1 = 0.9 and B2 = 0.95 with a delta (eps) = 1e-8.
Initially we trained a model on an NVIDIA RTX A6000 and the model used around 0.3 GB of GPU memory.
During training, double click on the Curve gym within the LiquidNN project to view the learning in real-time. Note, the accuracy is calculated as a rolling average of predictions that fall within a small tolerance of the target Sine curve.
Congratulations! You have now created your first Liquid Neural Net and trained it to follow a Sine curve in the SignalPop AI Designer!
[3] Closed-form Continuous-time Neural Models, by Ramin Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Aaron Ray, Max Tschaikowski, Gerald Teschl, Daniela Rus, 2021, arXiv:2106.13898