Posted on by ZoroDareVohnCoder

MyCaffe now supports Liquid Neural Networks!

In our latest release of the MyCaffe AI Platform, version 1.12.2.41, we now support Liquid Neural Networks as described in [1], [2], [3] and [4].

Liquid neural networks, first introduced by [1], are dynamic networks constructed “of linear first-order dynamical systems modulated via nonlinear interlinked gates,” resulting in models that “represent dynamical systems with varying (i.e., liquid) time-constants coupled to their hidden state, with outputs being computed by numerical differential equation solvers.”

The Liquid Neural Network is implemented in the new CFC Layer which supports both the CFC Cell and LTC Cell internal layers.

LiquidNetCfc Model

Using the new Curve Gym, data from a dynamically generated Sine curve is sent to the MyCaffe Liquid Neural Network which learns the curve over time.

Training Process

This training process is run iteratively as the curve progresses thus allowing the model to learn the Sine curve over time.

To try out a Liquid Neural Network in MyCaffe for yourself, check out the new Liquid Neural Network tutorial.

Temporal Fusion Transformer Models

In addition, in this release we have added new improvements and bug fixes to the MyCaffe Temporal Fusion Transformer models based on [5] and [6].  MyCaffe now supports a new in-memory temporal database MyCaffeTemporalDatabase that stores all temporal data in memory for faster training times. The new Favorita sample uses the in-memory temporal database during training to learn retail demand flows across a number of retail stores.

The Favorita dataset is a complex dataset comprising nearly 20 data streams of data that are all ‘fused’ together by the Temporal Fusion Transformer model to predict the demand flows across a large number of retail products.  The following data streams are in the Favorita dataset.

Static Numeric – none

Static Categorical – 9, shape = { B, 1 }

ItemNum (STATIC, CATEGORICAL)
StoreNum (STATIC, CATEGORICAL)
Store City ID (STATIC, CATEGORICAL)
Store State ID (STATIC, CATEGORICAL)
Store Type ID (STATIC, CATEGORICAL)
Store Cluster (STATIC, CATEGORICAL)
Item Family ID (STATIC, CATEGORICAL)
Item Class ID (STATIC, CATEGORICAL)
Item Perishable (STATIC, CATEGORICAL)

Historical Numeric – 6, shape = { B, 90, 1 }

Log Unit Sales (OBSERVED, NUMERIC)
Oil Price (OBSERVED, NUMERIC)
Transactions (OBSERVED, NUMERIC)
Day of Week (KNOWN, NUMERIC)
Day of Month (KNOWN, NUMERIC)
Month (KNOWN, NUMERIC)

Historical Categorical – 4, shape = { B, 90, 1 }

Open (KNOWN, CATEGORICAL)
On Promotion (KNOWN, CATEGORICAL)
Holiday Type (KNOWN, CATEGORICAL)
Holiday Locale ID (KNOWN, CATEGORICAL)

Future Numeric – 3, shape = { B, 30, 1 }

Day of Week (KNOWN, NUMERIC)
Day of Month (KNOWN, NUMERIC)
Month (KNOWN, NUMERIC)

Future Categorical – 4, shape = { B, 30, 1 }

Open (KNOWN, CATEGORICAL)
On Promotion (KNOWN, CATEGORICAL)
Holiday Type (KNOWN, CATEGORICAL)
Holiday Locale ID (KNOWN, CATEGORICAL)

This dataset is large and requires around 70 GB of RAM when fully loaded.  However, once loaded into memory, training commences quickly for there is no lag caused by loading data from disk. To help alleviate data loading times, the dataset may be loaded into the MyCaffeTemporalDatabase which holds the data in memory until unloaded.

After training, the model predicts the unit sales for each product sold at various locations.

TFT Favorita product demand flow Predictions

Further analysis shows how static variables impact the predictions.

TFT Favorita Static Variables

The impact of historical variables on the predictions is shown in the analytics.

TFT Favorita Historical Variables

And the impact of the future variables on the predictions is included in the analytics provided by the model.

TFT Favorita Future Variables

As shown above, the Store number, previous unit sales and day of the week are important data items that have a high impact on the predictions.

TFT Favorita Attention Scores

Visualizing the attention analytics confirms a strong 7-day periodic behavior which further confirms the importance of the day of week, likely due to stronger sales on weekends.

TFT Favorita Historical Variables Temporal View

Analyzing each data stream over time shows how these variables impact the predictions over time.

If you would like to try out these powerful models in MyCaffe, check out the Temporal Fusion Transformer tutorial.

New Features

The following new features have been added to this release.

  • CUDA 11.8.0.522/cuDNN 8.8.0.121/nvapi 515/driver 531.14
  • Windows 11 22H2
  • Windows 10 22H2, OS Build 19045.3448, SDK 10.0.19041.0
  • Added SQL support to TFT.electricity Dataset Creator
  • Added SQL support to TFT.traffic Dataset Creator
  • Added SQL support to TFT.volatility Dataset Creator
  • Added Liquid Neural Net support with new CfcLayer, CfcUnitLayer and LtcUnitLayer.
  • Added Temporal Database support with the new MyCaffeTemporalDatabase.
  • Added weight visualization for RNN type networks.
  • Added background loading to MyCaffeTemporalDatabase.
  • Added Temporal Database Windows Service support.
Bug Fixes

The following bug fixes have been made in this release.

For other great examples, including using Single Shot Multi-Box to detect gas leaks, or using Neural Style Transfer to create innovative and unique art, or creating Shakespeare sonnets with a CharNet, or beating PONG with Reinforcement Learning, check out the Examples page.

Happy Deep Learning with MyCaffe!

[1] Liquid Time-constant Networks, by Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, Radu Grosu, 2020, arXiv:2006.04439

[2] GitHub: Closed-form Continuous-time Models, by Ramin Hasani, 2021, GitHub

[3] Closed-form Continuous-time Neural Models, by Ramin Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Aaron Ray, Max Tschaikowski, Gerald Teschl, Daniela Rus, 2021, arXiv:2106.13898

[4] Closed-form continuous-time neural networks, by Ramin Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Aaron Ray, Max Tschaikowski, Gerald Teschl, Daniela Rus, 2022, nature machine intelligence

[5] Bryan Lim, Sercan O. Arik, Nicolas Loeff, and Tomas Pfister, Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Prediction, 2019, arXiv:1912.09360.

[6] GitHub: PlaytikaOSS/tft-torch, by Playtika Research, 2021, GitHub