In our latest release of the MyCaffe AI Platform, version 1.12.2.41, we now support Liquid Neural Networks as described in [1], [2], [3] and [4]. Liquid neural networks, first introduced by [1], are dynamic networks constructed “of linear first-order dynamical systems modulated via nonlinear interlinked gates,” resulting in models that “represent dynamical systems with varying …
Continue reading “MyCaffe now supports Liquid Neural Networks!”
Liquid neural networks, first introduced by [1], are networks constructed “of linear first-order dynamical systems modulated via nonlinear interlinked gates,” resulting in models that “represent dynamical systems with varying (i.e., liquid) time-constants coupled to their hidden state, with outputs being computed by numerical differential equation solvers.” The Closed-form Continuous-time Models (CfC) ‘are powerful sequential liquid …
Continue reading “Closed-form Continuous-time Liquid Neural Net Models – A Programmer’s Perspective”
In our latest release of the MyCaffe AI Platform, version 1.12.1.82, we now support Temporal Fusion Transformer (TFT) Models as described in [1] and [2]. These powerful models provide multi-horizon time-series predictions while outperforming DeepAR from Amazon, Deep State Space Models, MQRNN, TRMF, and traditional models such as ARIMA and ETS according to [1]. The …
Continue reading “MyCaffe now supports Temporal Fusion Transformer Models!”
In our last post, we looked at the organization of the data used by Temporal Fusion Transformer models as described in the Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting article by Lim et al. [1]. In this post we take a deeper dive into the architecture of the Temporal Fusion Transformer model and how …
Continue reading “Temporal Fusion Transformers – Model Data Flow”
Temporal Fusion Transformers, as described in the Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting article by Lim et al. [1], use a complex mix of inputs to provide multi-horizon forecasting for timeseries data. The first step in understanding these models is to understand the data inputs fed into the model and the predicted outputs …
Continue reading “Temporal Fusion Transformers – Data Organization”
In our latest release, version 1.12.0.60, we now support ChatGPT type architectures with Encoder/Decoder Transformer Models based on the open-source transformer-translator-pytorch GitHub project by Song [1]. ChatGPT uses encoder/decoder transformer models to learn the context of the input query, the context of the likely responses and a mapping between the two via attention layers. The …
With GPT and ChatGPT, transformer models have been proven to be very powerful AI models. However, how do they work on the inside? With this post, we use the SignalPop AI Designer to visually walk through the forward pass of a transformer model used for language translation. Before showing a visual walk-through we wanted to …
Continue reading “Visually Walking Through a Transformer Model”
While completing the MyCaffe implementation of the transformer encoder/decoder model for language translation, we ran into a very difficult bug to fix – in fact it was the kind of bug feared most when developing a model for with this bug everything appeared to work as expected when training. Yet, the model would train up …
When training any deep learning AI model, knowing your data is critical. This is especially important when training a transformer-based encoder/decoder model for data ordering is important. In this post, we analyze the Python open-source language translation encoder/decoder transformer model by Jaewoo (Kyle) Song [1] which is based on the ‘Attention Is All You Need‘ …
Continue reading “Data Flow when Training an Encoder/Decoder Model for Language Translation”
GPT is a great transformer model used to solve many natural language problems, however GPT only implements the encoder side of a full encoder/decoder transformer model as described by Vaswani et al. [1]. Only a few changes are needed to implement a full encoder/decoder transformer model as shown below (GPT portion inspired by the minGPT …
Continue reading “Converting a GPT Model into a full Encoder/Decoder Transformer Model”
