Large Art Prints Created with The SignalPop AI Designer

The SignalPop AI Designer and the MyCaffe AI Platform‘s implementation of Neural Style Transfer[1] and the VGG model[2] are now used to create large sized artistic prints – some as large as a ping-pong table! Creating these prints can run up to 2 trillion calculations and requires nearly all of the 50GB of video memory offered by the NVIDIA Quadro RTX 8000 GPU.

Trevor Kennison’s epic launch off Corbet’s Couloir, Jackson Hole, WY.

The print of Trevor Kennison was a collaborative effort between former US Ski Team photographer Jonathan Selkowitz, artist Noemí Ibarz of Barcelona, Spain and Signalpop who provided the artificial intelligence software, including both the SignalPop AI Designer and MyCaffe AI Platform.

Photo to Art Collaboration

Artificial intelligence helps bring together the creative talents of both the photographer and artist to create a new, collaborative piece of work!

A closer view of the print shows how the AI actually learns Noemí’s artistic style and paints Jonathan’s picture of Trevor with it to create a fantastic, new piece of art!

Trevor Print Detail #1
Trevor Print Detail #2
Trevor Print Detail #3

The neural style transfer, learns not only the colors of Noemí’s art, but also the brush strokes and even the texture of the medium on which her art was painted.

Art by Noemí Ibarz

Visit Instagram@noemi_ibarz to see more of Noemí’s fantastic colorful art!

Visit Selko Photo to see more of Jonathan’s beautiful photography that does an amazing job of capturing motion in a still image.

In an effort to help the High Fives Foundation (who helped Trevor get his jump back) we are auctioning off this print – to learn more about the auction and learn more about Trevor’s story, please see the auction site on RallyUp.


[1] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge, A Neural Algorithm of Artistic Style, 2015, arXiv:1508:06576.

[2] Karen Simonyan and Andrew Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, 2014, arXiv:1409.1556.

Siamese Net with KNN, Contrastive Loss and easy Python Programming

In our latest release, version 0.10.2.124, we have added support for K-Nearest Neighbor (KNN) to the Siamese Net as described by [1] and [2].

KNN provides the Siamese Net with another way to perform ‘One-Shot’ learning on difficult datasets that include images the network has never seen before.

How Does This Work?

Most of the steps are very similar to what was discussed in the last blog entry on the Siamese Net.  However, when using KNN, the Decode Layer builds a cache in GPU memory of encodings received for each label.

Siamese Net with KNN

During training, the encoding cache is filled, and then during testing and running, the cache is used to find the encodings with the largest number of ‘nearest neighbors’ (e.g. shortest distances) to the current encoding from the data we seek to find a label for.

The following steps take place within the KNN based Siamese Net:

1.) During training, the Data Layer is fed a pair of images where each image is stacked one after the other in the channel.

2.) Internally, the model splits the data into two separate data streams using the Slice Layer.  Each data stream is then fed into each of the two parallel nets making up the Siamese Net.

3.) Within the first network, the Decode Layer caches the encodings that it receives and stores the encodings by label.  The encoding cache is stored within GPU memory and persisted along with the model weights so that the cache can be used later when the net is run.

4.) As with the CENTROID based Siamese Net, the encodings are sent to the Contrastive Loss layer which calculates the loss from the distance between the two encodings received from each of the two parallel nets.

5.) When running the net, the Decode layer compares its input encodings(that we seek to find a label for) with each of the encodings within the label cache.  The distance between each is determined and the distances are sorted for each label and ordered by smallest distance first.  A set of the  k smallest distances for each label are averaged and the label with the smallest average distance is determined to be the detected label for the data.

6.) When running, the label distance averages are returned and the label with the smallest distance is determined to be the label detected.

Programming MyCaffe in Python

Using pythonnet makes programming MyCaffe in Python extremely easy and gives you access to virtually all aspects of MyCaffe.  To get started, just install pythonnet by running the following command from with in your 64-bit Python environment.

pip install pythonnet

Once installed, each MyCaffe namespace is accessed by adding a reference to each namespace via the clr.  For example the following references are used by the One-Shot Python Sample on GitHub.

Python References for One-Shot Learning Sample

These Python references are equivalent to the following references used in the C# sample.

C# References for One-Shot Learning Sample

After referencing the MyCaffe DLLs, each object within the referenced DLLs are just about as easy to use in Python as they are in C#!  For more discussion on using Python to program MyCaffe, see the section on Training and Testing with Python in the MyCaffe Programming Guide.

For an example on programming the MyCaffeControl with C# to learn the MNIST dataset using a Siamese Net with KNN, see the C# Siamese Net Sample on GitHub.

For an example on programming the MyCaffeControl with Python to learn the MNIST dataset using a Siamese Net with KNN, see the Python Siamese Net Sample on GitHub.

New Features

The following new features have been added to this release.

  • Added MyCaffe Image Database version 2 support with faster background loading and new query state support.
  • Added VerifyCompute to verify that the current device matches the compute requirement of the currently used CudaDnnDll.
  • Added multi-label support to Debug Layer inspection.
  • Added boost query hit percentage support to project properties.
  • Added new KNN support the Decode layer for use with Siamese Nets.
  • Added new AccuracyDecode layer for use with Siamese Nets.
  • Added boost persistence support for saving and loading image boost settings.
Bug Fixes

The following bugs have been fixed in this release.

  • Project results are now properly displayed after importing into a new project.
  • Project results are now deletable after closing a project.
  • Data loading optimized for faster image loading.
  • Fixed bugs in MyCaffe Image Database related to superboost probability and label balancing.

To try out this model and train it yourself, just check out our Tutorials for easy step-by-step instructions that will get you started quickly! For cool example videos, including an ATARI Pong video and Cart-Pole balancing video, check out our Examples page.


[1] Berkeley Artificial Intelligence (BAIR), Siamese Network Training with Caffe.

[2] G. Koch, R. Zemel and R. Salakhutdinov, Siamese Neural Networks for One-shot Image Recognition, ICML 2015 Deep Learning Workshop, 2015.

Siamese Net with Contrastive Loss now supported with CUDA 10.2 and cuDNN 7.6.5!

In our latest release, version 0.10.2.38, we have added support for Siamese Nets as described by [1] and [2], and do so with the newly released CUDA 10.2 and cuDNN 7.6.5!

The Siamese Net provides the ability to perform ‘One-Shot’ learning where an image that the network has never seen before, is quickly matched with already learned classifications – if such a match exists. Several examples using Siamese Nets for one-shot learning include: Image retrieval described by [3]; Content based retrieval described by [4]; and Railway asset detection described by [5].

How Does This Work?

Siamese Nets use two parallel networks that both share the same weights which are learned while digesting pairs of images.  The following sequence of steps occur while training a Siamese Net.

How the Siamese Net works

1.) Using the Data Layer, data is fed into the net as image pairs where each image is stacked one after the other along the data input channels. In order to provide a balanced training, pairs of images alternate between two images of the same class followed by two images of different classes.

2.) During training, the pairs of images loaded by the Data Layer are split by the Slice Layer which then feeds each image into one of two parallel networks that both share a set of learnable weights.

3.) Each network produces an encoding for each image fed to it.

4.) These encodings are then sent to the Contrastive Loss layer which calculates the loss from the distance between the two encodings where the loss is set to the squared distance between the two images when they are from the same class, and the squared difference between the margin and the distance when they are from different classes; which moves the image encodings toward one another when they are from the same class and further apart when they are not.

5.) During training, a Decode Layer calculates and stores the centroid encoding for each class.

6.) When running the network, the Decode Layer’s stored centroids are used to determine the shortest distance between the input image’s encoding and each classes encoding centroid. A minimum distance between the input image and a given class indicates that the input image matches the class.

In summary, training a Siamese Net directs encodings of similar classes to group together and encodings of different classes to move away from one another.  When running the Siamese Net, the input image is converted into its encoding which is then matched with the class for which it is closest.  In the event the encoding is not ‘close’ to any other class encoding centroids, the image is determined to be a part of an unknown, or new class.

Siamese Net to learn MNIST

The following Siamese Net is used to learn the MNIST dataset.

Siamese Net Model

As shown above, the model uses two parallel networks joined at the bottom by a Contrastive Loss layer.  Pairs of images are fed into the top of each network which then each produce the ‘feat‘ and ‘feat_p‘ encodings for the images.  These encoding pairs are then sent to the Contrastive Loss layer where the learning begins.

New Features

The following new features have been added to this release.

  • Added CUDA 10.2.89/cuDNN7.6.5.32 support.
  • Added label query hit percent and label query epoch information to projects.
  • Added new AccuracyEncoding layer for Siamese Nets.
  • Added new Decode layer for Siamese Nets.
  • Added new multiple image support to Data Layer for Siamese Nets.
  • Added new layer weight map visualization.
Bug Fixes

The following bugs have been fixed in this release.

  • Fixed bug related to VOC0712 Dataset Creator.
  • Fixed bug related to data overlaps occurring within Data layer.
  • Optimized cuDNN handle usage.
  • Optimized pinned memory usage.

To try out this model and train it yourself, just check out our Tutorials for easy step-by-step instructions that will get you started quickly! For cool example videos, including an ATARI Pong video and Cart-Pole balancing video, check out our Examples page.


[1] Berkeley Artificial Intelligence (BAIR), Siamese Network Training with Caffe.

[2] G. Koch, R. Zemel and R. Salakhutdinov, Siamese Neural Networks for One-shot Image Recognition, ICML 2015 Deep Learning Workshop, 2015.

[3] K. L. Wiggers, A. S. Britto, L. Heutte, A. L. Koerich and L. S. Oliveira, Image Retrieval and Pattern Spotting using Siamese Neural Network, arXiv, vol. 1906.09513, 2019.

[4] Y.-A. Chung and W.-H. Weng, Learning Deep Representations of Medical Images using Siamese CNNs with Application to Content-Based Image Retrieval, arXiv, vol. 1711.08490, 2017.

[5] D. J. Rao, S. Mittal and S. Ritika, Siamese Neural Networks for One-shot detection of Railway Track Switches, arXiv, vol. 1712.08036, 2017.

Maintenance Release #2 with CUDA 10.1.243/cuDNN 7.6.4 Support.

In our latest maintenance release, version 0.10.1.283, we have fixed numerous bugs and support the newly released NVIDIA CUDA 10.1.243 with cuDNN 7.6.4. You can now use Neural Style Transfer, Policy Gradient based reinforcement learning, Char-RNN LSTM based learning and much more with CUDA 10.1.243.

New Features

The following new features have been added to this release.

  • Added new Permute Layer for SSD.
  • Added new PriorBox Layer for SSD.
  • Added new AnnotatedData Layer for SSD.
  • Added new Normalization2 Layer for SSD.
  • Added new MultiBoxLoss Layer for SSD.
  • Added new DetectionEvaluate Layer for SSD.
  • Added new DetectionOutput Layer for SSD.
  • Added new VOC0712 Dataset Creator.
  • Added label query hit percent and label query epoch information to projects.
Bug Fixes

The following bugs have been fixed in this release.

  • Fixed bug related to flatten layer reporting incorrect sizing in model editor.
  • Fixed bug related to Split layer causing crash in model editor (e.g. DANN model).
  • Fixed bug in dataset export.
  • Fixed bug in image viewer – the viewer now resizes images to the window.

Check out our Tutorials for easy step-by-step instructions that will get you started quickly with several complex model types! For cool example videos, including an ATARI Pong video and Cart-Pole balancing video, check out our Examples page.

Maintenance Release with CUDA 10.1.243/cuDNN 7.6.3 Support.

In our latest maintenance release, version 0.10.1.221, we have fixed numerous bugs and support the newly released NVIDIA CUDA 10.1.243 with cuDNN 7.6.3. You can now use Neural Style Transfer, Policy Gradient based reinforcement learning, Char-RNN LSTM based learning and much more with CUDA 10.1.243.

New Features

The following new features have been added to this release.

  • Added legacy compute 3.5 support to CUDA 10.1 version of CudaDnn.DLL.
  • Added optional use of Image Database Service when performing TSNE analysis.
  • Added option to resize input images to TSNE analysis.
  • Added custom input image masking to TSNE analysis.
  • Added Find Layer to Model Editor.
Bug Fixes

The following bugs have been fixed in this release.

  • Fixed bug related to process warning, now properly handled when closing application.
  • Fixed bug related to project results not showing, now results are shown based on the snapshot load method.
  • Fixed bug related to TSNE and PCA analysis creating duplicate outputs.

Check out our Tutorials for easy step-by-step instructions that will get you started quickly with several complex model types! For cool example videos, including an ATARI Pong video and Cart-Pole balancing video, check out our Examples page.

Noisy-Net with Deep Q-Learning Now Supported with cuDNN 7.6.1!

In our latest release, version 0.10.1.169, we now support Noisy-Nets as described by Fortunato et al.[1], Prioritized Experience Replay as described by Schaul et al.[2] and Deep Q-Learning reinforcement learning as described by Castro et al.[3] and The Dopamine Team[4] and use these to train MyCaffe to beat the ATARI game ‘Breakout’!

The action values are displayed in an overlay at the bottom of the ATARI Gym and updated during each step of the game.

The Noisy-Net model used with reinforcement learning is fairly simple, comprising of three CONVOLUTION layers followed by two INNERPRODUCT layers each of which have noise turned ON.

Deep Q-Learning Noisy-Net model for ATARI breakout.

To try out this model and train it yourself, just check out our Tutorials for easy step-by-step instructions that will get you started quickly! For cool example videos, including a Cart-Pole balancing video, check out our Examples page.

New Features

The following new features have been added to this release

  •  CUDA 10.1.168/cuDNN 7.6.1/nvapi 410/driver 430.86 support added.
  • Windows 1903 support added.
  • Added Test support to RL trainers.
  • Added TestMany support to RL trainers.
  • Added DQN trainer support for Deep Q-Learning.
  • Added ATARI breakout ROM.
  • Added Noise support to InnerProduct layers for NoisyNets.
Bug Fixes

The following bug fixes have been made in this release.

  • Fixed bugs in MemoryLoss layer.
  • Fixed bugs in Convolution Editor, pad = 0 now accepted.

Happy Deep Learning!



[1] Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, Shane Legg, Noisy Networks for Exploration, arXiv:1706.10295, June 30, 2017.
[2] Tom Schaul, John Quan, Ioannis Antonoglou, David Silver, Prioritized Experience Replay, arXiv:1511.05952, November 18, 2015.
[3] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, Marc G. Bellemare, Dopamine: A Research Framework for Deep Reinforcement Learning, arXiv:1812.06110, December 14, 2018.
[4] The Dopamine Team (Google), GitHub:Google/Dopamine, GitHub, Licensed under the Apache 2.0 License. Source code available on GitHub at google/dopamine.
[5] The Arcade Learning Environment: An Evaluation Platform for General Agents, by Marc G. Bellemare, Yavar Naddaf, Joel Veness and Michael Bowling, 2012-2013. Source code available on GitHub at mgbellemare/Arcade-Learning-Environment.
[6] Stella – A multi-platform Atari 2600 VCS emulator by Bradford W. Mott, Stephen Anthony and The Stella Team, 1995-2018. Source code available on GitHub at stella-emu/stella

Dual RNN/RL Trainer with CUDA 10.1.168 / cuDNN 7.6 Now Supported!

In our latest release, version 0.10.1.145, we have added support for a new Dual Recurrent/Reinforcement Learning Trainer that also supports the newly released NVIDIA CUDA 10.1.168 (Update 1) with cuDNN 7.6. With the dual RNN/RL trainer you can train the same model first in an recurrent style learning and then train the same model with a second training pass that uses a reinforcement learning style of learning. In addition, this release includes dramatic optimizations that greatly improve the responsiveness of the application when opening and closing projects.

New Features

The following new features have been added to this release.

  • Dramatically improved application start-up time.
  • Dramatically improved application responsiveness to project actions (open, close, etc.)
  • Dramatically improved application responsiveness when changing project properties.
  • Added option to save image in all Image Viewer dialogs.
  • Added dual RNN/RL training model support with stages to distinguish between each during training.
  • New setting allows changing the maximum debug data/diff displayed.
  • Editor now automatically shows RNN or RL stage (if used).
  • Added new Finance Gym with simulated price stream.
  • Added ability to debug data of specific items on data or data link.
  • Added __half sized memory support to CONVOLUTION, POOLING, TANH, SIGMOID, RELU and INPUT layers.
  • Moved minimum compute from 3.5 to 5.3 (Maxwell) for half memory support.

The following bugs have been fixed in this release.

  • LSTM layer output now shows values in debug data window instead of all zeros.
  • Dummy Data Layer shape now supported in the editor.
  • Google Dream Evaluator now shows octaves properly.
  • Image Evaluator now shows debug layers properly.
  • Snapshot Update, Snapshot Load Method and Image Load Method now saved.
  • Custom trainers now properly take snapshots.
  • Large images in image viewer are no longer skewed and support scrolling.

Check out our Tutorials for easy step-by-step instructions that will get you started quickly with several complex model types! For cool example videos, including an ATARI Pong video and Cart-Pole balancing video, check out our Examples page.

Maintenance Release with CUDA 10.1/cuDNN 7.5 Support.

In our latest maintenance release, version 0.10.1.48, we have fixed numerous bugs and support the newly released NVIDIA CUDA 10.1 with cuDNN 7.5. You can now use Neural Style Transfer, Policy Gradient based reinforcement learning, Char-RNN LSTM based learning and much more with CUDA 10.1.

New Features

The following new features have been added to this release.

  • Added original image and final image sizing to Neural Style Evaluator.
  • Added update status when deleting datasets.
  • Updated SimpleGraphing with minor updates.
Bug Fixes

The following bugs have been fixed in this release.

  • Fixed bugs related to debug window updating very slowly.
  • Fixed bugs related to layer inspection which now shows the best results correctly.
  • Fixed bugs related to extremely slow training after 10k iterations.
  • Fixed bugs related to iterations not matching Project Graph x values.
  • Fixed bugs related to Project Control user interface during training.
  • Fixed bug related to Resource Window not updating after dataset changes.
  • Fixed bugs related to slow LSTM_SIMPLE training.

Check out our Tutorials for easy step-by-step instructions that will get you started quickly with several complex model types! For cool example videos, including an ATARI Pong video and Cart-Pole balancing video, check out our Examples page.

CUDA 10.1 / cuDNN 7.5 Now Supported!

In our latest release, version 0.10.1.21, we have added support for the newly released NVIDIA CUDA 10.1 with cuDNN 7.5. You can now use Neural Style Transfer, Policy Gradient based reinforcement learning, Char-RNN LSTM based learning and much more with CUDA 10.1.

New Features

The following new features have been added to this release.

  • Added image blending to Neural Style Transfer.
  • Added new CSV Dataset Creator.
  • Updated SimpleGraphing with minor updates.
Bug Fixes

The following bugs have been fixed in this release.

  • Fixed bugs related to deleted dataset causing project load crash.
  • Fixed bugs related to GPF caused when debugging data or diff link.

Check out our Tutorials for easy step-by-step instructions that will get you started quickly with several complex model types! For cool example videos, including an ATARI Pong video and Cart-Pole balancing video, check out our Examples page.

Neural Style Transfer now supported with cuDNN 7.4.2!

In our latest release, version 0.10.0.190, we have added Neural Style Transfer as described by [1] using the VGG model [2].

Neural Style Transfer

With Neural Style Transfer, the style of one photo (such as Vincent Van Gogh’s Starry Night) is learned by the AI model and applied to a content photo (such as the photo of train tracks) to produce a new art piece!

For example, selecting Edvard Munch’s The Scream as a style tells the application to learn the style of his painting and apply it to the content photo to produce a new and different art piece from the one created with Van Gogh’s style.

Edvard Munch’s The Scream

In another example, using Claude Monet’s Water Lilies 1916 as the style creates another new art piece.

Claude Monet’s Water Lilies 1916

How Does This Work?

As shown above, the neural style algorithm learns the style and applies it to the content, but how does this work?  When running a neural style transfer, the model used (e.g. VGG19) is augmented by adding a few new layers that allow for learning the style and then back-propagating both the style and content features back to create the final image.  As described by Li [3], the Gram matrix is the main work-horse used to learn the style in that it, in theory, “is equivalent to minimizing the Maximum Mean Discrepancy” and shows that that neural style transfer is “a process of distribution alignment of the neural activations between images.”  The Gram matrix is implemented by the GRAM layer.  Two additional layers play important roles in the process.  The EVENT layer is used to only apply the gradients that matter (e.g. zeroing out the others) and the PARAMETER layer holds the final learned results that produce the end result – a neural ‘stylized’ image.

To show how this works a little better, the model below displays a visualization of the actual augmented model (augmented from VGG19 and originally inspired by [4]) used for the neural style transfer.  With this example, the layers used to learn the style are: conv1_1, conv2_1, conv3_1, conv4_1 and conv5_1.  And, the layer used to learn the content is: conv4_2.  You can use whatever layers you want to create different output effects, but these are the ones used in this discussion.

Neural Style Transfer Model

During the neural style transfer process, the following steps take place.

  1. First, the style image is loaded into the data blob and a forward pass is performed on the augmented VGG19 model with GRAM layers already added.
  2. Next, the GRAM layer output blobs gram1_1, gram2_1, gram3_1, gram4_1 and gram5_1 are saved.
  3. The content image is then loaded into the data blob and another forward pass is performed on the augmented model.
  4. This time, the output blob of the conv4_2 CONVOLUTION output layer is saved.
  5. The model is further augmented by adding INPUT, EVENT and EUCLIDEAN_LOSS layers to match each GRAM layer.  A SCALAR layer is added to the content layer to help data scaling, and weight learning is disabled on the CONVOLUTION layers.
  6. Next, the style output blobs saved in steps #2 above are restored back into the solver model, augmented in step #5 above.
  7. And the content output blob saved in step #4 above is restored back into the augmented solver model as well.
  8. The solver is then directed to solve the model which begins the forward/backward set of iterations to solve the neural style transfer.
  9. On each forward pass, the Euclidean loss is calculated between the style INPUT and GRAM layers (EVENT layer acts as a passthrough on the forward pass).
  10. During the backward pass, the gradient calculated in step #9 is passed back through the EVENT layer which then applies only the gradients that have actually changed and zeros out the rest.
  11. The same process occurs with the content layer, but in this case the Euclidean loss is calculated between the content INPUT and the CONVOLUTION layer conv4_2 output.
  12. Again the gradient calculated in step #11 is passed back through an EVENT layer which then applies only the gradients that have actually changed and zeros out the rest.

The back propagation continues on up the network and deposits the final changes into the PARAMETER layer input1.  After running numerous iterations (sometimes just around 200), the style of the style image paints the content provided by the content image to produce a new piece of art!

New Features

The following new features have been added to this release.

  • Added new Neural Style Transfer Evaluator.
  • Added Database Optimizer.
Bug Fixes

The following bugs have been fixed in this release.

  • Fixed bugs in Import Project dialog including slow processing.

To try out this model and train it yourself, just check out our Tutorials for easy step-by-step instructions that will get you started quickly! For cool example videos, including an ATARI Pong video and Cart-Pole balancing video, check out our Examples page.


[1] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge, A Neural Algorithm of Artistic Style, 2015, arXiv:1508:06576.

[2] Karen Simonyan and Andrew Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, 2014, arXiv:1409.1556.

[3] Yanghao Li, Naiyan Wang, Jiaying Liu and Xiaodi Hou, Demystifying Neural Style Transfer, 2017, arXiv:1701.01036.

[4] ftokarev, GitHub:ftokarev/caffe-neural-style, 2017, GitHub.

The ‘Train Bridge‘ photo is provided by www.pexels.com under the CC0 License.

Each of the paintings, Starry Night by Vincent Van Gogh, The Scream by Edvund Munch and Water Lilies 1916 by Claude Monet were are all provided as public domain by commons.wikimedia.org.