Hi,
Today, I got progress on the software part of the project. I started searching for stylization / style transfer implementations and I found one that may be suitable for the Raspberry Pi.
Image stylization / style transfer is a process the can be described as "extracting the style" from an image and "applying it" to a second image.
An example would be to take a painting (ex take Vincent van Gogh's The Starry Night) and apply to a photo (Stanford campus):
(example from the neural-style GitHub Repo)
Two early techniques of image style transfer are described in the following articles:
- A Neural Algorithm of Artistic Style
(Leon A. Gatys, Alexander S. Ecker, Matthias Bethge - Sep, 2015)- video description: Two Minute Papers #6 (Youtube)
- implementation: jcjohnson/neural-style (GitHub)
- Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis
(Chuan Li, Michael Wand - Jan, 2016)- video description: Two Minute Papers #49 (YouTube)
- implementation: chuanli11/CNNMRF (GitHub)
I tried to run examples from the two implementation, but I quickly realized that these will not work for the project. Both techniques needs to train convolutional neural networks "on-the-fly" to apply a style to an input image. The process takes significant time (minutes on a powerful GPU, hours on CPU). This means there is no way to get this running on the Raspberry Pi.
Fortunately, the research was ongoing on the topic and some techniques for real-time stylization / style transfer were developed.
I found the the fast_neural_style example in the PyTorch repository which implements style transfer using pre-trained models. The example is based on the following two articles:
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution
(Justin Johnson, Alexandre Alahi, Li Fei-Fei - Mar, 2016)
- Instance Normalization: The Missing Ingredient for Fast Stylization
(Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky - Nov, 2016)
The example can be run as follows:
$ git clone https://github.com/pytorch/examples.git pytorch-examples
$ cd pytorch-examples/
$ pip3 install torch torchvision
$ python3 download_saved_models.py
$ python3 neural_style/neural_style.py eval --content-image images/content-images/amber.jpg --model saved_models/<model>.pth --output-image amber-candy.jpg --cuda 0
Here is what I got by running the examples:
- input image
- output images
- "candy" model
- "mosaic" model
- "rain princess" model
- "Udnie" model
- "candy" model
The example run in ~ 6.5 seconds each on CPU, so they are relatively fast. I think, it can be ported to the Raspberry Pi and hopefully it will run in a reasonable amount of time.
Cheers,
Attila
Refs :
- A Neural Algorithm of Artistic Style
(Leon A. Gatys, Alexander S. Ecker, Matthias Bethge - Sep, 2015) - Perceptual Losses for Real-Time Style Transfer and Super-Resolution
(Justin Johnson, Alexandre Alahi, Li Fei-Fei - Mar, 2016)
- Instance Normalization: The Missing Ingredient for Fast Stylization
(Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky - Nov, 2016) - Towards Fast Neural Style Transfer
(Connor Shorten, towardsdatascience.com) - chuanli11/CNNMRF - code for paper "Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis"
(GitHub) - jcjohnson/neural-style - Torch implementation of neural style algorithm
(GitHub) - pytorch/examples - fast_neural_style
(GitHub)
Top Comments
-
dubbie
-
Cancel
-
Vote Up
+3
Vote Down
-
-
Sign in to reply
-
More
-
Cancel
Comment-
dubbie
-
Cancel
-
Vote Up
+3
Vote Down
-
-
Sign in to reply
-
More
-
Cancel
Children