<?xml version="1.0" encoding="UTF-8" ?>
<?xml-stylesheet type="text/xsl" href="https://community.element14.com/cfs-file/__key/system/syndication/rss.xsl" media="screen"?><rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/"><channel><title>Artificial Intelligence: Inference for Machine Learning</title><link>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning</link><description /><dc:language>en-US</dc:language><generator>Telligent Community 12</generator><item><title>Artificial Intelligence: Inference for Machine Learning</title><link>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning</link><pubDate>Sun, 27 Feb 2022 20:13:53 GMT</pubDate><guid isPermaLink="false">93d5dcb4-84c2-446f-b2cb-99731719e767:880c3939-2e7d-48c5-9b70-5674233690e7</guid><dc:creator>pchan</dc:creator><comments>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning#comments</comments><description>Current Revision posted to Documents by pchan on 2/27/2022 8:13:53 PM&lt;br /&gt;
&lt;p&gt;&lt;span&gt;&lt;div class="e14-navigation-buttons"&gt;&lt;a href="/w/documents/4964/element14-essentials-artificial-intelligence-ii?ICID=essentials-ai3-doc"&gt;
                    &lt;button class="e14-nav-button-prev"&gt;Previous&lt;/button&gt;
                &lt;/a&gt;&lt;a href="/w/documents/23124/artificial-intelligence-face-applications?ICID=essentials-ai3-doc"&gt;
                    &lt;button class="e14-nav-button-next"&gt;Next&lt;/button&gt;
                &lt;/a&gt;&lt;/div&gt;&lt;script&gt;e14.func.queueScripts.add(function(){ e14.func.repositionNavButtons(); })&lt;/script&gt;&lt;/span&gt;&lt;/p&gt;
&lt;div style="background-color:#f9f9f9;border:1px solid #c6c6c6;display:block;float:right;padding:6px 8px;width:212px;"&gt;&lt;a class="jivecontainerTT-hover-container jive-link-community-small" href="/learn/learning-center/?ICID=essentials-ai3-doc"&gt;element14 Learning Center&lt;/a&gt;&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="border:1px solid #c6c6c6;clear:both;padding:8px;"&gt;
&lt;div style="display:inline-block;padding-right:16px;vertical-align:top;"&gt;&lt;a href="/learn/learning-center/essentials/?ICID=essentials-ai3-doc"&gt;&lt;img alt="image"  src="/e14/assets/legacy/2017/learningess.png" width="175px" /&gt;&lt;/a&gt;&lt;/div&gt;
&lt;div style="display:inline-block;"&gt;
&lt;p class="essTitle" style="font-size:18px;font-weight:bold;margin:0;" title="Inference for Machine Learning - Artificial Intelligence"&gt;&lt;span style="color:#007fac;"&gt;Artificial Intelligence III: &lt;/span&gt; Inference for Machine Learning&lt;/p&gt;
&lt;p style="margin:0;padding:8px;"&gt;&lt;span style="font-size:11px;padding-right:10px;"&gt;&lt;em&gt;Sponsored by&lt;/em&gt;&lt;/span&gt;&lt;a href="https://www.xilinx.com/" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;&lt;img loading="lazy" alt="image" class="essSponsor" style="vertical-align:top;" title="Xilinx"  src="/e14/assets/main/mfg-group-assets/xilinxLogo.png" width="120px" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;hr /&gt;
&lt;div style="clear:both;line-height:1.5;"&gt;&lt;span style="padding-right:6px;"&gt;&lt;a class="jive-link-anchor-small" href="#intro"&gt;1. Introduction&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#object"&gt;2. Objectives&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec3"&gt;3. Neural Networks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec4"&gt;4. Frameworks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec5"&gt;5. Training&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec6"&gt;6. Implementing ML Inference &lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec7"&gt;7. Typical Applications&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec8"&gt;8. Conclusion&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-wiki-small" href="/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii?ICID=essentials-ai3-doc"&gt;Related Components&lt;/a&gt;&lt;/span&gt; | &lt;span style="font-weight:bold;padding:0px 6px;"&gt;&lt;a href="#test"&gt;Test Your Knowledge &lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/LinkArrow.gif" /&gt;&lt;/a&gt;&lt;/span&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;p style="color:#007fac;font-size:15px;margin:0;padding:8px 0px;"&gt;&lt;a name="intro"&gt;&lt;/a&gt;&lt;strong&gt;1. Introduction&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;We have all seen films where humankind implements an artificial intelligence (AI) system, something goes wrong, and it turns against its creators. 2001: A Space Odyssey and the Terminator films are classic examples. But what is artificial intelligence, really, and how do we use it in our applications, especially at the edge?&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;When it comes to working with artificial intelligence, we really are standing on the shoulders of giants. The term AI itself was first used by John McCarthy in 1956, when the earliest conference on artificial intelligence was held. Even before the AI term had been coined, however, there was considerable research being undertaken on the question of whether or not machines think. One early leading researcher was Alan Turing, the Bletchley Park code breaker, who posed the Turing Test. The test revolves around the question of whether we will ever be able to build a computer that can sufficiently imitate a human, to the point where a suspicious judge can&amp;#39;t tell the difference between what is human and what is machine.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;AI research has been a hot topic over the years. The late 1990s saw IBM&amp;#39;s Deep Blue supercomputer beat chess grandmasters. More recently, we have seen significant interest in artificial intelligence and machine learning at the edge, deployed within embedded systems.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;But what is machine learning? According to the book Machine Learning by Tom M. Mitchell, Professor at Carnegie Mellon University, &amp;quot;Machine learning is the study of computer algorithms that allow computer programs to automatically improve through experience.&amp;quot;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;A simple example of this is providing a machine learning algorithm with a data set of pictures with labels. The machine learning (ML) algorithm will be able to identify features and elements of the pictures, so that when a new unlabeled picture is input, the ML algorithm is able to determine how likely the image is to contain any of the learned features and elements. The classic example of this is the machine learning algorithm being able to identify pictures of animals, fruits, and so on.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These machine learning algorithms can, therefore, be said to have two distinct elements: training and inference. Training is process in which the features and elements that need to be identified are learned, while inference is using the trained behavior to examine unlabeled inputs to identify the elements and features once deployed.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="object"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin:0;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;&lt;strong&gt;2. Objectives&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 8px 0px;"&gt;This Essentials learning module will introduce you to the basics of artificial intelligence (AI) and machine learning (ML) inference. &lt;em&gt;The learning objectives are as follows:&lt;/em&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Gain an understanding of inference and its role in AI and ML systems.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Learn about neural networks and what they consist of.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Become familiar with the different network types that can be used for AI inference.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; See examples of different methodologies and frameworks which can be used for creation, training, and inference of AI and ML.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec3"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;3. Neural Networks&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;At the heart of the machine learning inference implementation is the neural network. This neural network is instantiated and trained to provide the required behaviour. Training can be very compute intensive, as large data sets are used to train the network. As such, the training most often takes place using a high-performance GPU unit rather than the deployed end solution. This enables much faster training of the network than would be achieved if the deployed system was used alone.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The term neural network is very generic, and includes a significant number of distinct subcategories whose names are normally used to identify the exact type of network being implemented. Neural networks are modeled upon the human cerebral cortex, in that each neuron receives an input, processes it, and communicates the processed signal to another neuron. Therefore, neural networks typically consist of an input layer, internal layer(s), and an output layer.&lt;/p&gt;
&lt;div style="float:left;padding:12px 10px 5px 0px;width:310px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_diagram.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox"  src="/e14/assets/legacy/2021/AI3_diagram.png" /&gt;&lt;/a&gt;
&lt;p style="font-size:11px;line-height:12px;margin:0;padding-top:3px;"&gt;&lt;em&gt;Figure 1: A Simple Neural Network &lt;/em&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;There are several classes of neural networks, with different classes used for different applications (e.g. speech recognition or image classification). Neural networks which pass the output of one layer to another without forming a cycle are called Feedforward Neural Networks (FNNs). Those which contain directed cycles where there is feedback, for example the Elman network, are called Recurrent Neural Networks (RNNs). One very commonly used term in many machine learning applications is Deep Neural Networks (DNNs). These are neural networks which have several hidden layers enabling a more complex machine learning task to be implemented.&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.1 Network Types&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The structure of the neural network, the number of stages, and which types of elements are used has been, and indeed continues to be, the subject of considerable research to obtain the best performance. Common network structures have evolved. Convolutional Neural Networks (CNNs), for example, are a type of feedforward neural network. Long Short-Term Memory (LSTM) is a type of Recurrent Neural Network. CNNs are exceedingly popular for image classification, while LSTM is a popular network for Natural Language Processing (NLP).&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Commonly used CNN network structures include:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AlexNet:&lt;/strong&gt; Invented by Alex Krizhevsky as part of his PhD Studies. AlexNet is an 8-layer deep network which completed the ImageNet Large Scale Visual Recognition Challenge in 2012.&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;GoogLeNet:&lt;/strong&gt; A 22-layer CNN first described in 2014 by Google.&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;VGG-16:&lt;/strong&gt; A 16-layer CNN described by the Visual Geometry Group at the University of Oxford in 2014.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.2 Convolutional Neural Networks &lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;What separates embedded vision applications from other, simpler machine learning applications is that they have a two-dimensional input format. As such, in machine learning implementations, a Convolutional Neural Network is used, as it can process two-dimensional inputs. As outlined above, several different CNN implementations exist which have evolved over the years. However, they consist of multiple stages of the same basic functional blocks, with different parameterizations throughout the implementation. These stages are:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Convolution:&lt;/strong&gt; In this stage, the input image is broken down into several overlapping smaller tiles by the application of a filter. The results from this convolution are used to create a feature map.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Rectified Linear Unit (reLU):&lt;/strong&gt; This activation layer is used to create an activation map following a convolution.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Max Pooling:&lt;/strong&gt; Performs subsampling between layers to reduce resolution.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Fully Connected:&lt;/strong&gt; This is a matrix multiplication layer which applies both weights and a bias to the input. It is called a fully connected layer because the input and output are fully connected.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These networks and stages are defined and trained within an ML framework. The ML framework provides not only a range of libraries, but also pre-existing network models thanks to its Model Zoo, and pre-trained weights that are applied to the model to quickly get networks up and running. Training is required to define the parameters for the convolution (filter weights) and fully connected layers (weights and bias), while the pool and reLU elements require no parameterization. Both the pool and convolution layers, however, require parameterization to define the size of the filter kernel and the stride. How we determine the values which are used for these is called training. One of the advantages of using a CNN is the relative ease of training the network.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec4"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;4. Frameworks&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Of course, machine learning inference is a complex area, and starting from scratch could be daunting. However, several frameworks exist which enable us to get started creating our solution. Here are some of the most commonly used frameworks:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;TensorFlow:&lt;/strong&gt; Developed by the Google Brain Team and first released publicly in 2015, TensorFlow provides both Python and C APIs.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Caffe:&lt;/strong&gt; This convolutional architecture for fast-feature embedding was developed by the University of California at Berkley. Caffe supports both C and Python.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;PyTorch:&lt;/strong&gt; A Python implementation of torch primarily developed by the FaceBook AI research group.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;tinyML:&lt;/strong&gt; TensorFlow Lite, developed to enable ML implementations on Arm Cortex-M class devices, ESP32 devices, and other edge processors.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These frameworks provide everything required to instantiate a network, train the network, and deploy the network on the target device.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec5"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;5. Training&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The training of a network requires a large data set consisting of many thousands of correct and incorrect sources.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;There are three types of training routinely used for an ML network:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Supervised Training:&lt;/strong&gt; A tagged data set is used. The network iteratively makes output predictions based on the tagged input data. These output predictions are corrected by a supervisor based upon the tag data, meaning that when untagged data is input, the output prediction is correct within an accuracy. Supervised training is used for the majority of neural networks.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Unsupervised Training:&lt;/strong&gt; Untagged data sources are used. These are used for neural network implementations with no correct answer (e.g. clustering and association).&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Reinforcement Learning:&lt;/strong&gt; Uses the interaction with the network&amp;#39;s environment to maximize the notion of cumulative reward.&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding-top:12px;"&gt;When structured training is performed, an additional layer is introduced, called the loss function. This stage enables the training algorithm to know if the network correctly identified the input image. During training, both forward and backward propagation is used to determine if the image is correctly classified, and updates are made to the weights and biases based on the error signals and calculated gradients. To apply the image sets and calculate the network coefficients as quickly and as efficiently as possible, large farms of Graphical Processing Units (GPUs) are used. GPU farms are used because the goal of training is to generate the weights and biases within the minimal time frame. Therefore, power efficiency, real-time response, and determinism for each image frame is not as critical as it is within the final inference deployment. Therefore, GPU farms provide the most efficient mechanism for determining weights and biases.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec6"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;6. Implementing ML Inference &lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;While the training and creation of an ML solution will take place using GPU farms, the ML inference application will be deployed on many different edge technologies, such as the following examples.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Edge-Based Applications&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;CPU:&lt;/strong&gt; General-purpose CPUs, such as Arm-based microcontrollers like the Arm Cortex-M3 processor&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;GPU:&lt;/strong&gt; Graphics Processor Units, such as the Nvidia System-on-Module&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;FPGA:&lt;/strong&gt; Field Programmable Gate Array, such as Xilinx heterogeneous SoCs and MPSoCs like the Zynq-7000 or the &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii#mpsoc"&gt;Zynq UltraScale+ MPSoC&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Cloud-Based Applications&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;High Performance CPU:&lt;/strong&gt; x86 or Arm-based higher performance server processor&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Graphics Cards:&lt;/strong&gt; High-performance graphics co-processors&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Acceleration Card:&lt;/strong&gt; FPGA-based acceleration cards connected to the x86 server hardware using PCIe&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Each end implementation technology will have different deployment needs. This is where technology-specific tool chains are used for the deployment. Examples of this include the &lt;a class="jive-link-external-small" href="https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;Vitis AI tool&lt;/a&gt; from Xilinx or the &lt;a class="jive-link-external-small" href="https://www.tensorflow.org/lite/microcontrollers/build_convert#model_conversion" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;TensorFlow Lite converter&lt;/a&gt;.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;One large deciding factor in this situation is the capability of the underlying hardware. For example, many neural networks are defined and trained as floating-point implementations. However, at the edge, fixed-point representation may make for easier and more efficient implementations. Using fixed point may also enable the use of lower-cost hardware. Even in the cloud, FP32 implementation may still provide a faster, more powerful, and efficient implementation.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Machine learning applications are increasingly using more efficient, reduced precision, fixed-point number systems such as INT8 representation. The use of fixed-point reduced precision number systems comes without a significant loss in accuracy, when compared to a traditional floating point 32 (FP32) approach. However, fixed-point mathematics is considerably easier to implement than floating point. This move to INT8 provides for more efficient, faster solutions in some implementations. To support this generation of reduced precision number systems, the automated &lt;a class="jive-link-external-small" href="https://github.com/hotzeng/Ristretto-caffe" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;Ristretto tool for Caffe&lt;/a&gt; enables the training of networks using fixed-point number systems. This removes the need (and associated potential performance impact) of converting floating-point training data into fixed-point training data.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;If you are targeting a Xilinx programmable logic device, the Vitis AI tool also provides several tools which support implementation of models in programmable logic.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Optimizer:&lt;/strong&gt; Optimizes the floating-point model to reduce model complexity&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Quantizer:&lt;/strong&gt; Quantizes the floating-point model weights and biases into an INT8 representation&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Compiler:&lt;/strong&gt; Compiles the quantized model to be deployed on the Xilinx Deep Learning Processor Unit (DPU)&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Using Vitis AI, networks defined within Caffe, TensorFlow, and PyTorch can be deployed on programmable logic solutions. This allows AI developers to leverage the benefits which come with implementation of a solution in programmable logic.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec7"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;7. Typical Applications&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Edge-based ML applications are increasingly growing in popularity. Typical examples include use in both the industrial and automotive solutions spaces.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Within the automotive space, ML can be used for several different solutions. One implementation is in road sign recognition and classification, as part of an Advanced Driver Assistance program. Another application is internal cabin monitoring, where the behavior of the occupants is used to set up environmental conditions.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;When ML is deployed at the edge, one of the main aspects of its use for industrial applications can be predictive maintenance, where sensors and data are used to predict the failure or wear out of materials ending their lifecycle.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec8"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;8. Conclusion&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Embedded systems at the edge are increasingly deployed using machine learning algorithms to enable intelligent processing. Throughout this Essentials course, we have examined the history of AI and ML, neural networks, network types, frameworks, training, and deployment at the edge. Now, all that is left is for you to try creating your first ML application!&lt;/p&gt;
&lt;p style="font-size:11px;margin:0;padding-top:12px;"&gt;*Trademark. &lt;strong&gt;Xilinx is a trademark of Xilinx Inc.&lt;/strong&gt;&amp;nbsp; Other logos, product and/or company names may be trademarks of their respective owners.&lt;/p&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;border:1px solid #a9aaaa;padding:12px 12px 0px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_profile.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox essProfile" style="float:left;padding-right:10px;vertical-align:middle;"  height="115" src="/e14/assets/legacy/2021/AI3_profile.png" /&gt;&lt;/a&gt;
&lt;div&gt;
&lt;p style="color:#007fac;font-size:15px;margin:0;padding-bottom:10px;"&gt;Shop our wide range of &lt;strong&gt;components, development boards, and evaluation kits used for artificial intelligence and machine learning solutions.&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;span class="e14-button-large e14-button-primary"&gt;&lt;a class="jive-link-wiki-small" title="Shop Now" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii"&gt;Shop Now&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:20px;"&gt;&lt;a name="test"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="color:#007fac;font-size:18px;margin:0;padding-bottom:8px;"&gt;&lt;strong&gt;Test Your Knowledge&lt;/strong&gt;&lt;span style="float:right;font-size:11px;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a href="/e14/assets/legacy/2021/100x100_AI3Badge.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox" style="float:left;padding:0px 10px 5px 0px;vertical-align:top;"  height="80" src="/e14/assets/legacy/2021/100x100_AI3Badge.png" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-bottom:8px;"&gt;Are you ready to demonstrate your knowledge of Inference for Machine Learning? &lt;strong&gt;Then take this 15-question quiz to see how much you&amp;#39;ve learned.&lt;/strong&gt; To earn the AI III Badge, read through the learning module, score 100% on the Quiz, leave us some feedback in the comments section, and give the learning module a star rating.&lt;/p&gt;
&lt;p style="margin:0;padding:0px;"&gt;&lt;div class="voltquiz" data-formid="a5c10660-9227-47a4-8252-886bb231de44" data-formtype="quiz" data-success-message="" data-fail-message="" data-pass-mark="OTk=" &gt;&lt;/div&gt;&amp;nbsp;&lt;/p&gt;
&lt;div class="e14-quiz-document-metadata e14-init-hidden"&gt;&lt;br /&gt;&lt;br /&gt;&lt;/div&gt;
&lt;div class="migration-injected-attachments"&gt;
&lt;div style="font-weight:bold;margin:15px 0 5px 0;"&gt;Attachments:&lt;/div&gt;
&lt;div&gt;&lt;/div&gt;
&lt;table style="border:0;"&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td width="160"&gt;&lt;span id="attid_https://www.element14.com/community/api/core/v3/attachments/313552" class="_se_attachment"&gt;&lt;a href="https://community.element14.com/cfs-file/__key/communityserver-wikis-components-files/00-00-00-02-50/AI3.pdf"&gt;community.element14.com/.../AI3.pdf&lt;/a&gt;&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;span style="font-size:80%;"&gt;AI3.pdf&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;&lt;div style="clear:both;"&gt;&lt;/div&gt;

&lt;div style="font-size: 90%;"&gt;Tags: fpga_essentials, machine_learning, xilinx, inference_for_machine_learning, ai, artificial_intelligence, ess_module&lt;/div&gt;
</description></item><item><title>Artificial Intelligence Inference for Machine Learning</title><link>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning/revision/6</link><pubDate>Sun, 27 Feb 2022 18:08:39 GMT</pubDate><guid isPermaLink="false">93d5dcb4-84c2-446f-b2cb-99731719e767:880c3939-2e7d-48c5-9b70-5674233690e7</guid><dc:creator>pchan</dc:creator><comments>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning#comments</comments><description>Revision 6 posted to Documents by pchan on 2/27/2022 6:08:39 PM&lt;br /&gt;
&lt;p&gt;&lt;span&gt;&lt;div class="e14-navigation-buttons"&gt;&lt;a href="/w/documents/4964/element14-essentials-artificial-intelligence-ii?ICID=essentials-ai3-doc"&gt;
                    &lt;button class="e14-nav-button-prev"&gt;Previous&lt;/button&gt;
                &lt;/a&gt;&lt;a href="/w/documents/23124/artificial-intelligence-face-applications?ICID=essentials-ai3-doc"&gt;
                    &lt;button class="e14-nav-button-next"&gt;Next&lt;/button&gt;
                &lt;/a&gt;&lt;/div&gt;&lt;script&gt;e14.func.queueScripts.add(function(){ e14.func.repositionNavButtons(); })&lt;/script&gt;&lt;/span&gt;&lt;/p&gt;
&lt;div style="background-color:#f9f9f9;border:1px solid #c6c6c6;display:block;float:right;padding:6px 8px;width:212px;"&gt;&lt;a class="jivecontainerTT-hover-container jive-link-community-small" href="/learn/learning-center/?ICID=essentials-ai3-doc"&gt;element14 Learning Center&lt;/a&gt;&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="border:1px solid #c6c6c6;clear:both;padding:8px;"&gt;
&lt;div style="display:inline-block;padding-right:16px;vertical-align:top;"&gt;&lt;a href="/learn/learning-center/essentials/?ICID=essentials-ai3-doc"&gt;&lt;img alt="image"  src="/e14/assets/legacy/2017/learningess.png" width="175px" /&gt;&lt;/a&gt;&lt;/div&gt;
&lt;div style="display:inline-block;"&gt;
&lt;p class="essTitle" style="font-size:18px;font-weight:bold;margin:0;" title="Inference for Machine Learning - Artificial Intelligence"&gt;&lt;span style="color:#007fac;"&gt;Artificial Intelligence III: &lt;/span&gt; Inference for Machine Learning&lt;/p&gt;
&lt;p style="margin:0;padding:8px;"&gt;&lt;span style="font-size:11px;padding-right:10px;"&gt;&lt;em&gt;Sponsored by&lt;/em&gt;&lt;/span&gt;&lt;a href="https://www.xilinx.com/" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;&lt;img loading="lazy" alt="image" class="essSponsor" style="vertical-align:top;" title="Xilinx"  src="/e14/assets/main/mfg-group-assets/xilinxLogo.png" width="120px" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;hr /&gt;
&lt;div style="clear:both;line-height:1.5;"&gt;&lt;span style="padding-right:6px;"&gt;&lt;a class="jive-link-anchor-small" href="#intro"&gt;1. Introduction&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#object"&gt;2. Objectives&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec3"&gt;3. Neural Networks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec4"&gt;4. Frameworks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec5"&gt;5. Training&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec6"&gt;6. Implementing ML Inference &lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec7"&gt;7. Typical Applications&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec8"&gt;8. Conclusion&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-wiki-small" href="/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii?ICID=essentials-ai3-doc"&gt;Related Components&lt;/a&gt;&lt;/span&gt; | &lt;span style="font-weight:bold;padding:0px 6px;"&gt;&lt;a href="#test"&gt;Test Your Knowledge &lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/LinkArrow.gif" /&gt;&lt;/a&gt;&lt;/span&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;p style="color:#007fac;font-size:15px;margin:0;padding:8px 0px;"&gt;&lt;a name="intro"&gt;&lt;/a&gt;&lt;strong&gt;1. Introduction&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;We have all seen films where humankind implements an artificial intelligence (AI) system, something goes wrong, and it turns against its creators. 2001: A Space Odyssey and the Terminator films are classic examples. But what is artificial intelligence, really, and how do we use it in our applications, especially at the edge?&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;When it comes to working with artificial intelligence, we really are standing on the shoulders of giants. The term AI itself was first used by John McCarthy in 1956, when the earliest conference on artificial intelligence was held. Even before the AI term had been coined, however, there was considerable research being undertaken on the question of whether or not machines think. One early leading researcher was Alan Turing, the Bletchley Park code breaker, who posed the Turing Test. The test revolves around the question of whether we will ever be able to build a computer that can sufficiently imitate a human, to the point where a suspicious judge can&amp;#39;t tell the difference between what is human and what is machine.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;AI research has been a hot topic over the years. The late 1990s saw IBM&amp;#39;s Deep Blue supercomputer beat chess grandmasters. More recently, we have seen significant interest in artificial intelligence and machine learning at the edge, deployed within embedded systems.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;But what is machine learning? According to the book Machine Learning by Tom M. Mitchell, Professor at Carnegie Mellon University, &amp;quot;Machine learning is the study of computer algorithms that allow computer programs to automatically improve through experience.&amp;quot;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;A simple example of this is providing a machine learning algorithm with a data set of pictures with labels. The machine learning (ML) algorithm will be able to identify features and elements of the pictures, so that when a new unlabeled picture is input, the ML algorithm is able to determine how likely the image is to contain any of the learned features and elements. The classic example of this is the machine learning algorithm being able to identify pictures of animals, fruits, and so on.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These machine learning algorithms can, therefore, be said to have two distinct elements: training and inference. Training is process in which the features and elements that need to be identified are learned, while inference is using the trained behavior to examine unlabeled inputs to identify the elements and features once deployed.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="object"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin:0;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;&lt;strong&gt;2. Objectives&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 8px 0px;"&gt;This Essentials learning module will introduce you to the basics of artificial intelligence (AI) and machine learning (ML) inference. &lt;em&gt;The learning objectives are as follows:&lt;/em&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Gain an understanding of inference and its role in AI and ML systems.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Learn about neural networks and what they consist of.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Become familiar with the different network types that can be used for AI inference.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; See examples of different methodologies and frameworks which can be used for creation, training, and inference of AI and ML.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec3"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;3. Neural Networks&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;At the heart of the machine learning inference implementation is the neural network. This neural network is instantiated and trained to provide the required behaviour. Training can be very compute intensive, as large data sets are used to train the network. As such, the training most often takes place using a high-performance GPU unit rather than the deployed end solution. This enables much faster training of the network than would be achieved if the deployed system was used alone.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The term neural network is very generic, and includes a significant number of distinct subcategories whose names are normally used to identify the exact type of network being implemented. Neural networks are modeled upon the human cerebral cortex, in that each neuron receives an input, processes it, and communicates the processed signal to another neuron. Therefore, neural networks typically consist of an input layer, internal layer(s), and an output layer.&lt;/p&gt;
&lt;div style="float:left;padding:12px 10px 5px 0px;width:310px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_diagram.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox"  src="/e14/assets/legacy/2021/AI3_diagram.png" /&gt;&lt;/a&gt;
&lt;p style="font-size:11px;line-height:12px;margin:0;padding-top:3px;"&gt;&lt;em&gt;Figure 1: A Simple Neural Network &lt;/em&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;There are several classes of neural networks, with different classes used for different applications (e.g. speech recognition or image classification). Neural networks which pass the output of one layer to another without forming a cycle are called Feedforward Neural Networks (FNNs). Those which contain directed cycles where there is feedback, for example the Elman network, are called Recurrent Neural Networks (RNNs). One very commonly used term in many machine learning applications is Deep Neural Networks (DNNs). These are neural networks which have several hidden layers enabling a more complex machine learning task to be implemented.&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.1 Network Types&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The structure of the neural network, the number of stages, and which types of elements are used has been, and indeed continues to be, the subject of considerable research to obtain the best performance. Common network structures have evolved. Convolutional Neural Networks (CNNs), for example, are a type of feedforward neural network. Long Short-Term Memory (LSTM) is a type of Recurrent Neural Network. CNNs are exceedingly popular for image classification, while LSTM is a popular network for Natural Language Processing (NLP).&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Commonly used CNN network structures include:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AlexNet:&lt;/strong&gt; Invented by Alex Krizhevsky as part of his PhD Studies. AlexNet is an 8-layer deep network which completed the ImageNet Large Scale Visual Recognition Challenge in 2012.&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;GoogLeNet:&lt;/strong&gt; A 22-layer CNN first described in 2014 by Google.&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;VGG-16:&lt;/strong&gt; A 16-layer CNN described by the Visual Geometry Group at the University of Oxford in 2014.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.2 Convolutional Neural Networks &lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;What separates embedded vision applications from other, simpler machine learning applications is that they have a two-dimensional input format. As such, in machine learning implementations, a Convolutional Neural Network is used, as it can process two-dimensional inputs. As outlined above, several different CNN implementations exist which have evolved over the years. However, they consist of multiple stages of the same basic functional blocks, with different parameterizations throughout the implementation. These stages are:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Convolution:&lt;/strong&gt; In this stage, the input image is broken down into several overlapping smaller tiles by the application of a filter. The results from this convolution are used to create a feature map.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Rectified Linear Unit (reLU):&lt;/strong&gt; This activation layer is used to create an activation map following a convolution.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Max Pooling:&lt;/strong&gt; Performs subsampling between layers to reduce resolution.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Fully Connected:&lt;/strong&gt; This is a matrix multiplication layer which applies both weights and a bias to the input. It is called a fully connected layer because the input and output are fully connected.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These networks and stages are defined and trained within an ML framework. The ML framework provides not only a range of libraries, but also pre-existing network models thanks to its Model Zoo, and pre-trained weights that are applied to the model to quickly get networks up and running. Training is required to define the parameters for the convolution (filter weights) and fully connected layers (weights and bias), while the pool and reLU elements require no parameterization. Both the pool and convolution layers, however, require parameterization to define the size of the filter kernel and the stride. How we determine the values which are used for these is called training. One of the advantages of using a CNN is the relative ease of training the network.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec4"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;4. Frameworks&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Of course, machine learning inference is a complex area, and starting from scratch could be daunting. However, several frameworks exist which enable us to get started creating our solution. Here are some of the most commonly used frameworks:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;TensorFlow:&lt;/strong&gt; Developed by the Google Brain Team and first released publicly in 2015, TensorFlow provides both Python and C APIs.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Caffe:&lt;/strong&gt; This convolutional architecture for fast-feature embedding was developed by the University of California at Berkley. Caffe supports both C and Python.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;PyTorch:&lt;/strong&gt; A Python implementation of torch primarily developed by the FaceBook AI research group.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;tinyML:&lt;/strong&gt; TensorFlow Lite, developed to enable ML implementations on Arm Cortex-M class devices, ESP32 devices, and other edge processors.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These frameworks provide everything required to instantiate a network, train the network, and deploy the network on the target device.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec5"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;5. Training&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The training of a network requires a large data set consisting of many thousands of correct and incorrect sources.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;There are three types of training routinely used for an ML network:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Supervised Training:&lt;/strong&gt; A tagged data set is used. The network iteratively makes output predictions based on the tagged input data. These output predictions are corrected by a supervisor based upon the tag data, meaning that when untagged data is input, the output prediction is correct within an accuracy. Supervised training is used for the majority of neural networks.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Unsupervised Training:&lt;/strong&gt; Untagged data sources are used. These are used for neural network implementations with no correct answer (e.g. clustering and association).&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Reinforcement Learning:&lt;/strong&gt; Uses the interaction with the network&amp;#39;s environment to maximize the notion of cumulative reward.&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding-top:12px;"&gt;When structured training is performed, an additional layer is introduced, called the loss function. This stage enables the training algorithm to know if the network correctly identified the input image. During training, both forward and backward propagation is used to determine if the image is correctly classified, and updates are made to the weights and biases based on the error signals and calculated gradients. To apply the image sets and calculate the network coefficients as quickly and as efficiently as possible, large farms of Graphical Processing Units (GPUs) are used. GPU farms are used because the goal of training is to generate the weights and biases within the minimal time frame. Therefore, power efficiency, real-time response, and determinism for each image frame is not as critical as it is within the final inference deployment. Therefore, GPU farms provide the most efficient mechanism for determining weights and biases.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec6"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;6. Implementing ML Inference &lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;While the training and creation of an ML solution will take place using GPU farms, the ML inference application will be deployed on many different edge technologies, such as the following examples.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Edge-Based Applications&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;CPU:&lt;/strong&gt; General-purpose CPUs, such as Arm-based microcontrollers like the Arm Cortex-M3 processor&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;GPU:&lt;/strong&gt; Graphics Processor Units, such as the Nvidia System-on-Module&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;FPGA:&lt;/strong&gt; Field Programmable Gate Array, such as Xilinx heterogeneous SoCs and MPSoCs like the Zynq-7000 or the &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii#mpsoc"&gt;Zynq UltraScale+ MPSoC&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Cloud-Based Applications&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;High Performance CPU:&lt;/strong&gt; x86 or Arm-based higher performance server processor&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Graphics Cards:&lt;/strong&gt; High-performance graphics co-processors&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Acceleration Card:&lt;/strong&gt; FPGA-based acceleration cards connected to the x86 server hardware using PCIe&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Each end implementation technology will have different deployment needs. This is where technology-specific tool chains are used for the deployment. Examples of this include the &lt;a class="jive-link-external-small" href="https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;Vitis AI tool&lt;/a&gt; from Xilinx or the &lt;a class="jive-link-external-small" href="https://www.tensorflow.org/lite/microcontrollers/build_convert#model_conversion" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;TensorFlow Lite converter&lt;/a&gt;.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;One large deciding factor in this situation is the capability of the underlying hardware. For example, many neural networks are defined and trained as floating-point implementations. However, at the edge, fixed-point representation may make for easier and more efficient implementations. Using fixed point may also enable the use of lower-cost hardware. Even in the cloud, FP32 implementation may still provide a faster, more powerful, and efficient implementation.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Machine learning applications are increasingly using more efficient, reduced precision, fixed-point number systems such as INT8 representation. The use of fixed-point reduced precision number systems comes without a significant loss in accuracy, when compared to a traditional floating point 32 (FP32) approach. However, fixed-point mathematics is considerably easier to implement than floating point. This move to INT8 provides for more efficient, faster solutions in some implementations. To support this generation of reduced precision number systems, the automated &lt;a class="jive-link-external-small" href="https://github.com/hotzeng/Ristretto-caffe" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;Ristretto tool for Caffe&lt;/a&gt; enables the training of networks using fixed-point number systems. This removes the need (and associated potential performance impact) of converting floating-point training data into fixed-point training data.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;If you are targeting a Xilinx programmable logic device, the Vitis AI tool also provides several tools which support implementation of models in programmable logic.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Optimizer:&lt;/strong&gt; Optimizes the floating-point model to reduce model complexity&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Quantizer:&lt;/strong&gt; Quantizes the floating-point model weights and biases into an INT8 representation&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Compiler:&lt;/strong&gt; Compiles the quantized model to be deployed on the Xilinx Deep Learning Processor Unit (DPU)&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Using Vitis AI, networks defined within Caffe, TensorFlow, and PyTorch can be deployed on programmable logic solutions. This allows AI developers to leverage the benefits which come with implementation of a solution in programmable logic.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec7"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;7. Typical Applications&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Edge-based ML applications are increasingly growing in popularity. Typical examples include use in both the industrial and automotive solutions spaces.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Within the automotive space, ML can be used for several different solutions. One implementation is in road sign recognition and classification, as part of an Advanced Driver Assistance program. Another application is internal cabin monitoring, where the behavior of the occupants is used to set up environmental conditions.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;When ML is deployed at the edge, one of the main aspects of its use for industrial applications can be predictive maintenance, where sensors and data are used to predict the failure or wear out of materials ending their lifecycle.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec8"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;8. Conclusion&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Embedded systems at the edge are increasingly deployed using machine learning algorithms to enable intelligent processing. Throughout this Essentials course, we have examined the history of AI and ML, neural networks, network types, frameworks, training, and deployment at the edge. Now, all that is left is for you to try creating your first ML application!&lt;/p&gt;
&lt;p style="font-size:11px;margin:0;padding-top:12px;"&gt;*Trademark. &lt;strong&gt;Xilinx is a trademark of Xilinx Inc.&lt;/strong&gt;&amp;nbsp; Other logos, product and/or company names may be trademarks of their respective owners.&lt;/p&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;border:1px solid #a9aaaa;padding:12px 12px 0px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_profile.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox essProfile" style="float:left;padding-right:10px;vertical-align:middle;"  height="115" src="/e14/assets/legacy/2021/AI3_profile.png" /&gt;&lt;/a&gt;
&lt;div&gt;
&lt;p style="color:#007fac;font-size:15px;margin:0;padding-bottom:10px;"&gt;Shop our wide range of &lt;strong&gt;components, development boards, and evaluation kits used for artificial intelligence and machine learning solutions.&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;span class="e14-button-large e14-button-primary"&gt;&lt;a class="jive-link-wiki-small" title="Shop Now" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii"&gt;Shop Now&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:20px;"&gt;&lt;a name="test"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="color:#007fac;font-size:18px;margin:0;padding-bottom:8px;"&gt;&lt;strong&gt;Test Your Knowledge&lt;/strong&gt;&lt;span style="float:right;font-size:11px;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a href="/e14/assets/legacy/2021/100x100_AI3Badge.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox" style="float:left;padding:0px 10px 5px 0px;vertical-align:top;"  height="80" src="/e14/assets/legacy/2021/100x100_AI3Badge.png" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-bottom:8px;"&gt;Are you ready to demonstrate your knowledge of Inference for Machine Learning? &lt;strong&gt;Then take this 15-question quiz to see how much you&amp;#39;ve learned.&lt;/strong&gt; To earn the AI III Badge, read through the learning module, score 100% on the Quiz, leave us some feedback in the comments section, and give the learning module a star rating.&lt;/p&gt;
&lt;p style="margin:0;padding:0px;"&gt;&lt;div class="voltquiz" data-formid="a5c10660-9227-47a4-8252-886bb231de44" data-formtype="quiz" data-success-message="" data-fail-message="" data-pass-mark="OTk=" &gt;&lt;/div&gt;&amp;nbsp;&lt;/p&gt;
&lt;div class="e14-quiz-document-metadata e14-init-hidden"&gt;&lt;br /&gt;&lt;br /&gt;&lt;/div&gt;
&lt;div class="migration-injected-attachments"&gt;
&lt;div style="font-weight:bold;margin:15px 0 5px 0;"&gt;Attachments:&lt;/div&gt;
&lt;div&gt;&lt;/div&gt;
&lt;table style="border:0;"&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td width="160"&gt;&lt;span id="attid_https://www.element14.com/community/api/core/v3/attachments/313552" class="_se_attachment"&gt;&lt;a href="https://community.element14.com/cfs-file/__key/communityserver-wikis-components-files/00-00-00-02-50/AI3.pdf"&gt;community.element14.com/.../AI3.pdf&lt;/a&gt;&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;span style="font-size:80%;"&gt;AI3.pdf&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;&lt;div style="clear:both;"&gt;&lt;/div&gt;

&lt;div style="font-size: 90%;"&gt;Tags: fpga_essentials, machine_learning, xilinx, inference_for_machine_learning, ai, artificial_intelligence, ess_module&lt;/div&gt;
</description></item><item><title>Artificial Intelligence Inference for Machine Learning</title><link>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning/revision/5</link><pubDate>Sun, 27 Feb 2022 18:04:44 GMT</pubDate><guid isPermaLink="false">93d5dcb4-84c2-446f-b2cb-99731719e767:880c3939-2e7d-48c5-9b70-5674233690e7</guid><dc:creator>pchan</dc:creator><comments>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning#comments</comments><description>Revision 5 posted to Documents by pchan on 2/27/2022 6:04:44 PM&lt;br /&gt;
&lt;p&gt;&lt;span&gt;&lt;div class="e14-navigation-buttons"&gt;&lt;a href="/w/documents/4964/element14-essentials-artificial-intelligence-ii?ICID=essentials-ai3-doc"&gt;
                    &lt;button class="e14-nav-button-prev"&gt;Previous&lt;/button&gt;
                &lt;/a&gt;&lt;a href="/w/documents/23124/artificial-intelligence-face-applications?ICID=essentials-ai3-doc"&gt;
                    &lt;button class="e14-nav-button-next"&gt;Next&lt;/button&gt;
                &lt;/a&gt;&lt;/div&gt;&lt;script&gt;e14.func.queueScripts.add(function(){ e14.func.repositionNavButtons(); })&lt;/script&gt;&lt;/span&gt;&lt;/p&gt;
&lt;div style="background-color:#f9f9f9;border:1px solid #c6c6c6;display:block;float:right;padding:6px 8px;width:212px;"&gt;&lt;a class="jivecontainerTT-hover-container jive-link-community-small" href="/learn/learning-center/?ICID=essentials-ai3-doc"&gt;element14 Learning Center&lt;/a&gt;&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="border:1px solid #c6c6c6;clear:both;padding:8px;"&gt;
&lt;div style="display:inline-block;padding-right:16px;vertical-align:top;"&gt;&lt;a href="/learn/learning-center/essentials/?ICID=essentials-ai3-doc"&gt;&lt;img alt="image"  src="/e14/assets/legacy/2017/learningess.png" width="175px" /&gt;&lt;/a&gt;&lt;/div&gt;
&lt;div style="display:inline-block;"&gt;
&lt;p class="essTitle" style="font-size:18px;font-weight:bold;margin:0;" title="Inference for Machine Learning - Artificial Intelligence"&gt;&lt;span style="color:#007fac;"&gt;Artificial Intelligence III: &lt;/span&gt; Inference for Machine Learning&lt;/p&gt;
&lt;p style="margin:0;padding:8px;"&gt;&lt;span style="font-size:11px;padding-right:10px;"&gt;&lt;em&gt;Sponsored by&lt;/em&gt;&lt;/span&gt;&lt;a href="https://www.xilinx.com/" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;&lt;img loading="lazy" alt="image" class="essSponsor" style="vertical-align:top;" title="Xilinx"  src="/e14/assets/main/mfg-group-assets/xilinxLogo.png" width="120px" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;hr /&gt;
&lt;div style="clear:both;line-height:1.5;"&gt;&lt;span style="padding-right:6px;"&gt;&lt;a class="jive-link-anchor-small" href="#intro"&gt;1. Introduction&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#object"&gt;2. Objectives&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec3"&gt;3. Neural Networks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec4"&gt;4. Frameworks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec5"&gt;5. Training&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec6"&gt;6. Implementing ML Inference &lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec7"&gt;7. Typical Applications&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec8"&gt;8. Conclusion&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii"&gt;Related Components&lt;/a&gt;&lt;/span&gt; | &lt;span style="font-weight:bold;padding:0px 6px;"&gt;&lt;a href="#test"&gt;Test Your Knowledge &lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/LinkArrow.gif" /&gt;&lt;/a&gt;&lt;/span&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;p style="color:#007fac;font-size:15px;margin:0;padding:8px 0px;"&gt;&lt;a name="intro"&gt;&lt;/a&gt;&lt;strong&gt;1. Introduction&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;We have all seen films where humankind implements an artificial intelligence (AI) system, something goes wrong, and it turns against its creators. 2001: A Space Odyssey and the Terminator films are classic examples. But what is artificial intelligence, really, and how do we use it in our applications, especially at the edge?&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;When it comes to working with artificial intelligence, we really are standing on the shoulders of giants. The term AI itself was first used by John McCarthy in 1956, when the earliest conference on artificial intelligence was held. Even before the AI term had been coined, however, there was considerable research being undertaken on the question of whether or not machines think. One early leading researcher was Alan Turing, the Bletchley Park code breaker, who posed the Turing Test. The test revolves around the question of whether we will ever be able to build a computer that can sufficiently imitate a human, to the point where a suspicious judge can&amp;#39;t tell the difference between what is human and what is machine.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;AI research has been a hot topic over the years. The late 1990s saw IBM&amp;#39;s Deep Blue supercomputer beat chess grandmasters. More recently, we have seen significant interest in artificial intelligence and machine learning at the edge, deployed within embedded systems.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;But what is machine learning? According to the book Machine Learning by Tom M. Mitchell, Professor at Carnegie Mellon University, &amp;quot;Machine learning is the study of computer algorithms that allow computer programs to automatically improve through experience.&amp;quot;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;A simple example of this is providing a machine learning algorithm with a data set of pictures with labels. The machine learning (ML) algorithm will be able to identify features and elements of the pictures, so that when a new unlabeled picture is input, the ML algorithm is able to determine how likely the image is to contain any of the learned features and elements. The classic example of this is the machine learning algorithm being able to identify pictures of animals, fruits, and so on.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These machine learning algorithms can, therefore, be said to have two distinct elements: training and inference. Training is process in which the features and elements that need to be identified are learned, while inference is using the trained behavior to examine unlabeled inputs to identify the elements and features once deployed.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="object"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin:0;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;&lt;strong&gt;2. Objectives&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 8px 0px;"&gt;This Essentials learning module will introduce you to the basics of artificial intelligence (AI) and machine learning (ML) inference. &lt;em&gt;The learning objectives are as follows:&lt;/em&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Gain an understanding of inference and its role in AI and ML systems.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Learn about neural networks and what they consist of.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Become familiar with the different network types that can be used for AI inference.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; See examples of different methodologies and frameworks which can be used for creation, training, and inference of AI and ML.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec3"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;3. Neural Networks&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;At the heart of the machine learning inference implementation is the neural network. This neural network is instantiated and trained to provide the required behaviour. Training can be very compute intensive, as large data sets are used to train the network. As such, the training most often takes place using a high-performance GPU unit rather than the deployed end solution. This enables much faster training of the network than would be achieved if the deployed system was used alone.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The term neural network is very generic, and includes a significant number of distinct subcategories whose names are normally used to identify the exact type of network being implemented. Neural networks are modeled upon the human cerebral cortex, in that each neuron receives an input, processes it, and communicates the processed signal to another neuron. Therefore, neural networks typically consist of an input layer, internal layer(s), and an output layer.&lt;/p&gt;
&lt;div style="float:left;padding:12px 10px 5px 0px;width:310px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_diagram.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox"  src="/e14/assets/legacy/2021/AI3_diagram.png" /&gt;&lt;/a&gt;
&lt;p style="font-size:11px;line-height:12px;margin:0;padding-top:3px;"&gt;&lt;em&gt;Figure 1: A Simple Neural Network &lt;/em&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;There are several classes of neural networks, with different classes used for different applications (e.g. speech recognition or image classification). Neural networks which pass the output of one layer to another without forming a cycle are called Feedforward Neural Networks (FNNs). Those which contain directed cycles where there is feedback, for example the Elman network, are called Recurrent Neural Networks (RNNs). One very commonly used term in many machine learning applications is Deep Neural Networks (DNNs). These are neural networks which have several hidden layers enabling a more complex machine learning task to be implemented.&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.1 Network Types&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The structure of the neural network, the number of stages, and which types of elements are used has been, and indeed continues to be, the subject of considerable research to obtain the best performance. Common network structures have evolved. Convolutional Neural Networks (CNNs), for example, are a type of feedforward neural network. Long Short-Term Memory (LSTM) is a type of Recurrent Neural Network. CNNs are exceedingly popular for image classification, while LSTM is a popular network for Natural Language Processing (NLP).&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Commonly used CNN network structures include:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AlexNet:&lt;/strong&gt; Invented by Alex Krizhevsky as part of his PhD Studies. AlexNet is an 8-layer deep network which completed the ImageNet Large Scale Visual Recognition Challenge in 2012.&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;GoogLeNet:&lt;/strong&gt; A 22-layer CNN first described in 2014 by Google.&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;VGG-16:&lt;/strong&gt; A 16-layer CNN described by the Visual Geometry Group at the University of Oxford in 2014.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.2 Convolutional Neural Networks &lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;What separates embedded vision applications from other, simpler machine learning applications is that they have a two-dimensional input format. As such, in machine learning implementations, a Convolutional Neural Network is used, as it can process two-dimensional inputs. As outlined above, several different CNN implementations exist which have evolved over the years. However, they consist of multiple stages of the same basic functional blocks, with different parameterizations throughout the implementation. These stages are:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Convolution:&lt;/strong&gt; In this stage, the input image is broken down into several overlapping smaller tiles by the application of a filter. The results from this convolution are used to create a feature map.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Rectified Linear Unit (reLU):&lt;/strong&gt; This activation layer is used to create an activation map following a convolution.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Max Pooling:&lt;/strong&gt; Performs subsampling between layers to reduce resolution.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Fully Connected:&lt;/strong&gt; This is a matrix multiplication layer which applies both weights and a bias to the input. It is called a fully connected layer because the input and output are fully connected.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These networks and stages are defined and trained within an ML framework. The ML framework provides not only a range of libraries, but also pre-existing network models thanks to its Model Zoo, and pre-trained weights that are applied to the model to quickly get networks up and running. Training is required to define the parameters for the convolution (filter weights) and fully connected layers (weights and bias), while the pool and reLU elements require no parameterization. Both the pool and convolution layers, however, require parameterization to define the size of the filter kernel and the stride. How we determine the values which are used for these is called training. One of the advantages of using a CNN is the relative ease of training the network.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec4"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;4. Frameworks&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Of course, machine learning inference is a complex area, and starting from scratch could be daunting. However, several frameworks exist which enable us to get started creating our solution. Here are some of the most commonly used frameworks:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;TensorFlow:&lt;/strong&gt; Developed by the Google Brain Team and first released publicly in 2015, TensorFlow provides both Python and C APIs.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Caffe:&lt;/strong&gt; This convolutional architecture for fast-feature embedding was developed by the University of California at Berkley. Caffe supports both C and Python.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;PyTorch:&lt;/strong&gt; A Python implementation of torch primarily developed by the FaceBook AI research group.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;tinyML:&lt;/strong&gt; TensorFlow Lite, developed to enable ML implementations on Arm Cortex-M class devices, ESP32 devices, and other edge processors.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These frameworks provide everything required to instantiate a network, train the network, and deploy the network on the target device.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec5"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;5. Training&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The training of a network requires a large data set consisting of many thousands of correct and incorrect sources.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;There are three types of training routinely used for an ML network:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Supervised Training:&lt;/strong&gt; A tagged data set is used. The network iteratively makes output predictions based on the tagged input data. These output predictions are corrected by a supervisor based upon the tag data, meaning that when untagged data is input, the output prediction is correct within an accuracy. Supervised training is used for the majority of neural networks.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Unsupervised Training:&lt;/strong&gt; Untagged data sources are used. These are used for neural network implementations with no correct answer (e.g. clustering and association).&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Reinforcement Learning:&lt;/strong&gt; Uses the interaction with the network&amp;#39;s environment to maximize the notion of cumulative reward.&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding-top:12px;"&gt;When structured training is performed, an additional layer is introduced, called the loss function. This stage enables the training algorithm to know if the network correctly identified the input image. During training, both forward and backward propagation is used to determine if the image is correctly classified, and updates are made to the weights and biases based on the error signals and calculated gradients. To apply the image sets and calculate the network coefficients as quickly and as efficiently as possible, large farms of Graphical Processing Units (GPUs) are used. GPU farms are used because the goal of training is to generate the weights and biases within the minimal time frame. Therefore, power efficiency, real-time response, and determinism for each image frame is not as critical as it is within the final inference deployment. Therefore, GPU farms provide the most efficient mechanism for determining weights and biases.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec6"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;6. Implementing ML Inference &lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;While the training and creation of an ML solution will take place using GPU farms, the ML inference application will be deployed on many different edge technologies, such as the following examples.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Edge-Based Applications&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;CPU:&lt;/strong&gt; General-purpose CPUs, such as Arm-based microcontrollers like the Arm Cortex-M3 processor&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;GPU:&lt;/strong&gt; Graphics Processor Units, such as the Nvidia System-on-Module&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;FPGA:&lt;/strong&gt; Field Programmable Gate Array, such as Xilinx heterogeneous SoCs and MPSoCs like the Zynq-7000 or the &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii#mpsoc"&gt;Zynq UltraScale+ MPSoC&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Cloud-Based Applications&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;High Performance CPU:&lt;/strong&gt; x86 or Arm-based higher performance server processor&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Graphics Cards:&lt;/strong&gt; High-performance graphics co-processors&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Acceleration Card:&lt;/strong&gt; FPGA-based acceleration cards connected to the x86 server hardware using PCIe&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Each end implementation technology will have different deployment needs. This is where technology-specific tool chains are used for the deployment. Examples of this include the &lt;a class="jive-link-external-small" href="https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;Vitis AI tool&lt;/a&gt; from Xilinx or the &lt;a class="jive-link-external-small" href="https://www.tensorflow.org/lite/microcontrollers/build_convert#model_conversion" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;TensorFlow Lite converter&lt;/a&gt;.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;One large deciding factor in this situation is the capability of the underlying hardware. For example, many neural networks are defined and trained as floating-point implementations. However, at the edge, fixed-point representation may make for easier and more efficient implementations. Using fixed point may also enable the use of lower-cost hardware. Even in the cloud, FP32 implementation may still provide a faster, more powerful, and efficient implementation.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Machine learning applications are increasingly using more efficient, reduced precision, fixed-point number systems such as INT8 representation. The use of fixed-point reduced precision number systems comes without a significant loss in accuracy, when compared to a traditional floating point 32 (FP32) approach. However, fixed-point mathematics is considerably easier to implement than floating point. This move to INT8 provides for more efficient, faster solutions in some implementations. To support this generation of reduced precision number systems, the automated &lt;a class="jive-link-external-small" href="https://github.com/hotzeng/Ristretto-caffe" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;Ristretto tool for Caffe&lt;/a&gt; enables the training of networks using fixed-point number systems. This removes the need (and associated potential performance impact) of converting floating-point training data into fixed-point training data.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;If you are targeting a Xilinx programmable logic device, the Vitis AI tool also provides several tools which support implementation of models in programmable logic.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Optimizer:&lt;/strong&gt; Optimizes the floating-point model to reduce model complexity&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Quantizer:&lt;/strong&gt; Quantizes the floating-point model weights and biases into an INT8 representation&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Compiler:&lt;/strong&gt; Compiles the quantized model to be deployed on the Xilinx Deep Learning Processor Unit (DPU)&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Using Vitis AI, networks defined within Caffe, TensorFlow, and PyTorch can be deployed on programmable logic solutions. This allows AI developers to leverage the benefits which come with implementation of a solution in programmable logic.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec7"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;7. Typical Applications&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Edge-based ML applications are increasingly growing in popularity. Typical examples include use in both the industrial and automotive solutions spaces.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Within the automotive space, ML can be used for several different solutions. One implementation is in road sign recognition and classification, as part of an Advanced Driver Assistance program. Another application is internal cabin monitoring, where the behavior of the occupants is used to set up environmental conditions.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;When ML is deployed at the edge, one of the main aspects of its use for industrial applications can be predictive maintenance, where sensors and data are used to predict the failure or wear out of materials ending their lifecycle.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec8"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;8. Conclusion&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Embedded systems at the edge are increasingly deployed using machine learning algorithms to enable intelligent processing. Throughout this Essentials course, we have examined the history of AI and ML, neural networks, network types, frameworks, training, and deployment at the edge. Now, all that is left is for you to try creating your first ML application!&lt;/p&gt;
&lt;p style="font-size:11px;margin:0;padding-top:12px;"&gt;*Trademark. &lt;strong&gt;Xilinx is a trademark of Xilinx Inc.&lt;/strong&gt;&amp;nbsp; Other logos, product and/or company names may be trademarks of their respective owners.&lt;/p&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;border:1px solid #a9aaaa;padding:12px 12px 0px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_profile.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox essProfile" style="float:left;padding-right:10px;vertical-align:middle;"  height="115" src="/e14/assets/legacy/2021/AI3_profile.png" /&gt;&lt;/a&gt;
&lt;div&gt;
&lt;p style="color:#007fac;font-size:15px;margin:0;padding-bottom:10px;"&gt;Shop our wide range of &lt;strong&gt;components, development boards, and evaluation kits used for artificial intelligence and machine learning solutions.&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;span class="e14-button-large e14-button-primary"&gt;&lt;a class="jive-link-wiki-small" title="Shop Now" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii"&gt;Shop Now&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:20px;"&gt;&lt;a name="test"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="color:#007fac;font-size:18px;margin:0;padding-bottom:8px;"&gt;&lt;strong&gt;Test Your Knowledge&lt;/strong&gt;&lt;span style="float:right;font-size:11px;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a href="/e14/assets/legacy/2021/100x100_AI3Badge.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox" style="float:left;padding:0px 10px 5px 0px;vertical-align:top;"  height="80" src="/e14/assets/legacy/2021/100x100_AI3Badge.png" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-bottom:8px;"&gt;Are you ready to demonstrate your knowledge of Inference for Machine Learning? &lt;strong&gt;Then take this 15-question quiz to see how much you&amp;#39;ve learned.&lt;/strong&gt; To earn the AI III Badge, read through the learning module, score 100% on the Quiz, leave us some feedback in the comments section, and give the learning module a star rating.&lt;/p&gt;
&lt;p style="margin:0;padding:0px;"&gt;&lt;div class="voltquiz" data-formid="a5c10660-9227-47a4-8252-886bb231de44" data-formtype="quiz" data-success-message="" data-fail-message="" data-pass-mark="OTk=" &gt;&lt;/div&gt;&amp;nbsp;&lt;/p&gt;
&lt;div class="e14-quiz-document-metadata e14-init-hidden"&gt;&lt;br /&gt;&lt;br /&gt;&lt;/div&gt;
&lt;div class="migration-injected-attachments"&gt;
&lt;div style="font-weight:bold;margin:15px 0 5px 0;"&gt;Attachments:&lt;/div&gt;
&lt;div&gt;&lt;/div&gt;
&lt;table style="border:0;"&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td width="160"&gt;&lt;span id="attid_https://www.element14.com/community/api/core/v3/attachments/313552" class="_se_attachment"&gt;&lt;a href="https://community.element14.com/cfs-file/__key/communityserver-wikis-components-files/00-00-00-02-50/AI3.pdf"&gt;community.element14.com/.../AI3.pdf&lt;/a&gt;&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;span style="font-size:80%;"&gt;AI3.pdf&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;&lt;div style="clear:both;"&gt;&lt;/div&gt;

&lt;div style="font-size: 90%;"&gt;Tags: fpga_essentials, machine_learning, xilinx, inference_for_machine_learning, ai, artificial_intelligence, ess_module&lt;/div&gt;
</description></item><item><title>Artificial Intelligence III Inference for Machine Learning</title><link>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning/revision/4</link><pubDate>Fri, 18 Feb 2022 20:35:24 GMT</pubDate><guid isPermaLink="false">93d5dcb4-84c2-446f-b2cb-99731719e767:880c3939-2e7d-48c5-9b70-5674233690e7</guid><dc:creator>pchan</dc:creator><comments>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning#comments</comments><description>Revision 4 posted to Documents by pchan on 2/18/2022 8:35:24 PM&lt;br /&gt;
&lt;p&gt;&lt;span&gt;&lt;div class="e14-navigation-buttons"&gt;&lt;a href="/learn/learning-center/online-learning/essentials/w/documents/4964/element14-essentials-artificial-intelligence-ii?ICID=essentials-ai3-doc"&gt;
                    &lt;button class="e14-nav-button-prev"&gt;Previous&lt;/button&gt;
                &lt;/a&gt;&lt;/div&gt;&lt;script&gt;e14.func.queueScripts.add(function(){ e14.func.repositionNavButtons(); })&lt;/script&gt;&lt;/span&gt;&lt;/p&gt;
&lt;div style="background-color:#f9f9f9;border:1px solid #c6c6c6;display:block;float:right;padding:6px 8px;width:212px;"&gt;&lt;a class="jivecontainerTT-hover-container jive-link-community-small" href="/learn/learning-center/"&gt;element14 Learning Center&lt;/a&gt;&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="border:1px solid #c6c6c6;clear:both;padding:8px;"&gt;
&lt;div style="display:inline-block;padding-right:16px;vertical-align:top;"&gt;&lt;a href="/learn/learning-center/online-learning/essentials/"&gt;&lt;img alt="image"  src="/e14/assets/legacy/2017/learningess.png" width="175px" /&gt;&lt;/a&gt;&lt;/div&gt;
&lt;div style="display:inline-block;"&gt;
&lt;p class="essTitle" style="font-size:18px;font-weight:bold;margin:0;" title="Inference for Machine Learning - Artificial Intelligence"&gt;&lt;span style="color:#007fac;"&gt;Artificial Intelligence III: &lt;/span&gt; Inference for Machine Learning&lt;/p&gt;
&lt;p style="margin:0;padding:8px;"&gt;&lt;span style="font-size:11px;padding-right:10px;"&gt;&lt;em&gt;Sponsored by&lt;/em&gt;&lt;/span&gt;&lt;a href="https://www.xilinx.com/" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;&lt;img loading="lazy" alt="image" class="essSponsor" style="vertical-align:top;" title="Xilinx"  src="/e14/assets/main/mfg-group-assets/xilinxLogo.png" width="120px" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;hr /&gt;
&lt;div style="clear:both;line-height:1.5;"&gt;&lt;span style="padding-right:6px;"&gt;&lt;a class="jive-link-anchor-small" href="#intro"&gt;1. Introduction&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#object"&gt;2. Objectives&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec3"&gt;3. Neural Networks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec4"&gt;4. Frameworks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec5"&gt;5. Training&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec6"&gt;6. Implementing ML Inference &lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec7"&gt;7. Typical Applications&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec8"&gt;8. Conclusion&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii"&gt;Related Components&lt;/a&gt;&lt;/span&gt; | &lt;span style="font-weight:bold;padding:0px 6px;"&gt;&lt;a href="#test"&gt;Test Your Knowledge &lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/LinkArrow.gif" /&gt;&lt;/a&gt;&lt;/span&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="background-color:#e4f2f8;border:1px solid #b8d7e5;float:right;margin:20px 0px 5px 5px;padding:12px;width:155px;"&gt;
&lt;p style="color:#007fac;font-weight:bold;margin:0;"&gt;Also Available:&lt;/p&gt;
&lt;p style="margin:0;padding-top:5px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/4563/element14-essentials-artificial-intelligence"&gt;Artificial Intelligence&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:5px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/4964/element14-essentials-artificial-intelligence-ii"&gt;Artificial Intelligence II: AI at the Edge&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;p style="color:#007fac;font-size:15px;margin:0;padding:8px 0px;"&gt;&lt;a name="intro"&gt;&lt;/a&gt;&lt;strong&gt;1. Introduction&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;We have all seen films where humankind implements an artificial intelligence (AI) system, something goes wrong, and it turns against its creators. 2001: A Space Odyssey and the Terminator films are classic examples. But what is artificial intelligence, really, and how do we use it in our applications, especially at the edge?&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;When it comes to working with artificial intelligence, we really are standing on the shoulders of giants. The term AI itself was first used by John McCarthy in 1956, when the earliest conference on artificial intelligence was held. Even before the AI term had been coined, however, there was considerable research being undertaken on the question of whether or not machines think. One early leading researcher was Alan Turing, the Bletchley Park code breaker, who posed the Turing Test. The test revolves around the question of whether we will ever be able to build a computer that can sufficiently imitate a human, to the point where a suspicious judge can&amp;#39;t tell the difference between what is human and what is machine.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;AI research has been a hot topic over the years. The late 1990s saw IBM&amp;#39;s Deep Blue supercomputer beat chess grandmasters. More recently, we have seen significant interest in artificial intelligence and machine learning at the edge, deployed within embedded systems.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;But what is machine learning? According to the book Machine Learning by Tom M. Mitchell, Professor at Carnegie Mellon University, &amp;quot;Machine learning is the study of computer algorithms that allow computer programs to automatically improve through experience.&amp;quot;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;A simple example of this is providing a machine learning algorithm with a data set of pictures with labels. The machine learning (ML) algorithm will be able to identify features and elements of the pictures, so that when a new unlabeled picture is input, the ML algorithm is able to determine how likely the image is to contain any of the learned features and elements. The classic example of this is the machine learning algorithm being able to identify pictures of animals, fruits, and so on.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These machine learning algorithms can, therefore, be said to have two distinct elements: training and inference. Training is process in which the features and elements that need to be identified are learned, while inference is using the trained behavior to examine unlabeled inputs to identify the elements and features once deployed.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="object"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin:0;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;&lt;strong&gt;2. Objectives&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 8px 0px;"&gt;This Essentials learning module will introduce you to the basics of artificial intelligence (AI) and machine learning (ML) inference. &lt;em&gt;The learning objectives are as follows:&lt;/em&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Gain an understanding of inference and its role in AI and ML systems.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Learn about neural networks and what they consist of.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Become familiar with the different network types that can be used for AI inference.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; See examples of different methodologies and frameworks which can be used for creation, training, and inference of AI and ML.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec3"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;3. Neural Networks&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;At the heart of the machine learning inference implementation is the neural network. This neural network is instantiated and trained to provide the required behaviour. Training can be very compute intensive, as large data sets are used to train the network. As such, the training most often takes place using a high-performance GPU unit rather than the deployed end solution. This enables much faster training of the network than would be achieved if the deployed system was used alone.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The term neural network is very generic, and includes a significant number of distinct subcategories whose names are normally used to identify the exact type of network being implemented. Neural networks are modeled upon the human cerebral cortex, in that each neuron receives an input, processes it, and communicates the processed signal to another neuron. Therefore, neural networks typically consist of an input layer, internal layer(s), and an output layer.&lt;/p&gt;
&lt;div style="float:left;padding:12px 10px 5px 0px;width:310px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_diagram.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox"  src="/e14/assets/legacy/2021/AI3_diagram.png" /&gt;&lt;/a&gt;
&lt;p style="font-size:11px;line-height:12px;margin:0;padding-top:3px;"&gt;&lt;em&gt;Figure 1: A Simple Neural Network &lt;/em&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;There are several classes of neural networks, with different classes used for different applications (e.g. speech recognition or image classification). Neural networks which pass the output of one layer to another without forming a cycle are called Feedforward Neural Networks (FNNs). Those which contain directed cycles where there is feedback, for example the Elman network, are called Recurrent Neural Networks (RNNs). One very commonly used term in many machine learning applications is Deep Neural Networks (DNNs). These are neural networks which have several hidden layers enabling a more complex machine learning task to be implemented.&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.1 Network Types&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The structure of the neural network, the number of stages, and which types of elements are used has been, and indeed continues to be, the subject of considerable research to obtain the best performance. Common network structures have evolved. Convolutional Neural Networks (CNNs), for example, are a type of feedforward neural network. Long Short-Term Memory (LSTM) is a type of Recurrent Neural Network. CNNs are exceedingly popular for image classification, while LSTM is a popular network for Natural Language Processing (NLP).&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Commonly used CNN network structures include:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AlexNet:&lt;/strong&gt; Invented by Alex Krizhevsky as part of his PhD Studies. AlexNet is an 8-layer deep network which completed the ImageNet Large Scale Visual Recognition Challenge in 2012.&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;GoogLeNet:&lt;/strong&gt; A 22-layer CNN first described in 2014 by Google.&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;VGG-16:&lt;/strong&gt; A 16-layer CNN described by the Visual Geometry Group at the University of Oxford in 2014.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.2 Convolutional Neural Networks &lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;What separates embedded vision applications from other, simpler machine learning applications is that they have a two-dimensional input format. As such, in machine learning implementations, a Convolutional Neural Network is used, as it can process two-dimensional inputs. As outlined above, several different CNN implementations exist which have evolved over the years. However, they consist of multiple stages of the same basic functional blocks, with different parameterizations throughout the implementation. These stages are:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Convolution:&lt;/strong&gt; In this stage, the input image is broken down into several overlapping smaller tiles by the application of a filter. The results from this convolution are used to create a feature map.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Rectified Linear Unit (reLU):&lt;/strong&gt; This activation layer is used to create an activation map following a convolution.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Max Pooling:&lt;/strong&gt; Performs subsampling between layers to reduce resolution.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Fully Connected:&lt;/strong&gt; This is a matrix multiplication layer which applies both weights and a bias to the input. It is called a fully connected layer because the input and output are fully connected.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These networks and stages are defined and trained within an ML framework. The ML framework provides not only a range of libraries, but also pre-existing network models thanks to its Model Zoo, and pre-trained weights that are applied to the model to quickly get networks up and running. Training is required to define the parameters for the convolution (filter weights) and fully connected layers (weights and bias), while the pool and reLU elements require no parameterization. Both the pool and convolution layers, however, require parameterization to define the size of the filter kernel and the stride. How we determine the values which are used for these is called training. One of the advantages of using a CNN is the relative ease of training the network.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec4"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;4. Frameworks&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Of course, machine learning inference is a complex area, and starting from scratch could be daunting. However, several frameworks exist which enable us to get started creating our solution. Here are some of the most commonly used frameworks:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;TensorFlow:&lt;/strong&gt; Developed by the Google Brain Team and first released publicly in 2015, TensorFlow provides both Python and C APIs.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Caffe:&lt;/strong&gt; This convolutional architecture for fast-feature embedding was developed by the University of California at Berkley. Caffe supports both C and Python.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;PyTorch:&lt;/strong&gt; A Python implementation of torch primarily developed by the FaceBook AI research group.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;tinyML:&lt;/strong&gt; TensorFlow Lite, developed to enable ML implementations on Arm Cortex-M class devices, ESP32 devices, and other edge processors.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These frameworks provide everything required to instantiate a network, train the network, and deploy the network on the target device.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec5"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;5. Training&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The training of a network requires a large data set consisting of many thousands of correct and incorrect sources.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;There are three types of training routinely used for an ML network:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Supervised Training:&lt;/strong&gt; A tagged data set is used. The network iteratively makes output predictions based on the tagged input data. These output predictions are corrected by a supervisor based upon the tag data, meaning that when untagged data is input, the output prediction is correct within an accuracy. Supervised training is used for the majority of neural networks.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Unsupervised Training:&lt;/strong&gt; Untagged data sources are used. These are used for neural network implementations with no correct answer (e.g. clustering and association).&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Reinforcement Learning:&lt;/strong&gt; Uses the interaction with the network&amp;#39;s environment to maximize the notion of cumulative reward.&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding-top:12px;"&gt;When structured training is performed, an additional layer is introduced, called the loss function. This stage enables the training algorithm to know if the network correctly identified the input image. During training, both forward and backward propagation is used to determine if the image is correctly classified, and updates are made to the weights and biases based on the error signals and calculated gradients. To apply the image sets and calculate the network coefficients as quickly and as efficiently as possible, large farms of Graphical Processing Units (GPUs) are used. GPU farms are used because the goal of training is to generate the weights and biases within the minimal time frame. Therefore, power efficiency, real-time response, and determinism for each image frame is not as critical as it is within the final inference deployment. Therefore, GPU farms provide the most efficient mechanism for determining weights and biases.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec6"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;6. Implementing ML Inference &lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;While the training and creation of an ML solution will take place using GPU farms, the ML inference application will be deployed on many different edge technologies, such as the following examples.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Edge-Based Applications&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;CPU:&lt;/strong&gt; General-purpose CPUs, such as Arm-based microcontrollers like the Arm Cortex-M3 processor&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;GPU:&lt;/strong&gt; Graphics Processor Units, such as the Nvidia System-on-Module&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;FPGA:&lt;/strong&gt; Field Programmable Gate Array, such as Xilinx heterogeneous SoCs and MPSoCs like the Zynq-7000 or the &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii#mpsoc"&gt;Zynq UltraScale+ MPSoC&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Cloud-Based Applications&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;High Performance CPU:&lt;/strong&gt; x86 or Arm-based higher performance server processor&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Graphics Cards:&lt;/strong&gt; High-performance graphics co-processors&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Acceleration Card:&lt;/strong&gt; FPGA-based acceleration cards connected to the x86 server hardware using PCIe&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Each end implementation technology will have different deployment needs. This is where technology-specific tool chains are used for the deployment. Examples of this include the &lt;a class="jive-link-external-small" href="https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;Vitis AI tool&lt;/a&gt; from Xilinx or the &lt;a class="jive-link-external-small" href="https://www.tensorflow.org/lite/microcontrollers/build_convert#model_conversion" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;TensorFlow Lite converter&lt;/a&gt;.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;One large deciding factor in this situation is the capability of the underlying hardware. For example, many neural networks are defined and trained as floating-point implementations. However, at the edge, fixed-point representation may make for easier and more efficient implementations. Using fixed point may also enable the use of lower-cost hardware. Even in the cloud, FP32 implementation may still provide a faster, more powerful, and efficient implementation.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Machine learning applications are increasingly using more efficient, reduced precision, fixed-point number systems such as INT8 representation. The use of fixed-point reduced precision number systems comes without a significant loss in accuracy, when compared to a traditional floating point 32 (FP32) approach. However, fixed-point mathematics is considerably easier to implement than floating point. This move to INT8 provides for more efficient, faster solutions in some implementations. To support this generation of reduced precision number systems, the automated &lt;a class="jive-link-external-small" href="https://github.com/hotzeng/Ristretto-caffe" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;Ristretto tool for Caffe&lt;/a&gt; enables the training of networks using fixed-point number systems. This removes the need (and associated potential performance impact) of converting floating-point training data into fixed-point training data.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;If you are targeting a Xilinx programmable logic device, the Vitis AI tool also provides several tools which support implementation of models in programmable logic.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Optimizer:&lt;/strong&gt; Optimizes the floating-point model to reduce model complexity&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Quantizer:&lt;/strong&gt; Quantizes the floating-point model weights and biases into an INT8 representation&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Compiler:&lt;/strong&gt; Compiles the quantized model to be deployed on the Xilinx Deep Learning Processor Unit (DPU)&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Using Vitis AI, networks defined within Caffe, TensorFlow, and PyTorch can be deployed on programmable logic solutions. This allows AI developers to leverage the benefits which come with implementation of a solution in programmable logic.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec7"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;7. Typical Applications&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Edge-based ML applications are increasingly growing in popularity. Typical examples include use in both the industrial and automotive solutions spaces.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Within the automotive space, ML can be used for several different solutions. One implementation is in road sign recognition and classification, as part of an Advanced Driver Assistance program. Another application is internal cabin monitoring, where the behavior of the occupants is used to set up environmental conditions.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;When ML is deployed at the edge, one of the main aspects of its use for industrial applications can be predictive maintenance, where sensors and data are used to predict the failure or wear out of materials ending their lifecycle.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec8"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;8. Conclusion&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Embedded systems at the edge are increasingly deployed using machine learning algorithms to enable intelligent processing. Throughout this Essentials course, we have examined the history of AI and ML, neural networks, network types, frameworks, training, and deployment at the edge. Now, all that is left is for you to try creating your first ML application!&lt;/p&gt;
&lt;p style="font-size:11px;margin:0;padding-top:12px;"&gt;*Trademark. &lt;strong&gt;Xilinx is a trademark of Xilinx Inc.&lt;/strong&gt;&amp;nbsp; Other logos, product and/or company names may be trademarks of their respective owners.&lt;/p&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;border:1px solid #a9aaaa;padding:12px 12px 0px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_profile.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox essProfile" style="float:left;padding-right:10px;vertical-align:middle;"  height="115" src="/e14/assets/legacy/2021/AI3_profile.png" /&gt;&lt;/a&gt;
&lt;div&gt;
&lt;p style="color:#007fac;font-size:15px;margin:0;padding-bottom:10px;"&gt;Shop our wide range of &lt;strong&gt;components, development boards, and evaluation kits used for artificial intelligence and machine learning solutions.&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;span class="e14-button-large e14-button-primary"&gt;&lt;a class="jive-link-wiki-small" title="Shop Now" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii"&gt;Shop Now&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:20px;"&gt;&lt;a name="test"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="color:#007fac;font-size:18px;margin:0;padding-bottom:8px;"&gt;&lt;strong&gt;Test Your Knowledge&lt;/strong&gt;&lt;span style="float:right;font-size:11px;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a href="/e14/assets/legacy/2021/100x100_AI3Badge.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox" style="float:left;padding:0px 10px 5px 0px;vertical-align:top;"  height="80" src="/e14/assets/legacy/2021/100x100_AI3Badge.png" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-bottom:8px;"&gt;Are you ready to demonstrate your knowledge of Inference for Machine Learning? &lt;strong&gt;Then take this 15-question quiz to see how much you&amp;#39;ve learned.&lt;/strong&gt; To earn the AI III Badge, read through the learning module, score 100% on the Quiz, leave us some feedback in the comments section, and give the learning module a star rating.&lt;/p&gt;
&lt;p style="margin:0;padding:0px;"&gt;&lt;div class="voltquiz" data-formid="a5c10660-9227-47a4-8252-886bb231de44" data-formtype="quiz" data-success-message="" data-fail-message="" data-pass-mark="OTk=" &gt;&lt;/div&gt;&amp;nbsp;&lt;/p&gt;
&lt;div class="e14-quiz-document-metadata e14-init-hidden"&gt;&lt;br /&gt;&lt;br /&gt;&lt;/div&gt;
&lt;div class="migration-injected-attachments"&gt;
&lt;div style="font-weight:bold;margin:15px 0 5px 0;"&gt;Attachments:&lt;/div&gt;
&lt;div&gt;&lt;/div&gt;
&lt;table style="border:0;"&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td width="160"&gt;&lt;span id="attid_https://www.element14.com/community/api/core/v3/attachments/313552" class="_se_attachment"&gt;&lt;a href="https://community.element14.com/cfs-file/__key/communityserver-wikis-components-files/00-00-00-02-50/AI3.pdf"&gt;community.element14.com/.../AI3.pdf&lt;/a&gt;&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;span style="font-size:80%;"&gt;AI3.pdf&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;&lt;div style="clear:both;"&gt;&lt;/div&gt;

&lt;div style="font-size: 90%;"&gt;Tags: fpga_essentials, machine_learning, xilinx, inference_for_machine_learning, ai, artificial_intelligence, ess_module&lt;/div&gt;
</description></item><item><title>Artificial Intelligence III Inference for Machine Learning</title><link>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning/revision/3</link><pubDate>Sat, 18 Dec 2021 02:32:01 GMT</pubDate><guid isPermaLink="false">93d5dcb4-84c2-446f-b2cb-99731719e767:880c3939-2e7d-48c5-9b70-5674233690e7</guid><dc:creator>dkibbey</dc:creator><comments>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning#comments</comments><description>Revision 3 posted to Documents by dkibbey on 12/18/2021 2:32:01 AM&lt;br /&gt;
&lt;p&gt;&lt;span&gt;&lt;div class="e14-navigation-buttons"&gt;&lt;a href="/learn/learning-center/online-learning/essentials/w/documents/4964/element14-essentials-artificial-intelligence-ii?ICID=essentials-ai3-doc"&gt;
                    &lt;button class="e14-nav-button-prev"&gt;Previous&lt;/button&gt;
                &lt;/a&gt;&lt;/div&gt;&lt;script&gt;e14.func.queueScripts.add(function(){ e14.func.repositionNavButtons(); })&lt;/script&gt;&lt;/span&gt;&lt;/p&gt;
&lt;div style="background-color:#f9f9f9;border:1px solid #c6c6c6;display:block;float:right;padding:6px 8px;width:212px;"&gt;&lt;a class="jivecontainerTT-hover-container jive-link-community-small" href="/learn/learning-center/"&gt;element14 Learning Center&lt;/a&gt;&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="border:1px solid #c6c6c6;clear:both;padding:8px;"&gt;
&lt;div style="display:inline-block;padding-right:16px;vertical-align:top;"&gt;&lt;a href="/learn/learning-center/online-learning/essentials/"&gt;&lt;img alt="image"  src="/e14/assets/legacy/2017/learningess.png" width="175px" /&gt;&lt;/a&gt;&lt;/div&gt;
&lt;div style="display:inline-block;"&gt;
&lt;p class="essTitle" style="font-size:18px;font-weight:bold;margin:0;" title="Inference for Machine Learning - Artificial Intelligence"&gt;&lt;span style="color:#007fac;"&gt;Artificial Intelligence III: &lt;/span&gt; Inference for Machine Learning&lt;/p&gt;
&lt;p style="margin:0;padding:8px;"&gt;&lt;span style="font-size:11px;padding-right:10px;"&gt;&lt;em&gt;Sponsored by&lt;/em&gt;&lt;/span&gt;&lt;a href="https://www.xilinx.com/" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;&lt;img loading="lazy" alt="image" class="essSponsor" style="vertical-align:top;" title="Xilinx"  src="/e14/assets/legacy/logos/xilinxLogo.png" width="120px" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;hr /&gt;
&lt;div style="clear:both;line-height:1.5;"&gt;&lt;span style="padding-right:6px;"&gt;&lt;a class="jive-link-anchor-small" href="#intro"&gt;1. Introduction&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#object"&gt;2. Objectives&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec3"&gt;3. Neural Networks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec4"&gt;4. Frameworks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec5"&gt;5. Training&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec6"&gt;6. Implementing ML Inference &lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec7"&gt;7. Typical Applications&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec8"&gt;8. Conclusion&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii"&gt;Related Components&lt;/a&gt;&lt;/span&gt; | &lt;span style="font-weight:bold;padding:0px 6px;"&gt;&lt;a href="#test"&gt;Test Your Knowledge &lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/LinkArrow.gif" /&gt;&lt;/a&gt;&lt;/span&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="background-color:#e4f2f8;border:1px solid #b8d7e5;float:right;margin:20px 0px 5px 5px;padding:12px;width:155px;"&gt;
&lt;p style="color:#007fac;font-weight:bold;margin:0;"&gt;Also Available:&lt;/p&gt;
&lt;p style="margin:0;padding-top:5px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/4563/element14-essentials-artificial-intelligence"&gt;Artificial Intelligence&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:5px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/4964/element14-essentials-artificial-intelligence-ii"&gt;Artificial Intelligence II: AI at the Edge&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;p style="color:#007fac;font-size:15px;margin:0;padding:8px 0px;"&gt;&lt;a name="intro"&gt;&lt;/a&gt;&lt;strong&gt;1. Introduction&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;We have all seen films where humankind implements an artificial intelligence (AI) system, something goes wrong, and it turns against its creators. 2001: A Space Odyssey and the Terminator films are classic examples. But what is artificial intelligence, really, and how do we use it in our applications, especially at the edge?&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;When it comes to working with artificial intelligence, we really are standing on the shoulders of giants. The term AI itself was first used by John McCarthy in 1956, when the earliest conference on artificial intelligence was held. Even before the AI term had been coined, however, there was considerable research being undertaken on the question of whether or not machines think. One early leading researcher was Alan Turing, the Bletchley Park code breaker, who posed the Turing Test. The test revolves around the question of whether we will ever be able to build a computer that can sufficiently imitate a human, to the point where a suspicious judge can&amp;#39;t tell the difference between what is human and what is machine.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;AI research has been a hot topic over the years. The late 1990s saw IBM&amp;#39;s Deep Blue supercomputer beat chess grandmasters. More recently, we have seen significant interest in artificial intelligence and machine learning at the edge, deployed within embedded systems.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;But what is machine learning? According to the book Machine Learning by Tom M. Mitchell, Professor at Carnegie Mellon University, &amp;quot;Machine learning is the study of computer algorithms that allow computer programs to automatically improve through experience.&amp;quot;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;A simple example of this is providing a machine learning algorithm with a data set of pictures with labels. The machine learning (ML) algorithm will be able to identify features and elements of the pictures, so that when a new unlabeled picture is input, the ML algorithm is able to determine how likely the image is to contain any of the learned features and elements. The classic example of this is the machine learning algorithm being able to identify pictures of animals, fruits, and so on.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These machine learning algorithms can, therefore, be said to have two distinct elements: training and inference. Training is process in which the features and elements that need to be identified are learned, while inference is using the trained behavior to examine unlabeled inputs to identify the elements and features once deployed.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="object"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin:0;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;&lt;strong&gt;2. Objectives&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 8px 0px;"&gt;This Essentials learning module will introduce you to the basics of artificial intelligence (AI) and machine learning (ML) inference. &lt;em&gt;The learning objectives are as follows:&lt;/em&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Gain an understanding of inference and its role in AI and ML systems.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Learn about neural networks and what they consist of.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Become familiar with the different network types that can be used for AI inference.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; See examples of different methodologies and frameworks which can be used for creation, training, and inference of AI and ML.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec3"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;3. Neural Networks&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;At the heart of the machine learning inference implementation is the neural network. This neural network is instantiated and trained to provide the required behaviour. Training can be very compute intensive, as large data sets are used to train the network. As such, the training most often takes place using a high-performance GPU unit rather than the deployed end solution. This enables much faster training of the network than would be achieved if the deployed system was used alone.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The term neural network is very generic, and includes a significant number of distinct subcategories whose names are normally used to identify the exact type of network being implemented. Neural networks are modeled upon the human cerebral cortex, in that each neuron receives an input, processes it, and communicates the processed signal to another neuron. Therefore, neural networks typically consist of an input layer, internal layer(s), and an output layer.&lt;/p&gt;
&lt;div style="float:left;padding:12px 10px 5px 0px;width:310px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_diagram.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox"  src="/e14/assets/legacy/2021/AI3_diagram.png" /&gt;&lt;/a&gt;
&lt;p style="font-size:11px;line-height:12px;margin:0;padding-top:3px;"&gt;&lt;em&gt;Figure 1: A Simple Neural Network &lt;/em&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;There are several classes of neural networks, with different classes used for different applications (e.g. speech recognition or image classification). Neural networks which pass the output of one layer to another without forming a cycle are called Feedforward Neural Networks (FNNs). Those which contain directed cycles where there is feedback, for example the Elman network, are called Recurrent Neural Networks (RNNs). One very commonly used term in many machine learning applications is Deep Neural Networks (DNNs). These are neural networks which have several hidden layers enabling a more complex machine learning task to be implemented.&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.1 Network Types&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The structure of the neural network, the number of stages, and which types of elements are used has been, and indeed continues to be, the subject of considerable research to obtain the best performance. Common network structures have evolved. Convolutional Neural Networks (CNNs), for example, are a type of feedforward neural network. Long Short-Term Memory (LSTM) is a type of Recurrent Neural Network. CNNs are exceedingly popular for image classification, while LSTM is a popular network for Natural Language Processing (NLP).&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Commonly used CNN network structures include:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AlexNet:&lt;/strong&gt; Invented by Alex Krizhevsky as part of his PhD Studies. AlexNet is an 8-layer deep network which completed the ImageNet Large Scale Visual Recognition Challenge in 2012.&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;GoogLeNet:&lt;/strong&gt; A 22-layer CNN first described in 2014 by Google.&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;VGG-16:&lt;/strong&gt; A 16-layer CNN described by the Visual Geometry Group at the University of Oxford in 2014.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.2 Convolutional Neural Networks &lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;What separates embedded vision applications from other, simpler machine learning applications is that they have a two-dimensional input format. As such, in machine learning implementations, a Convolutional Neural Network is used, as it can process two-dimensional inputs. As outlined above, several different CNN implementations exist which have evolved over the years. However, they consist of multiple stages of the same basic functional blocks, with different parameterizations throughout the implementation. These stages are:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Convolution:&lt;/strong&gt; In this stage, the input image is broken down into several overlapping smaller tiles by the application of a filter. The results from this convolution are used to create a feature map.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Rectified Linear Unit (reLU):&lt;/strong&gt; This activation layer is used to create an activation map following a convolution.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Max Pooling:&lt;/strong&gt; Performs subsampling between layers to reduce resolution.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Fully Connected:&lt;/strong&gt; This is a matrix multiplication layer which applies both weights and a bias to the input. It is called a fully connected layer because the input and output are fully connected.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These networks and stages are defined and trained within an ML framework. The ML framework provides not only a range of libraries, but also pre-existing network models thanks to its Model Zoo, and pre-trained weights that are applied to the model to quickly get networks up and running. Training is required to define the parameters for the convolution (filter weights) and fully connected layers (weights and bias), while the pool and reLU elements require no parameterization. Both the pool and convolution layers, however, require parameterization to define the size of the filter kernel and the stride. How we determine the values which are used for these is called training. One of the advantages of using a CNN is the relative ease of training the network.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec4"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;4. Frameworks&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Of course, machine learning inference is a complex area, and starting from scratch could be daunting. However, several frameworks exist which enable us to get started creating our solution. Here are some of the most commonly used frameworks:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;TensorFlow:&lt;/strong&gt; Developed by the Google Brain Team and first released publicly in 2015, TensorFlow provides both Python and C APIs.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Caffe:&lt;/strong&gt; This convolutional architecture for fast-feature embedding was developed by the University of California at Berkley. Caffe supports both C and Python.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;PyTorch:&lt;/strong&gt; A Python implementation of torch primarily developed by the FaceBook AI research group.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;tinyML:&lt;/strong&gt; TensorFlow Lite, developed to enable ML implementations on Arm Cortex-M class devices, ESP32 devices, and other edge processors.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These frameworks provide everything required to instantiate a network, train the network, and deploy the network on the target device.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec5"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;5. Training&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The training of a network requires a large data set consisting of many thousands of correct and incorrect sources.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;There are three types of training routinely used for an ML network:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Supervised Training:&lt;/strong&gt; A tagged data set is used. The network iteratively makes output predictions based on the tagged input data. These output predictions are corrected by a supervisor based upon the tag data, meaning that when untagged data is input, the output prediction is correct within an accuracy. Supervised training is used for the majority of neural networks.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Unsupervised Training:&lt;/strong&gt; Untagged data sources are used. These are used for neural network implementations with no correct answer (e.g. clustering and association).&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Reinforcement Learning:&lt;/strong&gt; Uses the interaction with the network&amp;#39;s environment to maximize the notion of cumulative reward.&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding-top:12px;"&gt;When structured training is performed, an additional layer is introduced, called the loss function. This stage enables the training algorithm to know if the network correctly identified the input image. During training, both forward and backward propagation is used to determine if the image is correctly classified, and updates are made to the weights and biases based on the error signals and calculated gradients. To apply the image sets and calculate the network coefficients as quickly and as efficiently as possible, large farms of Graphical Processing Units (GPUs) are used. GPU farms are used because the goal of training is to generate the weights and biases within the minimal time frame. Therefore, power efficiency, real-time response, and determinism for each image frame is not as critical as it is within the final inference deployment. Therefore, GPU farms provide the most efficient mechanism for determining weights and biases.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec6"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;6. Implementing ML Inference &lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;While the training and creation of an ML solution will take place using GPU farms, the ML inference application will be deployed on many different edge technologies, such as the following examples.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Edge-Based Applications&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;CPU:&lt;/strong&gt; General-purpose CPUs, such as Arm-based microcontrollers like the Arm Cortex-M3 processor&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;GPU:&lt;/strong&gt; Graphics Processor Units, such as the Nvidia System-on-Module&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;FPGA:&lt;/strong&gt; Field Programmable Gate Array, such as Xilinx heterogeneous SoCs and MPSoCs like the Zynq-7000 or the &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii#mpsoc"&gt;Zynq UltraScale+ MPSoC&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Cloud-Based Applications&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;High Performance CPU:&lt;/strong&gt; x86 or Arm-based higher performance server processor&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Graphics Cards:&lt;/strong&gt; High-performance graphics co-processors&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Acceleration Card:&lt;/strong&gt; FPGA-based acceleration cards connected to the x86 server hardware using PCIe&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Each end implementation technology will have different deployment needs. This is where technology-specific tool chains are used for the deployment. Examples of this include the &lt;a class="jive-link-external-small" href="https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;Vitis AI tool&lt;/a&gt; from Xilinx or the &lt;a class="jive-link-external-small" href="https://www.tensorflow.org/lite/microcontrollers/build_convert#model_conversion" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;TensorFlow Lite converter&lt;/a&gt;.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;One large deciding factor in this situation is the capability of the underlying hardware. For example, many neural networks are defined and trained as floating-point implementations. However, at the edge, fixed-point representation may make for easier and more efficient implementations. Using fixed point may also enable the use of lower-cost hardware. Even in the cloud, FP32 implementation may still provide a faster, more powerful, and efficient implementation.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Machine learning applications are increasingly using more efficient, reduced precision, fixed-point number systems such as INT8 representation. The use of fixed-point reduced precision number systems comes without a significant loss in accuracy, when compared to a traditional floating point 32 (FP32) approach. However, fixed-point mathematics is considerably easier to implement than floating point. This move to INT8 provides for more efficient, faster solutions in some implementations. To support this generation of reduced precision number systems, the automated &lt;a class="jive-link-external-small" href="https://github.com/hotzeng/Ristretto-caffe" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;Ristretto tool for Caffe&lt;/a&gt; enables the training of networks using fixed-point number systems. This removes the need (and associated potential performance impact) of converting floating-point training data into fixed-point training data.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;If you are targeting a Xilinx programmable logic device, the Vitis AI tool also provides several tools which support implementation of models in programmable logic.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Optimizer:&lt;/strong&gt; Optimizes the floating-point model to reduce model complexity&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Quantizer:&lt;/strong&gt; Quantizes the floating-point model weights and biases into an INT8 representation&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Compiler:&lt;/strong&gt; Compiles the quantized model to be deployed on the Xilinx Deep Learning Processor Unit (DPU)&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Using Vitis AI, networks defined within Caffe, TensorFlow, and PyTorch can be deployed on programmable logic solutions. This allows AI developers to leverage the benefits which come with implementation of a solution in programmable logic.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec7"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;7. Typical Applications&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Edge-based ML applications are increasingly growing in popularity. Typical examples include use in both the industrial and automotive solutions spaces.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Within the automotive space, ML can be used for several different solutions. One implementation is in road sign recognition and classification, as part of an Advanced Driver Assistance program. Another application is internal cabin monitoring, where the behavior of the occupants is used to set up environmental conditions.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;When ML is deployed at the edge, one of the main aspects of its use for industrial applications can be predictive maintenance, where sensors and data are used to predict the failure or wear out of materials ending their lifecycle.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec8"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;8. Conclusion&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Embedded systems at the edge are increasingly deployed using machine learning algorithms to enable intelligent processing. Throughout this Essentials course, we have examined the history of AI and ML, neural networks, network types, frameworks, training, and deployment at the edge. Now, all that is left is for you to try creating your first ML application!&lt;/p&gt;
&lt;p style="font-size:11px;margin:0;padding-top:12px;"&gt;*Trademark. &lt;strong&gt;Xilinx is a trademark of Xilinx Inc.&lt;/strong&gt;&amp;nbsp; Other logos, product and/or company names may be trademarks of their respective owners.&lt;/p&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;border:1px solid #a9aaaa;padding:12px 12px 0px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_profile.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox essProfile" style="float:left;padding-right:10px;vertical-align:middle;"  height="115" src="/e14/assets/legacy/2021/AI3_profile.png" /&gt;&lt;/a&gt;
&lt;div&gt;
&lt;p style="color:#007fac;font-size:15px;margin:0;padding-bottom:10px;"&gt;Shop our wide range of &lt;strong&gt;components, development boards, and evaluation kits used for artificial intelligence and machine learning solutions.&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;span class="e14-button-large e14-button-primary"&gt;&lt;a class="jive-link-wiki-small" title="Shop Now" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii"&gt;Shop Now&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:20px;"&gt;&lt;a name="test"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="color:#007fac;font-size:18px;margin:0;padding-bottom:8px;"&gt;&lt;strong&gt;Test Your Knowledge&lt;/strong&gt;&lt;span style="float:right;font-size:11px;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a href="/e14/assets/legacy/2021/100x100_AI3Badge.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox" style="float:left;padding:0px 10px 5px 0px;vertical-align:top;"  height="80" src="/e14/assets/legacy/2021/100x100_AI3Badge.png" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-bottom:8px;"&gt;Are you ready to demonstrate your knowledge of Inference for Machine Learning? &lt;strong&gt;Then take this 15-question quiz to see how much you&amp;#39;ve learned.&lt;/strong&gt; To earn the AI III Badge, read through the learning module, score 100% on the Quiz, leave us some feedback in the comments section, and give the learning module a star rating.&lt;/p&gt;
&lt;p style="margin:0;padding:0px;"&gt;&lt;div class="voltquiz" data-formid="a5c10660-9227-47a4-8252-886bb231de44" data-formtype="quiz" data-success-message="" data-fail-message="" data-pass-mark="OTk=" &gt;&lt;/div&gt;&amp;nbsp;&lt;/p&gt;
&lt;div class="e14-quiz-document-metadata e14-init-hidden"&gt;&lt;br /&gt;&lt;br /&gt;&lt;/div&gt;
&lt;div class="migration-injected-attachments"&gt;
&lt;div style="font-weight:bold;margin:15px 0 5px 0;"&gt;Attachments:&lt;/div&gt;
&lt;div&gt;&lt;/div&gt;
&lt;table style="border:0;"&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td width="160"&gt;&lt;span id="attid_https://www.element14.com/community/api/core/v3/attachments/313552" class="_se_attachment"&gt;&lt;a href="https://community.element14.com/cfs-file/__key/communityserver-wikis-components-files/00-00-00-02-50/AI3.pdf"&gt;community.element14.com/.../AI3.pdf&lt;/a&gt;&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;span style="font-size:80%;"&gt;AI3.pdf&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;&lt;div style="clear:both;"&gt;&lt;/div&gt;

&lt;div style="font-size: 90%;"&gt;Tags: fpga_essentials, machine_learning, xilinx, inference_for_machine_learning, ai, artificial_intelligence, ess_module&lt;/div&gt;
</description></item><item><title>element14 Essentials: Artificial Intelligence III</title><link>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning/revision/2</link><pubDate>Mon, 08 Nov 2021 23:04:09 GMT</pubDate><guid isPermaLink="false">93d5dcb4-84c2-446f-b2cb-99731719e767:880c3939-2e7d-48c5-9b70-5674233690e7</guid><dc:creator>dychen</dc:creator><comments>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning#comments</comments><description>Revision 2 posted to Documents by dychen on 11/8/2021 11:04:09 PM&lt;br /&gt;
&lt;p&gt;&lt;span&gt;&lt;div class="e14-navigation-buttons"&gt;&lt;a href="/learn/learning-center/online-learning/essentials/w/documents/4964/element14-essentials-artificial-intelligence-ii?ICID=essentials-ai3-doc"&gt;
                    &lt;button class="e14-nav-button-prev"&gt;Previous&lt;/button&gt;
                &lt;/a&gt;&lt;/div&gt;&lt;script&gt;e14.func.queueScripts.add(function(){ e14.func.repositionNavButtons(); })&lt;/script&gt;&lt;/span&gt;&lt;/p&gt;
&lt;div style="background-color:#f9f9f9;border:1px solid #c6c6c6;display:block;float:right;padding:6px 8px;width:212px;"&gt;&lt;a class="jivecontainerTT-hover-container jive-link-community-small" href="/learn/learning-center/"&gt;element14 Learning Center&lt;/a&gt;&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="border:1px solid #c6c6c6;clear:both;padding:8px;"&gt;
&lt;div style="display:inline-block;padding-right:16px;vertical-align:top;"&gt;&lt;a href="/learn/learning-center/online-learning/essentials/"&gt;&lt;img alt="image"  src="/e14/assets/legacy/2017/learningess.png" width="175px" /&gt;&lt;/a&gt;&lt;/div&gt;
&lt;div style="display:inline-block;"&gt;
&lt;p class="essTitle" style="font-size:18px;font-weight:bold;margin:0;" title="Inference for Machine Learning - Artificial Intelligence"&gt;&lt;span style="color:#007fac;"&gt;Artificial Intelligence III: &lt;/span&gt; Inference for Machine Learning&lt;/p&gt;
&lt;p style="margin:0;padding:8px;"&gt;&lt;span style="font-size:11px;padding-right:10px;"&gt;&lt;em&gt;Sponsored by&lt;/em&gt;&lt;/span&gt;&lt;a href="https://www.xilinx.com/" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;&lt;img loading="lazy" alt="image" class="essSponsor" style="vertical-align:top;" title="Xilinx"  src="/e14/assets/legacy/logos/xilinxLogo.png" width="120px" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;hr /&gt;
&lt;div style="clear:both;line-height:1.5;"&gt;&lt;span style="padding-right:6px;"&gt;&lt;a class="jive-link-anchor-small" href="#intro"&gt;1. Introduction&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#object"&gt;2. Objectives&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec3"&gt;3. Neural Networks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec4"&gt;4. Frameworks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec5"&gt;5. Training&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec6"&gt;6. Implementing ML Inference &lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec7"&gt;7. Typical Applications&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec8"&gt;8. Conclusion&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii"&gt;Related Components&lt;/a&gt;&lt;/span&gt; | &lt;span style="font-weight:bold;padding:0px 6px;"&gt;&lt;a href="#test"&gt;Test Your Knowledge &lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/LinkArrow.gif" /&gt;&lt;/a&gt;&lt;/span&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="background-color:#e4f2f8;border:1px solid #b8d7e5;float:right;margin:20px 0px 5px 5px;padding:12px;width:155px;"&gt;
&lt;p style="color:#007fac;font-weight:bold;margin:0;"&gt;Also Available:&lt;/p&gt;
&lt;p style="margin:0;padding-top:5px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/4563/element14-essentials-artificial-intelligence"&gt;Artificial Intelligence&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:5px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/4964/element14-essentials-artificial-intelligence-ii"&gt;Artificial Intelligence II: AI at the Edge&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;p style="color:#007fac;font-size:15px;margin:0;padding:8px 0px;"&gt;&lt;a name="intro"&gt;&lt;/a&gt;&lt;strong&gt;1. Introduction&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;We have all seen films where humankind implements an artificial intelligence (AI) system, something goes wrong, and it turns against its creators. 2001: A Space Odyssey and the Terminator films are classic examples. But what is artificial intelligence, really, and how do we use it in our applications, especially at the edge?&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;When it comes to working with artificial intelligence, we really are standing on the shoulders of giants. The term AI itself was first used by John McCarthy in 1956, when the earliest conference on artificial intelligence was held. Even before the AI term had been coined, however, there was considerable research being undertaken on the question of whether or not machines think. One early leading researcher was Alan Turing, the Bletchley Park code breaker, who posed the Turing Test. The test revolves around the question of whether we will ever be able to build a computer that can sufficiently imitate a human, to the point where a suspicious judge can&amp;#39;t tell the difference between what is human and what is machine.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;AI research has been a hot topic over the years. The late 1990s saw IBM&amp;#39;s Deep Blue supercomputer beat chess grandmasters. More recently, we have seen significant interest in artificial intelligence and machine learning at the edge, deployed within embedded systems.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;But what is machine learning? According to the book Machine Learning by Tom M. Mitchell, Professor at Carnegie Mellon University, &amp;quot;Machine learning is the study of computer algorithms that allow computer programs to automatically improve through experience.&amp;quot;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;A simple example of this is providing a machine learning algorithm with a data set of pictures with labels. The machine learning (ML) algorithm will be able to identify features and elements of the pictures, so that when a new unlabeled picture is input, the ML algorithm is able to determine how likely the image is to contain any of the learned features and elements. The classic example of this is the machine learning algorithm being able to identify pictures of animals, fruits, and so on.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These machine learning algorithms can, therefore, be said to have two distinct elements: training and inference. Training is process in which the features and elements that need to be identified are learned, while inference is using the trained behavior to examine unlabeled inputs to identify the elements and features once deployed.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="object"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin:0;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;&lt;strong&gt;2. Objectives&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 8px 0px;"&gt;This Essentials learning module will introduce you to the basics of artificial intelligence (AI) and machine learning (ML) inference. &lt;em&gt;The learning objectives are as follows:&lt;/em&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Gain an understanding of inference and its role in AI and ML systems.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Learn about neural networks and what they consist of.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; Become familiar with the different network types that can be used for AI inference.&lt;/p&gt;
&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; See examples of different methodologies and frameworks which can be used for creation, training, and inference of AI and ML.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec3"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;3. Neural Networks&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;At the heart of the machine learning inference implementation is the neural network. This neural network is instantiated and trained to provide the required behaviour. Training can be very compute intensive, as large data sets are used to train the network. As such, the training most often takes place using a high-performance GPU unit rather than the deployed end solution. This enables much faster training of the network than would be achieved if the deployed system was used alone.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The term neural network is very generic, and includes a significant number of distinct subcategories whose names are normally used to identify the exact type of network being implemented. Neural networks are modeled upon the human cerebral cortex, in that each neuron receives an input, processes it, and communicates the processed signal to another neuron. Therefore, neural networks typically consist of an input layer, internal layer(s), and an output layer.&lt;/p&gt;
&lt;div style="float:left;padding:12px 10px 5px 0px;width:310px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_diagram.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox"  src="/e14/assets/legacy/2021/AI3_diagram.png" /&gt;&lt;/a&gt;
&lt;p style="font-size:11px;line-height:12px;margin:0;padding-top:3px;"&gt;&lt;em&gt;Figure 1: A Simple Neural Network &lt;/em&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;There are several classes of neural networks, with different classes used for different applications (e.g. speech recognition or image classification). Neural networks which pass the output of one layer to another without forming a cycle are called Feedforward Neural Networks (FNNs). Those which contain directed cycles where there is feedback, for example the Elman network, are called Recurrent Neural Networks (RNNs). One very commonly used term in many machine learning applications is Deep Neural Networks (DNNs). These are neural networks which have several hidden layers enabling a more complex machine learning task to be implemented.&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.1 Network Types&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The structure of the neural network, the number of stages, and which types of elements are used has been, and indeed continues to be, the subject of considerable research to obtain the best performance. Common network structures have evolved. Convolutional Neural Networks (CNNs), for example, are a type of feedforward neural network. Long Short-Term Memory (LSTM) is a type of Recurrent Neural Network. CNNs are exceedingly popular for image classification, while LSTM is a popular network for Natural Language Processing (NLP).&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Commonly used CNN network structures include:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AlexNet:&lt;/strong&gt; Invented by Alex Krizhevsky as part of his PhD Studies. AlexNet is an 8-layer deep network which completed the ImageNet Large Scale Visual Recognition Challenge in 2012.&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;GoogLeNet:&lt;/strong&gt; A 22-layer CNN first described in 2014 by Google.&lt;/p&gt;
&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;VGG-16:&lt;/strong&gt; A 16-layer CNN described by the Visual Geometry Group at the University of Oxford in 2014.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.2 Convolutional Neural Networks &lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;What separates embedded vision applications from other, simpler machine learning applications is that they have a two-dimensional input format. As such, in machine learning implementations, a Convolutional Neural Network is used, as it can process two-dimensional inputs. As outlined above, several different CNN implementations exist which have evolved over the years. However, they consist of multiple stages of the same basic functional blocks, with different parameterizations throughout the implementation. These stages are:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Convolution:&lt;/strong&gt; In this stage, the input image is broken down into several overlapping smaller tiles by the application of a filter. The results from this convolution are used to create a feature map.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Rectified Linear Unit (reLU):&lt;/strong&gt; This activation layer is used to create an activation map following a convolution.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Max Pooling:&lt;/strong&gt; Performs subsampling between layers to reduce resolution.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Fully Connected:&lt;/strong&gt; This is a matrix multiplication layer which applies both weights and a bias to the input. It is called a fully connected layer because the input and output are fully connected.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These networks and stages are defined and trained within an ML framework. The ML framework provides not only a range of libraries, but also pre-existing network models thanks to its Model Zoo, and pre-trained weights that are applied to the model to quickly get networks up and running. Training is required to define the parameters for the convolution (filter weights) and fully connected layers (weights and bias), while the pool and reLU elements require no parameterization. Both the pool and convolution layers, however, require parameterization to define the size of the filter kernel and the stride. How we determine the values which are used for these is called training. One of the advantages of using a CNN is the relative ease of training the network.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec4"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;4. Frameworks&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Of course, machine learning inference is a complex area, and starting from scratch could be daunting. However, several frameworks exist which enable us to get started creating our solution. Here are some of the most commonly used frameworks:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;TensorFlow:&lt;/strong&gt; Developed by the Google Brain Team and first released publicly in 2015, TensorFlow provides both Python and C APIs.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Caffe:&lt;/strong&gt; This convolutional architecture for fast-feature embedding was developed by the University of California at Berkley. Caffe supports both C and Python.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;PyTorch:&lt;/strong&gt; A Python implementation of torch primarily developed by the FaceBook AI research group.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;tinyML:&lt;/strong&gt; TensorFlow Lite, developed to enable ML implementations on Arm Cortex-M class devices, ESP32 devices, and other edge processors.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;These frameworks provide everything required to instantiate a network, train the network, and deploy the network on the target device.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec5"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;5. Training&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;The training of a network requires a large data set consisting of many thousands of correct and incorrect sources.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;There are three types of training routinely used for an ML network:&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Supervised Training:&lt;/strong&gt; A tagged data set is used. The network iteratively makes output predictions based on the tagged input data. These output predictions are corrected by a supervisor based upon the tag data, meaning that when untagged data is input, the output prediction is correct within an accuracy. Supervised training is used for the majority of neural networks.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Unsupervised Training:&lt;/strong&gt; Untagged data sources are used. These are used for neural network implementations with no correct answer (e.g. clustering and association).&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Reinforcement Learning:&lt;/strong&gt; Uses the interaction with the network&amp;#39;s environment to maximize the notion of cumulative reward.&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding-top:12px;"&gt;When structured training is performed, an additional layer is introduced, called the loss function. This stage enables the training algorithm to know if the network correctly identified the input image. During training, both forward and backward propagation is used to determine if the image is correctly classified, and updates are made to the weights and biases based on the error signals and calculated gradients. To apply the image sets and calculate the network coefficients as quickly and as efficiently as possible, large farms of Graphical Processing Units (GPUs) are used. GPU farms are used because the goal of training is to generate the weights and biases within the minimal time frame. Therefore, power efficiency, real-time response, and determinism for each image frame is not as critical as it is within the final inference deployment. Therefore, GPU farms provide the most efficient mechanism for determining weights and biases.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec6"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;6. Implementing ML Inference &lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;While the training and creation of an ML solution will take place using GPU farms, the ML inference application will be deployed on many different edge technologies, such as the following examples.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Edge-Based Applications&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;CPU:&lt;/strong&gt; General-purpose CPUs, such as Arm-based microcontrollers like the Arm Cortex-M3 processor&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;GPU:&lt;/strong&gt; Graphics Processor Units, such as the Nvidia System-on-Module&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;FPGA:&lt;/strong&gt; Field Programmable Gate Array, such as Xilinx heterogeneous SoCs and MPSoCs like the Zynq-7000 or the &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii#mpsoc"&gt;Zynq UltraScale+ MPSoC&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Cloud-Based Applications&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;High Performance CPU:&lt;/strong&gt; x86 or Arm-based higher performance server processor&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Graphics Cards:&lt;/strong&gt; High-performance graphics co-processors&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;Acceleration Card:&lt;/strong&gt; FPGA-based acceleration cards connected to the x86 server hardware using PCIe&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Each end implementation technology will have different deployment needs. This is where technology-specific tool chains are used for the deployment. Examples of this include the &lt;a class="jive-link-external-small" href="https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;Vitis AI tool&lt;/a&gt; from Xilinx or the &lt;a class="jive-link-external-small" href="https://www.tensorflow.org/lite/microcontrollers/build_convert#model_conversion" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;TensorFlow Lite converter&lt;/a&gt;.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;One large deciding factor in this situation is the capability of the underlying hardware. For example, many neural networks are defined and trained as floating-point implementations. However, at the edge, fixed-point representation may make for easier and more efficient implementations. Using fixed point may also enable the use of lower-cost hardware. Even in the cloud, FP32 implementation may still provide a faster, more powerful, and efficient implementation.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Machine learning applications are increasingly using more efficient, reduced precision, fixed-point number systems such as INT8 representation. The use of fixed-point reduced precision number systems comes without a significant loss in accuracy, when compared to a traditional floating point 32 (FP32) approach. However, fixed-point mathematics is considerably easier to implement than floating point. This move to INT8 provides for more efficient, faster solutions in some implementations. To support this generation of reduced precision number systems, the automated &lt;a class="jive-link-external-small" href="https://github.com/hotzeng/Ristretto-caffe" rel="nofollow ugc noopener noreferrer" target="_blank"&gt;Ristretto tool for Caffe&lt;/a&gt; enables the training of networks using fixed-point number systems. This removes the need (and associated potential performance impact) of converting floating-point training data into fixed-point training data.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;If you are targeting a Xilinx programmable logic device, the Vitis AI tool also provides several tools which support implementation of models in programmable logic.&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Optimizer:&lt;/strong&gt; Optimizes the floating-point model to reduce model complexity&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Quantizer:&lt;/strong&gt; Quantizes the floating-point model weights and biases into an INT8 representation&lt;/p&gt;
&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" style="vertical-align:middle;"  src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" /&gt;&lt;/a&gt; &lt;strong&gt;AI Compiler:&lt;/strong&gt; Compiles the quantized model to be deployed on the Xilinx Deep Learning Processor Unit (DPU)&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Using Vitis AI, networks defined within Caffe, TensorFlow, and PyTorch can be deployed on programmable logic solutions. This allows AI developers to leverage the benefits which come with implementation of a solution in programmable logic.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec7"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;7. Typical Applications&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Edge-based ML applications are increasingly growing in popularity. Typical examples include use in both the industrial and automotive solutions spaces.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Within the automotive space, ML can be used for several different solutions. One implementation is in road sign recognition and classification, as part of an Advanced Driver Assistance program. Another application is internal cabin monitoring, where the behavior of the occupants is used to set up environmental conditions.&lt;/p&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;When ML is deployed at the edge, one of the main aspects of its use for industrial applications can be predictive maintenance, where sensors and data are used to predict the failure or wear out of materials ending their lifecycle.&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a name="sec8"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;color:#007fac;font-size:15px;margin-top:16px;padding:4px 8px;vertical-align:middle;"&gt;
&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;8. Conclusion&lt;/strong&gt;&lt;/div&gt;
&lt;div style="display:inline-block;font-size:11px;text-align:right;vertical-align:top;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:12px;"&gt;Embedded systems at the edge are increasingly deployed using machine learning algorithms to enable intelligent processing. Throughout this Essentials course, we have examined the history of AI and ML, neural networks, network types, frameworks, training, and deployment at the edge. Now, all that is left is for you to try creating your first ML application!&lt;/p&gt;
&lt;p style="font-size:11px;margin:0;padding-top:12px;"&gt;*Trademark. &lt;strong&gt;Xilinx is a trademark of Xilinx Inc.&lt;/strong&gt;&amp;nbsp; Other logos, product and/or company names may be trademarks of their respective owners.&lt;/p&gt;
&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;div style="background-color:#e7f2f5;border:1px solid #a9aaaa;padding:12px 12px 0px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_profile.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox essProfile" style="float:left;padding-right:10px;vertical-align:middle;"  height="115" src="/e14/assets/legacy/2021/AI3_profile.png" /&gt;&lt;/a&gt;
&lt;div&gt;
&lt;p style="color:#007fac;font-size:15px;margin:0;padding-bottom:10px;"&gt;Shop our wide range of &lt;strong&gt;components, development boards, and evaluation kits used for artificial intelligence and machine learning solutions.&lt;/strong&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;span class="e14-button-large e14-button-primary"&gt;&lt;a class="jive-link-wiki-small" title="Shop Now" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii"&gt;Shop Now&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p style="clear:both;margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p style="margin:0;padding-top:20px;"&gt;&lt;a name="test"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="color:#007fac;font-size:18px;margin:0;padding-bottom:8px;"&gt;&lt;strong&gt;Test Your Knowledge&lt;/strong&gt;&lt;span style="float:right;font-size:11px;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p style="margin:0;"&gt;&lt;a href="/e14/assets/legacy/2021/100x100_AI3Badge.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox" style="float:left;padding:0px 10px 5px 0px;vertical-align:top;"  height="80" src="/e14/assets/legacy/2021/100x100_AI3Badge.png" /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p style="margin:0;padding-bottom:8px;"&gt;Are you ready to demonstrate your knowledge of Inference for Machine Learning? &lt;strong&gt;Then take this 15-question quiz to see how much you&amp;#39;ve learned.&lt;/strong&gt; To earn the AI III Badge, read through the learning module, score 100% on the Quiz, leave us some feedback in the comments section, and give the learning module a star rating.&lt;/p&gt;
&lt;p style="margin:0;padding:0px;"&gt;&lt;div class="voltquiz" data-formid="a5c10660-9227-47a4-8252-886bb231de44" data-formtype="quiz" data-success-message="" data-fail-message="" data-pass-mark="OTk=" &gt;&lt;/div&gt;&amp;nbsp;&lt;/p&gt;
&lt;div class="e14-quiz-document-metadata e14-init-hidden"&gt;&lt;br /&gt;&lt;br /&gt;&lt;/div&gt;
&lt;div class="migration-injected-attachments"&gt;
&lt;div style="font-weight:bold;margin:15px 0 5px 0;"&gt;Attachments:&lt;/div&gt;
&lt;div&gt;&lt;/div&gt;
&lt;table style="border:0;"&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td width="160"&gt;&lt;span id="attid_https://www.element14.com/community/api/core/v3/attachments/313552" class="_se_attachment"&gt;&lt;a href="https://community.element14.com/cfs-file/__key/communityserver-wikis-components-files/00-00-00-02-50/AI3.pdf"&gt;community.element14.com/.../AI3.pdf&lt;/a&gt;&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;span style="font-size:80%;"&gt;AI3.pdf&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;&lt;div style="clear:both;"&gt;&lt;/div&gt;

&lt;div style="font-size: 90%;"&gt;Tags: fpga_essentials, machine_learning, xilinx, inference_for_machine_learning, ai, artificial_intelligence, ess_module&lt;/div&gt;
</description></item><item><title>element14 Essentials: Artificial Intelligence III</title><link>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning/revision/1</link><pubDate>Tue, 25 May 2021 20:22:10 GMT</pubDate><guid isPermaLink="false">93d5dcb4-84c2-446f-b2cb-99731719e767:880c3939-2e7d-48c5-9b70-5674233690e7</guid><dc:creator>pchan</dc:creator><comments>https://community.element14.com/learn/learning-center/essentials/w/documents/5217/artificial-intelligence-inference-for-machine-learning#comments</comments><description>Revision 1 posted to Documents by pchan on 5/25/2021 8:22:10 PM&lt;br /&gt;
&lt;span&gt;&lt;div class="e14-navigation-buttons"&gt;&lt;a href="/learn/learning-center/online-learning/essentials/w/documents/4964/element14-essentials-artificial-intelligence-ii?ICID=essentials-ai3-doc"&gt;
                    &lt;button class="e14-nav-button-prev"&gt;Previous&lt;/button&gt;
                &lt;/a&gt;&lt;/div&gt;&lt;script&gt;e14.func.queueScripts.add(function(){ e14.func.repositionNavButtons(); })&lt;/script&gt;&lt;/span&gt;&lt;div style="padding:6px 8px;border:1px solid #c6c6c6;background-color:#f9f9f9;display:block;width:212px;float:right;"&gt;&lt;a class="jivecontainerTT-hover-container jive-link-community-small" href="/learn/learning-center/"&gt;element14 Learning Center&lt;/a&gt;&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div style="padding:8px;clear:both;border:1px solid #c6c6c6;"&gt;&lt;div style="display:inline-block;vertical-align:top;padding-right:16px;"&gt;&lt;a href="/learn/learning-center/online-learning/essentials/"&gt;&lt;img alt="image" src="/e14/assets/legacy/2017/learningess.png" width="175px"  /&gt;&lt;/a&gt;&lt;/div&gt;&lt;div style="display:inline-block;"&gt;&lt;p class="essTitle" style="margin:0;font-weight:bold;font-size:18px;" title="Inference for Machine Learning - Artificial Intelligence"&gt;&lt;span style="color:#007fac;"&gt;Artificial Intelligence III: &lt;/span&gt; Inference for Machine Learning&lt;/p&gt;&lt;p style="margin:0;padding:8px;"&gt;&lt;span style="font-size:11px;padding-right:10px;"&gt;&lt;em&gt;Sponsored by&lt;/em&gt;&lt;/span&gt;&lt;a href="https://www.xilinx.com/" rel="nofollow ugc noopener" target="_blank"&gt;&lt;img loading="lazy" alt="image" class="essSponsor" src="/e14/assets/legacy/logos/xilinxLogo.png" style="vertical-align:top;" title="Xilinx" width="120px"  /&gt;&lt;/a&gt;&lt;/p&gt;&lt;/div&gt;&lt;hr /&gt;&lt;div style="clear:both;line-height:1.5;"&gt;&lt;span style="padding-right:6px;"&gt;&lt;a class="jive-link-anchor-small" href="#intro"&gt;1. Introduction&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#object"&gt;2. Objectives&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec3"&gt;3. Neural Networks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec4"&gt;4. Frameworks&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec5"&gt;5. Training&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec6"&gt;6. Implementing ML Inference &lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec7"&gt;7. Typical Applications&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-anchor-small" href="#sec8"&gt;8. Conclusion&lt;/a&gt;&lt;/span&gt; | &lt;span style="padding:0px 6px;"&gt;&lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii"&gt;Related Components&lt;/a&gt;&lt;/span&gt; | &lt;span style="font-weight:bold;padding:0px 6px;"&gt;&lt;a href="#test"&gt;Test Your Knowledge &lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/LinkArrow.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div style="float:right;width:155px;border:1px solid #b8d7e5;padding:12px;margin:20px 0px 5px 5px;background-color:#e4f2f8;"&gt;&lt;p style="margin:0;color:#007fac;font-weight:bold;"&gt;Also Available:&lt;/p&gt;&lt;p style="margin:0;padding-top:5px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/4563/element14-essentials-artificial-intelligence"&gt;Artificial Intelligence&lt;/a&gt;&lt;/p&gt;&lt;p style="margin:0;padding-top:5px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/4964/element14-essentials-artificial-intelligence-ii"&gt;Artificial Intelligence II: AI at the Edge&lt;/a&gt;&lt;/p&gt;&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;p style="margin:0;color:#007fac;font-size:15px;padding:8px 0px;"&gt;&lt;a name="intro"&gt;&lt;/a&gt;&lt;strong&gt;1. Introduction&lt;/strong&gt;&lt;/p&gt;&lt;p style="margin:0;"&gt;We have all seen films where humankind implements an artificial intelligence (AI) system, something goes wrong, and it turns against its creators. 2001: A Space Odyssey and the Terminator films are classic examples. But what is artificial intelligence, really, and how do we use it in our applications, especially at the edge?&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;When it comes to working with artificial intelligence, we really are standing on the shoulders of giants. The term AI itself was first used by John McCarthy in 1956, when the earliest conference on artificial intelligence was held. Even before the AI term had been coined, however, there was considerable research being undertaken on the question of whether or not machines think. One early leading researcher was Alan Turing, the Bletchley Park code breaker, who posed the Turing Test. The test revolves around the question of whether we will ever be able to build a computer that can sufficiently imitate a human, to the point where a suspicious judge can&amp;#39;t tell the difference between what is human and what is machine.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;AI research has been a hot topic over the years. The late 1990s saw IBM&amp;#39;s Deep Blue supercomputer beat chess grandmasters. More recently, we have seen significant interest in artificial intelligence and machine learning at the edge, deployed within embedded systems.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;But what is machine learning? According to the book Machine Learning by Tom M. Mitchell, Professor at Carnegie Mellon University, &amp;quot;Machine learning is the study of computer algorithms that allow computer programs to automatically improve through experience.&amp;quot;&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;A simple example of this is providing a machine learning algorithm with a data set of pictures with labels. The machine learning (ML) algorithm will be able to identify features and elements of the pictures, so that when a new unlabeled picture is input, the ML algorithm is able to determine how likely the image is to contain any of the learned features and elements. The classic example of this is the machine learning algorithm being able to identify pictures of animals, fruits, and so on.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;These machine learning algorithms can, therefore, be said to have two distinct elements: training and inference. Training is process in which the features and elements that need to be identified are learned, while inference is using the trained behavior to examine unlabeled inputs to identify the elements and features once deployed.&lt;/p&gt;&lt;p style="margin:0;"&gt;&lt;a name="object"&gt;&lt;/a&gt;&lt;/p&gt;&lt;p style="margin:0;color:#007fac;font-size:15px;padding:4px 8px;margin-top:16px;background-color:#e7f2f5;vertical-align:middle;"&gt;&lt;strong&gt;2. Objectives&lt;/strong&gt;&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 8px 0px;"&gt;This Essentials learning module will introduce you to the basics of artificial intelligence (AI) and machine learning (ML) inference. &lt;em&gt;The learning objectives are as follows:&lt;/em&gt;&lt;/p&gt;&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; Gain an understanding of inference and its role in AI and ML systems.&lt;/p&gt;&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; Learn about neural networks and what they consist of.&lt;/p&gt;&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; Become familiar with the different network types that can be used for AI inference.&lt;/p&gt;&lt;p style="margin:0;padding:4px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; See examples of different methodologies and frameworks which can be used for creation, training, and inference of AI and ML.&lt;/p&gt;&lt;p style="margin:0;"&gt;&lt;a name="sec3"&gt;&lt;/a&gt;&lt;/p&gt;&lt;div style="color:#007fac;font-size:15px;padding:4px 8px;margin-top:16px;background-color:#e7f2f5;vertical-align:middle;"&gt;&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;3. Neural Networks&lt;/strong&gt;&lt;/div&gt;&lt;div style="display:inline-block;vertical-align:top;text-align:right;font-size:11px;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;&lt;/div&gt;&lt;p style="margin:0;padding-top:12px;"&gt;At the heart of the machine learning inference implementation is the neural network. This neural network is instantiated and trained to provide the required behaviour. Training can be very compute intensive, as large data sets are used to train the network. As such, the training most often takes place using a high-performance GPU unit rather than the deployed end solution. This enables much faster training of the network than would be achieved if the deployed system was used alone.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;The term neural network is very generic, and includes a significant number of distinct subcategories whose names are normally used to identify the exact type of network being implemented. Neural networks are modeled upon the human cerebral cortex, in that each neuron receives an input, processes it, and communicates the processed signal to another neuron. Therefore, neural networks typically consist of an input layer, internal layer(s), and an output layer.&lt;/p&gt;&lt;div style="float:left;width:310px;padding:12px 10px 5px 0px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_diagram.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox" src="/e14/assets/legacy/2021/AI3_diagram.png"  /&gt;&lt;/a&gt;&lt;p style="margin:0;line-height:12px;font-size:11px;padding-top:3px;"&gt;&lt;em&gt;Figure 1: A Simple Neural Network &lt;/em&gt;&lt;/p&gt;&lt;/div&gt;&lt;p style="margin:0;padding-top:12px;"&gt;There are several classes of neural networks, with different classes used for different applications (e.g. speech recognition or image classification). Neural networks which pass the output of one layer to another without forming a cycle are called Feedforward Neural Networks (FNNs). Those which contain directed cycles where there is feedback, for example the Elman network, are called Recurrent Neural Networks (RNNs). One very commonly used term in many machine learning applications is Deep Neural Networks (DNNs). These are neural networks which have several hidden layers enabling a more complex machine learning task to be implemented.&lt;/p&gt;&lt;p style="margin:0;clear:both;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.1 Network Types&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;The structure of the neural network, the number of stages, and which types of elements are used has been, and indeed continues to be, the subject of considerable research to obtain the best performance. Common network structures have evolved. Convolutional Neural Networks (CNNs), for example, are a type of feedforward neural network. Long Short-Term Memory (LSTM) is a type of Recurrent Neural Network. CNNs are exceedingly popular for image classification, while LSTM is a popular network for Natural Language Processing (NLP).&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;Commonly used CNN network structures include:&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;AlexNet:&lt;/strong&gt; Invented by Alex Krizhevsky as part of his PhD Studies. AlexNet is an 8-layer deep network which completed the ImageNet Large Scale Visual Recognition Challenge in 2012.&lt;/p&gt;&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;GoogLeNet:&lt;/strong&gt; A 22-layer CNN first described in 2014 by Google.&lt;/p&gt;&lt;p style="margin:0;padding:8px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;VGG-16:&lt;/strong&gt; A 16-layer CNN described by the Visual Geometry Group at the University of Oxford in 2014.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;&lt;em&gt;- 3.2 Convolutional Neural Networks &lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;What separates embedded vision applications from other, simpler machine learning applications is that they have a two-dimensional input format. As such, in machine learning implementations, a Convolutional Neural Network is used, as it can process two-dimensional inputs. As outlined above, several different CNN implementations exist which have evolved over the years. However, they consist of multiple stages of the same basic functional blocks, with different parameterizations throughout the implementation. These stages are:&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;Convolution:&lt;/strong&gt; In this stage, the input image is broken down into several overlapping smaller tiles by the application of a filter. The results from this convolution are used to create a feature map.&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;Rectified Linear Unit (reLU):&lt;/strong&gt; This activation layer is used to create an activation map following a convolution.&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;Max Pooling:&lt;/strong&gt; Performs subsampling between layers to reduce resolution.&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;Fully Connected:&lt;/strong&gt; This is a matrix multiplication layer which applies both weights and a bias to the input. It is called a fully connected layer because the input and output are fully connected.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;These networks and stages are defined and trained within an ML framework. The ML framework provides not only a range of libraries, but also pre-existing network models thanks to its Model Zoo, and pre-trained weights that are applied to the model to quickly get networks up and running. Training is required to define the parameters for the convolution (filter weights) and fully connected layers (weights and bias), while the pool and reLU elements require no parameterization. Both the pool and convolution layers, however, require parameterization to define the size of the filter kernel and the stride. How we determine the values which are used for these is called training. One of the advantages of using a CNN is the relative ease of training the network.&lt;/p&gt;&lt;p style="margin:0;"&gt;&lt;a name="sec4"&gt;&lt;/a&gt;&lt;/p&gt;&lt;div style="color:#007fac;font-size:15px;padding:4px 8px;margin-top:16px;background-color:#e7f2f5;vertical-align:middle;"&gt;&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;4. Frameworks&lt;/strong&gt;&lt;/div&gt;&lt;div style="display:inline-block;vertical-align:top;text-align:right;font-size:11px;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;&lt;/div&gt;&lt;p style="margin:0;padding-top:12px;"&gt;Of course, machine learning inference is a complex area, and starting from scratch could be daunting. However, several frameworks exist which enable us to get started creating our solution. Here are some of the most commonly used frameworks:&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;TensorFlow:&lt;/strong&gt; Developed by the Google Brain Team and first released publicly in 2015, TensorFlow provides both Python and C APIs.&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;Caffe:&lt;/strong&gt; This convolutional architecture for fast-feature embedding was developed by the University of California at Berkley. Caffe supports both C and Python.&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;PyTorch:&lt;/strong&gt; A Python implementation of torch primarily developed by the FaceBook AI research group.&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;tinyML:&lt;/strong&gt; TensorFlow Lite, developed to enable ML implementations on Arm Cortex-M class devices, ESP32 devices, and other edge processors.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;These frameworks provide everything required to instantiate a network, train the network, and deploy the network on the target device.&lt;/p&gt;&lt;p style="margin:0;"&gt;&lt;a name="sec5"&gt;&lt;/a&gt;&lt;/p&gt;&lt;div style="color:#007fac;font-size:15px;padding:4px 8px;margin-top:16px;background-color:#e7f2f5;vertical-align:middle;"&gt;&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;5. Training&lt;/strong&gt;&lt;/div&gt;&lt;div style="display:inline-block;vertical-align:top;text-align:right;font-size:11px;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;&lt;/div&gt;&lt;p style="margin:0;padding-top:12px;"&gt;The training of a network requires a large data set consisting of many thousands of correct and incorrect sources.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;There are three types of training routinely used for an ML network:&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;Supervised Training:&lt;/strong&gt; A tagged data set is used. The network iteratively makes output predictions based on the tagged input data. These output predictions are corrected by a supervisor based upon the tag data, meaning that when untagged data is input, the output prediction is correct within an accuracy. Supervised training is used for the majority of neural networks.&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;Unsupervised Training:&lt;/strong&gt; Untagged data sources are used. These are used for neural network implementations with no correct answer (e.g. clustering and association).&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;Reinforcement Learning:&lt;/strong&gt; Uses the interaction with the network&amp;#39;s environment to maximize the notion of cumulative reward.&lt;/p&gt;&lt;p style="margin:0;clear:both;padding-top:12px;"&gt;When structured training is performed, an additional layer is introduced, called the loss function. This stage enables the training algorithm to know if the network correctly identified the input image. During training, both forward and backward propagation is used to determine if the image is correctly classified, and updates are made to the weights and biases based on the error signals and calculated gradients. To apply the image sets and calculate the network coefficients as quickly and as efficiently as possible, large farms of Graphical Processing Units (GPUs) are used. GPU farms are used because the goal of training is to generate the weights and biases within the minimal time frame. Therefore, power efficiency, real-time response, and determinism for each image frame is not as critical as it is within the final inference deployment. Therefore, GPU farms provide the most efficient mechanism for determining weights and biases.&lt;/p&gt;&lt;p style="margin:0;"&gt;&lt;a name="sec6"&gt;&lt;/a&gt;&lt;/p&gt;&lt;div style="color:#007fac;font-size:15px;padding:4px 8px;margin-top:16px;background-color:#e7f2f5;vertical-align:middle;"&gt;&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;6. Implementing ML Inference &lt;/strong&gt;&lt;/div&gt;&lt;div style="display:inline-block;vertical-align:top;text-align:right;font-size:11px;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;&lt;/div&gt;&lt;p style="margin:0;padding-top:12px;"&gt;While the training and creation of an ML solution will take place using GPU farms, the ML inference application will be deployed on many different edge technologies, such as the following examples.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Edge-Based Applications&lt;/strong&gt;&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;CPU:&lt;/strong&gt; General-purpose CPUs, such as Arm-based microcontrollers like the Arm Cortex-M3 processor&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;GPU:&lt;/strong&gt; Graphics Processor Units, such as the Nvidia System-on-Module&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;FPGA:&lt;/strong&gt; Field Programmable Gate Array, such as Xilinx heterogeneous SoCs and MPSoCs like the Zynq-7000 or the &lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii#mpsoc"&gt;Zynq UltraScale+ MPSoC&lt;/a&gt;&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;&lt;strong&gt;Cloud-Based Applications&lt;/strong&gt;&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;High Performance CPU:&lt;/strong&gt; x86 or Arm-based higher performance server processor&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;Graphics Cards:&lt;/strong&gt; High-performance graphics co-processors&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;Acceleration Card:&lt;/strong&gt; FPGA-based acceleration cards connected to the x86 server hardware using PCIe&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;Each end implementation technology will have different deployment needs. This is where technology-specific tool chains are used for the deployment. Examples of this include the &lt;a class="jive-link-external-small" href="https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html" rel="nofollow ugc noopener" target="_blank"&gt;Vitis AI tool&lt;/a&gt; from Xilinx or the &lt;a class="jive-link-external-small" href="https://www.tensorflow.org/lite/microcontrollers/build_convert#model_conversion" rel="nofollow ugc noopener" target="_blank"&gt;TensorFlow Lite converter&lt;/a&gt;.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;One large deciding factor in this situation is the capability of the underlying hardware. For example, many neural networks are defined and trained as floating-point implementations. However, at the edge, fixed-point representation may make for easier and more efficient implementations. Using fixed point may also enable the use of lower-cost hardware. Even in the cloud, FP32 implementation may still provide a faster, more powerful, and efficient implementation.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;Machine learning applications are increasingly using more efficient, reduced precision, fixed-point number systems such as INT8 representation. The use of fixed-point reduced precision number systems comes without a significant loss in accuracy, when compared to a traditional floating point 32 (FP32) approach. However, fixed-point mathematics is considerably easier to implement than floating point. This move to INT8 provides for more efficient, faster solutions in some implementations. To support this generation of reduced precision number systems, the automated &lt;a class="jive-link-external-small" href="https://github.com/hotzeng/Ristretto-caffe" rel="nofollow ugc noopener" target="_blank"&gt;Ristretto tool for Caffe&lt;/a&gt; enables the training of networks using fixed-point number systems. This removes the need (and associated potential performance impact) of converting floating-point training data into fixed-point training data.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;If you are targeting a Xilinx programmable logic device, the Vitis AI tool also provides several tools which support implementation of models in programmable logic.&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;AI Optimizer:&lt;/strong&gt; Optimizes the floating-point model to reduce model complexity&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;AI Quantizer:&lt;/strong&gt; Quantizes the floating-point model weights and biases into an INT8 representation&lt;/p&gt;&lt;p style="margin:0;padding:12px 0px 0px 45px;"&gt;&lt;a href="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif"&gt;&lt;img loading="lazy" alt="image" src="/e14/assets/legacy/gen/small_square_bullet_oj5x5.gif" style="vertical-align:middle;"  /&gt;&lt;/a&gt; &lt;strong&gt;AI Compiler:&lt;/strong&gt; Compiles the quantized model to be deployed on the Xilinx Deep Learning Processor Unit (DPU)&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;Using Vitis AI, networks defined within Caffe, TensorFlow, and PyTorch can be deployed on programmable logic solutions. This allows AI developers to leverage the benefits which come with implementation of a solution in programmable logic.&lt;/p&gt;&lt;p style="margin:0;"&gt;&lt;a name="sec7"&gt;&lt;/a&gt;&lt;/p&gt;&lt;div style="color:#007fac;font-size:15px;padding:4px 8px;margin-top:16px;background-color:#e7f2f5;vertical-align:middle;"&gt;&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;7. Typical Applications&lt;/strong&gt;&lt;/div&gt;&lt;div style="display:inline-block;vertical-align:top;text-align:right;font-size:11px;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;&lt;/div&gt;&lt;p style="margin:0;padding-top:12px;"&gt;Edge-based ML applications are increasingly growing in popularity. Typical examples include use in both the industrial and automotive solutions spaces.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;Within the automotive space, ML can be used for several different solutions. One implementation is in road sign recognition and classification, as part of an Advanced Driver Assistance program. Another application is internal cabin monitoring, where the behavior of the occupants is used to set up environmental conditions.&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;"&gt;When ML is deployed at the edge, one of the main aspects of its use for industrial applications can be predictive maintenance, where sensors and data are used to predict the failure or wear out of materials ending their lifecycle.&lt;/p&gt;&lt;p style="margin:0;"&gt;&lt;a name="sec8"&gt;&lt;/a&gt;&lt;/p&gt;&lt;div style="color:#007fac;font-size:15px;padding:4px 8px;margin-top:16px;background-color:#e7f2f5;vertical-align:middle;"&gt;&lt;div style="display:inline-block;vertical-align:top;width:75%;"&gt;&lt;strong&gt;8. Conclusion&lt;/strong&gt;&lt;/div&gt;&lt;div style="display:inline-block;vertical-align:top;text-align:right;font-size:11px;width:24%;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/div&gt;&lt;/div&gt;&lt;p style="margin:0;padding-top:12px;"&gt;Embedded systems at the edge are increasingly deployed using machine learning algorithms to enable intelligent processing. Throughout this Essentials course, we have examined the history of AI and ML, neural networks, network types, frameworks, training, and deployment at the edge. Now, all that is left is for you to try creating your first ML application!&lt;/p&gt;&lt;p style="margin:0;padding-top:12px;font-size:11px;"&gt;*Trademark. &lt;strong&gt;Xilinx is a trademark of Xilinx Inc.&lt;/strong&gt;&amp;nbsp; Other logos, product and/or company names may be trademarks of their respective owners.&lt;/p&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div style="border:1px solid #a9aaaa;background-color:#e7f2f5;padding:12px 12px 0px;"&gt;&lt;a href="/e14/assets/legacy/2021/AI3_profile.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox essProfile" height="115" src="/e14/assets/legacy/2021/AI3_profile.png" style="float:left;padding-right:10px;vertical-align:middle;"  /&gt;&lt;/a&gt;&lt;div&gt;&lt;p style="margin:0;font-size:15px;color:#007fac;padding-bottom:10px;"&gt;Shop our wide range of &lt;strong&gt;components, development boards, and evaluation kits used for artificial intelligence and machine learning solutions.&lt;/strong&gt;&lt;/p&gt;&lt;p style="margin:0;"&gt;&lt;span class="e14-button-large e14-button-primary"&gt;&lt;a class="jive-link-wiki-small" href="/learn/learning-center/online-learning/essentials/w/documents/5218/related-components-for-element14-essentials-artificial-intelligence-iii" title="Shop Now"&gt;Shop Now&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;&lt;p style="margin:0;padding:0px;clear:both;"&gt;&amp;nbsp;&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;p style="margin:0;padding-top:20px;"&gt;&lt;a name="test"&gt;&lt;/a&gt;&lt;/p&gt;&lt;p style="margin:0;color:#007fac;font-size:18px;padding-bottom:8px;"&gt;&lt;strong&gt;Test Your Knowledge&lt;/strong&gt;&lt;span style="font-size:11px;float:right;"&gt;&lt;a class="jive-link-anchor-small" href="#top"&gt;&lt;strong&gt;Back to Top&lt;/strong&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;&lt;p style="margin:0;"&gt;&lt;a href="/e14/assets/legacy/2021/100x100_AI3Badge.png"&gt;&lt;img loading="lazy" alt="image" class="nolightbox" height="80" src="/e14/assets/legacy/2021/100x100_AI3Badge.png" style="vertical-align:top;float:left;padding:0px 10px 5px 0px;"  /&gt;&lt;/a&gt;&lt;/p&gt;&lt;p style="margin:0;padding-bottom:8px;"&gt;Are you ready to demonstrate your knowledge of Inference for Machine Learning? &lt;strong&gt;Then take this 15-question quiz to see how much you&amp;#39;ve learned.&lt;/strong&gt; To earn the AI III Badge, read through the learning module, score 100% on the Quiz, leave us some feedback in the comments section, and give the learning module a star rating.&lt;/p&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-document-metadata e14-init-hidden"&gt; &lt;/div&gt;&lt;div class="e14-quiz-question"&gt;      [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;      [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;      [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;      [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;      [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;      [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;      [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;      [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;     [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;      [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;     [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;     [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;      [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;     [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;&lt;p style="margin:0;padding:0px;"&gt;&amp;nbsp;&lt;/p&gt;&lt;div class="e14-quiz-question"&gt;      [QUIZ QUESTION PLACEHOLDER]&lt;/div&gt;
&lt;div class="migration-injected-attachments"&gt;&lt;div style="font-weight:bold;margin:15px 0 5px 0;"&gt;Attachments:&lt;/div&gt;&lt;div&gt;&lt;/div&gt;&lt;table style="border:0;"&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td width="160"&gt;&lt;span class="_se_attachment" id="attid_https://www.element14.com/community/api/core/v3/attachments/313552"&gt;&lt;a href="https://community.element14.com/cfs-file/__key/communityserver-wikis-components-files/00-00-00-02-50/AI3.pdf"&gt;community.element14.com/.../AI3.pdf&lt;/a&gt;&lt;/span&gt;&lt;/td&gt;&lt;td&gt;&lt;span style="font-size:80%;"&gt;AI3.pdf&lt;/span&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;div style="clear:both;"&gt;&lt;/div&gt;

&lt;div style="font-size: 90%;"&gt;Tags: fpga_essentials, machine_learning, xilinx, inference_for_machine_learning, ai, artificial_intelligence, ess_module&lt;/div&gt;
</description></item></channel></rss>