Layers

Artificial Neural Networks are made up of layers:

Input Layer

The input layer of an ANN is where data is fed into the ANN. The input layer is one neuron or node thick (there is only ever one input layer per ANN), however can contain an infinite number of input nodes on the vertical plane (or within it). For example, if we tasked an ANN to learn from website user statistics we may have about 50 data points or fields to learn from, and each of these would make up an individual input. On the other hand we may have an ANN that is learning how to recognise facial expressions from images and each input might be a single pixel, or group of pixels and therefore can be many thousands in number. The input layer only ever talks to downstream layers, there is no cross talk within the input layer. In fact, there is no actual processing in the Input layer at all, it is simply an input interface to the rest of the ANN, but is key in dictating the input size and ultimately the structure of our Artificial Neural Network.

Hidden Layer

The hidden layer of an Artificial Neural Network is where the majority of the processing and learning happens. It is composed of layers of Artificial Neurons. Each layer can vary in the number of individual neurons, and all upstream hidden layer neurons connect with all neurons in the next hidden layer. Weighted (random to start with) numerical input data/values are received from the input layer and fed into each first hidden layer neuron where a process called activation takes place. Activation is a threshold calculation that decides if the combined inputs to a particular hidden layer neuron are sufficient to trigger an output. The activation threshold is the same in every neuron in the network. This process of summation and activation continues left to right through the hidden layer meaning that each neuron contains a different output value, which is important for learning.

Output Layer

Immediately after the last hidden layer is the output layer, which unsurprisingly is where the ANN outputs its final result(s). Like the input layer, the output layer is only one neuron/node thick but can contain multiple neuron outputs within that layer. For example if you had some known outputs of "Bad customer" or "Good customer" the output layer would comprise of 2 output neurons. Unlike the input layer, activation occurs within the output layer before churning out the final result.

ANN learning

Forward Propagation

Described above is the structure of an Artificial Neural Network and how the flow of data goes left to right, input to hidden to output layer using weighted summation and activation at each neuron, ultimately leading to an output value. Remember, the goal of the ANN is to achieve an output based on an input - it has to learn how to get the known output target value from the provided input(s). When the ANN goes left to right for the first time, weights are random, and the calculated output is understandably nowhere near the target output (know as loss). This first attempt or first pass is called Forward Propagation. What needs to happen now is the ANN has to adjust to get closer to the target output value. It does this by adjusting all the weights in the ANN (which started off randomly for this very reason). This is called Back Propagation.

Back Propagation

Back Propagation is the process of an ANN systematically going backwards (output to hidden to input layer) over the ANN to adjust each weight within the system to achieve a single goal of being able for the next round of Forward Propagation to get closer to the target output. The maths behind this is fairly complex and out of the scope of this overview however it involves working out the derivative with respect to activation of all neurons or nodes in the ANN. This forward and back propagation happens many thousands of times (each cycle is called an epoch) until the ANN achieves the expected target output.

Activation

Activation occurs before the output of each hidden layer and output layer neuron and as stated above it works based on a threshold value. E.g. if a certain value is higher than X, then give an output of Y. The calculation or formula used for activation is consistent throughout the ANN however many different ones can be used. The type of activation formulae used is one of the key ways to optimise ANN learning efficiency.

Weights and biases

Weights were mentioned a few times above. Weights are values that reside between the output from each neuron in the ANN and modify the input going into each hidden and output layer neuron by simply multiplying output by weight. It is the weights that change as the ANN learns and are fundamentally how AI and Machine Learning is achieved. When AI/ML training has finished, it is the weights that are stored and used. The weights are literally the manifestation of the learning process of an ANN and how the output is achieved from set of inputs.

Biases are also commonly used in ANN and if in place act as a positive reinforcer on all neurons in all layers - ensuring there is always an input value to start with. Biases remain constant in value however they have weights that change during the learning process. Biases have been shown to positively influence the learning process.

Size and shape of the Artificial Neural Network, and optimisation

Artificial Neural Networks have a single job to do, which they do exceptionally - and that is to learn how to get from a set of inputs to a set of outputs. However, of course, the data they handle can vary massively in size and complexity, and this ultimately must influence the size and shape of our ANN.

The input layer of an ANN is directly related to the number of input fields, and the same is true of the output layer, so these are easy to define in terms of size. However, the hidden layer's composition - which can contain as many layers and as many neurons within each layer as we want - needs a bit of thought.

It is a safe bet to assume that the more complex the input data and/or learning task, the greater the requirement for a more robust hidden layer section e.g. if you are carrying out image processing with 10,000 inputs, clearly a hidden layer 2 thick with 3 neurons in each layer is not going to be sufficient.

The honest answer is, you simply do not know when you first start with a new ANN the optimal make-up of the hidden layer(s) is. You therefore have to go through a period of trial and error, optimisation and scaling up.

Overall number of Artificial Neuron connections

The total number of neuron to neuron connections has an impact on processing power requirements, and in an Artificial Neural Network it is calculated by multiplying the number of neurons in each layer by each other. E.g. if you have an input layer of 10 neurons, 3 hidden layers of 10 neurons each, and an output layer of 2, you would have 10 x 10 x 10 x 10 x 2 = 20,000 neuron connections. For image recognition systems you can easily end up with millions of artificial neuronal connections.

Fennaio AI and ML software and systems contain ANN automatic optimisation helpers as standard.

Artificial Neural Network Software

Fennaio designs, develops and provides Artificial Neural Network software than can be used for all Machine Learning and Artificial Intelligence systems.

We have available off-the-shelf turnkey Artificial Neural Network/Machine Learning/Deep Learning AI software and can create bespoke software and systems solutions where required.

Please get in touch with us to discuss your Artificial Neural Network requirements.