Neural Networks: Understanding Using Visual Basic

Introduction:

Neural Networks as a subject was the most difficult one to learn when I started taking interest in AI. Although, Internet was full of NN tutorials but they all seemed cryptic and too much mathematical. Anything which was available was hard to digest for a beginner in this field.
I swept through tons of code and tutorials just to understand what the hell ,much hyped, NNs were. Then, one fine day, Eureka!
I finally understood them and their practical applications. I immediately decided to write code in Visual Basic(my fav lang). After an hour of writing code and few more for tweaking it, I finally produced something which I had only dreamt of. It was one of the the best moments of my life. Then I finally uploaded .........history is very boring.
The real thing: There isn't a single NN tutorial available for Visual Basic Programmers. That is why I decided to write this tutorial.

Introduction to Neural Networks:

Neural Network or more appropriately Artificial Neural Network is basically a mathematical model of what goes in our mind(or brain). The brain of all the advanced living creatures consists of neurons, a basic cell, which when interconnected produces what we call Neural Network. The sole purpose of a Neuron is to receive electrical signals, accumulate them and see further if they are strong enough to pass forward.
So simple in its basic functionality but the interconnections of these produces beings(me, u and others) capable of writing about them. phew! The real thing lies not in neurons but the complex pattern in which they are interconnected. NNs are just like a game of chess, easy to learn but hard to master. As the moves of chess are simple, yet the succession of moves is what makes the game complex and fun to play. Imagine, a chess game in which you are allowed only one single move. Would that game be fun to play?
In the same way, a single neuron is useless. Well, practically useless. It is the complex connection between them and values attached with them(explained later) which makes brains capable of thinking and having a sense of consciousness(much debated).
Basic Working:

As explained earlier, a neuron is basically a cell which accumulates electrical signals with different strengths. What it does more is that it compares the accumulated signal with one predefined value unique to every neuron. This value is called bias. Well, now I think is time to explain with an image. So here is it:


Image Source: http://www.interwet.psu.edu/f41.gif

The circles in the image represents neurons. This network or more appropriately this network topology is called feed-forward multi layered neural network. It is the most basic and most widely used network. I will be explaining this network only.
The network is called multi layered because it consists of more than two layers. The neurons are arranged in a number of layers, generally three. They are input, hidden/middle and output layers. The names signify the function of the layer.
This network is feed-forward, means the values are propagated in one direction only. There are many other topologies in which values can be looped or move in both forward and backward direction. But, this network allows the movement of values only from input layer to output layer. The functions of various layers are explained below:

• Input layer: As it says, this layer takes the inputs(the values you pass) and forwards it to hidden layer. You can just imagine input layer as a group of neurons whose sole task is to pass the numeric inputs to the next level. Well it depends on the implementation but, in my implementation the numeric input has to be in range 0 and 1 (both inclusive). The larger the number greater its strength. E.g. 0.51 is stronger than 0.39 but 0.93412 is stronger still. But, the interpretation of this strength depends upon the implementation and the problem NN is required to solve. E.g. for an OCR you connect every pixel with its respective input neuron and darker the pixel, higher the signal/input strength. In short shades of gray would correspond to input strength. But at last again, remember: Input layer never processes data, it just hands over it.
• Middle layer: This layer is the real thing behind the network. Without this layer, network would not be capable of solving complex problems. There can be any number or middle or hidden layers. But, for most of the tasks, one is sufficient. The number of neurons in this layer is crucial. There is no formula for calculating the number, just hit and trial works.
  This layer takes the input from input layer, does some calculations and forwards to the next layer, in most cases it is the output layer.
• Output layer: This layer consists of neurons which output the result to you. This layer takes the value from the previous layer, does calculations and gives the final result. Basically, this layer is just like hidden layer but instead of passing values to the next layer, the values are treated as output.
• Dendrites: No! it is not some creature from X-Files, but a name given to straight lines joining two neurons of consecutive layers, which you can see in the image. They are just a passage(or method) through which values are passed from one layer to the next. There is a value attached with dendrite called weight. The weight associated with a dendrites basically determines the importance of incoming value. A weight with larger value determines that the value from that particular neuron is of higher significance. To achieve this what we do is multiply the incoming value with weight. So no matter how high the value is, if the weight is low the multiplication yields the final low value. I know it is hard to explain and understand, but take a look at code and you'll understand this concept more clearly.

So now lets get to the code and define the basic elements we'll need in NNs

'Don't forget to write option base 1 into the code
' or else this net will not work
'Coded by Paras Chopra
'paras_chopra@fastmail.fm
'http://naramcheez.netfirms.com

'Please don't forget to give comments, credits and most important your VOTE!


Option Base 1
Option Explicit

Const e = 2.7183 'Mathematical const, used in sigmod function

Private Type Dendrite ' Dendrite connects one neuron to another and allows signal to pass from it
Weight As Double 'Weight it has
End Type

Private Type Neuron 'The main thing
Dendrites() As Dendrite 'Array of Dendrites
DendriteCount As Long 'Number of dendrites
Bias As Double 'The bias
Value As Double 'The value to be passed to next layer of neurons
Delta As Double 'The delta of neuron (used while learning)
End Type



Private Type Layer 'Layer containing number of neurons
Neurons() As Neuron 'Neurons in the layer
NeuronCount As Long 'Number of neurons
End Type

Private Type NeuralNetwork
Layers() As Layer 'Layers in the network
LayerCount As Long 'Number of layers
LearningRate As Double 'The learning rate of the network
End Type

Dim Network As NeuralNetwork ' Our main network

The code above is pretty self explanatory. And for the things you haven't understood, don't worry that belongs to belong to the training section, which I would be explaining a little later on. The only thing that you need to understand now is bias. Bias is just another value or parameter associated with a neuron. In my and most of the implementations this value is added to the accumulated incoming value. You will better understand this when we see code for running the NN. For now see the code for creating the net:

Function CreateNet(LearningRate As Double, ArrayOfLayers As Variant) As Integer '0 = Unsuccessful and 1 = Successful
Dim i, j, k As Integer
Network.LayerCount = UBound(ArrayOfLayers) 'Init number of layers
If Network.LayerCount < 2 Then 'Input and output layers must be there
    CreateNet = 0 'Unsuccessful
    Exit Function
End If
Network.LearningRate = LearningRate 'The learning rate
ReDim Network.Layers(Network.LayerCount) As Layer 'Redim the layers variable
For i = 1 To UBound(ArrayOfLayers) ' Initialize all layers
DoEvents
    Network.Layers(i).NeuronCount = ArrayOfLayers(i)
    ReDim Network.Layers(i).Neurons(Network.Layers(i).NeuronCount) As Neuron
    For j = 1 To ArrayOfLayers(i) 'Initialize all neurons
    DoEvents
        If i = UBound(ArrayOfLayers) Then 'We will not init dendrites for it because output layers doesn't have any
            Network.Layers(i).Neurons(j).Bias = GetRand 'Set the bias to random value
            Network.Layers(i).Neurons(j).DendriteCount = ArrayOfLayers(i - 1)
            ReDim Network.Layers(i).Neurons(j).Dendrites(Network.Layers(i).Neurons(j).DendriteCount) As Dendrite 'Redim the dendrite var
            For k = 1 To ArrayOfLayers(i - 1)
                DoEvents
                Network.Layers(i).Neurons(j).Dendrites(k).Weight = GetRand 'Set the weight of each dendrite
            Next k
        ElseIf i = 1 Then 'Only init dendrites not bias
            DoEvents 'Do nothing coz it is input layer
        Else
            Network.Layers(i).Neurons(j).Bias = GetRand 'Set the bias to random value
            Network.Layers(i).Neurons(j).DendriteCount = ArrayOfLayers(i - 1)
            ReDim Network.Layers(i).Neurons(j).Dendrites(Network.Layers(i).Neurons(j).DendriteCount) As Dendrite 'Redim the dendrite var
            For k = 1 To ArrayOfLayers(i - 1)
                DoEvents
                Network.Layers(i).Neurons(j).Dendrites(k).Weight = GetRand 'Set the weight of each dendrite
            Next k
        End If
    Next j
Next i
CreateNet = 1
End Function

Basically, what we are doing above is:

• Creating an array of Neurons and dendrites.
• Looping through all the neurons and assigning their respective baises a random value.
• Looping through all the dendrites and assigning their respective weights a random value.

Now lets see, how do we run a neural network:

Function Run(ArrayOfInputs As Variant) As Variant 'It returns the output in form of array
Dim i, j, k As Integer
If UBound(ArrayOfInputs) <> Network.Layers(1).NeuronCount Then
    Run = 0
    Exit Function
End If
For i = 1 To Network.LayerCount
DoEvents
    For j = 1 To Network.Layers(i).NeuronCount
    DoEvents
        If i = 1 Then
            Network.Layers(i).Neurons(j).Value = ArrayOfInputs(j) 'Set the value of input layer
        Else
            Network.Layers(i).Neurons(j).Value = 0 'First set the value to zero
            For k = 1 To Network.Layers(i - 1).NeuronCount
                DoEvents
                Network.Layers(i).Neurons(j).Value = Network.Layers(i).Neurons(j).Value + Network.Layers(i - 1).Neurons(k).Value * Network.Layers(i).Neurons(j).Dendrites(k).Weight 'Calculating the value
            Next k
        Network.Layers(i).Neurons(j).Value = Activation(Network.Layers(i).Neurons(j).Value + Network.Layers(i).Neurons(j).Bias) 'Calculating the real value of neuron
        'Network.Layers(i).Neurons(j).Value = tanh(Network.Layers(i).Neurons(j).Value + Network.Layers(i).Neurons(j).Bias) 'Calculating the real value of neuron
        End If
    Next j
Next i
ReDim OutputResult(Network.Layers(Network.LayerCount).NeuronCount) As Double
For i = 1 To (Network.Layers(Network.LayerCount).NeuronCount)
    DoEvents
    OutputResult(i) = (Network.Layers(Network.LayerCount).Neurons(i).Value) 'The array of output result
Next i
Run = OutputResult
End Function

What we have done above may be summed up as:

• Assigning all the input neurons the values provided in the 'ArrayOfInputs' array. For every other type of neuron, the value assigned is 0 because we have to calculate that value.
• Now for every hidden layer (also for output layer), the value is sum of values of previous layers multiplied with the weight of dendrites.
• To the resulting value, add up the bias value.
• Pass the resulting value through Activation function.
• Give back the values of output neurons as final results.

I would like to explain that activation function. What this function does is that, no matter what number you pass, it always returns a number between 0 and 1. This is because we want the final value of every neuron between 0 and 1, but what if it comes to be something like 2.5 or 1.25, etc. Thus, in order to avoid this incompatibility with standards we use the activation function. The function is something like:

Private Function Activation(Value As Double)
'To crunch a number between 0 and 1
    Activation = (1 / (1 + Exp(Value * -1)))
End Function

I will suppose you wouldn't like to get into the mathematical details(hurray!). In case you like, you can easily google it and find out. Training:

Training is the most important part of a neural network and the one consisting of the most mathematics. We'll be using Backpropagation method for training the NN. I would be explaining only the basic idea how it is done and not all the details. And you need not worry, just apply the formulae written in the code and you can get the training up and ready. Always remember, it is not necessary to know the details of any system in order to get something practical and working out of it. The best example illustrating this principle is Charles Darwin(what?). Yes, at the time when he wrote 'On the Origin of Species', DNA was not known. So, he propounded the evolution without even knowing the method of how it is done i.e. how traits are passed on from parents to offspring.
Back to NN training, first lets see the code:

Function SupervisedTrain(inputdata As Variant, outputdata As Variant) As Integer '0=unsuccessful and 1 = successful
Dim i, j, k As Integer
If UBound(inputdata) <> Network.Layers(1).NeuronCount Then 'Check if correct amount of input is given
    SupervisedTrain = 0
    Exit Function
End If
If UBound(outputdata) <> Network.Layers(Network.LayerCount).NeuronCount Then 'Check if correct amount of output is given
    SupervisedTrain = 0
    Exit Function
End If
Call Run(inputdata) 'Calculate values of all neurons and set the input
'Calculate delta's
For i = 1 To Network.Layers(Network.LayerCount).NeuronCount
DoEvents
    Network.Layers(Network.LayerCount).Neurons(i).Delta = Network.Layers(Network.LayerCount).Neurons(i).Value * (1 - Network.Layers(Network.LayerCount).Neurons(i).Value) * (outputdata(i) - Network.Layers(Network.LayerCount).Neurons(i).Value) 'Deltas of Output layer
    For j = Network.LayerCount - 1 To 2 Step -1
    DoEvents
        For k = 1 To Network.Layers(j).NeuronCount
        DoEvents
            Network.Layers(j).Neurons(k).Delta = Network.Layers(j).Neurons(k).Value * (1 - Network.Layers(j).Neurons(k).Value) * Network.Layers(j + 1).Neurons(i).Dendrites(k).Weight * Network.Layers(j + 1).Neurons(i).Delta 'Deltas of Hidden Layers
        Next k
    Next j
Next i


For i = Network.LayerCount To 2 Step -1
DoEvents
    For j = 1 To Network.Layers(i).NeuronCount
    DoEvents
        Network.Layers(i).Neurons(j).Bias = Network.Layers(i).Neurons(j).Bias + (Network.LearningRate * 1 * Network.Layers(i).Neurons(j).Delta)  'Calculate new bias
        For k = 1 To Network.Layers(i).Neurons(j).DendriteCount
        DoEvents
            
            Network.Layers(i).Neurons(j).Dendrites(k).Weight = Network.Layers(i).Neurons(j).Dendrites(k).Weight + (Network.LearningRate * Network.Layers(i - 1).Neurons(k).Value * Network.Layers(i).Neurons(j).Delta) 'Calculate new weights
        Next k
    Next j
Next i
SupervisedTrain = 1
End Function

So lets see what we have here in arguments, inputdata a sample input array and outputdata, its corresponding output which we require the NN to produce.

• Run the NN using inputdata as inputs. Now, we have all the values of individual neurons.
• Loop through last layer to first layer.
• Calculate the difference between output values desired and output values produced.
• Using that difference, adjust the values of bias and weights accordingly.

This training procedure must be repeated for larger number of samples so that our NN can produce accurate results for untrained input samples.
Yes, I know I haven't explained the training part in details(well practically zero). You see, I had written this code 1-2 years back and was not actively involved with NNs. During that period I have forgotten all the mathematics behind Backpropagation. But anyway, If you want to delve into thedetails, Google is to your rescue. Practical Applications:

There are numerous applications of NNs limited only by your imagination. Innovation is key to success, so dude use NNs to create something which will revolutionize the world!
For the sake of writing :) a few applications are given below:

• Optical Character Recognization
• Stock market prediction
• Creating new art forms
• Modeling human behavior
• Loan risk analysis
• Classification of patterns, Image analysis, music generation, etc, etc.

Further reading:

There is more theory in this world about NNs than you can read in your lifetime. There are Hopfield networks, Recurrent networks, NeuroGenetics, etc. They are for people who really want to do serious research in this field. But in case, you want to read further, you can. Internet is a great free information highway. You can get anything you want on the internet if you know how to search effectively. Even if you don't, train a NN to fetch NN tutorials for you :-)
Thanks for reading this basic tutorial, Hope you enjoyed the ride. Please feel free to give
comments and votes.