Domain Specific Language (DSL) for Neural Network(s)

Howdy folks,

With this entry I would like to show the possibilities of DSL.
For those who don't know DSL, DSL as in Domain Specific Language,
is a modeling tool for modeling problems in an organization for example.

An example that everybody who's familiar with Visual Studio Standard Edition or higher knows is the Class Designer. It's a graphical tool within VS that allows you to model your classes, and it also generated code in the background.

Seeing this, I did some digging, and I came up on the subject of DSL's or DSL Tools.

As you already know probably, I've got a big interest in A.I., neural networks more specifically. Wouldn't it be cool that you could just draw you neural network structure and let VS generate all the network code for you?
In this way you can make balanced or unbalanced neural networks, which can have a big benefit.

I've made a little design of how the designer should look like.



The boxes in yellow are the DSL Domain classes, those classes are the domain model of the DSL, who describe the structure of the problem, those classes should be able to store all the 'problem information'.

The Red boxes are the shapes that are related to the domain model.
A neuron is represented by a Circle.
A Connection should be represented by a line in a certain color etc...
DSL let's you do lot's of fun things with it.

I’ve drawn in Visio how the designer itself should look like, this is an approximate drawing, but it's very close to the reality.
When my designer is finished, I’ll put some screenshots online, and maybe also some code.



This is how it looks like in visual studio (in the experimental hive):



This example shows the structure of a back propagation network that's able to solve the famous XOR problem. This problem is also addressed in another entry of mine regarding the Neural Network Simulator.

The Designer I’m building uses in fact the same framework, with the difference that there is an extra factory for creating the network, and the framework is extended with extended methods of the .NET 3.5 network.

Neural Network Simulation

Since I'm bitten by the A.I. microbe, I'm experimenting with all kinds of Neural Networks and Genetic Algorithms.
Therefore I created a little simulator to visualize the networks, for experimenting with it.

For the visualizing I used GLEE , a 'new' framework from Microsoft Research that's able to draw flowcharts, hierarchical diagrams, ...

I created the tool so that I can manage which network type that I want to simulate.

In this case I'll choose for the Back Propagation network.
The BP-Network is a network that 'learns' from it's mistakes,
it's trained supervised, meaning that the network knows it's input values and the expected output values for the given inputs.
In the scheme below you can see the layout of a BP-Network.
1 and 0 are the input nodes, who accepts numeric values.
2,3,4 are the hidden layers, expanding the learning possibilities
5 is the output node, which gives a number, and you'll probably have to round it to have a significant result. 

We can adjust the number of input , hidden and output-nodes.

Next after defining the network parameters we'll have to provide some input data.
The upper group-box contains the input values, the lower box contains the output values, the amount of values for input and output must be the same, ex. 4 inputs = 4 outputs. (Pretty normal I think)

So in the example below we provide input and output values for the famous XOR-problem.

Input 1	Input 2	Output
0	0	0
0	1	1
1	0	1
1	1	0

The table shows the input for the XOR problem, this data pattern is also inserted in the application below.

After the data-input we'll go to the training part.
Here we'll train the network until all inputs give the right output,

and re-train it until the data-errors are 0. meaning that the output-values are more accurate, or should be more accurate to 0.0 or 1.0 (or the output values you trained it against).

After the training happened, we'll test the network.
As you can see below, in the application you can provide your own input pattern.
In this example we provide 0 and 1 and the output is 0.95, which is rounded -> 1.

If you look at the table above then indeed the output is 1 when the inputs are 0 and 1.

Currently I'm working on the 3 other network types to simulate, BAM, SON and Adeline,
when that part is finished, then I'll make the tool public so that more people can play with it, and understand the working of neural networks better.

I'm also going to include the possibility to remove node links, so that you can train an unbalanced network, which sometimes is trained quicker and can give more accurate output. But that process is more like trail and error :).

Hope you enjoyed this introduction, if you have any questions or comments, don't hesitate to leave them behind or to mail them to me.

Regards,

F.

Picture SOM

Howdy,

This article is about the use of SOM's, I've made an interesting application to demonstrate the use of this self organizing maps.

This program loads pictures and groups them with each other.
So that matching or simular pictures are put together.
I saw this once on a longhorn demo, and I wanted to know how this stuff worked.

Step 1:

So the first step of this project was to download hot women, not nude, otherwise I get in trouble with my girlfriend ;), I found a site (trough another article also) fobpro where you can download thumbnails of woman in simular positions, and with simular backgrounds etc. Which is excellent to test on.

Step 2:

Step 2 was getting data from pictures.
I began with getting the histogram of an image. (greyscale and hsl), an histogram is actually a series of buckets where you put values in, meaning, you look at every pixel and calculate a value from it, say from 0 to 255, then you take for example 16 buckets, with range 255 / 16 and you put the pixel-values in the right bucket, at the end we have the counts of the ammount of values that are in each bucket, and we have our histogram. These 'buckets' will be our inputs for a neural network later on.

Second thing I did was taking Color and Texture area's from the image, in this case, we devide the image (thumbnailed to a size of 50x50) in 9 pieces. (Note that overlapping areas would be better) For each area, we calculate the average color, and texture value ( high value = noisy area, low value is not-noisy area )
This results in 9 values / picture. So 9 input vectors.



On the pic you see on the left the histogram of the image, and the red lines are the area's where i can take the texture or color values from.

I made a library that can fetch these values or combination of, and put them in an input vector. So if we take a histogram and an area, then we have 16 + 9 inputs.

Step 3:

Here I made loaded and thumbnailed the images into memory, and made input vectors for each image, so that at the end I have a list of input vectors to train the neural SOM network with.

As feature map i took a matrix of 10 by 10, and i trained the network with a series of 200 pictures from fobpro.
This way we have only an average of 2 pictures / mapnode.

Step 4:

In step 4 i made the GUI, providing possibilities to manage which types of histograms or area's we would like to train on, also the map-size, and a few buttons to select the input directory, to train the network and to provide an image that's not in the trained collection.

Step 5:

Testing the application :)

First i selected an input directory, and trained the network.
For the thumbnails, it took a few minutes.



After the training has been done, we get the pictures distributed over the mapnodes, is the matrix of buttons on the right.
The number on the buttons tells me how many pictures are under each node.

I've selected the top-left node, and you see the simular pictures found under this node (3 pictures):



Now we take a look the the top-right node, which has 4 nice pictures below it:



If we look at the bottom-left node with 4 pixtures, then we see that the pictures
are of the same model in the same clothes, but in other positions, so it works!



This is the bottom-right node of 3 pictures:



Now let's test that the network really can group the pictures together!
I fed the network an image that was not in the collection of the training data,
and look, it has found the right picture-set! In the first set, there is another girl with the same clothes on, with different hair, the network looks at color and texture, histograms and areas, in this case the histogram had 64 buckets, and the area's 9 divisions. So the input vector for this network was 64 + 64 + 9 + 9 nodes.

The other girl matched the texture area's and texture histogram pretty closely, also the color histogram is almost the same, and the color area's which are averages will be almost the same, so it's correct that the girl is in this collection.



Let's feed another one:



Conclusion:

The SOM network can do amazing stuff, there are tons of other applications where you can use it for. The training did not take very long, because all thumbs are resized before getting the data from it.

The next step would be face recognition, and edge detection, so that I can actually filter persons out of the collection, that would be really cool ;)

I also found some information about hierachically growing SOM networks, that way the network has a granularity for the spreading of the nodes.
For example if we take the color som, and we get different shades of red, then the shades of red would come under a node which is also a som network, that way we have toplevel red, green, etc, and sublevel under the red, the shades of red, this way we can group even better.

The next thing i'm going to try is to finetune this application and make it a WCF service (.NET 3.0) and feed it my entire picture collection, so that I can make a kind of picasa form grouping pictures :)

Now, it's just therapy to learn about A.I and make some pretty cool thing with it, in a pretty-learning environment. Not that the pictures supplied here are only for educational purposes :)

I hope you got somewhat wiser about neural networks, and the possibilies... I will come back later with more fun stuff :)

Regards,

Whizzo

Color SOM

With this article i would like to demostrate the Color SOM (Self Organizing Map)

The basics of this application comes to creating and training a neural network that groups colors with eachother.

The neural network itself is a SOM (Self Organizing Map) or SON (Self Organizing Network) map. It has 3 input nodes, and in this example a 40 by 40 matrix of output nodes. In this network there are no 'hidden' layers.

The architecture looks like this:



We have our input values, (RGB color), in this case 3 inputs, so the input vector is R,G,B (values from 0 to 255)
We also have a feature map of size X and Y, (40 x 40 in my examples)
Every node of this feature map has a feature vector to an input-node.
This is called the feature vector, and every vector has a weight (Wij)

Now if we look to what I fabricated, then we have to initialize the weights of the vectors connecting each input-node to each output-node with gradient values (left pic) or with random values (right pic).



Then I made a data-set where I trained the network with.
In my example i have trained a network with 9 colors, and a one with random colors ( with 1600 unique colors in it).
While the network is training, it is going to adjust the weights of the feature vectors so that a 'grouping' is brought to life (using epoch- and sigmoid functions etc, look in "my links" to find a tutorial on ai-junkie, which is pretty good).

If I initialize the "feature vector-weights" with gradient values
((0,0,0) on the top-left and (255,255,255) on the bottom-right)
then we get, with the same map-size and color inputs offcourse, always the same end-result if the network is fully trained.

Example of training a gradient colormap with 9 colors as input dataset:



Example of training a gradient colormap (40x40) with 1600 unique colors:



If I train the map with random initialization and random colors as inputs,
then I get a simular result.
When I do it with 9 colors and random initialization, then the 9 colors will be on another place each time we (re)train the network.

Now, think about what for use this could have?
The network groups simular colors in the case, bottomline is, that it can group values with each other, I will show you later on a nice example where I used this technique to group pictures with each other in a nice learning environment ;)

I hope I could give you a little more information or insight about is topic, I will surely discuss this later with a little more depth.

If you don't know anything of A.I, then i strongly suggest to take a look at my links, and follow a couple of tutorials, you will be on your way pretty soon.

Regards,

Whizzo

Artificial Intelligence

Howdy,

Since a while ago i gained interest in A.I., I'm interested in it since a long time, but never had the time to really get started with it.

But hippieyee, when it was my birthday, a bought a couple of books and actually read them also. Then the A.I vibes came really to life :)

After reading the book(s) i started to implement a framework for neural networks,
because the N.N. part of A.I interests me alot.
It's pretty cool how many simularities you can see in it, in relation with a human brain.

Now, i learned that there are many different kinds of neural nets, with the framework, based on explenation from a book, i think i managed to get the biggest kinds implemented, that's ADELINE, BackPropagation, BAM and SOM (or SON).

If you look at the working of those networks, you'll see that there are certain simularities in it. I'll go deeper on that later.

My Object Oriented Neural Network Framework looks like this:



Like you see i only show the classnames, because the complete content doesn't fit on the screen.

The thing that all N.N. have in common is a basenode, from there we make a feedforward-node, SON-Node (for self organizing maps), InputNode for giving input values and a baseNetwork, that actually consists of BaseNodes.



Then we have the BaseLink, it's a link between the Neuron-nodes (a neuron is an element of a Neural Network). We have the adeline links, Backprop links with the epochlink (for learning) and the BAM and SON links.



With this construction we can make all combinations of neural networks. Maybe you will have to inherit something to extend functionality.

The usual practices to learn something about neural networks (which i encountered the most) are:

• Self Origanizing Nets


• Color SOM


• XOR problem


• ... there are others also offcourse.

I also made a deviation, that i picked op on www.generation5.org, and that's detecting picture simularity with neural networks.

I'll post more on those subjects later.

Regards,

F.