Domain Specific Language (DSL) for Neural Network(s)

Howdy folks,

With this entry I would like to show the possibilities of DSL.
For those who don't know DSL, DSL as in Domain Specific Language,
is a modeling tool for modeling problems in an organization for example.

An example that everybody who's familiar with Visual Studio Standard Edition or higher knows is the Class Designer. It's a graphical tool within VS that allows you to model your classes, and it also generated code in the background.

Seeing this, I did some digging, and I came up on the subject of DSL's or DSL Tools.

As you already know probably, I've got a big interest in A.I., neural networks more specifically. Wouldn't it be cool that you could just draw you neural network structure and let VS generate all the network code for you?
In this way you can make balanced or unbalanced neural networks, which can have a big benefit.

I've made a little design of how the designer should look like.



The boxes in yellow are the DSL Domain classes, those classes are the domain model of the DSL, who describe the structure of the problem, those classes should be able to store all the 'problem information'.

The Red boxes are the shapes that are related to the domain model.
A neuron is represented by a Circle.
A Connection should be represented by a line in a certain color etc...
DSL let's you do lot's of fun things with it.

I’ve drawn in Visio how the designer itself should look like, this is an approximate drawing, but it's very close to the reality.
When my designer is finished, I’ll put some screenshots online, and maybe also some code.



This is how it looks like in visual studio (in the experimental hive):



This example shows the structure of a back propagation network that's able to solve the famous XOR problem. This problem is also addressed in another entry of mine regarding the Neural Network Simulator.

The Designer I’m building uses in fact the same framework, with the difference that there is an extra factory for creating the network, and the framework is extended with extended methods of the .NET 3.5 network.

Object Cloning Using IL in C#

This subject is inspired on a session I followed on the TechDays 2008,
that addressed the fact that IL (Intermediate Language) can be used to clone objects, among other things, and that it's not evil at all, and it can be pretty performant also.

You only have to see that you don't overuse it tho, because otherwise the readability of your code is reduced, which is not a good thing for the maintenance.
And wrong usage of reflection (what IL is, or at least uses) can also result in poor performance.

This being said, I tested this on my own, with some self written code and comments,
just to test that it was really true what Mr. Roy Osherove told me ;)

Below you can see a screenshot of a console application that does the tests.

As you see, sometimes the IL code is even faster the the normal cloning on a simple class Person with a couple of fields in it that are filled in with some random values.

Let's take a look at the code, first we declare a simple class Person
I've added comments in the code so that people that don't understand IL too much, can also understand what's happening.

Person class definition:

public class Person
{
    private int _id;
    private string _name;
    private string _firstName;
    private string _field1, _field2, _field3;

    public Person()
    {

    }

    public int ID
    {
        get { return _id; }
        set { _id = value; }
    }

    public string Name
    {
        get { return _name; }
        set { _name = value; }
    }

    public string FirstName
    {
        get { return _firstName; }
        set { _firstName = value; }
    }
}

TestCode:

The code below is a nice piece of code, read through the comments and you'll understand what's stated.

class Program
{
    /// <summary>
    /// Delegate handler that's used to compile the IL to.
    /// (This delegate is standard in .net 3.5)
    /// </summary>
    /// <typeparam name="T1">Parameter Type</typeparam>
    /// <typeparam name="TResult">Return Type</typeparam>
    /// <param name="arg1">Argument</param>
    /// <returns>Result</returns>
    public delegate TResult Func<T1, TResult>(T1 arg1);
    /// <summary>
    /// This dictionary caches the delegates for each 'to-clone' type.
    /// </summary>
    static Dictionary<Type, Delegate> _cachedIL = new Dictionary<Type, Delegate>();

    /// <summary>
    /// The Main method that's executed for the test.
    /// </summary>
    /// <param name="args"></param>
    static void Main(string[] args)
    {
        DoCloningTest(1000);
        DoCloningTest(10000);
        DoCloningTest(100000);
        //Commented because the test takes long ;)
        //DoCloningTest(1000000);

        Console.ReadKey();
    }

    /// <summary>
    /// Do the cloning test and printout the results.
    /// </summary>
    /// <param name="count">Number of items to clone</param>
    private static void DoCloningTest(int count)
    {
        // Create timer class.
        HiPerfTimer timer = new HiPerfTimer();
        double timeElapsedN = 0, timeElapsedR = 0, timeElapsedIL = 0;

        Console.WriteLine("--> Creating {0} objects...", count);
        timer.StartNew();
        List<Person> personsToClone = CreatePersonsList(count);
        timer.Stop();
        Person temp = CloneObjectWithIL(personsToClone[0]);
        temp = null;
        Console.WriteLine("\tCreated objects in {0} seconds", timer.Duration);

        Console.WriteLine("- Cloning Normal...");
        List<Person> clonedPersons = new List<Person>(count);
        timer.StartNew();
        foreach (Person p in personsToClone)
        {
            clonedPersons.Add(CloneNormal(p));
        }
        timer.Stop();
        timeElapsedN = timer.Duration;

        Console.WriteLine("- Cloning IL...");
        clonedPersons = new List<Person>(count);
        timer.StartNew();
        foreach (Person p in personsToClone)
        {
            clonedPersons.Add(CloneObjectWithIL<Person>(p));
        }
        timer.Stop();
        timeElapsedIL = timer.Duration;

        Console.WriteLine("- Cloning Reflection...");
        clonedPersons = new List<Person>(count);
        timer.StartNew();
        foreach (Person p in personsToClone)
        {
            clonedPersons.Add(CloneObjectWithReflection(p));
        }
        timer.Stop();
        timeElapsedR = timer.Duration;

        Console.WriteLine();
        Console.ForegroundColor = ConsoleColor.Green;
        Console.WriteLine("----------------------------------------");
        Console.WriteLine("Object count:\t\t{0}", count);
        Console.WriteLine("Cloning Normal:\t\t{0:00.0000} s", timeElapsedN);
        Console.WriteLine("Cloning IL:\t\t{0:00.0000} s", timeElapsedIL);
        Console.WriteLine("Cloning Reflection:\t{0:00.0000} s", timeElapsedR);
        Console.WriteLine("----------------------------------------");
        Console.ResetColor();
    }

    /// <summary>
    /// Create a list of persons with random data and a given number of items.
    /// </summary>
    /// <param name="count">Number of persons to generate</param>
    /// <returns>List of generated persons</returns>
    private static List<Person> CreatePersonsList(int count)
    {
        Random r = new Random(Environment.TickCount);
        List<Person> persons = new List<Person>(count);
        for (int i = 0; i < count; i++)
        {
            Person p = new Person();
            p.ID = r.Next();
            p.Name = string.Concat("Slaets_", r.Next());
            p.FirstName = string.Concat("Filip_", r.Next());
            persons.Add(p);
        }
        return persons;
    }

    /// <summary>
    /// Clone one person object with reflection
    /// </summary>
    /// <param name="p">Person to clone</param>
    /// <returns>Cloned person</returns>
    private static Person CloneObjectWithReflection(Person p)
    {
        // Get all the fields of the type, also the privates.
        FieldInfo[] fis = p.GetType().GetFields(System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.NonPublic);
        // Create a new person object
        Person newPerson = new Person();
        // Loop through all the fields and copy the information from the parameter class
        // to the newPerson class.
        foreach (FieldInfo fi in fis)
        {
            fi.SetValue(newPerson, fi.GetValue(p));
        }
        // Return the cloned object.
        return newPerson;
    }

    /// <summary>
    /// Generic cloning method that clones an object using IL.
    /// Only the first call of a certain type will hold back performance.
    /// After the first call, the compiled IL is executed.
    /// </summary>
    /// <typeparam name="T">Type of object to clone</typeparam>
    /// <param name="myObject">Object to clone</param>
    /// <returns>Cloned object</returns>
    private static T CloneObjectWithIL<T>(T myObject)
    {
        Delegate myExec = null;
        if (!_cachedIL.TryGetValue(typeof(T), out myExec))
        {
            // Create ILGenerator
            DynamicMethod dymMethod = new DynamicMethod("DoClone", typeof(T), new Type[] { typeof(T) }, true);
            ConstructorInfo cInfo = myObject.GetType().GetConstructor(new Type[] { });

            ILGenerator generator = dymMethod.GetILGenerator();

            LocalBuilder lbf = generator.DeclareLocal(typeof(T));
            //lbf.SetLocalSymInfo("_temp");

            generator.Emit(OpCodes.Newobj, cInfo);
            generator.Emit(OpCodes.Stloc_0);
            foreach (FieldInfo field in myObject.GetType().GetFields(System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.NonPublic))
            {
                // Load the new object on the eval stack... (currently 1 item on eval stack)
                generator.Emit(OpCodes.Ldloc_0);
                // Load initial object (parameter)          (currently 2 items on eval stack)
                generator.Emit(OpCodes.Ldarg_0);
                // Replace value by field value             (still currently 2 items on eval stack)
                generator.Emit(OpCodes.Ldfld, field);
                // Store the value of the top on the eval stack into the object underneath that value on the value stack.
                //  (0 items on eval stack)
                generator.Emit(OpCodes.Stfld, field);
            }
            
            // Load new constructed obj on eval stack -> 1 item on stack
            generator.Emit(OpCodes.Ldloc_0);
            // Return constructed object.   --> 0 items on stack
            generator.Emit(OpCodes.Ret);

            myExec = dymMethod.CreateDelegate(typeof(Func<T, T>));
            _cachedIL.Add(typeof(T), myExec);
        }
        return ((Func<T, T>)myExec)(myObject);
    }

    /// <summary>
    /// Clone a person object by manually typing the copy statements.
    /// </summary>
    /// <param name="p">Object to clone</param>
    /// <returns>Cloned object</returns>
    private static Person CloneNormal(Person p)
    {
        Person newPerson = new Person();
        newPerson.ID = p.ID;
        newPerson.Name = p.Name;
        newPerson.FirstName = p.FirstName;
        return newPerson;
    }
}

The basic thing that it does is, create a DynamicMethod, get the ILGenerator, emit code in the method, compile it to a delegate, and execute the delegate.
The delegate is cached so that the IL is not generated each time a cloning should take place, so we loose only one time performance, when the first object is cloned (the IL has to be created and compiled at runtime).

Hopefully this article is of use for some people, if so, let me know.

Regards