vrijdag 3 juli 2009

"Outlook could not create the work file. Check the TEMP environment variable"

This was the error I had when I had installed Office 2007
Word also had this error :

"Word could not create the work file. Check the TEMP environment variable"

After a while of searching and cursing, I've finally found the solution!

First I've took a look at the environment variables, but they pointed to the right directory -> %USERPROFILE%\Local Settings\Temp a lot of sites suggested that that was the problem, but the path was/is valid when I typed it in the explorer.

After a lot of configuration swimming I encountered that outlook also uses registry keys, more specificly these:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders


The 'cache' key of both registry entries.

When opening regedit, I saw that the 'User Shell Folders' key was pointing to a k: drive (which I didn't have): k:\Local\Microsoft\Windows\Temporary Internet Files

The 'Shell Folders' was pointing to C:\Users\Whizzo\AppData\Local\Microsoft\Windows\Temporary Internet Files

So I've changed the k drive to my user profile's temp folder:
%USERPROFILE%\Local Settings\Temp

And BAM, the problem was solved!!

I hope this is helpfull for someone ;)

woensdag 3 september 2008

Hierarchical Temporal Memory

It's been a while since I've posted a new topic, so I was thinking on what topic to post about on my blog. After a while of hard thinking and sleepless nights (just kidding) it stroked me... The idea entered my mind :)

I while ago I stumbled on an article somewhere regarding HTM (Hierarchical Temporal Memory)... 'What?' you may think? Well, it's an algorithm in A.I. invented by numenta if I'm not mistaken, and it's meant for classifying objects, or at least it can be used for it.

Imagine that you have a cat and a bowl of food in front of the cat, while the cat steps towards the bowl, the image of the bowl changes each step the cat takes.
The cat-brain must then determine whether the bowl is still the same bowl, it all happens in the sub-consciousness of the cat's brain, it's logical that it's the same bowl,
although the bowl is with every step different in size and perspective and if it’s a bummer the lighting may also be different.

Now if we want to translate it to image analysis with A.I., whether to know that an object is still in an image when we present a series of pictures (or movie) this methodology/technology becomes useful :)
It's an algorithm that functions in function of time.
You can pretty much compare it with an Self Organizing Map (SOM) network but in function of time, and with multiple layers.
Each layer is a specialization of the previous layer until the top layer has a classification answer with a certain belief that the thing in the picture is still the same thing.

The basic structure (from the algorithm document) is something like this:



I've found a particular document (paper) explaining the working of the algorithm and
for a more detailed and scientific explanation, I'll have to refer you to their amazing paper.
You can find it here.
As stubborn and curious I am, I'm trying to create something similar, but with my own twist of thought, actually I was thinking of such a mechanism for a long time before I’ve had found this algorithm, only the 'how to start on it' was still missing in my mind, this paper gave me a few ideas on how to create my own program/simulation.

I'll post my findings when my experiments with this algorithm are successful :)
Along with maybe if it's possible a basic peace of code for people who are interested in it for research or self-development.

I must mention that I've looked around on the web and didn't find a piece of code (except the numenta site) that implements this algorithm... such a bummer..
It has probably something to do with the patents obtained on this technology I guess.

Anyway, meanwhile receive my best regards and until the following post ;)

F.

donderdag 7 augustus 2008

My first steps in embedded programming

Since I've always been interested in automatisation (domotics) and because I saw the introduction of a new module regarding embedded programming of GHI Electronics, I've bought a development board that makes it able to program through the .NET Micro Framework.

In the beginning I even had never heard of .NET Micro Framework, didn't know that it existed.

I got really excited because this board provides all kinds of interfaces you can think of, analog in/outputs, digital in/outputs, USB interfaces, Ethernet port, serial ports, parallel ports, etc..., even a little Piezo (=speaker) where you can buzz around with ;)

You can program managed in the micro framework, and when you deploy it, the framework uploads the code to the board, and then you can test it standalone.
It's great for making 'smart' devices, and since you can address the FAT file system on USB thumb drives/SD Cards, makes it possible that you can go really far in it.

The device also supports strong keyed assemblies, if you don't have a the right key, then the board will be flashed :) (Which is great :d, except when you've lost it...)

Here are some images of the device:

This is the development board, as you can see, all kinds of ports are present on it.



This is the master module, that contains the code (firmware).
It has also a lot of pins, where you can attach you own electronics on, if you know what you are doing :) The pins are located on the left and richt side of the PCB.
This module is located under the LCD display of the development board.



At the time of writing I've successfully succeeded at letting a LED blink on the board with a given frequency, and even fading of the LED worked ;)
I know, it's stupid but you have to start somewhere :)

vrijdag 27 juni 2008

Dirty check on objects

A problem that's pretty common is to determine whether an object has been changed, how do we know an object is changed?
You can use for example datasets; datasets keep track of changes, and can update only changed rows.

But in most of the projects that I had to work on, datasets weren't used, or as less as possible, they used their own domain model for storing and transmitting data from point a to point b.

Another problem they encountered was the fact that some of the objects marked as serializable are in fact not really serializable (they are, but have issues), like the ObservableCollection used in combination mostly with WPF. That class is marked serializable, but once you attach the added en removed events, then you can't serialize it anymore, because the delegates aren't marked NonSerializable.

Knowing that these were problems, I've created a solution to bypass all these problems without inheriting or wrapping or using surrogates or whatever.

I've used code that I've blogged about earlier in the IL section:
» Deep cloning objects with IL

This code has the ability to clone objects with IL, you can choose for cloning deeply or shallow, or to exclude fields from being cloned, you can also adapt it to work with other attributes for usage in WCF for example.
It can clone ObserveableCollection because it strips (ignores) the delegates from it.

The code I've written makes it possible to check for changes in an object, it doesn't really matter which object, because it uses a MD5 hash code. The only problem is: knowing what has been changed, but that's not needed in my case.
Feel free to contribute for knowing where the object has been changed, without wrapping it, or adding stuff to it :)

Here's the code that does all the work:

    /// <summary>
    /// Class for checking wether objects have been changed or not.
    /// Clones the object in order to check, for stripping events etc,
    /// but checks against the original object.
    /// Author:     Slaets Filip
    /// Date:       25/06/2008
    /// </summary>
    
public static class DirtyChecker
    {
        
/// <summary>
        /// Cache for holding hashed objects-data.
        /// </summary>
        
private static List<DirtyEntity> _hashesCache = new List<DirtyEntity>();

        
/// <summary>
        /// Retrieve a hashcode for an object.
        /// </summary>
        /// <param name="objToCheck">Object to get hashcode for.</param>
        /// <returns>hexadecimal hashcode</returns>
        
private static string GetHashRecord(object objToCheck)
        {
            
// Create our hash-record of this object.
            
MemoryStream ms = new MemoryStream();
            
BinaryFormatter bf = new BinaryFormatter();
            
bf.Serialize(ms, objToCheck);
            
MD5 md5 MD5.Create();
            
ms.Position 0;
            byte
[] hash md5.ComputeHash(ms);
            
// Convert the hashvalue to a HEX string.
            
StringBuilder sb = new StringBuilder();
            foreach 
(byte outputByte in hash)
            {
                
// Convert each byte to a Hexadecimal lower case string
                
sb.Append(outputByte.ToString("X2").ToLower());
            
}
            
return sb.ToString();
        
}

        
/// <summary>
        /// Check wether an object has changed somewhere or not.
        /// </summary>
        /// <param name="objToCheck">Object to check dirtyness of.</param>
        /// <returns>True when something has been changed in the object.</returns>
        
public static bool IsDirty(object objToCheck)
        {
            
if (objToCheck.GetType().GetCustomAttributes(typeof(SerializableAttribute), true).Length == 0)
            {
                
throw new ArgumentException("Object is not serializable, mark it with [Serializable]""objToCheck");
            
}

            
string name = string.Format("{0}.CloneHelper`1[[{1}]]"typeof(DirtyChecker).Namespace, objToCheck.GetType().AssemblyQualifiedName);
            
// Let's invoke the cloning mechanism
            
Type cloneType Type.GetType(name);
            if 
(cloneType == null)
                
throw new InvalidOperationException("Check the CloneHelper location, must be in same assembly and namespace as DirtyChecker!");
            
MethodInfo mi cloneType.GetMethod("Clone"new Type[] { objToCheck.GetType(), typeof(CloneType) });
            object 
toHashObject mi.Invoke(nullnew object[] { objToCheck, CloneType.ShallowCloning });

            string 
hashRecord GetHashRecord(toHashObject);
            
DirtyEntity entity = new DirtyEntity(objToCheck.GetHashCode(), hashRecord);
            if 
(!_hashesCache.Contains(entity))
            {
                _hashesCache.Add(entity)
;
                return false;
            
}

            DirtyEntity foundEntity 
_hashesCache[_hashesCache.IndexOf(entity)];
            if 
(foundEntity.HashRecord.Equals(entity.HashRecord, StringComparison.InvariantCulture))
            {
                
return false;
            
}
            
if (foundEntity.UpdateCascading)
            {
                foundEntity.HashRecord 
hashRecord;
            
}
            
return true;
        
}

        
/// <summary>
        /// Check wether an object has changed somewhere or not.
        /// </summary>
        /// <param name="objToCheck">Object to check dirtyness of.</param>
        /// <param name="cascaded">If the record is dirty, take the new record as base-record for checking dirtyness.</param>
        /// <returns>True when something has been changed in the object.</returns>
        
public static bool IsDirty(object objToCheck, bool cascaded)
        {
            
if (objToCheck.GetType().GetCustomAttributes(typeof(SerializableAttribute), true).Length == 0)
            {
                
throw new ArgumentException("Object is not serializable, mark it with [Serializable]""objToCheck");
            
}

            
string name = string.Format("{0}.CloneHelper`1[[{1}]]"typeof(DirtyChecker).Namespace, objToCheck.GetType().AssemblyQualifiedName);
            
// Let's invoke the cloning mechanism
            
Type cloneType Type.GetType(name)
            
MethodInfo mi cloneType.GetMethod("Clone"new Type[] { objToCheck.GetType() });
            object 
toHashObject mi.Invoke(nullnew object[] { objToCheck });

            string 
hashRecord GetHashRecord(toHashObject);
            
DirtyEntity entity = new DirtyEntity(objToCheck.GetHashCode(), hashRecord, cascaded);
            if 
(!_hashesCache.Contains(entity))
            {
                _hashesCache.Add(entity)
;
                return false;
            
}

            DirtyEntity foundEntity 
_hashesCache[_hashesCache.IndexOf(entity)];
            if 
(foundEntity.HashRecord.Equals(entity.HashRecord, StringComparison.InvariantCulture))
            {
                
return false;
            
}
            
if (foundEntity.UpdateCascading || cascaded)
            {
                foundEntity.HashRecord 
hashRecord;
            
}
            
return true;
        
}

        
/// <summary>
        /// Class for storing hashing data in the cache.
        /// </summary>
        
private class DirtyEntity
        {
            
private int _hashCodeOfObject;
            private string 
_hashRecord;
            private bool 
_updateCascading;

            public 
DirtyEntity(int hc, string hr)
            {
                _hashCodeOfObject 
hc;
                
_hashRecord hr;
                
_updateCascading = true;
            
}

            
public DirtyEntity(int hc, string hr, bool cascading)
            {
                _hashCodeOfObject 
hc;
                
_hashRecord hr;
                
_updateCascading cascading;
            
}

            
public int HashCodeOfObject
            {
                
get return _hashCodeOfObject}
                
set { _hashCodeOfObject = value; }
            }

            
public string HashRecord
            {
                
get return _hashRecord}
                
set { _hashRecord = value; }
            }

            
public bool UpdateCascading
            {
                
get return _updateCascading}
                
set { _updateCascading = value; }
            }

            
public override int GetHashCode()
            {
                
return base.GetHashCode();
            
}

            
public override bool Equals(object obj)
            {
                
if (obj is DirtyEntity)
                {
                    DirtyEntity de 
obj as DirtyEntity;
                    return 
de._hashCodeOfObject == this._hashCodeOfObject;
                
}
                
return base.Equals(obj);
            
}
        }
    }


bool IsDirty(object objToCheck) and bool IsDirty(object objToCheck, bool cascaded) the first one checks that the object is dirty against a previous version of the object, the previous version being the version when the IsDirty method is called for the first time.

The second method has the cascaded parameter, when it's set to true, then the method will compare against the previous called IsDirty result, each time it's called, the internal hashing-cache will be updated.

An example of usage:

// Create a list of 100 persons...
List<Person> list CreatePersonsList(100);

// Initialize the dirty checker to know how the object looks like
bool dirty1 DirtyChecker.IsDirty(list);

// Change something
list[0].Addresses[0].City "Jut City";

// Check again to see that something has been changed.
bool dirty2 DirtyChecker.IsDirty(list);

// Output:
// dirty1 = false;
// dirty2 = true;


Voila, if it's useful, let me know please ;)

Cheers!

woensdag 25 juni 2008

.NET to PHP Connector

Howdy,

Wouldn't it be easy to call in .NET a PHP function, and retrieve an answer in the form of an .NET object?

I've looked around on the net and I haven't found a library that does such a thing.
Because I've made a site in PHP for managing kittens and nests (cat-site) etc, where users can add stuff as they want, the client requested to manage the clients and the nests etc, which is pretty logical, but implementing a controlpanel in PHP that does all that was requested was a little too much work, because I had a little budget to take in account.

So I've created a .NET to PHP connector (that I can re-use in other projects) for connecting to the kitten-server and retrieve information about nests etc...

I've also implemented it that way that there is a possibility to easyly change the security of the PHP-Connector.
For the moment the default security is base64. I'm sure there are better ways to send data from point a to point b, but the issue is that if you encode stuff, for sending it to the server, then you have to decode it on the other side also, and you need to decode it with PHP, while it's encoded in .NET, it's not always that easy.

The architecture is pretty easy, it's straight forward, and based on async calls to the server. The usage is also pretty straight forward:



  • Put the gateway PHP file and code file on the server;
  • Instantiate the Connector for connecting to the gateway URL;
  • Call the connector.SendQuery(involvedObject, "methodname", [arguments]);
  • Catch the receive answer event which gives you back your 'involvedObject' and an Answer object containing the server's response.


With this mechanism, I've created an administration tool, which is session-based with login system etc, you can name the gateway as you like, the mechanism only accepts POST requests, and the request are encoded... you can also define methods that may never be called from the connector for security reasons...

I know this system is not super secure, but you can adapt it to you own whishes, and extend it to your believes, it's just a primer where you can base yourselves on.

I will upload the code to a server soon when my project is ready and the code is stable enough, (it's almost ready ;))

Maye a graph would make it more clear how the system works :

Class diagram:



If you have any comments or suggestions, please feel free to leave them ;)

Regards,
F.

vrijdag 20 juni 2008

Object Deep Cloning using IL in C# - version 1.1

Howdy,

Meanwhile i've been busy with the cloning stuff in IL, and on demand of mr. Sam
I've adapted the code so that it can also clone LINQ entities (EntitySet etc)

The code is not yet in a final stage but it is becoming more and more stable as I refine the code, so, if you have comments or suggestions, please let me know...

I've added in this version:


  • Deep cloning until 1 level (more level's, i'm not yet sure)

  • Use of attributes for defining which field should be deepcloned, shallowcloned or not be cloned at all. You can change the 'CloneAttribute' by a system attribute if you like, when you for example transfer stuff through WCF or something like that. Just map you own attribute to the cloning attribute in the function -> 'GetCloneTypeForField' and it's normally OK.

  • The cloning code is put together in a CloneHelper static generic class, so that you can use it in combination with the ICloneable interface for example, or just in some framework as possible cloning mechanism.



Now let's take a look at the code:

using System;
using 
System.Collections.Generic;
using 
System.Text;
using 
System.Reflection;
using 
System.Reflection.Emit;
using 
System.Threading;
using 
System.Collections;

namespace 
Cloning
{
    
/// <summary>
    /// Enumeration that defines the type of cloning of a field.
    /// Used in combination with the CloneAttribute
    /// </summary>
    
public enum CloneType
    {
        None,
        ShallowCloning,
        DeepCloning
    }

    
/// <summary>
    /// CloningAttribute for specifying the cloneproperties of a field.
    /// </summary>
    
[AttributeUsage(AttributeTargets.Field, AllowMultiple = false)]
    
public class CloneAttribute : Attribute
    {
        
private CloneType _clonetype;
        public 
CloneAttribute()
        {

        }

        
public CloneType CloneType
        {
            
get return _clonetype}
            
set { _clonetype = value; }
        }
    }

    
/// <summary>
    /// Class that clones objects
    /// </summary>
    /// <remarks>
    /// Currently can deepclone to 1 level deep.
    /// Ex. Person.Addresses (Person.List<Address>) 
    /// -> Clones 'Person' deep
    /// -> Clones the objects of the 'Address' list deep
    /// -> Clones the sub-objects of the Address object shallow. (at the moment)
    /// </remarks>
    
public static class CloneHelper<T>
    where T : 
class
    
{
        
#region Declarations
        
// Dictionaries for caching the (pre)compiled generated IL code.
        
private static Dictionary<Type, Delegate> _cachedILShallow = new Dictionary<Type, Delegate>();
        private static 
Dictionary<Type, Delegate> _cachedILDeep = new Dictionary<Type, Delegate>();
        
// This is used for setting the fixed cloning, of this is null, then
        // the custom cloning should be invoked. (use Clone(T obj) for custom cloning)
        
private static CloneType? _globalCloneType CloneType.ShallowCloning;

        #endregion

        #region
 Public Methods

        
/// <summary>
        /// Clone an object with Deep Cloning or with a custom strategy 
        /// such as ShallowCloning and/or DeepCloning combined (use the CloneAttribute)
        /// </summary>
        /// <param name="obj">Object to perform cloning on.</param>
        /// <returns>Cloned object.</returns>
        
public static T Clone(T obj)
        {
            _globalCloneType 
= null;
            return 
CloneObjectWithILDeep(obj);
        
}

        
/// <summary>
        /// Clone an object with one strategy (DeepClone or ShallowClone)
        /// </summary>
        /// <param name="obj">Object to perform cloning on.</param>
        /// <param name="cloneType">Type of cloning</param>
        /// <returns>Cloned object.</returns>
        /// <exception cref="InvalidOperationException">When a wrong enum for cloningtype is passed.</exception>
        
public static T Clone(T obj, CloneType cloneType)
        {
            
if (_globalCloneType != null)
                _globalCloneType 
cloneType;
            switch 
(cloneType)
            {
                
case CloneType.None:
                    
throw new InvalidOperationException("No need to call this method?");
                case 
CloneType.ShallowCloning:
                    
return CloneObjectWithILShallow(obj);
                case 
CloneType.DeepCloning:
                    
return CloneObjectWithILDeep(obj);
                default
:
                    
break;
            
}
            
return default(T);
        
}

        
#endregion

        #region
 Private Methods

        
/// <summary>    
        /// Generic cloning method that clones an object using IL.    
        /// Only the first call of a certain type will hold back performance.    
        /// After the first call, the compiled IL is executed.    
        /// </summary>    
        /// <typeparam name="T">Type of object to clone</typeparam>    
        /// <param name="myObject">Object to clone</param>    
        /// <returns>Cloned object (shallow)</returns>    
        
private static T CloneObjectWithILShallow(T myObject)
        {
            Delegate myExec 
= null;
            if 
(!_cachedILShallow.TryGetValue(typeof(T), out myExec))
            {
                DynamicMethod dymMethod 
= new DynamicMethod("DoShallowClone"typeof(T), new Type[] { typeof(T) }, Assembly.GetExecutingAssembly().ManifestModule, true);
                
ConstructorInfo cInfo myObject.GetType().GetConstructor(new Type[] { });
                
ILGenerator generator dymMethod.GetILGenerator();
                
LocalBuilder lbf generator.DeclareLocal(typeof(T));
                
generator.Emit(OpCodes.Newobj, cInfo);
                
generator.Emit(OpCodes.Stloc_0);
                foreach 
(FieldInfo field in myObject.GetType().GetFields(System.Reflection.BindingFlags.Instance
                | System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Public))
                {
                    generator.Emit(OpCodes.Ldloc_0)
;
                    
generator.Emit(OpCodes.Ldarg_0);
                    
generator.Emit(OpCodes.Ldfld, field);
                    
generator.Emit(OpCodes.Stfld, field);
                
}
                generator.Emit(OpCodes.Ldloc_0)
;
                
generator.Emit(OpCodes.Ret);
                
myExec dymMethod.CreateDelegate(typeof(Func<T, T>));
                
_cachedILShallow.Add(typeof(T), myExec);
            
}
            
return ((Func<T, T>)myExec)(myObject);
        
}

        
/// <summary>
        /// Generic cloning method that clones an object using IL.
        /// Only the first call of a certain type will hold back performance.
        /// After the first call, the compiled IL is executed. 
        /// </summary>
        /// <param name="myObject">Type of object to clone</param>
        /// <returns>Cloned object (deeply cloned)</returns>
        
private static T CloneObjectWithILDeep(T myObject)
        {
            Delegate myExec 
= null;
            if 
(!_cachedILDeep.TryGetValue(typeof(T), out myExec))
            {
                
// Create ILGenerator            
                
DynamicMethod dymMethod = new DynamicMethod("DoDeepClone"typeof(T), new Type[] { typeof(T) }, Assembly.GetExecutingAssembly().ManifestModule, true);
                
ILGenerator generator dymMethod.GetILGenerator();
                
LocalBuilder cloneVariable generator.DeclareLocal(myObject.GetType());

                
ConstructorInfo cInfo myObject.GetType().GetConstructor(Type.EmptyTypes);
                
generator.Emit(OpCodes.Newobj, cInfo);
                
generator.Emit(OpCodes.Stloc, cloneVariable);

                foreach 
(FieldInfo field in typeof(T).GetFields(System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Public))
                {
                    
if (_globalCloneType == CloneType.DeepCloning)
                    {
                        
if (field.FieldType.IsValueType || field.FieldType == typeof(string))
                        {
                            generator.Emit(OpCodes.Ldloc, cloneVariable)
;
                            
generator.Emit(OpCodes.Ldarg_0);
                            
generator.Emit(OpCodes.Ldfld, field);
                            
generator.Emit(OpCodes.Stfld, field);
                        
}
                        
else if (field.FieldType.IsClass)
                        {
                            CopyReferenceType(generator, cloneVariable, field)
;
                        
}
                    }
                    
else
                    
{
                        
switch (GetCloneTypeForField(field))
                        {
                            
case CloneType.ShallowCloning:
                                {
                                    generator.Emit(OpCodes.Ldloc, cloneVariable)
;
                                    
generator.Emit(OpCodes.Ldarg_0);
                                    
generator.Emit(OpCodes.Ldfld, field);
                                    
generator.Emit(OpCodes.Stfld, field);
                                    break;
                                
}
                            
case CloneType.DeepCloning:
                                {
                                    
if (field.FieldType.IsValueType || field.FieldType == typeof(string))
                                    {
                                        generator.Emit(OpCodes.Ldloc, cloneVariable)
;
                                        
generator.Emit(OpCodes.Ldarg_0);
                                        
generator.Emit(OpCodes.Ldfld, field)
                                        
generator.Emit(OpCodes.Stfld, field);
                                    
}
                                    
else if (field.FieldType.IsClass)
                                        CopyReferenceType(generator, cloneVariable, field)
;
                                    break;
                                
}
                            
case CloneType.None:
                                {
                                    
// Do nothing here, field is not cloned.
                                
}
                                
break;
                        
}
                    }
                }
                generator.Emit(OpCodes.Ldloc_0)
;
                
generator.Emit(OpCodes.Ret);
                
myExec dymMethod.CreateDelegate(typeof(Func<T, T>));
                
_cachedILDeep.Add(typeof(T), myExec);
            
}
            
return ((Func<T, T>)myExec)(myObject);
        
}

        
/// <summary>
        /// Helper method to clone a reference type.
        /// This method clones IList and IEnumerables and other reference types (classes)
        /// Arrays are not yet supported (ex. string[])
        /// </summary>
        /// <param name="generator">IL generator to emit code to.</param>
        /// <param name="cloneVar">Local store wheren the clone object is located. (or child of)</param>
        /// <param name="field">Field definition of the reference type to clone.</param>
        
private static void CopyReferenceType(ILGenerator generator, LocalBuilder cloneVar, FieldInfo field)
        {
            
if (field.FieldType.IsSubclassOf(typeof(Delegate)))
            {
                
return;
            
}
            LocalBuilder lbTempVar 
generator.DeclareLocal(field.FieldType);

            if 
(field.FieldType.GetInterface("IEnumerable") != null && field.FieldType.GetInterface("IList") != null)
            {
                
if (field.FieldType.IsGenericType)
                {
                    Type argumentType 
field.FieldType.GetGenericArguments()[0];
                    
Type genericTypeEnum Type.GetType("System.Collections.Generic.IEnumerable`1[" + argumentType.FullName + "]");

                    
ConstructorInfo ci field.FieldType.GetConstructor(new Type[] { genericTypeEnum });
                    if 
(ci != null && GetCloneTypeForField(field) == CloneType.ShallowCloning)
                    {
                        generator.Emit(OpCodes.Ldarg_0)
;
                        
generator.Emit(OpCodes.Ldfld, field);
                        
generator.Emit(OpCodes.Newobj, ci);
                        
generator.Emit(OpCodes.Stloc, lbTempVar);
                        
generator.Emit(OpCodes.Ldloc, cloneVar);
                        
generator.Emit(OpCodes.Ldloc, lbTempVar);
                        
generator.Emit(OpCodes.Stfld, field);
                    
}
                    
else
                    
{
                        ci 
field.FieldType.GetConstructor(Type.EmptyTypes);
                        if 
(ci != null)
                        {
                            generator.Emit(OpCodes.Newobj, ci)
;
                            
generator.Emit(OpCodes.Stloc, lbTempVar);
                            
generator.Emit(OpCodes.Ldloc, cloneVar);
                            
generator.Emit(OpCodes.Ldloc, lbTempVar);
                            
generator.Emit(OpCodes.Stfld, field);
                            
CloneList(generator, field, argumentType, lbTempVar);
                        
}
                    }
                }
            }
            
else
            
{
                ConstructorInfo cInfo 
field.FieldType.GetConstructor(new Type[] { });
                
generator.Emit(OpCodes.Newobj, cInfo);
                
generator.Emit(OpCodes.Stloc, lbTempVar);
                
generator.Emit(OpCodes.Ldloc, cloneVar);
                
generator.Emit(OpCodes.Ldloc, lbTempVar);
                
generator.Emit(OpCodes.Stfld, field);
                foreach 
(FieldInfo fi in field.FieldType.GetFields(System.Reflection.BindingFlags.Instance
                    | System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Public))
                {
                    
if (fi.FieldType.IsValueType || fi.FieldType == typeof(string))
                    {
                        generator.Emit(OpCodes.Ldloc_1)
;
                        
generator.Emit(OpCodes.Ldarg_0);
                        
generator.Emit(OpCodes.Ldfld, field);
                        
generator.Emit(OpCodes.Ldfld, fi);
                        
generator.Emit(OpCodes.Stfld, fi);
                    
}
                }
            }
        }

        
/// <summary>
        /// Makes a deep copy of an IList of IEnumerable
        /// Creating new objects of the list and containing objects. (using default constructor)
        /// And by invoking the deepclone method defined above. (recursive)
        /// </summary>
        /// <param name="generator">IL generator to emit code to.</param>
        /// <param name="listField">Field definition of the reference type of the list to clone.</param>
        /// <param name="typeToClone">Base-type to clone (argument of List<T></param>
        /// <param name="cloneVar">Local store wheren the clone object is located. (or child of)</param>
        
private static void CloneList(ILGenerator generator, FieldInfo listField, Type typeToClone, LocalBuilder cloneVar)
        {
            Type genIEnumeratorTyp 
Type.GetType("System.Collections.Generic.IEnumerator`1[" + typeToClone.FullName + "]");
            
Type genIEnumeratorTypLocal Type.GetType(listField.FieldType.Namespace + "." + listField.FieldType.Name + "+Enumerator[[" + typeToClone.FullName + "]]");
            
LocalBuilder lbEnumObject generator.DeclareLocal(genIEnumeratorTyp);
            
LocalBuilder lbCheckStatement generator.DeclareLocal(typeof(bool));
            
Label checkOfWhile generator.DefineLabel();
            
Label startOfWhile generator.DefineLabel();
            
MethodInfo miEnumerator listField.FieldType.GetMethod("GetEnumerator");
            
generator.Emit(OpCodes.Ldarg_0);
            
generator.Emit(OpCodes.Ldfld, listField);
            
generator.Emit(OpCodes.Callvirt, miEnumerator);
            if 
(genIEnumeratorTypLocal != null)
            {
                generator.Emit(OpCodes.Box, genIEnumeratorTypLocal)
;
            
}
            generator.Emit(OpCodes.Stloc, lbEnumObject)
;
            
generator.Emit(OpCodes.Br_S, checkOfWhile);
            
generator.MarkLabel(startOfWhile);
            
generator.Emit(OpCodes.Nop);
            
generator.Emit(OpCodes.Ldloc, cloneVar);
            
generator.Emit(OpCodes.Ldloc, lbEnumObject);
            
MethodInfo miCurrent genIEnumeratorTyp.GetProperty("Current").GetGetMethod();
            
generator.Emit(OpCodes.Callvirt, miCurrent);
            
Type cloneHelper Type.GetType(typeof(CloneHelper<T>).Namespace + "." typeof(CloneHelper<T>).Name + "[" + miCurrent.ReturnType.FullName + "]");
            
MethodInfo miDeepClone cloneHelper.GetMethod("CloneObjectWithILDeep", BindingFlags.Static | BindingFlags.NonPublic);
            
generator.Emit(OpCodes.Call, miDeepClone);
            
MethodInfo miAdd listField.FieldType.GetMethod("Add");
            
generator.Emit(OpCodes.Callvirt, miAdd);
            
generator.Emit(OpCodes.Nop);
            
generator.MarkLabel(checkOfWhile);
            
generator.Emit(OpCodes.Nop);
            
generator.Emit(OpCodes.Ldloc, lbEnumObject);
            
MethodInfo miMoveNext = typeof(IEnumerator).GetMethod("MoveNext");
            
generator.Emit(OpCodes.Callvirt, miMoveNext);
            
generator.Emit(OpCodes.Stloc, lbCheckStatement);
            
generator.Emit(OpCodes.Ldloc, lbCheckStatement);
            
generator.Emit(OpCodes.Brtrue_S, startOfWhile);
        
}

        
/// <summary>
        /// Returns the type of cloning to apply on a certain field when in custom mode.
        /// Otherwise the main cloning method is returned.
        /// You can invoke custom mode by invoking the method Clone(T obj)
        /// </summary>
        /// <param name="field">Field to examine</param>
        /// <returns>Type of cloning to use for this field.</returns>
        
private static CloneType GetCloneTypeForField(FieldInfo field)
        {
            
object[] attributes field.GetCustomAttributes(typeof(CloneAttribute), true);
            if 
(attributes == null || attributes.Length == 0)
            {
                
if (!_globalCloneType.HasValue)
                    
return CloneType.ShallowCloning;
                else
                    return 
_globalCloneType.Value;
            
}
            
return (attributes[0as CloneAttribute).CloneType;
        
}

        
#endregion
    
}
}


If you have any remarks, please notify me, or when you can use this code, please notify me, you may use it as you whish :)

Regards,

F.

woensdag 18 juni 2008

Object Deep Cloning using IL in C# - version 1.0

Howdy,

If you've read the rest of my blog, then you'll have noticed that I have another post that addresses cloning of objects using IL (Intermediate Language)
If you haven't, then you can find the post 'Object cloning using IL in C#'.

The reason I published this post is because the code in my previous post (mentioned above) makes a shallow copy of an object. It copies only references of classes, value-types are off course copied by value. So if you have a class with only value types or strings (immutable), then the shallow copy will act as a deep copy (sort of).

Now, imagine that you have (like in most projects) objects that contain other objects, like a person object that has one legal address and different optional addresses. Then you'll probably create a Person class and an Address class,
in the person class you'll make a property which has the type 'Address' and a property that's a list of Addresses.

If you would take a shallow clone, and you change an address in the clone, then it will also be changed in the original person.address object.

So, i've created a solution that clones the 'Person' object in this case, with the addresses, but it makes new instances of each address.

The only constraints however are that the 'to-clone' objects MUST have a default constructor for instantiating the class, if the system can make an instance, then the private fields will automatically be copied.

Like I mentioned in my previous post, you'll only loose performance the first time, because of all the lookups and the generating of the IL-code.
After that, the compiled code will be executed.
I tried to optimize the IL code as much as possible; maybe it can be optimized even more.

For example:

If you would clone the address of a person in normal C# code then you can do:

public Person Clone(Person p) 

    Person clone 
= new Person()
    
clone.Address = new Address()
    
// normally you should check on null, 
    // but let's assume that it's never null. 
    
clone.Address.ID p.Address.ID;
    return 
clone;
}

or you can do:

public Person Clone(Person p) 

    Person clone 
= new Person();
    
Address a = new Address();
    
clone.Address a;
    
// normally you should check on null, 
    // but let's assume that it's never null. 
    
a.ID p.Address.ID;
    return 
clone;
}

The performance of the second option is 'higher'.

You could ask, why?
Well, if you would disassemble these two code samples, then you'll notice that the second code needs fewer instructions.

First sample decompiled:

.locals init (
     [0] class Cloning.Person clone)
L_0000: nop
L_0001: newobj instance void Cloning.Person::.ctor()
L_0006: stloc.0
L_0007: ldloc.0
L_0008: newobj instance void Cloning.Address::.ctor()
L_000d: callvirt instance void Cloning.Person::set_Address(class Cloning.Address)
L_0012: nop
L_0013: ldloc.0
L_0014: callvirt instance class Cloning.Address Cloning.Person::get_Address()
L_0019: ldarg.1
L_001a: callvirt instance class Cloning.Address Cloning.Person::get_Address()
L_001f: callvirt instance int32 Cloning.Address::get_AddressID()
L_0024: callvirt instance void Cloning.Address::set_AddressID(int32)

L_0029: nop
L_002a: ldloc.0
L_002b: ret


Second sample decompiled:

.locals init (
     [0] class Cloning.Person clone,
     [1] class Cloning.Address a)
L_0000: nop
L_0001: newobj instance void Cloning.Person::.ctor()
L_0006: stloc.0
L_0007: newobj instance void Cloning.Address::.ctor()
L_000c: stloc.1
L_000d: ldloc.0
L_000e: ldloc.1
L_000f: callvirt instance void Cloning.Person::set_Address(class Cloning.Address)
L_0014: nop
L_0015: ldloc.1
L_0016: ldarg.1
L_0017: callvirt instance class Cloning.Address Cloning.Person::get_Address()
L_001c: callvirt instance int32 Cloning.Address::get_AddressID()
L_0021: callvirt instance void Cloning.Address::set_AddressID(int32)
L_0026: nop
L_0027: ldloc.0
L_0028: ret


As you see in the code of the first sample, you need 5 callvirt's, and it's more IL code,
in the second sample you have 4 callvirt's and less code, because we have a kind of shortcut to the address object
from the local store [1].

When you create an object, you push it onto the stack and then you store it in a local store (local variable).
If you use a.ID then you need to lookup the ID from the local store reference,
else if you use clone.Address.ID then you need to lookup the Address reference from the person object (also in a store), and then lookup the ID reference from the retrieved Address reference, and then you can do an action with it.
(lookat line L_0013 to L_0024 of first sample and line L_0015 to L_0021 in second sample)

In that manner I tried to optimize all the calls in the IL code, so that I can do the required job in as less instructions as possible.

Here is the result:

Now let's take a look at the code:

Our revisited Person class with an Address class:


using System;
using 
System.Collections.Generic;
using 
System.Text;

namespace 
Cloning
{
    
public class Person
    {
        
private int _id;
        private string 
_name;
        private string 
_firstName;
        private string 
_field1, _field2, _field3;

        public 
Person()
        {
            
this.Addresses = new List<Address>();
        
}

        
public int ID
        {
            
get return _id}
            
set { _id = value; }
        }

        
public string Name
        {
            
get return _name}
            
set { _name = value; }
        }

        
public string FirstName
        {
            
get return _firstName}
            
set { _firstName = value; }
        }

        
private Address _address;

        public 
Address Address
        {
            
get return _address}
            
set { _address = value; }
        }

        
private List<Address> _addresses;

        public 
List<Address> Addresses
        {
            
get return _addresses}
            
set { _addresses = value; }
        }

    }

    
public class Address
    {
        
public Address()
        {
            
this.AddressID -1;
        
}

        
public Address(int aid)
        {
            
this.AddressID aid;
        
}

        
private int _addressID;

        public int 
AddressID
        {
            
get return _addressID}
            
set { _addressID = value; }
        }

        
private string _street;

        public string 
Street
        {
            
get return _street}
            
set { _street = value; }
        }

        
private string _city;

        public string 
City
        {
            
get return _city}
            
set { _city = value; }
        }

    }
}


The Cloning class used for testing:

using System;
using 
System.Collections.Generic;
using 
System.Text;
using 
System.Reflection;
using 
System.Reflection.Emit;
using 
System.Threading;
using 
System.Collections;

namespace 
Cloning
{
    
/// <summary>    
    /// Delegate handler that's used to compile the IL to.    
    /// (This delegate is standard in .net 3.5)    
    /// </summary>    
    /// <typeparam name="T1">Parameter Type</typeparam>    
    /// <typeparam name="TResult">Return Type</typeparam>    
    /// <param name="arg1">Argument</param>    
    /// <returns>Result</returns>    
    
public delegate TResult Func<T1, TResult>(T1 arg1);

    public class 
Cloning
    {
        
/// <summary>    
        /// This dictionary caches the delegates for each 'to-clone' type.    
        /// </summary>    
        
private static Dictionary<Type, Delegate> _cachedIL = new Dictionary<Type, Delegate>();
        private static 
Dictionary<Type, Delegate> _cachedILDeep = new Dictionary<Type, Delegate>();
        private 
LocalBuilder _lbfTemp;

        
/// <summary>    
        /// Clone one person object with reflection    
        /// </summary>    
        /// <param name="p">Person to clone</param>    
        /// <returns>Cloned person</returns>    
        
public static Person CloneObjectWithReflection(Person p)
        {
            FieldInfo[] fis 
p.GetType().GetFields(System.Reflection.BindingFlags.Instance |
                System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.NonPublic)
;
            
Person newPerson = new Person();
            foreach 
(FieldInfo fi in fis)
            {
                fi.SetValue(newPerson, fi.GetValue(p))
;
            
}
            
return newPerson;
        
}

        
/// <summary>    
        /// Clone a person object by manually typing the copy statements.    
        /// </summary>    
        /// <param name="p">Object to clone</param>    
        /// <returns>Cloned object</returns>    
        
public static Person CloneNormal(Person p)
        {
            Person newPerson 
= new Person();
            
newPerson.ID p.ID;
            
newPerson.Name p.Name;
            
newPerson.FirstName p.FirstName;
            
newPerson.Address = new Address();
            
newPerson.Address.AddressID p.Address.AddressID;
            
newPerson.Address.City p.Address.City;
            
newPerson.Address.Street p.Address.Street;
            if
(newPerson.Addresses!=null)
            {
                newPerson.Addresses 
= new List<Address>();
                foreach 
(Address a in newPerson.Addresses)
                {
                    newPerson.Addresses.Add(a)
;
                
}
            }
            
return newPerson;
        
}

        
/// <summary>    
        /// Generic cloning method that clones an object using IL.    
        /// Only the first call of a certain type will hold back performance.    
        /// After the first call, the compiled IL is executed.    
        /// </summary>    
        /// <typeparam name="T">Type of object to clone</typeparam>    
        /// <param name="myObject">Object to clone</param>    
        /// <returns>Cloned object</returns>    
        
public static T CloneObjectWithILShallow<T>(T myObject)
        {
            Delegate myExec 
= null;
            if 
(!_cachedIL.TryGetValue(typeof(T), out myExec))
            {
                
// Create ILGenerator (both DM declarations work)
                // DynamicMethod dymMethod = new DynamicMethod("DoClone", typeof(T), 
                //      new Type[] { typeof(T) }, true);
                
DynamicMethod dymMethod = new DynamicMethod("DoClone"typeof(T), 
                    
new Type[] { typeof(T) }, Assembly.GetExecutingAssembly().ManifestModule, true);
                
ConstructorInfo cInfo myObject.GetType().GetConstructor(new Type[] { });
                
ILGenerator generator dymMethod.GetILGenerator();
                
LocalBuilder lbf generator.DeclareLocal(typeof(T));
                
generator.Emit(OpCodes.Newobj, cInfo);
                
generator.Emit(OpCodes.Stloc_0);
                foreach 
(FieldInfo field in myObject.GetType().GetFields(
                        System.Reflection.BindingFlags.Instance 
                        | System.Reflection.BindingFlags.NonPublic 
                        | System.Reflection.BindingFlags.Public))
                {
                    generator.Emit(OpCodes.Ldloc_0)
;
                    
generator.Emit(OpCodes.Ldarg_0);
                    
generator.Emit(OpCodes.Ldfld, field);
                    
generator.Emit(OpCodes.Stfld, field);
                
}
                generator.Emit(OpCodes.Ldloc_0)
;
                
generator.Emit(OpCodes.Ret);
                
myExec dymMethod.CreateDelegate(typeof(Func<T, T>));
                
_cachedIL.Add(typeof(T), myExec);
            
}
            
return ((Func<T, T>)myExec)(myObject);
        
}

        
public T CloneObjectWithILDeep<T>(T myObject)
        {
            Delegate myExec 
= null;
            if 
(!_cachedILDeep.TryGetValue(typeof(T), out myExec))
            {
                
// Create ILGenerator (both DM declarations work)
                // DynamicMethod dymMethod = new DynamicMethod("DoClone", typeof(T), 
                //      new Type[] { typeof(T) }, true);
                
DynamicMethod dymMethod = new DynamicMethod("DoClone"typeof(T), 
                    
new Type[] { typeof(T) }, Assembly.GetExecutingAssembly().ManifestModule, true);
                
ConstructorInfo cInfo myObject.GetType().GetConstructor(new Type[] { });
                
ILGenerator generator dymMethod.GetILGenerator();
                
LocalBuilder lbf generator.DeclareLocal(typeof(T));
                
generator.Emit(OpCodes.Newobj, cInfo);
                
generator.Emit(OpCodes.Stloc_0);

                foreach 
(FieldInfo field in typeof(T).GetFields(System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Public))
                {
                    
if (field.FieldType.IsValueType || field.FieldType == typeof(string))
                        CopyValueType(generator, field)
;
                    else if 
(field.FieldType.IsClass)
                        CopyReferenceType(generator, field)
;
                
}
                generator.Emit(OpCodes.Ldloc_0)
;
                
generator.Emit(OpCodes.Ret);
                
myExec dymMethod.CreateDelegate(typeof(Func<T, T>));
                
_cachedILDeep.Add(typeof(T), myExec);
            
}
            
return ((Func<T, T>)myExec)(myObject);
        
}

        
private void CreateNewTempObject(ILGenerator generator, Type type)
        {
            ConstructorInfo cInfo 
type.GetConstructor(new Type[] { });
            
generator.Emit(OpCodes.Newobj, cInfo);
            
generator.Emit(OpCodes.Stloc, _lbfTemp);
        
}

        
private void CopyValueType(ILGenerator generator, FieldInfo field)
        {
            generator.Emit(OpCodes.Ldloc_0)
;
            
generator.Emit(OpCodes.Ldarg_0);
            
generator.Emit(OpCodes.Ldfld, field);
            
generator.Emit(OpCodes.Stfld, field);
        
}

        
private void CopyValueTypeTemp(ILGenerator generator, FieldInfo fieldParent, FieldInfo fieldDetail)
        {
            generator.Emit(OpCodes.Ldloc_1)
;
            
generator.Emit(OpCodes.Ldarg_0);
            
generator.Emit(OpCodes.Ldfld, fieldParent);
            
generator.Emit(OpCodes.Ldfld, fieldDetail);
            
generator.Emit(OpCodes.Stfld, fieldDetail);
        
}

        
private void PlaceNewTempObjInClone(ILGenerator generator, FieldInfo field)
        {
            
// Get object from custom location and store it in right field of location 0
            
generator.Emit(OpCodes.Ldloc_0);
            
generator.Emit(OpCodes.Ldloc, _lbfTemp);
            
generator.Emit(OpCodes.Stfld, field);
        
}

        
private void CopyReferenceType(ILGenerator generator, FieldInfo field)
        {
            
// We have a reference type.
            
_lbfTemp generator.DeclareLocal(field.FieldType);
            if 
(field.FieldType.GetInterface("IEnumerable") != null)
            {
                
// We have a list type (generic).
                
if (field.FieldType.IsGenericType)
                {
                    
// Get argument of list type
                    
Type argType field.FieldType.GetGenericArguments()[0];
                    
// Check that it has a constructor that accepts another IEnumerable.
                    
Type genericType Type.GetType("System.Collections.Generic.IEnumerable`1[" 
                            
+ argType.FullName + "]");
                    
                    
ConstructorInfo ci field.FieldType.GetConstructor(new Type[] { genericType });
                    if 
(ci != null)
                    {
                        
// It has! (Like the List<> class)
                        
generator.Emit(OpCodes.Ldarg_0);
                        
generator.Emit(OpCodes.Ldfld, field);
                        
generator.Emit(OpCodes.Newobj, ci);
                        
generator.Emit(OpCodes.Stloc, _lbfTemp);
                        
PlaceNewTempObjInClone(generator, field);
                    
}
                }
            }
            
else
            
{
                CreateNewTempObject(generator, field.FieldType)
;
                
PlaceNewTempObjInClone(generator, field);
                foreach 
(FieldInfo fi in field.FieldType.GetFields(System.Reflection.BindingFlags.Instance
                    | System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Public))
                {
                    
if (fi.FieldType.IsValueType || fi.FieldType == typeof(string))
                        CopyValueTypeTemp(generator, field, fi)
;
                    else if 
(fi.FieldType.IsClass)
                        CopyReferenceType(generator, fi)
;
                
}
            }
        }

    }
}


The Program class with the test code in it.

namespace Cloning
{
    
class Program
    {
        
static void Main(string[] args)
        {
            
// Do some cloning tests...
            
Cloning.TestCloning tc = new Cloning.TestCloning();
            
tc.DoTest();
        
}
    }
}


Explanation:

As you see in the code, the model is a little bit expanded.
But if you shuffle a bit with the code, then you can put it in 1 method again,
I only did this to improve readability, so that you can understand what's happening, and how the code is generated.

I also left out the comments from the IL parts, if you want to know what those statements mean, then look at my other post regarding IL cloning (Object cloning using IL in C#), it's explained there.

Method list of cloning class with a short description:


  • Person CloneObjectWithReflection(Person p)
    This method clones an object using reflection (very slow).

  • Person CloneNormal(Person p)
    This method clones the person class manually.
    I think a lot of people will do/user this method :)


  • T CloneObjectWithILShallow(T myObject)
    Make a shallow copy of an object using IL in a generic method.

  • T CloneObjectWithILDeep(T myObject)
    Make a deep copy of an object using IL in a generic method.

  • void CreateNewTempObject(ILGenerator generator, Type type)
    Generate IL-code to create a new object of a certain type and store it in a local store. (local variable)

  • void CopyValueType(ILGenerator generator, FieldInfo field)
    Generate IL-code to copy the values of a value type to the clone.

  • void CopyValueTypeTemp(ILGenerator generator, FieldInfo fieldParent, FieldInfo fieldDetail)
    Generate IL-code to copy the values of a value type from a store location to the destination address in the clone.

  • void PlaceNewTempObjInClone(ILGenerator generator, FieldInfo field)
    Generate IL-code to reference an object from a store to an address in the clone.

  • void CopyReferenceType(ILGenerator generator, FieldInfo field)
    Generate IL-code to copy the values of a reference type (class) to the clone. Instantiate new objects if needed.


I didn't test LINQ support, i guess that an extra 'exception' has to be added for the IQueryable or something like that, and also Arrays should be added, but that's for in version 2.0 or something like that ;), you can see how the system works and expand it to your needs...

I hope that this post and piece of code is useful for some people, if so, please let me know through a comment, thanks ;)

Regards,
F.