Simple Metadata Driven Linq To Sql Type Generator

As part of a larger body of work, I implement a “Dynamic” Linq to Sql Type generator that was based on metadata. A number of people have asked about the source code for this generator so I have decided to publish this portion of the code as a stand alone component.

DumpingCoreMemory.Data.Linq.Generation assembly

• LinqToSqlTypeGenerator: This class generates the types based on metadata provided at runtime.
• EntityDescriptor: Instances of this class define entities that the LinqToSqlTypeGenerator class will generate concrete runtime types for
• EntityPropertyDescriptor: Instances of this class define properties of the concrete runtime types that the LinqToSqlTypeGenerator will generate

How To Use the Type Generator

Included in the attached source code zip file is a contrived example, which briefly demonstrates how the type generator can be used. The example assumes a database which contains a table called TableA which contains three columns Id, ColumnA and ColumnB. In the example code this table is to be mapped to a Type called EntityA with three properties Identity, PropertyA and PropertyB. The type EntityA is to be derived from an interface IEntity. This is to enable demonstration of one of the ways a generated Type can be manipulated. The code for the mapping is as follows:

private static List<EntityDescriptor> GenerateEntityDescriptors()
{
 var entityDescriptors = new List<EntityDescriptor>
 {
  new EntityDescriptor
   {
    BaseType = typeof (IEntity),
    EntityName = "EntityA",
    TableName = "TableA",
    EntityProperties = new List<EntityPropertyDescriptor>
       {
        new EntityPropertyDescriptor
         {
          ColumnName = "Id",
          Name = "Identity",
          Type = typeof (Int32),
          IsIdentity = true
         },
        new EntityPropertyDescriptor
         {
          ColumnName = "ColumnA",
          Name = "PropertyA",
          Type = typeof (String)
         },
        new EntityPropertyDescriptor
         {
          ColumnName = "ColumnB",
          Name = "PropertyB",
          Type = typeof (Int32)
         }
       },
   }
 };
 return entityDescriptors;
}

The generator is invoked which generates the types, in the example I pass in an assembly file name, this is not necessary but for the purpose of the example it is useful as we can have a look at the generated types.

var entityDescriptors = GenerateEntityDescriptors();
var generator = new LinqToSqlTypeGenerator();
IList<Type> generatedTypes;

#warning Ensure that the process that invokes this method has rights to write to the disk
// Passing the assemblyFileName parameter is not necessary. 
// The generator will write the assembly to the specified filename.
// This allows us to reuse the generated types or just take a look at them with ILSpy
generator.GenerateTypes("LinqToSqlGeneratedTypes", "LinqToSqlGeneratedTypesAssembly.dll",
      entityDescriptors,
      out generatedTypes);

Using ILSpy we can inspect the generated code that was written to the assembly LinqToSqlGeneratedTypesAssembly.dll.



As one would expect we see a simple object with mapping attributes to the underlying database table called TableA. The next step is to activate and use the type for this we have a number of options:

1. Statically reference the type, this is not shown in the test application as it is trivial to reference the assembly and instantiate its types. It also kind of defeats the purpose of generating types dynamically
2. Cast to a known base type, in the test application the base type of the generated Type was specified as IEntity. The instance of the type is cast to an IEntity type so that properties can be accessed, the EntityDescriptor for the type specified a matching property for the IEntity interfaces specified Identity property
3. Use a dynamic type to contain the type. This allows bypass of static type checking to allow access to properties of a type at runtime
4. Use reflection to get or set properties of the underlying instance, using the name of the property for access

The following code snippet shows instantiation of an instance of the generated type and access by options 2,3 and 4.

object instance = Activator.CreateInstance(generatedTypes[0]);

var instanceAsIEntity = (IEntity) instance;
dynamic o = instance;

Console.WriteLine("Accessed as interface: {0}", instanceAsIEntity.Identity);
Console.WriteLine("Accessed as dynamic: {0}", o.Identity);
Console.WriteLine("Accessed by reflection: {0}", GetPropertyValueByReflection(instance, "Identity"));

Persisting instances of the types to a database is straight forward. The method PersistEntity in the test application shows how to insert an instance to a database.

private static void PersistEntity(object instance)
{
 using (var sqlConnection = new SqlConnection("Server=localhost;Database=Test;Trusted_Connection=True;"))
 {
  using (var context = new DataContext(sqlConnection))
  {
   ITable table = context.GetTable(instance.GetType());

   table.InsertOnSubmit(instance);

   context.SubmitChanges();
  }    
 }
}

For simplicity the code sets the properties via the dynamic object reference. In a real world example the instance may be bound to a UI control prior to CRUD operations being performed.

Limitations

There is no abstraction of the underlying types from the data store. So if the underlying column is of type nvarchar then the property of the generated entity must be a String. The generator does not support type coercion or conversion which can get messy. The WCF service that allows the definition of the types metadata ensures that this cannot occur

There is no abstraction from the underlying data stores foreign key relationships. So if a type is mapped to a table that requires a foreign key value to be specified which is not exposed via the generated type then new rows cannot be inserted

There is no support for nested relationships. i.e a Thing contains a list of Parts. For the use cases of the overall project this is not a key requirement and can be added with more elaborate metadata and a change to the type generator

Configurable OData Service - Part 3 – Custom Data Service Provider

In this post I am going to delve into the implementation of the Custom Data Service Provider that will unify the Type metadata, Dynamic Type generation and WCF OData services.
For brevity of this post I am not going to cover all of the classes required for the implementation of the Custom Data Service Provider just those that are key to this implementation. Implementing a Custom Data Service Provider is actually  a relatively straight forward task once you get your head around the interfaces involved. The details behind the steps required in Creating a Data Service Provider has been covered in a series of excellent posts by Alex D James on his blog. I have based my implementation on some of the concepts expressed in those posts.

Class Model Overview


There are a number of classes required to expose the underlying custom data source as a service. I will discuss each one in turn to give a simple overview.

DynamicDataService is an abstract base class that derives from DataService<DynamicDataContext> and the IServiceProvider interface. The purpose of this class is to and allow the WCF OData service infrastructure to instantiate the various custom data service provider types via the object GetService(Type serviceType) method of the IServiceProvider interface.

DynamicDataContext is the class that represents the underlying data source that will be exposed via an OData service. It derives from DataContext and the IDynamicDataContext interface.

The IDynamicDataContext interface is a custom interface to abstract the implementation so that the classes of the custom data provider can be unit tested. It derives from DataContext because it is a LINQ to SQL based data source, I will discuss this class in more detail later in this post.

DomainModelService inherits from the abstract base class DynamicDataService. It implements the two abstract methods from the base class:

public abstract IDynamicDataServiceMetadataProvider GetMetadataProvider();


public abstract IDataServiceQueryProvider GetQueryProvider(IDataServiceMetadataProvider metadata);

The DomainModelService generates types using the Data Type Generator to generate the Entity types based on metadata that is provided to the service when it is instantiated. The metadata is also used to create the DynamicDataServiceMetadataProvider instance, the DomainModelService class will be discussed in more detail later in this post.

DynamicDataServiceMetadataProvider derives from the IDynamicDataServiceMetadataProvider interface, the IDynamicDataServiceMetadataProvider interface is derived from the IDataServiceMetadataProvider interface, its purpose is to simply facilitate unit testing. The DynamicDataServiceMetadataProvider class is simply a container for the metadata for the OData service.

DynamicDataServiceQueryProvider derives from the IDataServiceQueryProvider interface and provides the implementation of the IQueryable GetQueryRootForResourceSet(ResourceSet resourceSet)

DynamicDataServiceUpdateProvider derives from IDataServiceUpdateProvider interface and provides the mechanisms by which the OData service framework can create update and delete the exposed data type entities.

In summary the key classes of the implementation are the DomainModelService and the DynamicDataContext. The other classes in reality are pretty much boiler plate and depend mainly on the implementation of the DynamicDataContext.

DomainModelService

The DomainModelService is the primary touch point for exposing the dynamically defined entities types via the WCF OData service framework.

[ServiceBehavior(IncludeExceptionDetailInFaults = true)]
public class DomainModelService : DynamicDataService
{
    private const String AssemblyName = "DomainModel";

    private Dictionary<EntityDescriptor, Type> _entityToTypeMap;
    private Dictionary<ResourceSet, Type> _resourceSetToTypeMap;
    private IDynamicTypeGenerator _dynamicTypeGenerator;
   
    /// <summary>
    /// Initializes a new instance of the <see cref="DomainModelService"/> class.
    /// </summary>
    /// <param name="domainDescriptor">The <see="DomainModelDescriptor"> that this service has to represent.</param>
    public DomainModelService(DomainModelDescriptor domainDescriptor) : this(domainDescriptor, DynamicTypeGenerator.Instance)
    {
    }

    /// <summary>
    /// Initializes a new instance of the <see cref="DomainModelService"/> class.
    /// </summary>
    /// <param name="domainDescriptor">The <see="DomainModelDescriptor"> that this service has to represent.</param>
    /// <param name="dynamicTypeGenerator">The type genrator to use for generating the types</param>
    public DomainModelService(DomainModelDescriptor domainDescriptor, IDynamicTypeGenerator dynamicTypeGenerator)
    {
        _dynamicTypeGenerator = dynamicTypeGenerator;
        _entityToTypeMap = GenerateTypes(domainDescriptor);
    }

    /// <summary>
    /// Initializes the configuration for the service instance
    /// </summary>
    /// <param name="config">Configuration settings for the service.</param>
    public static void InitializeService(DataServiceConfiguration config)
    {
        config.SetEntitySetAccessRule("*", EntitySetRights.All);
        config.UseVerboseErrors = true;

        config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
        config.DataServiceBehavior.AcceptProjectionRequests = true;
    }

    /// <summary>
    /// Gets instance of an <see cref="IDynamicDataServiceMetadataProvider"/> derived type for this odata service.
    /// </summary>
    /// <returns>An instance of an <see cref="DynamicDataServiceMetadataProvider"/></returns>
    public override IDynamicDataServiceMetadataProvider GetMetadataProvider()
    {
        DynamicDataServiceMetadataProvider metaDataProvider = new DynamicDataServiceMetadataProvider(AssemblyName, "DomainModelService");
        _resourceSetToTypeMap = new Dictionary<ResourceSet, Type>();

        foreach (var entity in _entityToTypeMap.Keys)
        {
            Type t = _entityToTypeMap[entity];
            var resourceType = new ResourceType(t, ResourceTypeKind.EntityType, null, AssemblyName, entity.EntityName, false);

            foreach (var property in entity.EntityProperties)
            {
                ResourcePropertyKind kind = ResourcePropertyKind.Primitive;

                if (property.IsIdentity)
                {
                    kind |= ResourcePropertyKind.Key;
                }
               
                var resourceProperty = new ResourceProperty(property.Name,
                    kind, ResourceType.GetPrimitiveResourceType(property.Type));

                resourceType.AddProperty(resourceProperty);
            }

            metaDataProvider.AddResourceType(resourceType);
            ResourceSet resourceSet = new ResourceSet(String.Format("{0}s",entity.EntityName), resourceType);
            metaDataProvider.AddResourceSet(resourceSet);
            _resourceSetToTypeMap.Add(resourceSet, t);
        }

        return metaDataProvider;
    }

    /// <summary>
    /// Gets instance of an <see cref="IDataServiceMetadataProvider"/> derived type for this odata service.
    /// </summary>
    /// <param name="metadataProvider">The <see cref="IDataServiceMetadataProvider"/></param>
    /// <returns>An instance of an <see cref="DynamicDataServiceQueryProvider"/></returns>
    public override IDataServiceQueryProvider GetQueryProvider(IDataServiceMetadataProvider metadataProvider)
    {
        return new DynamicDataServiceQueryProvider(metadataProvider);
    }

    /// <summary>
    /// Create the underlying data source for this odata service
    /// </summary>
    /// <returns>An instance of an <see cref="DynamicDataContext"/> class</returns>
    protected override DynamicDataContext CreateDataSource()
    {
        return new DynamicDataContext(_resourceSetToTypeMap,
            ConfigurationManager.ConnectionStrings[DomainModelServiceSettings.Current.TargetStoreConnectionStringName].ConnectionString);
    }

    /// <summary>
    /// Generates types dynamically that represent the <see cref="EntityDescriptor"/> entities of the <paramref name="domainDescriptor"/>
    /// </summary>
    /// <param name="domainDescriptor">The <see cref="DomainModelDescriptor"/> that the types are to be generated for.</param>
    /// <returns>A <see cref="Dictionary"/> that contains mapping between an <see cref="EntityDescriptor"/> and the generated type</returns>
    private Dictionary<EntityDescriptor, Type> GenerateTypes(DomainModelDescriptor domainDescriptor)
    {
        Dictionary<EntityDescriptor, Type> map = new Dictionary<EntityDescriptor, Type>();
        AssemblyBuilder assemblyBuilder = null;
        ModuleBuilder moduleBuilder = null;
       
        _dynamicTypeGenerator.CreateDynamicAssembly(AssemblyName, out assemblyBuilder, out moduleBuilder);

        foreach (var entity in domainDescriptor.Entities)
        {
            map.Add(entity, _dynamicTypeGenerator.CreateLinqToSqlType(AssemblyName, assemblyBuilder, moduleBuilder, entity));
        }

        if (DomainModelServiceSettings.Current.StoreGeneratedAssemblies)
        {
            assemblyBuilder.Save(String.Format("{1}.dll", domainDescriptor.Version, AssemblyName));
        }

        return map;
    }
}

The service requires an instance of a DomainModelDescriptor class to be provided when the service is constructed. The DomainModelDescriptor defines the domain model entities and their properties that are to be exposed as an OData service. The constructor generates a set of types that map to the EntityDescriptor types that are contained within the DomainModelDescriptor instance.



When the OData framework invokes the on the object GetService(Type serviceType) method of the IServiceProvider interface this in turn invokes the two overridden methods GetMetadataProvider() and  GetQueryProvider(IDataServiceMetadataProvider metadata) methods.

Of the two methods only the GetMetadataProvider() is of interest. This method creates an instance of a DynamicDataServiceMetadataProvider type and populates it with the required ResourceSet, ResourceTypes, ResourceProperty values that will enable the OData framework to describe the underlying data source structure to an OData client.

Another key method is the CreateDataSource() method that creates the underlying data source for the OData service. In this implementation the data source is an instance of a DynamicDataContext. The DynamicDataServiceQueryProvider and DynamicDataServiceUpdateProvider types both leverage the functionality of the DynamicDataContext provider to perform their function.

DynamicDataContext

The DynamicDataContext class is a simple Linq To Sql DataContext derived class that is responsible for marshalling calls between the Custom Data Service Provider components and the underlying data store.

public class DynamicDataContext : DataContext, IDynamicDataContext
{
    private Dictionary<ResourceSet, Type> _resourceSetToTypeMap;


    /// <summary>
    /// Initializes a new instance of the <see cref="DynamicDataContext"/> class
    /// </summary>
    /// <param name="resourceSetToTypeMap">A <see cref="Dictionary"/> that contains a mapping from a <see cref="ResourceSet"/> to an underlying type</param>
    /// <param name="connection">The connection string for this <see cref="DataContext"/> derived type to use</param>
    public DynamicDataContext(Dictionary<ResourceSet, Type> resourceSetToTypeMap, String connection) : base(connection) 
    {
        _resourceSetToTypeMap = resourceSetToTypeMap;
    }


    /// <summary>
    /// Gets an instance of an <see cref="IQueryable"/> for the <paramref name="resourceSet"/>
    /// </summary>
    /// <param name="resourceSet">The resource set to retrieve the <see cref="IQueryable"/> instance for</param>
    /// <returns>
    /// An instance of an <see cref="IQueryable"/> type
    /// </returns>
    public IQueryable GetQueryableForResourceSet(ResourceSet resourceSet)
    {
        Type t = _resourceSetToTypeMap[resourceSet];


        return GetTableForType(t);
    }


    /// <summary>
    /// Add a resource to the underlying store
    /// </summary>
    /// <param name="resource">The resource to add to the store</param>
    public void AddResource(object resource)
    {
        ITable table = GetTableForType(resource.GetType());
        IEntity baseType = (IEntity) resource;


        if (baseType.Identifier == Guid.Empty)
        {
            baseType.Identifier = Guid.NewGuid();
        }


        table.Attach(resource);
    }


    /// <summary>
    /// Remove a resource from the underlying store
    /// </summary>
    /// <param name="resource">The resource to remove</param>
    public void RemoveResource(object resource)
    {
        ITable table = GetTableForType(resource.GetType());


        table.DeleteOnSubmit(resource);
    }


    /// <summary>
    /// Commit changes to the underlying storeB
    /// </summary>
    public void CommitChanges()
    {
        this.SubmitChanges();
    }


    private ITable GetTableForType(Type type)
    {
        MethodInfo method = typeof(DataContext).GetMethod("GetTable", Type.EmptyTypes);
        MethodInfo generic = method.MakeGenericMethod(type);
        return (ITable) generic.Invoke(this, null);
    }
}

As you can see there is nothing particularly complex about the DynamicDataContext. The GetQueryableForResourceSet simply maps ResourceSet to a type to provide an IQueryable object for querying data, which the OData framework will expose. When modifying or deleting entities in the data store the GetTableForType method uses some simple reflection to get the underlying ITable derived object to which changes are submitted.

Now we have a OData service that can be dynamically configured based on metadata. In the next post, Part 4 – WCF Service Host, I will cover the activation and instantiation of a DomainModelService.

Configurable OData Service - Part 2 – Data Type Generator

In the this post I am going to cover some of the implementation details of the Data Type generation component of the configurable OData service.
The primary purpose of this component is to dynamically generate data types based on some metadata. The metadata describes the properties of the types and how the type maps to the underlying data store. The metadata is defined by an end user via a WPF application and is stored in a database via an intermediary WCF service that manages the metadata.
Metadata 

Reflection Type Generation

Given metadata for a Type to be generated at runtime, take for example the following metadata definition:

EntityDescriptor entityMetadata = new EntityDescriptor()
{
    BaseType = typeof(IEntity),
    EntityName = "GeneratedEntity",
    EntityProperties = new List<EntityPropertyDescriptor>()
    {
        new EntityPropertyDescriptor()
        {
            ColumnName = "Id",
            IsIdentity = true,
            IsNullable = false,
            Name = "Identifier",
            Type = typeof(Guid)
        },
        new EntityPropertyDescriptor()
        {
            ColumnName = "Comment",
            IsIdentity = false,
            IsNullable = false,
            Name = "Comment",
            Type = typeof(String)
        },

TableName = "TestTable"
    }
};

The first step in generating the dynamic types is to create an instance of an AssemblyBuilder and a ModuleBuilder into which the Type, based on metadata will be emitted.

ModuleBuilder moduleBuilder;
AssemblyBuilder assemblyBuilder;


DynamicTypeGenerator.Instance.CreateDynamicAssembly("TestAssembly", out assemblyBuilder, out moduleBuilder);

Now we have an AssemblyBuilder instance and an instance of a ModuleBuilder which will be used in subsequent calls to the DynamicTypeGenerator instance to emit types into. For the purpose of this example only one Type is to be emitted into the Assembly.

Given the above entity metadata a call to the CreateLinqToSqlType method of the DynamicTypeGenerator instance passing the type metadata generates the underlying Type for the metadata description.

DynamicTypeGenerator.Instance.CreateLinqToSqlType("TestAssembly", assemblyBuilder, moduleBuilder, entityMetadata);

The CreateLinqToSqlType as expected generates a simple LinqToSql data type entity that has attributes applied to its properties to support the mapping of the Type to an underlying database. There is not anything extremely complicated about this method. It uses a number of the ReflectionEmit constructs to generate the Types and the ILGenerator class to emit IL into the get and setters of the Types properties.

Given the entityMetadata above the following is the generated C# code extracted using ilSpy



Looking a bit closer at the GeneratedEntity Type above we see the following:

• A Table attribute has been applied to the Type


• Each of the Types properties has a Column attribute applied


• The identity property has the IsPrimaryKey set to true


• The generated Type  derives from an interface IEntity

So what does this give us?

• A type that can be used via LinqToSql to query, update and delete entities in the underlying data store


• An abstraction from the underlying data stores table and column names. It is not a requirement for the table and generated type names to match. This also applies to the properties


• A type that derives from a statically types known interface type called IEntity. This allows code that consumes these types to cast to a known defined type. Thus reducing the amount of reflection based referencing required to manipulate these generated types

What doesn't this give us?

• There is no abstraction of the underlying types from the data store. So if the underlying column is of type nvarchar then the property of the generated entity must be a String. The generator does not support type coercion or conversion which can get messy. The WCF service that allows the definition of the types metadata ensures that this cannot occur


• There is no abstraction from the underlying data stores foreign key relationships. So if a type is mapped to a table that requires a foreign key value to be specified which is not exposed via the generated type then new rows cannot be inserted


• There is no support for nested relationships. i.e a Thing contains a list of Parts. For the use cases of the overall project this is not a key requirement and can be added with more elaborate metadata and a change to the type generator

So we now have one of the key building blocks of the configurable OData service. In the next post, Part 3 – Custom Data Service Provider, we will see how this fits into place.