Showing posts with label Linq. Show all posts
Showing posts with label Linq. Show all posts

Wednesday, 5 December 2012

Simple Metadata Driven Linq To Sql Type Generator

As part of a larger body of work, I implement a “Dynamic” Linq to Sql Type generator that was based on metadata. A number of people have asked about the source code for this generator so I have decided to publish this portion of the code as a stand alone component.

DumpingCoreMemory.Data.Linq.Generation assembly

  • LinqToSqlTypeGenerator: This class generates the types based on metadata provided at runtime.
  • EntityDescriptor: Instances of this class define entities that the LinqToSqlTypeGenerator class will generate concrete runtime types for
  • EntityPropertyDescriptor: Instances of this class define properties of the concrete runtime types that the LinqToSqlTypeGenerator will generate
ClassDiagram

How To Use the Type Generator

Included in the attached source code zip file is a contrived example, which briefly demonstrates how the type generator can be used. The example assumes a database which contains a table called TableA which contains three columns Id, ColumnA and ColumnB. In the example code this table is to be mapped to a Type called EntityA with three properties Identity, PropertyA and PropertyB. The type EntityA is to be derived from an interface IEntity. This is to enable demonstration of one of the ways a generated Type can be manipulated. The code for the mapping is as follows:
private static List<EntityDescriptor> GenerateEntityDescriptors()
{
 var entityDescriptors = new List<EntityDescriptor>
 {
  new EntityDescriptor
   {
    BaseType = typeof (IEntity),
    EntityName = "EntityA",
    TableName = "TableA",
    EntityProperties = new List<EntityPropertyDescriptor>
       {
        new EntityPropertyDescriptor
         {
          ColumnName = "Id",
          Name = "Identity",
          Type = typeof (Int32),
          IsIdentity = true
         },
        new EntityPropertyDescriptor
         {
          ColumnName = "ColumnA",
          Name = "PropertyA",
          Type = typeof (String)
         },
        new EntityPropertyDescriptor
         {
          ColumnName = "ColumnB",
          Name = "PropertyB",
          Type = typeof (Int32)
         }
       },
   }
 };
 return entityDescriptors;
}

The generator is invoked which generates the types, in the example I pass in an assembly file name, this is not necessary but for the purpose of the example it is useful as we can have a look at the generated types.
var entityDescriptors = GenerateEntityDescriptors();
var generator = new LinqToSqlTypeGenerator();
IList<Type> generatedTypes;

#warning Ensure that the process that invokes this method has rights to write to the disk
// Passing the assemblyFileName parameter is not necessary. 
// The generator will write the assembly to the specified filename.
// This allows us to reuse the generated types or just take a look at them with ILSpy
generator.GenerateTypes("LinqToSqlGeneratedTypes", "LinqToSqlGeneratedTypesAssembly.dll",
      entityDescriptors,
      out generatedTypes);

Using ILSpy we can inspect the generated code that was written to the assembly LinqToSqlGeneratedTypesAssembly.dll.

GeneratedCode

As one would expect we see a simple object with mapping attributes to the underlying database table called TableA. The next step is to activate and use the type for this we have a number of options:
  1. Statically reference the type, this is not shown in the test application as it is trivial to reference the assembly and instantiate its types. It also kind of defeats the purpose of generating types dynamically
  2. Cast to a known base type, in the test application the base type of the generated Type was specified as IEntity. The instance of the type is cast to an IEntity type so that properties can be accessed, the EntityDescriptor for the type specified a matching property for the IEntity interfaces specified Identity property
  3. Use a dynamic type to contain the type. This allows bypass of static type checking to allow access to properties of a type at runtime
  4. Use reflection to get or set properties of the underlying instance, using the name of the property for access

The following code snippet shows instantiation of an instance of the generated type and access by options 2,3 and 4.
object instance = Activator.CreateInstance(generatedTypes[0]);

var instanceAsIEntity = (IEntity) instance;
dynamic o = instance;

Console.WriteLine("Accessed as interface: {0}", instanceAsIEntity.Identity);
Console.WriteLine("Accessed as dynamic: {0}", o.Identity);
Console.WriteLine("Accessed by reflection: {0}", GetPropertyValueByReflection(instance, "Identity"));

Persisting instances of the types to a database is straight forward. The method PersistEntity in the test application shows how to insert an instance to a database.
private static void PersistEntity(object instance)
{
 using (var sqlConnection = new SqlConnection("Server=localhost;Database=Test;Trusted_Connection=True;"))
 {
  using (var context = new DataContext(sqlConnection))
  {
   ITable table = context.GetTable(instance.GetType());

   table.InsertOnSubmit(instance);

   context.SubmitChanges();
  }    
 }
}

For simplicity the code sets the properties via the dynamic object reference. In a real world example the instance may be bound to a UI control prior to CRUD operations being performed.

Limitations


  • There is no abstraction of the underlying types from the data store. So if the underlying column is of type nvarchar then the property of the generated entity must be a String. The generator does not support type coercion or conversion which can get messy. The WCF service that allows the definition of the types metadata ensures that this cannot occur
  • There is no abstraction from the underlying data stores foreign key relationships. So if a type is mapped to a table that requires a foreign key value to be specified which is not exposed via the generated type then new rows cannot be inserted
  • There is no support for nested relationships. i.e a Thing contains a list of Parts. For the use cases of the overall project this is not a key requirement and can be added with more elaborate metadata and a change to the type generator
  • Thursday, 4 October 2012

    Linq To Entities query with Nullable<Guid> exception

    While developing a WCF service I came across one of those runtime errors where the exception details are obscure and provide no real hint as to what one is doing wrong in the code.  The service was exposing data and the get methods of the service supported a simple paging continuation mechanism based on the previous set of data retrieved from the service.
    Exception: System.NotSupportedException: Unable to create a constant value of type 'System.Object'. Only primitive types ('such as Int32, String, and Guid') are supported in this context.   at System.Data.Objects.ELinq.ExpressionConverter.ConstantTranslator.TypedTranslate(ExpressionConverter parent, ConstantExpression linq)
       at System.Data.Objects.ELinq.ExpressionConverter.BinaryTranslator.TypedTranslate(ExpressionConverter parent, BinaryExpression linq)
       at System.Data.Objects.ELinq.ExpressionConverter.TranslateLambda(LambdaExpression lambda, DbExpression input)
       at System.Data.Objects.ELinq.ExpressionConverter.MethodCallTranslator.OneLambdaTranslator.Translate(ExpressionConverter parent, MethodCallExpression call, DbExpression& source, DbExpressionBinding& sourceBinding, DbExpression& lambda)
       at System.Data.Objects.ELinq.ExpressionConverter.MethodCallTranslator.OneLambdaTranslator.Translate(ExpressionConverter parent, MethodCallExpression call)
       at System.Data.Objects.ELinq.ExpressionConverter.MethodCallTranslator.TypedTranslate(ExpressionConverter parent, MethodCallExpression linq)
       at System.Data.Objects.ELinq.ExpressionConverter.TranslateExpression(Expression linq)
       at System.Data.Objects.ELinq.ExpressionConverter.MethodCallTranslator.UnarySequenceMethodTranslator.Translate(ExpressionConverter parent, MethodCallExpression call)
       at System.Data.Objects.ELinq.ExpressionConverter.MethodCallTranslator.TypedTranslate(ExpressionConverter parent, MethodCallExpression linq)
       at System.Data.Objects.ELinq.ExpressionConverter.Convert()
       at System.Data.Objects.ELinq.ELinqQueryState.GetExecutionPlan(Nullable`1 forMergeOption)
       at System.Data.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
       at System.Data.Objects.ObjectQuery`1.System.Collections.Generic.IEnumerable<T>.GetEnumerator()

    The method that the client invokes is the following:
    public IList<DataContracts.TBL_Incident> GetIncidents(Guid? skipToken)
    {
        List<DataContracts.TBL_Incident> result = new List<DataContracts.TBL_Incident>();
    
        ExceptionHandledOperation(delegate
        {
            using (IContextProxy context = _contextFactory.CreateObjectContext())
            {
                IQueryable<Domain.EntityModel.TBL_Incident> incidents;
    
                if (!skipToken.HasValue)
                {
                    incidents = (from i in context.TBL_Incident
                                 orderby i.IncidentId
                                 select i).Take(EntityModelServiceSettings.Current.EntitySetPageSize);
                }
                else
                {
                    incidents = (from i in context.TBL_Incident
                                 orderby i.IncidentId
                                 where i.IncidentId.CompareTo(skipToken) > 0
                                 select i).Take(EntityModelServiceSettings.Current.EntitySetPageSize);
                }
    
                foreach (var i in incidents)
                {
    #warning optimise generated method so that this can be done in less LOC
                    var mappedIncident = new DataContracts.TBL_Incident();
                    DataContracts.TBL_Incident.MapTBL_Incident(i, mappedIncident);
                    result.Add(mappedIncident);
                }
            }
        });
    
        return result;
    }
    

    The skipToken parameter allows the client to pass the last identifier from the previous page retrieved from the service.
    sw.Start();
    
    var incidents = client.GetIncidents(null);
    
    Console.WriteLine("Set: {0} Time: {1}", incidents.Count(), sw.Elapsed);
    
    incidents = client.GetIncidents(incidents.Last().IncidentId);
    
    Console.WriteLine("Set: {0} Time: {1}", incidents.Count(), sw.Elapsed);

    The cause of the exception is the service side statement that appends the skipToken to the query.
    incidents = (from i in context.TBL_Incident
                orderby i.IncidentId
                where i.IncidentId.CompareTo(skipToken) > 0
                select i).Take(EntityModelServiceSettings.Current.EntitySetPageSize);

    The problem is that the compiler does not have a problem with the fact that the call passes the skipToken which is of type Nullable<Guid>. The problem is when the LINQ to Entity classes attempt to map the expression to a SQL expression.

    To fix the problem it is simply a case of passing the skipToken.Value to the CompareTo() method.
    incidents = (from i in context.TBL_Incident
                orderby i.IncidentId
                where i.IncidentId.CompareTo(skipToken.Value) > 0
                select i).Take(EntityModelServiceSettings.Current.EntitySetPageSize);

    Wednesday, 14 March 2012

    Configurable OData Service - Part 2 – Data Type Generator

    In the this post I am going to cover some of the implementation details of the Data Type generation component of the configurable OData service.
    The primary purpose of this component is to dynamically generate data types based on some metadata. The metadata describes the properties of the types and how the type maps to the underlying data store. The metadata is defined by an end user via a WPF application and is stored in a database via an intermediary WCF service that manages the metadata.
    Metadata metadata

    Reflection Type Generation

    Given metadata for a Type to be generated at runtime, take for example the following metadata definition:
    EntityDescriptor entityMetadata = new EntityDescriptor()
    {
        BaseType = typeof(IEntity),
        EntityName = "GeneratedEntity",
        EntityProperties = new List<EntityPropertyDescriptor>()
        {
            new EntityPropertyDescriptor()
            {
                ColumnName = "Id",
                IsIdentity = true,
                IsNullable = false,
                Name = "Identifier",
                Type = typeof(Guid)
            },
            new EntityPropertyDescriptor()
            {
                ColumnName = "Comment",
                IsIdentity = false,
                IsNullable = false,
                Name = "Comment",
                Type = typeof(String)
            },
    TableName = "TestTable"     } };

    The first step in generating the dynamic types is to create an instance of an AssemblyBuilder and a ModuleBuilder into which the Type, based on metadata will be emitted.
    ModuleBuilder moduleBuilder;
    AssemblyBuilder assemblyBuilder;
    DynamicTypeGenerator.Instance.CreateDynamicAssembly("TestAssembly", out assemblyBuilder, out moduleBuilder);

    Now we have an AssemblyBuilder instance and an instance of a ModuleBuilder which will be used in subsequent calls to the DynamicTypeGenerator instance to emit types into. For the purpose of this example only one Type is to be emitted into the Assembly.

    Given the above entity metadata a call to the CreateLinqToSqlType method of the DynamicTypeGenerator instance passing the type metadata generates the underlying Type for the metadata description.

    DynamicTypeGenerator.Instance.CreateLinqToSqlType("TestAssembly", assemblyBuilder, moduleBuilder, entityMetadata);

    The CreateLinqToSqlType as expected generates a simple LinqToSql data type entity that has attributes applied to its properties to support the mapping of the Type to an underlying database. There is not anything extremely complicated about this method. It uses a number of the ReflectionEmit constructs to generate the Types and the ILGenerator class to emit IL into the get and setters of the Types properties.

    Given the entityMetadata above the following is the generated C# code extracted using ilSpy

    GeneratedType

    Looking a bit closer at the GeneratedEntity Type above we see the following:


    • A Table attribute has been applied to the Type

    • Each of the Types properties has a Column attribute applied

    • The identity property has the IsPrimaryKey set to true

    • The generated Type  derives from an interface IEntity

    IEntity

    So what does this give us?


    • A type that can be used via LinqToSql to query, update and delete entities in the underlying data store

    • An abstraction from the underlying data stores table and column names. It is not a requirement for the table and generated type names to match. This also applies to the properties

    • A type that derives from a statically types known interface type called IEntity. This allows code that consumes these types to cast to a known defined type. Thus reducing the amount of reflection based referencing required to manipulate these generated types

    What doesn't this give us?


    • There is no abstraction of the underlying types from the data store. So if the underlying column is of type nvarchar then the property of the generated entity must be a String. The generator does not support type coercion or conversion which can get messy. The WCF service that allows the definition of the types metadata ensures that this cannot occur

    • There is no abstraction from the underlying data stores foreign key relationships. So if a type is mapped to a table that requires a foreign key value to be specified which is not exposed via the generated type then new rows cannot be inserted

    • There is no support for nested relationships. i.e a Thing contains a list of Parts. For the use cases of the overall project this is not a key requirement and can be added with more elaborate metadata and a change to the type generator

    So we now have one of the key building blocks of the configurable OData service. In the next post, Part 3 – Custom Data Service Provider, we will see how this fits into place.

    Sunday, 4 March 2012

    WCF Data Services + Reflection + Custom Data Provider = Configurable OData Service

    Part 1 Intro

    I really enjoy a good technical challenge, in this day and age of highly evolved frameworks and tool kits it is good to get a set of requirements for which there is not an obvious out of the box solution for. One that requires a little bit more than a few LOC and a bit of configuration tweaking.

    Background

    The challenge which I was presented with, was in the context of a windows workflow 4 (WF4) project for which I am in the midst of designing and developing a number of components. The project is to integrate WF4 into the main product line. The aim is to enable end user business analysts or customer service consultants to design a set of workflow processes that are unique to each customer.
    The main components of this solution are the following:
    • Workflow Designer – A WPF application to enable the design, management and deployment of workflow processes
    • Workflow Activities – A set of custom activities that encapsulate common business actions
    • A Workflow Engine – A host component to manage the execution, lifecycle and versioning of the various running workflows that a end user can design
    • A Domain Model Service – A service which exposes a consistent, version invariant view of the applications data model
    Future blog entries will cover some of the other components. This is the first post in a series that will cover some of the background behind the Domain Model Service

    Requirements

    • Enable Users Define Entities
      • The Domain Model Service should allow an end user to define a Domain Model entity via a designer invoking an API
    • Support CRUD on Entities
      • The Domain Model Service should support CRUD operations on all Entities that are defined
    • Enable Database Abstraction
      • Expose Entities in a manner that allows attribute names and entity names to differ from that of the underlying data store. This prevents implementation leaking out to the upper layers
    • Enable Versioning
      • A workflow instance can potentially execute indefinitely. It is therefore a requirement to maintain multiple versions of the Domain Model
    • Enable Query Of Entities
      • Allow an Entity to be queried by any combination of its attributes

    The Solution

    ComponentView

    Metadata Store – This component is the store of all Domain Models and the entities that are defined for each of the Domain Models. The data within this store is managed by a different set of components, which are beyond the scope of this post.

    WCF Service – This component provides the hosting and lifecycle management for the WCF Data Services (OData), which are defined within the metadata. This component exposes the OData services each of which exposes a version if the Domain Model at a different unique URL. It would have been possible to host each version of the Domain Model in its own process but this is not required for the initial implementation. The Domain Model services could be hosted in an own process if required in the future.

    Data Type Generator – This component generates the set of entities for a given Domain Model through the use of reflection.

    Domain Model Service – This component is the realisation of a concrete implementation of a Domain Model version. This is a WCF Data Services (OData) service that uses the Custom Data Service Provider as the source and sink of the Domain Model data that it exposes.

    Custom Data Service Provider – This component provides the data and the mechanisms for querying an updating Domain Model objects for a given Domain Model Service instance.

    In the following posts I will delve into more detail for each of the components.

    Part 2 – Data Type Generator
    Part 3 – Custom Data Service Provider
    Part 4 – WCF Service Host
    Part 5 - Put it all together