Custom WCF UserNamePasswordValidator Performance Impact

Recently I have had to perform some debug/analysis on an unusual case of clients receiving timeouts on a critical WCF service in a production environment. I use the term “unusual” to describe the timeouts due to the nature in which these timeouts occur combined with the operation behaviour of the service when they are occurring. That and the timeouts are extremely severe, every client experiences timeouts, in my experience with WCF is not an expected operational behaviour of the WCF stack. Under normal high load scenarios a percentage of clients will experience timeouts or receive the expected Server To Busy fault when the WCF service has breached its configured throttle values.
In this specific case knowing the operation capabilities of the WCF service in question I knew that the amount of load on the server although high was well within the capacity of the service in question. Nor was the load significant enough to breach the configured service throttles. I eventually got to the bottom of the issue through debug and crash dump analysis. It was an unexpected/undocumented behaviour of the WCF stack coupled/influenced by a specific behaviour of a custom UserNamePasswordValidator used by the WCF service.
I thought it would be worth describing the problem in more detail in the hope it might help others who experience WCF Service performance issues, that done seem to "make sense", giving one additional diagnostic check to ad to the toolbox.

Symptoms

The symptoms of the service outage are as follows:

• A high percentage of clients, > 90%, receive timeouts or Server To Busy exceptions
• The Instances service performance Counter is at or close to the the configured or defaulted ServiceThrottlingElement.MaxConcurrentInstances value
• The Calls Per Second Rate service performance counter is low or at 0 for periods of time
• If the service is deployed in an IIS app pool the ASP.NET Application Performance Counter  Pipeline Instance Count will be at or close to its configured or defaulted value
• The Security Validation And Authentication Failures performance counter can be non zero
• The service appears to be deadlocked or being starved of requests, the service is not writing log files or updating database etc. but clients experience timeouts or very slow reponses
• Very low CPU utilisation by the process in question

 

Cause

After generating a process memory dump, from the WCF Service. After examining all of the threads stacks by executing the EEStack command it was clearly evident that most of the threads in the process where blocked on the unmanaged windows API method GetQueuedCompletionStatus which is a blocking I/O call. One call was blocked on the Validate method of the UserNamePasswordValidator.
It transpires that the the Validate method of the custom UserNamePasswordValidator used by the WCF service, talks to a proprietary back office system to validate a clients username and password. To prevent a client from brute forcing a users password it will delay the username/password validation response for 5 seconds after the nth unsuccessful attempt. This has very serious consequences for the throughput of the WCF service.

 

Demonstration Code And Results

To demonstrate this problem I have put together a simple demo base on one of the WCF sample applications. It is the calculator application that implements a simple custom UserNamePasswordValidator. The code itself is not really the interesting part and is pretty trivial, the intent is to demonstrate the symptoms.

 

Setup

Run the Setup.Bat file in the .\UserNamePasswordValidator\CS folder. Note you should run this from a VS2010 command prompt with administrative privileges. As it creates a test certificate and grants access to the private key for encryption/signing the message credentials.

 

Clean-up

Run the CleanUp.bat file in the .\UserNamePasswordValidator\CS folder. Note you should run this from a VS2010 command prompt with administrative privileges. AS it deletes the test certificate created by setup.bat
Open the solution in VS2010 running as a administrator as the code attempts to register a http.sys listener on port 8001.
There are a number of application settings that can be used to modify the behaviour of the client and service code.
In the LoadTestClient there are the following settings:

<applicationSettings>
 <LoadTestClient.Properties.Settings>
   <!-- The number of tasks to create with valid passwords -->
   <setting name="NumberTasksValidPassword" serializeAs="String">
    <value>10</value>
   </setting>
   <!-- The number of tasks to create with invalid passwords -->
   <setting name="NumberTasksInvalidPassword" serializeAs="String">
    <value>2</value>
   </setting>
 </LoadTestClient.Properties.Settings>
</applicationSettings>

In the service there is one setting to configure the sleep time for invalid requests:

<applicationSettings>
 <service.Properties.Settings>
  <!-- The sleep time for a request with an invalid password -->
  <setting name="InvalidCredentialSleep" serializeAs="String">
   <value>1000</value>
  </setting>
 </service.Properties.Settings>
</applicationSettings>

Test Runs

To demonstrate the behaviour I have put together a few test cases, that vary each of the parameters of the test and show the results.

10 Tasks valid credentials , 0 Tasks invalid credentials

Form the graphs above we can see 10 service instances performing approximately 85 calls/second. The calls outstanding counter averages < 1. This is normal operation of the service.

10 Tasks valid credentials , 2 Tasks invalid credentials, Invalid Credential sleep time 1 sec

As we can see can see from the above, the results are relatively similar.

10 Tasks valid credentials , 3 Tasks invalid credentials, Invalid Credential sleep time 1 sec

As we can see from the above we have reached a tipping point. When the number of clients making invalid requests increases to 3, we can see initially the service respond times and calls/second are ok. When the three client tasks with invalid passwords start the throughput of the service is severely throttled to approximately 10 calls/sec. I would also draw your attention to the Calls Duration metric which is close to the other two runs.

10 Tasks valid credentials , 2 Tasks invalid credentials, Invalid Credential sleep time 5 sec

When the number of clients making invalid requests is set to 2 but the sleep time is increased to 5 seconds we see a very erratic calls/second profile. Again the Calls Duration metric at the service level is in line with the other tests, but if measured from the client we would see a higher Calls Duration. The Calls Outstanding metric is also not zero more often than in the previous tests.

10 Tasks valid credentials , 4 Tasks invalid credentials, Invalid Credential sleep time 1 sec

When the number of clients making invalid requests is set to 4 with a sleep time of 1 seconds, we can see initially the service respond times and calls/second are ok. When the  When the 4 client tasks with invalid passwords start the calls/second profile of the service is very erratic. Again the Calls Duration metric at the service level is in line with the other tests.

In a real world environment it is clear that you will not see such clear patterns to the calls/second metric. It is more likely that the service will in fact not process any requests if the client invalid call pattern is more random and at a higher volume.

Crash Dump Analysis

If we take a crash dump of the WCF Service process while it is in the “Deadlocked/Blocked” state we can examine the processes threads.

0:000> !threads
ThreadCount:      12
UnstartedThread:  0
BackgroundThread: 11
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
                                       PreEmptive                                                   Lock
   ID  OSID        ThreadOBJ     State GC       GC Alloc Context                  Domain           Count APT Exception
0    1  16d0 0000000000205eb0      a020 Enabled  0000000000000000:0000000000000000 00000000001f9780     1 MTA
2    2  184c 000000000020c1f0      b220 Enabled  0000000000000000:0000000000000000 00000000001f9780     0 MTA (Finalizer)
6    a  1708 000000001b9df9d0   a009220 Enabled  0000000000000000:0000000000000000 00000000001f9780     0 MTA (Threadpool Completion Port)
8    6  1ae8 000000001b927570   a009220 Enabled  0000000000000000:0000000000000000 00000000001f9780     0 MTA (Threadpool Completion Port)
9    4  19fc 000000001b926cf0   8009220 Enabled  0000000000000000:0000000000000000 00000000001f9780     0 MTA (Threadpool Completion Port)
10    8  1958 000000001b9e1e40   8009220 Enabled  0000000000000000:0000000000000000 00000000001f9780     0 MTA (Threadpool Completion Port)
11    b   e84 000000001b9e6c60   8009220 Enabled  00000000033bb8c0:00000000033bd178 00000000001f9780     0 MTA (Threadpool Completion Port)13    c   e18 000000001b9f2260   8009220 Enabled  0000000000000000:0000000000000000 00000000001f9780     0 MTA (Threadpool Completion Port)
14    5  1a58 000000001b90c4a0   8009220 Enabled  0000000000000000:0000000000000000 00000000001f9780     0 MTA (Threadpool Completion Port)
17    9   728 000000001af1f7c0   8009220 Enabled  0000000000000000:0000000000000000 00000000001f9780     0 MTA (Threadpool Completion Port)
18    d  1aec 000000001af1ef40   8009220 Enabled  0000000000000000:0000000000000000 00000000001f9780     0 MTA (Threadpool Completion Port)
19    3  1be0 000000001aed7e00   8009220 Enabled  0000000000000000:0000000000000000 00000000001f9780     0 MTA (Threadpool Completion Port)


We can see that all of the threads are running but the Exception column which lists the last thrown exception (if any) for the thread, shows (ThreadPool Completion Port). We can exclude a few of the threads from inspection, for example thread 0 is the thread executing the Microsoft.ServiceModel.Samples.CalculatorService.Main() method. Thread 2 is the CLR Finalizer thread.

If we look at a few of the other threads call stacks, shortened for brevity, we see the following:

---------------------------------------------
Thread  17
Current frame: ntdll!NtRemoveIoCompletion+0xa
Child-SP         RetAddr          Caller, Callee
000000001d5deb40 000007fefdf616ad KERNELBASE!GetQueuedCompletionStatus+0x39, calling ntdll!ZwRemoveIoCompletion
000000001d5deb50 000007fef4d46b1c clr!GCHolderBase<1,0,0,1>::Pop+0x2c, calling clr!Thread::EnablePreemptiveGC
000000001d5deb80 000007fef4e309ff clr!Thread::InternalReset+0x140, calling clr!Thread::SetBackground
000000001d5deba0 00000000777aa4e1 kernel32!GetQueuedCompletionStatusStub+0x11, calling kernel32!GetQueuedCompletionStatus
000000001d5debb0 000007fef4d7dd49 clr!Thread::LeaveRuntime+0x9, calling clr!Thread::LeaveRuntimeNoThrow
000000001d5debe0 000007fef4f1eb2f clr!ThreadpoolMgr::CompletionPortThreadStart+0x113, calling kernel32!GetQueuedCompletionStatusStub
000000001d5dec50 000007fef4d46a14 clr!operator delete+0x45, calling kernel32!GetLastErrorStub---------------------------------------------
Thread  18
Current frame: ntdll!NtRemoveIoCompletion+0xa
Child-SP         RetAddr          Caller, Callee
000000001ce7ec10 000007fefdf616ad KERNELBASE!GetQueuedCompletionStatus+0x39, calling ntdll!ZwRemoveIoCompletion
000000001ce7ec20 000007fef4d46b1c clr!GCHolderBase<1,0,0,1>::Pop+0x2c, calling clr!Thread::EnablePreemptiveGC
000000001ce7ec50 000007fef4e309ff clr!Thread::InternalReset+0x140, calling clr!Thread::SetBackground
000000001ce7ec70 00000000777aa4e1 kernel32!GetQueuedCompletionStatusStub+0x11, calling kernel32!GetQueuedCompletionStatus
000000001ce7ec80 000007fef4d7dd49 clr!Thread::LeaveRuntime+0x9, calling clr!Thread::LeaveRuntimeNoThrow
000000001ce7ecb0 000007fef4f1eb2f clr!ThreadpoolMgr::CompletionPortThreadStart+0x113, calling kernel32!GetQueuedCompletionStatusStub
000000001ce7ed20 000007fef4d46a14 clr!operator delete+0x45, calling kernel32!GetLastErrorStub

As mentioned earlier GetQueuedCompletionStatus is a blocking operation.

Looking at Thread 8 we can see that it is the only thread executing the Validate method of the CustomUsernameValidator.

Thread   8
Current frame: ntdll!NtDelayExecution+0xa
Child-SP         RetAddr          Caller, Callee
000000001c4cd200 000007fefdf61203 KERNELBASE!SleepEx+0xab, calling ntdll!ZwDelayExecution
000000001c4cd220 000007fee14f2f9a (MethodDesc 000007fee148ffb0 +0x4a System.Xml.XmlBufferReader.GetString(Int32, Int32)), calling (MethodDesc 000007fef39e5540 +0 System.String.CtorCharArrayStartLength(Char[], Int32, Int32))
000000001c4cd270 000007fefdf6358b KERNELBASE!SleepEx+0x12d, calling ntdll!RtlActivateActivationContextUnsafeFast
000000001c4cd2a0 000007fef4d76455 clr!EESleepEx+0x22, calling kernel32!SleepExStub
000000001c4cd2d0 000007fef4f33fba clr!Thread::UserSleep+0x97
000000001c4cd310 000007fee14e95bf (MethodDesc 000007fee1470360 +0x2f System.Xml.XmlBaseReader.get_Value()), calling clr!JIT_WriteBarrier_Fast
000000001c4cd320 000007fef4f3410c clr!ThreadNative::Sleep+0x100, calling clr!Thread::UserSleep
000000001c4cd3d0 000007fee166fca5 (MethodDesc 000007fee148f5e8 +0x85 System.Xml.XmlBaseReader.ReadElementString())
000000001c4cd3e0 000007fee167ed39 (MethodDesc 000007fee148d818 +0x19 System.Xml.XmlDictionaryReader.ReadStartElement(System.Xml.XmlDictionaryString, System.Xml.XmlDictionaryString))
000000001c4cd3e8 000007ff00160be6 (MethodDesc 000007ff001b4f98 +0x116 Microsoft.ServiceModel.Samples.CalculatorService+CustomUserNameValidator.Validate(System.String, System.String)), calling (MethodDesc 000007fef3a5cd98 +0 System.Threading.Thread.Sleep(Int32))

Summary

In my opinion it is evident from the test results and the stack trace analysis presented in this post that there is some inherent "queuing/pipelining" in the processing of messages with credentials. Whether or not this is due to the fact that a custom UserNamePasswordValidator is being used is unclear. It is also evident that this “pipeline” is sensitive to execution latency of the Validate method of the UserNamePasswordValidator, possibly as a side effect of being a shallow queue.

Either a number of requests of moderate latency or a smaller number of requests with high latency can have a impact on the overall performance of the service. It is therefore something to be aware of when implementing custom UserNamePasswordValidator types that perform credential validation that may be subject to non trivial execution latency either due to I/O or CPU.

In my opinion this behaviour could be exploited to create an effective DOS attack, as the attacker does not have to create a huge request load to affect the WCF services performance. In the specific case that I discovered in the production environment I modified the code to mitigate the problem while remaining secure.

WCF Data Services NullReferenceException browsing metadata

I have recently discovered an issue with malformed Uri base addresses and WCF Data Services when querying the metadata for the service.
The issue causes an exception in the System.Data.Services.HttpContextServiceHost class which causes a http 500 in the client browser. Searching the internet did not yield much additional information in this problem.




The offending line in the HttpContextServiceHost is the when the code accesses the UriTemplateMatch property, which is null, a NullReferenceException is thrown.

internal void VerifyQueryParameters()
       {
           HashSet<string> set = new HashSet<string>(StringComparer.Ordinal);
          NameValueCollection queryParameters = this.operationContext.IncomingRequest.UriTemplateMatch.QueryParameters;

To discover the reason why this property is null, I need to debug much further back in the request processing phase of the WCF Data Services stack.

Problem has its root in the the public string SelectOperation(ref Message message) of the System.ServiceModel.Dispatcher.WebHttpDispatchOperationSelector class. This operation is called during the http request that is made to the base address. You can see from the call stack below:



From what I understand from stepping through the code the method public string SelectOperation(ref Message message) is attempting to determine the operation to invoke from the request message. The matched operation is then assigned to the message properties collection which is accessed later, via the OperationContext.IncomingRequest.UriTemplateMatch. The method public string SelectOperation(ref Message message) makes a call to the method  protected virtual string SelectOperation(ref Message message, out bool uriMatched) which in turn calls an internal private method private bool CanUriMatch(UriTemplateTable methodSpecificTable, Uri to, HttpRequestMessageProperty prop, Message message, out string operationName) it is this method where the failure to match occurs and thus the OperationContext.IncomingRequest.UriTemplateMatch property is null.

public string SelectOperation(ref Message message)
{
    bool flag;
    if (message == null)
    {
        throw DiagnosticUtility.ExceptionUtility.ThrowHelperArgumentNull("message");
    }
    string property = this.SelectOperation(ref message, out flag);
    message.Properties.Add("UriMatched", flag);
    if (property != null)
    {
        message.Properties.Add("HttpOperationName", property);
        if (DiagnosticUtility.ShouldTraceInformation)
        {
            TraceUtility.TraceEvent(TraceEventType.Information, 0xf0025, SR2.GetString(SR2.TraceCodeWebRequestMatchesOperation, new object[] { message.Headers.To, property }));
        }
    }
    return property;
}
protected virtual string SelectOperation(ref Message message, out bool uriMatched)
{
    if (message == null)
    {
        throw DiagnosticUtility.ExceptionUtility.ThrowHelperArgumentNull("message");
    }
    uriMatched = false;
    if (this.methodSpecificTables != null)
    {
        UriTemplateTable table;
        if (!message.Properties.ContainsKey(HttpRequestMessageProperty.Name))
        {
            return this.catchAllOperationName;
        }
        HttpRequestMessageProperty requestProp = message.Properties[HttpRequestMessageProperty.Name] as HttpRequestMessageProperty;
        if (requestProp == null)
        {
            return this.catchAllOperationName;
        }
        string method = requestProp.Method;
        Uri to = message.Headers.To;
        if (to == null)
        {
            return this.catchAllOperationName;
        }
        if (this.helpUriTable != null)
        {
            UriTemplateMatch match = this.helpUriTable.MatchSingle(to);
            if (match != null)
            {
                uriMatched = true;
                this.AddUriTemplateMatch(match, requestProp, message);
                if (method == "GET")
                {
                    return "HelpPageInvoke";
                }
                WebHttpDispatchOperationSelectorData property = new WebHttpDispatchOperationSelectorData {
                    AllowedMethods = new List<string> { "GET" }
                };
                message.Properties.Add("HttpOperationSelectorData", property);
                return this.catchAllOperationName;
            }
        }
        if (this.methodSpecificTables.TryGetValue(method, out table))
        {
            string str2;
            uriMatched = this.CanUriMatch(table, to, requestProp, message, out str2);
            if (uriMatched)
            {
                return str2;
            }
        }
        if (this.wildcardTable != null)
        {
            string str3;
            uriMatched = this.CanUriMatch(this.wildcardTable, to, requestProp, message, out str3);
            if (uriMatched)
            {
                return str3;
            }
        }
        if (this.ShouldRedirectToUriWithSlashAtTheEnd(table, message, to))
        {
            return "";
        }
        List<string> list2 = null;
        foreach (KeyValuePair<string, UriTemplateTable> pair in this.methodSpecificTables)
        {
            if (((pair.Key != method) && (pair.Key != "*")) && (pair.Value.MatchSingle(to) != null))
            {
                if (list2 == null)
                {
                    list2 = new List<string>();
                }
                if (!list2.Contains(pair.Key))
                {
                    list2.Add(pair.Key);
                }
            }
        }
        if (list2 != null)
        {
            uriMatched = true;
            WebHttpDispatchOperationSelectorData data2 = new WebHttpDispatchOperationSelectorData {
                AllowedMethods = list2
            };
            message.Properties.Add("HttpOperationSelectorData", data2);
        }
    }
    return this.catchAllOperationName;
}
private void AddUriTemplateMatch(UriTemplateMatch match, HttpRequestMessageProperty requestProp, Message message)
{
    match.SetBaseUri(match.BaseUri, requestProp);
    message.Properties.Add("UriTemplateMatchResults", match);
}
private bool CanUriMatch(UriTemplateTable methodSpecificTable, Uri to, HttpRequestMessageProperty prop, Message message, out string operationName)
{
    operationName = null;
    UriTemplateMatch match = methodSpecificTable.MatchSingle(to);
    if (match != null)
    {
        operationName = match.Data as string;
        this.AddUriTemplateMatch(match, prop, message);
        return true;
    }
    return false;
}

When the Uri is matched the matched property is assigned to the “UriTemplateMatchResults” property of the request message. It is this property that is accessed by the OperationContext.IncomingRequest.UriTemplateMatch property and is the reason that it is returned as null and the cause of the exception.

WCF Data Services Reflection Provider with Derived Types

Recently while doing some POC investigative development work, on the potential use of WCF Data Services aka the Microsoft implementation of OData feeds, I discovered some behaviour while testing out the Reflection Provider that I was unable to explain or find an answer too online. So I decided to do some more digging to figure out what was causing this unexpected behaviour.

Background

WCF Data Services supports multiple Data Provider models for exposing data as a Open Data Protocol (OData) feed, these are:

• Entity Framework Provider
• Reflection Provider
• Custom Data Service Provider

I was specifically interested in delving into the capabilities of the Reflection Provider. The reason being the data that I wanted to expose, as an OData feed, a data model based on an existing set of data classes, this rules out the use of the Entity Framework provider as a mechanism to expose the data. This left the oprions of the Reflection Provider or rolling my own Custom Data Service Provider as options for exposing the data.

My used case is actually more complicated than simply exposing a set of POCO as an OData feed. So I was prepared for my investigation to lead me to the conclusion that I would have to roll my own Custom Data Service Provider. Something instinct was telling me I would need to do anyway. But it was worth the effort to spend some time up front figuring out if the Reflection Provider would be a suitable candidate implementation. As it would mean less LOC for the final implementation.

The Behaviour

The behaviour that I discovered while building and testing the POC was in relation to exposing types that are derived from a base type which is a key requirement for the code that will need to be implemented. Specifically the “problem” behaviour is the Reflection Provider is unable to expose the metadata and data for a types derived from a base type,where the derived type is not located in the same assembly as the base type. To clarify this point further I have put together a simple demo application that demonstrates the “problem”.

Simple Object Graph

The above class diagram shows a simple hierarchy of types. BaseEntityTypeA and BaseEntityTypeB are identical in terms of definition as are the derived types of DerivedTypeA and DerivedTypeB. What is not shown is that DerivedTypeB is defined in a different assembly from that of the rest of the types.

Simple Data Service

The types are exposed from the following simple data service.

[System.ServiceModel.ServiceBehavior(IncludeExceptionDetailInFaults = true)]
public class EntityDataservice : DataService<EntityDataItems>
{
    // This method is called only once to initialize
    //service-wide policies.
    public static void InitializeService(IDataServiceConfiguration config)
    {
        config.SetEntitySetAccessRule("*", EntitySetRights.All);
        config.UseVerboseErrors = true;
    }
}

public class EntityDataItems
{
    private static List<DerivedTypeA> _derivedA;
    private static List<DerivedTypeB> _derivedB;

    static EntityDataItems()
    {
        _derivedA = new List<DerivedTypeA>();
        _derivedB = new List<DerivedTypeB>();

        for (int i = 0; i < 10; i++)
        {
            _derivedA.Add(new DerivedTypeA() { Id = i, Name = i.ToString() });
            _derivedB.Add(new DerivedTypeB() { Id = i, Name = i.ToString() });
        }
    }

    public IQueryable<BaseEntityTypeA> DerivedA
    {
        get { return _derivedA.AsQueryable<DerivedTypeA>(); }
    }

    public IQueryable<BaseEntityTypeB> DerivedB
    {
        get { return _derivedB.AsQueryable<DerivedTypeB>(); }
    }
}

Query Endpoint

When the service is fired up and we browse to the base address of the service, http://localhost:8080/EntityDataservice, we see the following:

Query Metadata

So far so good, so lets query the metadata from the service http://localhost:8080/EntityDataservice/$metadata



So its at this point that we notice that there is something not quite right with the service. The definition of DerivedTypeA is exposed in the metadata feed but there is no definition of DerivedTypeB exposed from the service.

<EntityType Name="DerivedTypeA" BaseType="BaseEntityTypes.BaseEntityTypeA">
    <Property Name="Name" Type="Edm.String" Nullable="true" />
</EntityType>

Query DerivedTypeA Data

When we query the DerivedTypeA data feed from http://localhost:8080/EntityDataservice/DerivedA we see the following snippet of data:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<feed xml:base="http://localhost:8080/EntityDataservice/" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom">
  <title type="text">DerivedA</title>
  <id>http://localhost:8080/EntityDataservice/DerivedA</id>
  <updated>2012-02-20T15:39:43Z</updated>
  <link rel="self" title="DerivedA" href="DerivedA" />
  <entry>
    <id>http://localhost:8080/EntityDataservice/DerivedA(0)</id>
    <title type="text"></title>
    <updated>2012-02-20T15:39:43Z</updated>
    <author>
      <name />
    </author>
    <link rel="edit" title="BaseEntityTypeA" href="DerivedA(0)" />
    <category term="BaseEntityTypes.DerivedTypeA" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
    <content type="application/xml">
      <m:properties>
        <d:Id m:type="Edm.Int32">0</d:Id>
        <d:Name>0</d:Name>
      </m:properties>
    </content>
  </entry>
  <entry>
    <id>http://localhost:8080/EntityDataservice/DerivedA(1)</id>
    <title type="text"></title>
    <updated>2012-02-20T15:39:43Z</updated>
    <author>
      <name />
    </author>
    <link rel="edit" title="BaseEntityTypeA" href="DerivedA(1)" />
    <category term="BaseEntityTypes.DerivedTypeA" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
    <content type="application/xml">
      <m:properties>
        <d:Id m:type="Edm.Int32">1</d:Id>
        <d:Name>1</d:Name>
      </m:properties>
    </content>
  </entry>
</feed>

Query DerivedTypeB Data

When we query the DerivedTypeA data feed from http://localhost:8080/EntityDataservice/DerivedB we see the following in the browser:



The response data from the service shows the following

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<feed xml:base="http://localhost:8080/EntityDataservice/" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom">
  <title type="text">DerivedB</title>
  <id>http://localhost:8080/EntityDataservice/DerivedB</id>
  <updated>2012-02-20T15:40:51Z</updated>
  <link rel="self" title="DerivedB" href="DerivedB" />
  <m:error>
    <m:code></m:code>
    <m:message xml:lang="en-IE">Internal Server Error. The type 'DerivedEntityTypes.DerivedTypeB' is not a complex type or an entity type.</m:message>
  </m:error>

Debugging

So what specifically the problem with the Reflection Provider, to answer this question its time to step into the .net framework. From reflecting on the DataService<T> code using ILSpy I found the method CreateProvider() which conditionally instantiates the ReflectionServiceProvider.



The method that we are interested in is highlighted in the above image dataProviderInstance.PopulateMetadata(); It is this method of the ReflectionServiceProvider instance that generates the metadata for the types exposed by the Data Service.

Drilling down to into this method we find a call to PopulateMetadataForDerivedTypes it is this method that populates the metadata for the types derived from the base entity types.

Debugging a little further we see that the ReflectionServiceProvider.PopulateMetadataForDerivedTypes iterates over the collection of known resource types, in the test case this is BaseEntityTypeA and BaseEntityTypeB




For each known resource type the ReflectionServiceProvider.PopulateMetadataForDerivedTypes get the assembly of the known type being evaluated



For each type in the assembly the method checks if the type can be assigned to the root known type. If so it is added to the collection of derived types that are further processed to generate the metadata.



You can clearly see that this does not support having derived types in a different assembly.

Conclusion

In my opinion this seems like a reasonable strategy to employ as to do other wise would require reflecting over every non CLR or .net framework in the current app domain. It would be nice to see this behaviour of the ReflectionServiceProvider documented.

In general this confirmed my initial suspicion that the ReflectionServiceProvider  would have some restrictive limitations for what I intend on implementing. I think it is better for most non EF4 based implementations of a WCF Data Service to skip the ReflectionServiceProvider and jump straight to a Custom Data Service Provider. Hopefully my findings will save someone a bit of time when using the ReflectionServiceProvider in the future.


The code is available here: GitHub ReflectionProviderTest