tag:blogger.com,1999:blog-28423084264997033022024-03-06T00:53:34.880+00:00Dumping core memoryPersisting bits of my memory to the interwebAnonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.comBlogger35125tag:blogger.com,1999:blog-2842308426499703302.post-42933517240648565732016-10-01T18:20:00.000+01:002016-10-22T18:20:21.472+01:00Calculate Real Time Percentiles On Streamed Data<p>When monitoring the operational performance of a sub system or component one can use the <a href="https://msdn.microsoft.com/en-us/library/system.diagnostics.performancecounter(v=vs.110).aspx" target="_blank">PerformanceCounter</a> classes to define a custom counter. The <a href="https://msdn.microsoft.com/en-us/library/system.diagnostics.performancecounter%28v=vs.110%29.aspx" target="_blank">PerformanceCounter</a> classes can create counters that can measure rates, averages, percentages, differences or instantaneous values.These counter types cover a broad range of measurement use cases.<br><br>One of the common features of these performance counter types is that they are single values computed at an instance of time. Without recording successive value and aggregating the results they cannot provide any indication of the distribution of the values measured.<br><br>Often when monitoring the operational performance of a subsystem it is extremely useful to determine the values of the high order percentiles e.g. 90th, 95th, 99th. One of the difficulties in determining the values for these percentiles on a operational system that produces a continuous stream of metric values, for example response time, is the storage requirements for the sample data.<br><br>There are a number of algorithms that have been proposed that can compute percentile values on a stream of data values utilising a limited of fixed amount of storage space. After some investigation I decided to implement the <a href="http://www.cs.wustl.edu/~jain/papers/ftp/psqr.pdf" target="_blank">P<sup>2</sup> algorithm</a> (P squared algorithm).<br><br>It is beyond the scope of this post to discuss the <a href="http://www.cs.wustl.edu/%7Ejain/papers/ftp/psqr.pdf" target="_blank">P<sup>2</sup> algorithm</a> in detail. It is however worth outlining some of the pros and cons of P<sup>2</sup> algorithm.</p> <h2>Pros</h2> <ul> <li>The amount of memory per percentile value calculated is fixed <li>The execution time for calculating a given percentile when a new observation is added is linear</li></ul> <h2>Cons</h2> <ul> <li>The algorithm does not store the observations there for it performs an estimation of a given percentile. The estimate will deviate from the actual value. The P2 does not have an upper bound for this error. In practical terms the deviation is not that significant as will be shown later in this post</li></ul> <h3>P<sup>2</sup> Implementation</h3> <p>To use the PSquared type to calculate a number of percentiles an instance is constructed passing a list of percentiles expressed as values in the range of 0 – 1.</p><pre class="brush: csharp;">var pSquared = new PSquared(new List<double> {0d, 0.1d, 0.2d, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1});
</pre>
<p>As samples of data are generated or measured they are added to the instance</p><pre class="brush: csharp;">pSquared.AddSample(d);
</pre>
<p>To calculate a given percentile value call the Result method passing the percentile to calculate</p><pre class="brush: csharp;">var percentile = pSquared.Result(0.9);</pre>
<h3>P<sup>2</sup> Performance</h3>
<p>Using test data generated with <a href="https://www.r-project.org/">R</a> using the following functions <strong>x <-runif(200)</strong>, <strong>quantile(x)</strong>. This produced test data and expected quantiles values to compare against those generated by the P<sup>2</sup> implementation.</p>
<p>
<table cellspacing="0" cellpadding="2" width="401" border="1">
<tbody>
<tr bgcolor="#4f81bd">
<td valign="top" width="147"><strong>Expected</strong></td>
<td valign="top" width="124"><strong>Actual</strong></td>
<td valign="top" width="128"><strong>% Error</strong></td></tr>
<tr>
<td valign="top" width="147">0.002780183</td>
<td valign="top" width="137">0.00471922292652855</td>
<td valign="top" width="159">-37.7260207657616</td></tr>
<tr>
<td valign="top" width="147">0.250419949</td>
<td valign="top" width="137">0.247412034395431</td>
<td valign="top" width="159">0.803984453455285</td></tr>
<tr>
<td valign="top" width="147">0.515937582</td>
<td valign="top" width="137">0.508948571191173</td>
<td valign="top" width="159">0.90717849799471</td></tr>
<tr>
<td valign="top" width="147">0.757101159</td>
<td valign="top" width="137">0.747366391997607</td>
<td valign="top" width="159">0.860886247617353</td></tr>
<tr>
<td valign="top" width="147">0.996617917</td>
<td valign="top" width="137">0.992925258643177</td>
<td valign="top" width="159">0.2473180945208</td></tr></tbody></table></p>
<p>As can be seen from the results that the error in all percentile is less than <strong>1%</strong>. When the execution time is measured per call to the <em>AddSample()</em> method the average execution time is <strong>1 us.</strong></p>The code is available here: <span style="font-size: large"><a href="https://github.com/DerekGn/Dumping-Core-Memory/tree/master/Statistics" target="_blank">GitHub Statistics</a></span>Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-29062165248243485182016-09-28T20:28:00.000+01:002016-09-28T20:32:55.928+01:00Interoperability between Nanopb and googles Protocol BuffersAs part of a project to build an AVR controlled toaster oven reflow controller I decided to use Googles <a href="https://developers.google.com/protocol-buffers/">Protocol Buffers</a> and <a href="https://github.com/nanopb/nanopb">Nanopb</a> to serialise the command and control data between the .net pc software and the atmega32u2 based control board. The details of the toaster oven reflow controller will be covered in a separate blog entry.<br>There are where a number of compatibility issues between the two libraries that I found workarounds for. I didn't find a huge amount of information about this topic on line so I decided to share what I found.<br> <h2>Version Compatibility</h2>The first issue is one of <a href="https://developers.google.com/protocol-buffers/">Protocol Buffers</a> and <a href="https://github.com/nanopb/nanopb">Nanopb</a> protocol version compatibility. <a href="https://github.com/nanopb/nanopb">Nanopb</a> only supports version 2, proto2 definitions, and the current .net <a href="https://www.nuget.org/packages/Google.Protobuf">Google.Protobuf</a> only supports version 3, proto3 definitions.<br>For nanopb a message must be defined as follows:<br><pre class="brush: cpp;">syntax = "proto2";
message Request {
enum RequestType {
REQUESTONE = 0;
REQUESTTWO = 1;
}
required RequestType Command = 1;
}
</pre>For Google Protocol Buffers the corresponding message must be defined as follows:<br><pre class="brush: cpp;">syntax = "proto3";
message Request {
enum RequestType {
REQUESTONE = 0;
REQUESTTWO = 1;
}
RequestType Command = 1;
}
</pre>The version 3.0.0 release of Google Protocol Buffers proto2 is not supported, the <i><b>required</b></i> keyword has been deprecated as it not supported in proto3. <br><br>
<h2>Stack space and error strings</h2>
<p>Nanopb is designed to run on memory constrained devices. The atmega32u2 device has 1024 bytes of SRAM available for stack and heap storage. I discovered during the course of testing the controller that the code on the atmega32u2 device could not decode messages that contained more than 3 or 4 fields. Using the debugger I determined that this was due to stack overrun. Luckily the only side effect as there was not a huge amount of free SRAM memory available on the device. Looking at the memory usage of the Nanopb code I could see that there is some overhead in the definition of the fields for a message. Each field defined consumes 10 bytes of SRAM for the number of fields defined in the messages that was 240 bytes.<br><br>When the atmega32u2 code is complied the static analysis of the compiler indicates the following:</p>
<p>Program Memory Usage : 17732 bytes 54.1 % Full<br>Data Memory Usage : 747 bytes 72.9 % Full </p>
<p>Looking through the code for Nanopb I noticed that the error strings in the stream methods where not loaded from flash memory. Which is default behaviour in AVR code unless steps are taken to instruct the compiler, using the <a href="http://www.atmel.com/webdoc/AVRLibcReferenceManual/pgmspace_1pgmspace_strings.html">PROGMEM</a> macro to locate the literal string constants in flash.</p><pre class="brush: cpp;">PB_RETURN_ERROR(stream, "end-of-stream");
</pre>
<p>Launching the debugger and browsing through the IRAM of the device shows the strings taking up memory.</p>
<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcAU726QSZ16dpngx2B1OmEnpfnQzPWDMmlArWQIrYgSyWn9elSbqP0eae6XHhs75KD0fk4G3-qFRyeCIXfG9aPVhm95l_v1_rV_kejWwX-GCKw-c7kdB80FlBrbniwCCTAEuAXRvzUM8/s1600-h/image%25255B3%25255D.png"><img title="image" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8_6cPNUZllrGKz-M8h2TZolas7rl23kGtYTf461eQGGwy9RRe8BBAYlZABYPAQvTy9-oqqSKMx6XmMlIYm8VbAtLBYrm2AWNxqw8uV7yn66uUg2-xedkXE9IMujVWDkIG3ZbTW0mrW6E/?imgmax=800" width="644" height="189"></a><br>Nanopb does support turning off error messages by using a compiler flag PB_NO_ERRMSG. Defining this flag for the build results in the following:</p>
<p>Program Memory Usage : 16248 bytes 49.6 % Full<br>Data Memory Usage : 377 bytes 36.8 % Full</p>
<p>Basically a small, ~5% reduction in flash memory but a 50% reduction in SRAM usage.</p>Derek Goslinhttp://www.blogger.com/profile/07515915090983714883noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-23728673330078349352015-10-28T20:05:00.000+00:002015-10-28T20:43:45.683+00:00SAMA5D3 Xplained .Net Micro Framework Port<p>I have been working on a port of the .net micro framework for the SAMA5D3 Xplained board for the last few months on and off as I have had the time. The port is based on the 4.4 code of the recent <a href="https://github.com/NETMF/netmf-interpreter/releases/tag/v4.4-RTW-20-Oct-2015">.Net Micro Framework</a> release.</p> <p>I have now reached the first milestone I have a partially functioning <em>PortBooter</em>. It initialises the hardware and loads the <em>PortBooterDecompressor</em> into SRAM. The <em>PortBooterDecompressor</em> then proceeds to load the <em>PortBooter</em> application into DRAM and jumps to the <em>BootEntry</em> entry point.</p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgo6MWQHFEvaaefXF2FCst-ZieNveI2RoKc7fcb3BOt8uS2YmqdKa_F19CvQc9lbQkZPKoON5Aq-r_VclvSX69uG9VgXyOMMw7QIjPI8EksaWWDcYq1s6DhATP_Ak8lzXWg38RxveKUpsUu/s1600-h/image3.png"><img title="image" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="image" src="http://lh3.googleusercontent.com/-NOUQep75DTI/VjEzgEfa9fI/AAAAAAAAAbc/6m-XATNC7ug/image_thumb1.png?imgmax=800" width="644" height="406"></a></p> <p>The next pieces of work required are to implement the flash and storage drivers. Build the <em>PortBooterClient</em> and test its interaction with the <em>PortBooter.</em></p> <p>Later I will develop the additional drivers for the other hardware devices such as USB, I2C, SPI etc. There is still some way to go but I think I have gotten over the first hurdle.</p> <p>At some point in the future I may simplify the boot process as the decompressed boot loader is not necessary as the <em>PortBooter</em> should fit in the SRAM section of the SAMA5D3 device.</p> Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-42839583352113082532015-04-07T18:59:00.000+01:002015-04-07T20:08:53.876+01:00Who is ELMO GMAS<p>I recently purchased an <a href="http://www.atmel.com/tools/ATSAMA5D3-XPLD.aspx" target="_blank">ATMEL SAMA5D3 Xplained</a> board play around with developing some bare metal ARM code and possibly at some point in the future when I learn a bit more port the .net micro framework to this board. Also this board can run a few different flavours of embedded Linux and is supported by the GCC compiler.</p> <p>One of the first issues that I came across is when I connected this board was that it is detected as an ELMO GMAS device.</p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVYKfqHFgtZqcku2Mm4NI5rf3-BLDc5KDz_WHI2LdFtyQU3hMF8k_ZflZNgc1xPNDTOiBgmOwTGigk2AqitzB7V3lwChydZCHmOpwKXTl0WhVzLEbM6cN7yHX2O35JD9gErQeqY_zMAMyR/s1600-h/image2.png"><img title="image" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="image" src="http://lh4.ggpht.com/-z-VyzvIj_fg/VSQoggANryI/AAAAAAAAAZM/ypaGNE4MsLA/image_thumb.png?imgmax=800" width="137" height="244"></a></p> <p>From a quick search I found very little info on why the board was detected as an ELMO GMAS device. Being unsure if this incorrect detection would cause problems with some of the flash tools I set about fixing the problem.</p> <p>First step is to install the latest <a href="http://www.atmel.com/tools/atmelsam-bain-systemprogrammer.aspx" target="_blank">SAM-BA</a> tool from ATMEL. This contains the correct driver definition for the board.</p> <ul> <li>Right click on the ELMO GMAS device and click update driver <li>Select Browse my computer for driver software</li></ul> <blockquote> <p><a href="http://lh6.ggpht.com/-8jjCJbi1e0E/VSQohPYZccI/AAAAAAAAAZU/LZCASf54dgo/s1600-h/image%25255B3%25255D.png"><img title="image" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhO3342fWs7R-jWD8D_nTI7Wtqhkx2CmwOidIfmRsXHIVBBXj0LHkmhRnw9H2Ok7bLk7muxuGiExncH8-uCuu-9HI_1K8n1_FtALZ88B6iGj5jUirnNimHhbgdXYqdGCHwWRpohGaIBNF4u/?imgmax=800" width="244" height="180"></a></p></blockquote> <ul> <li>Select <em>Let me pick from a list of device drivers on my computer</em></li></ul> <blockquote> <p><a href="http://lh4.ggpht.com/-jOXJS7POBoM/VSQoiSEapbI/AAAAAAAAAZk/e3ghDzG2qgs/s1600-h/image%25255B9%25255D.png"><img title="image" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="image" src="http://lh5.ggpht.com/-rk_EdKxJcsQ/VSQoiyZVr5I/AAAAAAAAAZs/YILdMIpykBk/image_thumb%25255B3%25255D.png?imgmax=800" width="244" height="180"></a></p></blockquote> <ul> <li>Click have disk</li></ul> <blockquote> <p><a href="http://lh6.ggpht.com/-FTg03UmqXE0/VSQojl1bQmI/AAAAAAAAAZ0/42DCj1Pzdlg/s1600-h/image%25255B12%25255D.png"><img title="image" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="image" src="http://lh5.ggpht.com/-gUEkeJMwwds/VSQokLncLII/AAAAAAAAAZ8/cxEFmk1WcdI/image_thumb%25255B4%25255D.png?imgmax=800" width="244" height="180"></a></p></blockquote> <ul> <li>Browse to the driver folder of SAM-BA</li></ul> <blockquote> <p><a href="http://lh3.ggpht.com/-elkgG46BeyI/VSQok0fbq2I/AAAAAAAAAaE/wKJ4d5K-N1o/s1600-h/image%25255B15%25255D.png"><img title="image" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="image" src="http://lh4.ggpht.com/-qDgL2Vc5Jl4/VSQolSL6eiI/AAAAAAAAAaM/H0WQx2noqyk/image_thumb%25255B5%25255D.png?imgmax=800" width="244" height="132"></a></p></blockquote> <ul> <li>Click ok then click next</li></ul> <blockquote> <p><a href="http://lh6.ggpht.com/-QQ2qTHRFQC8/VSQolxFC1ZI/AAAAAAAAAaU/00mo7u3QrLg/s1600-h/image%25255B18%25255D.png"><img title="image" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="image" src="http://lh4.ggpht.com/-yUK5IZC8twY/VSQomd9i36I/AAAAAAAAAac/W9khojjksq4/image_thumb%25255B6%25255D.png?imgmax=800" width="244" height="180"></a></p></blockquote> <ul> <li>A warning will be displayed, click ok as we are sure that this is the correct driver</li></ul> <blockquote> <p><a href="http://lh5.ggpht.com/-NVSRVckFe0k/VSQom6uFjCI/AAAAAAAAAak/5Jp_AYK8EAE/s1600-h/image%25255B21%25255D.png"><img title="image" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="image" src="http://lh4.ggpht.com/-NmzyiKYuT7s/VSQonpK56tI/AAAAAAAAAas/6SEZWJ1g94Y/image_thumb%25255B7%25255D.png?imgmax=800" width="244" height="108"></a></p></blockquote> <ul> <li>The driver will be installed and the device will show up as an AT91 USB to Serial converter</li></ul> <blockquote> <p><a href="http://lh3.ggpht.com/-uAue997ohmI/VSQooPm-RHI/AAAAAAAAAa0/tNn4a2r0Q_c/s1600-h/image%25255B24%25255D.png"><img title="image" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="image" src="http://lh5.ggpht.com/-K4boXxwqaaA/VSQoo-99TVI/AAAAAAAAAa8/39pgHPCfCEs/image_thumb%25255B8%25255D.png?imgmax=800" width="244" height="180"></a></p></blockquote> <p>When the board is booted up in BootROM mode i.e. the JP5 jumper is removed you may have to install a driver for the device. The same AT91 driver can be used</p> Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-62147171858926911672014-06-01T18:00:00.000+01:002016-10-21T21:14:21.806+01:00Part2: Creating a SAML 2.0 Token using WIF 4.5In <a href="http://corememorydump.blogspot.ie/2014/05/part1-creating-saml-20-token-using-wif.html" target="_blank">Part 1</a> I demonstrated how to generate a simple SAML2 token using the WIF 4.5 libraries. To finish this out I will now describe how to read a SAML2 token that was previously generated with the <b>SamlTokenProcessor</b> class.<br />
<h2>
Read Token</h2>
In order to read an encrypted and signed SAML2 token an instance of a <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.tokens.securitytokenhandler%28v=vs.110%29.aspx" target="_blank">SecurityTokenHandler</a> class is required that has the correct configuration. Specifically we need an instance of an <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.tokens.saml2securitytokenhandler%28v=vs.110%29.aspx" target="_blank">Saml2SecurityTokenHandler</a> class. It is possible to create an instance of this class and configure it through code, an alternate approach is to use the WIF configuration classes to configure and create an instance of a <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.tokens.saml2securitytokenhandler%28v=vs.110%29.aspx" target="_blank">Saml2SecurityTokenHandler</a>. This is a preferred approach because it provides some level of flexibility to the <b>SamlTokenProcessor.</b> Below is an example of how to create the <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.tokens.saml2securitytokenhandler%28v=vs.110%29.aspx" target="_blank">Saml2SecurityTokenHandler</a> from the <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.configuration.identityconfiguration%28v=vs.110%29.aspx" target="_blank">IdentityConfiguration</a>.<br />
<pre class="brush: c-sharp; toolbar: false">IdentityConfiguration configuration = new IdentityConfiguration();
_saml2SecurityTokenHandler = configuration.SecurityTokenHandlers[typeof(Saml2SecurityToken)];
_saml2SecurityTokenHandler.Configuration.ServiceTokenResolver =
new X509CertificateStoreTokenResolver(StoreName.My, StoreLocation.CurrentUser);</pre>
<br />
The <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.tokens.saml2securitytokenhandler%28v=vs.110%29.aspx" target="_blank">Saml2SecurityTokenHandler</a> instance requires a security token resolver. This is a class that is able to resolve the signing and encrypting certificates that where originally used to create the SAML2 token. The <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.configuration.identityconfiguration%28v=vs.110%29.aspx" target="_blank">IdentityConfiguration</a> class expects a configuration section in the application configuration file.<br />
<pre class="brush: xhtml; toolbar: false"><configSections>
<section name="system.identityModel" type="System.IdentityModel.Configuration.SystemIdentityModelSection, System.IdentityModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" />
</configSections></pre>
<br />
The configuration required for the example application is as follows, this configuration is a basic configuration which is used for token validation, more on this later. The configuration can be used to configure replay detection for tokens or the maximum clock skew between the token generator and the token consumer.<br />
<pre class="brush: xhtml; toolbar: false"><system.identityModel>
<identityConfiguration>
<audienceUris>
<add value="http://rp.com/" />
</audienceUris>
<issuerNameRegistry type="System.IdentityModel.Tokens.ConfigurationBasedIssuerNameRegistry, System.IdentityModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<trustedIssuers>
<add thumbprint="0D274A03BDCA95015E712E2808F917C93159ED1B"/>
</trustedIssuers>
</issuerNameRegistry>
</identityConfiguration>
</system.identityModel></pre>
<br />
<b><span style="color: red;">Note: The thumbprint element value must be set to the thumbprint of the STS identity certificate that was generated or acquired from a CA. Also be careful copy and pasting the thumbprint as it can copy with BOM data in the string.</span></b><br />
<br />
Now that we have an instance of a <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.tokens.saml2securitytokenhandler%28v=vs.110%29.aspx" target="_blank">Saml2SecurityTokenHandler</a> it can be used to read a token from a stream.<br />
<pre class="brush: c-sharp; toolbar: false">public SecurityToken ReadSecurityToken(Stream stream)
{
SecurityToken securityToken;
using(XmlReader rdr = new XmlTextReader(stream))
{
securityToken = _saml2SecurityTokenHandler.ReadToken(rdr);
}
return securityToken;
}</pre>
<br />
<h2>
Validate Token</h2>
<br />
The <b>VerifySecurityToken</b> method of the <b>SamlTokenProcessor</b> class can verify a SAML2 token against the configuration specified in the configuration file. A prerequisite to verifying the SAML2 token generated is to ensure that the signing token is present in the certificate store under TrustedPeople.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9fDrxZdAtX4GDfbbe7prvcTt86yIUiJIDhOJTod6OQbnp4tK4trl_s1IZeWPnQzKfZntx3LgQyF1Fd_QH4TMpzAqD6jXgi2RclcLFUTO6Y-Cgn3GwDcnqQ4BPSOJyqqZLMI7pv0Ol2tVL/s1600-h/image%25255B3%25255D.png"><img alt="image" border="0" height="302" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEil5tuHGV819DBt7hRd1NX4bVt_Ag8Je79v24WkaMBCrg8EL1NyJcjOBqYfG3BDR8gOXJ2BjhrgZOa_rCNl4PC0uEPXTMS4JhOzTkSY3HZfKPrsa426FvDNqWMfuscVn_jCgMD74TH7aOO3/?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="644" /></a><br />
<br />
The <b>VerifySecurityToken</b> method will verify the attributes of the token against the configuration. As a test the audienceUri can be modified to cause the verification to fail.<br />
<pre class="brush: xhtml; toolbar: false"><audienceUris>
<add value="http://rp2.com/" />
</audienceUris></pre>
<span style="font-family: "calibri";">When the test application is run using this configuration an <a href="http://msdn.microsoft.com/en-us/library/microsoft.identitymodel.tokens.audienceurivalidationfailedexception.aspx" target="_blank">AudienceUriValidationFailedException</a> exception will be thrown.</span><br />
<span style="font-family: "calibri";"> </span>
<br />
<span style="color: red;"><b>System.IdentityModel.Tokens.AudienceUriValidationFailedException: ID1038: The AudienceRestrictionCondition was not valid because the specified Audience is not present in AudienceUris.</b></span><br />
<span style="color: red;"><b>Audience: '<a href="http://rp.com/%27">http://rp.com/'</a></b></span><br />
<span style="color: red;"><b> at System.IdentityModel.Tokens.Saml2SecurityTokenHandler.ValidateToken(SecurityToken token)</b></span><br />
<span style="color: red;"><b> at CoreMemoryDump.Saml.SamlTokenProcessor.VerifySecurityToken(SecurityToken securityToken) in c:\Users\Office\Documents\Visual Studio 2012\Projects\CoreMemoryDump.Saml\CoreMemoryDump.Saml\SamlToken</b></span><br />
<span style="color: red;"><b>Processor.cs:line 92</b></span><br />
<span style="color: red;"><b> at CoreMemoryDump.SamlTest.Program.Main(String[] args) in c:\Users\Office\Documents\Visual Studio 2012\Projects\CoreMemoryDump.Saml\CoreMemoryDump.SamlTest\Program.cs:line 47</b></span><br />
<br />
Similarly modifying the TrustedIssuers list will elicit an exception indicating that the issuer is not trusted.<br />
<br />
<h2>
Summary</h2>
<br />
The intent of this example code was to show that it is possible to generate, read and verify a SAML token using the base libraries from the WIF 4.5 framework. This example provides a basic implementation that can be extended to enable greater flexibility for generating SAML tokens. <br />
<br />
<br />
The code is available here: <span style="font-size: large;"><a href="https://github.com/DerekGn/Dumping-Core-Memory/tree/master/DumpingCoreMemory.Saml" target="_blank">GitHub DumpingCoreMemory.Saml</a></span>Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-14234813023045325102014-05-01T19:58:00.000+01:002016-10-21T21:15:09.648+01:00Part1: Creating a SAML 2.0 Token using WIF 4.5To create a SAML token we will need a set of certificates for the Relying Party (RP) and Secure Token Service (STS). These certificates plus keys will be used for the signing and encryption of the SAML 2.0 token. In the part 2 of this series the keys will be used for signature verification and decryption.<br />
First we need a certificate for the certificate authority that will be used as the issuer for the STS and Relying Party. Note you will be prompted for a password for the CA private key file, use a simple password as you will be prompted for this password when creating the certificates for the RP.<br />
<strong><span style="color: black;">makecert -n "<span style="color: darkred;">CN=RootCATest</span>" -r -sv RootCATest.pvk RootCATest.cer</span></strong><br />
To create the certificate for the STS and Relying Party the following command can be executed from a batch file.<br />
<strong>makecert -sk %1 -iv RootCATest.pvk -n CN=%1 -ic RootCATest.cer –sr localmachine -ss my -sky exchange -pe %1.cer</strong><br />
For example <strong><em>createcert.bat RP</em></strong> will create the certificate for the RP. Note that all of the certificate creation commands must be executed from a VS2012/2010 command prompt that is running with administrator privileges.<br />
Later we will place the certificates in the relevant certificate stores for use later in the process of developing the SAML token generator.<br />
<h2>
Token Generation</h2>
To generate a SAML token a <a href="http://msdn.microsoft.com/query/dev11.query?appId=Dev11IDEF1&l=EN-US&k=k%28System.IdentityModel.Tokens.SecurityTokenDescriptor%29;k%28SecurityTokenDescriptor%29;k%28TargetFrameworkMoniker-.NETFramework,Version%3Dv4.5%29;k%28DevLang-csharp%29&rd=true" target="_blank">SecurityTokenDescriptor</a> must be created. This is a place holder for the attributes of an issued token.<br />
<pre class="brush: c-sharp; nogutter: true; toolbar: false">private SecurityTokenDescriptor BuildBaseSecurityTokenDescriptor(Uri appliesToAddress, String issuer, TimeSpan validityPeriod,
IEnumerable<Claim> claims, X509Certificate2 encryptingCertificate, X509Certificate2 signingCertificate)
{
DateTime utcNow = DateTime.UtcNow;
SecurityTokenDescriptor securityTokenDescriptor = new SecurityTokenDescriptor();
securityTokenDescriptor.AppliesToAddress = appliesToAddress.ToString();
securityTokenDescriptor.TokenIssuerName = issuer;
securityTokenDescriptor.Lifetime = new Lifetime(utcNow, utcNow.Add(validityPeriod));
securityTokenDescriptor.SigningCredentials = new X509SigningCredentials(signingCertificate);
securityTokenDescriptor.Subject = new ClaimsIdentity(claims);
securityTokenDescriptor.TokenType = "http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV2.0";
securityTokenDescriptor.EncryptingCredentials = new EncryptedKeyEncryptingCredentials(encryptingCertificate);
return securityTokenDescriptor;
}</pre>
<br />
To generate the token a <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.tokens.securitytokenhandlercollection%28v=vs.110%29.aspx" target="_blank">SecurityTokenHandlerCollection</a> is constructed with a collection of SecurityToken handlers. To generate a SAML token we use a <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.tokens.saml2securitytokenhandler%28v=vs.110%29.aspx" target="_blank">Saml2SecurityTokenHandler</a>, to generate the Saml2 token and a <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.tokens.encryptedsecuritytokenhandler%28v=vs.110%29.aspx" target="_blank">EncryptedSecurityTokenHandler</a> to encrypt and sign the token when the token is serialized.<br />
<pre class="brush: c-sharp; toolbar: false">public SamlTokenProcessor()
{
_securityTokenHandlerCollection = new SecurityTokenHandlerCollection();
_securityTokenHandlerCollection.Add(new Saml2SecurityTokenHandler());
_securityTokenHandlerCollection.Add(new EncryptedSecurityTokenHandler());
}</pre>
<br />
Calling the <strong>CreateSecurityToken</strong> method generates a <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.tokens.saml2securitytoken%28v=vs.110%29.aspx" target="_blank">Saml2SecurityToken</a> instance.<br />
<pre class="brush: c-sharp; toolbar: false">SamlTokenProcessor processor = new SamlTokenProcessor();
X509Certificate2 encryptingCertificate = CertificateResolver.ResolveCertificate(StoreLocation.LocalMachine, StoreName.My, "CN=RP");
X509Certificate2 signingCertificate = CertificateResolver.ResolveCertificate(StoreLocation.LocalMachine, StoreName.My, "CN=STS");
List<Claim> claims = new List<Claim>() { new Claim("Claim", "Value") };
SecurityToken token = processor.CreateSecurityToken(new Uri("http://rp.com"), "sts", new TimeSpan(0, 0, 10), claims,
encryptingCertificate, signingCertificate);
using(Stream stream = new FileStream("token.xml", FileMode.OpenOrCreate))
{
processor.SerializeSecurityToken(token, stream);
}</pre>
<br />
The <strong>SerializeSecurityToken</strong> method serializes the token to a stream. This applies the signing and encryption to the resultant tokens XML representation. Below is an example token shortenedshortend for brevity.<br />
<pre class="brush: c-sharp; toolbar: false"><?xml version="1.0" encoding="utf-8"?>
<EncryptedAssertion xmlns="urn:oasis:names:tc:SAML:2.0:assertion">
<xenc:EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element" xmlns:xenc="http://www.w3.org/2001/04/xmlenc#">
<xenc:EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" />
<KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
<e:EncryptedKey xmlns:e="http://www.w3.org/2001/04/xmlenc#">
<e:EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p">
<DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
</e:EncryptionMethod>
<KeyInfo>
<o:SecurityTokenReference xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
<X509Data>
<X509IssuerSerial>
<X509IssuerName>CN=RootCATest</X509IssuerName>
<X509SerialNumber>78425684050690387325969986111748521388</X509SerialNumber>
</X509IssuerSerial>
</X509Data>
</o:SecurityTokenReference>
</KeyInfo>
<e:CipherData>
<e:CipherValue>NSmOOCR6CzGCj+lY2PF....</e:CipherValue>
</e:CipherData>
</e:EncryptedKey>
</KeyInfo>
<xenc:CipherData>
<xenc:CipherValue>e5kGcfcel/YN+Qjz....</xenc:CipherValue>
</xenc:CipherData>
</xenc:EncryptedData>
</EncryptedAssertion></pre>
<br />
In the second part of this two part blog I will demonstrate how to reverse the process and extract the claims from the encrypted SAML2 token.<br />
<br />
The code is available here: <span style="font-size: large;"><a href="https://github.com/DerekGn/Dumping-Core-Memory/tree/master/DumpingCoreMemory.Saml" target="_blank">GitHub DumpingCoreMemory.Saml</a></span>Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-59215813972293818582014-04-01T00:00:00.000+01:002014-06-27T20:17:51.659+01:00Creating and Consuming SAML 2.0 Tokens using WIF<p>With the introduction of <a href="http://msdn.microsoft.com/en-us/library/hh377151%28v=vs.110%29.aspx" target="_blank">Windows Identity Foundation 4.5</a> Microsoft removed the Visual Studio templates that could be used to generate a custom WCF STS service. The custom STS could then be used to issue SAML 2.0 tokens based on custom credentials passed with the WS-Trust request to the custom STS.</p> <p>The underlying classes are still available to use for creating a custom STS, without the templates available it is still possible to create a custom WCF STS service but it requires a bit more effort.</p> <p>There is however an alternate means to generate SAML 2.0 tokens using WIF that forgoes the need to create a custom STS. This is useful in scenarios involving proprietary systems that have custom protocols and authentication, that also do not lend themselves well to integration with ADFS etc.</p> <p>To explain how this can be achieved I will break the problem down into two parts:</p> <p><a href="http://corememorydump.blogspot.ie/2014/05/part1-creating-saml-20-token-using-wif.html" target="_blank">Part 1: Creating a SAML 2.0 Token using WIF 4.5</a></p> <p>Part2: Consuming a SAML 2.0 Token using WIF 4.5</p> Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-20409811399348230682014-03-22T16:47:00.001+00:002014-03-22T16:49:50.061+00:00mfNordic library released on CodeplexAs part of the development effort for the home monitoring system I implemented a .net micro framework library for interfacing to a <a href="http://www.nordicsemi.com/" target="_blank">Nordic semiconductor</a> <a href="http://www.nordicsemi.com/eng/Products/2.4GHz-RF/nRF24L01P" target="_blank">NRF24L01P</a> radio module. I have released the source code on <a href="https://mfnordic.codeplex.com/" title="https://mfnordic.codeplex.com/">https://mfnordic.codeplex.com/</a> under the MS-PL license.<br />
<br />
<h2>
Using the library</h2>
<pre class="brush: c-sharp">var radio = new Nrf24L01P(SPI_Devices.SPI1, Pins.GPIO_PIN_D0, Pins.GPIO_PIN_D1, Pins.GPIO_PIN_D2);</pre>
<br />
The Nrf24L01P class needs the SPI port, the chips select, chip enable and interrupt line that the physical radio device is connected too.<br />
<br />
The radio can be left with the default settings or can be configured via properties. Below are some example properties.<br />
<br />
<pre class="brush: c-sharp"> radio.Features = radio.Features | Feature.AckPayload | Feature.DynamicPayload;
//Note the number of address bytes must match the configured/default radio.AddressWidth property
radio.TransmitAddress = new byte[] {0,0,0,0,0};
radio.Mode = Mode.Receiver;
radio.Channel = 125;</pre>
<br />
The class will raise events when data is transmitted or received or errors occur. These events can be hooked, each has an event argument that contains relevant context information.<br />
<br />
<pre class="brush: c-sharp"> _radio.OnDataReceived += RadioOnDataReceived;
_radio.OnTransmitFailed += RadioOnTransmitFailed;
_radio.OnTransmitSuccess += RadioOnTransmitSucess;
_radio.OnDataReceiveFailure += RadioDataReceivedFailure;
static void RadioDataReceivedFailure(object sender, DataReceiveFailureEventArgs e)
{
Debug.Print("OnDataReceiveFailure occured on Pipe: " + e.Pipe);
}
static void RadioOnDataReceived(object sender, DataReceivedEventArgs e)
{
Debug.Print("RadioOnDataReceived occured on Pipe: " + e.Pipe + " " + e.Payload.Length + "bytes received");
}</pre>
<br />
The NRF24L01P supports up to 6 logical radio data pipelines, all sharing some basic settings namely:<br />
<br />
<ul>
<li>CRC enabled/disabled (CRC always enabled when Enhanced ShockBurst™ is enabled) </li>
<li>CRC encoding scheme </li>
<li>RX address width </li>
<li>Frequency channel </li>
<li>Air data rate </li>
<li>LNA gain</li>
</ul>
<br />
The library exposes the data pipelines via an indexer. This allows properties and methods of individual pipes to be accessed.<br />
<br />
<pre class="brush: c-sharp"> radio[Pipe.Pipe0].AutoAcknowledgment = true;
radio[Pipe.Pipe0].DynamicPayload = true;
radio[Pipe.Pipe0].Enabled = true;</pre>
<br />
Acknowledge packets can be queued on a given pipe line using the QueueAcknowledgePacket method.<br />
<br />
<pre class="brush: c-sharp"> radio[Pipe.Pipe0].QueueAcknowledgePacket(new byte[] {0,1,2,3});</pre>
<br />
In transmitter mode packets can be transmitted using the Transmit method. The Boolean parameter indicates if the packet should be transmitted with an expected acknowledge from the receiver.<br />
<br />
<pre class="brush: c-sharp"> radio.Transmit(new byte[] { 0, 1, 2, 3 }, false);</pre>
<br />
Finally to power the radio devices RF section up.<br />
<br />
<pre class="brush: c-sharp"> radio.PowerUp = true;</pre>
<br />
To enable the radio device, which will set the CE signal to the device, which will active the RX or TX mode of the radio device.<br />
<br />
<pre class="brush: c-sharp"> radio.Enable = true;</pre>
Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-6758982873320058922014-03-19T00:00:00.000+00:002014-03-20T21:18:44.294+00:00Home Monitoring System – Version 1.0<p>Having worked on my home monitoring system on an off for the last number of weeks I have finally gotten to version 1 for the host board and the sensor nodes.</p> <h2>Temperature And Humidity Sensor Node</h2> <p>I completed the development and testing of the wireless temperature and humidity sensor nodes.</p> <p><a href="http://lh4.ggpht.com/-Otv3EoiAGwM/UyoZywqqYWI/AAAAAAAAAWI/yxb5OyvLR-A/s1600-h/WP_20140319_00114.jpg"><img title="WP_20140319_001" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="WP_20140319_001" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiM49DLqHX6w2CHofi93rfquEv24HYdiDrXUYB2YEKGdTsBZqWZ_0dPlgdzA_hvBBx-4QAVk5GWgTxzn0W2IoeKRseIiDGsVuQq-rKtoSgUuasR1A-ZBWAs0Bg01j6UNBIWWJ21Fb-M_weS/?imgmax=800" width="244" height="139"></a></p> <p>This board was installed in an enclosure as shown below. The board was designed to fit snuggly into the enclosure which takes two AAA batteries to power the sensor.</p> <p><a href="http://lh5.ggpht.com/-ODK3SmF02Aw/UyoZz2lHVPI/AAAAAAAAAWY/mQ6zkyKoNWw/s1600-h/WP_20140319_002%25255B2%25255D.jpg"><img title="WP_20140319_002" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="WP_20140319_002" src="http://lh6.ggpht.com/-WQVsT_SUU2U/UyoZ0Qi-K_I/AAAAAAAAAWc/FFTT3R6KBrA/WP_20140319_002_thumb%25255B3%25255D.jpg?imgmax=800" width="244" height="139"></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWby77AYeiXf_eNmaDmnOhxy_ta-dR5og2oHYfgogsIJloKO7RE7yPVhAXOPL_JeQeI62LdgVkoGCaryEvAln9mTmJ6_SehaIsLSr9x7d4Zc6TZirNjGQHsYQH-tD_PkeeKMen7Ci6z9JX/s1600-h/WP_20140319_003%25255B6%25255D.jpg"><img title="WP_20140319_003" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="WP_20140319_003" src="http://lh4.ggpht.com/-nq3gQm5sP4E/UyoZ1V3ILAI/AAAAAAAAAWs/KXfmcFc-O_w/WP_20140319_003_thumb%25255B3%25255D.jpg?imgmax=800" width="244" height="139"></a></p> <p>The sensor implements a simple channel scanning and handshake protocol which allows it to determine the radio frequency channel the wireless hub is listening for packets. When the channel is found by sending a <em>NodeHello</em> packet to which the wireless hub replies with a <em>NodeHelloAck</em> packet. The <em>NodeHelloAck</em> packet contains configuration data that the node uses to setup it logical RF channel.</p> <p>Once the handshake has completed the node enters a loop where it sleeps for a period, which is provided as part of the handshake data, wakes up and transmits the latest temperature, humidity and battery voltage readings to the wireless base station.</p> <h2>Wireless Hub</h2> <p>I chose a <a href="http://netduino.com/netduinoplus2/specs.htm" target="_blank">Netduino Plus 2</a> as the hardware for the wireless hub. The hardware setup is pretty straight forward. An NRF24L01P radio module is connected to power and the SPI port on the Netduino.</p> <p><a href="http://lh6.ggpht.com/-W5_5mtyBNKM/UyosgwUA3fI/AAAAAAAAAXA/6mIzkC-5kvU/s1600-h/WP_20140319_004%25255B8%25255D.jpg"><img title="WP_20140319_004" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="WP_20140319_004" src="http://lh3.ggpht.com/-YHlBRlJ3BwM/UyoshVaTQEI/AAAAAAAAAXE/ekIy_w9a8SQ/WP_20140319_004_thumb%25255B5%25255D.jpg?imgmax=800" width="244" height="139"></a></p> <p>The Netduino implements a simple web server, based on <a href="https://embeddedwebserver.codeplex.com/" target="_blank">embeddedwebserver</a>, that hosts a set of pages that renders the sensor data from each of the sensors, allows configuration changes to be made and log file viewing. Since the Netduino has limited memory and CPU most of the processing for data rending is offloaded onto the browser. The sensor data page uses a JavaScript graphing component called <a href="http://code.shutterstock.com/rickshaw/" target="_blank">Rickshaw</a>. The pages are rendered on the client side browser as a single page application using <a href="http://AngularJs.org" target="_blank">AngularJs</a> as the MVVM framework and <a href="http://getbootstrap.com/" target="_blank">Bootstrap</a> to provide a responsive UI that scales to different device screen sizes.</p> <h2>Sensor Data Display</h2> <p><a href="http://lh5.ggpht.com/-xlOa85E5vQk/UyoshyAntKI/AAAAAAAAAXQ/pVuoaVd9CsU/s1600-h/Monitoring%25255B3%25255D.png"><img title="Monitoring" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="Monitoring" src="http://lh5.ggpht.com/-wQk3Y3Pd25k/UyosieJhWcI/AAAAAAAAAXU/w4At9CM0qDE/Monitoring_thumb%25255B1%25255D.png?imgmax=800" width="431" height="484"></a></p> <p>The sensor data is rendered on the main page of the application. The interface allows selection of a number of days of data. Multiple sensor feeds can be rendered, the graphing component supports hoover details when the mouse is over a point on the graph.</p> <h2>Nodes Configuration</h2> <p><a href="http://lh3.ggpht.com/-k6cXQqUF09g/Uyosi7ZdpTI/AAAAAAAAAXc/oSeIid1u79w/s1600-h/Nodes%25255B3%25255D.png"><img title="Nodes" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="Nodes" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjkC5Dwe8JzFwKS18elGyDKK4RpjtE0x58eUtkd7ukQ5M_IXoraMvto8GftAgLUG5TZ9x4lST74hOVCztufTunL93l6bD_lbxSzCw8a7Y66pcu3O62pWFAqYHRr7Z877EUkzqKzJrRIvJ7/?imgmax=800" width="644" height="318"></a></p> <p>The nodes configuration page shows active nodes and the node types. The nodes name can be modified.</p> <h2>System Log</h2> <p><a href="http://lh4.ggpht.com/-qgrIm49mu0Y/UyoskKlVMzI/AAAAAAAAAXw/W_CyXSiqav4/s1600-h/log%25255B3%25255D.png"><img title="log" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="log" src="http://lh4.ggpht.com/-HsZUa-766yg/Uyosk5zSSmI/AAAAAAAAAX4/iGD4wCc4rxU/log_thumb%25255B1%25255D.png?imgmax=800" width="644" height="393"></a></p> <p>The logging level is set on the configuration page from 0 to 4, 0 = None 4 = Debug</p> <h2>Configuration</h2> <p><a href="http://lh5.ggpht.com/-PrywqkFdaGU/UyoslY0hz3I/AAAAAAAAAX8/jh1KTarbVTo/s1600-h/Configuration%25255B3%25255D.png"><img title="Configuration" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="Configuration" src="http://lh5.ggpht.com/-1sdf5hLgmEg/Uyosl3Fj9DI/AAAAAAAAAYE/caA2YO3TA2o/Configuration_thumb%25255B1%25255D.png?imgmax=800" width="644" height="202"></a></p> <p>The configuration page allows the following to be configured:</p> <ul> <li>The radio channel <li>The radio device address <li>The log level <ul> <li>None <li>Info <li>Warn <li>Error <li>Debug</li></ul> <li>Update interval, the rate that the sensor will push readings</li></ul> <h2>Future Additions</h2> <ul> <li>Minor web UI clean up and changes</li> <li>The ability to export sensor data</li> <li>The ability to have a node release its allocated id</li></ul> <p>I would also like to add some additional sensor types, maybe air quality, water,power etc. Later I would like to use the data gathered to directly control the space heating from the netduino, via a relay node that can be used to advance and retard the heating system based on evaluation of the temperature data. For example on a very cold day fire up the heat at an earlier time than the heating control system would normally.</p> Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com4tag:blogger.com,1999:blog-2842308426499703302.post-12587856563270862732014-01-20T11:18:00.001+00:002014-01-20T11:37:46.637+00:00Home Monitoring System – Sensor NodeFinally the boards that I designed for the home monitoring system arrived from <a href="https://www.blogger.com/oshpark.com" target="_blank">OSH Park</a>. Son now I can get down to building the the boards. The photo below shows the board in the bottom half of the enclosure that I am planning to use.<br />
<a href="http://lh5.ggpht.com/-S64QuR5LZ0w/UtlK_9L8UgI/AAAAAAAAAUg/5Wou1sFRShg/s1600-h/WP_20140116_003%25255B16%25255D.jpg"><img alt="WP_20140116_003" border="0" src="http://lh4.ggpht.com/-EZroGWaE0qI/UtlLAtc05LI/AAAAAAAAAUk/wlhHP53fA-o/WP_20140116_003_thumb%25255B13%25255D.jpg?imgmax=800" height="139" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="WP_20140116_003" width="244" /></a><br />
To ensure that I don't release the blue smoke I decided to build and test the node in a number of steps. For the first step I soldered the components of the power supply and tested the output voltage.<br />
<a href="http://lh4.ggpht.com/-Zyvi2HjVxdw/UtljSS3BmOI/AAAAAAAAAU4/VvFKOtY_qeQ/s1600-h/WP_20140116_004%25255B11%25255D.jpg"><img alt="WP_20140116_004" border="0" src="http://lh4.ggpht.com/-LvtGYGcVO04/UtljSz6nIZI/AAAAAAAAAU8/8AkcuL_9NiE/WP_20140116_004_thumb%25255B8%25255D.jpg?imgmax=800" height="139" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="WP_20140116_004" width="244" /></a><br />
This first step was successful connecting a bench power supply set to 2.8 volts yielded an output of 3.3 volts.<br />
<a href="http://lh3.ggpht.com/-9c9-addYAIs/UtljTY6It4I/AAAAAAAAAVI/tg7CtaIpzfk/s1600-h/WP_20140116_005%25255B3%25255D.jpg"><img alt="WP_20140116_005" border="0" src="http://lh3.ggpht.com/-7lZrzhk7pgU/UtljT1TOlKI/AAAAAAAAAVM/NhaYWddLUCg/WP_20140116_005_thumb.jpg?imgmax=800" height="139" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="WP_20140116_005" width="244" /></a><br />
The next step was to solder the remaining capacitors and ICs and resistors. When designing the board I used 0603 for the resistor and capacitor footprints. This was a mistake because they are difficult to hand solder due to the size. Next time I will use the next size up.<br />
<br />
The other SMD that was difficult to solder was the LCC8 humidity sensor. In fact I initially soldered it on upside down and had to remove it, in the process of which I destroyed the part. Below is an image of the final board with the debug and ISP header but without the radio module this will be added later.<br />
<a href="http://lh5.ggpht.com/-v6i-C-Xb8SI/UtljUpMl2ZI/AAAAAAAAAVU/aNypbyLdS_Q/s1600-h/WP_20140117_004%25255B3%25255D.jpg"><img alt="WP_20140117_004" border="0" src="http://lh6.ggpht.com/-fcpHzXPsJ6c/UtljVB0QzBI/AAAAAAAAAVc/J8Wi2XnmORQ/WP_20140117_004_thumb.jpg?imgmax=800" height="139" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="WP_20140117_004" width="244" /></a><br />
When I initially attached the ISP I found that it would not connect to the board. The reason for this was that the ISP clock was set to the default of 1Mhz. The ATmega48PA is shipped with internal RC oscillator at 8.0MHz and with the fuse CKDIV8 programmed, resulting in 1.0MHz system clock. Reducing the ISP clock to 1/4 of the CPU frequency resolved the issue.<br />
<a href="http://lh4.ggpht.com/-rjnVZRcjFTs/UtqwrnkT1nI/AAAAAAAAAVw/Ygn-11jpXWc/s1600-h/AVRSettings3.png"><img alt="AVRSettings" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8gpEnPZ_X646qlnBj_3YMeNPWUPFLBM_cmBRRHiTdd_Brzoxedn2VcDv8gtRmbKtlk4LiUt944sRtUkDjR76dZzV3dZBZzAkGjP49HCni_pNA8TxEmdwcHz0KtvHlFRhmHwFyGv3uSgNT/?imgmax=800" height="484" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="AVRSettings" width="564" /></a><br />
When I resolved the connection issue to the device I was able to flash the firmware and start testing the sensor board. The first check was to determine if the ADC voltage monitoring circuit worked. This is where I ran into another problem. I powered the board with a bench power supply @ 2.8 volts. Reading the battery voltage via the ADC resulted in a value of 2.1 volts. After examining the battery sensing circuit it initially looked like the p channel FET was not switching on fully causing a voltage drop of 0.6 volts across the Source to Gate. Checking the Drain voltage measured 2.1 volts which was wrong because it was not switched on, i.e. 0 volts applied.<br />
<br />
On a hunch I checked the datasheet of the FET against the footprint on the board. I discovered the PMBFJ177 footprint is different the Gate and Drain are interchanged. I will need to get another J177 FET that matches the board footprint.<br />
<br />
Once these issues where resolved I was able to debug the code that reads the humidity and temperature from the <span class="st">CC2D33S sensor </span>using the AVR dragon.<br />
<br />
The next step is to add the radio module and figure out a strategy to debug and test the radio module and its code.Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-75229946051149146762014-01-04T00:00:00.000+00:002014-01-06T13:48:45.691+00:00Home Monitoring System – Phase 1<p>Having spent a great deal of time and effort upgrading my 1970s built home to make it habitable in the winter, yes it was really that bad, and as an added benefit improve its energy efficiency. Now that the improvements have been made I would now like to monitor the effectiveness or lack thereof if that happens to be the case. Unfortunately I have will have no data to compare the data acquired from this proposed system, on the up side I will have data to hopefully make further improvements.</p> <p>My initial plan is to focus on <a href="http://www.merriam-webster.com/dictionary/space%20heating" target="_blank">space heating</a> as this is generally the biggest pain point in older houses.</p> <h2>Goals</h2> <ul> <li>Capture Temperature and Humidity <ul> <li>Temperature and Humidity are related and have an effect on the <a href="http://en.wikipedia.org/wiki/Thermal_comfort" target="_blank">Thermal Comfort</a> of the living space</li></ul> <li>Small Size Sensors <ul> <li>The sensors must be small and inconspicuous as I will be placing a number of them around the house</li></ul> <li>Battery powered <ul> <li>AA or AAA batteries, preferably AAA due to the size requirement</li></ul> <li>Efficient Power Usage <ul> <li>Aim to be operational for minimum of a year without requiring a battery change</li></ul></li></ul> <h2>Solution</h2> <p><a href="http://lh6.ggpht.com/-1ntyl5RI7jc/Ushm422S9UI/AAAAAAAAATA/XGT0TIsgsGQ/s1600-h/Sensor%25255B10%25255D.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px; padding-top: 0px" title="Sensor" border="0" alt="Sensor" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHNTksI1Q0U_JPAAay3AR_jkWjw0PO2RPevhVcN9MSXLkJV9VYg_zotH44E4P0L0BWg-YAqahegzUMQX_J1K8DZYI1gYiUY-MFgjoFexiaoDSEKHhYMomEXFg0Y-bAgVccSRqad_CUypJN/?imgmax=800" width="358" height="484"></a></p> <p>For phase 1 the wireless sensors will communicate with the Control Hub which will log and aggregate the data that the sensor nodes periodically send. The control hub will expose a set of web pages to manage the sensor network devices and view and export data.</p> <h5>Sensor Node</h5> <p>The sensor node is based around an Atmel <a href="http://www.atmel.com/devices/atmega48pa.aspx" target="_blank">ATMEGA48PA</a> 8-bit CPU. This was chosen due to the fact that it is a picoPower device which means that it is capable minimising power down to 0.1µA in sleep mode. </p> <p>The wireless link will use an <a href="http://www.nordicsemi.com/eng/Products/2.4GHz-RF/nRF24L01P/%28language%29/eng-GB" target="_blank">nRF24L01+</a> module these are extremely cheap and can be picked up on eBay for 80 cent which is remarkable given the chip itself is about 2-3 euro in single unit prices. This device operates in the 2.4Ghz space and is optimised for low power applications. Consuming sub μA current in power down mode.</p> <p>To measure humidity and temperature the sensor chosen was the <a href="http://www.google.ie/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&ved=0CD0QFjAC&url=http%3A%2F%2Fwww.ge-mcs.com%2Fdownload%2Fmoisture-humidity%2F920-558B-LR.pdf&ei=LHPIUo7oLY3d7QaizIGQCw&usg=AFQjCNFzH0UP8RKNyeX0SqCrxPu-6wn8CA&bvm=bv.58187178,d.ZGU" target="_blank">CC2D33S</a> sensor from GE. This sensor has a 3% accuracy and operates from –40 to 125°C. It costs about 6 ~ 8 euro in single unit quantity. It has an I<sup>2</sup>C bus interface and can operate in a sleep mode where it consumes 0.6 µA (typical) and 750 µA (typical) when operating. It also has a small footprint compared to some of the cheaper Chinese sensors available.</p> <p>For power management I decided to add a boost regulator based on the 3.3 volt version of the <a href="http://www.skyworksinc.com/uploads/documents/AAT1217_202050B.pdf" target="_blank">AAT1217</a> chip. This was chosen due to cost and the fact that it does not need an external Schottky diode which are difficult to find with a low forward voltage drop and reasonable load current.</p> <p><a href="http://lh4.ggpht.com/-IL9f9ejRl38/UsqygQG87PI/AAAAAAAAATY/vy19Yu2z-SA/s1600-h/PowerSupply%25255B2%25255D.png"><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="PowerSupply" border="0" alt="PowerSupply" src="http://lh3.ggpht.com/-xVVAiMnBRd0/Usqygy69lyI/AAAAAAAAATg/uBS7RN25sUA/PowerSupply_thumb.png?imgmax=800" width="244" height="122"></a></p> <p>The boost regulator will operate down to 0.5 volts ensuring that the sensor node will remain operational as the input voltage drops.</p> <p>I have also added the option of not installing the boost convertor and its components by placing a shut from VBAT to VCC.</p> <p><a href="http://lh3.ggpht.com/-61hQpkY12Bg/Usqyhtayf3I/AAAAAAAAATo/kPW8T-nzSAc/s1600-h/Shunt%25255B2%25255D.png"><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="Shunt" border="0" alt="Shunt" src="http://lh5.ggpht.com/-OJp0N3-7rf8/UsqyiFhietI/AAAAAAAAATw/93nQUPAs3mg/Shunt_thumb.png?imgmax=800" width="244" height="135"></a></p> <p>To monitor the input voltage to the sensor node I added a simple FET switch switch to a resistor that that the microcontroller will sample when it is performing a sensor update and send the input voltage, temperature and the humidity data.</p> <p><a href="http://lh5.ggpht.com/-xoOdqfcnwLs/Usqyi5c7mSI/AAAAAAAAAT4/ng4HoUWsL5M/s1600-h/BatSense%25255B2%25255D.png"><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="BatSense" border="0" alt="BatSense" src="http://lh3.ggpht.com/-rQjyeEBpzgg/Usqyjk6GvJI/AAAAAAAAAT8/EmcwpAvoLQ8/BatSense_thumb.png?imgmax=800" width="244" height="228"></a></p> <p>I will use the following <a href="http://newageenclosures.com/cart.php?m=product_detail&p=45&c=11" target="_blank">enclosure</a> to house the sensor as its neat and tidy. The PCB to fit the chosen enclosure was designed with <a href="www.kicad-pcb.org" target="_blank">KiCad</a> and a small batch was ordered through <a href="http://oshpark.com/" target="_blank">OSH Park</a>. A render of the PCB is shown below.</p> <p><a href="http://lh4.ggpht.com/-HPMBKQYsHbY/UsqykENx0AI/AAAAAAAAAUI/20eDew8mub4/s1600-h/i%25255B2%25255D.png"><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="i" border="0" alt="i" src="http://lh4.ggpht.com/-qSmqvi5amIs/UsqykzUaV7I/AAAAAAAAAUM/1rdruTL1R2I/i_thumb.png?imgmax=800" width="244" height="87"></a></p> <p>Once the boards arrive I will begin the building and debugging. I will add some follow up posts recording my progress. I think the main risk to the sensor nodes is the radio module. The 2.4 Ghz band is quite congested and the range performance of the 2.4 Ghz modules can be affected by obstacles such as internal doors, floors and walls.</p> <h5>Control Hub</h5> <p>The Control Hub is the next piece of the system, I have not made much progress on the implementation of this yet. I do however have a set of requirements:</p> <ul> <li>Expose a simple web interface</li> <ul> <li>To view data</li> <li>To manage the sensor nodes, update interval etc.</li></ul> <li>On board storage</li> <ul> <li>Should not require external storage for storing reading</li></ul> <li>Expose a programmatic interface</li> <ul> <li>Maybe some kind of restful interface</li></ul></ul> Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-58248362286039395142013-09-01T19:00:00.000+01:002016-09-29T11:30:51.791+01:00Dynamic resource pool with Weak ReferencesSometimes its necessary to allocate a number of expensive resources at runtime. These resources can be expensive in terms of threads or memory. They maybe individually expensive, have a large memory footprint, have a high number of threads or cumulatively expensive for example many instances have to be allocated. It may also be the case that the creation of the resource objects is an expensive task in itself.<br />
<br />
To address this problem it is usual to implement an <a href="http://en.wikipedia.org/wiki/Object_pool_pattern">object pool</a>. This pattern is one of the creational design patterns, the intent of this pattern is to reuse and share objects that are expensive to create. It can also be put to use to manage objects that encapsulate expensive resources.<br />
<br />
One of the implementation constraints of an <a href="http://en.wikipedia.org/wiki/Object_pool_pattern">object pool</a> implementation is the management of the number resources that are allocated by the object pool. The general implementation of an object pool supposes a fixed number of resources that are allocated when the pool is constructed and are destroyed when the object pool is destroyed.<br />
<br />
For the specific problem I am trying to solve, fixed resource allocation is too restrictive, I need some elasticity in the pool. I require the pool to shrink and expand base on the number of outstanding requests for objects.<br />
<br />
There are a number of options available to implement this elasticity for example<br />
<ul>
<li>A cache configured with sliding window flushing of least recently used resources </li>
<li>A background thread that manages last access time and release resources</li>
</ul>
Both of the implementations above would allow the resource allocation to be elastic. Both have the disadvantage of there response time to clean up of unused resources contained within the pool. This is due to the fact that both will have some periodic interval for clean up. The sliding wind value for the cache and the polling interval for the threaded solution. Tuning an optimal “rate of elasticity” adds to the complexity of the solution. It also means that one needs to tune this rate for different applications that have different resource allocation profiles.<br />
<br />
The solution that I have chosen is to allocate resources and maintain references with Weak References inside the resource pool. This allows the pool to respond to the request for resources and also to memory pressure on the system.<br />
<h3>
Dynamic Resource Pool</h3>
First lets define an interface for the resource pool. Using a generic interface allows the implementation of the pool to be abstracted from the implementation of the object pool.<br />
<br />
<pre class="brush: c-sharp; toolbar: false">/// <summary>
/// A resource pool that dynamically allocates and releases resources of type T
/// </summary>
/// <typeparam name="T"></typeparam>
public interface IDynamicResourcePool<T> where T : class
{
/// <summary>
/// Acquires a resource from the pool.
/// </summary>
/// <returns>An instance of a resource from the pool</returns>
T AcquireResource();
/// <summary>
/// Release a resource back to the pool
/// </summary>
/// <param name="resource">The resource to release</param>
void ReleaseResource(T resource);
}</pre>
<br />
Dissecting the <b>DynamicResourcePool</b> first we look at the local variables and the constructor for the class.<br />
<br />
<pre class="brush: c-sharp; toolbar: false">private readonly ConcurrentQueue<WeakReference> _resources;
private readonly PerformanceCounter _rateOfCreation;
private readonly PerformanceCounter _rateOfRelease;
private readonly PerformanceCounter _rateOfAquire;
private readonly PerformanceCounter _totalResources;
private readonly Func<T> _factoryFunc;
/// <summary>
/// Construct an instance of a <see cref="DynamicResourcePool{T}"/>
/// </summary>
/// <param name="resourceFactoryFunc">The factory used to create resources</param>
public DynamicResourcePool(Func<T> resourceFactoryFunc)
{
_resources = new ConcurrentQueue<WeakReference>();
_factoryFunc = resourceFactoryFunc;
if(PerformanceCounterCategory.Exists(CounterMetadata.CategoryName))
{
String instanceName = GetType().GetGenericArguments()[0].Name;
_rateOfCreation = new PerformanceCounter(CounterMetadata.CategoryName, CounterMetadata.CreationCounterName, instanceName, false);
_rateOfRelease = new PerformanceCounter(CounterMetadata.CategoryName, CounterMetadata.ReleasedCounterName, instanceName, false);
_rateOfAquire = new PerformanceCounter(CounterMetadata.CategoryName, CounterMetadata.AquiredCounterName, instanceName, false);
_totalResources = new PerformanceCounter(CounterMetadata.CategoryName, CounterMetadata.TotalCreatedCounterName, instanceName, false);
}
}</pre>
<br />
The <b>ConcurrentQueue<WeakReference></b> is used to store the weak references that contain the allocated resources. There are also a set of performance counters for tracking the performance characteristics of the <b>DynamicResourcePool</b> at runtime. The performance counters will be used later in this post to examine the <b>DynamicResourcePool</b>.<br />
<br />
Looking at the <b>AcquireResource</b> operation excluding some code for incrementing and decrementing counters its actually pretty simple. Try dequeue a resource from the resources queue if either a resource cannot be dequeued or the dequeued resource is null, i.e. it had been garbage collected, then create a new resource via the factory. Note: because a queue is used as the resource container the release and acquire of resources are round robin. This behaviour could be made configurable by delegating the internal acquire and release operations to an Action<T> or Func<T> delegate like the factory method.<br />
<br />
<pre class="brush: c-sharp; toolbar: false">/// <summary>
/// Acquires a resource from the pool.
/// </summary>
/// <returns>An instance of a resource from the pool</returns>
public T AcquireResource()
{
IncrementCounter(_rateOfAquire);
WeakReference weakReference;
T resource = default(T);
if(_resources.TryDequeue(out weakReference))
{
resource = (T)(weakReference.Target);
if(resource == null)
{
DecrementCounter(_totalResources);
}
}
if(resource == null)
{
if(_factory != null)
{
IncrementCounter(_totalResources);
IncrementCounter(_rateOfCreation);
resource = _factoryFunc();
}
}
return resource;
}</pre>
<br />
The Release method is really simple. The resource is simply queued back in the resource pool.<br />
<pre class="brush: c-sharp; toolbar: false">/// <summary>
/// Release a resource back to the pool
/// </summary>
/// <param name="resource">The resource to release</param>
/// <exception cref="ArgumentNullException">Thrown when the argument is resource is null</exception>
public void ReleaseResource(T resource)
{
IncrementCounter(_rateOfRelease);
if(resource == null)
{
throw new ArgumentNullException("resource");
}
_resources.Enqueue(new WeakReference(resource));
}</pre>
<br />
<h3>
Test Client</h3>
<br />
To demonstrate the behaviour of the <b>DynamicResourcePool</b> the test client performs the following:<br />
<ul>
<li>Defines a resource type, <b>ByteBuffer</b> </li>
<li>Creates a <b>DynamicResourcePool</b> <<b>ByteBuffer</b>> </li>
<li>Starts a number of Tasks to allocate and release <b>ByteBuffer</b> resources </li>
<li>Forces a garbage collection </li>
<li>At each stage reports the memory consumption of the process</li>
</ul>
<br />
After running the test client the following output is displayed.<br />
<br />
<a href="http://lh5.ggpht.com/-MQe2qkS8OJg/UiReSCtYyXI/AAAAAAAAARk/yL55y3v7CfA/s1600-h/image%25255B3%25255D.png"><img alt="image" border="0" height="199" src="https://lh5.ggpht.com/-YDqP1a5Q2s4/UiReSYf_eWI/AAAAAAAAARs/gR5-pXLBuRM/image_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="644" /></a><br />
<br />
The <i>Memory to be Allocated</i> figure indicates the amount of bytes that the <b>ByteBuffer</b> instances will contain, this is 10Mb each times 20 tasks. The <i>Memory used after Allocation</i> shows the initial processes allocated managed memory plus the memory allocated for the 20 <b>ByteBuffer</b> instances. The test client forces a garbage collection cycle, since there are no tasks running at this point all <b>ByteBuffer</b> instances are collected.<br />
<br />
Looking at the performance counters we can see the behaviour of the <b>DynamicResourcePool</b>. First the <i>Total Resources Created</i> counter shows that it starts at zero and rises linearly to 20, which corresponds to the allocation profile of the Tasks.<br />
<br />
<a href="http://lh6.ggpht.com/-BNozyxS5bjM/UiR8LKPBrOI/AAAAAAAAAR8/UZTxxsqKGbA/s1600-h/Total%25255B3%25255D.gif"><img alt="Total" border="0" height="442" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-Pe3Kr1EklEfm1eg683swN2Hn-zdrTFdYaST6C9BVBe98_7FZ7mrQgyBRLEgg7XWdFZLIoCTbyRhD735lPD_F_fpu4iWEzhDQ_2IIEQUNgyQxUXrw6bWDbDAItRqSndb2V-s2mt3SIES4/?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="Total" width="644" /></a><br />
<br />
The <i>Resources Created</i> counter shows that the <b>DynamicResourcePool</b> creates resources initially and then stops when the <i>Total Resources Created</i> reaches 20 as expected.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYPty5m2llzHRl1qXznpWzqvha71b315Vbe2wOvXZCrsJ5O6FGt66XhB1ldLaul87MdUBr5ranIeBCe3EU-sBb9nbnL0zmAzV_1FM1zYWuTUP48bjSdNJ41eD7qPyMcnqNvrkcgeO9epSx/s1600-h/created%25255B3%25255D.gif"><img alt="created" border="0" height="442" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj88DxOkidVgRfyLtyIhw6bFJOp1eXqBOBUAeGIVtmLaUgatcqgQFe7nkNjXtVLsk5FIboIy3myVtaBNj3pIkl-66kxDNbSK7pdpbgTJ0DSP0BEzAD5TogDKtgLr7ElERhPi_FTlJafE-Jb/?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="created" width="644" /></a><br />
<br />
The <i>Resources Acquired</i> counter shows the rate of resources acquired. The <i>Resources Release</i> counter tracks the acquired counter due to the nature of the test client code. The <i>Resources Acquired</i> counter reaches about 200 per second which is as expected as the test code acquires and releases a resource every 100 milliseconds on 20 separate threads.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNYMtyyCAmW6um9V-N3ioXlY3c0WokOHgBfk-5HcCnaKYPQvRWAwNMffMfSL7B18qmOhGG0hYuINuzIrWw6b5wwxhcR6Ke0kIsLuw-vmbley0yhAqB1LW7kknDhZMyUZBbvgyNwImRKKNG/s1600-h/Acquired%25255B3%25255D.gif"><img alt="Acquired" border="0" height="442" src="https://lh4.ggpht.com/-GystYW7C18g/UiR8NKXPerI/AAAAAAAAASk/tf3d5haYQUU/Acquired_thumb%25255B1%25255D.gif?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="Acquired" width="644" /></a><br />
<br />
<h3>
Further Enhancements</h3>
<ul>
<li>Allow acquire and release strategy to be configured. </li>
<li>Allow the pool to pre allocate resources at construction time</li>
</ul>
<br />
<h3>
Source Code</h3>
<br />
<span style="color: red;"><i>Note: you will need to launch either VS or the test client as administrator as it registers performance counters. </i></span><br />
The code is available here: <span style="font-size: large;"><a href="https://github.com/DerekGn/Dumping-Core-Memory/tree/master/DumpingCoreMemory.ResoucePooling" target="_blank">GitHub DumpingCoreMemory.ResoucePooling</a></span>Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-32567303851474522152013-07-31T08:15:00.003+01:002016-09-29T10:25:39.386+01:00C++/CLI Asynchronous events using EventHandler<T>While developing some code to expose a native C++ library as a managed component to be used in a C# component. I found that I needed to raise an event asynchronously from the C++/CLI class. Since I have a preference to use <a href="http://msdn.microsoft.com/en-us/library/system.eventhandler.aspx">EventHandler</a> and <a href="http://msdn.microsoft.com/en-us/library/db0etb8x.aspx">EventHandler<TEventArgs></a> delegates for exposing events rather than define the delegate and the event separately.<br />
<br />
So naively, as its been a while since I wrote C++ and my first time writing C++/CLI, I proceeded to simply define an event as shown in the following example snippet:<br />
<pre class="brush: c-sharp; toolbar: false">ref class EventSource
{
public:
event EventHandler^ SyncEvent;
}</pre>
<br />
And invoke it as in the following example snippet:<br />
<br />
<pre class="brush: c-sharp; toolbar: false">void EventSource::RaiseSyncEvent(void)
{
SyncEvent->BeginInvoke(this, EventArgs::Empty, gcnew AsyncCallback(this, &EventSource::Callback), nullptr);
}</pre>
Two which I received the following compiler error:<br />
<b><span style="color: red;">Error 1 error <a href="http://msdn.microsoft.com/query/dev10.query?appId=Dev10IDEF1&l=EN-US&k=k%28C3918%29;k%28VS.ERRORLIST%29&rd=true">C3918</a>: usage requires 'EventSource::SyncEvent' to be a data member </span></b><br />
<br />
In the case above this is caused by the fact that you cannot access the backing field for the event delegate. If we decompile the code with <a href="http://ilspy.net/">ILSpy</a> we can see that the compiler generates the following C# code:<br />
<br />
<a href="http://lh6.ggpht.com/-HBra5BJDKuo/UfhApcDMh6I/AAAAAAAAAQ0/ufMeNyobpIg/s1600-h/image%25255B7%25255D.png"><img alt="image" border="0" height="192" src="https://lh4.ggpht.com/-CqXxM_KOM44/UfhAp_eoKjI/AAAAAAAAAQ4/cDsG0Gonb2E/image_thumb%25255B3%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="644" /></a><br />
<br />
This is due to the fact that the event syntax used to define the SyncEvent is called <b><i>trivial event</i>.</b> A <i><b>trivial event</b></i> is similar to a <i><span style="color: #333333;"><b>trivial property</b></span></i> where by the compiler provides the backing store field and the assessor methods. <br />
<br />
The C++/CLI compiler also provides a raise_xxxxxxx method as shown below, which is kind of neat as the C# compiler requires that you write the equivalent code to insure thread safety when accessing the delegate.<br />
<br />
<a href="http://lh5.ggpht.com/-7XVDx7_pQNk/UfhFmeh_CRI/AAAAAAAAARM/KVZgZ8CcG2Q/s1600-h/image%25255B11%25255D.png"><img alt="image" border="0" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQmGYQhKgomvUuXY9kyiMhH2V6YAnzPex-dgWjeSmav30lDIsT7sVG0A9fkDVGck15UbJnIVPWdQXEM_MvWOleVYU6YXUonkXf6XDnBsxl2M3P0G1Oe4iXvTAHWTGd7vh08QDluDu5d_Fa/?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="644" /></a><br />
<br />
To get access to the underlying event delegate the <i>nontrivial event</i> syntax must be used. This requires the definition of the assessor methods. The <i>add</i> and <i>remove</i> are mandatory and the <i>raise</i> method is optional.<br />
<br />
Here is where I discovered another small problem. There are no examples of how to use <a href="http://msdn.microsoft.com/en-us/library/system.eventhandler.aspx">EventHandler</a> and <a href="http://msdn.microsoft.com/en-us/library/db0etb8x.aspx">EventHandler<TEventArgs></a> delegates on the Microsoft web site. All of the examples use the explicit delegate and event definition syntax.<br />
<br />
To define a non trivial event first define the private field/property.<br />
<pre class="brush: c++; toolbar: false">ref class EventSource
{
private:
EventHandler^ m_asyncEvent;</pre>
<br />
Then define the explicit add and remove methods.<br />
<pre class="brush: c-sharp; toolbar: false">event EventHandler^ AsyncEvent
{
void add(EventHandler^ eventHandler)
{
m_asyncEvent += eventHandler;
}
void remove(EventHandler^ eventHandler)
{
m_asyncEvent -= eventHandler;
}
}</pre>
<br />
You can now invoke the event asynchronously via BeginInvoke, note you must check for null as the compiler no longer generates the nice raise method.<br />
<pre class="brush: c++; toolbar: false">void EventSource::RaiseAsyncEvent(void)
{
if(m_asyncEvent != nullptr)
{
m_asyncEvent->BeginInvoke(this, EventArgs::Empty, gcnew AsyncCallback(this, &EventSource::Callback), nullptr);
}
}</pre>
<br />
Note that a call-back must provided to clean up the IAsyncResult instance by calling the EndInvoke method, see <a href="http://msdn.microsoft.com/en-us/library/2e08f6yc%28v=vs.100%29.aspx">here</a> for further reference.<br />
<pre class="brush: c++; toolbar: false">void EventSource::Callback(IAsyncResult^ ar)
{
Console::WriteLine("EventSource::Callback invoked on Thread Id: {0}", Thread::CurrentThread->ManagedThreadId);
AsyncResult^ result = (AsyncResult^) ar;
EventHandler^ caller = (EventHandler^) result->AsyncDelegate;
caller->EndInvoke(ar);
}</pre>
<br />
<br />
The code is available here: <span style="font-size: large;"><a href="https://github.com/DerekGn/Dumping-Core-Memory/tree/master/AsyncEvent" target="_blank">GitHub AsyncEvent</a></span>Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-59903850179537403712013-07-24T09:00:00.000+01:002016-09-29T10:15:38.323+01:00A Bounded Task DispatcherDuring a recent implementation I had to resolve some information from a database in response to a network event. These network events are delivered in a serialised order via a custom protocol. In order to decouple the processing of event messages from the query of the database, which is an order of magnitude slower than the event rate, it was necessary to dispatch the event processing onto a separate thread. This would allow the events to be processed in parallel. A secondary requirement was to limit the number of parallel threads to some upper bound.<br />
<br />
To execute event processing on a separate thread there are a number of options:<br />
<ol>
<li>Create a <a href="http://msdn.microsoft.com/en-us/library/system.threading.thread.aspx">Thread</a> explicitly and manage its lifecycle </li>
<li>Use the <a href="http://msdn.microsoft.com/en-us/library/system.threading.threadpool.aspx">ThreadPool</a> to dispatch the request processing </li>
<li>Use a <a href="http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx">BackGroundWorker</a> to dispatch request processing </li>
<li>Create a <a href="http://msdn.microsoft.com/en-us/library/system.threading.tasks.task.aspx">Task</a> and manage its lifecycle</li>
</ol>
There are a number of disadvantages to the options 1 and 2, which basically boil down to the amount of plumbing code required to manage and synchronise thread execution and lifecycle. For example handling exceptions, return data etc.<br />
<br />
Option 3 is not really a viable option, it is on the list just for completeness, I would not use, or recommend, the <a href="http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx">BackGroundWorker</a> for dispatching requests for a backend component as it is primarily a UI component.<br />
<br />
The .Net framework 4.0 introduced the <a href="http://msdn.microsoft.com/en-us/library/system.threading.tasks%28v=vs.100%29.aspx">System.Threading.Tasks</a> namespace which contains a number of classes and types that simplify the work of writing concurrent and asynchronous code. The <a href="http://msdn.microsoft.com/en-us/library/system.threading.tasks.task.aspx">Task</a> class is the type from <a href="http://msdn.microsoft.com/en-us/library/system.threading.tasks%28v=vs.100%29.aspx">System.Threading.Tasks</a> that the bounded dispatcher is based on.<br />
<br />
<h3>
The Task dispatcher code walk through</h3>
The <b>TaskDispatcher</b> type encapsulates the management of the dispatched tasks. Looking at the constructor we can see it takes an argument that specifies the maximum number of tasks that can be dispatched in parallel.<br />
<pre class="brush: c-sharp; toolbar: false">/// <summary>
/// The <see cref="TaskDispatcher"/> constructor
/// </summary>
/// <param name="maxTasks">The maximum number of tasks that can be dispatched before the call to <seealso cref="Dispatch"/> blocks</param>
public TaskDispatcher(Int32 maxTasks)
{
_executingTasks = new List<Task>();
_tasks = new Semaphore(maxTasks, maxTasks);
_tokenSource = new CancellationTokenSource();
}</pre>
<br />
The <b>_executingTasks</b> is a list that references the executing tasks. The <b>_tasks</b> <a href="http://msdn.microsoft.com/en-us/library/system.threading.semaphore%28v=vs.100%29.aspx">Semaphore</a> is used to bound the upper limit of Tasks that can be dispatched by the <b>TaskDispatcher</b>. The<b> _tokenSource</b> is a <a href="http://msdn.microsoft.com/en-us/library/System.Threading.CancellationToken%28v=vs.100%29.aspx">CancellationToken</a> that is use to cancel executing tasks.<br />
<br />
<h3>
Dispatching Tasks</h3>
<pre class="brush: c-sharp; toolbar: false">/// <summary>
/// Dispatches an <see cref="Action"/> for execution.
/// </summary>
/// <param name="action">The <see cref="Action"/> to dispatch</param>
/// <param name="completionCallback">The callback action to invoke when the task completes</param>
/// <param name="dispatchTimeout">The <see cref="TimeSpan"/> to wait for a task to be dispatched</param>
/// <exception cref="TimeoutException">An exception thrown when the <see cref="action"/> cannot be dispatched within the <paramref name="dispatchTimeout"/> </exception>
public void Dispatch(Action<CancellationToken> action, Action<Task,Exception> completionCallback, TimeSpan dispatchTimeout)
{
if (_tasks.WaitOne(dispatchTimeout))
{
lock (_executingTasks)
{
var task = new Task(() => action(_tokenSource.Token));
task.ContinueWith(t => Completion(t, completionCallback));
task.Start();
_executingTasks.Add(task);
}
}
else
{
throw new TimeoutException(String.Format("Unable to dispatch action within timeout {0}", dispatchTimeout));
}
}</pre>
To dispatch a task the Dispatch method is called, passing a parameter action of type Action<CancellationToken> and a parameter completionCallback of type Action<Task, Exception> delegate which is invoked when the dispatched action delegate completes. <br />
<br />
The thread invoking the Dispatch method attempts to enter the _tasks semaphore by calling the WaitOne method. If the semaphore is not signalled within the timeout period then an exception is thrown to indicate that a Task cannot be dispatched.<br />
<br />
If the thread can enter the _tasks semaphore a Task is created passing a CancellationToken to the constructor, the CancellationToken is used to control the executing Tasks cancellation. The Task is added to the list of executing tasks, and the calling thread returns.<br />
<br />
If there are multiple competing threads trying to enter the _tasks semaphore there is no guaranteed order as to which competing thread will be signalled. This is due to the behaviour of the semaphore. Also it is possible that to threads enter the Semaphore, thread A followed by thread B. Due to pre-emptive scheduling thread B may acquire the lock on the _executingTasks before thread A.<br />
<br />
<h3>
On Completion</h3>
<br />
<pre class="brush: c-sharp; toolbar: false">/// <summary>
/// The completion callback thats called when a dispatch task was completed
/// </summary>
/// <param name="completedTask">The <see cref="Task"/> that completed</param>
/// <param name="completionCallback">The callback that is invoked when the dispatched <see cref="Task"/> executes to completion</param>
private void Completion(Task completedTask, Action<Task,Exception> completionCallback)
{
_tasks.Release();
lock (_executingTasks)
{
_executingTasks.Remove(completedTask);
}
completionCallback(completedTask, completedTask.Exception);
}</pre>
On completion of an instance of a Task the private method Completion is called. This method releases the _tasks semaphore and removes the Task from the _executingTasks list. It then invokes the completionCallback delegate which was provided to the completedTask when it was constructed. If there is an exception associated with the completed task it is passed to the completion call-back.<br />
<br />
<h3>
Cancelling Tasks</h3>
<pre class="brush: c-sharp; toolbar: false">/// <summary>
/// Cancel all executing <see cref="Task"/>
/// </summary>
public void CancelAll()
{
_tokenSource.Cancel();
Task.WaitAll(_executingTasks.ToArray());
}</pre>
To cancel executing tasks the CancelAll method of the task dispatcher is invoked. This method sets the _tokenSource to cancelled. This sets the isCancellationRequested property to true. Dispatched actions should check this property for example:<br />
<pre class="brush: c-sharp; toolbar: false">static void CancellableTask(CancellationToken cancellationToken)
{
for (int i = 0; i < 100000; i++)
{
// Simulating work.
Thread.SpinWait(5000000);
if (cancellationToken.IsCancellationRequested)
{
// Perform cleanup if necessary.
//...
// Terminate the operation.
break;
}
}
}</pre>
<br />
Note: An errant task method that does not honour the CancellationToken will block the call to CancelAll method. This behaviour could of the CancelAll method could be modified by causing the Task.WaitAll to wait a period of time and then terminate the tasks that did not terminate with the timeout. This would be drastic and should be avoided, if it is possible modify the behaviour of the dispatched methods to hour honour the CancellationToken <br />
<br />
<h3>
Further enhancements</h3>
<br />
The code presented here is the basic functionality of a bounded task dispatcher some further improvements could be made. For example:<br />
<br />
<ul>
<li>Make the entire Dispatch class generic so that the action parameter type can be specified for example as a Func delegate so a return value can be passed back from the dispatched method.</li>
<br />
<li>Add methods to allow individual tasks to be cancelled. </li>
</ul>
The code is available here: <span style="font-size: large;"><a href="https://github.com/DerekGn/Dumping-Core-Memory/tree/master/DumpingCoreMemory.Threading" target="_blank">GitHub DumpingCoreMemory.Threading</a></span>Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-49473728035171258212013-06-20T12:12:00.000+01:002013-06-20T12:12:28.092+01:00Windows Azure SDK 1.7 + Windows Server App Fabric 1.1 Install ErrorWhile setting up my development machine for some proof of concept development for migration of an existing application to Windows Azure, I experienced an issue with installing Windows Azure SDK 1.7, released in June, on the machine that had Windows Server AppFabric 1.1 installed previously.<br />
<span style="font-family: Verdana; font-size: xx-small;">MSI (s) (58:BC) [12:08:37:660]: Executing op: ActionStart(Name=RollbackRegisterEventManifest,,)<br />MSI (s) (58:BC) [12:08:37:662]: Executing op: CustomActionSchedule(Action=RollbackRegisterEventManifest,ActionType=3393,Source=BinaryData,Target=CAQuietExec,CustomActionData="wevtutil.exe" um "C:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK\2012-06\bin\plugins\Caching\Microsoft.ApplicationServer.Caching.EventDefinitions.man")<br />MSI (s) (58:BC) [12:08:37:664]: Executing op: ActionStart(Name=RegisterEventManifest,,)<br />MSI (s) (58:BC) [12:08:37:666]: Executing op: CustomActionSchedule(Action=RegisterEventManifest,ActionType=3073,Source=BinaryData,Target=CAQuietExec,CustomActionData="wevtutil.exe" im "C:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK\2012-06\bin\plugins\Caching\Microsoft.ApplicationServer.Caching.EventDefinitions.man")<br />MSI (s) (58:50) [12:08:37:705]: Invoking remote custom action. DLL: C:\Windows\Installer\MSI4266.tmp, Entrypoint: CAQuietExec<br /><b><span style="color: red;">CAQuietExec: Provider Microsoft-Windows Server AppFabric Caching is already installed with GUID {a77dcf21-545f-4191-b3d0-c396cf2683f2}.</span></b><br />CAQuietExec: Configuration error.<br />CAQuietExec: Error 0x80073aa2: Command line returned an error.<br />CAQuietExec: Error 0x80073aa2: CAQuietExec Failed<br />CustomAction RegisterEventManifest returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)<br />Action ended 12:08:37: InstallFinalize. Return value 3.</span><br />
The workaround is to remove the AppFabric Cache component first and then install the Windows Azure SDK 1.7, there is some more information here: <a href="http://social.msdn.microsoft.com/Forums/en-US/windowsazuredevelopment/thread/cafcc159-578a-469f-b1d5-59a4b9c6b5c4/" title="http://social.msdn.microsoft.com/Forums/en-US/windowsazuredevelopment/thread/cafcc159-578a-469f-b1d5-59a4b9c6b5c4/">http://social.msdn.microsoft.com/Forums/en-US/windowsazuredevelopment/thread/cafcc159-578a-469f-b1d5-59a4b9c6b5c4/</a>Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-6240064484305660032013-02-01T00:06:00.000+00:002013-02-02T11:38:42.126+00:00A small ARM race<p>I have been on a quest lately to find a computer that meets the following criteria:</p> <ul> <li>Cheap enough to let the kids have it <li>I don't really care if the kids break it <li>I would like them to maybe learn something about computers, Don’t want app junkies <li>I’m old school and want the kids to learn to use a keyboard and a mouse, not mash their paws on a screen <li>I can reuse some old hardware, keyboard + mouse + monitor <li>Low power consumption</li></ul> <p>The initial logical answer is the <a href="www.raspberrypi.org/">Raspberry PI</a>, yes its cheap but, I’m of the opinion based on discussion on the web that it is a bit under powered as a computer and is more suited to embedded control projects then as a low cost PC like device.</p> <p>Next up on the list is the <a href="http://beagleboard.org/">Beagleboard-xm</a> it has a bit “power” than the <a href="www.raspberrypi.org/">Raspberry PI</a> but it is still considered an embedded board. Having the opportunity to play around with a <a href="http://beagleboard.org/">Beagleboard-xm</a> I was impressed buy its low power consumption, about 750mA @ 5 volts which works out at about 3.75 watts, that was running <a href="http://www.angstrom-distribution.org/">Ångström</a> while playing 720p video. Unfortunately that is all this board is capable of due to the video processor. But the gnome UI was smooth and use able.</p> <p>I the installed <a href="http://www.ubuntu.com/">Ubuntu</a> 12.10 on the micro SD card, which is what most of these small arm board boot from, the result was less that impressive. The response from the UI was pretty slow.</p> <p>Next on the list is the <a href="http://pandaboard.org/content/pandaboard-es">Pandaboard ES</a> which is a step above the <a href="http://beagleboard.org/">Beagleboard-xm</a>, it has 1GB of ram and a 1.2 Ghz processor and is capable of 1080p. This looks like a pretty capable board but it is more expensive than the <a href="http://beagleboard.org/">Beagleboard-xm</a>. Unfortunately I’ve not had the chance to play with one yet but its on the list of contenders.</p> <p>I recently found a new entry into the small low powered <a href="http://www.howchip.com/shop/item.php?it_id=BRIX5250A">ARMBrix Zero</a> from a Korean company.</p> <p><img title="1st Sample of ARMBRIX Zero Rev0" alt="" src="http://www.armbrix.com/wp-content/uploads/2013/01/ARMBRIX_Zero.jpg" width="600" height="356"></p> <p> Unfortunately the first batch has sold out. Hopefully there will be more. This is a pretty powerful board it has a Samsung Exynos 5250 dual core ARM Cortex-A15 which is a pretty powerful processor on a board that has 2GB of ram, HDMI, USB 3.0 and a SATA port, 1080p video decoding all for $145 + shipping.</p> <p>The catch is that there was only one pre order batch available to order which was for an engineering evaluation run. The people behind the board are in the process of testing the design. I for one am eagerly awaiting the next run.</p> <p>Only problem is I might not let the kids have it :></p> Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-38654191492004412052013-01-29T20:59:00.001+00:002013-01-29T21:00:55.877+00:00IASA Ireland Chapter<p>Attended the IASA IT Architects – 2013 Kick Off Event on Thursday 24th. Wasn't really too sure what to expect but was pleasantly surprised. The presentation covered the usual who and what is IASA, information about the Irish chapter, training and certification.</p> <p>The really big news is that the Irish chapter is to become the IT architecture network within ICS, ICS and IASA have signed a partnership agreement with <a href="http://www.ics.ie/">ICS</a>.</p> <p>Expect to hear more from March onwards as the details are worked out.</p> Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-63511640742105074832013-01-29T00:00:00.000+00:002013-01-31T22:21:37.642+00:00Custom WCF UserNamePasswordValidator Performance ImpactRecently I have had to perform some debug/analysis on an unusual case of clients receiving timeouts on a critical WCF service in a production environment. I use the term “unusual” to describe the timeouts due to the nature in which these timeouts occur combined with the operation behaviour of the service when they are occurring. That and the timeouts are extremely severe, every client experiences timeouts, in my experience with WCF is not an expected operational behaviour of the WCF stack. Under normal high load scenarios a percentage of clients will experience timeouts or receive the expected <b>Server To Busy</b> fault when the WCF service has breached its configured throttle values.<br />
In this specific case knowing the operation capabilities of the WCF service in question I knew that the amount of load on the server although high was well within the capacity of the service in question. Nor was the load significant enough to breach the configured service throttles. I eventually got to the bottom of the issue through debug and crash dump analysis. It was an unexpected/undocumented behaviour of the WCF stack coupled/influenced by a specific behaviour of a custom <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.selectors.usernamepasswordvalidator%28v=vs.100%29.aspx">UserNamePasswordValidator</a> used by the WCF service.<br />
I thought it would be worth describing the problem in more detail in the hope it might help others who experience WCF Service performance issues, that done seem to "make sense", giving one additional diagnostic check to ad to the toolbox.<br />
<br />
<h3>
Symptoms</h3>
The symptoms of the service outage are as follows:<br />
<ul>
<li>A high percentage of clients, > 90%, receive timeouts or Server To Busy exceptions</li>
<li>The <a href="http://msdn.microsoft.com/en-us/library/ms733922.aspx">Instances</a> service performance Counter is at or close to the the configured or defaulted <a href="http://msdn.microsoft.com/en-us/library/system.servicemodel.configuration.servicethrottlingelement.maxconcurrentinstances%28v=vs.100%29.aspx">ServiceThrottlingElement.MaxConcurrentInstances</a> value </li>
<li>The <a href="http://msdn.microsoft.com/en-us/library/ms731786.aspx">Calls Per Second Rate</a> service performance counter is low or at 0 for periods of time </li>
<li>If the service is deployed in an IIS app pool the ASP.NET Application Performance <a href="http://msdn.microsoft.com/en-us/library/fxk122b4%28v=vs.100%29.aspx">Counter Pipeline Instance Count</a> will be at or close to its configured or defaulted value </li>
<li>The <a href="http://msdn.microsoft.com/en-us/library/ms731722.aspx">Security Validation And Authentication Failures</a> performance counter can be non zero </li>
<li>The service appears to be deadlocked or being starved of requests, the service is not writing log files or updating database etc. but clients experience timeouts or very slow reponses </li>
<li>Very low CPU utilisation by the process in question</li>
</ul>
<h3>
</h3>
<h3>
Cause</h3>
After generating a process memory dump, from the WCF Service. After examining all of the threads stacks by executing the <a href="http://msdn.microsoft.com/en-us/library/bb190764.aspx">EEStack</a> command it was clearly evident that most of the threads in the process where blocked on the unmanaged windows API method <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/aa364986%28v=vs.85%29.aspx">GetQueuedCompletionStatus</a> which is a blocking I/O call. One call was blocked on the <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.selectors.usernamepasswordvalidator.validate%28v=vs.100%29.aspx">Validate</a> method of the <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.selectors.usernamepasswordvalidator%28v=vs.100%29.aspx">UserNamePasswordValidator</a>.<br />
It transpires that the the Validate method of the custom <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.selectors.usernamepasswordvalidator%28v=vs.100%29.aspx">UserNamePasswordValidator</a> used by the WCF service, talks to a proprietary back office system to validate a clients username and password. To prevent a client from brute forcing a users password it will delay the username/password validation response for 5 seconds after the nth unsuccessful attempt. This has very serious consequences for the throughput of the WCF service.<br />
<h3>
</h3>
<h3>
Demonstration Code And Results</h3>
To demonstrate this problem I have put together a simple demo base on one of the WCF sample applications. It is the calculator application that implements a simple custom UserNamePasswordValidator. The code itself is not really the interesting part and is pretty trivial, the intent is to demonstrate the symptoms.<br />
<h4>
</h4>
<h4>
Setup</h4>
Run the Setup.Bat file in the .\UserNamePasswordValidator\CS folder. <i>Note you should run this from a VS2010 command prompt with administrative privileges. As it creates a test certificate and grants access to the private key for encryption/signing the message credentials.</i><br />
<h4>
</h4>
<h4>
Clean-up</h4>
Run the CleanUp.bat file in the .\UserNamePasswordValidator\CS folder. <i>Note you should run this from a VS2010 command prompt with administrative privileges. AS it deletes the test certificate created by setup.bat</i><br />
Open the solution in VS2010 running as a administrator as the code attempts to register a http.sys listener on port 8001.<br />
There are a number of application settings that can be used to modify the behaviour of the client and service code.<br />
In the <i>LoadTestClient</i> there are the following settings:<br />
<pre class="brush: xhtml; toolbar: false"><applicationSettings>
<LoadTestClient.Properties.Settings>
<!-- The number of tasks to create with valid passwords -->
<setting name="NumberTasksValidPassword" serializeAs="String">
<value>10</value>
</setting>
<!-- The number of tasks to create with invalid passwords -->
<setting name="NumberTasksInvalidPassword" serializeAs="String">
<value>2</value>
</setting>
</LoadTestClient.Properties.Settings>
</applicationSettings></pre>
<br />
In the service there is one setting to configure the sleep time for invalid requests:<br />
<pre class="brush: xhtml; toolbar: false"><applicationSettings>
<service.Properties.Settings>
<!-- The sleep time for a request with an invalid password -->
<setting name="InvalidCredentialSleep" serializeAs="String">
<value>1000</value>
</setting>
</service.Properties.Settings>
</applicationSettings></pre>
<br />
<h3>
Test Runs</h3>
<br />
To demonstrate the behaviour I have put together a few test cases, that vary each of the parameters of the test and show the results.<br />
<br />
<h2>
10 Tasks valid credentials , 0 Tasks invalid credentials</h2>
<br />
<a href="http://lh3.ggpht.com/-7SBSdQdBGs0/UQgz3d5Jo7I/AAAAAAAAAO4/vhdmYuN3RbY/s1600-h/image%25255B2%25255D.png"><img alt="image" border="0" height="209" src="http://lh5.ggpht.com/-5CPsX0inwDc/UQgz4pu-wCI/AAAAAAAAAO8/e1zSEI34Cxk/image_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom: 0px; border-left: 0px; border-right: 0px; border-top: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="244" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi56T8OxUWiza8SnAgXNiIPt9MInQR_BsmrlDMR0FQMY__80bD-zRw39AcRM-2lIBXs07Zayb-QopPhKx-eS5c2T6r7gc6tihPIOYZKhORo-eCs7rRlnWMqfa1JH70R_jGZ0-evAN1QQaUf/s1600-h/image%25255B13%25255D.png"><img alt="image" border="0" height="69" src="http://lh6.ggpht.com/-3T_mkNqswWg/UQgz6Qkw5sI/AAAAAAAAAPE/Aoh4mhrAhkE/image_thumb%25255B6%25255D.png?imgmax=800" style="background-image: none; border-bottom: 0px; border-left: 0px; border-right: 0px; border-top: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="244" /></a><br />
<br />
Form the graphs above we can see 10 service instances performing approximately 85 calls/second. The calls outstanding counter averages < 1. This is normal operation of the service.<br />
<br />
<h2>
10 Tasks valid credentials , 2 Tasks invalid credentials, Invalid Credential sleep time 1 sec</h2>
<br />
<a href="http://lh4.ggpht.com/--oLO_UVZ8yQ/UQgz7hHd3jI/AAAAAAAAAOI/mPQIx6mpBWc/s1600-h/image%25255B14%25255D.png"><img alt="image" border="0" height="209" src="http://lh4.ggpht.com/-i7PTHbrmHMM/UQgz8qT5EBI/AAAAAAAAAOQ/uGVPZzHMEi4/image_thumb%25255B8%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; margin: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="244" /></a><a href="http://lh4.ggpht.com/-TIihWvFOdFw/UQgz9W4-77I/AAAAAAAAAOY/OeeGd5I1GHA/s1600-h/image%25255B17%25255D.png"><img alt="image" border="0" height="66" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHCFppQbETv19qpVgm7GE4Z5kAiDS2UKMRl8jrX8Y9J2pWsU1FzLs7-S278vVuLOyc5x31rJmG-R2VjIGgQig6CgXgccioMzvGqW53L6z3rBU4bV5d9teYpqXej7l7eY24_-FjQRHzFmQg/?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; margin: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="244" /></a><br />
<br />
As we can see can see from the above, the results are relatively similar.<br />
<br />
<h2>
10 Tasks valid credentials , 3 Tasks invalid credentials, Invalid Credential sleep time 1 sec</h2>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbEjhyfXRLLVi2K9TFXjCQJUQsXbgV2ToGQaPT8UM0RToTd7VTMrmb0DpuafOCgLMjfEaZC0V6c8_ThLbhLrbZGTDG4au_Vn8ObVnm6Eyu7tkcZjc28FoQxtTqIjGv2cErlThZvi1UJ0e8/s1600-h/image%25255B20%25255D.png"><img alt="image" border="0" height="209" src="http://lh5.ggpht.com/-Moet8jPKBXM/UQhQn3EN4PI/AAAAAAAAAPQ/-MFHXG5M91g/image_thumb%25255B11%25255D.png?imgmax=800" style="background-image: none; border-bottom: 0px; border-left: 0px; border-right: 0px; border-top: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="244" /></a><a href="http://lh5.ggpht.com/-V_6t9rYQ_RM/UQhQo_k5j6I/AAAAAAAAAPU/OdnZCx0fwWI/s1600-h/image%25255B18%25255D.png"><img alt="image" border="0" height="68" src="http://lh5.ggpht.com/-0QNQlTrZ_Kg/UQhQp85xtHI/AAAAAAAAAPc/FnCuKscJblo/image_thumb%25255B9%25255D.png?imgmax=800" style="background-image: none; border-bottom: 0px; border-left: 0px; border-right: 0px; border-top: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="244" /></a><br />
<br />
As we can see from the above we have reached a tipping point. When the number of clients making invalid requests increases to 3, we can see initially the service respond times and calls/second are ok. When the three client tasks with invalid passwords start the throughput of the service is severely throttled to approximately 10 calls/sec. I would also draw your attention to the <b>Calls Duration</b> metric which is close to the other two runs.<br />
<br />
<h2>
10 Tasks valid credentials , 2 Tasks invalid credentials, Invalid Credential sleep time 5 sec</h2>
<br />
<a href="http://lh4.ggpht.com/-2NB-CXzJpMM/UQhQrH3P6eI/AAAAAAAAAPo/NkCZYf1UT4g/s1600-h/image%25255B11%25255D.png"><img alt="image" border="0" height="209" src="http://lh5.ggpht.com/-zI_XMenT4Ko/UQhQsgamb3I/AAAAAAAAAPw/Uv44mRqzmMw/image_thumb%25255B4%25255D.png?imgmax=800" style="background-image: none; border-bottom: 0px; border-left: 0px; border-right: 0px; border-top: 0px; display: inline; margin: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="244" /></a><a href="http://lh3.ggpht.com/-btn_Y68p5Pw/UQhQtjcoCCI/AAAAAAAAAP4/rgyDXHvYhjc/s1600-h/image%25255B16%25255D.png"><img alt="image" border="0" height="66" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVVHSPKRMtMu7LsLqJUYVXAtm23Nsi62pvjNH1c5U6AsHPDHguIme_tIi3tOnK1bB3Khmu7hfYoeYWut7UoljuwZJ3eAUl0cg8rimL-5tEOORNu4_F4o55SRm7NHe54Ec9PPA-mhV2UfvK/?imgmax=800" style="background-image: none; border-bottom: 0px; border-left: 0px; border-right: 0px; border-top: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="244" /></a><br />
<br />
When the number of clients making invalid requests is set to 2 but the sleep time is increased to 5 seconds we see a very erratic calls/second profile. Again the <b>Calls Duration</b> metric at the service level is in line with the other tests, but if measured from the client we would see a higher Calls Duration. The <b>Calls Outstanding</b> metric is also not zero more often than in the previous tests.<br />
<br />
<h2>
10 Tasks valid credentials , 4 Tasks invalid credentials, Invalid Credential sleep time 1 sec</h2>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1r6yHjqMBWoqqsrxGkIgGrcrjgCeUF4frG8Acfd8CwiDqFHcOVX92fGCQo60nYTrXZvb8eVfFhwTLJcg1GazyQMZH-JuDvdoYgSJjM0DGns8l-a68WiMnUYjKJgYFlLCQ-FDZWdgXo0ii/s1600-h/image%25255B23%25255D.png"><img alt="image" border="0" height="209" src="http://lh6.ggpht.com/-YNJPLwx3cTs/UQhQwX2I_VI/AAAAAAAAAQM/_R-_Gofswyg/image_thumb%25255B12%25255D.png?imgmax=800" style="background-image: none; border-bottom: 0px; border-left: 0px; border-right: 0px; border-top: 0px; display: inline; margin: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="244" /></a><br />
<br />
When the number of clients making invalid requests is set to 4 with a sleep time of 1 seconds, we can see initially the service respond times and calls/second are ok. When the When the 4 client tasks with invalid passwords start the calls/second profile of the service is very erratic. Again the <b>Calls Duration</b> metric at the service level is in line with the other tests.<br />
<br />
In a real world environment it is clear that you will not see such clear patterns to the calls/second metric. It is more likely that the service will in fact not process any requests if the client invalid call pattern is more random and at a higher volume.<br />
<br />
<h3>
Crash Dump Analysis</h3>
<br />
If we take a crash dump of the WCF Service process while it is in the “Deadlocked/Blocked” state we can examine the processes threads.<br />
<br />
<span style="font-family: Courier New; font-size: xx-small;">0:000> !threads<br />ThreadCount: 12<br />UnstartedThread: 0<br />BackgroundThread: 11<br />PendingThread: 0<br />DeadThread: 0<br />Hosted Runtime: no<br /> PreEmptive Lock<br /> ID OSID ThreadOBJ State GC GC Alloc Context Domain Count APT Exception<br />0 1 16d0 0000000000205eb0 a020 Enabled 0000000000000000:0000000000000000 00000000001f9780 1 MTA<br />2 2 184c 000000000020c1f0 b220 Enabled 0000000000000000:0000000000000000 00000000001f9780 0 MTA (Finalizer)<br />6 a 1708 000000001b9df9d0 a009220 Enabled 0000000000000000:0000000000000000 00000000001f9780 0 MTA <b><span style="color: red;">(Threadpool Completion Port)</span></b><br />8 6 1ae8 000000001b927570 a009220 Enabled 0000000000000000:0000000000000000 00000000001f9780 0 MTA <b><span style="color: red;">(Threadpool Completion Port)</span></b><br />9 4 19fc 000000001b926cf0 8009220 Enabled 0000000000000000:0000000000000000 00000000001f9780 0 MTA <span style="color: red;"><b>(Threadpool Completion Port</b>)</span><br />10 8 1958 000000001b9e1e40 8009220 Enabled 0000000000000000:0000000000000000 00000000001f9780 0 MTA <b><span style="color: red;">(Threadpool Completion Port)</span></b><br />11 b e84 000000001b9e6c60 8009220 Enabled 00000000033bb8c0:00000000033bd178 00000000001f9780 0 MTA <span style="color: red;"><b>(Threadpool Completion Port)</b></span>13 c e18 000000001b9f2260 8009220 Enabled 0000000000000000:0000000000000000 00000000001f9780 0 MTA <b><span style="color: red;">(Threadpool Completion Port</span></b>)<br />14 5 1a58 000000001b90c4a0 8009220 Enabled 0000000000000000:0000000000000000 00000000001f9780 0 MTA <b><span style="color: red;">(Threadpool Completion Port)</span></b><br />17 9 728 000000001af1f7c0 8009220 Enabled 0000000000000000:0000000000000000 00000000001f9780 0 MTA <b><span style="color: red;">(Threadpool Completion Port)</span></b><br />18 d 1aec 000000001af1ef40 8009220 Enabled 0000000000000000:0000000000000000 00000000001f9780 0 MTA <b><span style="color: red;">(Threadpool Completion Port)</span></b><br />19 3 1be0 000000001aed7e00 8009220 Enabled 0000000000000000:0000000000000000 00000000001f9780 0 MTA <b><span style="color: red;">(Threadpool Completion Port)</span></b></span><br />
<br />
<br />
We can see that all of the threads are running but the Exception column which lists the last thrown exception (if any) for the thread, shows (ThreadPool Completion Port). We can exclude a few of the threads from inspection, for example thread 0 is the thread executing the Microsoft.ServiceModel.Samples.CalculatorService.Main() method. Thread 2 is the CLR Finalizer thread.<br />
<br />
If we look at a few of the other threads call stacks, shortened for brevity, we see the following:<br />
<br />
<span style="font-size: xx-small;">---------------------------------------------<br />Thread 17<br />Current frame: ntdll!NtRemoveIoCompletion+0xa<br />Child-SP RetAddr Caller, Callee<br />000000001d5deb40 000007fefdf616ad KERNELBASE!<b><u><span style="font-size: x-small;">GetQueuedCompletionStatus</span></u></b>+0x39, calling ntdll!ZwRemoveIoCompletion<br />000000001d5deb50 000007fef4d46b1c clr!GCHolderBase<1,0,0,1>::Pop+0x2c, calling clr!Thread::EnablePreemptiveGC<br />000000001d5deb80 000007fef4e309ff clr!Thread::InternalReset+0x140, calling clr!Thread::SetBackground<br />000000001d5deba0 00000000777aa4e1 kernel32!GetQueuedCompletionStatusStub+0x11, calling kernel32!GetQueuedCompletionStatus<br />000000001d5debb0 000007fef4d7dd49 clr!Thread::LeaveRuntime+0x9, calling clr!Thread::LeaveRuntimeNoThrow<br />000000001d5debe0 000007fef4f1eb2f clr!ThreadpoolMgr::CompletionPortThreadStart+0x113, calling kernel32!GetQueuedCompletionStatusStub<br />000000001d5dec50 000007fef4d46a14 clr!operator delete+0x45, calling kernel32!GetLastErrorStub</span><span style="font-size: xx-small;">---------------------------------------------<br />Thread 18<br />Current frame: ntdll!NtRemoveIoCompletion+0xa<br />Child-SP RetAddr Caller, Callee<br />000000001ce7ec10 000007fefdf616ad KERNELBASE!<b><u><span style="font-size: x-small;">GetQueuedCompletionStatus</span></u></b>+0x39, calling ntdll!ZwRemoveIoCompletion<br />000000001ce7ec20 000007fef4d46b1c clr!GCHolderBase<1,0,0,1>::Pop+0x2c, calling clr!Thread::EnablePreemptiveGC<br />000000001ce7ec50 000007fef4e309ff clr!Thread::InternalReset+0x140, calling clr!Thread::SetBackground<br />000000001ce7ec70 00000000777aa4e1 kernel32!GetQueuedCompletionStatusStub+0x11, calling kernel32!GetQueuedCompletionStatus<br />000000001ce7ec80 000007fef4d7dd49 clr!Thread::LeaveRuntime+0x9, calling clr!Thread::LeaveRuntimeNoThrow<br />000000001ce7ecb0 000007fef4f1eb2f clr!ThreadpoolMgr::CompletionPortThreadStart+0x113, calling kernel32!GetQueuedCompletionStatusStub<br />000000001ce7ed20 000007fef4d46a14 clr!operator delete+0x45, calling kernel32!GetLastErrorStub</span><br />
<br />
As mentioned earlier <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/aa364986%28v=vs.85%29.aspx">GetQueuedCompletionStatus</a> is a blocking operation.<br />
<br />
Looking at Thread 8 we can see that it is the only thread executing the <b>Validate</b> method of the <b>CustomUsernameValidator</b>.<br />
<br />
<span style="font-size: xx-small;">Thread 8<br />Current frame: ntdll!NtDelayExecution+0xa<br />Child-SP RetAddr Caller, Callee<br />000000001c4cd200 000007fefdf61203 KERNELBASE!SleepEx+0xab, calling ntdll!ZwDelayExecution<br />000000001c4cd220 000007fee14f2f9a (MethodDesc 000007fee148ffb0 +0x4a System.Xml.XmlBufferReader.GetString(Int32, Int32)), calling (MethodDesc 000007fef39e5540 +0 System.String.CtorCharArrayStartLength(Char[], Int32, Int32))<br />000000001c4cd270 000007fefdf6358b KERNELBASE!SleepEx+0x12d, calling ntdll!RtlActivateActivationContextUnsafeFast<br />000000001c4cd2a0 000007fef4d76455 clr!EESleepEx+0x22, calling kernel32!SleepExStub<br />000000001c4cd2d0 000007fef4f33fba clr!Thread::UserSleep+0x97<br />000000001c4cd310 000007fee14e95bf (MethodDesc 000007fee1470360 +0x2f System.Xml.XmlBaseReader.get_Value()), calling clr!JIT_WriteBarrier_Fast<br />000000001c4cd320 000007fef4f3410c clr!ThreadNative::Sleep+0x100, calling clr!Thread::UserSleep<br />000000001c4cd3d0 000007fee166fca5 (MethodDesc 000007fee148f5e8 +0x85 System.Xml.XmlBaseReader.ReadElementString())<br />000000001c4cd3e0 000007fee167ed39 (MethodDesc 000007fee148d818 +0x19 System.Xml.XmlDictionaryReader.ReadStartElement(System.Xml.XmlDictionaryString, System.Xml.XmlDictionaryString))<br />000000001c4cd3e8 000007ff00160be6 (MethodDesc 000007ff001b4f98 +0x116 <b>Microsoft.ServiceModel.Samples.CalculatorService+CustomUserNameValidator.Validate(System.String, System.String)), calling (MethodDesc 000007fef3a5cd98 +0 System.Threading.Thread.Sleep(Int32))</b></span><br />
<br />
<h3>
Summary</h3>
<br />
In my opinion it is evident from the test results and the stack trace analysis presented in this post that there is some inherent "queuing/pipelining" in the processing of messages with credentials. Whether or not this is due to the fact that a custom <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.selectors.usernamepasswordvalidator%28v=vs.100%29.aspx">UserNamePasswordValidator</a> is being used is unclear. It is also evident that this “pipeline” is sensitive to execution latency of the Validate method of the <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.selectors.usernamepasswordvalidator%28v=vs.100%29.aspx">UserNamePasswordValidator</a>, possibly as a side effect of being a shallow queue.<br />
<br />
Either a number of requests of moderate latency or a smaller number of requests with high latency can have a impact on the overall performance of the service. It is therefore something to be aware of when implementing custom <a href="http://msdn.microsoft.com/en-us/library/system.identitymodel.selectors.usernamepasswordvalidator%28v=vs.100%29.aspx">UserNamePasswordValidator</a> types that perform credential validation that may be subject to non trivial execution latency either due to I/O or CPU. <br />
<br />
In my opinion this behaviour could be exploited to create an effective DOS attack, as the attacker does not have to create a huge request load to affect the WCF services performance. In the specific case that I discovered in the production environment I modified the code to mitigate the problem while remaining secure.
<br />
<iframe src="https://skydrive.live.com/embed?cid=C6E922F6206F9897&resid=C6E922F6206F9897%21203&authkey=AF_9WU9A8EzMMZc" width="98" height="120" frameborder="0" scrolling="no"></iframe>Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-71747354188781563852012-12-05T21:00:00.000+00:002012-12-05T21:00:00.990+00:00Simple Metadata Driven Linq To Sql Type GeneratorAs part of a larger <a href="http://corememorydump.blogspot.ie/2012/02/wcf-data-services-reflection-custom.html">body of work</a>, I implement a “Dynamic” <a href="http://corememorydump.blogspot.ie/2012/03/configurable-odata-service-part-2-data.html">Linq to Sql Type generator</a> that was based on metadata. A number of people have asked about the source code for this generator so I have decided to publish this portion of the code as a stand alone component.<br />
<h3>
DumpingCoreMemory.Data.Linq.Generation assembly</h3>
<ul>
<li><b>LinqToSqlTypeGenerator:</b> This class generates the types based on metadata provided at runtime. </li>
<li><b>EntityDescriptor:</b> Instances of this class define entities that the <b>LinqToSqlTypeGenerator</b> class will generate concrete runtime types for </li>
<li><b>EntityPropertyDescriptor:</b> Instances of this class define properties of the concrete runtime types that the <b>LinqToSqlTypeGenerator</b> will generate</li>
</ul>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5JoWR4c98kcSR0DwAr9arvz7FmAgPcTm4EHZMolvBdjX2RzOTBLaAhcBoLwBZc7tLIKilBxAFNiPUC8RNRSW2VPOMnYykVl8ogeKqv4YX6fWAZdbxfpRqKe6FJWfwjrIZegv0rxXE7tu7/s1600-h/ClassDiagram%25255B4%25255D.png"><img alt="ClassDiagram" border="0" height="172" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8avHAn7VCIdGv_DQnlQvu9oJ5OnQhETFA20GoY_t1gAXQrbHdKZXDF_igdoe0rlMhqvA6o2RLCj_XJgDws30BJJaQ2363efd2aa7i1Uwi-pZSKHR4nppwGCEDKZout6dkDJ1IWpwmp_sF/?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="ClassDiagram" width="644" /></a><br />
<h3>
How To Use the Type Generator</h3>
Included in the attached source code zip file is a contrived example, which briefly demonstrates how the type generator can be used. The example assumes a database which contains a table called <b>TableA</b> which contains three columns <b>Id</b>, <b>ColumnA</b> and <b>ColumnB</b>. In the example code this table is to be mapped to a Type called <b>EntityA</b> with three properties <b>Identity</b>, <b>PropertyA</b> and <b>PropertyB</b>. The type <b>EntityA</b> is to be derived from an interface <b>IEntity.</b> This is to enable demonstration of one of the ways a generated Type can be manipulated. The code for the mapping is as follows:<br />
<pre class="brush: c-sharp; toolbar: false">private static List<EntityDescriptor> GenerateEntityDescriptors()
{
var entityDescriptors = new List<EntityDescriptor>
{
new EntityDescriptor
{
BaseType = typeof (IEntity),
EntityName = "EntityA",
TableName = "TableA",
EntityProperties = new List<EntityPropertyDescriptor>
{
new EntityPropertyDescriptor
{
ColumnName = "Id",
Name = "Identity",
Type = typeof (Int32),
IsIdentity = true
},
new EntityPropertyDescriptor
{
ColumnName = "ColumnA",
Name = "PropertyA",
Type = typeof (String)
},
new EntityPropertyDescriptor
{
ColumnName = "ColumnB",
Name = "PropertyB",
Type = typeof (Int32)
}
},
}
};
return entityDescriptors;
}</pre>
<br />
The generator is invoked which generates the types, in the example I pass in an assembly file name, this is not necessary but for the purpose of the example it is useful as we can have a look at the generated types.<br />
<pre class="brush: c-sharp; toolbar: false">var entityDescriptors = GenerateEntityDescriptors();
var generator = new LinqToSqlTypeGenerator();
IList<Type> generatedTypes;
#warning Ensure that the process that invokes this method has rights to write to the disk
// Passing the assemblyFileName parameter is not necessary.
// The generator will write the assembly to the specified filename.
// This allows us to reuse the generated types or just take a look at them with ILSpy
generator.GenerateTypes("LinqToSqlGeneratedTypes", "LinqToSqlGeneratedTypesAssembly.dll",
entityDescriptors,
out generatedTypes);</pre>
<br />
Using <a href="http://ilspy.net/">ILSpy</a> we can inspect the generated code that was written to the assembly LinqToSqlGeneratedTypesAssembly.dll.<br />
<br />
<a href="http://lh5.ggpht.com/-Y04pcKj50Dw/ULz-RrWKMUI/AAAAAAAAANI/XWiztFtzw2A/s1600-h/GeneratedCode%25255B5%25255D.png"><img alt="GeneratedCode" border="0" height="484" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtzWSEXyVUOJJwuibHmRarWb8PxI_QvBZp4S1EIG3v8ZRQKJvR-iTV6wQ3tai4z4Td2tqxv4T1xjN5s0cVmtPLYMHbgqIV2ypAjC-TGldbm1MO_7U0Afp55eueGscuvYHDXutfmaDIPIaG/?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="GeneratedCode" width="308" /></a><br />
<br />
As one would expect we see a simple object with mapping attributes to the underlying database table called <b>TableA</b>. The next step is to activate and use the type for this we have a number of options:<br />
<ol>
<li>Statically reference the type, this is not shown in the test application as it is trivial to reference the assembly and instantiate its types. It also kind of defeats the purpose of generating types dynamically </li>
<li>Cast to a known base type, in the test application the base type of the generated Type was specified as <b>IEntity</b>. The instance of the type is cast to an <b>IEntity</b> type so that properties can be accessed, the <b>EntityDescriptor </b>for the type specified a matching property for the <b>IEntity</b> interfaces specified <b>Identity</b> property </li>
<li>Use a <a href="http://msdn.microsoft.com/query/dev10.query?appId=Dev10IDEF1&l=EN-US&k=k%28DYNAMIC_CSHARPKEYWORD%29;k%28DYNAMIC%29;k%28TargetFrameworkMoniker-%22.NETFRAMEWORK%2cVERSION%3dV4.0%22%29;k%28DevLang-CSHARP%29&rd=true">dynamic</a> type to contain the type. This allows bypass of static type checking to allow access to properties of a type at runtime </li>
<li>Use reflection to get or set properties of the underlying instance, using the name of the property for access</li>
</ol>
<br />
The following code snippet shows instantiation of an instance of the generated type and access by options 2,3 and 4.<br />
<pre class="brush: c-sharp; toolbar: false">object instance = Activator.CreateInstance(generatedTypes[0]);
var instanceAsIEntity = (IEntity) instance;
dynamic o = instance;
Console.WriteLine("Accessed as interface: {0}", instanceAsIEntity.Identity);
Console.WriteLine("Accessed as dynamic: {0}", o.Identity);
Console.WriteLine("Accessed by reflection: {0}", GetPropertyValueByReflection(instance, "Identity"));</pre>
<br />
Persisting instances of the types to a database is straight forward. The method <b>PersistEntity</b> in the test application shows how to insert an instance to a database.<br />
<pre class="brush: c-sharp; toolbar: false">private static void PersistEntity(object instance)
{
using (var sqlConnection = new SqlConnection("Server=localhost;Database=Test;Trusted_Connection=True;"))
{
using (var context = new DataContext(sqlConnection))
{
ITable table = context.GetTable(instance.GetType());
table.InsertOnSubmit(instance);
context.SubmitChanges();
}
}
}</pre>
<br />
For simplicity the code sets the properties via the dynamic object reference. In a real world example the instance may be bound to a UI control prior to CRUD operations being performed.<br />
<br />
<h4>
Limitations</h4>
<br />
<li>There is no abstraction of the underlying types from the data store. So if the underlying column is of type nvarchar then the property of the generated entity must be a String. The generator does not support type coercion or conversion which can get messy. The WCF service that allows the definition of the types metadata ensures that this cannot occur </li>
<li>There is no abstraction from the underlying data stores foreign key relationships. So if a type is mapped to a table that requires a foreign key value to be specified which is not exposed via the generated type then new rows cannot be inserted </li>
<li>There is no support for nested relationships. i.e a Thing contains a list of Parts. For the use cases of the overall project this is not a key requirement and can be added with more elaborate metadata and a change to the type generator <br /><iframe frameborder="0" height="120" scrolling="no" src="https://skydrive.live.com/embed?cid=C6E922F6206F9897&resid=C6E922F6206F9897%21202&authkey=AI6YFZxYY4oVc8Y" width="98"></iframe></li>
Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-78238301951549234702012-10-04T19:00:00.000+01:002012-12-03T11:35:52.869+00:00Linq To Entities query with Nullable<Guid> exceptionWhile developing a WCF service I came across one of those runtime errors where the exception details are obscure and provide no real hint as to what one is doing wrong in the code. The service was exposing data and the get methods of the service supported a simple paging continuation mechanism based on the previous set of data retrieved from the service. <br />
<span style="font-size: xx-small;"><b><span style="color: red;">Exception: System.NotSupportedException: Unable to create a constant value of type 'System.Object'. Only primitive types ('such as Int32, String, and Guid') are supported in this context.</span></b> at System.Data.Objects.ELinq.ExpressionConverter.ConstantTranslator.TypedTranslate(ExpressionConverter parent, ConstantExpression linq)<br /> at System.Data.Objects.ELinq.ExpressionConverter.BinaryTranslator.TypedTranslate(ExpressionConverter parent, BinaryExpression linq)<br /> at System.Data.Objects.ELinq.ExpressionConverter.TranslateLambda(LambdaExpression lambda, DbExpression input)<br /> at System.Data.Objects.ELinq.ExpressionConverter.MethodCallTranslator.OneLambdaTranslator.Translate(ExpressionConverter parent, MethodCallExpression call, DbExpression& source, DbExpressionBinding& sourceBinding, DbExpression& lambda)<br /> at System.Data.Objects.ELinq.ExpressionConverter.MethodCallTranslator.OneLambdaTranslator.Translate(ExpressionConverter parent, MethodCallExpression call)<br /> at System.Data.Objects.ELinq.ExpressionConverter.MethodCallTranslator.TypedTranslate(ExpressionConverter parent, MethodCallExpression linq)<br /> at System.Data.Objects.ELinq.ExpressionConverter.TranslateExpression(Expression linq)<br /> at System.Data.Objects.ELinq.ExpressionConverter.MethodCallTranslator.UnarySequenceMethodTranslator.Translate(ExpressionConverter parent, MethodCallExpression call)<br /> at System.Data.Objects.ELinq.ExpressionConverter.MethodCallTranslator.TypedTranslate(ExpressionConverter parent, MethodCallExpression linq)<br /> at System.Data.Objects.ELinq.ExpressionConverter.Convert()<br /> at System.Data.Objects.ELinq.ELinqQueryState.GetExecutionPlan(Nullable`1 forMergeOption)<br /> at System.Data.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)<br /> at System.Data.Objects.ObjectQuery`1.System.Collections.Generic.IEnumerable<T>.GetEnumerator()</span><br />
The method that the client invokes is the following:<br />
<pre class="brush: csharp">public IList<DataContracts.TBL_Incident> GetIncidents(Guid? skipToken)
{
List<DataContracts.TBL_Incident> result = new List<DataContracts.TBL_Incident>();
ExceptionHandledOperation(delegate
{
using (IContextProxy context = _contextFactory.CreateObjectContext())
{
IQueryable<Domain.EntityModel.TBL_Incident> incidents;
if (!skipToken.HasValue)
{
incidents = (from i in context.TBL_Incident
orderby i.IncidentId
select i).Take(EntityModelServiceSettings.Current.EntitySetPageSize);
}
else
{
incidents = (from i in context.TBL_Incident
orderby i.IncidentId
where i.IncidentId.CompareTo(skipToken) > 0
select i).Take(EntityModelServiceSettings.Current.EntitySetPageSize);
}
foreach (var i in incidents)
{
#warning optimise generated method so that this can be done in less LOC
var mappedIncident = new DataContracts.TBL_Incident();
DataContracts.TBL_Incident.MapTBL_Incident(i, mappedIncident);
result.Add(mappedIncident);
}
}
});
return result;
}
</pre>
<br />
The <b>skipToken</b> parameter allows the client to pass the last identifier from the previous page retrieved from the service.<br />
<pre class="brush: csharp">sw.Start();
var incidents = client.GetIncidents(null);
Console.WriteLine("Set: {0} Time: {1}", incidents.Count(), sw.Elapsed);
incidents = client.GetIncidents(incidents.Last().IncidentId);
Console.WriteLine("Set: {0} Time: {1}", incidents.Count(), sw.Elapsed);</pre>
<br />
The cause of the exception is the service side statement that appends the <b>skipToken</b> to the query.<br />
<pre class="brush: csharp">incidents = (from i in context.TBL_Incident
orderby i.IncidentId
where i.IncidentId.CompareTo(skipToken) > 0
select i).Take(EntityModelServiceSettings.Current.EntitySetPageSize);</pre>
<br />
The problem is that the compiler does not have a problem with the fact that the call passes the <b>skipToken</b> which is of type <b>Nullable<Guid></b>. The problem is when the LINQ to Entity classes attempt to map the expression to a SQL expression.<br />
<br />
To fix the problem it is simply a case of passing the skipToken.Value to the CompareTo() method.<br />
<pre class="brush: csharp">incidents = (from i in context.TBL_Incident
orderby i.IncidentId
where i.IncidentId.CompareTo(skipToken.Value) > 0
select i).Take(EntityModelServiceSettings.Current.EntitySetPageSize);</pre>
Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-35991551583305432712012-08-07T19:00:00.000+01:002012-08-07T19:00:02.577+01:00WCF Data Services NullReferenceException browsing metadataI have recently discovered an issue with malformed Uri base addresses and WCF Data Services when querying the metadata for the service.<br />
The issue causes an exception in the <b>System.Data.Services.HttpContextServiceHost</b> class which causes a http 500 in the client browser. Searching the internet did not yield much additional information in this problem.<br />
<br />
<br />
<a href="http://lh5.ggpht.com/-Ze_Qhmq5rwQ/UBv_-o2HMdI/AAAAAAAAALQ/TRHTcXpP9g0/s1600-h/wcfdataservicesexception3.png"><img alt="wcfdataservicesexception" border="0" height="123" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZPsJP-GQcMgn040cI7vRRWrTOpwA6zCl5XfJom998mpTEdwtKlgYpdD6LPImA0BnyT2O0lUK-I04cKszJGLPSyCzopYkjtDO5mgUQYAzEKtcBVveaD8JlBGBE4IWhw8IwT2f6gJ0uAHgy/?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="wcfdataservicesexception" width="644" /></a><br />
<br />
The offending line in the <b>HttpContextServiceHost</b> is the when the code accesses the <b>UriTemplateMatch</b> property, which is null, a <b>NullReferenceException</b> is thrown.<br />
<br />
<pre class="brush:csharp">internal void VerifyQueryParameters()
{
HashSet<string> set = new HashSet<string>(StringComparer.Ordinal);
<b>NameValueCollection queryParameters = this.operationContext.IncomingRequest.UriTemplateMatch.QueryParameters;</b>
</pre>
To discover the reason why this property is null, I need to debug much further back in the request processing phase of the WCF Data Services stack. <br />
<br />
Problem has its root in the the <b>public string SelectOperation(ref Message message)</b> of the <b>System.ServiceModel.Dispatcher.WebHttpDispatchOperationSelector</b> class. This operation is called during the http request that is made to the base address. You can see from the call stack below:<br />
<br />
<a href="http://lh6.ggpht.com/-SEk_Xlmyb0s/UCDSe3vSq9I/AAAAAAAAALw/28w3zhhBLc8/s1600-h/image%25255B5%25255D.png"><img alt="image" border="0" height="144" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgblK5g79xMM0wZZTR1OgIg3but2KGNtSNMFYelDQHvZDGxfarX_hqEqbc9SY8H7RkOsrszZtrDqjLvdoZTIkKK_01Jt5lASJFx1VNlxPW58LUTwvFeYAQPJW01Lg-Oeta8enCFaqkCW9U/?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="644" /></a><br />
<br />
From what I understand from stepping through the code the method <b>public string SelectOperation(ref Message message)</b> is attempting to determine the operation to invoke from the request message. The matched operation is then assigned to the message properties collection which is accessed later, via the <b>OperationContext.IncomingRequest.UriTemplateMatch</b>. The method <b>public string SelectOperation(ref Message message)</b> makes a call to the method <b>protected virtual string SelectOperation(ref Message message, out bool uriMatched)</b> which in turn calls an internal private method <b>private bool CanUriMatch(UriTemplateTable methodSpecificTable, Uri to, HttpRequestMessageProperty prop, Message message, out string operationName)</b> it is this method where the failure to match occurs and thus the <b>OperationContext.IncomingRequest.UriTemplateMatch</b> property is null.<br />
<pre class="brush:csharp">public string SelectOperation(ref Message message)
{
bool flag;
if (message == null)
{
throw DiagnosticUtility.ExceptionUtility.ThrowHelperArgumentNull("message");
}
string property = this.SelectOperation(ref message, out flag);
message.Properties.Add("UriMatched", flag);
if (property != null)
{
message.Properties.Add("HttpOperationName", property);
if (DiagnosticUtility.ShouldTraceInformation)
{
TraceUtility.TraceEvent(TraceEventType.Information, 0xf0025, SR2.GetString(SR2.TraceCodeWebRequestMatchesOperation, new object[] { message.Headers.To, property }));
}
}
return property;
}
protected virtual string SelectOperation(ref Message message, out bool uriMatched)
{
if (message == null)
{
throw DiagnosticUtility.ExceptionUtility.ThrowHelperArgumentNull("message");
}
uriMatched = false;
if (this.methodSpecificTables != null)
{
UriTemplateTable table;
if (!message.Properties.ContainsKey(HttpRequestMessageProperty.Name))
{
return this.catchAllOperationName;
}
HttpRequestMessageProperty requestProp = message.Properties[HttpRequestMessageProperty.Name] as HttpRequestMessageProperty;
if (requestProp == null)
{
return this.catchAllOperationName;
}
string method = requestProp.Method;
Uri to = message.Headers.To;
if (to == null)
{
return this.catchAllOperationName;
}
if (this.helpUriTable != null)
{
UriTemplateMatch match = this.helpUriTable.MatchSingle(to);
if (match != null)
{
uriMatched = true;
this.AddUriTemplateMatch(match, requestProp, message);
if (method == "GET")
{
return "HelpPageInvoke";
}
WebHttpDispatchOperationSelectorData property = new WebHttpDispatchOperationSelectorData {
AllowedMethods = new List<string> { "GET" }
};
message.Properties.Add("HttpOperationSelectorData", property);
return this.catchAllOperationName;
}
}
if (this.methodSpecificTables.TryGetValue(method, out table))
{
string str2;
uriMatched = this.CanUriMatch(table, to, requestProp, message, out str2);
if (uriMatched)
{
return str2;
}
}
if (this.wildcardTable != null)
{
string str3;
uriMatched = this.CanUriMatch(this.wildcardTable, to, requestProp, message, out str3);
if (uriMatched)
{
return str3;
}
}
if (this.ShouldRedirectToUriWithSlashAtTheEnd(table, message, to))
{
return "";
}
List<string> list2 = null;
foreach (KeyValuePair<string, UriTemplateTable> pair in this.methodSpecificTables)
{
if (((pair.Key != method) && (pair.Key != "*")) && (pair.Value.MatchSingle(to) != null))
{
if (list2 == null)
{
list2 = new List<string>();
}
if (!list2.Contains(pair.Key))
{
list2.Add(pair.Key);
}
}
}
if (list2 != null)
{
uriMatched = true;
WebHttpDispatchOperationSelectorData data2 = new WebHttpDispatchOperationSelectorData {
AllowedMethods = list2
};
message.Properties.Add("HttpOperationSelectorData", data2);
}
}
return this.catchAllOperationName;
}
private void AddUriTemplateMatch(UriTemplateMatch match, HttpRequestMessageProperty requestProp, Message message)
{
match.SetBaseUri(match.BaseUri, requestProp);
message.Properties.Add("UriTemplateMatchResults", match);
}
private bool CanUriMatch(UriTemplateTable methodSpecificTable, Uri to, HttpRequestMessageProperty prop, Message message, out string operationName)
{
operationName = null;
UriTemplateMatch match = methodSpecificTable.MatchSingle(to);
if (match != null)
{
operationName = match.Data as string;
this.AddUriTemplateMatch(match, prop, message);
return true;
}
return false;
}</pre>
<br />
When the Uri is matched the matched property is assigned to the “UriTemplateMatchResults” property of the request message. It is this property that is accessed by the <b>OperationContext.IncomingRequest.UriTemplateMatch</b> property and is the reason that it is returned as null and the cause of the exception.Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-62853840965048662152012-07-18T19:30:00.000+01:002012-07-18T19:30:00.309+01:00WIX Registry Key Exists Custom ActionThe <a href="http://wix.sourceforge.net/manual-wix3/wix_xsd_registrysearch.htm" target="_blank">RegistrySearch Element</a> in WIX has a bug where by you cannot check the existence of a Key with out a value. ClickOnce installers with a custom bootstrapper manifest also inherits this issue as I believe that it is a problem in the underlying Windows Installer.<br />
For a deployment component that I am currently building for an application I have a requirement to check if the .Net Platform Update 1 is installed as per the instructions <a href="http://blogs.msdn.com/b/rjacobs/archive/2011/04/29/how-to-detect-if-the-net-framework-platform-update-1-is-installed.aspx" target="_blank">here</a>.<br />
<br />
The key in question is <span style="color: #3d85c6;">HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\SKUs\.NETFramework,Version=v4.0.1</span>, which has no default value.<br />
<br />
After taking a look around on the internet I could not find a code snippet that was flexible and reusable. Most where simple hard coded examples on how to work around the problem. So I decided to build my own Custom Action and add it to an assembly of custom actions that I share and reuse between deployment components.<br />
<h2>
The Custom Action</h2>
The code is pretty straight forward. A key name, as specified by the session property <b>REGKEY</b>, is opened from the registry hive as specified by <b>REGHIVE</b> session property. The session property <b>REGKEYEXISTS</b> is used to store the result of the check.<br />
<pre class="brush: csharp><br /><p>public class CustomActions<br>{<br> /// <summary><br> /// Checks the registry for the existance of a key.<br> /// </summary><br> /// <remarks><br> /// Uses two session properties<br> /// REGKEY - The key to find<br> /// REGHIVE - The hive to use to search for the key<br> /// REGKEYEXISTS - indicates if the key exists<br> /// <br> /// Supports the following values:<br> /// HKLM - HKEY_LOCAL_MACHINE<br> /// HKCC - HKEY_CURRENT_CONFIG<br> /// HKCR - HKEY_CLASSES_ROOT<br> /// HKU - HKEY_USERS<br> /// HKCU - HKEY_CURRENT_USER<br> /// </remarks><br> /// <param name=" param>=""> /// <returns>Returns ActionResult.Success if the key can be found</returns>
/// <exception cref="ArgumentNullException">Thrown if the <paramref name="session"/> parameter is null.</exception>
[CustomAction]
public static ActionResult CheckRegistryKeyExists(Session session)
{
if (null == session)
{
throw new ArgumentNullException("session");
}
try
{
#if DEBUG
Debugger.Launch();
#endif
session.Log("CheckRegistryKeyExists: Begin");
String key = session["REGKEY"];
String hive = session["REGHIVE"];
RegistryKey registryKey = null;
session.Log("CheckRegistryKeyExists: Looking up key {0} in Hive {1}", key, hive);
switch (hive)
{
case "HKLM":
registryKey = Registry.LocalMachine.OpenSubKey(key);
break;
case "HKCC":
registryKey = Registry.CurrentConfig.OpenSubKey(key);
break;
case "HKCR":
registryKey = Registry.ClassesRoot.OpenSubKey(key);
break;
case "HKU":
registryKey = Registry.Users.OpenSubKey(key);
break;
case "HKCU":
registryKey = Registry.CurrentUser.OpenSubKey(key);
break;
default:
session.Log("Unknown hive {0}", hive);
break;
}
session["REGKEYEXISTS"] = registryKey == null ? "0" : "1";
if(registryKey != null)
{
registryKey.Close();
}
session.Log("CheckRegistryKeyExists: End");
}
catch (Exception ex)
{
session.Log("CheckRegistryKeyExists: exception: {0}", ex.Message);
throw;
}
return ActionResult.Success;
}
}
</pre>
<br />
<h2>
Using the Custom Action</h2>
<br />
To use the custom action in an installer first embed the custom action as a Binary Element.<br />
<pre >="" class="brush:xml><br /><p><Binary Id=" p="">< SourceFile="$(var.CustomActions.TargetDir)\Installer.CustomActions.CA.dll" CustomActions?>Then define the Custom Action and sequence it for execution, in this case it is before LaunchConditions.
<CustomAction Id="CheckRegistryKeyExists" BinaryKey="CustomActions" DllEntry="CheckRegistryKeyExists" Execute="immediate" Return="check" />
<InstallUISequence>
<Custom Action="CheckRegistryKeyExists" Before="LaunchConditions">NOT Installed</Custom>
</InstallUISequence>
</pre>
Then specify the required properties and setup the condition and message.<br />
<pre class="brush: csharp><br /><p><Property Id=" regkey?><![cdata[software\microsoft\.netframework\v4.0.30319\skus\.netframework,version="v4.0.1]]></property><br "><Property Id="REGHIVE">HKLM</Property>
<Property Id="REGKEYEXISTS">0</Property>
<Condition Message='!(loc.Update401Missing)'>Installed OR REGKEYEXISTS="1"</Condition>
</pre>
<br />
When the custom action is run with logging <b>msiexec /i Workflow.Designer.Installer.msi /lv out.txt </b>the following output is displayed to aid with debugging.<br />
<br />
<span style="color: #646b86;"><span style="font-size: small;">Action start 14:38:41: CheckRegistryKeyExists.<br />MSI (c) (A0:FC) [14:38:41:385]: Invoking remote custom action. DLL: C:\MSIA543.tmp, Entrypoint: CheckRegistryKeyExists<br />MSI (c) (A0:24) [14:38:41:387]: Cloaking enabled.<br />MSI (c) (A0:24) [14:38:41:387]: Attempting to enable all disabled privileges before calling Install on Server<br />MSI (c) (A0:24) [14:38:41:387]: Connected to service for CA interface.<br />SFXCA: Extracting custom action to temporary directory: C:\MSIA543.tmp-\<br />SFXCA: Binding to CLR version v4.0.30319<br />Calling custom action Emex.Installer.CustomActions!Emex.Installer.CustomActions.CustomActions.CheckRegistryKeyExists<br />CheckRegistryKeyExists: Begin<br />CheckRegistryKeyExists: Looking up key SOFTWARE\Microsoft\.NETFramework\v4.0.30319\SKUs\.NETFramework,Version=v4.0.1 in Hive HKLM<br />MSI (c) (A0!F4) [14:39:09:498]: PROPERTY CHANGE: Modifying REGKEYEXISTS property. Its current value is '0'. Its new value: '1'.<br />CheckRegistryKeyExists: End</span></span><br />
<br />
<h2>
Debugging</h2>
The statement <b>Debugger.Launch()</b> statement launches and attaches the debugger to the installer when the Custom Action is executed. In the above code it is surrounded with the #if DEBUG to prevent the debugger being launched in release mode.<br />
<br />
When the debug version of the installer is launched the following dialog is displayed. You can attach the VS project that contains the source for the Custom Action and simply step through the code as you normally would.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibrD13BD6SURIVllYtDz9D5Tz6WEDHZAHiXg_r_O05MgA_iSCpaIh4QqMUAli1D0T2RxsVaMA2jDy2H4rDdBhczZWuJdkglZFLcLoKHGtYMjKAvWHzc5AJoDzkeJ49_mXwEjiIoIu8W2xv/s1600-h/JITD%25255B2%25255D.png"><img alt="JITD" border="0" height="244" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjo70BqWB74Vf7HrD8thehH79rXkUHC7lnTl2bApLMnelLRtibQGdREFfJujiWzoi_VwnSKq9nz0gCaflBfiE8ZHTCa-n5Sfhlt-ca1W-G9wm_dz16MzttIRdtgadtg-4mkPRPi28hdJ2MK/?imgmax=800" style="background-image: none; border-bottom: 0px; border-left: 0px; border-right: 0px; border-top: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="JITD" width="225" /></a>Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-19657957115252791412012-07-17T19:00:00.000+01:002012-07-17T19:00:00.457+01:00WIX Light.exe LGHT0204: ICE03: ErrorRecently I have been developing a set of deployment components for the various system components of a workflow designer and execution subsystem. One of the deployment components required is to deploy a workflow and domain model designer to an end users PC.<br />
<a href="http://lh3.ggpht.com/-yHnS06rLKOc/UAWDY8pS_sI/AAAAAAAAAKY/QCG9FVkYcZE/s1600-h/WorkflowDesigner%25255B3%25255D.png"><img alt="WorkflowDesigner" border="0" height="357" src="http://lh5.ggpht.com/-ksULatR37iw/UAWDZ6KgNHI/AAAAAAAAAKg/fckIOUAsDbU/WorkflowDesigner_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="WorkflowDesigner" width="644" /></a><br />
Since there can be a wide variance as to what .net runtime components an end user has installed on their PC it was therefore a requirement to perform some up front checks as a prerequisite to an install on a given target PC.<br />
The application has three required components that must be deployed to a candidate client PC, these are as follows:<br />
<br />
<ul>
<li>.Net Framework 4.0</li>
<li>Multi-Targeting Pack for Microsoft .NET Framework 4.0.1 (KB2495638)</li>
<li>WCF Data Services 5.0 for OData V3</li>
</ul>
<br />
A WIX installer was created, the <b>Conditions.wxi</b> contains obviously enough the prerequisite conditions for the deployment. The <b>Product.wxs</b> contains all of the assembly references for the application files. The <b>Strings_en-us.wxl</b> obviously contains localised strings for the installer messages etc..<br />
<br />
<a href="http://lh3.ggpht.com/-HbyoUlNhXYg/UAWDanJTj3I/AAAAAAAAAKk/Iso-G6PO5Os/s1600-h/image%25255B2%25255D.png"><img alt="image" border="0" height="92" src="http://lh4.ggpht.com/-KycCUhKP1K0/UAWDbbs96rI/AAAAAAAAAKs/3LZHNR1JFTs/image_thumb.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; margin: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="244" /></a><br />
So the conditions where defined as shown in the following snippet:<br />
<br />
<pre class="brush:xml"><Property Id="UPDATE401">
<RegistrySearch Id="Update401x32" Type="raw" Root="HKLM" Key="SOFTWARE\Microsoft\.NETFramework\v4.0.30319\SKUs\.NETFramework,Version=v4.0.1" Win64="no" />
</Property><br />
<Property Id="UPDATE401X64">
<RegistrySearch Id="Update401x64" Type="raw" Root="HKLM" Key="SOFTWARE\Microsoft\.NETFramework\v4.0.30319\SKUs\.NETFramework,Version=v4.0.1" Win64="yes" />
</Property><br />
<Property Id="ODATA">
<RegistrySearch Id="ODataRegistry32" Type="raw" Root="HKLM" Key="SOFTWARE\Microsoft\Microsoft WCF Data Services\5.0" Name="Version" Win64="no" />
</Property><br />
<Property Id="ODATAX64">
<RegistrySearch Id="ODataRegistry64" Type="raw" Root="HKLM" Key="SOFTWARE\Microsoft\Microsoft WCF Data Services\5.0" Name="Version" Win64="yes" />
</Property><br />
<Condition Message='!(loc.Update401Missing)'><![CDATA[(UPDATE401 <> "" OR UPDATE401X64 <> "") OR Installed]]></Condition><br />
<Condition Message='!(loc.ODataMissing)'><![CDATA[(ODATA <> "" OR ODATAX64 <> "") OR Installed]]></Condition></pre>
<br />
The corresponding message strings are shown in the following snippet:<br />
<pre class="brush:xml"><String Id="ODataMissing" Overridable="yes">WCF Data Services 5.0 for OData V3 missing [<a href="http://www.microsoft.com/en-ie/download/details.aspx?id=29306]">http://www.microsoft.com/en-ie/download/details.aspx?id=29306]</a> </String>
<String Id="Update401Missing" Overridable="yes">Multi-Targeting Pack for Microsoft .NET Framework 4.0.1 (KB2495638) [<a href="http://www.microsoft.com/en-ie/download/details.aspx?id=27757]">http://www.microsoft.com/en-ie/download/details.aspx?id=27757]</a></String></pre>
<br />
In some deployments the end user will install the application I decided to add references to the download pages for the .Net framework components that this application requires to be deployed to the PC before the application will execute on the PC. In larger client environments the application and the dependant components will be deployed using automated deployment systems.<br />
<br />
On compiling the WIX application the following errors where reported.<br />
<br />
<span style="color: red; font-size: x-small;">Emex.Workflow.Designer.Installer\Conditions.wxi(20,0): error LGHT0204: ICE03: Invalid format string; Table: LaunchCondition, Column: Description, Key(s): (UPDATE401 <> "" OR UPDATE401X64 <> "") OR Installed<br />Emex.Workflow.Designer.Installer\Conditions.wxi(22,0): error LGHT0204: ICE03: Invalid format string; Table: LaunchCondition, Column: Description, Key(s): (ODATA <> "" OR ODATAX64 <> "") OR Installed</span><br />
<br />
At first this looks like the error is due to the condition expression being incorrect. But this is not the case. The problem is actually the message strings, the [ ] delimiters that I had used around the links, for no obvious reason, where the cause of the problem. To resolve the errors it was simply a case of escaping the delimiters as per the rules <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/aa368609%28v=vs.85%29.aspx" target="_blank">here</a>.<br />
<pre class="brush:xml"><String Id="ODataMissing" Overridable="yes">WCF Data Services 5.0 for OData V3 missing [\[]<a href="http://www.microsoft.com/en-ie/download/details.aspx?id=29306]">http://www.microsoft.com/en-ie/download/details.aspx?id=29306]</a></String></pre>Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-55410496826073629972012-06-14T19:00:00.000+01:002012-06-15T14:25:01.406+01:00Windows 8 Release PreviewI have been reading the interweb chatter about Windows 8 since it was released as beta so I decided to check it out myself and see what all of the fuss was about.<br />
<br />
So I finally got some time to download the Windows 8 release preview and install it on a VM. For the VM I used <a href="https://www.virtualbox.org/" target="_blank">VirtualBox</a> which in my opinion is pretty good, I actually have a preference for it over the VMWare tools, but that is a different discussion.<br />
<br />
<a href="http://lh5.ggpht.com/-Ekxm2JcbtjI/T9nHbIXv7tI/AAAAAAAAAJ4/jD7tDQmeqOU/s1600-h/screenshot_13.png"><img alt="screenshot_1" border="0" height="484" src="http://lh4.ggpht.com/-CqIu-LgDlHk/T9nHb2tLiXI/AAAAAAAAAKA/sLg0ezXFkAA/screenshot_1_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="screenshot_1" width="644" /></a><br />
<br />
Well what can I say I can definitely see why the new interface has been divisive. Its a radical change, more so than the changes from Windows 3.1 to Windows 95. From Windows 95 until Windows 7 we have been presented with evolutionary changes to the Start button, Start menu task bar combination. That's 17 years of relative familiarity, with Windows 8 we now have a a complete departure from this familiarity.<br />
<br />
Windows 8 is a derivation of the Metro interface that we have seen on Windows Phone and in terms of look and feel has some similarity to the Xbox 360 user interface.<br />
<br />
I can see where Microsoft are going with the UI and the integration with online services, Maps, Weather, Music, Xbox LIVE, Sky Drive etc.. Windows 8 is integrated with more online services than Windows 7, similar to the integration previously seen on Windows Phone.<br />
<br />
The fundamental question is does the UI work? In my opinion based on the limited exposure that I have had with the OS the answer is …… maybe. Depending on the device that you intend using the OS on and for what purpose.<br />
<br />
For general content consumption on a touch enabled tablet or laptop I think, with the caveat of not having used such a configuration, it looks like the right UI for this type of human device interaction.<br />
Another configuration that I think this OS will work well with is for a <a href="http://en.wikipedia.org/wiki/Home_theater_PC" target="_blank">HTPC</a>. Having used Windows 7 for a HTPC OS and tried several of the media centre application like <a href="http://xbmc.org/" target="_blank">XBMC</a>, I have found the results some what mixed. When using the media centre interfaces its a generally workable solution. When you have to step outside and use native windows applications, for scenarios where plugins don't exist you end up having to scale the fonts and tweak the display settings so that you can see the text from the normal TV to couch distances. These tweaks are never a very satisfactory solution and its possibly the reason that HTPC have never really caught on.<br />
<br />
The Windows 8 UI I believe alleviates most if not all of the problems for HTPC usage. Using a HTPC with a Kinect running Windows 8 OS could potentially be a Killer combination once the input gestures where useable and not to extreme, it would be super to flicking through the music or changing channel with a simple wave of a hand.<br />
<br />
Lazy Minority Report fantasies aside there is a lot of debate and discussion around the usability of Windows 8 in the traditional PC environment of a keyboard and mouse. And I have to say from my experience of using Windows 8 I would say its detractors have a point, I do not believe that it is just a problem with familiarity or learning the curve. There are a few things that don't work so well with Windows 8 with a keyboard and a mouse:<br />
<br />
<b>More mouse movement</b><br />
<br />
The landscape orientated screen real-estate in Windows 8 is much larger in some apps, notably Media Player. This leads you to have to scroll left and right quite a bit. With a mouse this is not much fun.<br />
<br />
<b>More interaction required</b><br />
<br />
Seemingly simple operations like shutting down the PC or closing applications from the Metro Start menu screen take more interaction clicks than in windows 7. See here for <a href="http://www.winsupersite.com/article/windows8/windows-8-tip-shut-restart-sleep-143414" target="_blank">shut down</a><br />
<br />
<b>Mouse accuracy</b><br />
<br />
I found launching some of the menu items from the lower left hand corner to be hit and miss, this may be down to the fact that i was running in a VM or just that I’m an idiot <img alt="Smile" class="wlEmoticon wlEmoticon-smile" src="http://lh6.ggpht.com/-vqavuTJg7wM/T9sPaIg-J-I/AAAAAAAAAKM/aXLkjt0sRg0/wlEmoticon-smile%25255B2%25255D.png?imgmax=800" style="border-bottom-style: none; border-left-style: none; border-right-style: none; border-top-style: none;" /><br />
<br />
<b>Cluttered Start</b><br />
<br />
When all of the applications are enabled on the Metro start screen it can get a bit cluttered looking. I put this down too the fact that there is no hierarchy available for nesting as with the old Windows start menuAnonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com0tag:blogger.com,1999:blog-2842308426499703302.post-90607607040122550862012-04-30T19:09:00.000+01:002012-05-08T09:20:11.464+01:00Configurable OData Service - Part 5 – End To End ExampleThis is the final post in the series on the configurable OData service. In this post I will cover an end to end example of how the <b>Domain Model Service</b> and be configured and used by an OData client. For the sake of simplicity I am going to expose a simple database with only one table, the table is called Employees. The script below shows the layout of the table.<br />
<br />
<pre class="brush:sql">CREATE TABLE [dbo].[Employees](
[EmployeeID] [uniqueidentifier] NOT NULL,
[LastName] [nvarchar](20) NOT NULL,
[FirstName] [nvarchar](10) NOT NULL,
[Title] [nvarchar](30) NULL,
[BirthDate] [datetime] NULL,
[HireDate] [datetime] NULL,
[Address] [nvarchar](60) NULL,
[City] [nvarchar](15) NULL,
[Region] [nvarchar](15) NULL,
[PostalCode] [nvarchar](10) NULL,
[Country] [nvarchar](15) NULL,
[HomePhone] [nvarchar](24) NULL,
[Extension] [nvarchar](4) NULL
CONSTRAINT [PK_Employees] PRIMARY KEY CLUSTERED
(
[EmployeeID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO</pre>
<br />
The <b>Domain Model Service</b> is deployed to IIS via its WIX based installer which takes care of configuring the service to execute under IIS.<br />
<br />
<a href="http://lh6.ggpht.com/-N636veWk7sA/T6GCzAPclDI/AAAAAAAAAGE/l1_R9krxzvo/s1600-h/DomainModelIIS%25255B3%25255D.png"><img alt="DomainModelIIS" border="0" height="216" src="http://lh6.ggpht.com/--VaNm-Xk2EU/T6GCz0ZdGeI/AAAAAAAAAGM/dOr1FGiymYU/DomainModelIIS_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="DomainModelIIS" width="644" /></a><br />
<br />
There are two services exposed from the <b>Domain Model Service</b><br />
<br />
<ul>
<li>DefinitionService.svc<br />
<ul>
<li>Implements the functionality to define the shape and structure of the exposed <b>Domain Model Service</b></li>
</ul>
</li>
<li>ManagementService.svc<br />
<ul>
<li>Implements functionality to start and stop instances of the <b>Domain Model Service</b></li>
</ul>
</li>
</ul>
<br />
Before jumping into the client console application I will just mention the configuration settings that are required to configure the service.<br />
<br />
<pre class="brush:xml"><?xml version="1.0" encoding="utf-8"?>
<domainModel
tablePrefixFilter=""
connectionStringName="DomainModelMetadata"
targetStoreConnectionStringName="TargetDatabase"
domainModelServiceBaseAddress="<a href="http://127.000.000.001:8080%22/">http://127.000.000.001:8080"/</a>></pre>
<br />
The previous configuration fragment contains the following elements:<br />
<ul>
<li>tablePrefixFilter<br />
<ul>
<li>A prefix that is applied by the service when enumerating tables from the database</li>
</ul>
</li>
<li>connectionStringName<br />
<ul>
<li>The connection string element name for the store that contains the Domain Model metadata</li>
</ul>
</li>
<li>targetStoreConnectionStringName<br />
<ul>
<li>The connection string element name for the target database that will be exposes as an OData service</li>
</ul>
</li>
<li>domainModelServiceBaseAddress<br />
<ul>
<li>The base address that a Domain Model Service will be exposed relative too. Notice that the service base address is http:// not https:// more on this later</li>
</ul>
</li>
</ul>
<span style="font-size: large;"><b>Listing the Target Schema</b></span><br />
<br />
After creating the proxy to the service. The first step is to retrieve the table metadata from the <b>Domain Model Service</b>, this is used as reference information to define a domain model.<br />
<pre class="brush:xml">DefinitionServiceClient client = new DefinitionServiceClient();
client.ClientCredentials.UserName.UserName = "xxxxxx";
client.ClientCredentials.UserName.Password = "xxxxxx";
client.ClientCredentials.ServiceCertificate.Authentication.CertificateValidationMode = System.ServiceModel.Security.X509CertificateValidationMode.None;
client.Open();
var tableMetadata = client.GetTableMetadata();
foreach (var item in tableMetadata)
{
Console.WriteLine(item.Name);
foreach (var column in item.Columns)
{
Console.WriteLine("\t" + column.Name);
}
}</pre>
<br />
For the example database this produces the following output:<br />
<br />
<a href="http://lh3.ggpht.com/-rS-oBccrQRA/T6OsWuJ9T4I/AAAAAAAAAGs/SYvX6gDcO0k/s1600-h/schemalist%25255B4%25255D.png"><img alt="schemalist" border="0" height="190" src="http://lh3.ggpht.com/-Uushs9gF0q4/T6OsXBDLyXI/AAAAAAAAAG0/vhLrd6dbtSI/schemalist_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="schemalist" width="644" /></a><br />
<br />
We now have the list of the tables and columns from the target database. This information is intended for consumption in a designer that will allow for a Domain Model Service to be defined by an end user.<br />
<br />
<span style="font-size: large;"><b>Defining and Activating Domain Model Service</b></span><br />
<br />
Now that we can query the table metadata the next step is to create a Domain Model Definition. This is achieved by creating a proxy to the definition service and calling the CreateDomainModelDefinition<br />
<pre class="brush:sql">DefinitionServiceClient client = new DefinitionServiceClient();
client.Open();
...
domainModelDefinition = client.CreateDomainModelDefinition(domainModelDefinition);</pre>
<br />
Then the Domain Model needs to be activated. This is achieved by calling the ActivateDomainModel method of the management service proxy.<br />
<br />
<pre class="brush:csharp">ManagementServiceClient managmentServiceClient = new ManagementServiceClient();
managmentServiceClient.ActivateDomainModel(domainModelDefinition.Identifier);</pre>
<br />
The queried metadata is mapped to the entities that are to be exposed by instances of a Domain Model service. In this example I will expose two versioned “views” of the Employees entity. As an example of how the exposed Domain Model Service entity can diverge from the underlying database I will expose the Employees entity as an entity called People. <br />
<br />
<span style="font-size: large;"><b>Version 1</b></span><br />
<pre class="brush:csharp">DomainModelEntity domainModelEntity = new DomainModelEntity();
domainModelEntity.TableName = "Employees";
domainModelEntity.TypeName = "People";
domainModelEntity.Properties = new List<DomainModelProperty>();
domainModelEntity.Properties.Add(new DomainModelProperty()
{
ColumnName = "EmployeeID",
PropertyName = "Identifier",
IsIdentity = true
});
domainModelEntity.Properties.Add(new DomainModelProperty()
{
ColumnName = "FirstName",
PropertyName = "Givename"
});
domainModelEntity.Properties.Add(new DomainModelProperty()
{
ColumnName = "LastName",
PropertyName = "Surname"
});
DomainModelDefinition domainModelDefinition = new DomainModelDefinition()
{
Entities = new List<DomainModelEntity>(),
Comments = "Version 1"
};
domainModelDefinition.Entities = new List<DomainModelEntity>();
domainModelDefinition.Entities.Add(domainModelEntity);</pre>
<br />
For version one of the Domain Model the Employee table is exposed as an entity called People. The three required fields are exposed as aliases of the underlying columns. The Employee is mapped to a field called Identifier. After the definition is created we can query the service at the address “http://localhost:8081/V1/DomainModelService” which will describe the atom for the “Peoples” entity set.<br />
<br />
<a href="http://lh6.ggpht.com/-ubY9fon-Glk/T6O3JgqAXYI/AAAAAAAAAHA/zuodCUqKMBQ/s1600-h/version1%25255B3%25255D.png"><img alt="version1" border="0" height="201" src="http://lh4.ggpht.com/-4f4J1gmtqg8/T6O3KPCsquI/AAAAAAAAAHI/gi3xaTJSYqw/version1_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="version1" width="644" /></a><br />
<br />
The metadata can be queried from this OData service at “http://localhost:8081/V1/DomainModelService/$metdata”, which shows only the three fields that where mapped.<br />
<br />
<a href="http://lh4.ggpht.com/--ITq0xmt9N8/T6O3Ls4KExI/AAAAAAAAAHQ/tGy5RctcWl0/s1600-h/version1metadata%25255B3%25255D.png"><img alt="version1metadata" border="0" height="313" src="http://lh5.ggpht.com/-zUZ1AeQGBzM/T6O3NmVH0cI/AAAAAAAAAHY/_ZTvy9C_1ms/version1metadata_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="version1metadata" width="644" /></a><br />
<br />
Using the <a href="http://www.silverlight.net/content/samples/odataexplorer/default.html" target="_blank">ODataExplorer</a> tool we can perform some query and updates to the data exposed by the service. <br />
<br />
<a href="http://lh6.ggpht.com/-2DZAXF0l7SU/T6O9wg_wDBI/AAAAAAAAAHk/-s1o0FmD2lo/s1600-h/OdataV1%25255B3%25255D.png"><img alt="OdataV1" border="0" height="334" src="http://lh6.ggpht.com/-f4u86wo0lVI/T6O9xLqOxqI/AAAAAAAAAHs/QOVo3fdXtLs/OdataV1_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="OdataV1" width="644" /></a><br />
<br />
Unfortunately the ODataExplorer does not allow you to insert entities, so for the example I pre seeded the table with a number of rows. And yes i am that unoriginal to come up with the names of people.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNgI0O8RCOWsuV30UzMqGCTX-fSQG1n5nkpdLd_42MqyP_5mgw9butP65T3Dv7Cy8EkGyTJkm6XvpOcgtxjQXKIfq0u0ABPs_fOMMBLO-ksklgRKAMCNwbC2HAeAiOySinWw9m38lVvYDK/s1600-h/OdataV1Data%25255B3%25255D.png"><img alt="OdataV1Data" border="0" height="318" src="http://lh5.ggpht.com/-CEtHbrCr1iQ/T6O9zzW2B6I/AAAAAAAAAH8/cYTKMM3V8Po/OdataV1Data_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="OdataV1Data" width="644" /></a><br />
<br />
The data can be modified using the ODataExplorer<br />
<br />
<a href="http://lh3.ggpht.com/-KtcssGgMj_Q/T6O90djq6sI/AAAAAAAAAIE/I2oatwJWDPg/s1600-h/OdataV1DataChange%25255B3%25255D.png"><img alt="OdataV1DataChange" border="0" height="484" src="http://lh6.ggpht.com/-Uqqnu_FP-MM/T6O90i4cljI/AAAAAAAAAIM/AZ1wDYLS2Rk/OdataV1DataChange_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="OdataV1DataChange" width="368" /></a><br />
<br />
We can see the changes reflected in the database<br />
<br />
<a href="http://lh4.ggpht.com/-yVxj9sc1zMo/T6O91POcB9I/AAAAAAAAAIU/Ax7ViUQ8-Rk/s1600-h/RawDatachanged%25255B3%25255D.png"><img alt="RawDatachanged" border="0" height="64" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigteFxoRTF7Zo-rWNE3xCHsJbDLIgoc6a9_j3vRB_yyN0VvBPwQbv0PqQbtcCbq4BjHBwDD_e8W2Rd02-aeuiT6vZI7zTsmicRfqpbsBL6xqk4j6tDRgiRJR1vOPuV0H8rwpAWPXpvb3Lo/?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="RawDatachanged" width="644" /></a><br />
<br />
Just to prove that this is a fully functioning OData service you can see that the data can also be queried.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpuGsFxcoZnXiaKNbbZudp-ai44mlv9wgUOJ75et8QqO2cZmhR-VXDzDR4f9LQbVxiChYHC3If9_fJgJKTEq0VKrEX-Anog514Z6XVANN2YhfvFzcNDOuGYQ6qBanx_EIUZ3DCjKHhnCaG/s1600-h/OdataV1DataQuery%25255B3%25255D.png"><img alt="OdataV1DataQuery" border="0" height="165" src="http://lh6.ggpht.com/-tqm-EOS6Bmc/T6O92NtqmnI/AAAAAAAAAIs/bocAzW4mUYg/OdataV1DataQuery_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="OdataV1DataQuery" width="644" /></a><br />
<br />
<span style="font-size: large;"><b>Version 2</b></span><br />
For Version 2 of the Domain Model we can add a more of the underlying fields to the entities. For the sake of brevity I will add only one the BirthDate.<br />
<br />
<pre class="brush:csharp">DomainModelEntity domainModelEntity = new DomainModelEntity();
domainModelEntity.TableName = "Employees";
domainModelEntity.TypeName = "People";
domainModelEntity.Properties = new List<DomainModelProperty>();
domainModelEntity.Properties.Add(new DomainModelProperty()
{
ColumnName = "EmployeeID",
PropertyName = "Identifier",
IsIdentity = true
});
domainModelEntity.Properties.Add(new DomainModelProperty()
{
ColumnName = "FirstName",
PropertyName = "Firstname"
});
domainModelEntity.Properties.Add(new DomainModelProperty()
{
ColumnName = "LastName",
PropertyName = "Lastname"
});
domainModelEntity.Properties.Add(new DomainModelProperty()
{
ColumnName = "BirthDate",
PropertyName = "DateOfBirth"
});
DomainModelDefinition domainModelDefinition = new DomainModelDefinition()
{
Entities = new List<DomainModelEntity>(),
Comments = "Version 2"
};
domainModelDefinition.Entities = new List<DomainModelEntity>();
domainModelDefinition.Entities.Add(domainModelEntity);
</pre>
<br />
Again the metadata can be queried from the instance of the service that exposes this version of the Domain Model. For the second version the Url is “http://localhost:8081/V2/DomainModelService/$metdata”.<br />
<br />
<a href="http://lh4.ggpht.com/-eZMy41rfTok/T6O92wRFiJI/AAAAAAAAAI0/vXJYPUa0D7s/s1600-h/version2metadata%25255B3%25255D.png"><img alt="version2metadata" border="0" height="323" src="http://lh6.ggpht.com/-tO7UwZewJfI/T6O93dAtDmI/AAAAAAAAAI8/g0kaAI9lsT0/version2metadata_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="version2metadata" width="644" /></a><br />
<br />
Again the ODataExplorer tool can be used to query data. <br />
<br />
<a href="http://lh6.ggpht.com/-SOz3AYLNq-I/T6O_MX999uI/AAAAAAAAAJE/Z3S1p-HGek4/s1600-h/OdataV2Data%25255B3%25255D.png"><img alt="OdataV2Data" border="0" height="166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieJjIOZmxeENGv5TkR5lx_Prjnjj7mrVr2zfX7SDwqBveqjKquvGZ8yYZM5eJanq5dAxhNayO6c4yqgRkfguwjWy5cfNWrqs7da2H4Nskl4WzEudSqzUQkxzWQtTHbBSNS1IczoQZhgcap/?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="OdataV2Data" width="644" /></a><br />
<br />
We can also update the entities to change the value of the underlying data in the database.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUS1akn3X09Ba11yI00e2RuJDn7k5YGb4JCdN0VVc_lPpFKyMFF5cX-0U9vyop44t9kryUopjTMlbCsB2oTPGefN7a4HhrusQEU8FVoSqg7YYGAUkNnoywwh70hYM1U71_eIxCTThCVWWH/s1600-h/OdataV2DataChange%25255B3%25255D.png"><img alt="OdataV2DataChange" border="0" height="484" src="http://lh4.ggpht.com/-sGF4TRWSP0Q/T6O_NjO25zI/AAAAAAAAAJc/Kq8UrznQPsw/OdataV2DataChange_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="OdataV2DataChange" width="365" /></a><br />
<br />
<a href="http://lh3.ggpht.com/-7dpmN4cxrc4/T6O_N2OkhoI/AAAAAAAAAJk/1WDV2PsHpDE/s1600-h/RawDatachangedV2%25255B3%25255D.png"><img alt="RawDatachangedV2" border="0" height="64" src="http://lh6.ggpht.com/--qd4tHxMiCo/T6O_Oe6JtxI/AAAAAAAAAJs/SOcDrxXmQAU/RawDatachangedV2_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="RawDatachangedV2" width="644" /></a><br />
<br />
Multiple versions of Domain Models can be exposed using this service, they can contain multiple entities.<br />
<br />
<b>Further Improvements</b><br />
<br />
This version of the Domain Model service satisfies the initial requirements but there are a number of areas in which improvements and enhancements can be made. The following is a brief discussion of these enhancements:<br />
<ul>
<li>Entity Identity<br />
</li>
<ul>
<li>At the moment the implementation makes the assumption that all entities exposed from the Domain Model have an identifier of type of System.Guid. This is fine for the intended target schema that this component will expose, but it is not a truly flexible approach. A better approach would be to allow for the data type of the identity field to be specified when creating the Domain Model. The DynamicDataContext class would then use a different mechanism to generate and manipulate the identity field.</li>
</ul>
<br />
<li>Security<br />
</li>
<ul>
<li>The DefinitionService and the ManagementService exposed by the Domain Model Service are secured using message level security. The Domain Model Service instances that are instantiated and exposed are not secure. They are exposed as http endpoints. With some additional configuration they can be exposed as https and use any of the following methods of <a href="http://msdn.microsoft.com/en-us/library/dd728284.aspx" target="_blank">security</a>.</li>
</ul>
<br />
<li>Relationships<br />
</li>
<ul>
<li>The current implementation supports only exposing entities that do not have relationships to other entities defined in the domain model. It is possible to enhance the service to support relationships between entity types by extending the <a href="http://msdn.microsoft.com/query/dev10.query?appId=Dev10IDEF1&l=EN-US&k=k%28SYSTEM.DATA.SERVICES.PROVIDERS.IDATASERVICEMETADATAPROVIDER%29;k%28IDATASERVICEMETADATAPROVIDER%29;k%28TargetFrameworkMoniker-%22.NETFRAMEWORK%2cVERSION%3dV4.0%22%29;k%28DevLang-CSHARP%29&rd=true" target="_blank">IDataServiceMetadataProvider</a> and <a href="http://msdn.microsoft.com/query/dev10.query?appId=Dev10IDEF1&l=EN-US&k=k%28SYSTEM.DATA.SERVICES.PROVIDERS.IDATASERVICEUPDATEPROVIDER%29;k%28IDATASERVICEUPDATEPROVIDER%29;k%28TargetFrameworkMoniker-%22.NETFRAMEWORK%2cVERSION%3dV4.0%22%29;k%28DevLang-CSHARP%29&rd=true" target="_blank">IDataServiceUpdateProvider</a> derived classes.</li>
</ul>
</ul>
<br />
Hopefully this series of posts has given a good overview of the implementation of a configurable OData service and given a good example of how a Custom Data provider can be implemented to source data.Anonymoushttp://www.blogger.com/profile/16239602787361416300noreply@blogger.com4