DataWang examples (wiki page)

I've added a new wiki page to the DataWang google code project that gives examples on how to use DataWang to generate random data for your entities.

Hope this demonstrates how simple it is to use DataWang!

That's DataWang...(Google Sets API)

Inspired by Mitchell & Webbs "NumberWang" I have created "DataWang"....

"DataWang is a .Net library that can generate useful, realistic data for use in the development/mocks/testing process.

Based on Google Sets, this library provides a .Net API over the Set functionality (by making http calls and parsing the html response). Google Sets currently does not expose a programmatic API - DataWang provides you with programmatic access to this API!

DataWang is a completely extensible framework for mock data generation. It is fully interface driven and all components are dependency injectable. You can provide your own implementations for any of the component interfaces should you wish to extend or modify its behaviour.

Google Sets often cannot generate the data you require - so DataWang will manufacture a list based on the results of making multiple Google Set calls - email addresses is a good example (a first name call + "@" + a domain name call). DataWang exposes these named sets as "Providers" and comes preloaded with several including, firstname, surname, email address and fullname and these can be extended to provide any custom data you want to randomly generate.

The final feature of DataWang is that it can automatically create lists of your custom entities that have their properties automagically set to random data values. All that is required is to add a "DataWangType" attribute to any class property to mark the category of data it should hold (eg: emailaddress). DataWang will then create a list of your custom entites with these properties set to random, meaningful values!"

Add a reference to DataWang.Core and then enter on the static "DataWang"...

Happy DataWanging....

WCFServiceWrapper: v0.2 release

Update: 15th July 2008 - Project now hosted on Codeplex, links below updated. v0.3 is underway, check out the issues list on Codeplex; these will all be fixed in this next release.

After creating WCFServiceWrapper I've been using it a fair bit to whip up wrappers to several existing business layer interfaces...

(Recap: WCFServiceWrapper is a .Net application to generate the service layer code required to expose any existing non-WCF interface via WCF).

I'm releasing v0.2 which includes...

* New WinForms UI
* Support for generating VB code. Console app cmdline switch /L:VB
* Bugfix: Primitive datatypes and void return types from interface methods where not being handled correctly.
* I have also refreshed the WCFServiceExample project to include this new version in the example zip.

Version 0.2 application source code and example project can be downloaded from the new Codeplex project here (The example is a VS2008 solution providing a simple business assembly and an empty WCF service project plus pre-built WCFServiceWrapper applications so you don't need to get the application source code and build it)


Here is a screenshot of the new Winform UI...


Once you have generated your WCF service code you simply include it in your project...


You can test this in Visual Studio 2008 very easily by pressing F5/running your WCF project to popup the builtin WCF Test Client

ToDo-ist

I've long been a fan (and sometime practitioner) of "Getting Things Done" (GTD) organisational approach. At the heart of being "organised" is having a good "todo" list...and for the last few months I've been using ToDo-ist to power my list.

I'm also a recent convert to "the cloud" - using online applications and services...the availability and reach makes "cloud" based stuff just so useful....and Todo-ist also neatly ties these two things together...and the real plus is that ToDo-ist also has an iGoogle gadget/widget plugin.

Usually I find a shiny new app/service/tool etc and post about it immediately only to find that it doesn't live up the hype or expectation - so this "recommendation" for ToDo-ist is based on several months of actually using it!

I really like the level of flexibility ToDo-ist offers (it terms of list structures) and has some nice shortcuts too (like putting "tod" or "tom" in the date due box to represent todays/tomorrows date). The iGoogle widget really seals it for me though. Anyway...if you are looking for an online ToDo list then I do recommend ToDo-ist.

(Note: I would also recommend, as with any cloud based service/application that you exercise some caution with the information/data you store in/on such services).

Ninject...

...hallelujah!


What is Ninject?! (taken from here)

"In 5 seconds

Ninject is the ninja of dependency injectors.

In 30 Seconds

Ninject is a lightning-fast, ultra-lightweight dependency injector for .NET applications. It helps you split your application into a collection of loosely-coupled, highly-cohesive pieces, and then glue them back together in a flexible manner. By using Ninject to support your software’s architecture, your code will become easier to write, reuse, test, and modify. "

One of my main bug bears with Spring.NET is the feature bloat (I really don't need 70% of the features) and the xml configuration hell (and the same can be said of many IoC/DI frameworks).

I'm using Castle/Windsor at the moment and even with this lighter weight framework the configuration bogs you down. I have my own solution using "Assemblers" to construct preset object tree implementations of interfaces - the "assemblers" themselves are created using IoC so I can swap out stuff still just not at a fine grained level. However brainer people than I have been attacking this problem and I've run across Ninject and Binsor, plus I've also given myself an idea for a tool - a UI to configure any IoC framework (using schema plugins)...

Anyway back to Ninject....sounds great, can't wait to try it out!

Wrap your existing business layer with WCF

[Update: 21-Jun-08, v0.2 released...more info here]

I've recently had some hands on exposure with WCF. My initial impressions are that its pretty slick. However I was pretty perturbed to learn how invasive WCF is - specifically the "DataContract" attributes required to control the entity serialisation.

When designing software I start in the middle - usually a vanilla interface that represents some activity - this activity is UI and Data layer neutral...its upstream clients could be a UI or equally a service, likewise I make no assumptions about the persistance (if any) that might be required. So I was pretty unimpressed to learn that to expose an activity via WCF I needed to retrofit some WCF specific attributes to my interfaces and data entities in order for them to be WCF friendly. This does not sit well with me. Suddenly a context neutral component needs to know that it *might* be part of a WCF service and therefore has to be attributed just in case it is....and what about my existing entities and interfaces - I've got to go and revisit these too? Arrrrghhh!

I had a scout around - WCF and its forerunner Indigo are not exactly shiny shiny new features these days...someone must have already fixed this or come up with an architecture that solves this...I got pretty close with Web Service Software Factory from the Patterns & Practice team and although I'm keen to look into this further it looked a fairly serious heavyweight bit of kit, maybe too heavyweight. There was some talk of customised serialisation and not using the DataContract attributes at all but I wasn't keen on this either - it was sidestepping all the WCF goodness it seemed. A forum posting mentioned this problem and offered an approach - mirror your business interfaces and entities with WCF versions and convert the entities between the WCF and business layers. I liked this as it meant my business layer remained neutral and "unpolluted" but I wasn't keen on the waste of a mapping call for each entity in the interface.

However I took the bait and decided that creating the WCF versions of interfaces and entities and the conversion to and from the business layer could be automated. Reflect the business interface to WCF enable, discover the interface methods and entities then blast them out as WCF versions using CodeDom....which I did...and the WCFServiceWrapper application was born!

What is WCFServiceWrapper?
WCFServiceWrapper is a .Net console application written in C# 2.0, VS2005. It can take any interface, extract the methods and entities and then generate the WCF versions including conversion code from the orginal entities and a default service implementation. Essentially it allows for a "vanilla" interface and entities to be exposed via WCF without having to write any code to do it.

Once the application has been run against the interface you wish to expose via WCF all that is required is to include the source code items generated in your project and configure the app.config and you are up and running. It is even possible to generate Visual Basic WCF source code (although I have not exposed this on the application; you would need to rebuild the application from the source code to enable VB output. In the next build I'll add a command line switch for VB - promise!).

The console application is built on a core assembly that does all the heavy lifting including...

• Discovery of the interface methods and entities
• Generation of the WCF code items. Includes rudimentary collection and generics support
• Saving of the WCF code items

This layout has been specifically chosen to enable other "clients"; for instance an add-in to Visual Studio could utilise this assembly.

The source code is here (v0.2 on project distributor)


Using WCFServiceWrapper
To use the application you need to have a built version of the assembly that contains the interface and entities. The interface and entities must be in the same assembly.

Use the command line switches
/A:[assembly filename including path]
/T:[interface name] - the interface name does not need to include the namespace.

To contol the namespaces...
/NI:[wcf interface namespace]
/NE:[wcf entity namespace]

To control the output folders
Either
/O:[output folder for both interface and entities]
Or
/OIF:[output interface folder]
/OEF:[output entity folder]

The latter allows separate folders for interface and entities.

Finally...

/OF+ overwrites existing files (default)
/OF- will not overwrite existing files (error if file already exists)

Once the WCF code has been generated you simply include it in your WCF project.


Example project
I have included an example solution to demonstrate WCFServiceWrapper in operation. This example solution contains a "Business Layer" project I want to expose via WCF. A blank WCF project is also included to host the WCF code generated items.

Note: This example solution is written using VS2008. You will need to compile the solution before and after running the WCFServiceWrapper application.

The example can be found here (updated for v0.2 release on project distributor)

1. Unzip the WCFWrapperExample
2. Open and compile the solution
3. Goto the WCFWrapper folder and run CreateWrapper.bat. This will create the WCF code items in the WCFWrapperExample\MyServiceLayer folder - you should have two items (interface and service implementation) in this folder and and new folder called Entities. Remember to "Show all files" on the solution treeview to see these new items and include them in the project.
4. Next is to open MyServiceLayer\CopyAndPaste.txt and follow the instructions. Then press F5 - the WCFTestClient application should open and allow you to test the OrderWidget method.

Final thoughts
I wrote this application to save time because I have lots of existing interfaces I want to expose via WCF and I didn't want to manually enable this. I have tested the app on the types of interfaces I commonly use - please be aware it has not been extensively tested!

The design also accomodates extension - you can write your own implementations for type discovery, code generation and publishing by following the code and interfaces provided in the TypeWrapper.Core assembly.

Although I have released this application I cannot guarantee any kind of support - I've made the source code available so if it doesn't cope with a specific problem then you can have a go yourself at resolving the issue. As I encounter problems I will fix them and post updates - if you have a problem then let me know and I can certainly take a look but I'm not promising anything! If this is popular at all then it might be worth moving to something like CodePlex to actually keep the source moving via collaboration.

The bottom line is that I have a day job and this is just a little app I created to make coding easier - I hope it can do the same for you!

Castle/Windsor schema enables Visual Studio intellisense

There has been a lot of noise recently about Inversion of Control (IoC) with .Net recently (stop sniggering at the back java guys!)....

I've been using IoC via the Spring.NET framework for over 2 years now - it's a completely different approach to coding and once you get your head around it everything just falls into place and development is a real joy again.

As I mention, Spring.NET is my framework of choice but a recent change in employer has seen me bump up against Castle/Windsor. First impressions are that I like it - it's not as powerful or feature rich as Spring but that's not always a bad thing! The one thing I did miss though was Visual Studio intellisense when editing the configurations - Spring has an online schema that can be associated with a Spring configuration. This got me thinking - if the VS intellisense can be hooked into that easily why not create one for Windsor configuration?

So I did...you can download it from my new google code site here. Remember to "Readme.txt" once you unzip for instructions!

Enjoy!

Monitoring (Part 1): Whatchoolookinat?

Continuing my "Business Agility" series (I promise to explain what I mean by this!) I wanted to talk about another area of software that is often over looked and lacking information...application performance monitoring.

To start with why is performance monitoring important and what has it got to do with "business agility"?

As with my previous post on debugging it is the lack of information that directly impacts the ability of the business to improve its software. The lack of performance information is a blind spot as to what is actually happening operationally with your software. Ignorance of a problem doesn’t always equal bliss! In my experience much business software is written without regard for performance. However this is not the sin - the sin in my eyes is that the ability to measure performance is often not thought about at all and if performance does become an issue then it is often too late to put in place a good, consistent monitoring framework. Performance Monitoring should be part of your day 1 design goals!

I also want to define a few things at this point before we really get into it.

1 - I design software from many perspectives ahead of performance - chiefly extensibility (agility) and maintainability; performance is not my primary concern (usually in business software however your software my depend upon its performance alone). By this I mean that I prefer to write cleaner, well constructed and understandable code or components rather than ones that may be more performant but less easy to follow, extend, debug or fix. If an elegant design comes up to you and smacks you across the face with a "I will run very slowly" message then take heed obviously but in my experience you can often use the cleaner design to introduce performance enhancing techniques like caching into the software to boost performance way more than a "a fancy, complicated algorithm that no one can debug".

2 - "Performance monitoring", in the sense I am interested in is "operational" performance. There are many tools that exist to analyse the performance of your software. These are primarily developer orientated, technical tools that can instrument your code in situ on a development PC. They provide incredibly detailed outputs around method and even statement level code. Whilst these are valuable, and they certainly are if your code has performance at its heart I do not think they paint the "operational" picture well; eg: how does your application perform once deployed on an operational platform? Many performance analysis tools are run during the development phase (and usually at the end when it is realised that your application *does* have a performance issue, think horse bolted scenario) and really only provide "relative" performance information - absolute timings will only be available once deployed to the production environment due to the disparity in desktop and server hardware architecture and OS resources and configuration. So what I am specifically talking about is extending these "relative" timings to also include "absolute" timings from an operational environment.

3 - The performance monitoring information itself, at its simplist level is nothing more than a timing against an operation. This is all you essentially need to identify performance problems - however you will definitely benefit from correlating parameter and variable values to the performance figures - eg: creating a "context" for your performance data. It might be a specific combination of values that trigger a spike in performance of an operation - being able to tie the two together really accelerates problem identification and resolution. I coined (I hope) the term "Time To Fix" (TTF) in my last post about debugging and this fits the same pattern. Performance information can also be recorded as part of your debugging trace - this will give you the vital correlation mentioned. For impact and sheer wow factor you can also use the Windows Performance Monitor system to visualise your applications performance - this works very well for operational monitoring.

The most important aspect of performance monitoring is that it should be considered fundamental to helping you diagnose performance problems and improve the quality of your software in general. As with providing better debugging information the more (performance) information you are armed with the better equipped you are to identify, remedy and improve things. I want to convince you that performance monitoring should be baked into your applications from day one and hope to demonstrate that through good component design (and I've even provided the code!) that the overhead is minimal and the reward worthwhile.


Before you the reader go any further and if you have not already acquainted yourself with the specifics of the languages/platform this post is technically targeted at (C#/Microsoft platforms) you might like to read this "primer post" on the background to this post and approach to software design.

Right, welcome back...

Ok, so what are talking about at a code level?

A component interface that allows us to invoke any implementation of a "performance monitor". Using dependency injection we can even dynamically configure our monitoring components to suit operational conditions and requirements. Imagine being able to gather performance information triggered by a specific time period, executing user or any other environmental property!

The ability to provide "flavours" of performance monitor based on scenario. Whilst I think that performance information should be available to all applications there are clearly a difference in requirements from your core enterprise service stacks to other smaller, possibly desktop orientated applications. This gives us different implementations to satisfy the "horses for courses" maxim.

It should be simple to use with a low footprint in the executing code being monitored.

It needs to be flexible enough to monitor any of your application code; eg: it should not be tied to method level only.

Show me the code...
I have loaded the code I will talk about onto my ProjectDistributor.Net site - it may be worth your while downloading this now so you can trace, at a code level, what I will be talking about next. Essentially the code provides....

The performance monitoring interface

A winform application that demonstrates the use of the performance monitoring interface and also provides custom PerfMon counter install/uninstall

One performance monitor implementation to provide a basic "stopwatch" to provide duration and average duration (when used in a iterative scenario like a loop).

One "rich" implementation that provides the ability to visualise your application performance by hooking into Windows PerfMon.

A set of PerfMon orientated utilities to help take the pain out of PerfMon integration.

An ASP.Net HttpModule to hook up your web apps to your custom PerfMon counters.

A simple webservice to demonstrate the Windows PerfMon monitor implementation.


Installing & Running the code
1. Unzip the code into a folder. You should have...



(Ignore the Resharper folders)

The solution is in the BuiltUtilities folder.

Common - this will house all the "common" BuildUtilities libraries
Common/BuildUtilities.Common.Monitoring - this has the monitoring class library. This has all the "common" code we require for monitoring including the performance monitor interface, interface implementations, perfmon hook http module and perfmon helpers.
Monitoring/WebService - The dummy web service that demonstrates the two implementations. This used the standard ASP.NET development webserver. The service is fixed to run on port 81.

Monitoring/WinForm - The main client, this calls the web service and additionally installs/uninstalls your custom perfmon counters.


2. Load up the solution "BuildUtilities.Monitoring.sln" found in the BuiltUtilities folder.

3. Make sure the winform project "TestPerfMonComponentMonitor" is your start up project. Press play/F5. You should have...



4. Install PerfMon counters. Simple - just click the "Install Counters" button!


5. Locate the webserver (otherwise your webservice will fail). You will need to find the webserver executable. It is located in the .Net framework install folder. If you click the "..." button you should be taken to the c:\windows\microsoft.net\framework folder...choose your framework version (minimum v2) folder and select WebDev.WebServer.EXE.

6. Locate webservice code. Click the "..." button and browse to the BuildUtilities\Monitoring\WebService\TestPerfMonService folder.

7. Start Webserver - just click "Start WebServer" - you should get a system tray popup...



8. Install & run DebugView from SysInternals/Technet - this will let you see the web service debug trace to prove the service is being called.

9. Click the "Single Dummy WebService Call" button. You should see a couple of lines of debug appear in DebugView

10. Click "Start PerfMon" - it should launch the Windows Performance Monitor. Click the big plus button on the toolbar to add a new counter. Select "BuildUtilities.Net Perfmon" from the "Performance object" dropdown. You should have a instance in the right hand side instance list.



11. Back to the main winform app...this is the big one....click the "Burst Dummy WebService Call" button and swap to PerfMon - you should see something like this (image too large to display inline).

Summary - end of part 1
As you can see we have a fully instrumented webservice application. Using PerfMon I can visually see what is happening with the webservice performance and I can refer to the DebugView capture to inspect the trace for the service parameter values - as you can imagine it would be pretty quick for you to hunt down, replicate, investigate and resolve any performance issue you might encounter armed with all this information.


It's taken a few late nights to get the code and this post into shape - in doing so I realised that it would be better to split it into multiple posts rather than one monster post. There is value to be had from this post on its own so it is better to get it published so that you can benefit (or not!) from the example right now.


I will be following up this post with a second part and in it I will try to talk about some of the oddities of Windows PerfMon and also more importantly how you can use these components in your own software.


My parting shot is this. How you do the monitoring is not that important; I am merely demonstrating the way I do it. The important thing here is that performance monitoring, like debug tracing is something that has to baked into the application from day 1 - and the reason to do this is to improve the quality of your software and reduce the time to fix issues once it is operational.

Home Automation: HomeEasy + xPLMonkey

Update 4th March '08: Mal Lansell (the xPL guru) commented!...check out the update to the xPL site with a dedicated HomeEasy page - excellent stuff Mal!

I've been involved with Home Automation to some degree over the last few years. I've dabbled with zWave and installed a small home network to some success. zWave is a good technology - its mesh networking capability and bi-directional signalling is superb but the chronic lack of devices and support in the UK has really crippled its success and with it my enthusiasm for it.

However, a new DIY project (fitting a new kitchen) meant an opportunity for more automation and by pure coincidence I happened upon the HomeEasy range of automation products that B&Q stock. They had exactly the things I needed to automate the kitchen under cupboard lights and kickboard L.E.Ds - a wireless PIR sensor and sockets and all at a very good price, the starter kit (3 sockets + remote) was £20 and the PIR about £11. Now when I walk into the kitchen from dusk to dawn the lights automatically switch on and switch themselves off 1 minute after leaving the room (this can be set on the PIR to 5s, 60s, 5m, 10m)...perfect!

I can recommend the HomeEasy range if you want a simple, cheap automation system that is easy to setup. I've had the odd situation where one of the sockets does appear to not receive the power off command from the PIR but its pretty rare - possibly related to the battery power in the PIR. There are plenty of fancy switches, remotes and even an outdoor specific range. The "ultimate" range even boasts SMS integration with a special base station that can convert SMS messages sent from your mobile to commands beamed around your house!

What is missing is PC integration, however - this is where xPLMonkey appears to fit in. I say appears as I've not been tempted to try this myself but the xPL site mentions support for the HomeEasy range. Using USB tranmitters (transceivers?) connected to a PC you can control HomeEasy receivers - eg: send commands to switch them on/off/dim. xPLMonkey acts as an abstraction layer to convert scripting commands into wireless automation commands. There is even support for MediaCenter!

So if you want to start your wireless home automation somewhere then HomeEasy seems like a decent and cheap enough place to start - xPLMonkey can sate in the inner geek in you if you want to take things further and combine automation with your PC - this obviously opens up more interaction possibilities like web control.

Another range that offers almost identical automation products is DomiaLite - very similar to HomeEasy (or rather HomeEasy is similar to DomiaLite as I think this was around first). Domia offers PC/Web integration with a dedicated controller unit but I think there is a subscription fee involved. Worth a look though....

If you do get HomeEasy working with xPLMonkey then drop me a line - I'm curious to hear about your experience with it!

Sysprepping a virtual PC

This is an aide-memoire for Sysprepping a Virtual PC. We use VPC/VServer extensively in our software development and testing and our regular "Sysprepper" Patrick fowarded me this excellent link in case we needed to do this ourselves...

http://blogs.technet.com/megand/articles/357570.aspx

Debugging: Through the looking glass

Update 10-Jan-2008: I have added a link to a full trace log to one of our service calls to demonstrate the type and quality of information you should be logging...


One of the biggest areas for improvement in software development is debugging - or put another way, reducing the time to understand and fix bugs - lets call this Time To Fix (TTF).

Reducing TTF has some benefits directly related to "Business Agility"- more bugs can be tackled and fixed in a given period or a set threshold (severity) of bugs can be fixed in a shorter time. Rather nicely the means to improve TTF actually reduces the number of bugs being produced over time - we'll cover how this works later on.

Objective
Drawing on my "Business Agility through Component Software" concept and from hands on development these past ~three years I have experienced first hand how inefficient bug fixing can be and how enormous time savings can be made with a better approach to designing and implementing your software. I want to demonstrate practical changes you could make to your software in order to increase your software quality and reduce your TTF.

"Through the looking glass"
Read this in order to set the scene on the type of software and environment that has spawned this article.

One of the biggest challenges faced by developers is when their software is deployed and testing starts. Once the software is installed outside of a development environment it is on an "operational" footing - you rarely have the luxury of any IDE or serious debugging tools as they are considered a risk to the platform or environment. Take away the IDE and dev tools and you are left with an enormous hulk of code that often does not provide a clue why it is broken and offers a tiny spyhole into why it has broken.

"Software not only has a responsibility to work but also to provide information on why it doesn't when it fails."

Marvellous idea eh? It's nothing new obviously - developers have been writing exceptions and debugging information to logfiles, event log etc for a long time and many libraries exist to help facilitate this (Enterprise Library, Log4Net).

However my experience is that the quality of information logged is usually very poor, generally very cryptic and almost no help in finding the bug. Why would this be the case though? Well lets start with the notion that the code even has bugs - preposterous! So why should I worry about the quality of information as exceptions just don't happen! Widget.cs, Line 450 - "NullReferenceException - Object reference not set to an instance of an object" is just not good enough to actually diagnose and solve a bug is it really?

The next issue is that often a condition or code path goes wrong long before an exception is actually thrown. Providing a stack trace gives you a pretty clear idea of where the exception occurred but many is the time I've seen heads scratched and the heard the words "...it shouldn't even be in this code..." uttered.

Developers when writing software generally don't think from an operational perspective - I think this is a key reason for the poor information available when trying to fix a bug. Information about what the code is actually doing is vital to quickly diagnosing problems. By this I mean providing a "trace" through the code as it executes. Each trace event should contain information such as ....

• Instance executing (vital in web environments) - usually a guid. This will help isolate a trace when stored in a common format or repository
• Location information (service name, assembly, class)
• Point in time (best recorded as UTC for global applications)
• Message/description of event

Here is a sample of one of our event traces (its been sanitised) to demonstrate the type and quantity of information that I am talking about. This trace is from a single call to one of our service stacks (Windows media licence request). The important thing to notice is that it records variable (data) values - its often the value of a specific variable that alters the course of the code so it is important to know what these values are especially if you hook logic to them. Also notice that the trace contains performance information about the duration of some of the operations - this is the subject of my next post.

Clearly this is can get very very verbose. Plus, the code will be littered with "trace" statements. There are Aspect Orientated Programming (AOP) techniques to transparently generate "Entering Method X" and "Leaving Method X" trace statements however knowing where you have been often doesn't tell you why - AOP tracing is not the answer by itself as it solves one issue (readability) but offers no more real information than a stack trace.

"Fundamental to tracing is recording the logic flow within a method"

My answer to tracing statements "cluttering" the code is simply that it is for the greater good.

What about all those trace statements, the amount of information output and impact on performance?

One thing I will say about the tracing/logging frameworks I have ever encountered - they are often too complicated especially when it comes to "levels" (info, warning, error, critical etc). This plethora of levels leads to many problems...

• Confusion as to the definition of each level
• Coding pace slows as developers struggle with applying these levels
• Inconsistent application of levels to events (developers get sloppy and cut and paste statements and forget to adjust the level - honest, it does happen!)

So to answer the performance question...my experience is that debugging/auditing can usually be simplified such...

• There are only two audit levels - "debug" and "business". A debug event is part of the verbose trace used to diagnose a bug and a business event is usually something of significance that must be recorded - for instance, record a modification to value.
• The debug events are only enabled when required to investigate a problem. Performance is only impacted for the minimum period necessary.
• The quantity of information is not a problem if you can isolate the information you need to solve the problem.
• Use "rolling" event sinks such as the event log debugview or rolling date/size limited files

So it all seems very clear - providing more information and of a better quality will to help diagnose and pinpoint the bug. This will clearly help reduce our TTF.

The developer can also directly help by explicitly seeking out exception conditions before they blow up in an "uncontrolled" manner - eg: your typical "NullReferenceException". I am talking about pre and post conditional code checks. All code be it an assembly, class or method operates within a set of limits - these may be datatype limits or conceptual limits (eg: a uk traffic light has three colours, you cannot set it any colour other than red, amber or green). Without "spoiling" my future post on Behaviour and Interface driven design these pre & post conditional checks should check the inputs, outputs and assumptions a developer may make before, during and after the execution of a specific piece of code. The typical objective is to "fail hard and fail fast" but I think it should be...

"Fail hard, fail fast and fail informatively"

In code terms this means checking the validity of each input parameter and where data is returned doing the same. When an exception is encountered throw the correct type of exception (OutOfRange, Format, ArgumentNull) to clearly show why it has failed and if possible a much more informative message however care must be taken depending upon how public the audience this message may have - you may be giving away too much information to a hacker. Remember though the point of exception is often not the culprit - a null parameter is not the fault of the method that receives it however tighening up here does bring some nice side-effects.

Consistency in how you organise and layout your code is not critical but certainly has a minor role in TTF. Pre and post checking organises the layout of a method by decluttering it. Pre checks are performed - the meat of the method can safely assume all is well with the world and hand off the result once complete to the post checks to ensure it has done what it should. The bonus in .Net is using #regions to hide these checks so clarifying the code even further. Another positive side effect - you can use your nice clear method comments to help you write the checks or vice-versa...once your code is complete you can comment the exact behaviour of the method. The former is my preferred course of coding and again the subject for another day.

What next?

What are the practical recommendations I can give you to help improve your debugging and TTF?

I believe the magic ingredient is to make the diagnostic output of your application part of your peer review and certainly part of your acceptance or QA tests. I cannot stress this enough - like any part of a system you rely on you need to test it and test it again to improve the quality. There is absolutely no point in putting ANY debugging/tracing events into your code if they have no value in diagnosing a problem. To ensure they do this job you should use your unit tests to test drive the trace output. I even suggest that you have unit tests dedicated to this - they should deliberately provoke a problem in a component in order to provide a set of debug events. There is no "programatic" way of inspecting and passing or failing these, its simply a means to provide a trace earlier in the development cycle and get it inspected for suitability. It is actually very interesting reading a trace back - I have spotted several bugs as the trace didn't make sense.

Another tip is to formalise the importance of debugging and tracing within the code. Generally there needs to be a contiguous trace throughout the software stack for reasons stated already. For a component to be part of the trace, create an interface called "ISupportsAuditing" - this interface has a single get/set property called "Auditor" - this provides a handle to an instance of the component that actually does the tracing/auditing. This auditor/tracer component can be created externally (in a web page, web service) and used to set the "Auditor" property (alternatively you can set it in the constructor). You can now easily place calls to the auditor/tracer within your component. Using an interface for the auditing component is a wise move too as it allows a soft-wiring of your code to the auditing/tracing component.

To summarise...

• Diagnosing and resolving bugs within operational software requires good quality information. This allows you to find the bug quicker and fix it - TTF is reduced and software quality is improved.
• Simplify the trace/audit levels to avoid confusion and inconsistency in application. You can do this by creating your own auditing/tracing sub system to simplify and wrap standard libraries like Enterprise Library or Log4Net.
• Formalise the auditing/tracing within the code by explicitly implementing an "I Support Auditing/Tracing" interface on complex or mission critical components.
• Road test the trace as soon as possible to ensure it does the job it is intended for. If it doesn't then it is pointless! Use unit tests to provide trace output for inspection and review these as part of peer/QA/code review and acceptance. Could you zero in on a bug (someone elses) from the trace alone? Could you offer 3 ideas for where/why the bug occured prompted from the trace alone?
• Ensure the information in the trace allows isolation and location of information - if there is a lot of it you will need to be able to find the particular call from potentially hundreds!
• I strongly recommend you use the System Windows Debug (Diagnostics.Trace) to capture output - then use DebugView to capture this to file. It has several options for file roll over which makes it suitable for long term capture (say in a soak test or staging environment).

Finally - my experience is that the operational aspect of software is often overlooked. This mistake is made again and again and the time it takes to fix a "bug" is never questioned. I think that this "time to fix" (TTF) can be reduced if you ensure that you provide the information you need to do this. To ensure that this information is of sufficient quality it needs to be inspected and tested as part of your software QA/testing process. Either way you are a winner by reducing your TTF - you either fix N bugs quicker so save on cost here or you can fix more bugs in the same amount of time thus improving the quality of your software. In both cases inspecting the debug trace often reveals subtle bugs as the trace just doesn't read "right".

Think about it - are you spending too long fixing your bugs?

Act 1; Scene 1

As previously mentioned I want to write about my "Business Agility through Component Software" concept. This encompasses several aspects of software design and crucially includes operational considerations.

I'll be writing a series of posts covering all of this...however in true "reuse" style I want to provide a common "set the scene" post to explain the work I have been doing and give context and background to the actual post that will reference it.

I work for a Digital Media "supply chain" company. We ingest (take delivery) of media files (WMA,WMV, metadata & images) and perform a number of processing steps on them before eventual delivery to a consumer via a digital retail front end (website, WMP Online Store, gadget etc). This past twelve months I've been involved in re-developing the delivery services on our core retailer platform.

These services are used to...

• Deliver a media file (WMA, WMV) to a PC/Client
• Generate Windows Media Licences (rental, perpetual, subscription/chained)
• Manipulate (resize, crop, watermark etc) and deliver product images
• Subscription management (create, authorise, etc)
• Licence management (counts, reset etc)
• Product delivery manifest (download, licence, image urls, description etc)

This is all based on Microsoft .NET 2.0 using C# with a smattering of C++ for custom COM components running on a client PC. My expertise is in the .Net code however I'm learning the COM stuff as it seems to crop up a lot in media development and a worthy area of knowledge. These services are all accessed via ASP.Net endpoints as either Web Services or custom Http Handlers.


All of these services are very much typical "service icebergs"...the endpoint interface is 1/9th of the implementation. Typically a service (http handler) has three or four parameters or a web service a simple object parameter with a handful of properties. Data returned is similarly pretty slim too.


The service endpoints are deployed per retailer - that is each retailer has a dedicated copy of the services; some of the services are redundant depending upon the retailer and their business model.


Overhaul...
Luckily a portion of the services were re-writes and I could learn from the mistakes made in the previous incarnation. These improvements were also incorporated into the approach/pattern/template for any new services created.


Ideas and objectives I wanted to incorporate included,

• Decoupling service endpoints from service implementation
• Service implementation must use dependency injection (DI) and Inversion of Control (IoC)
• Pluggable component architecture to allow customisation for retailer specific processing (using DI/IoC)
• Services must be able to provide a verbose, readable audit trail
• Use interface/behaviour driven design for components
• Understand what makes a useful design document, what level and type of information is actually useful committing to a document?
• Remove focus from data/database during design
• Utilise mock objects to provide known behaviour and improve development pace

The result of all this was about 50,000 lines of pure code. Most if not all of the objectives were achieved - most with a successful result.

The following posts/articles I will write will cover these topic areas - some at a conceptual level, some at a technical level or combination. My objective with these articles is to evangalise about these ideas and objectives and for you to embrace them in your projects to increase business agility and software quality.

I hope you stick with it!

Service.Resume();

Hello everyone...er...anyone?!

I'm dusting off the blog at last - and shocked to see my the last time I published was 6th Dec 2006.....oh gosh....oppps!

I've been very busy working on rewriting the entire set of services for our Media delivery and licensing platform and integrating them with several large projects. I've finally surfaced from this and want to start sharing my knowledge and experience of the past year working on things such as....

• Windows Media licensing (subscriptions, full licenses, revocation, generation)
• Type 1 Windows Media Player Online (Active) Stores
• ASP.NET as a service platform (http modules)
• Software design (Inversion of control, Dependency Injection, Spring.Net, Behaviour Driven Design)
• Business agility through software component design
• The future of Digital Media and general thoughts on where its all going and where technology fits in with it.

I've started the ball rolling by updating my most popular blog post (.Net custom configuration) to replace the missing images with new inline examples. In recreating the example I have also got the C# project zipped up and this is available for download from my Project Distributor group.

If you're an old subscriber to this blog then hopefully I can reward your patience and if you are new to it then welcome - I hope I'll provide some real value for you!

Cheers,

James

PhraseExpress - free download

Just seen this productivity util from the Maxivista guys.

PhraseExpress - http://www.phraseexpress.com/

Its a phrase/expression expansion tool that works in any application (not just msoffice) to expand any phrase/term into a full string...

eg: you type bbfn and it is automatically expanded into bye bye for now

Plus - its free for the personal use edition!

GAUNTLET OF PRODUCTIVITY!

Hello!....been a bit quiet on the blogging front...Windows Media DRM, Type 1 Active Stores and the like are keeping me very busy but this made me laugh...if you need a productivity boost this might be the gadget you need!

SendMSN - binaries build

Since I released the source code for SendMSN, a utility that lets you send a message to multiple MSN Messenger clients from the command line onto my ProjectDistributor site I've had quite a few requests for just the built version (exe).

So I've finally got round to putting a binaries only build up onto ProjectDistributor.

I originally wrote this for use in build scripts for instant notification of a failed build but there are many more uses I've heard it being put to - many applications offer a "Run External Command" feature and by using SendMSN they now have an Instant Messenger capability.

One such application is mControl - Windows Media Centre 2005 Home Automation application that I've had on trial recently and been really impressed with. It has a rich range of trigger and action functions and SendMSN would fit right to give you instant MSN alerts about security or any other aspect of your home automation installation.

As ever, any problems with the application, installation, configuration then drop me a comment and I'll attempt to fix the issue.

What does Vista MCE ("Diamond") look like?

Just watched this on Charlie Owens blog.

This is a tour of Windows Vista Media Centre (code name Diamond). I have to say that I'm mildly impressed but I believe that with "consumer" grade products like this (and by consumer grade I mean ready for Joe Public - no computer science degree required to operate) the proof is in the long term experience with the product; i.e does it become indispensible or did you get frustrated with it because of stability/reliability issues? "Eye-candy" stuff is great but if it forgets to record a TV program it will get the boot! I've been pleased with MCE2005 so working on the assumption this will be a "better" product I hope this will become a truely indispensible integral part of my home.

Great video though demonstrating the features and improvements and I can't wait to get some long term experience with it myself. I'm holding off buying a new media centre PC until "Diamond" is released.

Resharper 2.0.1 point release

Get it here

Hopefully this release addresses some of the performance issues numerous people have reported. The release notes which include a complete breakdown of bugs fixed & features is here.

Enjoy...I know I will :)

Orb 2.0 beta released

Find out more here...

It's been AJAX'ed up and is looking really nice...

"Orb 2.0 is all about MyCasting - Be your own producer:Create channels out of your music, photos, videos, and television."

What's Orb? It allows you to stream your recorded and live TV, music, pictures, videos from your PC to any other IP connected device (PDA, Smartphone, Laptop, another PC)...how cool is that? Very very very cool that's how cool!

Check out these posts on it if you've never heard of Orb....

My Orb blog post
Scotts Orb blog post

Riding the Z-Wave...

So finally I've got round to buying some new gadgets...and this time its the long awaited attempt at home automation using Z-Wave devices.

Why Z-Wave and not X10?
Tricky choice...Z-Wave uses RF and a mesh network approach with command confirmation - this is the primary reason for going for Z-Wave - rock solid control. Each Z-Wave module is not only a receiver but a repeater too; this means that the controller sends the command to any module in range and it passes it on to any other modules nearby until the command eventually reaches the intended module and vice-versa for the command confirmation response. By "in range" and "nearby" I mean about 30 metres (100ft) The more modules you have the more resilient and robust the command messaging becomes and the messaging is very quick too.

I'd read a lot about X10 and was really impressed by its support for almost anything you want to do; Z-Wave in the UK is poorly supported in terms of variety of modules, currently there is only three types of module available; Appliance module (simple on/off), Lamp module (on/off/dimmer) and a PIR motion detector. X10 also has proper wall light switches the main thing you need to control your lighting - this is a goal of my home automation project so I'm a bit disappointed that Z-Wave hasn't got these available....the good news is they are on the way...allegely! So I ruled out X10 as it looks a little flakey on the underlying technology side - lots of comments in forums about commands going missing and general problems with other electrical appliances interfering with the automation system (as it sends its commands one way over your house power cables) - I want something rock solid and cost effective...there are more expensive alternatives but the cost does rule them out when all I want to do is dabble my toe in the home automation pond.

Taking the plunge...
So I bought the following from LetsAutomate

• 1 x Appliance module
• 1 x Lighting module
• 1 x Master remote control (used to discover and program your modules)
• 1 x USB PC interface (This allows your PC to control the Z-Wave modules via automation software)

Set up & configuration
You need to register each individual module with the master remote control. Modules can be put into a "group" and these will all respond to the same commands. If you send a group the "On" command all the modules in the group will switch on togther. You can also set "scenes" - these are cross group preset settings that will be sent to the modules in the scene. For instance, you might have a group for the kitchen and another group for the living room. With this you could turn all the lights in each room on/off as one. You may also create a scene that switches two of the kitchen lights to 50% brightness and one of the living room lights to 75% brightness and turns the TV on - scenes can include any module not just lighting modules.

It's a simple affair to register a module with the master remote - just follow the prompts on screen on the master remote. The approach I'm taking is to put each module into its own group (you can have up to 64 groups) so I have control over each individual module and use scenes to control multiple modules. Once the modules are registered you can control them using the master remote - but thats not fun!..The remotes a brick and I want something a little more sexier...

PC integration
To integrate control of the modules with your PC you need to select your automation software and setup the USB controller. As a USB device the controller is doing nothing until you get some drivers for it so I went on the hunt for these and found them on the Homeseer website - download them here. Ok, so unzip these and install then plug in the USB controller - it should pick up the drivers fine - your controller is now registered with your PC for some automation action! From what I gather controllers like the USB or serial PC controller are "dumb" in that you must beam your module configuration from the master remote to them in order for them to recognise the modules. You cannot register the modules directly with these "dumb" controllers - I maybe wrong but this is the impression I get from the documents.

Home Automation Software
This post from MCE guru Chris Lanier prompted me to investigate mControl - this looks like a .NET based MCE/Web home automation application. Trial downloaded and installed and into the Z-Wave configuration tool application. Hit connect and it should find the USB controller ok - look at the messages window at the bottom for confirmation. You will need to beam the master control configuration to the USB controller now - there is a button in the application to "receive configuration" and a menu on the master to "send configuration" - once done the messages window shows a list of your modules. Exit this application and go into Media Centre, more programs...mControl.

I set up my "zones", Kitchen and Living Room. The application module is used to control a radio in the kitchen and the lamp module is in the living room. I had a play with the automation macros and had the living room lamp switching on and off automatically at set times during the day. There is a fairly rich set of events and actions you can perform but more playtime required to investigate further (for instance you can switch a module on/off/dim when you play a DVD in MCE!).

First impressions
My first impressions are excellent...this stuff works really well even with just two modules. Its quick to respond and hasn't broken the bank; all this kit cost about £175 and about 1 hour to setup - the modules are about £35 and when the wall lighting modules get released (please, please soon!) this will be a killer automation system than anyone can use not just geeky me!

I'm going to buy the PIR to install in the garage to provide automatic lighting or possibly intruder alerts (the PIR can be set to email me when in "Alarm" mode!). I'll probably buy another lighting module but thats it until they release the wall switches I think...