In Visual Studio 2010, you can now get your F# unit tests loaded by the IDE. First, create your F# test project. This is just a normal library referencing Microsoft.VisualStudio.QualityTools.UnitTestFramework and having test classes and so on. The only change you need to do is go into the configuration and set the output folder to "bin" for both debug and release (instead of bin\debug).

Next, create a C# Test Project, and delete the code file. Then add an existing item, and navigate to the bin directory of your test project output. Select the DLL and Add as Link.

Finally, right click the solution, and change the project dependencies so the C# test project depends on your F# test project.

That's it. Now the IDE will pickup your F# tests and allow you to manage, run, and debug them right from VS. You may have to restart the IDE after setting all this up for it to work smoothly. 

Using the data protection API, I found I needed to be able to generate values by hand that my apps can work with later. So I wrote a small command line utility to help out with that process.

Example:

dpapicmd [/user] [/decrypt] [/utf8] [/entropy:<entropy>] {/clipboard|<text>}
Text is read as Base64 bytes unless /utf8 is used. Decrypted input and encrypted output is always Base64.

C:\>dpapicmd /utf8 /entropy:dogstuff "I'm barking mad."
AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAOE510Ds5PkGpG2g7PxkgXwQAAAACAAAAAAAQZgAAAAEAACAAAACSClIQpWDawT26jRrsFr/HauG2NjUw963fPKH+AcXqlwAAAAAOgAAAAAIAACAAAABwxg
Osbkh6TfdTUzaiEGgKJ/ohL91VGHIpRDrBeR7wvyAAAADyvUh1W+bmzUFago/LybBvmmQD96x2vCOiJgpPxTNItEAAAAAmVJ3DoTnIAfUIsO2ea8hqs6Rpp77gvDn77XAzXfACRY3IGT7BjicYJ7Og
NQhzsrHCybBK0DRQchrPK5+XT8TR

C:\>dpapicmd /utf8 /dec /entropy:dogstuff AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAOE510Ds5PkGpG2g7PxkgXwQAAAACAAAAAAAQZgAAAAEAACAAAACSClIQpWDawT26jRrsFr/HauG2
NjUw963fPKH+AcXqlwAAAAAOgAAAAAIAACAAAABwxgOsbkh6TfdTUzaiEGgKJ/ohL91VGHIpRDrBeR7wvyAAAADyvUh1W+bmzUFago/LybBvmmQD96x2vCOiJgpPxTNItEAAAAAmVJ3DoTnIAfUIsO
2ea8hqs6Rpp77gvDn77XAzXfACRY3IGT7BjicYJ7OgNQhzsrHCybBK0DRQchrPK5+XT8TR

I'm barking mad.

Download: dpapicmd.exe (7.5 KB) (Requires .NET 2.0 because I'm too lazy to write it in C.)

And horrible hacky source: dpapicmd.cs.txt (4.25 KB)

Fluent NHibernate is a nice way to be able to use NHibernate without having to deal with all that unchecked XML. This morning I decided to find out how well it works with F#. Things went relatively smooth. I've converted some code samples from the Fluent NHibernate First Project. I suggest having that open to fill in any gaps. I've also included the full project code and DB script at the end of this article.

If you're not familiar at all with Fluent NHibernate, basically it takes advantage of lambda expressions as Expression<T> to provide a somewhat strongly typechecked reflection system. Thus, instead of having attributes with hard-coded strings, or XML files, you have expressions that target the properties of objects you wish to map. The Fluent NHibernate library then takes care of hooking it up to NHibernate, and away you go. Something like that anyways.

Classes

So, first, we define the "entities" like the C# project does. Here's the first little pain. F# doesn't really support C#'s idea of "automatic properties". You can have vals on a class, which act like fields (although, they are implemented as properties). Or, you can manually specify them, like you would in earlier versions of C# which didn't have the auto-gen-a-field-for-me.

F# doesn't encourage uninitialized values. If you have uninitialized fields (vals), you need to mark them with an attribute to say you know what you're doing. So, that adds a bit more code overhead. What I do here is to alias the DefaultValueAttribute to "DV". So the first bit of our classes looks like this:

Now, NHibernate relies on having virtual properties so that it can dynamically create code to do nifty things like lazy loading. In F#, creating a virtual member means defining an abstract member and providing an implementation, in the same class. Since we don't have automatic properties, this means we'll have to define the backing field ourselfs. In all, the full code for the virtual property "Store" (virtual so NHibernate can lazy-load it) is:

Not the pinnacle of short code, but not horrible all things considered. The rest of the entity mappings are rather straightforward so I'll skip them here. I stuck with vals for anything that didn't have to be virtual, to keep it more concise.

Mappings: Easy Start

OK, so now to the "real" work. Fluent NHiberate looks for classes that subclass ClassMap<T>. It then creates an instance of them, which allows you to call the mapping methods in the object's constructor. From the First Project:

          
            public
            class EmployeeMap
: ClassMap<Employee> { public EmployeeMap()
{ Id(x => x.Id); } } 
        

OK, so how do we convert this to F#? It's easy... except for that lambda. The lambda in this case compiles to an Expression<Func<Employee, object>>. F#'s compiler does not support Expression<T>. So, we turn to experimental support. Referencing the FSharp.PowerPack.Linq assembly gives us the Microsoft.FSharp.Linq.QuotationEvaluation module. This extends the F# quotation type, Expr, with "ToLinqExpression", which returns an untyped LINQ Expression object.

To get this untyped Expression (LINQ) out of an Expr (F#) and into an Expression<T> (F#) suitable for Fluent NHibernate's consumption, I started off writing this tiny helper module:

When an F# quotation of a lambda is turned into a LINQ expression, the root node is a useless MethodCallExpression. So, we unwrap that by taking it's argument and using it to generate a typed lambda Expression<T>.  

This is a good start. But many of the Expressions that Fluent NHibernate looks for have a return type of object. Instead of having to write "box" or "upcast" all over, I added another function called "ToLinqObj". This takes an Expr<'a -> 'b>, but returns an Expression<Func<'a, obj>>.

Finally, writing "ToLinq" and "ToLinqObj" seemed too verbose, so I added some operators to the LinqHelper module:

Now we can start mapping:

We start off by disabling LazyLoad because most of the properties are not virtual, and NHibernate will fail to validate the mapping. Instead, we explicitly LazyLoad things, like the Store reference.

Mappings: Modifying the Expressions

Unfortunately, things didn't stay so simple. The Fluent NHibernate methods "HasMany" and "HasManyToMany", for instance, don't work with F#'s type inference. This is because they have several overloads, a couple of which take expressions. If we try this in the StoreMap:

We get an error because F# doesn't know what x is. (error FS0055: This lookup uses a deprecated feature, where a class type is inferred from the use of a class field label. Consider using a type annotation to make it clear which class the field comes from.) Using ~@ to keep it strongly typed fails as well; it can't figure out the overload. This is nothing surprising -- overloading is the enemy of type inference.

So, what do we do? Type annotations are not what I consider fun. So instead, we'll add non-overloaded ClassMap<'t> type extensions into the LinqHelper module:

By providing these extensions that are explicit once, we enable type inference for the rest of the time. We can now finish the StoreMap:

The only type "annotation" we need here is the upcast for the HasManyX. This is because F# forces you to be explicit. Accessing "Staff" in the first quotation means type IList<Employee>, not seq<Employee>. The upcast will sort this out for us.

This compiles fine. But remember how I said using expressions was a "somewhat strongly typed" way of doing things? The expression trees are interpreted at runtime, which allows failures a more complex type system might prevent. This is one of the times. If we try to execute it as-is, we get this exception:

 ---> FluentNHibernate.Cfg.FluentConfigurationException: An invalid or incomplete configuration was used while creating a SessionFactory. Check PotentialReasons collection, and InnerException for more detail.

 ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentException: Not a member access Parameter name: member

Ouch. Fluent NHibernate does not appreciate our expression trees. Why? Using FSI, we can inspect what we're actually converting the F# quotations to:

type SomeClass() = member x.Stuff = [|1;2;3|];;
let myExpr : Expr<SomeClass -> seq<int>> = <@ fun x -> upcast x.Stuff @>;;
let myLinq = LinqHelper.ToLinq myExpr;;

> myLinq;;
val it : Linq.Expressions.Expression<Func<SomeClass,seq<int>>>
= x => (x.Stuff As IEnumerable`1)
    {Body = (x.Stuff As IEnumerable`1);
     NodeType = Lambda;
     Parameters = seq [x];
     Type = System.Func`2[FSI_0022+SomeClass,System.Collections.Generic.IEnumerable`1[System.Int32]];}

> myLinq.Body;;
val it : Linq.Expressions.Expression
= (x.Stuff As IEnumerable`1)
    {IsLifted = false;
     IsLiftedToNull = false;
     Method = null;
     NodeType = TypeAs;
     Operand = x.Stuff;
     Type = System.Collections.Generic.IEnumerable`1[System.Int32];}

So, what's going on? The upcast is modifying the F# quotation (as it should), and the ToLinqExpression is sticking this in as a "TypeAs" node. Fluent NHibernate does not seem to like this. Not. One. Bit.

But, it IS ok if we have a Convert node in the Expression tree. I imagine this is because C# generates such nodes in its expression trees (say, accessing an int member in a Func<T, object> expression). So, our last hack in making F# work right here is adding a fixup function to our LinqHelper, and using it from ToLinq:

Now it's happy with our expressions.

Finally

The rest of the First Project is pretty straightforward in F#. For instance, creating the Session Factory:

Everything seems to execute as it should. Fluent interfaces in F# generate a bit more noise, because of all the ignores that have to be added. I'm thinking of creating a type extension to obj to add Ignore as a method, to make it look a bit more uniform.

Future

It's my hope that F# has now addressed most of these issues -- 1.9.6 is 7 months old. Expression<T> is growing in importance, outside of LINQ querying, so I cannot see F# not being able to handle them easily and generating similar output to C#. It'd be really neat if F# would auto-convert quotations to Expression<T> when the expr is a syntactic argument like it does with delegates now. I'd also be surprised if abstract/virtual members still lack the ability to define accessibility.

Code: fhib.zip (4.76 KB)
I did not include Fluent NHibernate or any of its dependencies. The project expects them to be in the project's lib folder, so get them here if you don't have them and extract them to "lib".

I welcome comments on this and suggestions for making the code more concise.

This entry was triggered by this question. Someone asked how to use AES, and we got 2 sample classes that do it wrong. The flaw in both was that they shared the IV which means your ciphertext will can leak information. One answerer didn't believe me at first, but then got it and deleted his code. The other person got offended and said IVs, performing authentication, etc. are all "corner cases" and any problem is "contrived". So, I'm going to provide a bit of code and show two problems that arise from not generating a unique IV for each message and not authenticating the decrypted data.

First, I wrote this a while ago: What is an IV?. That describes what an IV is, and why you need a unique one for each message. Wikipedia also has good information on this. Now for the demo. I used the code at the bottom, but removed the hashing and random IV from the code. So it's just encrypting the with the same key and IV for each message -- very straightforward. Here are the messages and their ciphertext:

"Alice; Bob; Eve;: PerformAct1"
"Alice; Bob; Eve;: PerformAct2"

tZfHJSFTXYX8V38AqEfYVXU5Dl/meUVAond70yIKGHY=
tZfHJSFTXYX8V38AqEfYVcf9a3U8vIEk1LuqGEyRZXM=

Notice how the first block of ciphertext is the same? All messages starting with "Alice; Bob; Eve;" will have that same first block. That means an attacker, after getting this ciphertext once, now knows if any message is addressed the same way. Very, very straightforward, basic attack. Now, maybe for a specific implementation you have in mind, this is not an issue. But it's still improperly implemented cryptography.

For the next attack, we're going to show that even with a random IV, you need to authenticate your decrypted messages. This code generates a 64-bit integer and encrypts it with AES and a random key/IV. Then, it starts changing bytes until the decrypt succeeds. Presto: the attacker was able to present a completely different value, and the decryption was successful.

public static void Main() {
    var buff =newbyte[8];
    new Random().NextBytes(buff);
    var v = BitConverter.ToUInt64(buff, 0);
    Console.WriteLine("Value: "+ v.ToString());
    Console.WriteLine("Value (bytes): "+ BitConverter.ToString(BitConverter.GetBytes(v)));
    var aes = Aes.Create();
    aes.GenerateIV();
    aes.GenerateKey();
    var encBytes = aes.CreateEncryptor().TransformFinalBlock(BitConverter.GetBytes(v), 0, 8);
    Console.WriteLine("Encrypted: "+ BitConverter.ToString(encBytes));
    var dec = aes.CreateDecryptor();
    Console.WriteLine("Decrypted: "+ BitConverter.ToUInt64(dec.TransformFinalBlock(encBytes, 0, encBytes.Length), 0));
    for (int i = 0; i < 8; i++) {
        for (int x = 0; x < 250; x++) {
            encBytes[i]++;
            try {
                Console.WriteLine("Attacked: "+ BitConverter.ToUInt64(dec.TransformFinalBlock(encBytes, 0, encBytes.Length), 0));
                 return;
            } catch { }
        }
    }
}

          
            Here's
an example run:
          
        

          Value:
5686260040031435365

Value (bytes): 65-7A-92-1A-61-A7-E9-4E

Encrypted: F4-62-AC-02-2D-7D-43-6A-4D-97-68-4D-95-9F-8A-DF

Decrypted: 5686260040031435365

Attacked: 1603329786558177755
        

Since there's no authentication of the decrypted data, an attacker can just play with the ciphertext until it generates an acceptable value. Perhaps you have other mitigations in your implementation/application for this, but why rely on that?

Here's some demo code (I haven't tested it much, so it might have some major issues -- but not by design AFAIK). Note this just shows performing an encryption operation, including the IV in the message, and verifying the decrypted bytes. Other things like replay attacks are not considered. If you're trying to learn how to use crypto so you can drop it into an application, STOP, then go read enough to understand what you're doing and the implications for your particular application.

aesdemo.cs.txt (4.38 KB)

 EndpointAddress
address =new EndpointAddress("http://127.0.0.1:52620/Service.svc");
BasicHttpMessageInspectorBinding binding =new BasicHttpMessageInspectorBinding(new SilverlightFaultMessageInspector());
ServiceClient proxy =new ServiceClient(binding,
address); proxy.DoWorkCompleted += new System.EventHandler<System.ComponentModel.AsyncCompletedEventArgs>(proxy_DoWorkCompleted);
proxy.DoWorkAsync(); 

protectedvoid Application_EndRequest(object sender,
EventArgs e) {
if (HttpContext.Current.Request.PhysicalPath.EndsWith(".svc",
StringComparison.OrdinalIgnoreCase) &&

HttpContext.Current.Response.StatusCode == 500 &&

!HttpContext.Current.Request.Browser.Crawler &&

HttpContext.Current.Request.Browser.EcmaScriptVersion.Major > 0) {
//
Set 200 if its a faulted service request

HttpContext.Current.Response.StatusCode = 200;

}

}

publicstatic Func<int, int>
Add(int x)
{ return y
=> x + y;
} publicstatic Func<int, int>
Inc = Add(1); //
and "full application" of add looks like this: Add(1)(2)

var
res =new EmptyCheckEnumerable<object>(dc.Execute<object>("SELECT
* FROM foo")); if (!res.IsEmpty)
{ //
allocate resources and consume the IE - will only execute the SELECT once }

public DbDataChangeProvider(long lastVersion)
{ this.lastVersion = lastVersion; this.txScope = ChangeTracking.GetSnapshotScope();
var validV = ChangeTracking.GetValidVersionForAll(dataContext);
baseline = lastVersion
< validV; currentVersion = ChangeTracking.GetCurrentVersion(dataContext);
}

There are other great books out there such as Expert F#. The F# Dev center has links to many other "learn F#" articles. All of these are great.

But, something I found helpful is going "purely functional", and Haskell is the perfect vehicle. When you're forced to think only functional, and don't have the other "escapes" F# has, you bend your mind into understanding how you can accomplish things without using mutation or object-orientation.

The downside of Haskell is that many resources seem to be very challenging to get into. There's no doubt that the learning curve for Haskell can be tough. On top of that, many materials tend to dive right into monads and it tends to end up too scary. I've even bought several other good books on functional programming, but none of them were easily approachable. (They have good content, but you can't start from zero by using them.)

Enter Real World Haskell. This is a *very* easy to follow book and really drives functional programming home. It doesn’t assume you know anything about functional programming at all, so the learning curve is a gentle slope.

Even better? It's completely available online, so you can start reading it right now! Plus, it has reader-submitted comments which are of tremendous use, as they ask and answer many common questions that might arise as you read along, without interfering with the flow. You can read the entire book here: http://book.realworldhaskell.org/read/ [But buy it to support the excellent work the authors have done!]

I've found that my F# skills have gone up tremendously by reading Real World Haskell. For instance, I "sorta" understood F#'s computation expressions and builder. Say, enough to use them -- that's easy, like most things in F#), but understanding the concepts behind them? Starting to learn Haskell really brings the understanding around. This isn't to say that you'll eschew mutation and OO in F# -- such concepts can be very useful (and increase performance on the CLR). But at least you'll know when a more elegant solution is available.

(Plus, it's fun! As someone in #haskell on freenode put it to me: "Learning Haskell will f*ck with your brain, and you'll like it.")

[Yea, I’m not a web dev, and actively avoid it as much as possible, so I’m late to this party.]

The interesting thing about ASP.NET MVC is that it takes an opposite approach to ASP.NET in general. ASP.NET concepts try to “build up”, so as to shelter us from the evolved idiocy that is HTML, as well as clean up the inherently stateless nature of HTTP. ASP.NET MVC makes no attempt and forces you to deal with reality. Considering ASP.NET’s abstraction isn’t really perfect (example: databinding sucks), MVC’s approach is unfortunately refreshing.

Because of its “raw” nature, you’ll be writing a lot more HTML than you’d do with ASP.NET, and this HTML must line up with code on your server. To make this less of a pain, there are some HTML helper functions. Rob “ type inference” Conery has an overview here.

Here’s the signature for one of the functions:

    public static string CheckBox(this HtmlHelper helper, string htmlName, string text, string value, bool isChecked, object htmlAttributes);

The last parameter confused me. Why would it be an object? Am I supposed to pass in an IDictionary<string,string>? Just a long string? To make it more confusing, other helpers had two overloads:

    public static string TextArea(this HtmlHelper helper, string htmlName, object value, IDictionary<string, object> htmlAttributes);

    public static string TextArea(this HtmlHelper helper, string htmlName, object value, object htmlAttributes);

OK, so they explicitly called out the IDictionary there – THEN WHO WAS OBJECT HTMLATTRIBUTES?

Rob covers in his overview. The idea is that you’re supposed to use anonymous types to hack around the lack of tuples.

<%=Html.Whatever(arg1, bla, …,
                             new { @class=”cssx”, style=”x:f” …}) %>

What a great case of not-having-built-in-tuples-is-lame. It’s so lame, Microsoft’s own developer teams have to resort to weird (but quite creative!) hacks like this so that their syntax won’t completely suck*. Damn.

And now, a duck

For bonus points, there is another C# compiler feature that the MVC team could have [ab]used, and it would arguably have made more sense (although the syntax isn’t as tight). C# supports duck typing on collection initializers! So, they could create a class like this:

    publicclassHtmlAttributes : System.Collections.IEnumerable

    {

        public System.Collections.IEnumerator GetEnumerator() { ... }

        publicvoid Add(string name, object val) { ... }

    }

And then they can write this:

new HtmlAttributes { { "A", 123} , {"B", "test"} }

No, it’s not as tight as the anonymous type syntax, but it does make a lot more sense.  And tuples still make much more sense than either approach, and have benefits for the rest of the language to boot (death to out parameters!).

 *The only benefit I see in anonymous types is that, at compile time, you'll  know there are no key conflicts - but that is totally trivial in the way they use them, since all the keys are declared right there.

I dislike this one. Because we're not doing the pattern match ourselves, we end up yielding 'a option. But None is never valid because we guard on IsEmpty. This makes us stick a Seq.choose on with an identity function to strip off the Some. [Side note, is there no built-in identity function?] Bringing the match into the seq fixes the issue, but the code is pretty long:

I just read these two posts:
http://blogs.msdn.com/simonince/archive/2008/08/15/strongly-typed-primitives.aspx
http://www.thejoyofcode.com/Avoiding_Primitive_Obsession_to_tip_developers_into_the_pit_of_success.aspx

And that reminded me about something we recently did. One system we're working on uses a lot of string identifiers for many different types of objects. There are many, many of these stored and passed around, so keeping things efficient was of high concern. 

The downside of string IDs (really, using any common type as an ID) is that it's legal to pass any primitive of the same type. Strings and integers abound, both as IDs of other classes, as well as general use. So it's not unimaginable that someone could pass the wrong parameter some where. This could lead to runtime crashes or unexpected results (if the ID is actually a real record of another class of object). Finally, using common types for IDs reduces usability. The signature "public void Delete(int id)" leaves a lot to be desired.

We wanted to hit all these issues, in addition to keeping things simple. There are times when untyped data needs to be converted, and this should be easy and clear. We wanted to avoid having to define new types when we had new classes of objects to identify It is also customer-visible code, so C# is used.

Using a reference type was unacceptable, because it'd add at least 12 bytes overhead (I think more on x64). Using a struct fixes this, in addition to dealing with silly nullability issues. [If a type can be null, it should always be explicit. C#'s "references types can be null" makes this hard.]

The end result was quite simple. Wrap a string in a structure so equality and hashing pass through. But, take advantage and remove case/cultural sensitivity (since in many systems, data IDs are not case sensitive). Provide explicit conversions so you can easily convert to and from strings, but never by accident. (If the conversions were implicit, you're back in the starting point.) Finally, add a generic parameter that is never used. The generic parameter gives you distinct types without having to define them. Now the APIs can look like:

    public void Delete(Id<Product> id)...

    Dictionary<Id<Group>, List<Id<User>>> members...

When you do have hardcoded IDs, as the blog entries I mentioned do, you can convert easily: (Id<User>)"Admin". Nulls are treated as empty, all the time (empty may be a valid value anyways).

When a truly optional ID is needed, use nullable types: "Id<Whatever>?". This fully captures how values are handled. This is vastly better than "It's a reference type, so maybe null is allowed. Or maybe null will crash. Empty string might be considered null, or maybe empty string means optional." With explicit nullability, the type system says it all.

The best part is that there should be pretty much no overhead. I'd expect the equality functions to be inlined, and there's no memory overhead, since the struct is simply a string reference.

publicstruct Id<T>
: IEquatable<Id<T>> { public Id(string name)
{ this.name = name
?? "";
} readonlystring name; publicstaticexplicitoperatorstring(Id<T>
x) { return x.name
?? "";
} publicstaticexplicitoperator Id<T>(string s)
{ returnnew Id<T>(s);
} publicoverridebool Equals(object obj)
{ return !(obj is Id<T>)
? false :
((Id<T>)obj) == this;
} publicbool Equals(Id<T>
other) { return other
== this;
} publicoverrideint GetHashCode()
{ return (name
?? "").GetHashCode();
} publicoverridestring ToString()
{ return name
?? "";
} publicstaticbooloperator ==(Id<T>
a, Id<T> b) { return StringComparer.InvariantCultureIgnoreCase.Compare(a.name,
b.name) == 0; } publicstaticbooloperator !=(Id<T>
a, Id<T> b) { return !(a
== b); } }

In my last post, I said I'd write the sample in C# to compare to F#. Well, I grossly underestimated the power of query comprehensions. The C# version is almost the same length (formatting differences, mainly). I'm surprised and impressed. (Or maybe I'm writing F# like I'd write C#.) Edit: I think maybe sequence expressions could cut it down a bit...

...But... C# still can't do discriminated unions efficiently or effectively ;).

No support for anonymous methods/lambda statements. I first realised this when someone commented on one of my functional C# postings, asking "how can I write this in VB"? Turns out VB has no support for anonymous methods. Ouch.

This lack of capability rules out a so much functionality, I honestly cannot imagine writing anything significant today using VB.NET. But with LINQ and F# making such a big splashes, I have high hopes for the future of functional programming on the .NET languages, and I'm sure VB.NET will get itself fixed up.

Really, dropping anonymous methods in favour of stuff like XML literals? Maybe that's why it's listed as first time or casual programming ;). (No, I jest. Scheme is far better for first time programming.)

Usually I use WCF just by referencing the contracts directly and using my own generic helper methods. Unfortunately, I had to make use of the built-in async patterns that the codegen can provide. Hence, I ended up using the "Add Service Reference" dialog, which notices (unlike svcutil on the command line) that since I'm requesting async, it'll need to generate my interfaces even though they're already referenced. Maybe in .NET 4 we'll have a new async pattern that'll leverage Expression Trees to rid us of having to use the rather ugly FooAsync/BeingFoo patterns.

Things were working just fine until I added a reference to a new service. Adding the service reference didn't work; it failed silently. It added all the schema files to the project, but the Reference.cs file was empty. If I told it to NOT used referenced assemblies, then it generated everything just fine.

After switching to the command line and using svcutil, I got these errors:
Error: Cannot import wsdl:portType
Detail: An exception was thrown while running a WSDL import extension: System.ServiceModel.Description.DataContractSerializerMessageContractImporter
Error: Referenced type 'MyApp.Management.CoolFunction.CoolThingRule, MyApp.Management.Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' with data contract name 'CoolThingRule' in namespace 'http://schemas.MyApp.com/management/v2/00' cannot be used since it does not match imported DataContract. Need to exclude this type from referenced types.XPath to Error Source: //wsdl:definitions[@targetNamespace='http://schemas.MyApp.com/management/v2/00']/wsdl:portType[@name='ICoolThingManagement']

I regenerated without the referenced code and figured out that the problem was that one type referenced an enum which had some values without the [EnumMember] attribute (although it had a DataContractAttribute on the enum itself). Because of this, WCF wanted to represent the enum with a string and thus was incompatible. Adding the EnumMemberAttribute to all the values sorted out this issue.

Silverlight 2 is pretty nice. Compared to dealing with the nightmare of HTML, CSS, AJAX and whatever else, it's quite divine. Of particular interest is that Silverlight 2 has a mini Windows Communication Foundation stack that can do basic SOAP work (and some "Web 2.0" type things too I think). While the SL WCF stack works pretty well overall, it doesn't support SOAP faults.

First off, SOAP faults usually cause the server to return an HTTP 500. For whatever reasons, this isn't handled correctly between the browser and Silverlight, and results in an UnexpectedHttpResponseCode. OK, so go into the Global.asax and in Application_EndRequest change 500s to 200s. [Yes, this requires that you run WCF with AspNetCompatibility.]

Some progress. Silverlight now gives a more useful exception:

Error: System.Runtime.Serialization.SerializationException: OperationFormatter encountered an invalid Message body. Expected to find node type 'Element' with name 'MyFunctionResponse' and namespace 'http://schemas.contoso.com/coolservice/v2/00'. Found node type 'Element' with name 's:Fault' and namespace 'http://schemas.xmlsoap.org/soap/envelope/'

Hmm, it's getting the fault, but can't seem to handle it. I tried adding typed faults to my WCF contract, but the SL WCF stack doesn't have the FaultContractAttribute. It appears as if SL cannot read faults at all.

Enter HackFaults. That message provides two pieces of data from the SOAP message, the element name, and the namespace. We can hijack these to pass our fault information, then extract it from the exception message. H to the A to the C-K-Y.

First off, we need to get access to our WCF fault. We can do this with an System.ServiceModel.Dispatcher.IErrorHandler. I've attached the full code, but basically you add an attribute to your WCF service to wire up the error handler. If you're doing WCF work, you probably want this anyways for logging purposes and so on. I've attached the entire error handler code to this article. The real meat of the error handler is this little function:

public void ProvideFault(Exception error, MessageVersion version, ref Message fault)
{
    // HackFaults store the exception in the httpcontext
    var context = System.Web.HttpContext.Current;
    if (context !=null) {
        // Only works in AspNetCompat mode
        if (!context.Request.Browser.Crawler && context.Request.Browser.EcmaScriptVersion.Major > 0) {
            // This should rule out non-browsers
            context.Items.Add("HackFault", error);
        }
    }
}

I'm not sure if there's a better way to differentiate between browsers and real SOAP stacks. If there is let me know. Now, in our Global.asax, we want to pick this information up and replace the actual SOAP fault with our own hackfault:

          
            protected
            void Application_EndRequest(object sender,
EventArgs e) { //
In all fairness, I was drinking eiswein at the timeif (Context.Items.Contains("HackFault"))
{ Context.Response.ContentType ="text/xml";
Context.Response.StatusCode = 200;
Context.Response.ClearContent(); var hackEx = Context.Items["HackFault"] as Exception;
var hackFaultXml =string.Format( "<s:Envelope
xmlns:s=\"http://schemas.xmlsoap.org/soap/envelope/\"><s:Body><{0} xmlns=\"{1}\"
/></s:Body></s:Envelope>", hackEx.GetType().ToString(), Context.Server.HtmlEncode(hackEx.Message));
Context.Response.Output.Write(hackFaultXml); } } 
        

Now we have a message that has the bits of data we care about in the right places for the Silverlight exception to be ready for parsing. The SL-side parsing code is vile but easy:

          
            public
            static
            string ExtractHackFaultMessage(this Exception
ex) { //
HackFaults generate these Exceptions and messages://Without
debug://Could
not connect to server: System.Runtime.Serialization.SerializationException: [SFxInvalidMessageBody]//Arguments:FaultException,Something
silly//With
debug://Could
not connect to server: System.Runtime.Serialization.SerializationException: 


          
            OperationFormatter
encountered an invalid Message body. Expected to find node type 'Element' with 


name 'SomeOperationResponse' and namespace 'http://schemas.cool.com/yea/v2/00'. 


Found node type 'Element' with name 'FaultException' and namespace 'Something silly'
            if (!(ex is System.Runtime.Serialization.SerializationException)) returnnull;
var msg = ex.Message; string regexp; if (msg.Contains("[SFxInvalidMessageBody]"))
{ regexp =@"Arguments:(.*),(.*)";
} elseif (msg.Contains("OperationFormatter
encountered an invalid Message body")) { regexp =@"Found
node type 'Element' with name '(.*)' and namespace '(.*)'"; } else { //
Nope, it's some thing elsereturnnull;
} var match = System.Text.RegularExpressions.Regex.Match(msg,
regexp); if (match.Groups.Count
!= 3) returnnull; //
Not expected var exType = match.Groups[1].Value.Trim();
var exMsg = match.Groups[2].Value.Trim(); if (exType
== "System.ServiceModel.FaultException") return exMsg; elsereturn exType+ ":
"+ exMsg;
}
        

This little function will give me the fault information (and not mention FaultException if that's what it is) or null if it can't extract it. When dealing with errors, I can show an error like this: "Error: " + (ex.ExtractHackFaultMessage() ?? ex.ToString())

This gives me at least rudimentary error reporting from the server back to Silverlight. If this can be improved or fixed up, please tell me. (Typed fault exceptions would be nice, but then SL can't codegen the contracts.) At any rate, it works as a cheap hack until Silverlight Beta 2 (they gotta fix it by then, right? Right?).

... or, how the hell to use Vista and 2008's new ETW stuff with managed code. And, introducing ecmanaged: A decent way to do all this stuff.

Quick ETW Overview

Actually, the real ETW overview is here: http://msdn.microsoft.com/msdnmag/issues/07/04/ETW/ <-- This is some of the best overview and documentation on it (the other good stuff is the ecmangen documentation in the Windows SDK bin folder). The MSDN stuff is terribly confusing for the most part. Or maybe I'm too spolied by how easy it is to find stuff in the BCL. My overview is on what you gotta do to make things work in .NET.

ETW is a real pain to use with .NET. Even so, ETW starts off looking really promising. You define everything in a nice XML manifest file, and everything is based off that. But wait, everything? Shouldn't the manifest be the end-all? Yea, that'd make logical sense. No, you run some tools from the Windows SDK. First you run MC, which generates a .h header file. Managed devs are growning now -- why the hell should something as general as event tracing be language specific? The .h file contains the processed event descriptors, ready for C consumption. 

It worsens: MC also generated a resource script. You have to compile that with RC and it'll create a Win32 .res resource. Then you compile that into a binary (the C# compiler has the /win32res option). Then you go back and edit your XML manifest and make sure it points to the final binary. Wait, what? Yes. The resources that MC generates for RC contain all the messages that are in your XML manifest. Someone thought it was a really cute idea to go and make the Event Viewer not only read all the data from your manifest, but also have to go look it up from some binary resources.

Actually, this probably made sense to someone on the Windows team since I'm guessing they already have tools to go and localise Win32 resources or something. Unfortunately, it sucks and makes no sense for anyone NOT in their particular position. Now, I hope I'm wrong (I really, really want to be wrong), but I think there's no way to force the message strings to just stay in the XML file and be read from there. 

Finally, things get easy again. Just run "wevtutil install-manifest Some.man" (wevtutil is in system32). In fact, this utility is so user friendly, it even lets you type "im" instead of "install-manifest". At this point, assuming the other steps went well, your provider shows up in Event Viewer.

ECManGen

But wait, how do I actually make that manifest? This part is almost the easiest. In the Windows SDK, there's a lovely little tool called ECManGen. Just fire it up, and go to town adding Providers, Channels, Templates, and Events.

Providers are the main things that show up in your Event Viewer, such as MyApp-FooProduct-LameComp. Channels separate Admin/Operational/Debug and others. Templates are an argument list for Events. If you have, say, a bunch of events that take the same kinds of parameters, you can share templates among them (I find it helpful to create a "SingleStringTemplate".) It's very straightforward.

*Note: I can't actually get Admin channels to work. If I create an event and stick it in an Admin channel and set its level to Informational, MC complains (as does ECManGen) that the level has to be Critical, Error, or Informational. Uh, OK. Instead, just use Operational.

Except... ECManGen is a free utility. (Free? Perhaps not, seeing as the annual MS bill for a 4-person dev team is around $20,000 (counting just MSDN) -- but it's well worth it.) Part of the docs say: "NOTE: For the Manifest Generator Tool to function correctly, the file winmeta.xml (which contains pre-defined metadata values) must either be in the Include directory of the SDK or in the same directory as the Manifest Generator tool." OK, easy enough. Except... it doesn't work that easily. The only way I got it to work was to copy the xml file over to the same directory, *and* start ECManGen from that directory.

Oh yea, ECManGen won't open your manifest file if you pass it as an argument, so forget about cute VS integration. Just Google ecmangen and go rate up the bugs on Connect :).

Going Managed

OK, so you're not living in the last century and use decent tools -- how does this map to C#? First off, you create an EventProvider with the right Guid (the one from your manifest). Then you create an EventDescriptor for each event, matching up all the little parameters (the MSDN docs for EventDescriptor have more details). Finally, you can call WriteEvent, passing the EventDescriptor *by ref* for some reason (no, I can't figure out why).

Oh yea, and you have to hookup that Win32 resource to your C# project, so if you needed another resource (like another app manifest?), you'll have to go deal with merging them and all that hassle. And, don't forget to make sure the parameters you pass into the object[] array of WriteEvent line up with what your manifest has. And also, the .NET API won't even handle the Boolean->BOOL (4 byte) silliness for you. 

In summary, it's a lot of boring, error-prone work, and you'll have to repeat it every time you edit your manifest. Yuck. Maybe it's just easier to use the old event log stuff and forget about all this fancy ETW stuff.

However, the ETW features are pretty cool. As is ecmangen (well, for the most part). I couldn't tear myself away from the promises the new ETW stuff offered, so I sat down and wrote a hack tool to do everything for me. The goal is to have an XML file, then convert it right into a DLL I can use.

ecmanaged does exactly this. Sorta. It doesn't deal with complicated scenarios or performance counters, but handles straightforward event logging with templates in an easy and type-safe fashion.

Using ecmanaged

Usage: ecmanaged ecgen <manifest> /out:<target assembly> /namespace:<target namespace> /class:<target class>
Usage: ecmanaged msglocate <manifest> <messageFileName>
Usage: ecmanaged install <manifest> <targetManifest> <messageFileName>
Usage: ecmanaged uninstall <targetManifest>

Call ecgen with a manifest, and it'll go build an assembly with the wrappers for calling your provider events. It'll stick the Win32 resource on the assembly too, so it's ready to be used.

Call msglocate with the manifest and the message binary file, and it'll go fix the manifest to point to the right place.

Call install, and it'll copy the manifest to another location, fix the msg location, and call wevtutil on it. It copies because I'm assuming your original manifest is under source control, and hence shouldn't be fixed up.

Uninstall just calls wevutil uninstall-manifest.

Yey! Now you can just make a batch file to call this on your manifest and run it every time you change your manifest. What I do is stick this in a folder in my project called "Eventing". I check the resulting binary into source control (just like a 3rd party DLL). The downside is making sure you always update the DLL if you modify the manifest. I'm sure somehow msbuild can come to the rescue (and autoupdate when the manifest changes, so Intellisense keeps working), but this works for me.

I welcome all comments/questions/insults, but please, no death threats. Download: ecmanaged.exe (45 KB)

P.S. No, I don't hate Windows or the devs on it, although I hate unmanaged code (it IS 2008, innit?). Perhaps I'm just annoyed I have to make a Saving Throw versus Hang every time I click Send in Outlook 2007 (yes, patched up, even with Vista SP1... which made it worse). But I still think this whole ETW toolchain is terribly unpolished.

Rob Conery talks about geocoding with LINQ here. In his post, he provides some code using the Haversine formula to compute the distance between two points on earth. His function is declared as:

Func < double , double , double , double , double > CalcDistance = (lat1, lon1, lat2, lon2) => …

Further, this delegate is wrapped in a normal C# method. Now, read my previous post about calling functions in LINQ to SQL. Can you see where things are going to go wrong? That's right, a normal delegate or method can't be converted into SQL by LINQ, as the engine has nothing to work with. Only when our code is available as data can the LINQ to SQL engine do it's magic. Since there's some insinuation that LINQ to SQL just can't handle things like Haversine, I'll demonstrate how to do it.

If you want to use your own "complicated" functions with LINQ to SQL, you'll need to manually construct the predicate expression. It's not pretty, but, it does let you convert somewhat detailed functions, such as Haversine, inside of LINQ-to-SQL queries. This is not always the right approach: in some cases it'll be better to use a UDF or stored procedure. (If someone knows a native, better way, please let me know! TomasP's Expandable stuff looks cool, but I ran into some bugs on Beta 2. This is really something the compiler should help out with!).

To start off, we need to declare our function as an Expression Tree (I cannot vouch for the accuracy of this code; I'm merely demonstrating LINQ to SQL technique. For the geocoding details, read Rob's post.):

const double R = 6367;
constdouble RAD = Math.PI / 180;
staticExpression<Func<double, double, double, double, double>> dist =
(lat1, lon1, lat2, lon2) =>
    R * 2 *
    (
        Math.Asin(Math.Min(1,Math.Sqrt(
            (
            Math.Pow(Math.Sin(((lat1 * RAD - lat2 * RAD)) / 2.0), 2.0) +
            Math.Cos(lat1 * RAD) * Math.Cos(lat2 * RAD) *
            Math.Pow(Math.Sin(((lon1 * RAD - lon2 * RAD)) / 2.0), 2.0) 
            ) 
        ))) 
    );

OK, that was the easy part. We just had to wrap Expression<> around the type declaration and remove the method calls. But how do we pass this to our query? The Where method has this signature:

public static IQueryable<TSource> Where<TSource>(thisIQueryable<TSource> source, Expression<Func<TSource, bool>> predicate);

Hence, we need to provide a predicate of Expression<Func<TSource, bool>> for it do its magic. To work with expressions, we need to start using the System.Linq.Expressions namespace. For more clarity, I aliased E to System.Linq.Expressions.Expression. Building expressions isn't particularly hard, it's just annoying and time consuming. C#'s lack of "symbolics", or a way to use the compiler to create expressions to reference properties, etc. means we have to use strings to do so. I never did claim it'd be pretty.

We start off with some data from elsewhere in our application. In my example, I'm just going to declare some locals:

double coolLat = 43.641852;
double coolLong = -79.387298;
double maxDist = 100;

The first Expression we need is a ParameterExpression to refer to the source item from the table (i.e., the parameter the Where method is going to give to us). In my code, my table is called Accounts, hence my object type is Account. To create the parameter expression:

var acctParam = E.Parameter(typeof(Account), "a");

Next, we need to be able to reference the fields in the table. With LINQ to SQL, these are properties on our object. We create them like this:

var acctLon = E.Property(acctParam, "Longitude");
var acctLat = E.Property(acctParam, "Latitude");

The secret sauce is creating the invoke to our dist expression. This is where all the work comes into play and includes all our complicated code in the LINQ to SQL query. Fortunately, after building up our arguments separately, it's not that hard:

var distCalc = E.Invoke(dist, E.Constant(coolLat), E.Constant(coolLong), acctLat, acctLon);

We have to use ConstantExpressions for our local variables. Using constant allows us to capture the value of those variables. Now we're ready to finish off, by adding a less than comparison and turning it all into a <TSource, bool> LambdaExpression:

var maxComp = E.LessThan(distCalc, E.Constant(maxDist));
var pred = E.Lambda<Func<Account, bool>>(maxComp, acctParam);

Our query can now look like this:

MembersDataContext dc = newMembersDataContext();
var q = dc.Accounts.Where(pred);
Console.WriteLine(q.ToString());

When we run it, we see that LINQ to SQL is quite capable of handling our little bit of math:

exec sp_executesql N'SELECT [t0].[AccountId], [t0].[Latitude], [t0].[Longitude]
FROM [dbo].[Accounts] AS [t0]
WHERE (@p0 * ASIN(
(CASE
WHEN @p1 < SQRT(POWER(SIN(((@p2 * @p3) - ([t0].[Latitude] * @p4)) / @p5), @p6) + (COS(@p7 * @p8) * COS([t0].[Latitude] * @p9) *
    POWER(SIN(((@p10 * @p11) - ([t0].[Longitude] * @p12)) / @p13), @p14))) THEN @p1
ELSE SQRT(POWER(SIN(((@p2 * @p3) - ([t0].[Latitude] * @p4)) / @p5), @p6) + (COS(@p7 * @p8) * COS([t0].[Latitude] * @p9) *
    POWER(SIN(((@p10 * @p11) - ([t0].[Longitude] * @p12)) / @p13), @p14)))
END))) < @p15' ,N'@p0 float,@p1 float,@p2 float,@p3 float,@p4 float,@p5 float,@p6 float,@p7 float,@p8 float,@p9 float,
    @p10 float,@p11 float,@p12 float,@p13 float,@p14 float,@p15 float',@p0=12742,@p1=1,@p2=95.412000000000006,@p3=0.017453292519943295,
@p4=0.017453292519943295,@p5=2,@p6=2,@p7=95.412000000000006,@p8=0.017453292519943295,@p9=0.017453292519943295,@p10=102.63200000000001,
@p11=0.017453292519943295,@p12=0.017453292519943295,@p13=2,@p14=2,@p15=100

I still think there should be some kind of syntax so we could write it like we want to: (Where(a=>dist(coolLat, coolLong, a.Latitude, a.Longitude) > 10)). If anyone knows a built-in way to do it, please let me know. I'm sure it's just something simple I'm overlooking...

This was sparked by the issues raised by Rob Conery here. Basically, if you have some semi-complicated function that you need to apply to a LINQ-to-SQL query, how can you do it? This is somewhat covered by TomasP.NET: Building LINQ Queries at Runtime in C# and Joseph Albahari: Dynamically building LINQ expression predicates. I recommend those two articles; they're very good. I'm going to write a bit about it too. Some of it might be redundant, some of the ideas I took from those articles. The code is all mine.

So, let's say we want to get all accounts where the square root of the account ID is even. This will serve as our placeholder for a totally contrived example. Just calling our method in our LINQ query won't work because the LINQ-to-SQL code isn't going to know what to do with it. A method is just an opaque block of code with a name. Here's an example:

static void Main(string[] args) {
    MembersDataContext dc = newMembersDataContext();
    var q = dc.Accounts.Where(a => IsRightAccount(a));
    Console.WriteLine(q.ToString());
}
static bool IsRightAccount(Account a) {
    return Math.Sqrt(a.AccountId) % 2 == 0;
}

This code crashes with: Unhandled Exception: System.NotSupportedException: Method 'Boolean IsRightAccount(ConsoleApplication1.Account)' has no supported translation to SQL. Which should be expected, as LINQ to SQL cannot know what goes on inside that method and thus can't translate it.

Let's change things to make IsRightAccount be a Func delegate (from a lambda expression):

static Func<Account, bool> IsRightAccount = a => Math.Sqrt(a.AccountId) % 2 == 0;

Now we get: Unhandled Exception: System.NotImplementedException: The method or operation is not implemented. At System.Data.Linq.SqlClient.QueryConverter.VisitInvocation(InvocationExpression invoke). That's a somewhat strange exception, as I'd expect it to be a bit more helpful. At any rate, I'd expect it to crash, because that is just a delegate to a method. It's still an opaque block of code.

Enter the magic Tree of Expressions: Expression<T>. As I mentioned in my last post, Expression Trees provide "introspection" (reflection against code). In the examples above, the lambda that calls IsRightAccount for the Where clause actually turned into an InvocationExpression that represents a call to the delegate provided. Hence me saying that it is "opaque". What we need is to make sure that our code (our IsRightAccount calculation) is visible as data. When it's visible as data, then LINQ-to-SQL can go and say "Oh, you want to take the square root of the account ID, mod it by 2, and see if that's zero… now THAT I can do in SQL".

Declaring an Expression Tree is really simple. First, make sure you import System.Linq.Expressions if you don't want to fully qualify the name. Then, declare your tree just like any lambda Func, except this time make the type Expression<MyFunc>:

static Expression<Func<Account, bool>> IsRightAccount = a => Math.Sqrt(a.AccountId) % 2 == 0;

We will also change our Where clause to accommodate the fact that we are not calling a method:

var q = dc.Accounts.Where(IsRightAccount);

And presto! Our program now shows a SQL conversion:

SELECT [t0].[AccountId], [t0].[Email]
FROM [dbo].[Accounts] AS [t0]
WHERE (SQRT(CONVERT(Float,[t0].[AccountId])) % @p0) = @p1

Things get a bit more complicated when we try to stack expressions together. As far as I know, we must create the Expression manually (using the Expression static methods); the compiler won't help out. If anyone knows a built-in way around this, please let me know. Otherwise, see the links at the top of this article for more information and other workarounds.

This also applies if you have complex logic that doesn't directly map to a item -> bool predicate expression. In those cases, you can still encapsulate most of your code by using the compiler to generate the bulk of the Expression, and then just wrap it with a bit of hand-created expression. In my LINQ to the CRUD article, the code attached uses this approach to generate queries for the select/delete commands. Again, I will note that TomasP has written expansions so you can just write myCoolExpression.Expand(arg) rather than building everything by hand.

If you know of any other links or work done in this area, I'm very interested in seeing other approaches. Thanks!

Apparently, there is some considerable confusion over all the new C# language features. People who I would hope are reasonably intelligent are completely misunderstanding some C# fundamentals. Agreed, a lot of the new concepts introduced to C# 3.0 are might seem relatively foreign to C# users. Microsoft's marketing related to LINQ doesn't help much either. I'm going to try to clarify the top few things I've seen. I'll reference the C# spec (http://msdn2.microsoft.com/en-us/library/ms364047(vs.80).aspx) so you can know I'm being accurate.

Myth: LINQ is just a data access technology

While data access is obviously a large part of LINQ (even the name stands for Language Integrated Query), you can do a lot more than just access data. Re-using a previous example:

    args
        .Select(s => newThread(() => SomeLongProcess(s))) 
        .Process(t => t.Start())
        .ToList()
        .ForEach(t => t.Join());

There's no sign of data access there! The practical functional C# articles on my site go into more detail on this. Suffice to say, while a lot of new features were added to "make LINQ possible", much, much, more is possible than just creating queries. (Alternatively, you might chose to limit the meaning of LINQ, as I'd like to do. In this case, I wouldn't consider using lambdas, extensions, etc. as LINQ. Others may disagree. Marketing?)

Implicitly typed local variables (var keyword)

C# is still statically and strongly typed. But, there's a new feature that lets you declare local variables without specifying the type, if the type can be inferred from the initializing expression. From the spec:

var i = 5;
var s = "Hello";
var d = 1.0;
var numbers = new int[] {1, 2, 3};
var orders = new Dictionary<int,Order>();

The implicitly typed local variable declarations above are precisely equivalent to the following explicitly typed declarations:

int i = 5;
string s = "Hello";
double d = 1.0;
int[] numbers = new int[] {1, 2, 3};
Dictionary<int,Order> orders = new Dictionary<int,Order>();

Oddly enough, the C# spec doesn't even mention LINQ or anonymous types when it talks about "var" locals. Why is there confusion about this simple feature? Let's examine anonymous types:

C# anonymous types allow you to declare a type just by specifying its fields. From the spec: "C# 3.0 permits the new operator to be used with an anonymous object initializer to create an object of an anonymous type." For instance, the following code is a valid expression:

new { Name = "Michael" }

This produces an object of a new, anonymous, type, containing a single string property called Name. Hence, this code works:

Console.WriteLine(new { Name = "Michael" }.Name);

However, how can you assign such an object to a variable? Yes, that is the only "need" for the var keyword. There's no way to name the type, since it's anonymous. Regardless if you agree with anonymous types (versus a Tuple class), this is the place where you *need* to use var: assigning an anonymous type to a local.

As you may have noticed, I still haven't mentioned LINQ. Anonymous types are not LINQ specific. They are, however, particularly helpful for certain LINQ queries:

var topCustomers = MyDatabase.Customers.Where(c => c.GoldStar == true).Select(c => new {c.CustomerId, c.Name});

Because of this, people start to associate anonymous types only with LINQ queries and hence, var with LINQ only. The truth is that these features can be used anywhere.

What's the takeaway here? The var keyword simply allows the compiler to infer the type of the variable so you don't have to specify it. Nothing more, nothing less. Some people still want to explicitly annotate every single variable – hey, that's their choice. But don't be locked into this just because there was no option before. Me? I'll take more concise code any day!

As a side note (if this wasn't clear), the var keyword is NOT dynamic typing, just implicit typing (type inference).

Dynamic C#

Seems there's a lot of confusion about C# being dynamic. C# is not dynamically typed, as some seem to imply. I think perhaps some of the confusion comes from all the nice type inference that C# provides. Using the var keyword as shown above might make some people feel "oh, I'm saying var just like Javascript!". Adding to the confusion is the fact that C# is now a "semi"-functional language. For instance, from "why's (poignant) guide to ruby", we see this Ruby code:

5 . times { print " Odelay! " }

C# allows us to write in a similar style:

5.Times(() => Console.WriteLine("Odelay!"));

The next Ruby example in that guide goes like this:

exit unless " restaurant ". include? " aura "

In C#, we can write:

exit.Unless(() => "restaurant".Contains("aura"));

The Wikipedia article on Functional Programming lists a few features that dynamic languages usually have:

Eval

Sorta… you can create Expression<T> and execute them. On the other hand, you can't do anything like eval(myString) (which is just asking for runtime failure).

Higher-order functions

Definitely in there.

Runtime alteration of object or type system

No, not really. (I.e., maybe you can hack around with certain APIs to try to do some magic, but it's not a language feature.)

Functional Programming

Yes. But still, functions aren't really first class citizens…yet. Once we can start using method groups as Actions and Funcs, implicitly, then it'll get even better. This is an interesting presentation from Andrew Kennedy, Microsoft Research: C# is a functional programming language.

Closures

Yep, since anonymous methods were introduced in C# 2.0.

Continuations

Extremely limited, in the form of yield return.

Introspection

Not just reflection, but actually inspecting the actual code. C# 3.0 has this in the form of Expression<T> (see below).

Macros

No, not crappy C-style macros. Here, I'm thinking more like macros that'd let you create things like C# query comprehensions, *in source code*. (Which is what I was actually hoping when I saw the new query comprehension syntax… no such luck).

In summary, C# lets you gain a lot of benefits usually associated with dynamic programming, but without the nasty parts of dynamic typing.

A great paper on this subject is Static Typing Where Possible, Dynamic Typing When Needed: The End of the Cold War Between Programming Languages, by Erik Meijer and Peter Drayton of Microsoft.

Myth: Extension methods add methods to a class

This is a tricky one, since extension methods appear to be exactly that. This myth is also somewhat perpetuated by the C# spec: "In effect, extension methods make it possible to extend existing types and constructed types with additional methods." But, right before that, the real explanation is given: "Extension methods are static methods that can be invoked using instance method syntax."

Essentially, Extension methods allow us to use infixnotation with certain methods. This explains the line from the spec "in effect". A more helpful way to think of this is by thinking of the "." operator as an overloaded operator that also allows passing the first operand given to it as the first argument to specially marked methods.

An alternative* would be to define an operator like the F# pipeline operator (|>). In C#, this would let us write stuff like:

customers |> Seq.Where(c => c.Name == "Michael")

That doesn't look like an improvement. BUT, we no longer need to mark methods in a special way. We can just use them:

myArray |> Array.BinarySearch("s")

Why do we need infix notation anyways? Well, the normal prefix notation can be difficult to read:

Select(Where(customers, c => c.Cool == true), c => c.Name)
Array.BinarySearch(items, "S")

Extension methods just make those functions easier to pipeline. That's all folks. Think of them like this, and save yourself a headache about what "extending a type" means.

*My guess as to why extension methods are done they way they are is because it could confuse people if you have something like "item |> Stuff.SomeMethod("X")" where SomeMethod returns a function. Or, where you have "item |> Stuff.SomeMethod("X").SomethingElse("y"). I'm still annoyed that I can't use infix semantics where *I* want, but oh well.

Lambda expressions and Expression<T>

Spec: "Lambda expressions provide a more concise, functional syntax for writing anonymous methods.". Spec: "Expression trees permit lambda expressions to be represented as data structures instead of executable code. A lambda expression that is convertible to a delegate type D is also convertible to an expression tree of type System.Query.Expression<D>."

So, "lambda expressions" can either be just code (i.e., directly executable IL) OR they can get turned into a data structure, Expression<T>.

Adding to the confusion, a lambda expression can contain just an expression (i=> i + 1), or it can be a block of statements ( i => {Write(i); i++; Write(i); return i+1;} ). However, a lambda expression with a statement block body cannot become an Expression<T>. (As far as I know, VB's lambdas only allow for expression bodies, not blocks.)

An example:

Expression<Func<int, int>> inc = i => i + 1;

Is equivalent to:

ParameterExpression param_i = Expression.Parameter(typeof(int), "i");
var inc2 = Expression.Lambda<Func<int, int>>(
    Expression.Add(
        param_i, 
    Expression.Constant(1, typeof(int))),
param_i);

At runtime, you can then go inspect the actual code and decide what to do with it. This is exactly the premise for LINQ to SQL. When you create a LINQ-to-SQL query, it is turned into an expression like shown above. Then the LINQ-to-SQL APIs inspect and convert that expression tree into SQL statements.

Here, I can understand the confusion. The word "expression" is used in three distinct manners. Rightfully, Expression Trees should be referred to as Expression or Expression<T>, which could help clear up some of the confusion. Additionally, it doesn't help that lambdas have these different conversion rules (although working around it could be ugly, possibly).

Are there any other features that you've seen misused or you've had questions about? Let me know! I love comments, insults, and suggestions.

Want to correct me on something? Go right ahead! But, if you're going to say something like "C# doesn't have type inference", please make sure to either be an expert on the matter or be able to quote an authoritative source or show a proof. Thanks!

Don't skip parts I to III to get you up to speed. I have $300 up for grabs if these articles are incorrect in stating that it will greatly improve most C# apps:

    Practical Functional C# - Part I 
    Practical Functional C# - Part II 
    Practical Functional C# - Part III - Loops are Evil

Tell the compiler what you want, rather than how to do it. That's a key concept that can take you very far. However, as imperative programmers, this can sometimes be a hard concept. We are so used to instructing the processor, step-by-step, how do to things, and then only after we're all done, making sure the end effect is to our liking. Functional programming helps us change this.

Consider the following task: Write a function that checks a username against a few disallowed characters "!@#$%^&*". The normal C# implementation looks like this (null checks removed):

static bool IsBadUserName(string userName)
{
    var badChars = "!@#$%^&*";
    foreach (char c in badChars) {
        if (userName.Contains(c)) returntrue; 
    }
    returnfalse;
}

It's not horrible; there's only one branch, but it does require a bit of thought to sort out. Now let's take an approach where we deal with string as a set of characters to manipulate at once (requires C# 3.0 compiler):

static bool IsBadUserName2(string userName){ 
    var badChars = "!@#$%^&*";
    return userName
        .Intersect(badChars) 
        .Any();
}

We've reduced the function from a step-by-step loop into something that has only path. The bigger benefit is that there's no need to evaluate end conditions and so on for correctness: The code says exactly what it does. "Are there any bad characters in the user name?" We don't have to worry how it does what we asked, we just need to think in terms of what we want our result to be.

A common function in functional languages is called "map". Map takes a list of something and turns it into a list of something else. For instance, if we had a list of integers ("ourInts"), we could turn them into squares by saying "map ourInts by multiplying each value with itself". In C# (LINQ), they called map "Select". Here's a quick example:

var ourInts = new[] { 2, 5, 13 };
var squares = ourInts.Select(i => i * i);

Squares will contain the list { 4, 25, 169 }. What use is this? Well, it is an extremely common pattern to take some set of data, filter it, modify it a bit, and return a new set of data. Here's an example: You have a variable from containing semicolon-delimited email addresses. You want to turn these into an array of .NET's MailAddress objects to use with some other code. The loop isn't very pretty:

var tempAddresses = newList<MailAddress>();
foreach (string s in semicolonEmails.Split(';')) {
    var ts = s.Trim();
    if (ts == "") continue; 
    tempAddresses.Add(newMailAddress(ts)); 
}
var myAddresses = tempAddresses.ToArray();

But consider the functional approach:

var myAddresses = semicolonEmails
    .Split(';') 
    .Select(s => s.Trim())
    .Where(s => s != "") 
    .Select(s => new MailAddress(s))
    .ToArray();

In one statement, we transform the data three times, as well as add filter to remove empty items.

One place where LINQ falls flat on its face is when it comes to processing data. For some reason, there are no methods defined to do "ForEach" or "Process". (Even more interesting: List<T> does define these methods.) Process is a great pattern: on each item, it performs some action, then returns the original item. The code to define it is very simple and looks like this:

static IEnumerable<T> Process<T>(thisIEnumerable<T> source, Action<T> f)
{
    foreach (var item in source) {
        f(item); 
        yieldreturn item; 
    }
}

How is this of use? Well, let's chain together some functional and imperative processing. For instance, write a program that does some long process on all files passed in as arguments – on separate threads. If we take the purely imperative approach, our code looks like this:

static void Main(string[] args)
{
    var threads = newList<Thread>();
    foreach (var s in args) {
        Thread t = newThread(startLongProcess); 
        t.Start(s); 
        threads.Add(t); 
    }
    foreach (var t in threads) {
        t.Join();
    }
}
staticvoid startLongProcess(object data)
{
    SomeLongProcess((string)data);
}

Yes, we actually must declare a separate function just to invoke SomeLongProcess. Let's combine and use the functional approach now:

static void Main(string[] args)
{
    args
        .Select(s => newThread(() => SomeLongProcess(s))) 
        .Process(t => t.Start())
        .ToList()
        .ForEach(t => t.Join());
}

Which way is going to be easier to edit and change around? I don't know about you, but for me, going from ~12 to ~5 lines, removing extra variables, useless functions and flow control structures: that's a hands-down win in my book.

As a side note, threading, in fact, is a space that is extremely ripe for functional styles. I'm willing to bet that ".NET 4" will include threading extensions that rely heavily on functional concepts. For instance, it's easy to create a method that allows us to replace the previous program with this:

args.Parallel(SomeLongProcess);

But I'll talk about that another day.

In the next article, I'm going to cover C#'s new inner functions capability and how that can be used to help build up more complicated function chains. I'd also like some feedback on which kinds of areas of C# programming you've run into that seem to require more code than necessary.

A question many people have run into is: How does CRUD fit into LINQ-to-SQL? While LINQ-to-SQL (I'll abbreviate as DLINQ) provides a very fast and easy way for us to start querying our data, it doesn't handle updates as beautifully. It is particularly noticeable when you are doing multi-tier and hence cannot share a DataContext.

Let's consider an example database called "MembersDatabase" which has a table called "Accounts". Accounts has an int primary key, and a varcharEmail field. We use the DLINQ designer and create the dbml that generates a class called MembersDataContext. How does our app-tier code look? I'll give you a hint, it starts with "ug" and rhymes with "nasty":

// Select
int someId = 123; // Passed from another tier
Account someAccount; // Can't use implicit typing -- no anonymous types
using (var dc = newMembersDataContext()) {
    someAccount = dc.Accounts.SingleOrDefault(a => a.AccountId == someId);
}

// Insert
var myAccount = newAccount();
myAccount.Email = "me@contoso.com";
using (var dc = newMembersDataContext()) {
    dc.Accounts.Add(myAccount); 
    dc.SubmitChanges();
}

// Update
int myId = 1; // Id and email passed from another tier
string newEmail = "cool@cool.com";
var changedAccount = newAccount();
changedAccount.AccountId = myId;
changedAccount.Email = newEmail;
using (var dc = newMembersDataContext()) {
    dc.Accounts.Attach(changedAccount, true); 
    dc.SubmitChanges();
}

// Delete
int idToKill = 2; // Passed from another tier
using (var dc = newMembersDataContext()) {
    using (var txScope = new System.Transactions.TransactionScope()) {
        var acc = dc.Accounts.SingleOrDefault(a => a.AccountId == idToKill); 
        if (acc == null) thrownewChangeConflictException("Row not found."); 
        dc.Accounts.Remove(acc); 
        dc.SubmitChanges();
        txScope.Complete();
    }
}

It's not horrible; it's certainly better than anything before it. But, we can do better. I created some helper classes to do so (see attached file).

First, DatabaseBase<TContext>. This holds our tables, and provides DataContext helper functions Use and Query. Second, TableBase<TItem, TKey>. This actually provides our CRUD methods. I don't think anyone is overly interested in the implementation details (comment if I'm wrong), so here's how you declare your CRUD types:

class MembersDatabase : DatabaseBase<MembersDataContext>
{
    publicstaticreadonlyTableBase<Account, int> Accounts 
        = CreateTable(dc => dc.Accounts, a => a.AccountId);
}

That's it.

You create a new class to serve as your "database" class. All that's required here is to inherit from DatabaseBase.

Next, for each table, simply create a new field via CreateTable. The first parameter is a lambda function that selects the right table off the DataContext. The second parameter is a lambda expression that selects the primary key. Not much too it.

So, how does our previous chunk of code look with this small helper library?

// Select
int someId = 123; // Passed from another tier
var someAccount = MembersDatabase.Accounts.SelectByKey(someId);

// Insert
var myAccount = newAccount();
myAccount.Email = "me@contoso.com";
MembersDatabase.Accounts.Insert(myAccount);

// Update
int myId = 1; // Id and email passed from another tier
string newEmail = "me@me.com";
var changedAccount = newAccount();
changedAccount.AccountId = myId;
changedAccount.Email = newEmail;
MembersDatabase.Accounts.Update(changedAccount);

// Delete
int idToKill = 2; // Passed from another tier
MembersDatabase.Accounts.Delete(idToKill);

That's about 40% less code. It’s far more straightforward, being a single block.

To start using this code, just drop DatabaseBase.cs into your project. It adds DatabaseBase to System.Data.Linq. Then subclass as shown above, and you're on your way to LINQ updating bliss. What do you think?

DatabaseBase.cs (5.65 KB)

P.S. At any rate, I should get points for the Zelda pun, right?

Update: I forgot to mention, you'll want to turn UpdateCheck to Never on your columns in the LINQ-to-SQL designer.
Update: The code (including the Tuple class) for the .NET 3.5 RTM release is here: http://www.atrevido.net/blog/2008/06/26/LINQ+To+The+CRUD+RTM.aspx

What people do in their own time in the privacy of their homes is none of my business. However, when they mess with reading documentation, then it crosses the line and becomes annoying. How many times do VB developers need to be told that a null is "Nothing"? Consider this snippet from MSDN:

-------
The CreateUser method will return a null reference (Nothing in Visual Basic) if password is an empty string or a null reference (Nothing in Visual Basic), username is an empty string or a null reference (Nothing in Visual Basic) or contains a comma (,), passwordQuestion is not a null reference (Nothing in Visual Basic) and contains an empty string, or passwordAnswer is not a null reference (Nothing in Visual Basic) and contains an empty string.
-------

Five times in one paragraph! I know null type systems are annoying and lead to errors, but that seems a bit excessive. Seriously though, it'd make more sense to make VB developers learn a few words once, rather than having to mess up documentation just in case they get confused.

Be sure to read these first two articles:

    Practical Functional C# - Part I

    Practical Functional C# - Part II

OK let's start with a quick challenge. Write an accurate description of the following program, in English:

static void Main(string[] args)
{
    string output = "";
    if (args.Length > 0) output = args[0]; 
    if (args.Length > 1) {
        for (int i = 1; i < args.Length; i++) {
            output += ", " + args[i]; 
        }
    }
    Console.WriteLine(output);
}

Well? The specification might be something like "a program that writes its arguments separated by a comma and space". But how quickly could you determine that from the code? How much additional time does it take to determine there aren't any bugs? Every time a maintenance developer comes across this code, they have to analyze this code, determine the boundary conditions correctly, verify the indexing, and so on. Every time someone reads this code, she must pay a high tax. The saddest part is that this is an extremely common pattern.

Edit: As Chad Hower pointed out in the comments that you can remove the two if statements by performing a check inside the loop. That shortens it considerably and reduces some of the "tax" that has to be paid when you read it.

"Take this set of values and aggregate them into a single value". How many pieces of code do exactly this, but obscure it behind a for loop? Functional languages realize this and provide functions called "Fold" or "Reduce". Such functions take an accumulator function, apply it to every element in the list, then return the accumulator value when finished. C# 3.0, courtesy of LINQ, provides an equivalent function, called "Aggregate":

static void Main(string[] args)
{
    string output = args.DefaultIfEmpty("").Aggregate(
        (accum, item) => accum += ", " + item); 
    Console.WriteLine(output);
}

Here we are saying that we are going to aggregate args into a single string value. The lambda on the second line takes two arguments. The first is the accumulator and Aggregate passes it to each element ("threads" it through). The second parameter is the current item we are working with. The return value of our lambda is simply the concatenation of the current accumulated value, comma and space, and the current item. This return value becomes the accumulator for the next item. On the first execution, since we did not give it an explicit seed value, it just uses the first item. The final return value becomes the value that Aggregate returns to output. (Edited: Added DefaultIfEmpty -- this overload of Aggregate doesn't work on empty sequences.) Using the lambda provides nicer syntax than this equivalent code:

static void Main(string[] args)
{ 
    string output = args.DefaultIfEmpty("").Aggregate(joinCommaSpace); 
    Console.WriteLine(output);
}
static string joinCommaSpace(string a, string b)
{ 
    return a + ", " + b;
}

So why is this a good thing? Well, it goes back to the questions about the first set of code: how much effort is required to determine intent and correctness of a particular piece of code? In the imperative way, we need eight lines of code, with three distinct paths. Using a functional approach, we have three lines of code and only one code path. The only serious objection that I've heard is that this code is "unfamiliar". Well, sure, anything new might be unfamiliar, but that does not make it bad.

You wouldn't write your SQL code to make someone only familiar with C "comfortable" or "familiar", would you? You wouldn't write in C# the same way you'd write in C or BASIC. So why stick with outdated programming practices just because some "new" developer might get confused? Functional code is more concise, less error prone, and much more readable. Learning this style might take a couple of days, but it's an invaluable skill. At any rate, LINQ is built upon these concepts, so it will do people good to learn anyways.

Now, let's examine how to create our own functions to hide loops. This time, we're going to look at data access. A very common pattern in data access is creating a SqlDataReader, going through it, adding elements to a list. Like other patterns, overhead obscures the intent:

public static List<Person> GetAllPeople()
{
    // Setup command
    var comm = newSqlCommand("GetAllPeople");
    comm.CommandType = CommandType.StoredProcedure;

    // Setup connection
    using (var conn = newSqlConnection(Settings.ConnectionString)) {
        comm.Connection = conn; 
        conn.Open();

        // Loop and add people to our list
        using (var reader = comm.ExecuteReader()) {
            var people = newList<Person>();
            while (reader.Read()) {
                var p = newPerson();
                p.Name = reader.GetString(0); 
                p.Age = reader.GetInt32(1); 
                people.Add(p); 
            }
            // Done
            return people; 
        }
    }
}

Yes, something as simple as reading a list of two-field type can take 14 lines of code. Let's do something about that. The only unique part is where we create a Person from the SqlDataReader. Outside of that, it's a very straightforward, but large, pattern. Refactoring the pattern, we get this:

public static List<T> ListFromReader<T>(string connectionString,
SqlCommand command, Func<SqlDataReader, T> code)
{
    // Setup connection
    using (var conn = newSqlConnection(connectionString)) {
        command.Connection = conn; 
        conn.Open();

        // Loop into list
        using (var reader = command.ExecuteReader()) {
            var list = newList<T>();
            while (reader.Read()) {
                // Here we call the supplied code to add the right item
                T item = code(reader); 
                list.Add(item); 
            }
            return list; 
        }
    }
}

Now our code to GetAllPeople is very simple:

public static List<Person> GetAllPeople2()
{
    var comm = newSqlCommand("GetAllPeople");
    comm.CommandType = CommandType.StoredProcedure;

    return ListFromReader(Settings.ConnectionString, comm, 
        reader => newPerson {
            Name = reader.GetString(0), 
            Age = reader.GetInt32(1) 
        });
}

Just be looking at this code, we know all the data-API stuff is handled correctly. This approach is vastly superior to other common SQL "helpers", for example, an ExecuteReader method that gives us a SqlDataReader. First, we have no locals that need disposing or other cleanup: this function scopes the variables to our lambda. Second, we can focus on our actual logic (creating a Person) rather than dealing with loop conditions.

Edit: This is not specific to C# 3.0! You can definately achieve a lot of the same benefits in 2.0, except you need to replace the simple lambda syntax ( => ) with the much more verbose delegate (ArgType arg) { } (anonymous method) syntax. C# 3.0 just makes it much easier to write.

What other common loops do you encounter? I have a few more we'll cover in the next article. As always, comments, insults and suggestions are welcome.

Previous article: Practical Functional C# - Part I

Last time I demonstrated how to replicate the using keyword as a function that takes a function. In this article, I'm going to show some real-world cases where you'll see a major improvement by taking a more functional approach. I'm going to use WCF as an example space.

WCF allows us to define services as normal C# interfaces. We then use a factory to create a proxy for an interface, specifying the URI, binding types (HTTP, binary, message queue), and other options. However, our interest is how this actually looks from the client side; how we actually make calls.

Since a WCF call is actually invoking code on another machine, any number of bad things can happen to our client channel, resulting in exceptions. When this happens, we have to Abort the channel. But if things complete successfully, we just need to Close the channel. Of course, that's not easy enough: if the Close fails, we then need to Abort the channel anyways. Confused? Well, check out this code snippet that uses an interface called ICalculator to add two numbers:

void WcfExample()
{
    int a = 1; 
    int b = 2; 
    int sum; 
    var chanFactory = GetCachedFactory<ICalculator>();
    ICalculator calc = chanFactory.CreateChannel();
    bool error = true; 
    try {
        sum = calc.Add(a, b); 
        ((IClientChannel)calc).Close();
        error = false; 
    }
    finally {
        if (error) {
            ((IClientChannel)calc).Abort();
        }
    }
    Console.WriteLine(sum);
}

Ouch! Out of 17 lines of code, only five relate to our problem. The rest (70%) is pure, ugly overhead. This is a much more complicated pattern than the using pattern, and from experience, I can tell you it is error-prone. Fortunately, this pattern decomposes nicely. The only two unique things are the name of the interface and the statement that acts on the interface. Here is one way you might go about writing this generically:

TReturn UseService<TChannel, TReturn>(Func<TChannel, TReturn> code)
{
    var chanFactory = GetCachedFactory<TChannel>();
    TChannel channel = chanFactory.CreateChannel();
    bool error = true; 
    try {
        TReturn result = code(channel); 
        ((IClientChannel)channel).Close();
        error = false; 
        return result; 
    }
    finally {
        if (error) {
            ((IClientChannel)channel).Abort();
        }
    }
}

This is exactly like the previous code, but we've substituted TChannel and a function parameter named code instead of our actual types. The type of function we want is Func<TChannel, TReturn>. You can think of this as saying: "transforms a TChannel into a TReturn". Now, look at the beauty this allows on the client side:

void WcfExample2()
{
    int a = 1; 
    int b = 2; 
    int sum = UseService((ICalculator calc) => 
        calc.Add(a, b)); 
    Console.WriteLine(sum);
}

Presto! Our overhead went from 12 lines to zero. If that didn't sell you on the power of functions, I suggest finding another career. Now, there is a lambda there, but all it is saying is "Here is a function that takes one parameter of type ICalculator and returns the value of Add(a, b)." The C# compiler automatically infers the return type (int).

Some of you mentioned still using C# 2.0. Two things: First, you can use the 3.0 compiler and run on 2.0 because these examples don't reference any new assemblies. Second, you can achieve the same thing using anonymous methods, but they just look a bit uglier:

void WcfExample3()
{
    // C# 2.0 example with anonymous methods
    int a = 1; 
    int b = 2; 
    int sum = UseService<ICalculator,int>(delegate(ICalculator calc) {
        return calc.Add(a, b); 
    });
    Console.WriteLine(sum);
}

By this time, I hope you are thinking about places in your code where you can extract a larger pattern and use functions instead. The key concept to understand here is that your source code should reflect your solution. You should not have extra code sitting around just for the sake of appeasing the platform or meeting some "design pattern". Let the platform and libraries take care of the details, and let your code focus on actually solving problems. As always, I welcome your comments, suggestions, and insults, so please let me know what you think!

Next up, we'll take a look at loops, see why they are harmful, and then go about fixing them.

C# with lambda syntax and extension methods (in lieu of compositional operators) gets us so far, but the syntax and compiler could use a bit of polish. I'll show some exact examples later, but meanwhile this picture sums up my feelings at the moment:

Edited to add: Inspired by http://xkcd.com/297/

Edited to add: Just to be clear, this isn't a compiler or IDE issue: it's a spec issue (AFAIK). The C# spec simply doesn't allow certain things (like type inference from a method group).

My friend just dealt with an oursourced project. Yes, outsourced as in sending it to a place that charges a lot less money than, say, developers who actually know what they're doing.

One of their deliverables was a program that compressed an XML string into a gzip file. Should be a minor thing, right? The C#/.NET 2 code is less than 10 lines. Well, their first delivery produced files that contained the text "System.Byte[]". This was not accepted and they vowed to look into it in more as they were sure the code was correct.

Their next files were a bit larger and supposedly were really correct this time. But still, no zip program could read the data. Well, a quick look at the beginning of the file shows the bytes EF BB BF -- the UTF-8 BOM. The rest of the file was ASCII digits. Yes, they wrote the bytestream out as a UTF-8 interpretation.

If we define evolution as "the non-random survival of randomly mutating replicators", we can define their approach as "the non-random acceptance of randomly mutating programs."

Well, after almost exactly 6 hours, I've succeeded at installing SQL Server 2005 Reporting Services on a server with more than one website.

We're running Reporting Services on separate web servers. So, after the install of reporting services, you run their little configuration tool. This of course, accomplishes very little :). See, apparently Reporting Services wasn't designed to work on a server running, *gasp*, more than one application.

If you have a decent IIS install, the default website isn't there and thus requests to http://localhost/ aren't gonna work. Reporting Services doesn't take this into consideration, and happily tries to request http://localhost/ReportServer/ even after you've specified this in the config tool. If this is your issue, you'll get a “HTTP Error 400: Bad Request“ when trying to access the ReportManager (/Reports/) website.

You'll need to edit the config files in Program Files\.....\ReportManager and ReportServer. rsreportserver.config needs to point to http://the.reporting.host.name/ReportServer in the UrlRoot element. In RSWebApplication.config, ReportServerUrl will need to have the same value. The ReportServerVirtualDirectory element must be deleted. You will get a “The configuration file contains an element that is not valid. The ReportServerUrl element is not a configuration file element. “ message. This is because the config reading code apparently doesn't fail gracefully. What it's trying to say is “the ReportServerUrl and ReportServerVirtualDirectory elements are mutually exclusive”. I'm still unsure why there should be anything besides a URL...

Around here, you might notice a bunch of DCOM errors in your Event Log (or before this point). To fix these, you'll need to go into dcomcnfg and edit the COM security for My Computer. Give the account you're using (like Network Service or “MyReportingServicesAccount“) permissions for local activation and local launch. You need to reboot for these changes to take effect (I think). But don't reboot just yet...

Finally, you end up with a 401 Unauthorized when accessing the Reports site. You might have also noticed you are also unable to authenticate when browsing the Reports or ReportServer sites from your the local server. Why?
“Windows XP SP2 and Windows Server 2003 SP1 include a loopback check security feature that is designed to help prevent reflection attacks on your computer. Therefore, authentication fails if the FQDN or the custom host header that you use does not match the local computer name.” So I'm guessing NTLM susceptible to this type of attack, and Microsoft is saving us from it. Well, it also hoses us in this case because from what I can tell, ReportManager (the thing in the Reports vdir -- why it wasn't called ReportManager by default...) needs to connect to ReportServer. It sends a request, which is denied because of the loopback protection above. A quick registry edit fixes this: http://support.microsoft.com/default.aspx?scid=kb;en-us;896861

After that... you might have a working SQL Reporting Services 2005 install! (Next up: Getting it to work with SSL...)

Really, apart from the horrible setup/configuration, it's a very very fine product. I'm actually pretty impressed. The report I wanted to setup (and the subscription so it's mailed out) only took about 10 minutes (first time I've ever used RS)! I'm just at a loss why Microsoft makes it so hard to setup. This configuration can't be that unusual. And, even stranger, most (if not all) of this configuration issues could take care of these problems. In other words, their little configuration app should automatically fix this stuff (or at least give explicit instructions on how to do so). Or maybe I just didn't RTFM that well... but this is a Microsoft product... you're supposed to just shove the DVD in the drive and click next, right? <g>

P.S., if you're getting a “Object Reference not set to an Instance of an Object“ when you add a new subscription, ensuring everything else is 100% working should make it go away...

http://webproject.scottgu.com/Default.aspx

YEY!!! Finally, after 2 years, we get VS2003 functionality back in VS2005. The biggest pain point I have, every single day, is dealing with the vile VS2005 web sites. Microsoft has finally realised that this monstrosity spawned to soothe the demented minds of webmonkies who think HTML is a programming language is actually bad for real developers. Really, how does catering to the people who think “build“  refers to writing code help them? (Oh wait, I know. It allows them to gain access to “web developers“ while knowing that people who know what they are doing still won't defect.)

This is exactly like I predicted -- Microsoft will have to back down. I guess it took the final RTM launch for everyone to try to upgrade their apps and then find out that the new model is unusable. I wonder how many PSS cases/pieces of feedback they got.

Well, I can't wait to change our projects to this format. I was planning on restructuring our solution (22 projects in VS is unwieldly) anyways, and this will go great with it. I know a few developers who are gonna be real happy when they come back after the holidays! Maybe this means that doing a public refactoring in our solution won't take 45 seconds anymore? Maybe these apps will build in less than a minute (like every other C# app)?

Oh, BTW, I'm far from ungrateful, even though I might sound like that. I'm actually very happy that the ASP.NET team is doing this, despite the fact that I might roll my eyes and say “yea, about time!” :). But after going through all the pain I have... perhaps its understandable.

I was working on an application today, and I needed to add some data to every HyperLink on the ASP.NET page (for a custom authorization string). I thought it might be a common thing: needing to go through all the controls on a page, but apparently not. I didn't find any framework functionality, and the only code samples (just to see if I have the “best” way of doing things) led to some not-so-nice code (arraylists and recursion!). So, here's the best I've come up with (criticism, please):

        Stack<Control> remainingControls =newStack<Control>();
        remainingControls.Push(this);
        do
        {
            Control currentControl = remainingControls.Pop();
            foreach (Control item in currentControl.Controls)
            {
                if (item isHyperLink)
                {
                    HyperLink hl = (HyperLink)item;
                    hl.NavigateUrl = AddAuthToUrl(hl.NavigateUrl);
                }
                elseif (item.Controls.Count > 0)
                {
                    remainingControls.Push(item);
                }
            }
        } while (remainingControls.Count != 0);

I've spent a few hours trying to get the secure TCP (based on NegotiateStream) integrated security in .NET 2.0 working. While there is a page on this (Authentication with the TCP Channel), it fails to mention that you need one more property in addition to encrypt, impersonationLevel and authenticationMode. It's called “secure”, and it must be “true”. I didn't see it mentioned anywhere, except when I happened to browse the MSDN Forums: http://forums.microsoft.com/msdn/ShowPost.aspx?PostID=55225

I looked at his config, and realised I didn't have this “secure” property. Problem solved. Also, I recommend checking out http://pluralsight.com/wiki/default.aspx/Keith.GuideBook/HowToAddCIAToDotNETRemoting.html, which has a lot of information about Windows security in general, apart from some specifics of remoting and Kerberos. And, finally, yes, there's one more page where the secure attribute is listed (with some other docs) http://blogs.msdn.com/manishg/archive/2005/04/22/410879.aspx

OK, so perhaps there was some error between the user and the keyboard... but I'm very very excited to see this feature running.

I found a cool C# library for use with Asterisk (AGI) and .NET: MONO-TONE. It looks promising as an easy way to deal with AGI from C#. I think that I'll be extending it to support FastAGI as well, and contributing the changes back. Nice work Gabriel!

I just used the coolest feature in a while: VS2005's Call Browser.

I'm currently working on some firmware for IP phones and adapters. The chip is the Intel 8051, as used by Centrality Communications in the PA168. It's actually quite fun, programming for an 8-bit system. Apart from writing in C (instead of such high-level things like C#), writing for embedded systems like this adds its own interesting things, like having to decide where a variable will be stored (I think there is 3 different storage locations a variable can have with the 8051). Oh yea, and having to keep things really small, and really fast.

At any rate, I am not that familiar with the entire design of the system, and I just want to focus on adding features to the IAX2 implementation (cause SIP sucks!). A large part of my work is to figure out how things work. Having the Goto Definition (F12) is great for finding specific symbols, but doesn't help with the flow. Up until now, I've been Finding in Files for a specific method, then chasing things down by recursively Finding in Files until I figured out how things are called. This happens a lot, since these devices support 5 protocols and use #ifdefs and generic calls to interface with the different protocols. Add to that 8MB of source, and it's no small task.

This morning, I remembered I had seen a “Call Browser” window in Visual Studio 2005 a while ago. Edit: Apparently this isn't a new feature and has been around since VC6 (at least). Well, it's new to ME, and it's still very cool.

Here's an example. Let's say I want to add attended transfer (where you have a call, press transfer, dial a number, talk to the new call, then hangup to connect the two). I'm looking in the source I'm familar with (the IAX protocol area), and see iax2_hangup(). That's a packet-level call, so when someone physically hangs up, that, somehow, gets called. Where? Well... right click the function, Call Browser -> Show Calls To:


Click... click... click... bingo. Now I've got a complete grasp on the call flow that would send a IAX_COMMAND_HANGUP. My old way (which makes me feel stupid now) of browsing source doesn't even come close. A lot of programming these days is managing complexity. It's all about making the best use of our limited brainspace (some more limited than others). 17 lines/1 small diagram. That's all it takes for me to see this complete flow. How much mental capacity does that require versus browsing multiple locations in 4 different source code files?

I'm doing my own realtime support for Asterisk, in an attempt to make it scale. Asterisk is nice software, but straight out-of-CVS, the performance for high volume (say, over 20,000 clients) sucks. There are also other inconviences with using a file-based store to determine how to route calls. Mainly, it's inflexible and hard to achieve high-perf when everything is based on a large .conf file. Not to mention that Asterisk uses linked lists for everything so looking up any user is an O(N) op (and parsing the users file is O(N*N) by default!). So, I'm going to put my own logic as a replacement for some of the critical parts.

One of my concerns was performance. Since I'll have multiple Asterisk clusters banging on my .NET code (via SOAP), I wanted to ensure the whole end-to-end process was fast enough.

I used gSOAP to create the C code on Linux. gSOAP is seriously nice. At least an order of magnitude easier to use than I expected any SOAP library that works on Linux would be.

I created a simple test. I made a database with phone numbers and codecs. The idea is that when an incoming call comes in, Asterisk will use my code to SOAP over to my Windows machines, get the data, and then go on its merry way.

My Asterisk machine is a P4 2.4GHz, 512MB RAM (but, I have a Gnome session running on it). My Windows XP machine (I tested against my desktop) is a P4 3GHz HT, 1.5GB RAM. I'm running ASP.NET 2 Beta 1 and SQL Server 2000.

The test program consists of a loop (count 5000) that generates a random number, then uses gSOAP to ask for the codec for that number. Simple, tight.

The results on Linux are particularly impressive. Each instance of the test app only used a max of 4% processor, and under 1MB of RAM. The bottleneck was definately inside ASP.NET. To simulate more load from other machines in a cluster, I ran 1, 2, and 4 instances of the test program. Also note that background tasks on the XP machine used up about 10% of the CPU.

Results:
 Single process (5000 total requests): 
  Total time:                   18 seconds (0.0036s/request)
  Requests per second:   277
  ASP.NET/IIS CPU:          30%
  SQL Server CPU:            4%

 Dual process (10,000 total requests):
  Total time:                   23 seconds (0.0048s/request)
  Requests per second:   384
  ASP.NET/IIS CPU:          60%
  SQL Server CPU:            7%

 Quad process (20,000 total requests):
  Total time:                   42 seconds (0.0052s/request)
  Requests per second:   476
  ASP.NET/IIS CPU:          80%
  SQL Server CPU:           10%

These results are encouraging enough that I'm not worried of the performance impact of using SOAP with Asterisk. My target was to have a response in less than 0.1 seconds. Although, anything under 0.5s would be quite unnoticable to a client. Even in tests with more threads, my single request response time was always way under 0.1 seconds.

Also, as far as I know, Whidbey Beta 2 (the version I'll go live with) makes some performance improvements. And also, IIS6 on Windows 2003 is much faster than IIS5.1 on XP. At any rate, a single proc desktop machine serving 476 RPS? That's pretty damn good perf if you ask me!