Logo




Subscribe:
RSS 2.0 | Atom 1.0
Categories:

Sign In


[Giagnocavo]Michael::Write()

# Thursday, August 28, 2008
Tsk, tsk, Silverlight - Events are not async's friend
OK, so Silverlight 2 is still in Beta 2 and hopefully will have time to change. But I would have thought this fix would have gotten into Beta 2 (as far as I can tell, it has not). At any rate, it applies to async design in general.

Scenario: You are writing an async method and need to call code when your async process finishes. How do you expose this to your caller? For some reason (I'd love to know what it is), it seems to be getting more popular to expose an event to accomplish this.

Events suck: Let's use a simple case. You want to get two URLs, using the return of one to get the other, then print the result to the console. You have a few local variables you need to use as well. Let's see what it'd look like without async:

// Inside some method:
var url1 = Console.ReadLine();
var url2 = Console.ReadLine();
var someData = Console.ReadLine();
// onEx = some exception handler
try {
    var res1 = webThingy.Download(url1);
    var res2 = webThingy.Download(url2 + "?data=" + res1.Data);
    Console.WriteLine(someData + res2);
} catch (Exception ex) {
    onEx(ex);
}

Straightforward, eh? Now, with async + events? You _could_ go create a new object type and add all sorts of fields and methods and whatnot, but that's a lot of work and gets ugly quickly. Closures are a natural help here. So how does the event-based async code look?

// Inside some method:
var url1 = Console.ReadLine();
var url2 = Console.ReadLine();
var someData = Console.ReadLine();
// onEx = some exception handler
OnDownloadCompleteEventHandler first;
webThingy.OnDownloadComplete += first = (o, res1) => {
    if (res1.Exception != null) {
        onEx(res1.Exception);
        return;
    }
    webThingy.OnDownloadComplete -= first;
    webThingy.OnDownloadComplete += (o2, res2) => {
        if (res2.Exception != null) {
            onEx(res2.Exception);
            return;
        }
        try {
            Console.WriteLine(someData + res2.Data);
        } catch (Exception ex) { onEx(ex); }
    }
    try {
        webThingy.DownloadAsync(url2 + "?data=" + res1.Data);
    } catch (Exception ex) { onEx(ex); }
};
try {
    webThingy.DownloadAsync(url1);
} catch (Exception ex) { onEx(ex); }


I think this judges itself.

Better
The way it SHOULD be is that any async method should take two arguments, one to call for result, one for exception. Let's see how that would look:

// Inside some method:
var url1 = Console.ReadLine();
var url2 = Console.ReadLine();
var someData = Console.ReadLine();
// onEx = some exception handler
webThingy.DownloadAsync(url1, onEx, res1 => {
    webThingy.DownloadAsync(url2 + ?data=" + res1, onEx, res2 => {
        Console.WriteLine(someData + res2);
    });
});

This code isn't perfect, but it's sure a ton better than the event-based system. With a bit extra work, you could build a simple async framework. Every Async method could return an Async object that would allow you to consolidate stuff like exception handling and cancellation. But even without that, this code straight away is much superior.

The extra downside
Interestingly enough, .NET 1 had the concept of BeginXXX/EndXXX, but because there were no closures, it was always a bit more of a pain to implement. BeginXXX/EndXXX, while not making exception handling as easy, are at least a good start.

The cool thing about BeginXXX/EndXXX was that you could refactor them generically into nice async syntax. Heres a quick and dirty example [F#'s async stuff works similarly.]:

static Action<A1, Action<Exception>, Action<R>> ToAsync<A1, R>(
    Func<A1, AsyncCallback, object, IAsyncResult> begin,
    Func<IAsyncResult, R> end) {
    return (arg, onEx, cont) => {
        begin(arg, iar => {
            try { cont(end(iar)); } catch (Exception ex) { onEx(ex); }
        }, null);
    };
}
static void BeginAcceptAsync(this Socket s, int timeout,
    Action<Exception> onEx, Action<Socket> cont) {
    ToAsync<int, Socket>(s.BeginAccept, s.EndAccept)(timeout, onEx, cont);
}
static void Main() {
    Socket s = ...;
    s.BeginAcceptAsync(10,
        ex => Console.WriteLine(ex),
        sock => sock.Close());
}

(Yes, I think this is a legitimate use of type extensions, but again, only because the original library had a design flaw. And even the non-extension syntax wouldn't necesarily be bad.)

But, to my knowledge, since there's no way to reference an event, this is not possible with the async + event approach t hat is becoming all the rage (yes, SocketAsyncEventArgs, I'm talking about you too). If you know of a way to ease the pain of async+events, tell me.

In summary
It seems that events are a bad choice for code that is not loosely coupled, such as UIs. Even when loosely coupled, sometimes a simple delegate field would be a better choice, since you can compose (like tacking on a filter or otherwise augmenting the callback). But in the case of async, I cannot see how it is good.

Code
Thursday, August 28, 2008 1:48:31 AM UTC  #    Comments [1]  |  Trackback

Why are there still delegates?
Can someone explain to me the point of having delegate types in C#/.NET? They made sense in the non-generic .NET 1 world, but with generics they should have no reason to exist (except maybe purely as a type alias). All the delegates in common use are representable with a generic definition (think Action/Func<...>). But for some reason, you cannot automatically use an equivalent delegate in C#.

This manifests a bit more when you consider a local such as:

var inc = (int x) => i + 1;

As Expression<T> syntax takes the same form, the compiler cannot tell if it's an expression or a delegate. Additionally, since Func<int,int> is "just as good" as any other generic type, even if the compiler wanted to create a delegate, it wouldn't know to use Func<int,int>. Thus, you must specify it, and inner functions are too clunky to be used in C#.

The answer I got from Microsoft (not an official answer, just a response to a Connect item) was that C# might allow auto-conversion in version 4, but also that delegates aren't totally useless because recursive delegate types can't be declared in that fashion. But for the relatively few uses (fixed-point combinators?) of such types, built-in, specific support would be a small sacrifice, wouldn't it?

This is just something I've been wondering about.

Update: Thanks for Jonathan Pryor for pointing out in the comments what my limited functional mind missed: ref/out parameters and of course, and high-arity delegates. While these should be easy to work around, they are accurate technical reasons. Thanks.

Code
Thursday, August 28, 2008 12:18:44 AM UTC  #    Comments [11]  |  Trackback

# Wednesday, August 27, 2008
ASP.NET MVC Begs for Tuples

[Yea, I’m not a web dev, and actively avoid it as much as possible, so I’m late to this party.]

The interesting thing about ASP.NET MVC is that it takes an opposite approach to ASP.NET in general. ASP.NET concepts try to “build up”, so as to shelter us from the evolved idiocy that is HTML, as well as clean up the inherently stateless nature of HTTP. ASP.NET MVC makes no attempt and forces you to deal with reality. Considering ASP.NET’s abstraction isn’t really perfect (example: databinding sucks), MVC’s approach is unfortunately refreshing.

Because of its “raw” nature, you’ll be writing a lot more HTML than you’d do with ASP.NET, and this HTML must line up with code on your server. To make this less of a pain, there are some HTML helper functions. Rob “ type inference” Conery has an overview here.

Here’s the signature for one of the functions:

    public static string CheckBox(this HtmlHelper helper, string htmlName, string text, string value, bool isChecked, object htmlAttributes);

The last parameter confused me. Why would it be an object? Am I supposed to pass in an IDictionary<string,string>? Just a long string? To make it more confusing, other helpers had two overloads:

    public static string TextArea(this HtmlHelper helper, string htmlName, object value, IDictionary<string, object> htmlAttributes);

    public static string TextArea(this HtmlHelper helper, string htmlName, object value, object htmlAttributes);

OK, so they explicitly called out the IDictionary there – THEN WHO WAS OBJECT HTMLATTRIBUTES?

Rob covers in his overview. The idea is that you’re supposed to use anonymous types to hack around the lack of tuples.

<%=Html.Whatever(arg1, bla, …,
                             new { @class=”cssx”, style=”x:f” …}) %>

What a great case of not-having-built-in-tuples-is-lame. It’s so lame, Microsoft’s own developer teams have to resort to weird (but quite creative!) hacks like this so that their syntax won’t completely suck*. Damn.

And now, a duck

For bonus points, there is another C# compiler feature that the MVC team could have [ab]used, and it would arguably have made more sense (although the syntax isn’t as tight). C# supports duck typing on collection initializers! So, they could create a class like this:

    public class HtmlAttributes : System.Collections.IEnumerable

    {

        public System.Collections.IEnumerator GetEnumerator() { ... }

        public void Add(string name, object val) { ... }

    }

And then they can write this:

new HtmlAttributes { { "A", 123} , {"B", "test"} }

No, it’s not as tight as the anonymous type syntax, but it does make a lot more sense.  And tuples still make much more sense than either approach, and have benefits for the rest of the language to boot (death to out parameters!).

 *The only benefit I see in anonymous types is that, at compile time, you'll  know there are no key conflicts - but that is totally trivial in the way they use them, since all the keys are declared right there.

ASP.NET | Code
Wednesday, August 27, 2008 1:12:13 AM UTC  #    Comments [10]  |  Trackback

Extension Methods Suck
Been meaning to write this for a while, and I think I touched on it here, but I'd like to expand a bit. C# Extension Methods are just a hack to compensate for C#'s poor handling of functions in general. As far as I can tell, they were added solely so you can do this type of thing in LINQ:

  var squares = myInts.Select(i => i * i)

Instead of:

  var squares = Enumerable.Select(myInts, i => i * i)

In other words, they wanted to provide some simple infix syntax for functions. Well, that's a poor approach to the problem.

    - First off, it only works on function specially declared to be "extensions", which means your composition options are limited to whatever the library has built in. I can't send arguments to arbitrary static methods, like, say, "someVar.Console.WriteLine".

    - Second, since extension methods are defined solely by their method name, ambiguity is quite easy to come across. There's no way to qualify Foo.Function versus Bar.Function.

Pipeline                                                       
Other languages approach this with two simple things (which actually simplify the entire language/type system overall). First off, we need to be able to define function operators. I'll demonstrate with F#'s pipeline operator:

    let (|>) x f = f x

[As a side note, any language that lets you toss around operators and functions will allow this kind of syntax – it isn’t that F# had to have compiler support for this particular operator.]

In fake C#, it’d be something like (for fun, notice the lack of type inference):

B operator|> <A, B>(A x, Func<A, B> f) { return f(x); }

This means that the |> operator will take x on the left and apply it to f on the right side. If this existed in C#, you'd be able to write something like:

    "Hello" |> Console.WriteLine
or
    if (myInts |> Enumerable.Any) { .... }

Now we can pass in a parameter to static methods. But, hey, whaddya know? With this, Extension Methods are solved for all single-argument static methods! That was easy. But what about Select – it takes two parameters, so this won’t work.

Enter the Lambda
What if ALL functions took one parameter and output one parameter? If that were the case, then we’d be set. But how do allow more than one parameter? Well, what if, every time you declared a method with more than one parameter, it actually returned a method that took the next parameter? For example, we could write “Add” as:

        Func<int,int> Add(Func<int,int> a) { return b =>  a + b; }

This is known as the curried form of Add. We’d now call it as: Add(5)(6). We can do cute stuff like “var inc = Add(1)”. But, as the Add declaration shows, in C# this is too unwieldy (and this is a simple example!). The compiler should actually do all this for us, so we can just write our functions normally but use them as if they were written in curried form.

Now, if we simply swap the order of arguments for Enumerable.Select, we have our extension method ready to roll: Enumerable.Select(Func, IEnumerable) can be used so:

var squares = myInts |> Enumerable.Select(i => i * i)

Now the call to Select takes the lambda (i * i) and returns a function that takes an Enumerable. It is then given the myInts, and everyone is happy. This is just a quick, crap, explanation. Google can lead you to many more interesting resources about partial application, currying and so on.

At any rate, I think if C# had taken this approach, we'd all be much better off. To top it off, more functions would take their arguments in a proper style. As it is, uncurried versions of Extension Methods are incompatible with normal function pipelining. Oh well.


Code | FSharp
Wednesday, August 27, 2008 12:17:53 AM UTC  #    Comments [15]  |  Trackback

# Tuesday, August 26, 2008
ASP.NET MVC - Abusing Using (At VB's request?)
Someone on our team started using ASP.NET MVC for a new web interface we're doing. I must say I'm impressed with the level that the MVC team [ab]uses the C# compiler, mostly in a good way. On the plus side, they end up with a bit more compiler time checking than would be possible otherwise (we can only hope WPF will follow suit some day).

But one thing struck me odd was how their helper method for generating an HTML form works. The goal is to generate an HTML form tag with the right action, and they use lambdas as symbolic references to figure out the action. The next problem is making sure that the <form> gets closed with a </form>. The straightforward answer to this is "create a function that takes a function". The C# signature would be: void Form<A>(Expression<Action<A>>, Action). Then your ASPX code would be:

<% Html.Form<FooController>(x => x.Edit(someVar.FieldX), () => { %>
    Some Html <% SomeCode%>
<% }); %>

The code is nicely bracketed and works fine. But ASP.NET MVC doesn't actually do that. Instead, the Form method returns an IDisposable! The code to use it is:

<% using (Html.Form<FooController>(x => x.Edit(someVar.FieldX)) { %>
    Some Html <% SomeCode%>
<% } %>

Why do they use an IDisposable? The rest of the MVC framework seems to assumes people are somewhat familiar with lambdas, closures and what not. The only thing I can think of is that VB doesn't support anonymous methods. So in order to make it VB friendly, they come up with quite a strange use of IDisposable to abuse language support for it. Overall, I'm not sure if this is dumb or cute.

Code | ASP.NET
Tuesday, August 26, 2008 9:19:59 PM UTC  #    Comments [4]  |  Trackback

# Saturday, August 16, 2008
Reference cells in F# sequences
I saw this stack implementation (http://blogs.msdn.com/jaredpar/archive/2008/08/15/immutablestack-in-f.aspx) and wanted to see how I'd approach it:
 
#light
open System

type 'a ImmutableStack =
   | Empty
   | Stack
of 'a * 'a ImmutableStack
  
member x.Peek() = match x with | Stack (v, _) -> Some v | _ -> None
  
member x.Pop() = match x with | Stack (_, s) -> s | _ -> failwith "empty stack"
   member x.Push a = Stack(a, x)
  
member x.IsEmpty = x = Empty
 
(Well, I'd probably just use a list in most cases.) But interestingly, he has an All function, which returns a sequence of the entire stack. Chris suggested using sequence expressions and then recursively calling the function:
 
member x.All =
    match data with
        | Empty
-> Seq.empty
        | Value (v,n)
->
            seq {
               
yield v
               
yield! n.All }
 
This is pretty nice as far as syntax goes. For some reason, I wanted to see how it'd look without the recursion (which needs to generate a new enumerator, unless the compiler is doing some wicked awesome stuff, which wouldn't surprise me). Here are some of the things I came up with:
 
member x.All1 = 
  
let until f = Seq.generate (fun () -> ref x) f (fun _ -> ())
   until (
fun cur -> match (!cur) with 
                        | Stack(v, s)
-> cur := s
                                         Some v 
                        | _
-> None)
 
Seq.generate continues calling the function until None is returned, so this provides the loop termination. Another similar approach:
 
member x.All2 =
      
let until f = Seq.generate (fun () -> ref x) f (fun _ -> ())
      until (
fun cur -> try (!cur).Peek()
                        
finally if not (!cur).IsEmpty then cur := (!cur).Pop())

Since Peek already returns 'a option, we can use it directly. The problem is that we need to update cur to cur.Pop
and then return a value. The try/finally works, but the whole deal still doesn't seem very elegant. Sequence expressions allow us to yield:
 
member x.All3 =
      
let cur = ref x
      {
while not (!cur).IsEmpty do
          yield (!cur).Peek()
          
do cur := (!cur).Pop() }
      |> Seq.choose(
fun x -> x)

I dislike this one. Because we're not doing the pattern match ourselves, we end up yielding 'a option. But None is never valid because we guard on IsEmpty. This makes us stick a Seq.choose on with an identity function to strip off the Some. [Side note, is there no built-in identity function?] Bringing the match into the seq fixes the issue, but the code is pretty long:

member x.All4 =
      
let cur = ref x
      {
while not (!cur).IsEmpty do
          match (!cur).Peek() with
                      | None
-> do ()
                      | Some v
-> do cur := (!cur).Pop()
                                  
yield v }

 I think what bothers me the most here is that the while and match are redundant. The None case will never be matches because we guard on IsEmpty. Moving that into a separate function gives:

member x.All5 =
      let v = function Stack(vl, _) -> vl | _ -> failwith "dont call on empty"
      let cur = ref x
      {
while (!cur) <> Empty do
          yield v (!cur)
          
do cur := (!cur).Pop() }

I prefer All5 to All4, but still think All1 or All2 are nicer. (Chris's original is best as far as I can tell.) How else can this be done?

Edit: I lost sight of one of the principals of functional programming: Composition. Here's a simple update to All5 that, IMHO, vastly improves it:

member x.All6 =
      
let cur = ref x
      
let toVal = function Stack(v, _) -> v | _ -> failwith "dont call on empty"
      let next () = try toVal (!cur) finally cur := (!cur).Pop()
      {
while (!cur) <> Empty do yield next () }
 

Every line builds up and you don't have to keep its details "active" in your mind. This makes the sequence expression (which is the driver of the algorithm) easy to verify. And again, to clarify, in a real system I'd probably use what Chris suggested since it's the nicest syntax. The other versions are only seeing what it looks like if we toss recursion and use a reference cell.

Code | FSharp
Saturday, August 16, 2008 4:09:49 AM UTC  #    Comments [0]  |  Trackback

# Friday, August 15, 2008
Typing string IDs

I just read these two posts:
http://blogs.msdn.com/simonince/archive/2008/08/15/strongly-typed-primitives.aspx
http://www.thejoyofcode.com/Avoiding_Primitive_Obsession_to_tip_developers_into_the_pit_of_success.aspx

And that reminded me about something we recently did. One system we're working on uses a lot of string identifiers for many different types of objects. There are many, many of these stored and passed around, so keeping things efficient was of high concern. 

The downside of string IDs (really, using any common type as an ID) is that it's legal to pass any primitive of the same type. Strings and integers abound, both as IDs of other classes, as well as general use. So it's not unimaginable that someone could pass the wrong parameter some where. This could lead to runtime crashes or unexpected results (if the ID is actually a real record of another class of object). Finally, using common types for IDs reduces usability. The signature "public void Delete(int id)" leaves a lot to be desired.

We wanted to hit all these issues, in addition to keeping things simple. There are times when untyped data needs to be converted, and this should be easy and clear. We wanted to avoid having to define new types when we had new classes of objects to identify It is also customer-visible code, so C# is used.

Using a reference type was unacceptable, because it'd add at least 12 bytes overhead (I think more on x64). Using a struct fixes this, in addition to dealing with silly nullability issues. [If a type can be null, it should always be explicit. C#'s "references types can be null" makes this hard.]

The end result was quite simple. Wrap a string in a structure so equality and hashing pass through. But, take advantage and remove case/cultural sensitivity (since in many systems, data IDs are not case sensitive). Provide explicit conversions so you can easily convert to and from strings, but never by accident. (If the conversions were implicit, you're back in the starting point.) Finally, add a generic parameter that is never used. The generic parameter gives you distinct types without having to define them. Now the APIs can look like:

    public void Delete(Id<Product> id)...

    Dictionary<Id<Group>, List<Id<User>>> members...

When you do have hardcoded IDs, as the blog entries I mentioned do, you can convert easily: (Id<User>)"Admin". Nulls are treated as empty, all the time (empty may be a valid value anyways).

When a truly optional ID is needed, use nullable types: "Id<Whatever>?". This fully captures how values are handled. This is vastly better than "It's a reference type, so maybe null is allowed. Or maybe null will crash. Empty string might be considered null, or maybe empty string means optional." With explicit nullability, the type system says it all.

The best part is that there should be pretty much no overhead. I'd expect the equality functions to be inlined, and there's no memory overhead, since the struct is simply a string reference.


Here's the class.:
public struct Id<T> : IEquatable<Id<T>>
{
    public Id(string name) {
        this.name = name ?? "";
    }
    readonly string name;
    
    public static explicit operator string(Id<T> x) { return x.name ?? ""; }
    public static explicit operator Id<T>(string s) { return new Id<T>(s); }

    public override bool Equals(object obj) {
        return
            !(obj is Id<T>) ? false :
            ((Id<T>)obj) == this;
    }

    public bool Equals(Id<T> other) {
        return other == this;
    }

    public override int GetHashCode() {
        return (name ?? "").GetHashCode();
    }
    public override string ToString() {
        return name ?? "";
    }
    public static bool operator ==(Id<T> a, Id<T> b) {
        return StringComparer.InvariantCultureIgnoreCase.Compare(a.name, b.name) == 0;
    }
    public static bool operator !=(Id<T> a, Id<T> b) {
        return !(a == b);
    }
}
I'd be interested in seeing a more generic, yet very simple, solution: one that doesn't rely on the underlying type to be string, but still provides all the same functionality. I don't think it's possible, since there's no way to get a generic constraint that'd allow similar handling of "string" and "int?". Additionally, structs can't inherit, so you'd end up using "Id<string, Product>" everywhere, which is far from elegant.
Code
Friday, August 15, 2008 7:39:11 PM UTC  #    Comments [8]  |  Trackback

# Thursday, June 26, 2008
I underestimated the power of query comprehensions

In my last post, I said I'd write the sample in C# to compare to F#. Well, I grossly underestimated the power of query comprehensions. The C# version is almost the same length (formatting differences, mainly). I'm surprised and impressed. (Or maybe I'm writing F# like I'd write C#.) Edit: I think maybe sequence expressions could cut it down a bit...

...But... C# still can't do discriminated unions efficiently or effectively ;).


    1 // crudcreatecompare.cs: Generates LINQ CRUD table fields using the horribly named DatabaseBase code

    2 //

    3 // Tables look like: [Table(Name="dbo.Accounts")]

    4 // Columns look like this:

    5 //  [Column(Storage="_AccountName", DbType="VarChar(128) NOT NULL", CanBeNull=false, IsPrimaryKey=true)]

    6 //  [DataMember(Order=1)] // Exists if serialization is turned on; used to order key parameters

    7 // Emits:

    8 //  public static readonly TableHelper<Account, String> Accounts =

    9 //      CreateTable(dc => dc.Accounts, a => a.AccountName);

   10 

   11 using System;

   12 using System.Collections.Generic;

   13 using System.Data.Linq.Mapping;

   14 using System.IO;

   15 using System.Linq;

   16 using System.Reflection;

   17 using System.Runtime.Serialization;

   18 

   19 class Program

   20 {

   21     static void Main()

   22     {

   23         Console.WriteLine(

   24             new Program().generate(

   25             "C:\\yourlinq.dll"));

   26     }

   27 

   28     string generate(string asmpath)

   29     {

   30         var asm = Assembly.LoadFrom(asmpath);

   31         var lines = from t in asm.GetExportedTypes()

   32                     let ta = getAttr<TableAttribute>(t)

   33                     where ta != null

   34                     let name = pluralize(ta.Name.Replace("dbo.", ""))

   35                     orderby name

   36                     select genTable(t, name);

   37         return joinStrings("", lines);

   38     }

   39 

   40     string genTable(Type t, string tableName)

   41     {

   42         var keyProps =

   43             from p in t.GetProperties()

   44             let c = getAttr<ColumnAttribute>(p)

   45             where c != null && c.IsPrimaryKey

   46             let dm = getAttr<DataMemberAttribute>(p)

   47             orderby dm == null ? 0 : dm.Order

   48             select p;

   49         var tw = new StringWriter();

   50         tw.WriteLine("public static readonly TableHelper<{0}, {1}> {2} =",

   51             t.Name,

   52             joinStrings(", ", keyProps.Select(p => p.PropertyType.Name)),

   53             tableName);

   54         tw.WriteLine("\tCreateTable(dc => dc.{0}, {1});",

   55             tableName,

   56             joinStrings(", ", keyProps.Select(p => "a => a." + p.Name)));

   57         tw.WriteLine();

   58         return tw.ToString();

   59     }

   60 

   61     string joinStrings(string sep, IEnumerable<string> items)

   62     {

   63         return string.Join(sep, items.ToArray());

   64     }

   65     string pluralize(string s)

   66     {

   67         return s.EndsWith("s") ? s : s + "s";

   68     }

   69 

   70     T getAttr<T>(ICustomAttributeProvider icap)

   71     {

   72         var a = icap.GetCustomAttributes(typeof(T), true);

   73         return a.Length == 0 ? default(T) : (T)a[0];

   74     }

   75 }

Code | FSharp
Thursday, June 26, 2008 6:15:10 AM UTC  #    Comments [0]  |  Trackback

LINQ to the CRUD CreateTable generator in F#

With the DatabaseBase and TableHelper classes, you still have to generate a CreateTable field per table. Why do it by hand? I wrote an F# script to generate the statements. I must say, I'm loving F# more than I had hoped. The more I learn, the better it gets. I've noticed (even in C#) that using a functional style generally means less errors. This script worked without bugs the first time (i.e., as soon as it compiled), which is pretty cool (granted, it's not big, but I'm sure if I had tons of for loops, I woulda messed up somewhere). Maybe this weekend I'll try writing it in C# just to compare (pretty sure it'll be more than 59 lines!). And I'll preempt comments about readability: Yes, it may be difficult to read if you don't know F#, but more on that later...

I'd love feedback as to making it more "functional"; years of imperative programming don't die quickly. Also, I'm not very happy with the definition of chooseAttr, but I can't seem to get it to infer the type I want otherwise.

Anyways, here's the code. You'll need to specify the references when compiling: -r "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5\System.Data.Linq.dll" -r "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0\System.Runtime.Serialization.dll"


    1 // crudcreatetable.fsx: Generates LINQ CRUD table fields using the horribly named DatabaseBase code

    2 //

    3 // Tables look like: [Table(Name="dbo.Accounts")]

    4 // Columns look like this:

    5 //  [Column(Storage="_AccountName", DbType="VarChar(128) NOT NULL", CanBeNull=false, IsPrimaryKey=true)]

    6 //  [DataMember(Order=1)] // Exists if serialization is turned on; used to order key parameters

    7 // Emits:

    8 //  public static readonly TableHelper<Account, String> Accounts =

    9 //      CreateTable(dc => dc.Accounts, a => a.AccountName);

   10 

   11 #light

   12 open System

   13 open System.Reflection

   14 open System.Data.Linq.Mapping

   15 open System.Runtime.Serialization

   16 

   17 let getAttr<'target, 'a when 'a :> ICustomAttributeProvider> (ty : 'a) =

   18     match List.of_array (ty.GetCustomAttributes(typeof<'target>, true)) with

   19         | a::_ -> Some (a :?> 'target)

   20         | [] -> None

   21 let chooseAttr<'target, 'a when 'a :> ICustomAttributeProvider> (ty : 'a) =

   22     match getAttr<'target,_> ty with

   23         | Some(a) -> Some(ty,a)

   24         | None -> None

   25 

   26 let joinStrings (sep:string) items = items |> Seq.fold1 (fun acc x -> acc + sep + x)

   27 let pluralize (name:string) = if name.EndsWith("s") then name else name + "s"

   28 

   29 let generate(asmpath:string) =

   30     let genTable (t:Type, tableName) =

   31         let keyProps =

   32             t.GetProperties()

   33             |> Seq.choose (chooseAttr<ColumnAttribute,_>)

   34             |> Seq.filter(fun (p,c) -> c.IsPrimaryKey)

   35             |> Seq.map(fun (p,_) -> p, getAttr<DataMemberAttribute, _> p)

   36             |> Seq.orderBy(function | _,Some(dm) -> dm.Order | _ -> 0)

   37             |> Seq.map (fun (p,_) -> p)

   38         let tw = new IO.StringWriter()

   39         let pn fmt = Printf.twprintfn tw fmt

   40         pn "public static readonly TableHelper<%s, %s> %s ="

   41             t.Name

   42             (joinStrings ", " (keyProps |> Seq.map (fun p -> p.PropertyType.Name)))

   43             tableName

   44         pn "\tCreateTable(dc => dc.%s, %s);"

   45             tableName

   46             (joinStrings ", " (keyProps |> Seq.map (fun p -> "a => a." + p.Name)))

   47         pn ""

   48         tw.ToString()

   49 

   50     let asm = Assembly.LoadFrom asmpath // Don't use ReflectionOnly 'cause it won't resolve dependencies

   51     asm.GetExportedTypes ()

   52         |> Seq.choose (chooseAttr<TableAttribute,_>)

   53         |> Seq.map (fun (t,ta) -> t, ta.Name.Replace("dbo.", "") |> pluralize)

   54         |> Seq.orderBy (fun (t,_) -> t.Name)

   55         |> Seq.map genTable

   56         |> Seq.fold1 (+)

   57 

   58 generate "C:\\yourlinq.dll"

   59     |> Console.WriteLine

FSharp
Thursday, June 26, 2008 5:13:44 AM UTC  #    Comments [0]  |  Trackback

LINQ to the CRUD RTM

[Reposting because it appears to have been deleted somehow.]

A bit ago, I posted some info on doing CRUD operations using LINQ: http://www.atrevido.net/blog/2007/08/26/A+LINQ+To+The+CRUD.aspx

This is a lightweight way (no codegen at all, only 2 lines of code per table) to get some CRUD stuff with LINQ. It's not the most efficient or fantastic way of doing things, but it works fine in the several projects we've used it so far. And we get to use C# 3's expression trees, which is a fantastic and under-exploited feature. At any rate, it does show that doing disconnected work with LINQ is trivial.

The code I posted was for Beta 2 and no longer works. I've since added a few new features, but the basic idea remains the same as before. I'm just posting the new code as I've gotten a few emails and one comment about the old code no longer working.

DatabaseBase.cs (10.47 KB)

While working on it in a real project, I started using our Tuple struct, so you'll need that too:
Tuple.cs (2.8 KB)

As Scott Peterson pointed out, this class doesn't implement IEquatable<T>. Just add it and call the == operator.

As always I welcome any criticism.

Misc. Technology
Thursday, June 26, 2008 4:56:24 AM UTC  #    Comments [0]  |  Trackback

# Monday, March 31, 2008
The reason VB.NET is truly a second class citizen

No support for anonymous methods/lambda statements. I first realised this when someone commented on one of my functional C# postings, asking "how can I write this in VB"? Turns out VB has no support for anonymous methods. Ouch.

This lack of capability rules out a so much functionality, I honestly cannot imagine writing anything significant today using VB.NET. But with LINQ and F# making such a big splashes, I have high hopes for the future of functional programming on the .NET languages, and I'm sure VB.NET will get itself fixed up.

Really, dropping anonymous methods in favour of stuff like XML literals? Maybe that's why it's listed as first time or casual programming ;). (No, I jest. Scheme is far better for first time programming.)

Code
Monday, March 31, 2008 9:55:56 PM UTC  #    Comments [8]  |  Trackback

Empty WCF services references - no code generated

Usually I use WCF just by referencing the contracts directly and using my own generic helper methods. Unfortunately, I had to make use of the built-in async patterns that the codegen can provide. Hence, I ended up using the "Add Service Reference" dialog, which notices (unlike svcutil on the command line) that since I'm requesting async, it'll need to generate my interfaces even though they're already referenced. Maybe in .NET 4 we'll have a new async pattern that'll leverage Expression Trees to rid us of having to use the rather ugly FooAsync/BeingFoo patterns.

Things were working just fine until I added a reference to a new service. Adding the service reference didn't work; it failed silently. It added all the schema files to the project, but the Reference.cs file was empty. If I told it to NOT used referenced assemblies, then it generated everything just fine.

After switching to the command line and using svcutil, I got these errors:
Error: Cannot import wsdl:portType
Detail: An exception was thrown while running a WSDL import extension: System.ServiceModel.Description.DataContractSerializerMessageContractImporter
Error: Referenced type 'MyApp.Management.CoolFunction.CoolThingRule, MyApp.Management.Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' with data contract name 'CoolThingRule' in namespace 'http://schemas.MyApp.com/management/v2/00' cannot be used since it does not match imported DataContract. Need to exclude this type from referenced types.XPath to Error Source: //wsdl:definitions[@targetNamespace='http://schemas.MyApp.com/management/v2/00']/wsdl:portType[@name='ICoolThingManagement']

I regenerated without the referenced code and figured out that the problem was that one type referenced an enum which had some values without the [EnumMember] attribute (although it had a DataContractAttribute on the enum itself). Because of this, WCF wanted to represent the enum with a string and thus was incompatible. Adding the EnumMemberAttribute to all the values sorted out this issue.

Code
Monday, March 31, 2008 9:06:51 AM UTC  #    Comments [5]  |  Trackback

# Thursday, March 27, 2008
Hacking SOAP Faults into Silverlight 2 Beta 1

Silverlight 2 is pretty nice. Compared to dealing with the nightmare of HTML, CSS, AJAX and whatever else, it's quite divine. Of particular interest is that Silverlight 2 has a mini Windows Communication Foundation stack that can do basic SOAP work (and some "Web 2.0" type things too I think). While the SL WCF stack works pretty well overall, it doesn't support SOAP faults.

First off, SOAP faults usually cause the server to return an HTTP 500. For whatever reasons, this isn't handled correctly between the browser and Silverlight, and results in an UnexpectedHttpResponseCode. OK, so go into the Global.asax and in Application_EndRequest change 500s to 200s. [Yes, this requires that you run WCF with AspNetCompatibility.]

Some progress. Silverlight now gives a more useful exception:

Error: System.Runtime.Serialization.SerializationException: OperationFormatter encountered an invalid Message body. Expected to find node type 'Element' with name 'MyFunctionResponse' and namespace 'http://schemas.contoso.com/coolservice/v2/00'. Found node type 'Element' with name 's:Fault' and namespace 'http://schemas.xmlsoap.org/soap/envelope/'

Hmm, it's getting the fault, but can't seem to handle it. I tried adding typed faults to my WCF contract, but the SL WCF stack doesn't have the FaultContractAttribute. It appears as if SL cannot read faults at all.

Enter HackFaults. That message provides two pieces of data from the SOAP message, the element name, and the namespace. We can hijack these to pass our fault information, then extract it from the exception message. H to the A to the C-K-Y.

First off, we need to get access to our WCF fault. We can do this with an System.ServiceModel.Dispatcher.IErrorHandler. I've attached the full code, but basically you add an attribute to your WCF service to wire up the error handler. If you're doing WCF work, you probably want this anyways for logging purposes and so on. I've attached the entire error handler code to this article. The real meat of the error handler is this little function:

public void ProvideFault(Exception error, MessageVersion version, ref Message fault)
{
    // HackFaults store the exception in the httpcontext
    var context = System.Web.HttpContext.Current;
    if (context != null) {
        // Only works in AspNetCompat mode
        if (!context.Request.Browser.Crawler && context.Request.Browser.EcmaScriptVersion.Major > 0) {
            // This should rule out non-browsers
            context.Items.Add("HackFault", error);
        }
    }
}

I'm not sure if there's a better way to differentiate between browsers and real SOAP stacks. If there is let me know. Now, in our Global.asax, we want to pick this information up and replace the actual SOAP fault with our own hackfault:

protected void Application_EndRequest(object sender, EventArgs e)
{
    // In all fairness, I was drinking eiswein at the time
    if (Context.Items.Contains("HackFault")) {
        Context.Response.ContentType = "text/xml";
        Context.Response.StatusCode = 200;
        Context.Response.ClearContent();

        var hackEx = Context.Items["HackFault"] as Exception;
        var hackFaultXml = string.Format(
            "<s:Envelope xmlns:s=\"http://schemas.xmlsoap.org/soap/envelope/\"><s:Body><{0} xmlns=\"{1}\" /></s:Body></s:Envelope>",
            hackEx.GetType().ToString(), Context.Server.HtmlEncode(hackEx.Message));
        Context.Response.Output.Write(hackFaultXml);
    }
}

Now we have a message that has the bits of data we care about in the right places for the Silverlight exception to be ready for parsing. The SL-side parsing code is vile but easy:

public static string ExtractHackFaultMessage(this Exception ex)
{
    // HackFaults generate these Exceptions and messages:
    //Without debug:
    //Could not connect to server: System.Runtime.Serialization.SerializationException: [SFxInvalidMessageBody]
    //Arguments:FaultException,Something silly
    //With debug:
    //Could not connect to server: System.Runtime.Serialization.SerializationException: 
OperationFormatter encountered an invalid Message body. Expected to find node type 'Element' with
name 'SomeOperationResponse' and namespace 'http://schemas.cool.com/yea/v2/00'.
Found node type 'Element' with name 'FaultException' and namespace 'Something silly'
if (!(ex is System.Runtime.Serialization.SerializationException)) return null; var msg = ex.Message; string regexp; if (msg.Contains("[SFxInvalidMessageBody]")) { regexp = @"Arguments:(.*),(.*)"; } else if (msg.Contains("OperationFormatter encountered an invalid Message body")) { regexp = @"Found node type 'Element' with name '(.*)' and namespace '(.*)'"; } else { // Nope, it's some thing else return null; } var match = System.Text.RegularExpressions.Regex.Match(msg, regexp); if (match.Groups.Count != 3) return null; // Not expected var exType = match.Groups[1].Value.Trim(); var exMsg = match.Groups[2].Value.Trim(); if (exType == "System.ServiceModel.FaultException") return exMsg; else return exType+ ": " + exMsg; }

This little function will give me the fault information (and not mention FaultException if that's what it is) or null if it can't extract it. When dealing with errors, I can show an error like this: "Error: " + (ex.ExtractHackFaultMessage() ?? ex.ToString())

This gives me at least rudimentary error reporting from the server back to Silverlight. If this can be improved or fixed up, please tell me. (Typed fault exceptions would be nice, but then SL can't codegen the contracts.) At any rate, it works as a cheap hack until Silverlight Beta 2 (they gotta fix it by then, right? Right?).

HttpErrorHandler.cs.txt (2.49 KB)
Code
Thursday, March 27, 2008 7:19:36 AM UTC  #    Comments [1]  |  Trackback

# Wednesday, March 12, 2008
Medical and General Security

Nothing surprising

I've been waiting for this: http://www.nytimes.com/2008/03/12/business/12heart-web.html?_r=2&ref=technology&oref=slogin&oref=slogin

Certain pacemakers (Medtronic in this case) are easy to reprogram without any useful authentication. The result is that an attacker can kill someone remotely by modifying their pacemaker.

This certainly will not be the first time this happens. The response from Medtronic is idiotic:

"To our knowledge there has not been a single reported incident of such an event in more than 30 years of device telemetry use, which includes millions of implants worldwide"

It's funny seeing industries that typically have little to no security requirements in their products get rudely awakened. Another vendor, St. Jude, says something equally scary:

"used “proprietary techniques” to protect the security of its implants and had not heard of any unauthorized or illegal manipulation of them"

Who wants to bet there's some globally shared key at work? At any rate, we expected this kind of stuff because too many people can't think clearly about security (I'll be writing about [the lack of] VoIP security soon).

A growing problem

How should these devices secured in the first place? I'm not talking specifically about pacemakers, but all sorts of implants and enhancements that we will have during the next years, using security technology today.

First, they need to be remotely monitored. This is relatively easy to secure, as the risk is considerably less: information disclosure. For example, if each monitoring device had to have it's public key explicitly trusted for a particular patient, that'd be pretty easy. In the case that a key was disclosed (say, by capturing and attacking a monitoring device), the only access gained is read-only.

Making it even less risk, it's possible that the amount of effort required for such an attack exceeds the value of the information gained. For example, if an attacker can access a target's house, they could steal identification and request medical records be sent to them.

More importantly is editing of configuration. How do we determine who has access? In theory, we want any qualified medical professional to be able to change configuration in case of an emergency. Without a global network connected to the device, the device has no way to validate credentials, particularly revocation. Additionally, even assuming that every device has access to a global database, there would be too many authorised users to ensure security. (Just like large government databases.)

Is this a threat? Some people may think this is a far fetched idea. Certainly today this is not a widespread fear. It may be a neat way to carry off a attack against a single target, but I doubt it'd be effective for major attacks. But how long will it be until a large percentage of the population carries some kind of embedded device? Pacemakers, medicine delivery systems, vision implants, hearing, digestive -- the list goes on.

The bottom line is that humans will carry more embedded technology, and this technology must be secure *and* accessible. A system where losing your private key means surgery is not usuable.

The easy solution

As far as I can tell, the only solid way to ensure security with today's technology is to add a hard link. In order for anyone to modify configuration, the configuration device must establish itself over a physical connection. This ensures no remote attacks are possible. This would take away little to no convenience -- before editing yourself, you'd have to let them physically connect a device to you.

The same could be done for remote devices. Let's say your doctor wants to adjust your body remotely. You'd simply key the remote device[s] to your doctor, and key yourself to the remote device[s]. You've established a chain of trust that's easy to clear and recreate later. There is no global database, simply yourself and devices you touch.

This mimics what you have in the real world: You trust your doctor after you establish a relationship with him. You can then call him on the phone and you trust his advice to take more or less of the medicine.

A quick note on the details: The medical devices themselves don't need hard lines to the hard configuration interface. Indeed, your "hard link" could be a special device keyed to yourself. However, embedding this device into the body means you won't lose it and it'll be readily accessible to medical teams, even if you're unconscious.

To protect against damage to the hard link device, I suppose a backup key could be made authorized. You could then store it safe, by yourself (as in, with a bank's security deposit box, not the database of the device manufacturer).

The general solution

However, this only secures us as much as we can trust the authentication. But it still relies on manual revocation and trust editing. It may be acceptable to Verisign when they accidentally issue a certificate in Microsoft's name to an attacker, but it is not acceptable for humans. Specifically, in a short vulnerability window, you could die.

The real solution, and one that we're going to need eventually across all technology, is intelligence. Specifically, a machine intelligence that determines if what is happening or what is requested is dangerous. This is the only way that we will have security moving forward.

This kind of intelligence is what we use to protect ourselves now. If the water comes out glowing green, we decide we won't trust it, even though we do trust (in general) our public water system. If you see your doctor and he recommends moving from 5mg to 500mg of Xanax a day, you'll immediately revoke his trust.

Attacks will adopt this kind of intelligence. A hacker uses a vulnerability to gain access and then attack other systems from there. How long will it be until attacking programs themselves replace the work done by the hacker?

Our software and machines will have to adopt this kind of intelligence to thwart such attacks. It will no longer be "oh, sorry you got hit by malicious code from clicking on a hyperlink, please reinstall your OS". As long as humans can be killed by the devices in use, the stakes are too high for even tiny vulnerability windows.

 

Security
Wednesday, March 12, 2008 11:56:44 PM UTC  #    Comments [0]  |  Trackback

# Thursday, February 28, 2008
Writing events with System.Diagnostics.Eventing

... or, how the hell to use Vista and 2008's new ETW stuff with managed code. And, introducing ecmanaged: A decent way to do all this stuff.

Quick ETW Overview

Actually, the real ETW overview is here: http://msdn.microsoft.com/msdnmag/issues/07/04/ETW/ <-- This is some of the best overview and documentation on it (the other good stuff is the ecmangen documentation in the Windows SDK bin folder). The MSDN stuff is terribly confusing for the most part. Or maybe I'm too spolied by how easy it is to find stuff in the BCL. My overview is on what you gotta do to make things work in .NET.

ETW is a real pain to use with .NET. Even so, ETW starts off looking really promising. You define everything in a nice XML manifest file, and everything is based off that. But wait, everything? Shouldn't the manifest be the end-all? Yea, that'd make logical sense. No, you run some tools from the Windows SDK. First you run MC, which generates a .h header file. Managed devs are growning now -- why the hell should something as general as event tracing be language specific? The .h file contains the processed event descriptors, ready for C consumption. 

It worsens: MC also generated a resource script. You have to compile that with RC and it'll create a Win32 .res resource. Then you compile that into a binary (the C# compiler has the /win32res option). Then you go back and edit your XML manifest and make sure it points to the final binary. Wait, what? Yes. The resources that MC generates for RC contain all the messages that are in your XML manifest. Someone thought it was a really cute idea to go and make the Event Viewer not only read all the data from your manifest, but also have to go look it up from some binary resources.

Actually, this probably made sense to someone on the Windows team since I'm guessing they already have tools to go and localise Win32 resources or something. Unfortunately, it sucks and makes no sense for anyone NOT in their particular position. Now, I hope I'm wrong (I really, really want to be wrong), but I think there's no way to force the message strings to just stay in the XML file and be read from there. 

Finally, things get easy again. Just run "wevtutil install-manifest Some.man" (wevtutil is in system32). In fact, this utility is so user friendly, it even lets you type "im" instead of "install-manifest". At this point, assuming the other steps went well, your provider shows up in Event Viewer.

ECManGen

But wait, how do I actually make that manifest? This part is almost the easiest. In the Windows SDK, there's a lovely little tool called ECManGen. Just fire it up, and go to town adding Providers, Channels, Templates, and Events.

Providers are the main things that show up in your Event Viewer, such as MyApp-FooProduct-LameComp. Channels separate Admin/Operational/Debug and others. Templates are an argument list for Events. If you have, say, a bunch of events that take the same kinds of parameters, you can share templates among them (I find it helpful to create a "SingleStringTemplate".) It's very straightforward.

*Note: I can't actually get Admin channels to work. If I create an event and stick it in an Admin channel and set its level to Informational, MC complains (as does ECManGen) that the level has to be Critical, Error, or Informational. Uh, OK. Instead, just use Operational.

Except... ECManGen is a free utility. (Free? Perhaps not, seeing as the annual MS bill for a 4-person dev team is around $20,000 (counting just MSDN) -- but it's well worth it.) Part of the docs say: "NOTE: For the Manifest Generator Tool to function correctly, the file winmeta.xml (which contains pre-defined metadata values) must either be in the Include directory of the SDK or in the same directory as the Manifest Generator tool." OK, easy enough. Except... it doesn't work that easily. The only way I got it to work was to copy the xml file over to the same directory, *and* start ECManGen from that directory.

Oh yea, ECManGen won't open your manifest file if you pass it as an argument, so forget about cute VS integration. Just Google ecmangen and go rate up the bugs on Connect :).

Going Managed

OK, so you're not living in the last century and use decent tools -- how does this map to C#? First off, you create an EventProvider with the right Guid (the one from your manifest). Then you create an EventDescriptor for each event, matching up all the little parameters (the MSDN docs for EventDescriptor have more details). Finally, you can call WriteEvent, passing the EventDescriptor *by ref* for some reason (no, I can't figure out why).

Oh yea, and you have to hookup that Win32 resource to your C# project, so if you needed another resource (like another app manifest?), you'll have to go deal with merging them and all that hassle. And, don't forget to make sure the parameters you pass into the object[] array of WriteEvent line up with what your manifest has. And also, the .NET API won't even handle the Boolean->BOOL (4 byte) silliness for you. 

In summary, it's a lot of boring, error-prone work, and you'll have to repeat it every time you edit your manifest. Yuck. Maybe it's just easier to use the old event log stuff and forget about all this fancy ETW stuff.

However, the ETW features are pretty cool. As is ecmangen (well, for the most part). I couldn't tear myself away from the promises the new ETW stuff offered, so I sat down and wrote a hack tool to do everything for me. The goal is to have an XML file, then convert it right into a DLL I can use.

ecmanaged does exactly this. Sorta. It doesn't deal with complicated scenarios or performance counters, but handles straightforward event logging with templates in an easy and type-safe fashion.

Using ecmanaged

Usage: ecmanaged ecgen <manifest> /out:<target assembly> /namespace:<target namespace> /class:<target class>
Usage: ecmanaged msglocate <manifest> <messageFileName>
Usage: ecmanaged install <manifest> <targetManifest> <messageFileName>
Usage: ecmanaged uninstall <targetManifest>

Call ecgen with a manifest, and it'll go build an assembly with the wrappers for calling your provider events. It'll stick the Win32 resource on the assembly too, so it's ready to be used.

Call msglocate with the manifest and the message binary file, and it'll go fix the manifest to point to the right place.

Call install, and it'll copy the manifest to another location, fix the msg location, and call wevtutil on it. It copies because I'm assuming your original manifest is under source control, and hence shouldn't be fixed up.

Uninstall just calls wevutil uninstall-manifest.

Yey! Now you can just make a batch file to call this on your manifest and run it every time you change your manifest. What I do is stick this in a folder in my project called "Eventing". I check the resulting binary into source control (just like a 3rd party DLL). The downside is making sure you always update the DLL if you modify the manifest. I'm sure somehow msbuild can come to the rescue (and autoupdate when the manifest changes, so Intellisense keeps working), but this works for me.

I welcome all comments/questions/insults, but please, no death threats. Download: ecmanaged.exe (45 KB)

P.S. No, I don't hate Windows or the devs on it, although I hate unmanaged code (it IS 2008, innit?). Perhaps I'm just annoyed I have to make a Saving Throw versus Hang every time I click Send in Outlook 2007 (yes, patched up, even with Vista SP1... which made it worse). But I still think this whole ETW toolchain is terribly unpolished.

Code
Thursday, February 28, 2008 9:12:25 PM UTC  #    Comments [1]  |  Trackback