Logo




Subscribe:
RSS 2.0 | Atom 1.0
Categories:

Sign In


[Giagnocavo]Michael::Write()

# Friday, May 07, 2010
F# Unit Testing with Visual Studio 2010

In Visual Studio 2010, you can now get your F# unit tests loaded by the IDE. First, create your F# test project. This is just a normal library referencing Microsoft.VisualStudio.QualityTools.UnitTestFramework and having test classes and so on. The only change you need to do is go into the configuration and set the output folder to "bin" for both debug and release (instead of bin\debug).

Next, create a C# Test Project, and delete the code file. Then add an existing item, and navigate to the bin directory of your test project output. Select the DLL and Add as Link.

Finally, right click the solution, and change the project dependencies so the C# test project depends on your F# test project.

That's it. Now the IDE will pickup your F# tests and allow you to manage, run, and debug them right from VS. You may have to restart the IDE after setting all this up for it to work smoothly. 

Code | FSharp
Friday, May 07, 2010 4:57:57 AM UTC  #    Comments [2]  |  Trackback

# Thursday, May 06, 2010
Command line tool for DPAPI

Using the data protection API, I found I needed to be able to generate values by hand that my apps can work with later. So I wrote a small command line utility to help out with that process.

Example:

dpapicmd [/user] [/decrypt] [/utf8] [/entropy:<entropy>] {/clipboard|<text>}
Text is read as Base64 bytes unless /utf8 is used. Decrypted input and encrypted output is always Base64.

C:\>dpapicmd /utf8 /entropy:dogstuff "I'm barking mad."
AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAOE510Ds5PkGpG2g7PxkgXwQAAAACAAAAAAAQZgAAAAEAACAAAACSClIQpWDawT26jRrsFr/HauG2NjUw963fPKH+AcXqlwAAAAAOgAAAAAIAACAAAABwxg
Osbkh6TfdTUzaiEGgKJ/ohL91VGHIpRDrBeR7wvyAAAADyvUh1W+bmzUFago/LybBvmmQD96x2vCOiJgpPxTNItEAAAAAmVJ3DoTnIAfUIsO2ea8hqs6Rpp77gvDn77XAzXfACRY3IGT7BjicYJ7Og
NQhzsrHCybBK0DRQchrPK5+XT8TR

C:\>dpapicmd /utf8 /dec /entropy:dogstuff AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAOE510Ds5PkGpG2g7PxkgXwQAAAACAAAAAAAQZgAAAAEAACAAAACSClIQpWDawT26jRrsFr/HauG2
NjUw963fPKH+AcXqlwAAAAAOgAAAAAIAACAAAABwxgOsbkh6TfdTUzaiEGgKJ/ohL91VGHIpRDrBeR7wvyAAAADyvUh1W+bmzUFago/LybBvmmQD96x2vCOiJgpPxTNItEAAAAAmVJ3DoTnIAfUIsO
2ea8hqs6Rpp77gvDn77XAzXfACRY3IGT7BjicYJ7OgNQhzsrHCybBK0DRQchrPK5+XT8TR

I'm barking mad.

Download: dpapicmd.exe (7.5 KB) (Requires .NET 2.0 because I'm too lazy to write it in C.)

And horrible hacky source: dpapicmd.cs.txt (4.25 KB)

 

Code | Security
Thursday, May 06, 2010 5:53:33 PM UTC  #    Comments [0]  |  Trackback

# Wednesday, April 28, 2010
FluentNHibernate FSharp update for RTM
I've updated the code we use for FluentNHibernate with F# - using the final versions of both products. It's just some code to fixup F# quotations to LINQ expressions that FluentNHibernate can work with, and type extensions to make it easy to consume from F#.

The FluentNHibernate RTM zip has all the other binaries you need to get started.

I'm building against .NET 4, but the code should work on 2.0 as well.

FluentNHibernate.FSharp.dll (38.5 KB)

FluentNHibernate.FSharp.zip (6.95 KB) Source

And here's the FluentNHibernate sample first project using FluentNHibernate.FSharp:

FirstProject.zip (7.66 KB)

Here's an example of what some mapping code looks like:

type ProductMap() as m = inherit ClassMapQ<Product>() do

    let x = m.DefaultX

    m.Not.LazyLoad()

    (m.IdQ <@ x.Id @>).Done

    (m.MapQ <@ x.Name @>).Done

    (m.MapQ <@ x.Price @>).Done

    (m.HasManyToManyQ <@ seq x.StoresStockedIn @>)

        .LazyLoad()

        .Cascade.All()

        .Inverse()

        .Table("StoreProduct")

        .Done


Code | FSharp
Wednesday, April 28, 2010 9:56:22 PM UTC  #    Comments [0]  |  Trackback

# Thursday, July 16, 2009
F# Records and ASP.NET MVC Binding

I came across a thread on hubFS about deserialising F# records with JSON. After Mr. McNamara pointed out that DataContract serialization would work with F# records, I realised we can do the same for other serialisation systems, such as ASP.NET MVC.

ASP.NET MVC binding doesn't work with F# records for a few reasons. First, it requires a default constructor, and record types don't have one. Second, it needs settable properties, and records have read-only properties. Fortunately, the backing field for a record's property is a mutable field. The name is mangled (@ is appended), but otherwise we're ok to set that field.

With this, we can subclass the default model binder and add in code to construct records as well as set their fields directly. Unlike DataContract serializers, I didn't use FormatterServices.GetUninitializedObject to create the object, I use the F# reflection function MakeRecord. This is because I want to attempt to initialise all fields on the record type, to try to keep out nulls. This goes against how the rest of MVC's null handling goes, so perhaps it's not a great idea.

At any rate, here's the quite short code. A lot of things probably don't work, such as F# lists. Perhaps there should be a community project that collects F#-specific type helpers for different frameworks to make serialization, binding, etc. easier.

open System

open System.Web.Mvc

open Microsoft.FSharp.Reflection

 

type RecordDefaultModelBinder() =

    inherit DefaultModelBinder()

 

    let isrec = FSharpType.IsRecord

 

    /// Makes a record, trying to provide initialised values for each field   

    let rec makeDefaultRecord ty =

        let defval ty =

            if isrec ty then makeDefaultRecord ty

                        else match ty.GetConstructor(Type.EmptyTypes) with null -> null | c -> c.Invoke null

        let vals = FSharpType.GetRecordFields ty |> Array.map (fun x -> defval x.PropertyType)

        FSharpValue.MakeRecord(ty, vals)

 

    override this.CreateModel(cc, bc, ty) =

        // We have to avoid them calling Activator.CreateInstance on records

        if isrec ty then makeDefaultRecord(ty) else base.CreateModel(cc, bc, ty)

 

    override this.GetModelProperties(cc, bc) =

        // Default one filters out read-only, but we own the field

        if isrec bc.ModelType then

            let props = ComponentModel.TypeDescriptor.GetProperties(bc.ModelType)

                        |> Seq.cast<ComponentModel.PropertyDescriptor> // BCLFail

                        |> Seq.filter(fun p -> bc.PropertyFilter.Invoke(p.Name))

            ComponentModel.PropertyDescriptorCollection(Seq.to_array props)

        else

            base.GetModelProperties(cc, bc)

 

    override this.SetProperty(cc, bc, propDesc, value) =

        // To set a record property, set the mangled field

        if isrec bc.ModelType then

            let field = bc.ModelType.GetField(propDesc.Name + "@", Reflection.BindingFlags.Instance ||| Reflection.BindingFlags.NonPublic)

            field.SetValue(bc.Model, value)

        else

            base.SetProperty(cc, bc, propDesc, value)

ASP.NET | FSharp
Thursday, July 16, 2009 9:35:53 AM UTC  #    Comments [1]  |  Trackback

# Friday, May 15, 2009
F# CTP and ASP.NET MVC ASPX: Scoping disposables

If you use ASP.NET MVC with F# CTP (Monday brings Beta 1 and probably many changes), you might run into an issue of how to scope certain things. For example, an IDisposable is used to create blocks, for things like an HTML form. Example in C#:

    <% using (Html.BeginForm()) {%>

 

        <fieldset><legend>Fields</legend>...</fieldset>

 

    <% } %>

 

This is needed because VB and C# didn't have any easy function/block syntax (VB 10 should fix this), and many C# developers are still wary of higher order functions. How does this play out in F#? First off, whitespace is important. This can get really messy with the current F# ASP.NET integration. Basically, always open the script blocks on a separate line, and indent from the first column. Example:

<% let i = 0

   this.Response.Write(sprintf "i = %d" i) %>

(Note that the 'this' variable is bound to the current page.) This fails:

Compiler Error Message: FS0010: Unexpected keyword 'let' or 'use' in expression. Expected incomplete structured construct at or before this point or other token

Source Error:

Line 111:                let mutable parameterContainer = parameterContainer
Line 112:                __w.Write("\r\n") |> ignore
Line 113:                   let i = 0
Line 114:                   this.Response.Write(sprintf "i = %d" i) 
Line 115:                __w.Write("\r\n    <h2>Create</h2>\r\n\r\n    ") |> ignore

Note how the <% is counted as space, so the let starts off indented 3 spaces. Instead, we need to write it so:

<%

let i = 0

this.Response.Write(sprintf "i = %d" i) %>

This works fine. You can also put the %> on the next line if you like. Now, on to ASP.NET MVC's IDisposable usage. A straightforward use of the F# using function won't work:

<b>Demo</b>

 

<%

using (Mvc.Html.FormExtensions.BeginForm this.Html) (fun _ -> %>

 

    <p>Inside a form</p>

 

<% ) %>

Compiler Error Message: FS0191: The mutable variable '__w' is used in an invalid way. Mutable variables may not be captured by closures. Consider eliminating this use of mutation or using a heap-allocated mutable reference cell via 'ref' and '!'.

Source Error:

Line 114:                __w.Write("\r\n\r\n<b>Demo</b>\r\n\r\n") |> ignore
Line 115:                  
Line 116:                using (Mvc.Html.FormExtensions.BeginForm this.Html) (fun _ -> 
Line 117:                __w.Write("\r\n\r\n    <p>Inside a form</p>\r\n\r\n") |> ignore
Line 118:                   ) 

As the error says, this is because the __w variable is mutable, so we can't play with it inside a lambda. I'm not sure if this will be worked around -- they'd have to change the codegen quite a bit, I'd think. As a side note, the F# CTP does not support C# extension methods (hence the verbose calling of BeginForm), but F# will eventually -- maybe in the Beta.

The way we must scope is with a use binding. There's no way I see to accomplish this with whitespace alone. Due to the ASPX translation process, this would probably be very error prone. Instead, we can simply put the use binding inside a do expression:

<b>Demo</b>

 

<%

do (use form = Mvc.Html.FormExtensions.BeginForm this.Html %>

 

<b>Inside the form</b>

 

<% ) %>

 

<b>Outside of the form</b>

The parentheses setup the scope exactly how we want it.

ASP.NET | FSharp
Friday, May 15, 2009 8:46:39 PM UTC  #    Comments [0]  |  Trackback

# Thursday, April 23, 2009
Update on F# with Fluent NHibernate

Last time, I hacked up some rudimentary support for F# and Fluent NHibernate. It really looked ugly. I looked into what I could do to make it not look as sucky.

First off, having to write a full lambda for each mapping was annoying. F# quotations don't have to be lambdas, unlike C#'s expressions. So, instead of having to write, say, "fun x -> x.Foo", we can write "x.Foo", assuming there's a local variable x with the right type. The ClassMap subclass now expects these types of quotations than full lambdas.

Next, I experimented with using type extensions to overload functions like Id and Map, however I found out that F#, at least currently, does not add in type extensions for overload resolution. So I ended up having a new subclass of ClassMap<T>, 'T ClassMapQ. I used the Q suffix consistently to denote "quotations". I added type extensions for most of the other mapping types so that quotations could be used.

As to the question of having to tag on " |> ignore " to each mapping, I decided to write an extension propery for IMappingPart called Done, which is simply unit. Finally, the problem of lazy loading I took care of by setting "use_proxy_validator" to false, as mentioned here. The end result is that this mapping code:

type StoreMap() = inherit ClassMap<Store>() do
    base.Not.LazyLoad()
    base.Id ~@@ <@ fun x -> x.Id @> |> ignore
    base.Map ~@@ <@ fun x -> x.Name @> |> ignore

    (base.HasManyX <@ fun x -> upcast x.Staff @>)
        .LazyLoad()
        .Inverse().Cascade.All() |> ignore
    (base.HasManyToManyX <@ fun x -> upcast x.Products @>)
        .Cascade.All()
        .WithTableName("StoreProduct") |> ignore

Is now:

type StoreMap() as m = inherit ClassMapQ<Store>() do
    let x = m.DefaultX
    (m.IdQ <@ x.Id @>).Done
    (m.MapQ <@ x.Name @>).Done
    (m.HasManyQ <@ seq x.Staff @>)
        .LazyLoad()
        .Inverse().Cascade.All()
        .Done
    (m.HasManyToManyQ <@ seq x.Products @>)
        .LazyLoad()
        .Cascade.All()
        .WithTableName("StoreProduct")
        .Done

This is pretty enough that I'm satisfied with how well F# interops. Most of the things I figured out here will apply to other .NET OO APIs, not just this one.

I am hoping that VS2010 Beta 1 will ship soon, as F# is getting some interesting upgrades then. With that, it should be easier to extend the support to other parts of NHibernate, such as querying, and perhaps integrate in NHibernate.Linq.

FSharp
Thursday, April 23, 2009 7:06:45 PM UTC  #    Comments [0]  |  Trackback

# Wednesday, April 01, 2009
Using Fluent NHibernate with F#

Fluent NHibernate is a nice way to be able to use NHibernate without having to deal with all that unchecked XML. This morning I decided to find out how well it works with F#. Things went relatively smooth. I've converted some code samples from the Fluent NHibernate First Project. I suggest having that open to fill in any gaps. I've also included the full project code and DB script at the end of this article.

If you're not familiar at all with Fluent NHibernate, basically it takes advantage of lambda expressions as Expression<T> to provide a somewhat strongly typechecked reflection system. Thus, instead of having attributes with hard-coded strings, or XML files, you have expressions that target the properties of objects you wish to map. The Fluent NHibernate library then takes care of hooking it up to NHibernate, and away you go. Something like that anyways.

Classes

So, first, we define the "entities" like the C# project does. Here's the first little pain. F# doesn't really support C#'s idea of "automatic properties". You can have vals on a class, which act like fields (although, they are implemented as properties). Or, you can manually specify them, like you would in earlier versions of C# which didn't have the auto-gen-a-field-for-me.

F# doesn't encourage uninitialized values. If you have uninitialized fields (vals), you need to mark them with an attribute to say you know what you're doing. So, that adds a bit more code overhead. What I do here is to alias the DefaultValueAttribute to "DV". So the first bit of our classes looks like this:

#light

namespace FHib.Entities

 

open System.Collections.Generic

 

type DV = DefaultValueAttribute

 

type Employee() as this =

    [<DV>] val mutable Id : int

    [<DV>] val mutable FirstName : string

    [<DV>] val mutable LastName : string

Now, NHibernate relies on having virtual properties so that it can dynamically create code to do nifty things like lazy loading. In F#, creating a virtual member means defining an abstract member and providing an implementation, in the same class. Since we don't have automatic properties, this means we'll have to define the backing field ourselfs. In all, the full code for the virtual property "Store" (virtual so NHibernate can lazy-load it) is:

    abstract Store : Store with get,set

    [<DV(false)>] val mutable _store : Store

    override x.Store with get() = x._store and set(v) = x._store <- v

Not the pinnacle of short code, but not horrible all things considered. The rest of the entity mappings are rather straightforward so I'll skip them here. I stuck with vals for anything that didn't have to be virtual, to keep it more concise.

Mappings: Easy Start

OK, so now to the "real" work. Fluent NHiberate looks for classes that subclass ClassMap<T>. It then creates an instance of them, which allows you to call the mapping methods in the object's constructor. From the First Project:

public class EmployeeMap : ClassMap<Employee>   {   
  public EmployeeMap()   {   
    Id(x => x.Id);   
  }   
}  

OK, so how do we convert this to F#? It's easy... except for that lambda. The lambda in this case compiles to an Expression<Func<Employee, object>>. F#'s compiler does not support Expression<T>. So, we turn to experimental support. Referencing the FSharp.PowerPack.Linq assembly gives us the Microsoft.FSharp.Linq.QuotationEvaluation module. This extends the F# quotation type, Expr, with "ToLinqExpression", which returns an untyped LINQ Expression object.

To get this untyped Expression (LINQ) out of an Expr (F#) and into an Expression<T> (F#) suitable for Fluent NHibernate's consumption, I started off writing this tiny helper module:

module LinqHelper =

    open Microsoft.FSharp.Quotations

    open Microsoft.FSharp.Linq.QuotationEvaluation

    open System.Linq.Expressions

 

    let ToLinq (exp : Expr<'a -> 'b>) =

        let linq = exp.ToLinqExpression()

        let call = linq :?> MethodCallExpression

        let lambda = call.Arguments.[0] :?> LambdaExpression

        Expression.Lambda<Func<'a, 'b>>(lambda.Body, lambda.Parameters)

When an F# quotation of a lambda is turned into a LINQ expression, the root node is a useless MethodCallExpression. So, we unwrap that by taking it's argument and using it to generate a typed lambda Expression<T>.  

This is a good start. But many of the Expressions that Fluent NHibernate looks for have a return type of object. Instead of having to write "box" or "upcast" all over, I added another function called "ToLinqObj". This takes an Expr<'a -> 'b>, but returns an Expression<Func<'a, obj>>.

Finally, writing "ToLinq" and "ToLinqObj" seemed too verbose, so I added some operators to the LinqHelper module:

    let (~@) expr = ToLinq expr

    let (~@@) expr = ToLinqObj expr

Now we can start mapping:

open LinqHelper

 

type EmployeeMap() = inherit ClassMap<Employee>() do

    base.Not.LazyLoad()

    base.Id ~@@ <@ fun x -> x.Id @> |> ignore

    base.Map ~@@ <@ fun x -> x.FirstName @> |> ignore

    base.Map ~@@ <@ fun x -> x.LastName @> |> ignore

    (base.References ~@ <@ fun x -> x.Store @>).LazyLoad() |> ignore

We start off by disabling LazyLoad because most of the properties are not virtual, and NHibernate will fail to validate the mapping. Instead, we explicitly LazyLoad things, like the Store reference.

Mappings: Modifying the Expressions

Unfortunately, things didn't stay so simple. The Fluent NHibernate methods "HasMany" and "HasManyToMany", for instance, don't work with F#'s type inference. This is because they have several overloads, a couple of which take expressions. If we try this in the StoreMap:

    base.HasMany ~@@ <@ fun x -> x.Staff @> |> ignore

We get an error because F# doesn't know what x is. (error FS0055: This lookup uses a deprecated feature, where a class type is inferred from the use of a class field label. Consider using a type annotation to make it clear which class the field comes from.) Using ~@ to keep it strongly typed fails as well; it can't figure out the overload. This is nothing surprising -- overloading is the enemy of type inference.

So, what do we do? Type annotations are not what I consider fun. So instead, we'll add non-overloaded ClassMap<'t> type extensions into the LinqHelper module:

    type ClassMap<'t> with

        member x.HasManyX expr = (x.HasMany : Expression<Func<'t, seq<_>>> -> _) (ToLinq expr)

        member x.HasManyToManyX expr = (x.HasManyToMany : Expression<Func<'t, seq<_>>> -> _) (ToLinq expr)

By providing these extensions that are explicit once, we enable type inference for the rest of the time. We can now finish the StoreMap:

type StoreMap() = inherit ClassMap<Store>() do

    base.Not.LazyLoad()

    base.Id ~@@ <@ fun x -> x.Id @> |> ignore

    base.Map ~@@ <@ fun x -> x.Name @> |> ignore

 

    (base.HasManyX <@ fun x -> upcast x.Staff @>)

        .LazyLoad()

        .Inverse().Cascade.All() |> ignore

    (base.HasManyToManyX <@ fun x -> upcast x.Products @>)

        .Cascade.All()

        .WithTableName("StoreProduct") |> ignore

The only type "annotation" we need here is the upcast for the HasManyX. This is because F# forces you to be explicit. Accessing "Staff" in the first quotation means type IList<Employee>, not seq<Employee>. The upcast will sort this out for us.

This compiles fine. But remember how I said using expressions was a "somewhat strongly typed" way of doing things? The expression trees are interpreted at runtime, which allows failures a more complex type system might prevent. This is one of the times. If we try to execute it as-is, we get this exception:

 ---> FluentNHibernate.Cfg.FluentConfigurationException: An invalid or incomplete configuration was used while creating a SessionFactory. Check PotentialReasons collection, and InnerException for more detail.

 ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentException: Not a member access Parameter name: member

Ouch. Fluent NHibernate does not appreciate our expression trees. Why? Using FSI, we can inspect what we're actually converting the F# quotations to:

type SomeClass() = member x.Stuff = [|1;2;3|];;
let myExpr : Expr<SomeClass -> seq<int>> = <@ fun x -> upcast x.Stuff @>;;
let myLinq = LinqHelper.ToLinq myExpr;;

> myLinq;;
val it : Linq.Expressions.Expression<Func<SomeClass,seq<int>>>
= x => (x.Stuff As IEnumerable`1)
    {Body = (x.Stuff As IEnumerable`1);
     NodeType = Lambda;
     Parameters = seq [x];
     Type = System.Func`2[FSI_0022+SomeClass,System.Collections.Generic.IEnumerable`1[System.Int32]];}

> myLinq.Body;;
val it : Linq.Expressions.Expression
= (x.Stuff As IEnumerable`1)
    {IsLifted = false;
     IsLiftedToNull = false;
     Method = null;
     NodeType = TypeAs;
     Operand = x.Stuff;
     Type = System.Collections.Generic.IEnumerable`1[System.Int32];}

So, what's going on? The upcast is modifying the F# quotation (as it should), and the ToLinqExpression is sticking this in as a "TypeAs" node. Fluent NHibernate does not seem to like this. Not. One. Bit.

But, it IS ok if we have a Convert node in the Expression tree. I imagine this is because C# generates such nodes in its expression trees (say, accessing an int member in a Func<T, object> expression). So, our last hack in making F# work right here is adding a fixup function to our LinqHelper, and using it from ToLinq:

    let fixup (lexpr:LambdaExpression) =

        if lexpr.Body.NodeType <> ExpressionType.TypeAs then lexpr else

        let typeAs = lexpr.Body :?> UnaryExpression

        let newBody = Expression.Convert(typeAs.Operand, typeAs.Type)

        Expression.Lambda(lexpr.Type, newBody, lexpr.Parameters)

Now it's happy with our expressions.

Finally

The rest of the First Project is pretty straightforward in F#. For instance, creating the Session Factory:

let createSessionFactory() =

    Fluently.Configure()

        .Database(

            MsSqlConfiguration.MsSql2005.ConnectionString(fun csb ->

                csb.Is("server=(local);database=fhib;Integrated Security=true") |> ignore))

        .Mappings(fun m ->

            m.FluentMappings.AddFromAssemblyOf<Mappings.EmployeeMap>() |> ignore)

        .BuildSessionFactory()

Everything seems to execute as it should. Fluent interfaces in F# generate a bit more noise, because of all the ignores that have to be added. I'm thinking of creating a type extension to obj to add Ignore as a method, to make it look a bit more uniform.

Future

It's my hope that F# has now addressed most of these issues -- 1.9.6 is 7 months old. Expression<T> is growing in importance, outside of LINQ querying, so I cannot see F# not being able to handle them easily and generating similar output to C#. It'd be really neat if F# would auto-convert quotations to Expression<T> when the expr is a syntactic argument like it does with delegates now. I'd also be surprised if abstract/virtual members still lack the ability to define accessibility.

Code: fhib.zip (4.76 KB)
I did not include Fluent NHibernate or any of its dependencies. The project expects them to be in the project's lib folder, so get them here if you don't have them and extract them to "lib".

I welcome comments on this and suggestions for making the code more concise.

Code | FSharp
Wednesday, April 01, 2009 8:01:29 AM UTC  #    Comments [1]  |  Trackback

# Sunday, March 22, 2009
Using symmetric encryption to pass messages

This entry was triggered by this question. Someone asked how to use AES, and we got 2 sample classes that do it wrong. The flaw in both was that they shared the IV which means your ciphertext will can leak information. One answerer didn't believe me at first, but then got it and deleted his code. The other person got offended and said IVs, performing authentication, etc. are all "corner cases" and any problem is "contrived". So, I'm going to provide a bit of code and show two problems that arise from not generating a unique IV for each message and not authenticating the decrypted data.

First, I wrote this a while ago: What is an IV?. That describes what an IV is, and why you need a unique one for each message. Wikipedia also has good information on this. Now for the demo. I used the code at the bottom, but removed the hashing and random IV from the code. So it's just encrypting the with the same key and IV for each message -- very straightforward. Here are the messages and their ciphertext:

"Alice; Bob; Eve;: PerformAct1"
"Alice; Bob; Eve;: PerformAct2"

tZfHJSFTXYX8V38AqEfYVXU5Dl/meUVAond70yIKGHY=
tZfHJSFTXYX8V38AqEfYVcf9a3U8vIEk1LuqGEyRZXM=

Notice how the first block of ciphertext is the same? All messages starting with "Alice; Bob; Eve;" will have that same first block. That means an attacker, after getting this ciphertext once, now knows if any message is addressed the same way. Very, very straightforward, basic attack. Now, maybe for a specific implementation you have in mind, this is not an issue. But it's still improperly implemented cryptography.

For the next attack, we're going to show that even with a random IV, you need to authenticate your decrypted messages. This code generates a 64-bit integer and encrypts it with AES and a random key/IV. Then, it starts changing bytes until the decrypt succeeds. Presto: the attacker was able to present a completely different value, and the decryption was successful.

public static void Main() {
    var buff = new byte[8];
    new Random().NextBytes(buff);
    var v = BitConverter.ToUInt64(buff, 0);
    Console.WriteLine("Value: " + v.ToString());
    Console.WriteLine("Value (bytes): " + BitConverter.ToString(BitConverter.GetBytes(v)));
    var aes = Aes.Create();
    aes.GenerateIV();
    aes.GenerateKey();
    var encBytes = aes.CreateEncryptor().TransformFinalBlock(BitConverter.GetBytes(v), 0, 8);
    Console.WriteLine("Encrypted: " + BitConverter.ToString(encBytes));
    var dec = aes.CreateDecryptor();
    Console.WriteLine("Decrypted: " + BitConverter.ToUInt64(dec.TransformFinalBlock(encBytes, 0, encBytes.Length), 0));
    for (int i = 0; i < 8; i++) {
        for (int x = 0; x < 250; x++) {
            encBytes[i]++;
            try {
                Console.WriteLine("Attacked: " + BitConverter.ToUInt64(dec.TransformFinalBlock(encBytes, 0, encBytes.Length), 0));
    
            return;
            } catch { }
        }
    }
}

Here's an example run:
Value: 5686260040031435365
Value (bytes): 65-7A-92-1A-61-A7-E9-4E
Encrypted: F4-62-AC-02-2D-7D-43-6A-4D-97-68-4D-95-9F-8A-DF
Decrypted: 5686260040031435365
Attacked: 1603329786558177755

Since there's no authentication of the decrypted data, an attacker can just play with the ciphertext until it generates an acceptable value. Perhaps you have other mitigations in your implementation/application for this, but why rely on that?

Here's some demo code (I haven't tested it much, so it might have some major issues -- but not by design AFAIK). Note this just shows performing an encryption operation, including the IV in the message, and verifying the decrypted bytes. Other things like replay attacks are not considered. If you're trying to learn how to use crypto so you can drop it into an application, STOP, then go read enough to understand what you're doing and the implications for your particular application.

aesdemo.cs.txt (4.38 KB)

Code | Security
Sunday, March 22, 2009 11:46:56 PM UTC  #    Comments [0]  |  Trackback

# Thursday, December 11, 2008
Full Typed Faults in Silverlight 2 (FaultException)
Silverlight 2 doesn't support faults, so you can't catch them. That sucks, since WCF's FaultException<T> is pretty useful. In Beta 1 I did a vile hack to get some data. After reading online, Id determined that Silverlight should be able to intercept the fault message and interpret it. Unfortunately, some of the handy classes to insert yourself into the stack aren't in Silverlight's WCF. Fortunately, there is a sample that does most of it:

http://code.msdn.microsoft.com/silverlightws/Release/ProjectReleases.aspx?ReleaseId=1660

In the "Message Inspectors" sample. This sample has code to let you inspect messages, and it also has a sample where they throw "raw" faults (i.e., "wrapped exceptions", aka "ExceptionDetail"). Why they stopped there is beyond me. With a bit of hacking (I must say, this SOAP stuff looks very complex), I got it to recognize any fault type, and throw a FaultException<T>. Well, not a FaultException<T>, 'cause there is a neutered FaultException class already in System.ServiceModel from Silverlight. So I named mine "SLFaultException" (SL for Simple Lame).

To get the fault thrown, you need to hookup the SilverlightFaultMessageInspector. From the demo code:
            EndpointAddress address = new EndpointAddress("http://127.0.0.1:52620/Service.svc");
            BasicHttpMessageInspectorBinding binding = new BasicHttpMessageInspectorBinding(new SilverlightFaultMessageInspector());
            ServiceClient proxy = new ServiceClient(binding, address);
            proxy.DoWorkCompleted += new System.EventHandler<System.ComponentModel.AsyncCompletedEventArgs>(proxy_DoWorkCompleted);
            proxy.DoWorkAsync();
You also need to register the assembly's types so that the fault throwing code can find the fault detail to deserialize. There's no way in Silverlight to get all the loaded assemblies (as far as I know). So we do this:
   System.ServiceModel.SilverlightFaultMessageInspector.RegisterCurrentAssembly();

This registers the calling assembly's types to be searched when a fault detail is received.

Now the only thing that needs to be done is fixup the HTTP 500 status code to a 200. The MS sample has a behaviour to do this. But that's annoying, having another DLL to deploy and deal with. So I do this in the Global.asax:
        protected void Application_EndRequest(object sender, EventArgs e) {
if (HttpContext.Current.Request.PhysicalPath.EndsWith(".svc", StringComparison.OrdinalIgnoreCase) &&
HttpContext.Current.Response.StatusCode == 500 &&
!HttpContext.Current.Request.Browser.Crawler &&
HttpContext.Current.Request.Browser.EcmaScriptVersion.Major > 0) {
// Set 200 if its a faulted service request
HttpContext.Current.Response.StatusCode = 200;
}
}
From there, go ahead and catch SLFaultException<T> just like you would a FaultException<T>.

Overall, this proved far easier than I expected. I'm at a complete loss why they didn't ship faults in SL2. I mean, if I hacked _something_ out in an hour or two, I'm sure someone who had some clue of what they're doing could have done so easier. Heck, they even have the real WCF source -- I was stuck using Reflector for parts :P.

SilverlightFaultsBinary.zip (13.81 KB)

SilverlightFaultsSource.zip (61.55 KB)
Code
Thursday, December 11, 2008 2:04:45 AM UTC  #    Comments [4]  |  Trackback

# Wednesday, December 03, 2008
Windows Live Installer for Windows 2008
I wanted to install the new Windows Live -- Messenger looks better (normal windows instead of custom, yey!). Unfortunately, the installer (wlsetup-web) has an annoying version check. Fortunately, it's nothing OllyDbg couldn't patch.

For the paranoid, you can check it against the original:

>fc wlsetup-web.exe wlsetup-web.original.exe
Comparing files wlsetup-web.exe and WLSETUP-WEB.ORIGINAL.EXE
00019DDD: C3 6A
00019DDE: 90 20

I've only tested on Windows 2008, 32 Bit, English, and I only installed Messenger. But it appears to work just fine. (I had to manually uninstall the current Messenger version first.)

Here's to hoping the final version of the "new wave" of Live removes this check for us. Enjoy!

wlsetup-web.exe (990.5 KB)
Misc. Technology
Wednesday, December 03, 2008 2:48:06 AM UTC  #    Comments [1]  |  Trackback

# Tuesday, December 02, 2008
Command line WCF Proxy Generation for Silverlight 2 RTM

I have a scenario where my web service app exposes several interfaces, and I want to use all of them from the Silverlight UI. The code generated by SvcUtil isn't completely compatible with Silverlight's WCF, meaning you need to do a lot of fixup to make it work in SL. Looking in the forums, I see mentions of "slsvcutil" and "slwsdl", but I've been unable to actually find these programs. The only thing that Silverlight has is "Microsoft.Silverlight.ServiceReference.dll", in the "Program Files\Microsoft SDKs\Silverlight\v2.0\Tools\ServiceReference" folder.

I opened that DLL up in ildasm, and it appears to just be some classes that work on a ServiceContractGenerator. Digging in deeper, yep, they just go through the generated CodeDOM and fixup stuff that isn't compatible. For instance, IExtensibleDataObject isn't supported. SL WCF also doesn't handle having Sync and Async methods (i.e, SomeOp and Begin/EndSomeOp) since they both have the same action. (You'll get an "An item with the same key has already been added" error.)

My main problems with the cutesy "Add Service Reference" UI are:
 - I can't figure out how to tell it to use mutliple WSDL files and share the data types.
 - It generates a ton of files and VS gets all obsessive over them for some reason.
 - It generates a lot of extra code (like the useless ClientBase stuff, and the idiotic/useless "event based async pattern" code). Extra code just clutters up Intellisense, so why bother?

[Add Service Reference is great for demos and perhaps simple one-offs... I can't imagine dealing with it for anything slightly complicated.]

So, I took the MSDN code sample for the ServiceContractGenerator, and hacked it up to do what I needed. Then I added the 2 lines to pass it through the Silverlight ServiceReference fixup thing, and presto - things look great. Usage is simple:

Usage: slsvchack <clr namespace> <outputfile> <wsdl1> .. <wsdlN>

The options are hard-coded to what I use and makes the most sense for Silverlight apps (in my opinion). I want to do a full fledged "SLSvcUtil.exe", but I think the right approach there is to dissassemble SvcUtil and patch in the Silverlight.ServiceReference stuff. But, I don't have the time right now.

Code: slsvchack.cs.txt (2.95 KB) [It's in C# 'cause the MSDN sample was, and my edits were minor. ]
Binary: slsvchack.exe (8.5 KB) [I'm too lazy to link in the Microsoft.Silverlight.ServiceReference.dll, so you'll need the Silverlight SDK installed.]
Code
Tuesday, December 02, 2008 7:56:09 PM UTC  #    Comments [1]  |  Trackback

# Saturday, November 29, 2008
Follow up on Obfuscation and VistaDB
Recently I mentioned how a lot of companies use obfuscation unnecessarily, and it ends up hurting legitimate customers while doing nothing to prevent "crackers". Specifically, I mentioned VistaDB, as the obfuscation tool was injecting invalid IL, causing Mono to reject the assembly.

Jason Short replied to me and I detailed the exact problems with the obfuscator (along with a few F# scripts to unobfuscate and remove the bad IL). They then released a new build with the obfuscation removed -- which Mono now happily loads.

I just wanted to give kudos to VistaDB for doing this. Not many companies are smart enough to realise that their "protection" tools are useless and do a 180 on such a stance.
Code | Security
Saturday, November 29, 2008 7:12:05 PM UTC  #    Comments [0]  |  Trackback

# Tuesday, November 25, 2008
XmlSerialization for F# Types
On a current project, I'm using F# with ASMX Web Services (Mono, so no WCF). I started off declaring some types like this:

type Account() =
    [<DefaultValue>]
    val mutable Name : string
    [<DefaultValue>]
    val mutable Data : byte array


We have to tag "default value" to tell F# it should go generate the default (Unchecked.defaultof<'a>) -- in this case, null.

Unfortunately, the serialized XML is not what we want. Currently F# generates both a public field ("_Name") and a get/set property ("Name") to access it. I'm not sure why this is, but there's probably some interesting reason (the part of the spec is 10.2.7 Explicit Fields, but it doesn't mention the implementation). The workaround is simple: tag the field with "XmlIgnore".

As of F# 1.9.3.7, you can choose to target an attribute at the property or the field for a val in a class type. See: http://blogs.msdn.com/dsyme/archive/2007/11/30/full-release-notes-for-f-1-9-3-7.aspx (I didn't find this part in the spec.)

So, the revised declaration for the XmlSerialization-friendly type is:

type Account() =
    [<DefaultValue; field: XmlIgnore>]
    val mutable Name : string
    [<DefaultValue; field: XmlIgnore>]
    val mutable Data : byte array


I found that to add a lot of noise, so I aliased them:

type DV = DefaultValueAttribute
type XI = XmlIgnoreAttribute

type Account() =
    [<DV; field: XI>] val mutable Name : string
    [<DV; field: XI>] val mutable Data : byte array


Now it works just fine and creates the XML we'd expect.

One other thing might bite you: null. F# rightfully eschews null. If you have a function that returns one of these types, you'll find it won't let you return null:

> let test() : Account = null;;
  let test() : Account = null;;
  -----------------------^^^^^
stdin(6,24): error FS0043: The type 'Account' does not have 'null' as a proper value.

This is generally good, but unfortunately, interop with the unenlightened world is sometimes necessary. The right way to do this is:

let test() : Account = Unchecked.defaultof<Account>;;

That's a bit too verbose for my liking, since I have to go annotate the type. A simple function will get around that:

> let inline getnull() = Unchecked.defaultof<_>;;
val inline getnull : unit -> 'a

> let test() : Account = getnull();;
val test : unit -> Account

Yes, I know it's not null, but default, but I think the purpose is clear enough.

FSharp
Tuesday, November 25, 2008 2:16:33 AM UTC  #    Comments [0]  |  Trackback

# Saturday, November 08, 2008
Software protection
I've been meaning to write about this for a while. It's a very simple topic, but developers get all emotional and stop being rational as soon as the magic "code protection" and "piracy" words get invoked. I'd like to say I'm not promoting copyright infringement nor saying developers don't deserve to be compensated for their work. Now that that's out of the way...

The two things most developers want to stop are unauthorized installing (license enforcement) and "code protection". Code protection is a very weak concept, mainly revolving around thinking people are gonna steal your precious algorithms. Protection is easy to deal with, so I'm going to cover that now.

Before VMs like .NET were popular, most of the code protection I've seen revolved around the code that implements the license enforcement. Developers would write all sorts of nasty-clever-clever code to make things hard for the crackers. You see this sometimes when you run an application and it complains about a debugger being installed or running. With Java and .NET, disassembly got easier. This made it extra easy to patch any license code, since the dissassembled code was in a high level language like IL. The response, and our first enemy of the day, was obfuscation.

Obfuscation takes your assembly and screws up all the metadata. On top of that, it might go and rewrite sections of your code to obfuscate the flow of the program, or perhaps indirectly load strings. The downside of course is that debugging gets really hard cause all your method names are now unreadable, reflection is broken, etc. Depending on the techniques an obfuscator uses, you can run into some other troubles. For instance, whatever obfuscator VistaDB uses is really broken, as it generates bad IL that just happens to work on MS CLR, but crashes (rightly so) on Mono. Not to mention that certain IL tricks are not verifiable, hence you can't use the code in lower-trust scenarios.

But what does obfuscation accomplish? Crackers ALWAYS win. Even the "most difficult" license system with hardware dongles and activation get cracked. The response I usually hear is "well it raises the bar". So. What. "Raising the bar" is totally pointless. Bruce Schneier talked about this.

For physical security, raising the bar is good in general. For example, if you buy a safe, it'll prevent a lot of thieves from getting to the valuables. Sure, there are higher level thieves, but you've weeded out a lot of the population around you, and the benefit is very real. Now some punk kids can't just go in and vandalize and "casually steal" your valuables.

But for computerized tech, the "bar" is the highest level attacker. If your valuable is "cracking my serial verification code", as soon as the "high level theif" cracks it, he can go write a simple program anyone can download. So the REAL bar is "user googles for a crack". That's what needs to sink past all the emotional nonsense developers go through when protecting their code. No matter what kind of complex protection schemes you put in, then obfuscate it on top of that, if the product has value, _someone_ will crack it, and all your users can just download the crack.

This isn't a maybe, this isn't a "possibly", this isn't theoretic, this is the exact reality. There is *nothing* you as a developer can do to prevent this (apart from make your product suck so much no one cares). [If there is, I'd love to hear it.]

So, obfuscation has zero value in preventing cracks, serials from getting out. And it has downsides. Just read the VistaDB blogs/forums to see real world problems only because they use an obfuscator.

What about "protecting special algorithms"? From who? If your competitors are good, they'll figure things out regardless. If they suck, they won't be able to do much with it anyways. I think the biggest threat is some overseas group disassembling your code, slapping their logos on it, and reselling it. That's a clear and obvious loss if they are making sales. But, obfuscation isn't really going to stop it, just raise the bar a tiny bit. In this case, since you're dealing with a limited number of "pirate companies" that exist for profit, perhaps obfuscating has a bit of value. But think: If someone can not know your source code, not be able to provide support, etc. etc., but can still outsell you and your marketing, perhaps you have business issues.

The one other place I hear people using obfuscation is to protect an app from "casual hacking". WTF does that mean? You mean you're afraid your sales clerk might decompile the PoS application, but give up quickly? You think it means you can safely store passwords in the binary? I'm not sure what such developers are thinking, but I'm guessing they did a poor security analysis of the situation.

As a side note, this is not particular to VM platforms like Java and .NET. Check out Hex Rays. They do a fine job *decompiling* optimized native code. I've seen it in action; it makes it easy to take any native app, decompile it, figure it out, then work with the assembly code. So these .NET devs thinking they are so leet cause Reflector messes up and hence no one can figure it out... sigh.

Finally, a nice real-live demo. Look at Spore and other games using heavy DRM and protection mechanisms. Obviously Eletronic Arts has an unlimited budget for getting the "best" type of protection. Yet the protection proved utterly useless against piracy. Just goto ThePirateBay.org and search. Yet they certainly introduced more bugs and user hate. (Of course, the REAL motive behind such DRM is killing the used games market. For this, all they need is stuff that honest users won't break.)

P.S. The reason I finally wrote all this is because VistaDB just took the silliness to the next level. I got their 3.4 Trial, but it crashes on Mono because the obfuscator emits totally invalid IL code. Their official response was that Trials arent tested on Mono. I bought the product and the "stable" builds still have the same busted IL code. Awesome protection; stopping paying users from using the software rocks!

I suppose I could understand IF they had some awesome trade secrets. BUT, they provide a source code license. So an evil VistaDB competitor just buys a source code license to get all the details. How is obfuscation helping ANYONE here? (Note the runtime has no licensing; only the developer install.)

Code | Security
Saturday, November 08, 2008 12:06:00 AM UTC  #    Comments [4]  |  Trackback

# Monday, November 03, 2008
Common Conceptual Issues with F#
Several times I've had to explain a few basic F# things to new users. I ran into some of these too when I first looked at F#. If you've been looking at F#, but some things still don't click, I hope this will help break the chains of C/imperative languages. I'm assuming you know a bit about F#, perhaps from the Quick Tour or the F# Tutorial file that comes with the Visual Studio integration.

Functions always take and receive exactly one argument.
Despite anything you see, in F#, there's only one argument, and only one return value. Let's look at how this plays out (I suggest firing up F# interactive and playing around):

> let inc x = x + 1;;
val inc : int -> int

> let add x y = x + y;;
val add : int -> int -> int

[I like to read -> as "to", because it's short and sweet.] So, for the case of "inc x", we see the signature is quite simple and expected "int to int". But for add? We see "int to int to int". What this actually means is "a function that takes an integer, and returns a function that takes an integer and returns an integer".

The same signature in C# would look like this: Func<int, <Func<int,int>>> or "Func<int,int> Add(int x)". [Some day, think how odd it is to have two ways of representing the function.] So when we call the F# function, we're really passing in the first argument, getting a new function, then applying the second argument. Something like this:

1. add x y
2. (add x) y
3. closure1 y
4. final-result

"closure1" is what we get from the "partial" application of add. I'll touch on this in a minute.

So how do we _really_ pass in two arguments at once? We put two arguments into one value, a tuple. We write it like this:

> let add (x, y) = x + y;;
val add : int * int -> int

Notice the new type signature: "int by int to int" or "int int tuple to int". In C#, this would be:

int add(Tuple<int,int> arguments) { return Tuple.A + Tuple.B; }

But F# nicely provides support for tuples [via pattern matching], so we can write them much more naturally. This syntax also works on the way out:

> let pow23 x = x*x, x*x*x;;
val pow23 : int -> int * int
// "int to int int tuple"

> let square, cube = pow23 7;;
val square : int
val cube : int

> square, cube;;
val it : int * int = (49, 343)


There's never no value
Another common typo/misunderstanding is when creating a function with no arguments:

> let sayHi = printfn "Hi";;
val sayHi : unit

Hi
> sayHi;;
val it : unit = ()

Oh, what happened here? Remember how _everything_ takes and returns exactly one value? The same is true of functions that "don't return a value", such as printf. Instead of "not returning a value" like C's void, functions return a special type called unit with one value, ().

Armed with this, let's see what the sayHi definition actually says. It says "let a value called sayHi equal to the result of printfn". Well, since the result of printfn is unit, sayHi becomes unit. The execution happens immediately, and subsequent uses of sayHi just get the value, unit.

To actually do what we want, we need to take an argument. Then F# knows we're a function value:

> let sayHi() = printfn "Hi";;
val sayHi : unit -> unit

> sayHi;;
val it : (unit -> unit) = <fun:clo@0>

> sayHi();;
Hi
val it : unit = ()


By explicitly taking a unit parameter, sayHi becomes a function (type unit to unit). Notice that if we just write "sayHi", we're just going to get the _value_ of it (a function), not apply (execute) the function.

Partial application
OK, so a side effect of this system of "take on param and return a function that takes the next", is that we can compose with just parts of a function. For instance, to write an increment function, we can do this:

> let add x y = x + y;;
val add : int -> int -> int

> let inc x = add x 1;;
val inc : int -> int


So far, nothing interesting. However, we can also write this more effectively:

> let inc = add 1;;
val inc : (int -> int)


Aha! What's happening here is what happens "secretly" each time we call add with 2 parameters. We're just using the intermediate result, the closure of "add 1", and assigning that function value to inc. In C#, it'd be something like this:
public static Func<int, int> Add(int x) {
    return y => x + y;
}
public static Func<int, int> Inc = Add(1);
// and "full application" of add looks like this: Add(1)(2)
[BTW, this isn't really currying in F#. Currying is taking a method of type "a' * 'b -> 'c" and turning it into "'a -> 'b -> 'c". Since F# methods are "automatically curried", there's no need for a "curry" step (well, except perhaps when using .NET methods, which are always "tupled", but that's another story).]

[Side note: a function like "add" is superfluous, because in F#, operators are functions:
> (+);;
val it : (int -> int -> int) = <fun:clo@18>
> let inc = (+) 1;;
val inc : (int -> int)
]

The pipeline


F# appears to have all this complicated syntax, with |> <| >> << and so on. But, these operators are defined in F# code, and follow some basic rules. They aren't magic or have any special compiler support.

The most important function operator is |>. A quick search of the F# source shows:
C:\Program Files\FSharp-1.9.6.2\source\fsharp\FSharp.Core\prim-types.fs(2062):       
        let inline (|>) x f = f x


(Operators are surrounded in parentheses to define them and to use them as functions with prefix notation.)
The type signature is:
> (|>);;
val it : ('a -> ('a -> 'b) -> 'b) = <fun:clo@23>

This is "alpha to a function alpha to beta to beta". That probably didn't help. Perhaps looking at the type of function application will help:

> let apply f x = f x;;
val apply : ('a -> 'b) -> 'a -> 'b


This demonstrates that a function application is really just taking a function "alpha to beta", giving it an alpha, and getting a beta. [I'm open to suggestions on better ways to pronounce type arguments.]

So, glance back up at the pipeline operator. We can see it's really just function application _in reverse_. What is the use of such a construct? If you've used C# 3.0's LINQ extension methods or a Unix shell, you probably already know. By reversing the function application, we can write things in a much more natural order. To modify something from the F# Quick Tour:

> let filterTypes name =
-   System.AppDomain.CurrentDomain.GetAssemblies()
-   |> Seq.map (fun a -> a.GetTypes()) |> Seq.concat
-   |> Seq.map (fun a -> a.Name)
-   |> Seq.filter (fun s-> s.Contains name);;

val filterTypes : string -> seq<string>

> filterTypes "Coll";;
val it : seq<string>
= seq ["ICollection"; "EvidenceCollection"; "GCCollectionMode"; "CollectionBase"; ...]
>


Now, this is a bit embarrassing, but this took me a long time to get. I stared at this and re-read the F# manual for probably an hour. How could something so simple do such "complex" stuff?? Once I finally got used to the idea of functions being normal values, and the whole "one arg one value" bit, it snapped together. The other operators (<|, >>, <<) are pretty easy to follow once the basics are understood (going through prim-types.fs is a great experience).

What else?
I hope this all helps fit some pieces together. Feel free to use the MSN thing on the side of my site to ask questions or give me suggestions. Thanks!

Edit: Also check out F# function types: fun with tuples and currying (From an F# team member's blog.)


Code | FSharp
Monday, November 03, 2008 9:07:15 PM UTC  #    Comments [3]  |  Trackback