The other day I got an interesting email.  A client who we had written a payment processing system was having trouble with MQSeries (shudder), and was pinning the blame on our system.  The issue was that when MQSeries dispatched a message that eventually timed out (30 seconds), the payment server blocked until the timeout was returned.

At first, we thought that MQSeries was to blame.  After all, it's a most annoying piece of software (it starts up around 30 processes for some reason).  There's a reason that IBM's consulting division makes so much money :).  But we thought that serializing all connections was a bit bad, even for IBM.

Remoting seemed unlikely.  After all, how could anything be scalable if remoting used only one thread?  After some tests, we found out the cause.

Apparently there is a bug in remoting.  Or perhaps it's by design.  The result is that it appears as if remoting tries to keep one and only one thread per CPU active.  This could be a performance benefit, assuming your thread is using the CPU.  However, when your thread blocks, for instance, calling Thread.Sleep for more than a few hundred miliseconds, or calling WaitOne with an indefinite timout, remoting releases a new thread.  This is actually a decent scenario for most things, since it assures your CPUs are operating at the highest efficiency.

The problem was the MQSeries was being called via a MC++ interop library.  (IBM didn't have a .NET library when we wrote this, and apparently their new .NET library for MQSeries is pretty bad.)  Since it's unknown what happens inside of a P/Invoke request, no thread is released.

The ideal workaround would be to let remoting know that a new thread should be released.  However, I'm unsure of how to do this (or if it's even possible), and thus, the workaround is to manually multithread your server-side code where needed.