On Entitlements, Coral8 CCL, SQL, and LINQ

One of the nice things about having kids that are a little older is that it gives me time to putter around on my laptop while watching the Yankees games on TV. I am not doing day-to-day development in Coral8, having handed that aspect of the project over to HH. However, I wanted to see if the entitlements processing of our system could be done in Coral8, which made logical sense.

A brief recap: Our CEP system takes a number of atomic events, puts them through the Coral8 cruncher, and produces derived events. However, we don’t want everyone to see these derived events. We might have information in a derived event that a Prop Trader should not see, or we might have information about a certain financial sector that should be hidden from someone on a trading desk who does not cover that sector.

In addition, we have different kinds of notification mechanisms (GUIs, message buses, email, chat, SMS, etc) that should be utilized depending on the severity level of an event. We don’t want to send several hundred emails to a trader for informational events. However, we might want to email and SMS a trader if we have a “red alert” type of event.

So, we will turn to a familiar pattern called the Recipient List. This is one of the well-documented patterns in the book Enterprise Integration Patterns. I get a good amount of email that asks me for advice on becoming a trading systems developer. My advice is to run, not walk, to this website and book. Most of this stuff is old hack to experienced trading systems developers, but the use cases (especially the one by Jonathan Simon) is worth its weight in gold.

We have come up with a schema and database of entitlement information that marries our users/groups list, severity levels, notification mechanisms, and derived events. As every derived event gets generated by our CEP system, we want to put it through the “entitlements grinder” and come up with a Recipient List of who can see what information in the message, and how they want to be notified of its occurrence.

This seems to be a perfect task for a CEP engine. It can be just one more additional “enrichment filter” whose input we attach to the output of the derived event stream. The output of this enrichment filter consists of the (possibly modified) derived event and the Recipient List.

As an initial step, we implemented the Recipient List Generator as a single SQL query using SQL Server 2005. It is a single query that consists of 2 inner joins and 2 outer joins. It works fairly well.

When I was watching the Yankees game yesterday, I tried porting this query to Coral8. I could not get any variation of this query to compile properly, and when I tried to decompose the query into 4 streams, I got totally different results that what SQL Server gave me. Ideally, “Streaming SQL” languages should be a superset of SQL92. So, in Coral8, if I mirror each SQL Server table as a Coral8 Window with a “KEEP ALL” property, then I should be able to use my SQL Query directly. I would like to do something like this:

INSERT INTO RecipientListOutputStream
SELECT [my original SQL query]

I have given the guys from Coral8 a homework assignment, and asked them to try to take my query and schema and make it work in Coral8.

So, after a frustrating two hours in which I tried to decipher the Coral8 reference documentation and compiler, I decided to turn to another strategy. For shits-and-giggles, I decided to try to write my SQL query in LINQ. I downloaded the experimental Visual LINQ Query Builder from http://code.msdn.microsoft.com/vlinq. I created a new Visual Studio project, pointed the LINQ data sources to the entitlements database, and started plugging away on the VLINQ. In about ten minutes, I had a full LINQ query that implemented my SQL Server query.

(Note: VLINQ was fairly slow on my laptop, and I soon gave up on it, preferring to code the query in LINQ myself. However, Coral8 and other CEP vendors should look at it as a prototype of a visual code generator.)

LINQ has a lot of goodness to it. LINQ is pervasive, and all flavors of LINQ are being developed. I can very well imagine that Microsoft is looking at versions of LINQ that could handle streaming data. Right now, I think that it would be fairly easy to hook up LINQ queries in a pipeline that would handle simple queries on streaming data. Adding streaming SQL constructs is very doable.

If Microsoft was to come out with a Streaming LINQ that is available as part of .NET, how would this affect the world of CEP? An immediate casualty might be NEsper, but that’s OK, since NEsper is just Aaron’s side project right now. But, longer term, I think that a combination of WCF, Streaming LINQ, and a version of Microsoft Analysis Services that was further geared to real-time streams would be a killer to the rest of the CEP industry. (Of course, technology is one thing. Getting all of those Java and Linux bigots over to .Net is another thing.)

©2008 Marc Adler - All Rights Reserved