More Cloudy Thoughts

It seems that my inelegance in describing my idea of the event cloud has spurred some debate between Greg and Hans. I am going to try to clarify my thoughts around the event cloud, and see if it makes more sense.

I don’t want to give any of our “secret sauce” away in my postings, so I will try to map my thinking from the Equities Trading domain to another domain that I am not familiar with …. The domain of building security. (Apologies in advance to all of the sleuths who are reading this. My use cases will probably malign your esteemed field of study.)

Let’s say that we are writing an enterprise-wide CEP application for our MegaBank that will monitor our security system so that those nosey parkers from StanLehGoldBar Inc don’t break into our building and steal our secrets.

We are going to make the distinction between Atomic Events and Derived Events. A Derived Event is generated when something interesting is detected from the monitoring of one or more atomic events.

An atomic event may be saved in our CEP engine, depending on the use case. A derived event will be definitely be saved by our CEP system. Since a derived event represents an “interesting” condition, we may want to do reporting and analysis on derived events. We may also want to do some sophisticated searching of our derived events. In my mind, the persisted atomic and derived events form our event cloud.

Two or more derived events can be combined to form a new derived event. A derived event can be combined with an atomic event to form a new derived event. Let me illustrate this.

We have a lot of atomic events that come through our CEP system. Every person that passes through MegaBank’s doors generates a PersonEnteredBuilding atomic event. (This is analogous to a market data event.) We have a static database of all of MegaBank’s employees, and when the person in a PersonEnteredBuilding event does not match an entry in our employee database, the CEP system generates a NonEmployeeEntered derived event. This derived event will be persisted in our CEP engine.

We also have a derived PersonLeftBuilding event generated whenever a person leaves the building. This event is generated

(Let’s fantasize a bit, and assume we do a retina scan of everyone entering the building. Let’s stretch our imaginations a bit and say that we have a retina scan of all StanLehGoldBar Inc employees)

Let’s say that we do a join between a NonEmployeeEntered event and the static source of StanLehGoldBar employees. We can generate a CompetitorInBuilding derived event. This derived event will be saved in the CEP engine too.

We monitor all doors to our building. Whenever someone goes through the doors of the Equities trading floor, we generate a TradingFloorEntered atomic event. If someone goes through the trading floor doors for the first time of the day after 6:00 PM, we might want to consider that action to be a suspicious activity, and we want to generate a PossibleIntruderOnTradingFloor derived event.

Now, some smart security guard who is a user of our CEP system wants to create a new derived event. He wants to see if we have a competitor roaming around our Trading Floor at night. So, the security guard goes into our CEP GUI and creates a new, custom derived event called CompetitorOnTradingFloorAfterHours that gets generated 1) if we have a CompetitorInBuilding derived event 2) that does not have a “cancelling” PersonLeftBuilding event, 3) combined with a PossibleIntruderOnTradingFloor event. The fact that various events are “floating” around our event cloud makes it possible to easily create new derived events on the fly.

The security guard might also want to do a query of our event cloud to see if the competitor was “casing the joint” prior to the intrusion into the trading floor at night. The security guard might want to see how many times that particular competitor visited MegaBank over the past month. He might want to see how many times the competitor visited the trading floor during normal business hours. This sounds like the kind of query that you would do with a Data Warehouse, not a CEP engine.

So, we need to be able to represent the event cloud in an efficient way in the CEP engine. We need to be able to create new derived events dynamically, while the CEP engine is running. We need to be able to do ad-hoc analysis of the event cloud to improve our situational awareness. We need to be able to create new and interesting visualizations that let the user peek into what is going on inside the cloud.

Interesting stuff, right?

Tim Bass gives a list of techniques that can be used to perform analysis of events. These techniques include:

• Rule-Based Inference
  
• Bayesian Belief Networks (Bayes Nets)
  
• Dempster-Shafer’s Method
  
• Adaptive Neural Networks
  
• Cluster Analysis
  
• State-Vector Estimation
  

I have to admit that I have not delved into these areas before. However, we just hired someone with a BioInformatics background who we hope could do this stuff in his sleep. Who would have thought that, working for MegaBank, I would be exposed to such interesting areas of study?


©2008 Marc Adler - All Rights Reserved

How to do an UPSERT in Coral8

Assume that the LastTrades window has the retention policy of KEEP LAST PER Symbol.

Here is some code (provided by Mark of Coral8) to do an upsert.

INSERT INTO
LastTrades
SELECT
SPC.symbol,
SPC.price,
If LT.volume Is Null Then 0 Else LT.volume End If
FROM
StreamPriceCorrections SPC
Left Outer Join LastTrades LT ON SPC.Symbol = LT.Symbol;


If we need to do an update rather than an upsert, this piece of code works:


INSERT INTO
LastTrades
SELECT
SPC.symbol,
SPC.price,
LT.volume
FROM
StreamPriceCorrections SPC, LastTrades LT
ON SPC.Symbol = LT.Symbol;


©2008 Marc Adler - All Rights Reserved

Addendum - Cloudy Thinking

In my previous post, I don't think that I stated my question properly.

I was asking how to actually implement the Event Cloud in a Streaming SQL-like CEP engine, so that we can build up hierarchies of events. I was looking for schemas, SQL statements, best practices, etc.

I am well aware of POSETS, but what might be easy to implement in C# or Java-based data structures is not that easy in a quasi-relational system such as CORAL8.

So, I was hunting for advice from people who might have implemented event clouds in Coral8, Streambase, and Aleri, all three which are based on SQL.

I will follow up this post with a more concrete example.

©2008 Marc Adler - All Rights Reserved

Cloudy Thinking

One of the things that has been on my mind recently is how to represent the "event cloud".

One of the buzzphrases that comes out of the CEP movement is the term "event cloud". This is the huge amalgamation of all of the events that flows through your event processing system.

This week, I had the opportunity to talk with Mary Knox of the Gartner Group. Mary follows the world of CEP. Just like many of the executives at the various CEP companies have told me, Mary confirms that many companies in finance are using CEP for simple streams of processing .... a small algo trading system here, a pricing engine there, a portfolio analysis system in another place. Most shops seem to be "under-utilizing" their CEP engines. Our effort is probably one of the few out there that will really attempt to poke holes in the hype surrounding CEP.... either our application will validate the hype, or all of the CEP engines will flame out in a blaze of glory.

Mary has not heard of too many financial companies who are implementing the event cloud, so I wonder if we are breaking new ground in that area. We know what an event cloud is and what it is supposed to do, but how do we actually implement the event cloud in a system like Coral8 or Streambase? Are relational tables/windows the best way to represent the cloud? And, what about hierarchies of events that are derived from other events? How can be best represent that complex graph of inter-relationships with SQL-ish windows?

Is anyone out there doing similar work with the cloud?


©2008 Marc Adler - All Rights Reserved

Coral8 is Our Choice or “How the hell did we get here?”

When we went down to Orlando last fall to attend the Gartner Summit on Complex Event Processing, we went with eyes wide open. We were new to the domain of CEP, and one our missions was to try to pick a vendor for the CEP engine that would drive our efforts to produce a major CEP system for our Equities business.

There were a bunch of event-processing systems that were not under consideration because it seemed that they had moved into the strictly vertical area of Algo Trading. These CEP systems included Truviso and Skylar. We needed a general-purpose CEP system, and we wanted to only consider systems that still had a generalist product. An exception to this rule was Aleri, a company who had just come out with a Liquidity Management System as a separate product. We thought that Aleri would still keep its focus on the core CEP engine, so it warranted inclusion of our evaluation.

Apama fell into the Algo trading vertical, but Apama still has a general purpose CEP engine. However, when we tried to evaluate Apama, we were told that we had to go through the dog-and-pony marketing show, something that we did not want to do. I am not sure if this requirement was brought on by the purchase of Apama by Progress Software, a company who I think of as being Computer Associates Lite. I was also told about some interesting experiences between Apama and a major bank by a former colleague of mine whose opinion I trust, and this also influenced by decision to evaluate Apama. This was unfortunate, as I happen to side more with Apama on the whole EPL vs Stream SQL debate.

This left four systems: Streambase, Coral8, Esper, and Aleri.

Coral8 had always been the front-runner, mostly due to recommendations by some former colleagues who were at Merrill Lynch. I had always liked the “vibe” surrounding Coral8, and their openness at giving out eval copies of their software.

Readers of my blog know that I had strong negative opinions about Streambase because of the aggressiveness of their marketing department, an opinion which was shared by a lot of people out there. Nevertheless, their new CEO, Chris Risley, contacted me personally and told me that he had addressed my concerns. After Chris and Richard Tibbetts came down to NYC to meet with me, we decided to include Streambase in the evaluation.

I really wanted to try Esper, but there were a few things that worked against them. The primary factor was that the .NET version, NEsper, was something that was developed by the hard-working Aaron Crackajaxx for his business needs, and did not seem to be part of the mainline Esper product line. We are a .NET shop here, and we needed a product that supported .NET as a first-class citizen. If Aaron decided to become disinterested in Nesper, or if he moved on to another company, then where would we be? We also preferred a product that had an entire ecosystem built around it. So, we passed on Esper. However, Esper is still very much on my radar screen, and I am interested to see how Thomas continues to develop the company and the product.

We spent a good deal of time evaluating Aleri, and most of these experiences were detailed in past entries in this blog. We really wanted to see Aleri succeed, as they were a local company, staffed with a lot of very smart and gentile ex Bell Lab-ers. However, we felt that their product was not ready for us, mostly because of what I called the “spit and polish” issues. I won’t rehash the details now, but if you are interested, please go back and read the old entries in this blog. The areas that needed improvement in the Aleri product were the Aleri Studio, the documentation, and the integration of external data sources.

I met a good deal with Don DeLoach, the CEO of Aleri, and the one positive that will come from my rejection of Aleri is a renewed focus by Aleri on the aesthetics of their product. You can already see these efforts by reading the new Aleri Blog. From what Don had told me a few months ago, their 3.0 product will start to focus on easier integration of external data sources, and will have much improved documentation. I look forward to seeing their efforts come into fruition.

Coral8 was always the front-runner in our evaluation. Their engine is written in C++. They had a decent .NET SDK that let you build out-of-process adapters in C#, and also let you interact with the internals of the Coral8 engine. Their documentation was good, although a bit obtuse at times, and the documentation was backed up by a ton of whitepapers that are available on their website. Their Coral8 Studio gives you a source code view of development, and the GUI part of the Studio is updated after every compile of the source. The CEO of the company was the person who created Crystal Reports, and knows what it takes to build a software company. But, most of all, their CTO and President interacted with us all of the time, and was extremely receptive to our ideas on improving his product. I like when a CTO and the pre-sales engineer send me mail on a Sunday morning!

Streambase was a strong contender. They had some great features in the product that Coral8 has only just come out with (ie: windows that are bucketed by column value). Their GUI is very strong, and their documentation and tutorials are first class. However, I have to say that the interest that Coral8 has shown in our success, and the availability of their CTO was what tipped the odds in Coral8’s favor.

By now, you must be saying to yourself “Where’s the meat?” Didn’t we try to soak and stress the various engines? Didn’t we have an OPRA feed running into the engines in an effort to break them? Didn’t we monitor the use of the CPU and other computing resources? Well … no. To tell you the truth, we were relying on STAC Research to try to do that job for us. STAC is only now just starting to get up to speed in the CEP world, and we will be monitoring their efforts in this space. The general feeling is that most of these CEP engines perform in roughly the same manner, and if one of the CEP engines is 5% faster than Coral8, it is not going to sway our decision, since we are not that concerned right now with super low latency. We are more concerned with the intangibles; responsiveness of the support organization, evolution of the product (and our input into the roadmap), support for .NET as a first-class citizen, stability of the company, etc.

Despite choosing Coral8, we have been careful in our architecture to abstract the specific CEP engine, and no external system will know that Coral8 is driving our CEP system. In the same way that CEP engines have abstracted datasources by using pluggable adapters, we have abstracted the CEP engine. Yes, we have chosen Coral8, and so far, we are satisfied by our choice. But, we are also keeping our eyes open for how the other products evolve in the world of Complex Event Processing.



©2008 Marc Adler - All Rights Reserved

Of Webcasts, Vendors, and Perceptions

Some people have pinged me to find out why I never appeared on the webcast with Coral8 a few weeks ago. Basically, it was because of a mess-up by the folks at Incisive Media.

You had to call a certain phone number to participate in the webcast. The operator dutifully asked me for my name and my company. I was then transferred to what turned out to be a listen-only line which enabled me to hear the speakers, but prevented them from hearing me. After the presenter from The Aite Group gave his spiel, the presenters asked if I was on the line. From my perch on the trading floor, I screamed into the phone “I’m here! I’m here!” … but since I was on a listen-only line, nobody on the other side could hear me. I furiously sent emails to the guys at Coral8 and Incisive Media, but all were oblivious. I finally hung up, called back the operators, and explained that I was one of the presenters, and the operator put me on the presenters-only line, but by that time, John Morrell had started his 30-minute pitch for Coral8. I had another meeting scheduled for noon, so I had to leave the webcast.

I have to admit that I had no idea that I was going to be part of a Coral8 infomercial, and if I did, I would not have been part of the webcast. As a good member of MegaBank, I need to be extremely careful about the PERCEPTION of a relationship between a specific vendor and myself. In fact, recently, I felt that I had to cancel the recording of a CEP podcast when I found out that the people who set up the podcast were bringing the Chief Architect of BEA Systems to the interview to ask questions. Not only do I not want to be perceived as a shill for a specific vendor, but I do not want my name on a banner next to the name of a company whose products I am not using. Even if the BEA architect’s questions to me were vendor-neutral, having my name on a podcast that has the BEA name on it as the sponsor would imperceptibly tie me into BEA.

The SOA/Web Services conference that I spoke at a few weeks ago had no vendor tie-in. I was on a panel with someone from Bank of America and from Google. There were no SOA/WebServices vendors on the panel. I am also scheduled to speak about CEP at a big conference in a few months, and I will be careful not to mention the specific technology stack that we are using. I don’t mind being mentioned as a reference customer for a vendor, under the conditions that

- I am actually using the product
- I am completely satisfied with the product
- My current employer gives me permission

Being a member of the management layer at MegaBank and blogging at the same time is a tricky proposition. If you read something here, you need to be thoroughly convinced that I am not shilling for any vendor. I disparage all vendors equally. I am not like certain consulting firms, who have relationships with every vendor and will never render a negative opinion about any of those vendors IN PUBLIC.

©2008 Marc Adler - All Rights Reserved

KDB is Free for Personal Use

I haven't blogged in about a month ... I have a ton to say, but I have just been too busy with the new employees who have joined my Complex Event Processing team (welcome Hanno, Scott and Feng). But this great news from Tenerife Joel is just too good to pass up.

On this blog a few month ago, we got on the case of KX Systems for making it virtually impossible to download, evaluate and learn if you were not a member of a large financial institution. Several people commented here that they really wanted to check out KDB, but they did not know how to get access to it.

I guess that Simon and Niall have seen the light, and are now making KDB available for personal use.

I would like to take all of the credit for this revelation, but I suspect that one of the motivating factors for KX Systems was the free, unrestricted availability of Coral8 for personal use. Any random person can download a personal-use, single-CPU version of the Coral8 server and development studio for no charge .... this version does not have any restrictions nor any time limits (are you listening, Aleri?).

During our evaluation of CEP systems, we found that Coral8 was the only company that did not put up any barriers to evaluation. Aleri had an annoying time limit. Apama wouldn't even let us download anything without having to run through their marketing gauntlet. Streambase directly gave us a copy of their product, so I am not sure if Streambase has any barriers to evaluation.

All of the CEP vendors face the conundrum of giving out unrestricted copies of their software vs fully qualifying prospective customers. I can understand this from the standpoint of support ... your technical support staff has a limited amount of time, and it usually has to be spent supporting the larger institutions, the ones who will most likely be dishing out several hundred thousand dollars for a license. But, today's independent hacker might be tomorrow's corporate developer .... and goodwill goes a long way.

So, congrats to Simon and Niall from KX. Here's hoping that a new generation of KDB developers will come out of this effort, and as a result, a wider set of tools is available for KDB.

©2008 Marc Adler - All Rights Reserved

CEP Webcast

At 11:00-12:00 (EDT) on Tuesday March 11, I am going to be on Webcast dealing with CEP. Sponsored by Waters Magazine and featuring John Morrell from Coral8. For more info:

http://www.coral8.com/news/pr/184.html


©2008 Marc Adler - All Rights Reserved

Visualization Vendors and .NET

We have seen a whole slew of enterprise visualization products lately ... Tableau, Spotfire, RTView, and others that I cannot talk about.

Almost none of these vendors have components that can be embedded in a .NET UserControl. Almost all of these visualization suites are complete desktop or web-based apps. Many of these DO NOT deal with real-time, streaming data.

One of the vendors that has addressed the .NET component market is Panopticon. This company is slowly moving beyond there core competency of Heatmaps and are staring to offer various other useful visualizations.

To the visualization vendors who call me and are puzzled why I have not considered their products ... we need .NET components. We have a container that hosts different "applets" that run at the same time, each applet containing a .NET UserControl that, in turn, contain different visualizations.

©2008 Marc Adler - All Rights Reserved

New Coral8 5.2 Released

This is from the Coral8 website. You may recognize some of these enhancements as things that may have been deficiencies when I detailed my use case a few months. Some of the enhancements catch Coral8 up to some of the features that its competitors have. (ie: Bucket Windows)

Good job, boys!

----------------------------------------------------

Coral8 Engine Release 5.2 Now Available

Coral8 is pleased to announce the general availability of the Coral8 Engine Release 5.2. It is available for immediate download. The Coral8 Engine

Release 5.2 contains a wide array of new features and enhancements including:

• Queryable (Public) Windows


• Automatic Event Causality Tracking in CCL Pattern Queries


• Single-Match Patterns


• "Buckets" Window


• Edit and Debug Perspectives in Studio


• PAM User-Authentication Plugin


• Coral8 Portal


• Custom Charts API


• Streamlined UI for Managing Dashboards and Users


• Java 1.5 SDK (Preview)


• Messaging Layer Improvements


• Access-Control File Changes

©2008 Marc Adler - All Rights Reserved

2007 Bonus Satisfaction Poll

Here Is The City has its annual Bonus Satisfaction Poll going on. Here are some updates. Looks like the Back Office was not too happy at most firms.

©2008 Marc Adler - All Rights Reserved

CEP Engines and Object Caches

I am curious if any of the CEP vendors have support for getting data from distributed object caches. It is easy to feed data into a CEP engine through JMS (Tibco EMS), sockets, flat files, standard databases (SS, Sybase, Oracle, DB2). But, has anyone integrated an object cache like Gemfire, Tangasol, and Gigaspaces with any CEP engine?

I also notice that Gemfire RTE claims to have CEP capabilities.


©2008 Marc Adler - All Rights Reserved

Of Pipes, CEP, and Traders

In our Complex Event Processing system, we will eventually want to be able to give a front-end to our traders which will let the trader develop ad-hoc streaming queries on real-time streams. We also want to allow the traders to implement their own analytics by doing back-testing on several years’ worth of data.

I have been looking at Yahoo Pipes a bit recently, and it is conceptually the same thing as the kind of GUI that I have been envisioning for the traders. In Yahoo Pipes, you can take the output of one or more high-level objects and pipe it into the inputs of another high-level object.

Here is an example of a Yahoo Pipes gadget:




In our simple use case, let’s say that we define high-level objects that are of interest to traders. Such objects could be

a real-time market data stream,
a historical market data stream,
a real time order flow stream and cache,
a historical order flow stream,
a ticker-to-sector mapper and filter, and
some calculation modules.

These are the kinds of “queries” that we built in the Complex Event Processing engines (Streambase, Coral8, Aleri, NEsper) during our evaluation process. However, just like Domain-Specific Languages (DSL) can be created to make developing applications in a certain domain easier, we can make a domain-specific version of Yahoo Pipes that is geared for analyzing Equities data.

Once the trader has put together his Pipes, the result can be translated into one of the languages that the CEP engine supports. If we encapsulate the code generation module, then we can have code generators for Coral8, Streambase, Aleri, NEsper, and others. I am pretty sure that all of these engines allow you to dynamically register and unregister queries. You can dynamically read the status and output of the CEP engine and feed it back into the GUI.

Aleri and Streambase have GUI builders for their developers that look a lot like Yahoo Pipes. However, the GUIs are not packaged as toolkits; in other words, you could not take the Streambase GUI and adapt it for your own needs. Coral8 does not really have a fancy GUI builder; as mentioned here before, their GUI is fairly spartan. NEsper/Esper does not have any kind of GUI to speak of, as their model of embedding the engine inside of an application is different that the model used by the other vendors.

©2008 Marc Adler - All Rights Reserved

Digipede Video

John Powers of Digipede has weighed in with a short, 4 minute, painless video to watch on how you can treat a single multi-core machine as a grid.

http://powersunfiltered.com/2008/02/19/multi-core-and-grid-computing-new-digipede-video-shows-the-way/

John is one of the good guys of our industry.


©2008 Marc Adler - All Rights Reserved

IT and Pay Practices

Jules sent me this interesting link about Gartner's study of IT pay practices.

One of the things that I noticed during this last bonus season was, despite all of the sturm and drung in the news, companies managed to come up with the money to handsomely reward its most valued IT employees. Almost all of the people who I know and respect on Wall Street and in The City managed to receive upside surprises on their bonus numbers this year (or if they didn't get it all in bonus, then they got increases in their base).

All of the companies on Wall Street are moving at a rapid pace to nearshore/rightshort/offshore a lot of their IT. This includes my company as well. DLG reports that nearshoring to Panama is working very well for his company. He goes down to Panama every few weeks, sees what his devs are working on, and then returns to the US with a disease that they now use penicillin to cure. But, how do you properly motivate these employees? As Garter says, "Performance management and reward systems will need to recognize and motivate the “out-of-sight” workforce."

©2008 Marc Adler - All Rights Reserved

Cayuga and SOA Conference

Thanks to Marco for a pointer to Cornell University's Cayuga Project. It's a CEP engine, it comes with source code, and it's free. Unfortunately, my megabank needs a full ecosystem, and the support model of an actual company, so this disqualifies Cayuga from further consideration. However, it would be interesting to see if the Cayuga people go off and form a commercial venture around this. (Jeremy, are you reading this?)

The SOA/WebServices panel was a success. Thanks to Tom Steinthal for inviting me to participate. A lot of the crowd was interested in CEP and the interaction of CEP and SOA. After the panel discussion, I was mobbed by a lot of vendors... something that I explicitly wanted to avoid. I would rather have been mobbed my my bank's competitors to engage in some general information sharing. I am very curious to see if CEP has moved out of the very "linear" world of algo trading and pricing.

©2008 Marc Adler - All Rights Reserved

Fixnetix vs Wombat

I have heard of some companies starting to look at Fixnetix as a possible replacement and/or alternative to Wombat for low-latency market data feeds.

Has anyone done a bakeoff between the two? How about Activ?

Wombat is starting to promote its stack that include Coral8 for CEP/latency monitoring and OneTick for a tick database. Reuters has the same with Streambase and Vhayu. This means that Fixnetix and Activ will probably be looking towards that direction as well. Can an alliance with Aleri be in the cards for any of these vendors?


©2008 Marc Adler - All Rights Reserved

I Will Be At 2008/Web Service Conference in NYC

I will be on a panel at the 2008 Web Services/SOA on Wall Street show on Monday, February 11th.

http://www.lighthouse-partners.com/wsonws/del_speakers.htm

The panel starts at 11:00 AM, and goes for about 45-50 minutes. Topics of interest are the increasing use of blogs, Wikis, LinkedIn, Facebook, etc on Wall Street.

Warning: Since I grew up on the tough streets of Jamaica, Queens, be warned that, if you throw tomatoes at me, you do so at your own risk.

Here is a blurb about the panel:

Beyond Web 2.0...What Enterprise 2.0 Is...And What It Means For Wall Street

Beyond Web 2.0, Enterprise 2.0 is about deploying these new technologies and social practices in a corporate business context.

This session will explore the drivers pushing Enterprise 2.0 adoption, survey relevant technologies, and discuss how Wall Street and the financial markets are benefitting.

Tom Steinthal, Managing Director, Financial Services, BSG Alliance (Moderator)
Marc Adler, Senior Vice President, Equities and Head of Complex Event Processing, Citigroup
Michael Ogrinz, Principal Architect for Global Markets, Bank of America
Jonathan Rochelle, Senior Product Manager, Google
Jason Wood, Head of Research, RT Capital Management and Enterprise IT Blogger


©2008 Marc Adler - All Rights Reserved

KDB Consulting Redux

I have gotten some private emails from people over the past few weeks who have been intrigued about my posts about KDB+, and wanting to know how to be a KDB+ consultant. It may be easier to apply to medical school and become a heart surgeon than it is to bootstrap yourself in a KDB consulting career.

I certainly do not hold myself out as a career counselor. You should not look to me as the key to future riches. All I do is observe the marketplace. I deal alot in hiring people for our own needs, and I get a lot of calls from recruiters. I have a lot of colleagues on Wall Street and in The City. So I am able to gather a lot of different data points, some of which I share on this blog.

Google everything you can about KDB+. Apply to KX Systems for access to the coding areas. Grab a copy of "Q for Mortals" by Jeff Borror. Find the pockets of KDB people in your company and get them to explain their source code for you.

If you have gotten this far....

You will need to get a copy of KDB+, which is not easy to get. Most likely, you cannot go onto LimeWire or Kazaa and find a copy of KDB floating around. Even if you did, you need to find a license key.

You need to go here, and think about how you will convince Niall Dalton or Simon Garland to let you play around with KDB. Or, wait for the next SIFMA conference, find the KDB booth, and offer to take the KDB guys to the lounge at the Hilton. Or, come up with a product that might compliment KDB (like Panopticon did) and try to show Simon that it will increase the popularity of KDB. Maybe an English-to-Q translator? Maybe a true ODBC driver for KDB? Maybe a Q code obsfuscator?

Another avenue is try to approach First Derivatives. They are the primary partner for KX Systems, and the Yin to KX Systems' Yang. Hint .. they love Guiness.

Lest you think that I am picking on KDB ... the same thing can apply to any niche product. I suppose that you would not be able to wake up one day and decide that you will be a Vhayu consultant. In order to specialize in any kind of niche product, you need to leverage the existing resources for that product. This is one of the advantages of being employees of large financial firms or large consulting firms that specialize in financial services.

You need to get yourself into the proper mindset to deal with KDB. It is a very spartan product, hand-tuned and lightning fast at processing real-time flows of tick and order data. The actual Q.exe engine is about a 150K file. You do not get any of the nice, cushy bloat that you have with something like SQL Server and Oracle. Working with KDB brings you back to the days when you were writing DOS TSRs using good old INT 21H (for those of you born after 1980, a TSR is a Terminate and Stay Resident program .... and DOS was an ancient operating system that made Microsoft lots of money).

As if you are not convinced already .... the following code is from a Q file called holiday.q. Once you understand this code, send email to Niall or Simon of KX Systems, or to the people at First Derivatives, and show them that you can write this stuff without even looking (and this is mild ... wait until you get into the juicy stuff).


/day:(day;year)
dy:{"D"$string[y],x}

/residue
r:{y-x*y div x}

/adjust sat/sun
a:{d+0^(x,1)r[7]d:dy[y]z}

/goto dayofweek
b:{d+r[7]x-d:dy[y]z}

/good friday(1900-2099)
g:{d+:e:r[7](6*d:r[30]24+19*a:r[19]x)+5+2*r[4;x]+2*r[7]x;dy["0320";x]+d-7*(d=35)(d=34)&(e=6)&a>10}

/nyse holidays
nyse:(a[2]"0101";b[2]"0115";b[2]"0215";g;b[2]"0525";a[-1]"0704";b[2]"0901";b[5]"1122";a[-1]"1225")

nyse@/:\:2007 2008

©2008 Marc Adler - All Rights Reserved

.NET Developer Wanted (No Consultants)

A colleague of mine will shortly be looking to hire a great .NET developer as a full-time employee. I am putting out this job req for them, in case you know anyone who is looking to make a switch in the next few weeks. This is a good group to work. The manager is very technical, and gives the developers a chance to play with the latest technology.

Please send all responses and resumes to me, and I will pass it on to him.

.NET Developer

We are seeking highly motivated candidates to create and deliver technology solutions and support for the XXXX business area. The new hire will join a “ground floor” opportunity, working with business clients, offering creative solutions to their needs, and implementing them.

The current technical platform includes C#/.NET, WinForms, ASP.NET, SQL Server 2005, Microsoft Business
Intelligence (Integration, Reporting, & Analysis Services), SharePoint, and Visual C++. A “Road Map” outlining a series of potential projects has recently been presented to senior business clients and has been well-received – the door is wide open for Technology to create the IT Work Slate for 2008, deliver on these commitments, and make a large positive impact on the XXXXX business’ bottom line revenue.

Job Responsibilities include:

• Key development on critical .NET-based projects


• Working on all aspects of these projects, including front end work in WinForms and ASP.NET as well as backend database work in SQL Server 2005


• Support and minor enhancements of the existing Visual C++ application until the new product is fully launched


• Balancing fast time to market (client delivery) with proper Governance, Compliance, and Audit mandates


• Fostering close relationships with other teams in XXXX Technology and throughout the Technology organization

Required Skills:

• Strong C#/.NET development skills


• Experience with data-driven .NET projects


• Comfortable with data binding techniques in ASP.Net


• Comfortable with data binding techniques in WinForms


• Experience or interest in Agile development (Scrum, XP, etc)


• Strong team player, collaborative


• Persistence - ability to get things done


• Worked in an investment banking environment

Preferred Skills:

• MS SQL Server 2005


• Familiarity with Visual C++ a plus


• Interest and ability to work with new technologies - Powershell, WPF*, WCF

Colin Clark is with Streambase - Where's my Finder's Fee?

A few week ago, this blog mentioned the news that Colin disbanded Kaskad, and was looking for new opportunities. I also proposed that Streambase hire Colin and some of his gang, as they were all in the Boston area together.

Lo and behold, Colin is now with Streambase.

An example of the wonders of the blogging network in action!

Also interesting that Bill Harts is on the Streambase Advisory Board. When I was consulting with Goldman in the late 1980's, I was friendly with Bill, as our groups interacted. I actually interviewed with Bill and Mike Dubnow for a job in Fischer Black's group, but mercifully, they spared me the pain of having to try to understand the world of the Quants (leaving me the opportunity to go full-time into Magma Systems after I left Goldman). Emmanuel Derman has written extensively about the goings-on in Black's little world.

Mike Dubnow eventually became the first Managing Director from the tech side at Goldman, and has since retired. I think that Mike was one of the creators of the SLANG language at Goldman.

On Wall Street, we have an incestuous little circle.

IDE for KDB+ Development

QKDT

Eclipse plugins for Q development

©2008 Marc Adler - All Rights Reserved

When will they ever learn?

http://apama.typepad.com/my_weblog/2008/01/the-power-of-th.html


Sigh .... When will the guys at Streambase learn that dubious marketing may overshadow what seems to be a really good product .....


©2008 Marc Adler - All Rights Reserved

NYSE Buys Wombat

A friend of mine at one of the NYSE-owned companies jokingly warned me last week that, today, there would be an announcement that would knock my socks off.

I wasn't prepared for the onslaught of emails that I started receiving around 9:00 AM, announcing the acquisition of Wombat by the NYSE. The guys from Wombat, who sit in the next row from me on the trading floor, gave me a look like "What? Who me?", while secretly counting their new-found riches under the table.

My only concern about this deal is a possible conflict-of-interest that might arise when a supposedly "exchange-neutral" information provider gets bought out by one of the exchanges that it is supposed to deal with. Conspiracy theorists might have a field day when there seems to be abnormal latency with a non-NYE exchange.

As far as CEP goes, my thought is .... would Coral8 be on the acquisition block next, seeing how closely they cooperate with Wombat and seeing how Reuters is aligning themselves with Streambase?

©2008 Marc Adler - All Rights Reserved

Erin Erring

Who let Erin Burnett out of the cage to write this article?

Liz Claman was always my favorite CNBC anchor. Sorbonne graduate. Fluent in French. Flaming red hair. Unfortunately, Liz made the jump over to the Fox Business Channel, which I will refuse to watch unless they have Peter Griffin delivering the financial reports.

You would think that CNBC would have learned its lessons after the hubbub over Maria Bartiromo. They should stick with the really solid reporters who have financial domain knowledge, such as Joe Kernan and David Faber. Let's also get Emmanuel Derman in there as a reporter.


©2007 Marc Adler - All Rights Reserved

CEP and SOA

There has been a lot of activity in the CEP blogs about the uses of that mythical beast, the Service Oriented Architecture (SOA). This beast means different things to different people. From my standpoint, let me tell you what it means to me, and let me tell you what I need.

For my company, the possible adoption of CEP will mean a huge seed-change in the way that we develop applications and share information. Right now, we have the most important information in the trading and risk process going directly into huge, monolithic Gui's. Important order and risk information is being sucked into a GUI application, much like a vacuum cleaner sucks up a trail of dust.

Already, the CEP effort has been able to transform the order-monitoring GUI into an Order Service, publishing order information to subscribers over a message bus. If the CEP project is a total bust, then at least the one tangible is the liberation of order flow information.

As all of the important data --- orders, risk, positions --- gets published out by the source systems and gets consumed by other applications, we need to have a global catalog that developers can browse where they can find out how to access data and what operations can be performed on the data.

For example, if I needed to get a real-time flow of Greeks for certain vanilla US equity derivatives, I might use the catalog to ask a question like this:

"Where can I find real time flows of greeks for IBM, INTC, MSFT, and DELL options? I prefer to have this flow come through as XML, but if no XML is available, then give the data to me in any format. In addition, I need to know if you have a remote function that I can call that calculates theta on a certain security."

The catalog service might respond:

"You can get the US Equity Deriv Greek flows by subscribing to a Tibco EMS message bus. You need to subscribe to the Tibco broker at tcp://megacorpbroker:7001, using functional ID "foo" and password "baz". This service is publishing out each greek as a JMS MessageMap on the EMS topic equities.derivatives.greeks.us, and here is a list of properties that you can access in each message. Sorry, but we don't support XML.

Furthermore, here are a list of request/response operations that the Greeks Web Service supports. If you want to generate the proxy code to use these operations, the URL of the WSDL for the Greeks Web Service is at http://greekserver:8042/webservice.wsdl.

As an added bonus, if you send this XML string as a JMS TextMessage in the following format to this the EMS queue named equities.derivatives.greeks.services, then you will get a response on your private EMS temporary queue."

This is like a super-charged UDDI, but knows about things like message buses and JMS queues and topics. For me, this is what we need out of SOA. Everyone publishing and consuming real-time flows. Everyone making services available, both as Web Services and as request/responses over a message bus.

The catalog should be real-time itself. In other words, if a new flow or a new service becomes available, the catalog itself should notify listeners that something has changed. So, all applications might subscribe to a "catalog control" topic where real-time changes to the catalog services are broadcast.

Imagine now that we have a GUI in which users can pick and choose from various flows, dynamically create queries that will be registered with the CEP engine, and dynamically define derived events that will be created when the queries detect a situation. Most of the CEP engines support an API which makes this possible.

Should we write this SOA catalog ourselves? Can you recommend a product that already does this?


©2007 Marc Adler - All Rights Reserved

I Hate Vista

Slow, cumbersome, intrusive.

Nuff said....


©2007 Marc Adler - All Rights Reserved

Kudos to Streambase Support

Thanks to Steve Barber and some of the other members of the Streambase tech support team. I could not get an out-of-process, .NET-based adapter to communicate with the Streambase engine on my laptop, although it worked fine on my office machine.

The Streambase people worked during the slow holiday season to diagnose the problem. It was caused by a DLL called BMNET.DLL that was installed by my Cingular/ATT Communications Manager. I have a Cingular broadband card that I use to connect to the internet when I am on the road and don't have a free wireless connection to tap into. BMNET.DLL provides data acceleration to the Internet.

Microsoft references this problem here: http://support.microsoft.com/kb/910435



©2007 Marc Adler - All Rights Reserved

Aleri Evaluation

Just a small word about the Aleri evaluation. Several of you have repeatedly pinged me to find out what I thought of Aleri, so I am going to write down some of my impressions. The usual disclaimers apply, such as this is my opinion and does not necessarily represent the opinions of my employer.

My impressions were formed after a superficial evaluation of Aleri against some of the other vendors. I have not gotten to the point yet where I am testing the latency and throughput of the CEP engines. I have not soaked and stressed the CEP engines to see if any of them are leaking memory. I have not tried them on a variety of processors, and I have not examined their performance under multicore processors.

In a nutshell, my biggest area of concern with Aleri was the "spit-and-polish" of the product. They very well might have the fastest CEP engine out there. However, I was stymied by the quality of the documentation, and my perceptions of their Aleri Studio. It also seemed that they were more of a "system integrator" that some of the other CEP firms, taking separate products like OpenAdaptor and JPivot and trying to fit them into a holistic offering.

An example of this was reflected in the difficult time I had in getting Aleri integrated with SQL Server 2005 through Open Adaptor. The documentation was non-obvious, and it took many hours with their sales engineer to finally get it connected. I compare this to Streambase and Coral8, where it took all of 5 minutes to hook up an SQL Server 2005 database to their systems (disclaimer: there is a problem getting Streambase, Vista and SQL Server to work together, although Streambase has not yet released an official version for Vista).

That being said, the salient points are:

1) Aleri's management team (Don DeLoach and Jeff Wootton) fully realize their short-comings in the department of aesthetics, and have promised me that they are actively addressing it. I would highly recommend that Aleri look at Streambase, whose total package is good with regards to documentation and tutorials. (However, I still find a lot of pockets of geekiness in the Streambase documentation.)

2) The Aleri sales engineering team, led by Dave, got a variant of my initial use case to work. However, there are features that Aleri does not yet have, such as jumping windows and pattern matching, that make Coral8 and Streambase stand out.

3) Going through Open Adaptor is not fun. Streambase and Coral8 make it simple to write adapters in C#. The Aleri sales engineer told me that he usually has to help clients to get adapters to work. That is really not the message that a company wants to hear if they have many legacy systems to interface with.

4) Aleri has real-time OLAP, using JPivot. To my knowledge, they are the only CEP company to offer real-time OLAP. I did not really get to see this feature, but true real-time OLAP is something that a lot of financial companies are interested in. We want to be able to slice and dice our order flow in real time over different dimensions.

5) The Aleri Studio uses Eclipse, just like Streambase, and the icons even look exactly like Streambase's icons. However, the user interaction seemed a bit shaky at times, and there were moments when I got myself into "trouble" with the Aleri Studio by clicking on one thing before clicking on another thing. Again, Streambase seems more solid. And, Coral8 does not try to impose a GUI builder on the developer. The guys at Aleri told me that they are addressing these issues.

I was really pulling for Aleri, since the development center is about 15 minutes from my house in New Jersey. They are right down the block from the hallowed halls of Bell Labs in Mountainside, and some of the developers live in the next town over from me. You couldn't meet a company of nicer guys, and the CEO is a very low-key guy compared to other CEOs that I have met. I was impressed by the fact that, at the Gartner conference on CEP, he stood up in front of the audience and exhorted us to try different CEP products.

I anxiously look forward to the 3.0 version of Aleri's offerings, and to see tighter, easier integration between their various components, enhanced documentation, enhanced support for .NET, and a cleaner version of their Aleri Studio. Given the quality of the developers there, I am sure that this version will kick some butt.



©2007 Marc Adler - All Rights Reserved

Where are the CEP Customers?

It seems that, lately, every blog posting I make on CEP generates further blog postings from the vendors and the subject-matter experts in the CEP space. It's great to see the CEP blogs become more active, so that I can tap into the collective wisdom of people like Tim, Marco, Mark, etc.

However, where are the postings from other customers? Isn't there someone from Goldman, Lehman, Merrill, etc who wonder about the same things that I do? Or, do these people mainly purchase the pre-packaged algo trading packages that the vendors have to offer.

One thing that was very interesting was the copy of the Aite Group report on Complex Event Processing that somebody had forwarded to me. It seems that most of the CEP companies number their customers in the dozens, rather than the hundreds or thousands. It seems that we are either at the very beginning of the explosive group that Aite predicts, or not many companies are finding a use for CEP, relying instead on their legacy apps to deal with streaming events. Certainly, in my own company, we have hand-written code for the various algo and HFT trading systems, code that I am sure the developers have profiled the hell out of in order to get the max performance. We would be hard pressed to replace this code with a generic CEP system.

If you are a customer or potential customer of CEP, I offer the opportunity to send me private comments at magmasystems at yahoo.





©2007 Marc Adler - All Rights Reserved

Visualizations Update

Stephen Few is rapidly positioning himself as the guru of business visualizations. His name has been brought to my attention several times over the past few weeks as someone to pay attention to .... "a new Edward Tufte", if you will.

Few has an online library with a lot of free articles to read. Right now, I'm reading Multivariate Analysis using Heatmaps. This is especially worthwhile reading following last week's visit by Richard and Markus of Panopticon, who showed more reasons why we should graduate from the free Microsoft heatmap control to the more feature-laden, doubleplusunfree, Panopticon product. As Panopticon adds more features in the value chain, it will be increasingly difficult to justify using a free product.

------------------------------------

Which brings me to another point that I have been thinking of ... a point that I raised on my previous blog posting. In the field of Enterprise Software, where do the responsibilities of a vendor begin and where do they end?

Take Panopticon, for instance. You can bind a streaming "dataset" to Panopticon, and Panopticon will render a realtime updating Heatmap to visualize that dataset. Of course, you ask how you get data into Panopticon, and you come back with the concept of input adapters.

Then, gradually, you wonder if their input adapters cover KDB, Wombat, Reuters, Vhayu, OpenTick, generic JMS, sockets, etc.

Then you wonder if Panopticon has input adapters that take the output of CEP engines, like Coral8 and Streambase. Or, you have a crazy thought like Panopticon embedding a copy of Esper/NEsper inside of itself.

Then, you get really greedy and wonder if Panopticon provides built-in FIX adapters that will devour a FIX 4.4 stream of orders and executions and show you what exchanges are slow today.

Then you wonder what kinds of analytical tools Panopticon might interface with ... since Panopticon is doing parsing and analysis of the streaming data anyway, can't it just take an extra step and analyze the silly data.

But, then if you are demanding all of these things of Panopticon and Coral8, how do you hook them together? Does the dog wag the tail or does the tail wag the dog?

Or, do we just consider Panopticon a simple visualization tool, demanding nothing more of it then the ability to display brightly colored rectangles of streaming data, and likewise, do we ask nothing more of Coral8 than to do what it does best ... recognize patterns and perform filtering and aggregations.

As Dali-esque as these thoughts may appear, this is the kind of things that I need to consider. In my quest for an ecosystem around the CEP engine, do we ask for the CEP engine vendors to expand outwards, or do we take the outer layer of components (ie: the visualization and analysis tools) and ask them to expand inwards to meet the CEP engine. Whatever it is, my wish would be for a true plug-and-play architecture between the CEP engine, its input components, and its output components.



©2007 Marc Adler - All Rights Reserved

CEP Vendors and the Ecosystem

While my wife and son cavort around Australia and New Zealand for the next few weeks (I get to stay home and watch my daughter, who only has one week off from high school), I hope to be able to catch up on some of the blog posts that I owe people.

One of the things that is most important for me in choosing a CEP vendor is the ecosystem that surrounds the CEP engine. In a company such as mine, we need to interface with many different legacy systems. These legacy systems can hold crucial data, such as historical orders, customer trades, market data, volatility curves, customer and security reference data, etc. This data may reside statically in a database, be published out as flow over some kind of middleware, or interfaced with an object cache or data fabric. We have every color and shape of database technology in our firm, whether it be more traditional relational databases like Oracle, SQL Server, and Sybase, or newer tick databases like KDB+.

From the input and output points of the CEP engine, we need seamless integration with all sorts of systems. Most CEP engines have the concept of in-process and out-of-process adapters. In-process adapters are more performant that out-of-process adapters. We would love to see as many in-process adapters delivered out-of-the-box by our CEP vendor. We do not want to spend time writing our own in-process adapters.

So far, none of the CEP vendors support KDB+ as an out-of-the-box solution. In fact, many of the CEP vendors did not even know what KDB+ was. (Is the same true for Vhayu as well?) My feeling is that, if a CEP vendor is going to be successful on Wall Street, then they must support KDB+. Is it even feasible for the CEP vendors to provide an abstraction layer around KDB+, and let the CEP developer write all queries in SQL instead of writing them in K or Q?

One of the most important things that I would like to see from the CEP vendors are tools to enable the analysis of all of the data that pass through the CEP engine. Many groups might not have the budget to hire a specialized mathematician or quant to perform time-series analysis on the data. Learning specialized languages like R or SPlus might not be possible for smaller groups that do not have a mathematical bent. The same goes for packages like Mathematica and Matlab.

Would it be worth it for the CEP vendors to come out with a pre-packaged "stack" for various financial verticals that incorporates analysis tools? Or, would writing a detailed cookbook be better? And, where does the responsibility of the CEP vendor end? Should we expect the CEP vendor to provide a one-stop shop for all of our needs, or should be just expect the CEP vendors to provide strong integration points?

Better yet, does this open up an opportunity for a third party company to provide this service? Like the many laptop vendors who buy a motherboard (the CEP engine), and slap together a disk drive, CD drive, screen and keyboard to make a complete system?

In examining the various CEP vendors, I have come to the conclusion that the offerings from Streambase, Coral8 and Aleri are very similar. Given another year, I might expect each vendor to fill in the gaps with regards to their competitors' offerings, and at that point, we might have practically identical technologies from 3 different vendors. In my opinion, the real win for these CEP vendors will come in the analysis tools they provide.


©2007 Marc Adler - All Rights Reserved

At Eaton Airport, October 2007

My shirt is hanging out of my fleece, but nevertheless, the foliage was beautiful at Eaton Airport in Norwich, New York. My Mooney M20G is right behind me.

©2007 Marc Adler - All Rights Reserved

Blog Momentum

On December 18th, I looked at my ClustrMap and saw that I had 239 visitors on the previous day. While this is nothing compared to the big blogs out there, it is definitely an improvement from the 30-40 per day that I had last year. I am not sure if my involvement with CEP has anything to do with it, or if it is becoming more well known across the Wall Street community (I believe it's the latter).

The levels of comments have increased too. Also, the number of private emails that I get have also increased, as a lot of you share your opinions about working on Wall Street, evals of CEP vendors, etc.

Some of you wonder if I can get into trouble with my company for the blog. Let me tell you that there are a HUGE number of people from my company who read my blog, not only from the tech side, but from the business side as well. The former CTO of our company was a big blog reader of mine. The CIO of our company has his own internal blog, and I know that people have recommended my blog to him.

Take a look at what other big financial companies are doing with blogs and wikis. Enlightened management at financial institutions will embrace social networking and a more open community, not run scared at the thoughts of what a competitor might pick up from a blog. Wall Street and The City are very close-knit communities, and more information passes between the employees at Lehman and Goldman during Happy Hour than whatever can be revealed from a blog.


©2007 Marc Adler - All Rights Reserved

Bonus Season

Word is trickling across the Street about Goldman's bonuses. Everyone knows the numbers of an average bonus of over $600,000 per employee. But the whispers that I have heard from a number of people are that the IT employees below Managing Director level did not fare very well.

In conversations that I have had with a number of current and ex colleagues, we are speculating whether most Wall Street companies will turn off the bonus spout, and admonish employees to "Just try to go anywhere else! Nobody is hiring now!".

What will be the likely scenario is a 10% reduction across the board of the lowest performers. I am going long in the steak-knife manufacturers.

What you don't want to see is the de-motivation of the normal, punch-the-clock worker in the IT field. These are the people who score in the middle of the road on their yearly evaluations, the people who have been at a company long enough to know one system inside and out, the people who don't blog nor read blogs, the people who don't work on weekends and who don't think about the next great product that they want to write for the traders. These are the people that usually hold the keys to the systems and the data, the people who you need to convince to open the gates to your applications. Giving these people the doughnut bonus will slow down processes, resulting in further roadblocks for the people who do want to get things done.

Nevertheless, I still have open headcount for the CEP project and for the .NET client framework team.

©2007 Marc Adler - All Rights Reserved

Farewell Kaskad

A player in the CEP space, Kaskad, is no more. Colin Clark writes in to say that, since the Boston Stock Exchange (BSX) has ceased to function, Colin had to disband Kaskad. I assume that since Kaskad did the work for the BSX as consultants, the BSX maintained all or most of the IP rights. Or, perhaps, Kaskad felt that since their only client was no longer funding their development, they could not gain further VC money in this financial environment.

According to the Aite report on CEP vendors, Kaskad had 16 employees. Since they are based up in Boston, I wonder if Streambase is looking at adding some additional talent.

Colin is looking for new opportunities in the CEP space, so contact him if you have anything that might interest him. Colin was involved in the old NEON (New Era for Networks) back in the dotcom boom, so he has the entrepreneural streak running through him.

©2007 Marc Adler - All Rights Reserved

Streambase (yet again ...)

After vowing to bypass Streambase in my CEP engine evaluation, I may be forced to eat crow. I agreed to let Streambase into the evaluation process because I need to have two CEP engines in my project ... one as primary and one as "cold backup". And, for various reasons, Aleri and Esper did not pan out for me.

The new CEO of Streambase, Chris Ridley, came down to New York to meet with me, with his chief architect, Richard Tibbetts, in tow. They acknowledged some of the errors of their over-aggressive marketing, and told me about their sharpened focus on the financial industry marketplace.

They also let me have an eval version of Streambase that is not constrained by any license key, and in the interests of expediency, they graciously allowed me to bypass their eval agreement (which would have taken weeks to make it through my company's legal processes at this time of the year).

I installed Streambase on my laptop. My first impressions are ..... "slick". In other words, all the superficial, glossy stuff that gives the initial impression to a prospective customer is all there. Nice documentation with plenty of graphics, a great interactive tutorial, etc. I was "warned" that Streambase puts a lot of time into their studio and help system, and I can definitely concur. Nice job, guys.

I am going through the tutorials now. Several things jump out at me right away:

1) They use Eclipse as the foundation of their Streambase Studio. I am quickly becoming a fan of Eclipse, especially the way that you can automatically update Eclipse plugins.

2) The development methodology is more "file-based" than the other products. A familiar paradigm to Java/C# developers.

3) There are two ways to develop apps. The Event Flow method uses a GUI-based method. You can also program in StreamSQL. Unfortunately, there is no tie-in between the Event Flow and the Stream SQL files. In other words, unlike Coral8, if you make a change in the Event Flow, it does not get reflected in the StreamSQL file. In your project, you can have multiple Event Flow files and multiple StreamSQL files. I would love to be able to develop in either system, and have them automatically translated to the other system.

4) There are certain things that you need to do in the Event Flow system that you cannot do in StreamSQL. There are comments in their demo programs to this effect. I would welcome a document that outlined these differences.

5) I noticed that the icons used in the tool palette are identical to the ones that Aleri uses. Interesting. Someone looked at the other company's product.

6) Richard Tibbetts and Mark Tzimelson are very respectful to each other's work. Nice to see that kind of respect at the technical level.


©2007 Marc Adler - All Rights Reserved

Acropolis Shrugged

http://blogs.msdn.com/gblock/archive/2007/12/07/if-acropolis-is-no-more-what-s-our-commitment.aspx

CAB? Acropolis? CAB? Acropolis? CAB? Acropolis? CAB? Acropolis? CAB? Acropolis?

It looks like Acropolis may be going by the wayside, and the Patterns and Practice Group has decided that, for today, they will refocus on CAB.

This is precisely why we build our own .NET frameworks in my investment bank. Because, we need a high quality framework that has some domain knowledge of what capital markets needs. Goldman, Wachovia and Morgan Stanley have done the same.

And, this time last year, Microsoft came in and tried to get us to adopt CAB, and the week after that, they told us that Acropolis was the new flavor of the day. Thank the lord that we were focused on a mission to build what we wanted and needed, without all of the background noise from Microsoft. (Sorry Joe...)





©2007 Marc Adler - All Rights Reserved

Get Rich Quick with KDB+

I am convinced that the world needs more KDB+ consultants. The supply of these creatures is so small, that if you end up needing one in a hurry, you probably have to go through First Derivatives.

KDB+ is used by most of the Wall Street companies --- I can name Citigroup, Barclays, Bank of America, and Lehman as big KDB+ users --- to store tick and order data. KDB+ 's biggest competitor is probably Vhayu.

The main blockade to learning KDB+ is their programming languages - K and Q - which can make APL look verbose!

If you are affected by the upcoming layoffs on Wall Street, and if you are looking for a new, exciting career change, and you don't relish the idea of selling steak knives door-to-door, then there is room in this world to be a KDB+ consultant.


©2007 Marc Adler - All Rights Reserved

OpenAdaptor, SQL Server 2005, and Aleri

This posting was made in the interests of any Aleri or OpenAdaptor users who are trying to connect to a named SS2005 instance.

I have multiple "instances" installed of SQL Server 2005. According to the documentation at http://msdn2.microsoft.com/en-us/library/ms378428.aspx, this JDBC connection string should have worked:

jdbc:sqlserver://MAGMALAPTOP\RPT;databaseName=MyDatabase;integratedSecurity=true;

In Microsoft's JDBC Driver 1.2 for SQL Server 2005, there is a sample Java app called connectURL. With the connection string above, this sample app worked fine, and was able to connect to the RPT instance of my SS2005 database.

However, I could not get OpenAdaptor to work with this connect string. In case you are wondering why I was messing around with OpenAdaptor, it is because this is what Aleri uses for its adapters to external data sources.

After spending several hours this weekend trying to get Aleri to connect to SQL Server using the connection string above, I finally stumbled upon an alternative syntax for the connection string.

The new connection string is:

jdbc:sqlserver://MAGMALAPTOP;instanceName=RPT;databaseName=MyDatabase;integratedSecurity=true;

Notice that the instanceName is specified with a separate parameter.

So, there may be an issue with OpenAdaptor. Or, another theory that I have is that the backslash character in the connection string is being considered as an escape character.


©2007 Marc Adler - All Rights Reserved

Getting intermediate results in Streams

Let's say that we want to keep a running total of the number of shares that we have traded, and at 4:00 PM every day, we want to dump out the total. In Coral8, we can do something like this:

CREATE LOCAL STREAM Totals (TotalShares INTEGER);

INSERT INTO Totals
SELECT SUM(shares)
FROM TradeInputStream KEEP EVERY 1 DAY OFFSET BY 16 HOURS
OUTPUT EVERY 1 DAY OFFSET BY 16 HOURS;

This looks pretty straightforward. The Totals stream retains the totals until 4:00PM. At 4:00 every day, it outputs the total shares to any other stream that is "subscribed" to Totals, and then resets itself to start accumulating new totals.

This is something that CEP engines are good at, whether it be Coral8, Aleri, or Esper.

Now, let's enhance this a little bit.

Let's say we give the traders a .NET GUI application, and on this GUI is a "Status" button. The traders can press this button any time they want to know how many shares have been traded so far that day. So, at 2:00, a trader pushes a button on the GUI and we need to return to him the number of orders seen so far that day, the number of shares seen, the notional value of all orders, etc.

So, there are two questions:

1) How can we "dump out" these accumulators on demand? In other words, is there a way to tell these CEP engines to give me the contents of an aggregation stream AS OF THIS MOMENT ?

2) How can we "call into" our CEP engine to retrieve these values? Do the CEP engines support an API that I can use from within the GUI to say "Give me the current value of a certain variable in my module"? Something like

IntegerFieldValue field = Coral8Service.GetObject("ccl://localhost:6789/Default/SectorFlowAnalyzer", "sum(Shares)") as IntegerFieldValue;
int shares = field.Value;

In a standard C# application, this would be as simple as putting a Getter on a variable, and just calling the getter. If I was using Web Services, then I could call into a Web Service and just ask for the values of some variables or for some sort of object. But, from a C# app, how can I get the current value of a stream that is aggregating totals?

Another way of accumulating the total number of shares in a CEP engine is to step into the procedural world, and just define a variable. In Coral8, it would be something like this:

CREATE VARIABLE TotalShares INTEGER = 0;

ON TradeInputStream
SET TotalShares = TotalShares + TradeInputStream.shares;

Then, we would need a "pulse" to fire at 4:00PM every day, and upon this pulse firing, we could send the TotalShares to another stream.

I am sure that there are patterns in every CEP engine for accessing intermediate results, but something that is a no-brainer in a procedural language may not be so easy in a CEP vendor variant of SQL.



©2007 Marc Adler - All Rights Reserved

Coral8 and Transparency

I just tried to get some info on the Apama Event Processing solution (not their Algo Trading platform,just the simple ESP platform). I filled out a form, and now I have to wait for a Progress sales rep to call to arrange a demo. Even if I want to see an Apama webcast, I need to fill out a form.

Let's contrast this what Coral8 has to offer. Coral8 lets you download the entire developer platform, with all of the documentation included. Everything is included .... there are no important packages that are missing with the eval version. There is no 30-day license key that you have to get. There is no waiting for a salesperson to get in touch. As far as I know, you get everything is ready to go from the time you download the package.

I fail to understand why certain vendors make it so difficult to evaluate a package. In a big financial institution like the one I work for, if you use software in production and this software is not properly licensed and paid for, it is grounds for termination of your job.

Coral8 has the right attitude. Just get it into the hands of as many people as possible as spread the word around.

©2007 Marc Adler - All Rights Reserved

NEsper Docs

Good solid docs on the Java version. Unfortunately, the NEsper version references the Java docs. The NEsper version only comes with an auto-generated CHM file.

Looks like I will have to compile the examples and dig through the source code of the examples in order to see how to use NEsper. Thomas, it may make sense to outline the difference between Esper and NEsper in the master documentation ... maybe using highlighted text boxes to outline the differences.

It also may make sense to include the PDFs in the NEsper distrubution.


©2007 Marc Adler - All Rights Reserved

Per Se

One of the perks of working for Mega-Bank is that you get invited to sales and marketing functions that were previously out of reach to me when I was a consultant. Such was the case this past Wednesday, when I was invited to a marketing function by Autonomy at Per Se.

If you have never heard of Per Se, then maybe you have heard of The French Laundry, a Napa Valley eatery that is run by reknowned chef, Thomas Keller. Per Se is Keller's New York City version of the French Laundry, and is one of the most difficult reservations to get in New York.

Autonomy has quarterly executive briefings at Per Se, where they bring together the sales of management of Autonomy, various Autonomy business partners, current customers, and future prospects. In addition to a fantastic lunch, we got to see how Standard and Poors used Autonomy to help their analysts get through the millions of pages of regulatory filings. S&P has a team of PHD mathematicians that have have developed some fairly sophisticated models in Autonomy to help them extract the "meat" out of their stream of documents.

Autonomy seems to positioning themselves as a major add-on in the Sharepoint marketplace, adding very sophisticated document searching. It would be interested to compare Autonomy with things like Google Search and X1.

©2007 Marc Adler - All Rights Reserved

On to Esper/NEsper

I have had to expand the CEP evaluation process. I am going to start looking at NEsper, and maybe, Apama (after some recommendations by some of our Asia folks who seemed pleased at Apama's performance over heavy loads).

I just downloaded NEsper and I am starting to go over some of the docs. I am sure that Thomas and Aaron will correct me if I say anything incorrect about Esper. Two things stand out about the Esper offering:

1) No GUI Builder or visual tools, probably because .....

2) Esper/Nesper is a component designed to be incorporated into your application ... in other words, it is treated as a third-party .NET assembly, just like things like Syncfusion, Log4Net, etc.

So, unlike Coral8, where you run a separate Coral8 server process, you need to write an application that "contains" Esper/NEsper. While this solution does not favor quick, out-of-the-box prototyping as Coral8 does, it gives you more control over the CEP actions. Everything with Esper/Nesper is "in-process" to your application.

Also, Esper/Nesper is Open Source. I downloaded it, and I have the full source code sitting on my hard drive. I have to talk to the financial guys at my company, but I don't think that we would have to do the amount of financial due dilligence with an Open Source effort as we would with a company who does not follow the Open Source model. Maybe we will have to count the number of moths that fly out of Thomas' wallet.


©2007 Marc Adler - All Rights Reserved

Reducing Lock Contention

Here is a good blog posting on 10 Ways to Reduce Lock Contention. Even though our .NET-based client-side framework behaves well, I should take these hints and examine our framework with a fine-tooth comb.

I am also following all of the recent developments in Parallel Programming coming out of Microsoft. I wonder how much Joe Duffy's team interacted with the Accelerator guys from Microsoft Labs. I am also very interested to see if our Derivatives Analytics team, which is very strong in .NET, can leverage this new technology instead of/in additon to some of the proprietary technology offered by hardware acceleration vendors.

By the way ... I just started using Google reader. This blog is one of the blogs that Google Reader automatically recommended.

©2007 Marc Adler - All Rights Reserved

Some Random Comments

1) Go out and see the new Coen Brothers' film, No Country for Old Men. Absolutely startling. I saw it over a week ago, and I still can't stop thinking about it.

2) Please pay careful attention to the Comments section of each post here. Many of the CEP vendors are responding with interesting and important comments.

3) I need to start thinking about a notification framework for the CEP project. Notifications are a bit more complicated in financial firms where Chinese Walls have to exist, and where compliance officers are constantly over your shoulder. It's a good idea to involve the compliance guys right from the start, and to have THEM tell you where the Chinese Walls should be. Nothing will get their attention more that discovering that your prop traders are getting notification about your customer order flow!

4) I just subscribed to Alex's blog. Some interesting posts. Takeaways from skimming through his blog include:

a) I need to narrow down CPU pricing from the CEP vendors. Let's assume quad-core machines.

b) I am more curious about NEsper than I was before.

c) How will Esper and BEA stay in sync? Seems a bit annoying that BEA has seen fit to change some of the syntactic sugar of the original product.


©2007 Marc Adler - All Rights Reserved