Thursday, October 30, 2008

CEP - A Legend in its own Mind?

In the wake of the current financial crisis, several enterprising journalists have been trying to link the use of CEP, EDA, and SOA with the prevention of further financial woes. I have been sitting back and chuckling at many of these attempts to equate a three-letter acronym with financial salvation, especially ones written by people who don't actually work in finance and have never set foot on a trading floor. However, I cannot remain silent anymore.

I am reading an article that just appeared in Wall Street & Technology magazine titled Wall Street Firms Using CEP to Measure and Manage Risk. Directly under the title of the article is the proclamation:

New complex event processing applications promise to help firms get a better handle on their risk exposure, but can CEP erase Wall Street’s risk management woes?

This is an example of the sensationalistic headlines that have been crossing the various blogs and trade magazines in the past few weeks. All of a sudden, CEP has become a three-letter word for Financial Nirvana.

People .... I have news for you ....

CEP systems are simple tools. They take streams of data and produce some output whenever something in the streams fit a certain condition. That's all it is! CEP systems do not do predictive analysis. They won't tell you when you will be losing money in the future.

They are like compilers. Saying that CEP can help with risk management is the same thing as saying a C++ compiler can help you with risk management. CEP products and C++ compiler are merely tools. You are supposed to be providing the logic and the interpretive abilities.

All banks have risk management systems, and most (if not all) have NOT been written using CEP products. Most of them are old, legacy systems written in C++. And, most of them do the same things that CEP products do. They take real-time streams of market data, positions, trades, and P&L, and they crunch them together to produce some useful output. And believe it or not, they work well despite the fact that they are not using commodity CEP technology!

And, many risk systems are run as an end-of-day processes. They are not real-time. They produce management reports which are read every morning. It is not the fault of the risk management system or the underlying technology if management chooses to ignore a risk report or chooses to put on an exotic trade whose risk can't be measured, or chooses to go all-in with subprime mortgages.

Risk management systems were here long before CEP became a buzzword and will remain long after all of the commodity CEP products have vanished.

I can imagine that, with the journalistic feeding frenzy associating CEP and risk management, smart marketeers like Terry Cunningham, Mark Palmer, and Don DeLoach are grinning from ear to ear. Not only grinning, but also doing their bit to fan the flames, as any smart entrepreneur would do.

Let's look at the article a bit.

But CEP vendors say their software can give risk managers a better view of such counterparty risk. "No software can replace [good] judgment," says Jeff Wooten, VP of Aleri. "What [CEP] software can do is give you better information with which to make those judgments and a better understanding of where you stand."

Ugh. Why could a system written with Aleri give you a better view of your risk than a custom-coded system? Will Aleri help you write more sophisticated detection rules?

Undoubtedly, Aleri will enable you to write a new risk management system much quicker than if you were to code one from scratch. But, I don't see that the CEP system will give you "better information". (To be fair, since Aleri has a real-time OLAP product, you might get better information by using the Aleri-specific visualization tool. Maybe that is what Jeff was implying.)

"Our software can't help you predict what's going to happen with your counterparty; it can't help you predict that Lehman will declare bankruptcy," Wooten adds. "But it can help you know what your exposure to Lehman is."

Knowing your exposure to Lehman is conceptually a very simple task that can be done without the help of a CEP system. However, what if your exposure is tied up in complex derivatives? How would a CEP system help you there? I am pretty sure that all of the people who lost money in Lehman bonds knew precisely what their exposure was, without the help of a CEP system.

"It's predictive; it's [based on] probability and in some cases the CEP engine will grab something it doesn't need," Greene acknowledges. "But when you look at more-complex instruments that can take weeks or months to settle because of issues on the back end, the CEP engine that can help them automatically grab information ahead of time behind the scenes speeds that up."

Hmm ..... Can anyone decipher Spence's words for me? I think that Spence is kinda hinting about what I mentioned above, with the complexities involved in figuring out your P&L based on very complex structured products. But I would like to know exactly how Tibco Business Events assists in figuring out this exposure, and why it would be harder to do if you were to use some custom code or Excel.

Despite CEP vendors' promises, though, there are those who feel the value of CEP technology to risk management is finite, pointing to limitations of the technology itself and to the fact that risk management involves more than looking at numbers.

Bingo! And, directly after this statement, Tim Bass weighs in with some of his very correct opinions.

Another dissenting voice is Miles Kumaresan, head of quantitative trading at proprietary trading firm TransMarket Group. "The problem we have right now is the credit market, and that has nothing to do with complex risk models," he told WS&T in late September. "To do risk assessment you don't need CEP. It's much more important to actually use the risk numbers that are already available."

Double Bingo!

CEP vendors sell a tool. This tool enables you to take various real-time streams of data, and allows you to correlate the various streams in various ways. They are very useful tools. But, if you have a working risk management system, you don't need to go out and start rearchitecting your systems right away. The current crisis is bigger than any risk management system. No risk management system would have stopped SAC and Greenlight Capital from losing billions of dollars on the Volkswagon short squeeze. There is a herd mentality on Wall Street, and CEP/SOA/EDA was not going to stop SAC from accumulating this particular short position.

Where would a CEP system be useful? I might use CEP if I would build a new risk system that aggregates output from other legacy risk systems in order to present an enterprise-wide view of risk. I would use CEP to build a brand new risk management system, but only if I could not find one from a vendor that fit my needs. (And, if I might allow myself to jump on a fashionable buzzword, I might eventually find risk management applications in "the cloud", maybe as a service offered in Microsoft Azure?)

CEP, SOA, EDA are concepts and there are tools that implement these concepts. Don't ever mistake them for something that will give you immediate safety from all of the wolves out there.


©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Saturday, October 25, 2008

New Coral8 Users group on LinkedIn

There is a new "Coral8 Users" discussion group on LinkedIn. Please consider joining if you are a user of Coral8 or are interested in learning more about coral8.

Please note that this group is not being sponsored by Coral8 in any way. This is a user-sponsored group. It is a way for Coral8 users to help each other and to discuss Coral8 without involving the staff of Coral8 at all.



©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Forays into C++

We have a C#/.NET-based Reuters market data adapter that we wrote ourselves (well, actually, I wrote it). This is an out-of-process Coral8 adapter that reads real-time ticks from RMDS, turns then into Coral8 tuples, and feeds them into our Coral8 engine. The adapter has two caches in it, and since market data is last reliable, we can publish the last-reliable ticks from one of the caches at specific intervals.

When we hooked the market data adapter up to our Coral8 engine, we started seeing an immediate backup in the pending message queue. We were sending Coral8 about 3000 ticks per second. In addition, our Coral8 engine was processing our order flow, which could hit a max of another 3000 messages per second.

My first thought was that it was taking too long for Coral8 to deserialize the market data tuple, so I wanted to transform our market data adapter from an out-of-process adapter to an in-process adapter. Unfortunately, in-process adapters have to be written as a vanilla, Win32-based C/C++ DLL.

I grew up on C++. However, I have not touched a lick of C++ since diving into .NET in 2001. I was scared. I was frightened. However, I know that my teams needs some experience in writing in-process adapters for Coral8, and since my team consists of very strong .NET and Java people, I decided to bite the bullet and take one for the team.

Man, oh man! I can't believe how much I had forgotten! Strange symbols like '*' and '->' were coming from my fingers. Compilation errors sprang up everywhere. There was no Resharper to help me along. Linker errors to strange obfuscated functions that I swear I never wrote started appearing in the error window.

Perhaps I need to use a little bit of extern "C" here? Maybe there? Maybe everywhere!

Where was my .NET Thread class? Where was System.Timer? How do I do events? There is a new kind of pain that you have to go through in order to get callbacks to work within templated C++ classes. It took me hours to get simple callbacks to work.

Finally, after two days of work, I got the market data adapter ported over to Coral8. To start with, let's try reading the ticks from a flat data file instead of from Reuters. Whew, it works. I see the ticks appearing in the Coral8 stream viewer. Now, let's try Reuters. After a bit of tweaking and some crashes on the destructors, ticks are finally coming in.

Then, I get an email from our Coral8 developer telling me that the reason why the market data feed was slow in Coral8 was because of one of those mysterious coral8 queries that he wrote that clogged the system! So, my inproc adapter was not needed anymore. Market data was now zooming through our Coral8 engine.

Nevertheless, it was a fascinating experience to return to my C++ roots from a 7-year hiatus in the .NET world. Never again!




©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Coral8 and Performance

From the head of our CEP engine development team (used with his permission) :

I'm pleasantly surprised at how fast c8 is when one gets it right. (On the other hand, one misstep and performance goes over the cliff.)


This is one of the main problems with the variants of Streaming SQL that you find in many of the CEP engines. It is not at all transparent what goes on under the hood of all of the CEP engines when you are writing complex queries in Streaming SQL.

Unlike Microsoft, where you find very good profiling tools in SQL Server, the CEP vendors do not provide the necessary tools that will enable a developer to isolate bottlenecks. This has been an issue with Coral8, which they recognize and hope to remedy in the future.

Every time that I've cursed out Coral8 for what I might think is lousy performance, it has turned out that it was us that managed to write a query that clogged the engine. And, when you clog up Coral8, it can bring an entire machine down very quickly as the pending message count builds up.

The important thing is about choosing a CEP vendor is the level of technical support, and despite some turnover in Coral8's tech support staff (goodbye to the excellent Trahn), we have been able to have access to their chief architect and to the head of development when we have really gotten ourselves into trouble. Coral8 also recently hired a New York-based support person who used to be an architect with Gemstone. So, even though it will take this guy some time to get up to speed with Coral8, we are glad to have a local person to help us when we need the assistance.

Coral8 has just released version 5.5, which will be a major help to us in terms of real-time OLAP and entitled subscriptions. It is a major release that will force us to refactor some of our code. But, it is good to see that Coral8 is making it easier for us to implement our real-time dashboards where every user can possibly be entitled to see a different view of data. I feel that support for real-time OLAP is going to be a major selling point for CEP systems, as it is *really hard* to implement it totally on your own.


©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

CEP Updates from the Front

It feels like forever since I was able to find the time to blog last. There are millions of things to write and so little time.

We finally rolled out our CEP system to the heads of the Equities Trading department. It is sitting on the desks of the Head of Equities Trading and the Head of North American Trading, and will start to be rolled out to the individual traders in short order.

The CEP system gave us a nice "victory" last week. One one of the very big volume mornings, our system was showing that a certain customer type was not buying any financial stocks. The Head of Trading came to us and told us that our system was showing wrong data. Then, he started to call around to his individual traders, and what'dya know ... this customer was not buying financials.

So, it seems like our system is already starting to give the department heads the insight into trading activity that they haven't had before. And, it is giving them the information in a totally new visual paradigm that is so far away from the simple flashing grids that most traders are used to.

We have just started to integrate market data into our system ... but that's the subject of my next blog posting.

The big steps for 2009 will be to start to take our system global. This means having region-specific instances of CEP, and having each region feed into a central CEP hub in order to let people do cross-region correlations. This will bring us new challenges in terms of data quality, as we will have to deal with unifying symbologies, currencies, compliance issues, and more.

... Of course, this is assuming that we all have jobs next year!





©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Thursday, October 09, 2008

Orange Widgets

A recent posting from Jack Rusher of Aleri reminded me about Orange. I had looked at Orange a few months ago very briefly, but it looks like, over this past summer, the Orange Project ramped things up and put out the first official version of their system.

This looks a lot like what I want to do with a Yahoo Pipes-like system. The guys from the Microsoft Oslo team should also read this and see if they can get any additional inspiration. (It's nice to see that Aleri is also looking at this stuff ....)

I will be reading further to see what it would take to come up with an input adapter for Orange that can read a Coral8 tuple. Hopefully, they already have adapters that can read from SQL Server and KDB (yeah, right!)

And, what about Orange's capability to analyze real-time data? Can we put order flow through Orange? Do we have to aggregate order flow and feed Orange at one-minute intervals?

Visual tools for data mining are starting to become more and more widely used. There are tools like Tibco Spotfire and Tableau that are used a lot in Pharma and Financial Services. (There are some new ones on the horizon that I cannot talk about that makes these tools seem like MS-DOS prompts). But, Spotfire and tableau really do not handle real-time flows of data.

The elusive goal is to provide great visualization for true, real-time OLAP. On my CEP team, we had to write a custom real-time OLAP GUI ... how long before these kinds of tools become commoditized?



©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Wednesday, October 08, 2008

New Aite Report Slams Microsoft's Capital Markets Partner Program

A blog reader sent me this link to a new report by The Aite Group.

Below is a copy of the report summary from the Aite website. I will comment further in another blog posting.



Of vendors engaged with Microsoft's Capital Markets Partner Program, 55% rate Microsoft's overall satisfaction with the relationship as mediocre or poor.

Boston, MA, October 6, 2008 – A new report from Aite Group, LLC evaluates the effectiveness of Microsoft's Partner Program as a capital markets vertical strategy within a horizontal company. The report outlines what vendors engaged in a partnership with Microsoft can expect, best practices for other horizontal vendors looking to focus on capital markets, and ways in which vendors and capital markets firms can maximize their technology investment in Microsoft.



The report is based on interviews conducted with 21 partners in the Microsoft Partner Program who all offer capital markets solutions, as well as current and former Microsoft employees, customers, and non-partners working with Microsoft Capital Markets. Aite Group found that Microsoft's strategy of developing its vertical partnership program may have hurt its reputation, as 55% of partners in this program rate their overall satisfaction with Microsoft as mediocre or poor. Though Microsoft is criticized for its lack of knowledge of the capital markets vertical, it is viewed favorably for its architecture and engineering, as well as for its technical support.

"While Microsoft's horizontal rivals, including IBM and Oracle, were busy building capital markets offerings through acquisitions, Microsoft maintained a partner program whereby it supplied the underlying platform upon which other software providers could build vertically specific solutions," says Adam Honoré, senior analyst with Aite Group and author of this report. "As a vertical strategy, Microsoft is not the feared giant it is perceived to be in many horizontal venues. Instead, Microsoft has much work to do before it is perceived as anything other than a mediocre performer in its vertical go-to-market strategy."





©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Tuesday, October 07, 2008

Microsoft Oslo and our App

For some reason, Microsoft seems to think that I am an "influencer" in Wall Street, and that plying me with large steaks and bottles of wine will make me happy (they are certainly correct in the second assumption).

Last night, I had a very nice dinner with Robert Wahbe and Steven Martin of Microsoft. They are large in the Connected Systems Division, which is the group that is overseeing the development of the mysterious Microsoft Oslo product/framework. We had a happy group that included the famous DonXML, Ambrose from Infragistics, Bill Zack, and a charming lady from Waggoner Edstrom whose job it was to make sure that Robert and Steven did not disclose too many of Microsoft's secrets (although I ended up telling them a few little ditties about Microsoft that they were not aware of!).

I have been sworn to secrecy, so I am going to regurgitate what little known facts about Oslo has been leaked out through various blogs. Oslo contains three things:

1) A new language that enables you to create your own Domain-Specific Languages (DSL).

2) A visual modeling tool.

3) A repository for models that you build.


What would we do with Oslo in our CEP system?

We would like our traders to be able to eventually build their own queries/models, and "inject" them into our CEP system. We do not want the traders coding in Coral8's CCL or any other streaming SQL language for that matter. And, we want the traders to play in our "sandbox" without hurting themselves or the other users of the system.

If we could create a high-level DSL that, through the use of some sort of "adapter", was translated into Streaming SQL, then we could give that to the traders. A visual modeling tool would allow them to easily construct models (remember how enthusiastic I was about Yahoo Pipes a few months ago?).

Let's see what Microsoft comes up with, and how soon it will be until Oslo is ready to use on a trading floor.



©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Sunday, October 05, 2008

Benchmarking .NET-based Tranaction Engines (and the LSE)

Although somewhat dated (the infrastructure in from 2004, which can be considered to be light-years in the past), this is a useful read:

Benchmarking a High Performance Real-Time Transaction Engine Design

(Go to the bottom-left of the page, and click on the link that says "View or download")

Even more interesting is the slide deck that accompanied the presentation. Slides 24 to 32 give some insight into the .NET-based architecture of the London Stock Exchange.








©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Has LinkedIn Jumped the Shark?

There are a few annoying tendencies that I have been noticing about LinkedIn for the past several months:

1) Gratuitous recommendations. You open up your Inbox and find that some unremarkable colleague that you worked with five years ago is asking you for a recommendation, and promising to give you a stellar recommendation in return. Usually this ex-colleague is looking to change jobs, and wants to load up on praise. You then have to spend the next 15 minutes remembering what this person actually did on your team, and try to grasp at straws in order to say something positive about him. The verbiage associated with LinkedIn recommendations is always the same. As a potential employer, a LinkedIn recommendation is pure noise.

2) Unsolicited ads and spam on LinkedIn Groups. It seems that new groups pop up every day, taking a simple domain and dicing and slicing it 20 ways. There is a group for Wall Street, a group for Wall Streeters who are left-handed, a group for Wall Streeters who take the subway more than 5 stops, etc. It seems that the moderators of these groups will let anyone who has a pulse into the group.

I joined a group about Wall Street the other day, and the first discussion on the group was from a real estate agent from Prudential Realtors, looking to sell condos and co-ops in New York. Recruiters and outsourcers troll these groups, posting ads for opportunities and peddling their wares, thus avoiding having to pay any fees to LinkedIn for job postings. People like these cheapen the groups.

3) There are a large number of recruiters who send you LinkedIn invitations. Why in the world would I let a recruiter see my contacts? Some of these contacts are fairly high up in the food chain. I don't want a recruiter contacting them. I almost never accept invites from recruiters. There is an implication that, if you have a LinkedIn contact, you know that person and like them enough to let them participate in your network.


LinkedIn is a truly useful tool. However, I think that it has "jumped the shark". LinkedIn needs to give the user additional options to control the increasing amount of spam that is invading the network.


©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.