Scott was a big believer in a tool called CodeSmith to automatically generate code. In our group, he used it to take a Coral8 schema and generate serializers, deserializers, and something that we call "ServiceAgents". The problem was that Scott had the only CodeSmith license, so only Scott's machine was able to do the code generation. A few months ago, I had stumbled upon the T4 tool in a blog post, and I had asked Scott to switch over from CodeSmith to T4. Now, the code generation is done by a free tool that Microsoft includes in Visual Studio, so now everyone can experiment with automatic code generation.
A few weeks ago, I found an article in Visual Studio magazine about T4, and then I saw that Chris Donnan was experimenting with T4.
So, I decided to spend an hour last weekend and code up my own T4 Coral8 class generator. I did not bother to read the T4 manual, but I was impressed with what I could rig up in a very short time. This generates a class for every Coral8 CCS file in a certain directory.
This isn't world-class code .... I generated nullable types and I did not bother to generate null-checking code on the nullable fields. But, feel free to modify it.
I will certainly use T4 to generate classes and serializers/derserializers for Orinoco.
Congrats to Terry Cunningham, the former CEO of Coral8. It looks like he is now a Senior VP over at i365, which is a Seagate company. I guess that Terry will be wearing a dual hat, as he is also the Chairman of Aleri. One wonders if Seagate will eventually go into the CEP business :-)
Also, good luck to Mark Tsimelzon. A few people at Coral8 have remarked that Mark is no longer with the combined entity. With Mark's track record, I am sure that he will be involved with something new and interesting.
It looks like Coraleri has chosen Coral8's CCL to be the SQL-based language of choice for developing Coraleri applications. This is the first piece that is falling into place, and for me, it's good news. Now, we have to wait to for the Splash integration to happen, and then the second piece of the puzzle will be solved.
The latest Coral8 5.6.1 has lived up to its claims of major performance enhancements. My CEP team reports that CPU usage is down significantly in our tests on our most complex queries.
With Coral8 making big strides in performance, the case to keep the Aleri CEP engine around is weakening. Don DeLoach is adamant that none of their customers will need to rewrite a single line of code, and that Aleri has even put it in writing. I know that Don believes this with all of his heart, but I am still a bit skeptical. But I am willing to cut them a big piece of slack, as long it does not affect what we are doing with Coral8. We are continuing on as if nothing happened, and I am trusting in Don and the Coraleri team to make the transition completely transparent to us.
Which brings me to my next topic ... Sybase RAP and Coral8.
For this topic, I am putting on the other hat that I wear, which is Chief Architect of Equities. One of the jobs of the Architect is to keep abreast of the current technologies that we use to see if any of them will approach an "end-of-life" situation. Chief Architects needs to make sure that the company does not invest in any technology that might be risky in any way.
Almost every large investment bank has one of every technology. I am not giving away any of our secret sauce when I say that we have a number of applications that use Sybase ASE for the database. So, when Sybase comes knocking on my door, asking me if I am interested in their new CEP product, what do I think?
First, I think it is a bit curious that Sybase did not publicly mention the use of Coral8 as their CEP engine. I can now understand why. I can speculate that Sybase knew of the pending acquisition of Coral8 by Aleri, and wanted to avoid some of the confusion that goes along with a merger like this. If I knew that a company's product was based on another vendor's product, and that other vendor had just gotten acquired by another company, I would need to be absolutely sure that all of the moving parts were going to have a long life, and that we would not be faced with an end-of-life situation. I could imagine more that a few sweaty brows at Sybase when Coral8 said that they were selling to Aleri. I am sure that Sybase did not want this consternation to leak out to their potential customers. So, better not to mention the association between Coral8 and Sybase RAP. If this was the primary reason for the silence, then I can certainly appreciate that.
But, as the Architect, I would also want to know what Sybase's plans are for the future of their Coral8 engine. Sybase bought the source code to Coral8. Does Sybase plan on getting continuous enhancements from Coral8 as the Coral8 engine and language evolves? Does Sybase plan a separate fork of the source code so that, eventually, the Coral8 engine and language and the Sybase RAP engine and language are two different things? Will Sybase RAP eventually incorporate Aleri's Splash language? These are things that are very important to us. We would need to see Sybase's roadmap as well as Coraleri's roadmap.
Are there other details on the deal? Is this a one time source code tree fork? Will Sybase get new versions from Aleri? It would be interesting to hear if Sybase they will develop it itself further or if bugfixes, patches and new features will come from Aleri’s developers?
John Morrell from Coraleri responds with the very mysterious:
Sorry, we are not releasing more details on the terms of the deal. Those are business terms between Sybase and Aleri.
That's a bit scary. I can understand the financial terms not being released (although it would be interesting to see if Coraleri gets a piece of every Sybase RAP sale, as that would help with Coraleri's financials). But not giving transparency into Sybase's plans is not good news for Sybase, although it is a positive for Coraleri. I have no transparency into whether Sybase will be upgrading their engine based on the upcoming work of Coraleri, and that does not sit well with me.
Sybase RAP can be used without the CEP portion. However, I do not know what other parts of RAP are dependent on other third-party vendors, and whether these other parts are subject to any risk. And, it looks like it has some overlap with KDB+ and with Vhayu. (I wonder if there is a third-party comparison between the three.)
I will not be recommending Sybase RAP for consideration until some of these underlying questions are answered. The current situation just leaves too many questions unanswered. I would welcome a bit more transparency from Sybase.
A very curious situation between Sybase RAP and Coral8.....
I ran into Ivy Schmerken the other day. She has been an editor and writer at Wall Street and Technology for over 20 years, and I used to read here articles when I started on Wall Street back in 1986.
She told me that she had just written an article on the new Sybase RAP "CEP" product, and asked me what I thought of it. I said, "I know all about it. They licensed the Coral8 engine, and that's how they are doing the CEP part of things." I thought it was public knowledge on Wall Street that Sybase RAP was using Coral8, but I guess that Sybase didn't think so, as they never disclosed this fact to Ivy in her interview.
Ivy followed up on this fact with the people from Sybase, and then it came out that Sybase was using Coral8 under the hood. Ivy's news story was then picked up by other news sources like Finextra, and all of a sudden, it was out that Sybase was using Coral8.
This prompted an interesting post from John Morrell of Coraleri that was posted on the Coraleri blog.
I am frankly puzzled why Sybase did not disclose the Coral8 partnership up front. Surely they realize that most of Wall Street (at least the people who care about CEP) knew about the partnership. I was told about this by Sybase themselves. Wombat used Coral8 under the hood in their Acumen product, and in fact, it was Wombat's strong endorsement of Coral8 that helped us in our choice of CEP engines.
I would have thought that it would be to Sybase's benefit that they tout the fact that a tried and tested CEP engine was at the heart of their new offering. Can anyone shed some insight on the thought process here?
It is now March 9th, 2009 somewhere in the world. And, the atomic bomb news that I have been mysteriously referring to for the past two weeks is ….
ALERI BUYS CORAL8
Not quaking in your boots? Well, this IS earth-shattering news in the small community of Complex Event Processing. But, you guys were good guessers. No, I am not leaving my company (yet). No, I did not take the CTO position at a Complex Event Processing vendor (none has been offered yet). No, I did not get escorted out off of the trading floor by a gaggle of security guards (yet). No, I didn’t run off to play marimba with Magma.
However, it confirms the predictions of many people in the blogosphere that the consolidation amongst CEP vendors is starting to happen. And, I am sure that this consolidation is not over, both in the CEP space and in the infrastructure space. Already, I am hearing of financial troubles among several of the infrastructure players, and it would be really wise of everyone out there to re-engage your vendors in a little bit of financial due diligence. Make sure that their source code is in escrow …. not that you would ever do anything with it.
I had been predicting that Coral8 would be acquired for quite some time. Terry Cunningham had been personally bank-rolling the company since its inception, and unless Terry went short on the market, I am sure that his portfolio has suffered like the rest of ours. I can only speculate whether financial considerations inspired Terry to sell the company, or whether Terry just got the itch to move on to his next adventure (although he did take the role of Chairman of the new company). To be honest, when I heard that a large database company had purchased a source code license for Coral8, I thought that the end was near for our intrepid band of CEP’ers from Mountain View.
My curiosity was aroused when I received a dinner invitation from Terry about two weeks ago. Since I have been burning the midnight oil due to my new role and responsibilities, I politely declined, saying that I was just two exhausted to go out at night and play. However, Terry knew how to appeal to the closet “foodie” in me, and picked a very nice restaurant at Columbus Circle that I have only been able to dream about. Imagine my surprise when I got to the restaurant, and found Terry locked arm-in-arm with my other favorite CEO, Don DeLoach of Aleri. I thought that they had united to jointly serve me with the lawsuit that I deserved for publicly taking their products to task on my blog.
(I am taking a bit of literary license here … I actually found out about the merger the night before from another source, so I was not too surprised to see Don there.)
Why did they single me out for such attention? I guess that having a big mouth and a title pays off sometimes. But as a customer of the newly combined company, they wanted to know how a typical customer would react, and what effect the merger might have on us. Would we continue to buy licenses, or would we go running to a competitor? Or, did they want me to give them love and kisses on my blog?
So, in between lot of glasses of wine and a lot of arm-waving on my part, I expressed a number of concerns. But, also, there is lots of goodness in this merger.
There have always been some issues with Coral8. First and foremost, they had no financial domain expertise. They had attempted to rectify that recently by hiring Colin Clark, but I had my doubts whether that strategy would result in anything concrete. Second, they had no true presence in New York, and if you are going to market to the financial services industry, you have to at least have an office in New York City. Third, they had no European penetration, and I could imagine that Apama, Streambase and Aleri were cleaning Coral8’s clock in EMEA. Fourth, they were one of the only CEP vendors who did not have a vertical product for trading or risk, and they really had no intentions of writing one.
Here was this nice 40-person company, sitting in their nice, sunny offices in Mountain View, 3000 miles away from their main customer base, without a clue of what the capital markets companies really needed. However, Coral8 is an extremely nice company, very friendly to developers, writing lots of nice white papers, and putting out a good product that was less expensive than all of the other products (except Esper).
Then, there was Aleri. Lots of financial domain expertise, with a few products that were geared towards trading firms. Good European presence. Ex-Bell Labs rocket scientists working in a basement office in Mountainside, right across the street from their old Bell Labs haunts. Not a very developer-friendly company. Very hard to configure the product. Documentation that was not as good as Coral8's (although my last experience with their docs was the old 2.4 version). A better IDE than Coral8, but that’s not saying much as Coral8 has a minimalist IDE. Not very good at testing and QA (based on 2.4 experiences, and some recent feedback from some customers in London). Some sort of real-time OLAP strategy. More adapters geared towards financial services companies.
Now, Aleri has bought all of the assets of Coral8. What does the combined company have to offer us? It has a European presence, it has financial expertise, it has a good documentation writer, it has a person who knows .NET, it has people who are concerned about compelling visualizations, it has a procedural language called Splash, and it has two engines and two SQL-ish languages. And it has two pricing models, with Coral8 coming in at $15K per core and Aleri costing about $27K per core.
What did you say? Did you say two engines and two languages? What’s a customer to think?
First, let’s tackle the issues of the engineers. You have two very smart teams located on different sides of the USA. Each team is justifiably proud of the CEP engines that they wrote. One engine came out of a StanfordUniversity research group, and the other engine came out of a commercial effort from Bell Labs. (Correction: the engine is not the same one that they developed at Bell Labs.) Eventually, the combined CorAleri is going to have to choose one language and one engine. So, you will have two sets of development teams positioning themselves as the winner of the language and engine competition.
Every customer of Coral8 MUST program the engine in CCL, which is Coral8’s SQL-like language. However, even though Aleri supports an SQL language, almost nobody uses it, as the Aleri IDE does not support it. (The Aleri IDE likes to generate XML-based code.) Let’s assume that Coral8 wins the language and engine competition. However, in my vision, Aleri gets their licks in when CCL is supplemented with the Splash language.
So, in my ideal world, the language of CorAleri would be CCL with some Splash. And, since the language and the engine are linked tightly, Coral8 keeps the engine as well. However, Aleri would help the Coral8 engine team to make their engine even faster.
At first, CorAleri will have to support two engines and two languages. But, the longer you postpone the single-engine strategy, the more time you give the Coral8 and Aleri engineers to improve their individual engines in order to win the engine competition. Then, it becomes more difficult to have a single-engine strategy.
Don and Terry talked about a single product that includes BOTH engines. The Aleri engine would be the “performance engine” and the Coral8 engine would be the “deterministic engine”. If you check a certain checkbox, “stuff” would be sent to the Coral8 engine and if you check another checkbox, “stuff” would be sent to the Aleri engine.
You cannot have two engines running simultaneously on a single server. They would be competing for resources. We have to worry about administering another process on our servers. We wouldn’t be sure which engine was handling what. It makes monitoring more difficult.
If you give a customer too many choices, then the customer might get confused. Customers want transparency and simplicity. If you confuse a customer, then that customer might change over to a CEP vendor who offers a simpler solution. This means that a confused customer might run over to Apama or Streambase.
CorAleri must come up with a solid roadmap about the merging of the languages and engines, and they must make this roadmap available to customers.
We do not want to take 25% of each sprint to play catch-up on a moving target. The worst thing for us is to be surprised. With each release of the new CorAleri engine and language, we want the migration effort to be painless. We want to change as little of our existing CCL scripts as possible, and we don’t want to find that queries that previously worked do not work anymore.
To me, this is the greatest challenge to CorAleri and the greatest risk to us.
I would like to see the engine and language and the documentation stay with the Mountain View folks. What would the Aleri folks do? They could concentrate on writing more adapters, work on integration with other partners (ie: Netezza, Gemfire, RTI, etc), come up with a better IDE with debugging a monitoring support, come up with more Jack Rusher-type visualizations, real-time OLAP and analytics, and develop some more financial apps. This would give a clear line of separation of duties between the two development teams.
In summation, there is a lot to be excited about, but there is also risk that is associated with confusion.
I wish CorAleri good luck. They have some really good people. They have a sense of integrity. They have a new sense of purpose. And, if they could come up with a unified roadmap that makes sense, then it would be a company to be reckoned with in the CEP marketplace.
Scott posted this message on the Coral8 user's group on LinkedIn. Look at his stuff and give him some feedback.
Coral8 & PowerShell
If you use Coral8 on Windows, and you use Powershell for other development and admin tasks, you may be interested in the powershell navigation provider and cmdlets which I've uploaded to
Colin told me that I can post the good news .... he just accepted the job of Executive Vice President of Financial Services at Coral8, reporting directly to the CEO.
This is a great thing for Coral8.
As I have written here before, I feel that financial services domain knowledge was not one of Coral8's strong points, and I think that they would be the first to admit this. I personally think that it is difficult to target Wall Street firms from the dry air of Silicon Valley. I would venture to say that not one of the people who we interact with at Coral8 comes from a financial services background, and although they tried very hard, they were never able to give us a full-time support person in New York, let alone someone who knew anything about trading apps.
Now in the space of a few weeks, Coral8 bagged Mike DiStefano of Gemstone and Colin Clark, the ex-CEO of Kaskad. And, just when I thought that Coral8 was going to de-emphasize their efforts in financial services (not that I would have blamed them!).
Colin will be building up a financial services organization in the US and in continental Europe and the UK. Coral8 has no current sales organization on the right side of the pond, and when we go global with our CEP system, it is nice to have local support from the vendor.
What is great for us is that Colin actually cares about what we, as a capital markets firm, do with Coral8. And, let me tell you ... we have a laundry list for Colin that will keep him busy for many months. We have requests in the areas of real-time OLAP, entitled subscriptions, persistence, object cache connectivity, performance improvements with ad-hoc queries in the Coral8 public windows, better .NET support (which we have been promised already), better profiling and performance monitoring, better documentation, and more.
And, did I mention that Colin is a pilot? (My goal is to have our vendors stocked with pilots and percussionists.)
Welcome to Coral8, Colin. I hope you are ready for us, cause we are coming at you like one of those planes that you see in the Reno Air Races.
We have a C#/.NET-based Reuters market data adapter that we wrote ourselves (well, actually, I wrote it). This is an out-of-process Coral8 adapter that reads real-time ticks from RMDS, turns then into Coral8 tuples, and feeds them into our Coral8 engine. The adapter has two caches in it, and since market data is last reliable, we can publish the last-reliable ticks from one of the caches at specific intervals.
When we hooked the market data adapter up to our Coral8 engine, we started seeing an immediate backup in the pending message queue. We were sending Coral8 about 3000 ticks per second. In addition, our Coral8 engine was processing our order flow, which could hit a max of another 3000 messages per second.
My first thought was that it was taking too long for Coral8 to deserialize the market data tuple, so I wanted to transform our market data adapter from an out-of-process adapter to an in-process adapter. Unfortunately, in-process adapters have to be written as a vanilla, Win32-based C/C++ DLL.
I grew up on C++. However, I have not touched a lick of C++ since diving into .NET in 2001. I was scared. I was frightened. However, I know that my teams needs some experience in writing in-process adapters for Coral8, and since my team consists of very strong .NET and Java people, I decided to bite the bullet and take one for the team.
Man, oh man! I can't believe how much I had forgotten! Strange symbols like '*' and '->' were coming from my fingers. Compilation errors sprang up everywhere. There was no Resharper to help me along. Linker errors to strange obfuscated functions that I swear I never wrote started appearing in the error window.
Perhaps I need to use a little bit of extern "C" here? Maybe there? Maybe everywhere!
Where was my .NET Thread class? Where was System.Timer? How do I do events? There is a new kind of pain that you have to go through in order to get callbacks to work within templated C++ classes. It took me hours to get simple callbacks to work.
Finally, after two days of work, I got the market data adapter ported over to Coral8. To start with, let's try reading the ticks from a flat data file instead of from Reuters. Whew, it works. I see the ticks appearing in the Coral8 stream viewer. Now, let's try Reuters. After a bit of tweaking and some crashes on the destructors, ticks are finally coming in.
Then, I get an email from our Coral8 developer telling me that the reason why the market data feed was slow in Coral8 was because of one of those mysterious coral8 queries that he wrote that clogged the system! So, my inproc adapter was not needed anymore. Market data was now zooming through our Coral8 engine.
Nevertheless, it was a fascinating experience to return to my C++ roots from a 7-year hiatus in the .NET world. Never again!
From the head of our CEP engine development team (used with his permission) :
I'm pleasantly surprised at how fast c8 is when one gets it right. (On the other hand, one misstep and performance goes over the cliff.)
This is one of the main problems with the variants of Streaming SQL that you find in many of the CEP engines. It is not at all transparent what goes on under the hood of all of the CEP engines when you are writing complex queries in Streaming SQL.
Unlike Microsoft, where you find very good profiling tools in SQL Server, the CEP vendors do not provide the necessary tools that will enable a developer to isolate bottlenecks. This has been an issue with Coral8, which they recognize and hope to remedy in the future.
Every time that I've cursed out Coral8 for what I might think is lousy performance, it has turned out that it was us that managed to write a query that clogged the engine. And, when you clog up Coral8, it can bring an entire machine down very quickly as the pending message count builds up.
The important thing is about choosing a CEP vendor is the level of technical support, and despite some turnover in Coral8's tech support staff (goodbye to the excellent Trahn), we have been able to have access to their chief architect and to the head of development when we have really gotten ourselves into trouble. Coral8 also recently hired a New York-based support person who used to be an architect with Gemstone. So, even though it will take this guy some time to get up to speed with Coral8, we are glad to have a local person to help us when we need the assistance.
Coral8 has just released version 5.5, which will be a major help to us in terms of real-time OLAP and entitled subscriptions. It is a major release that will force us to refactor some of our code. But, it is good to see that Coral8 is making it easier for us to implement our real-time dashboards where every user can possibly be entitled to see a different view of data. I feel that support for real-time OLAP is going to be a major selling point for CEP systems, as it is *really hard* to implement it totally on your own.
I am reading a Powerpoint presentation by Prof. Eric Zivot on SPlus, and I was intrigued by the ad-hoc query facility that SPlus offers. For example, if you have TAQ tick data loaded into SPlus, you can use the following SPlus query to get the average price of Microsoft over a non-overlapping 5-minute window:
Coral8 recently implemented queryable "public windows" as a way to do ad-hoc queries. In order to get this feature out the door quickly, Coral8 embedded a version of SQLite into their system. So, you cannot use the Coral8 streaming query language, CCL, to do ad-hoc queries of windows. Instead, you need to use a dialect of SQL-92.
(Coral8 tells me that eventually. you will be able to use CCL for ad-hoc queries)
SPlus looks like it supports a lot of built-in functions for time-series financial data. This is something I would also like to see in Coral8. (I guess there is an opportunity for a little cottage industry to be built around enhancing Coral8 for capital markets firms...)
I wonder if any of the CEP vendors considered SPlus/R as a query language before deciding on Streaming SQL? (I know that JOS has been on the R bandwagon for a while at Barcap. He hasn't blogged too much about it recently though...)
During the Coral8 dinner the other night, I told Terry Cunningham that a stronger sense of community would help their product. There are two things that I would like to see:
1) An annual user conference. This can coincide with the annual Gartner Conference on CEP. This year's Gartner conference is in Stamford, Connecticut, which is very close to all of the Wall Street firms (and the home to many hedge funds).
One of the Coral8 customers at the dinner was extoling the virtues of Parallel Queries. Another user was giving tricks and tips for pumping in massive amounts of data into Coral8. Plus, some of the non-financial customers of Coral8 are doing some pretty cool things with event processing. There are a number of compelling stories that can be told at a users conference.
2) A user forum on the Coral8 website.
Oh ... and a full-time New York-based sales engineer would be fantastic, but I know that Coral8 is working hard on that. However, you need to be able to pass Mark's tech screening, so study up!
One of the nice things about having kids that are a little older is that it gives me time to putter around on my laptop while watching the Yankees games on TV. I am not doing day-to-day development in Coral8, having handed that aspect of the project over to HH. However, I wanted to see if the entitlements processing of our system could be done in Coral8, which made logical sense.
A brief recap: Our CEP system takes a number of atomic events, puts them through the Coral8 cruncher, and produces derived events. However, we don’t want everyone to see these derived events. We might have information in a derived event that a Prop Trader should not see, or we might have information about a certain financial sector that should be hidden from someone on a trading desk who does not cover that sector.
In addition, we have different kinds of notification mechanisms (GUIs, message buses, email, chat, SMS, etc) that should be utilized depending on the severity level of an event. We don’t want to send several hundred emails to a trader for informational events. However, we might want to email and SMS a trader if we have a “red alert” type of event.
So, we will turn to a familiar pattern called the Recipient List. This is one of the well-documented patterns in the book Enterprise Integration Patterns. I get a good amount of email that asks me for advice on becoming a trading systems developer. My advice is to run, not walk, to this website and book. Most of this stuff is old hack to experienced trading systems developers, but the use cases (especially the one by Jonathan Simon) is worth its weight in gold.
We have come up with a schema and database of entitlement information that marries our users/groups list, severity levels, notification mechanisms, and derived events. As every derived event gets generated by our CEP system, we want to put it through the “entitlements grinder” and come up with a Recipient List of who can see what information in the message, and how they want to be notified of its occurrence.
This seems to be a perfect task for a CEP engine. It can be just one more additional “enrichment filter” whose input we attach to the output of the derived event stream. The output of this enrichment filter consists of the (possibly modified) derived event and the Recipient List.
As an initial step, we implemented the Recipient List Generator as a single SQL query using SQL Server 2005. It is a single query that consists of 2 inner joins and 2 outer joins. It works fairly well.
When I was watching the Yankees game yesterday, I tried porting this query to Coral8. I could not get any variation of this query to compile properly, and when I tried to decompose the query into 4 streams, I got totally different results that what SQL Server gave me. Ideally, “Streaming SQL” languages should be a superset of SQL92. So, in Coral8, if I mirror each SQL Server table as a Coral8 Window with a “KEEP ALL” property, then I should be able to use my SQL Query directly. I would like to do something like this:
INSERT INTO RecipientListOutputStream SELECT [my original SQL query]
I have given the guys from Coral8 a homework assignment, and asked them to try to take my query and schema and make it work in Coral8.
So, after a frustrating two hours in which I tried to decipher the Coral8 reference documentation and compiler, I decided to turn to another strategy. For shits-and-giggles, I decided to try to write my SQL query in LINQ. I downloaded the experimental Visual LINQ Query Builder from http://code.msdn.microsoft.com/vlinq. I created a new Visual Studio project, pointed the LINQ data sources to the entitlements database, and started plugging away on the VLINQ. In about ten minutes, I had a full LINQ query that implemented my SQL Server query.
(Note: VLINQ was fairly slow on my laptop, and I soon gave up on it, preferring to code the query in LINQ myself. However, Coral8 and other CEP vendors should look at it as a prototype of a visual code generator.)
LINQ has a lot of goodness to it. LINQ is pervasive, and all flavors of LINQ are being developed. I can very well imagine that Microsoft is looking at versions of LINQ that could handle streaming data. Right now, I think that it would be fairly easy to hook up LINQ queries in a pipeline that would handle simple queries on streaming data. Adding streaming SQL constructs is very doable.
If Microsoft was to come out with a Streaming LINQ that is available as part of .NET, how would this affect the world of CEP? An immediate casualty might be NEsper, but that’s OK, since NEsper is just Aaron’s side project right now. But, longer term, I think that a combination of WCF, Streaming LINQ, and a version of Microsoft Analysis Services that was further geared to real-time streams would be a killer to the rest of the CEP industry. (Of course, technology is one thing. Getting all of those Java and Linux bigots over to .Net is another thing.)
Coral8 just released version 5.3. We asked them for a KDB+ adapter, and they delivered. It was our opinion that KDB+ is used so frequently in capital markets firms that it made perfect sense that the coral8 developer should be able to read data from KDB as easily as they can read data from Oracle or SQL Server. Right now, you still need to write Q queries in Coral8's KDB adapter in order to fetch data from KDB+ .... I was hoping for a way that a developer can write a simple SQL statement and have the KDB adapter translate the query into Q, but that will have to wait for a future version. We ask them to write this stuff and make it available in their core product in the hopes that a lot of people use it, debug it, and ask Coral8 for more enhancements. They did a very basic version, and it will be enhanced per customer demand.
Coral8 also released a Reuters market data adapter, and from what I understand, it will be offered as a separate product that costs a not-too-trivial amount of money. Our internal framework has built-in market data adapters, so we won't be leveraging the Coral8 adapter, especially since it seems like a very vanilla adapter.
It is good that Coral8 has become aware of all of the various adapters needed for firms that do trading. It has been about 6 months since we had to explain what a FIX message was to the Coral8 people, and they have caught on pretty quickly. Coral8 probably had the least amount of captial markets experience of any of the CEP vendors, and they are rapidly catching up.
I read with interest the "exciting announcement" that the banking products side of Aleri were bought by Wall Street Systems. I don't know quite what to make of this. Was this a much-needed infusion of capital? Does Aleri really want to concentrate solely on CEP? Was the banking side of Aleri under-performing? I had lunch yesterday with a vendor of products that are sold into the capital markets space, and the vendor mentioned that the main CEP products that are evaluated are Apama and Streambase, with Coral8 gaining more and more interest. Combined with the difficulty of seeling into capital markets right now, I wonder if this is the first shoe to drop at Aleri.
(A note to PR agencies and Aleri newsletter writers ... you must think that the lives of your readers are pretty drab if you consider the above announcement to be "exciting".... you need to get out of your offices and see what kind of party animals your potential customers are!)
On the plus side, Aleri seems to still be the only company willing to take the STAC challenge. Where are you, Coral8? Apama? Streambase? Esper?
Sprint 2 has completed, and we are starting up Sprint 3. One of the things that is weighing heavily on my mind is the subject of entitlements and Derived Events. Certain users should not see certain derived events at all. Certain users should only see the partial contents of certain derived events.
There is no standard entitlements framework out there. Every IB that I have been with has had multiple custom-built entitlement frameworks. Morgan Stanley had at least 3, and 2 years ago, there was a group that had just been formed in order to build the mother of all entitlement frameworks.
Our entitlements framework needs to work hand-in-hand with our message bus. Most CEP applications are fairly simplistic, and their output goes into a single system, so there is no need for entitlements. Other CEP applications want to publish everything out to everyone ... a surveillance-type CEP application might be crippled if its output is only going to the security guard who is in the middle of a doughnut break. We can't let the prop traders see agency flow, and vice-versa. We can't let certain people on a single desk see what others on the desk are doing ... but the head of the desk should be able to see everything, and if the head of the desk is on vacation, the notifications should be transmitted to a chain of proxies.
It would be great if this kind of feature was built into a CEP engine ... given a derived event and a list of fine-grained entitlements, produce a "recipient list" of what message bus topics we send the derived event to.
Here are two comments that I received yesterday, and my answers:
1) You bought an CEP Engine that doesn't support event clouds?
We feel that Coral8 does support event clouds, but we are looking for the best pattern to implement it. Mark, who is the CTO of Coral8, doesn't quite agree with the term "event cloud". His posting here highlights his argument. According to Mark, an "event cloud" can be represented as multiple event streams, something that Coral8 supports.
2) How and why did you abstract the CEP engine in your system?
First, the why. We want to insulate ourselves from any uncertainties concern the CEP engine, both in terms of the product itself and of the company. In this economic environment, we are concerned that some of these smallish CEP companies might be strained. Ones who are backed by Venture Capital might find their VC's getting worried and thinking that we are reliving those inglorious times from 2002 to 2003. Ones who are privately financed might find that the backers want to move into other areas. It is no secret that most firms are cutting back or delaying their software purchases, and the ones who get impacted first are the smaller niche companies.
We also want to have some flexibility in case the CEP engine itself does not function as advertised. Coral8 has given us great support, but we have not stressed it yet. We know other companies who have evaluated Coral8 who have foudn some shortcomings, things that the Coral8 staff have addressed. However, it is perfectly within the realm of possibility that we may need to consider another CEP engine should Coral8 fall on its face.
Now, the how ....
We are not using Coral8's native input and output adapters. We are not even reading databases using Coral8's PollFromDatabase and ReadFromDatabase adapters. We have an input server that is used to read static and real-time data and marshall that data into coral8 tuples. On the other side, we have an output server that takes the derived event tuples from Coral8, marshalls them into a common format, and does various kinds of alerting and visualizations. From the days that we did evaluations of other CEP vendors, we have layers in our input and output servers that deal with Aleri and Streambase. In other words, we have our own adapters! Changing from Coral8 to Streambase or Aleri involves a simple edit to our Spring-like configuration files.
In addition, we have the ability to farm out work to other engines, such as KDB+. We can then read the derived events that are generated by other systems (as long as they are in our common format) and put them into the CEP engine's "event cloud".
In our architecture, we have introduced extra hops in order to abstract the CEP engine. But, we are not that concerned, since we are dealing with analysis and alerting rather than trading.
Assume that the LastTrades window has the retention policy of KEEP LAST PER Symbol.
Here is some code (provided by Mark of Coral8) to do an upsert.
INSERT INTO LastTrades SELECT SPC.symbol, SPC.price, If LT.volume Is Null Then 0 Else LT.volume End If FROM StreamPriceCorrections SPC Left Outer Join LastTrades LT ON SPC.Symbol = LT.Symbol;
If we need to do an update rather than an upsert, this piece of code works:
INSERT INTO LastTrades SELECT SPC.symbol, SPC.price, LT.volume FROM StreamPriceCorrections SPC, LastTrades LT ON SPC.Symbol = LT.Symbol;
When we went down to Orlando last fall to attend the Gartner Summit on Complex Event Processing, we went with eyes wide open. We were new to the domain of CEP, and one our missions was to try to pick a vendor for the CEP engine that would drive our efforts to produce a major CEP system for our Equities business.
There were a bunch of event-processing systems that were not under consideration because it seemed that they had moved into the strictly vertical area of Algo Trading. These CEP systems included Truviso and Skylar. We needed a general-purpose CEP system, and we wanted to only consider systems that still had a generalist product. An exception to this rule was Aleri, a company who had just come out with a Liquidity Management System as a separate product. We thought that Aleri would still keep its focus on the core CEP engine, so it warranted inclusion of our evaluation.
Apama fell into the Algo trading vertical, but Apama still has a general purpose CEP engine. However, when we tried to evaluate Apama, we were told that we had to go through the dog-and-pony marketing show, something that we did not want to do. I am not sure if this requirement was brought on by the purchase of Apama by Progress Software, a company who I think of as being Computer Associates Lite. I was also told about some interesting experiences between Apama and a major bank by a former colleague of mine whose opinion I trust, and this also influenced by decision to evaluate Apama. This was unfortunate, as I happen to side more with Apama on the whole EPL vs Stream SQL debate.
This left four systems: Streambase, Coral8, Esper, and Aleri.
Coral8 had always been the front-runner, mostly due to recommendations by some former colleagues who were at Merrill Lynch. I had always liked the “vibe” surrounding Coral8, and their openness at giving out eval copies of their software.
Readers of my blog know that I had strong negative opinions about Streambase because of the aggressiveness of their marketing department, an opinion which was shared by a lot of people out there. Nevertheless, their new CEO, Chris Risley, contacted me personally and told me that he had addressed my concerns. After Chris and Richard Tibbetts came down to NYC to meet with me, we decided to include Streambase in the evaluation.
I really wanted to try Esper, but there were a few things that worked against them. The primary factor was that the .NET version, NEsper, was something that was developed by the hard-working Aaron Crackajaxx for his business needs, and did not seem to be part of the mainline Esper product line. We are a .NET shop here, and we needed a product that supported .NET as a first-class citizen. If Aaron decided to become disinterested in Nesper, or if he moved on to another company, then where would we be? We also preferred a product that had an entire ecosystem built around it. So, we passed on Esper. However, Esper is still very much on my radar screen, and I am interested to see how Thomas continues to develop the company and the product.
We spent a good deal of time evaluating Aleri, and most of these experiences were detailed in past entries in this blog. We really wanted to see Aleri succeed, as they were a local company, staffed with a lot of very smart and gentile ex Bell Lab-ers. However, we felt that their product was not ready for us, mostly because of what I called the “spit and polish” issues. I won’t rehash the details now, but if you are interested, please go back and read the old entries in this blog. The areas that needed improvement in the Aleri product were the Aleri Studio, the documentation, and the integration of external data sources.
I met a good deal with Don DeLoach, the CEO of Aleri, and the one positive that will come from my rejection of Aleri is a renewed focus by Aleri on the aesthetics of their product. You can already see these efforts by reading the new Aleri Blog. From what Don had told me a few months ago, their 3.0 product will start to focus on easier integration of external data sources, and will have much improved documentation. I look forward to seeing their efforts come into fruition.
Coral8 was always the front-runner in our evaluation. Their engine is written in C++. They had a decent .NET SDK that let you build out-of-process adapters in C#, and also let you interact with the internals of the Coral8 engine. Their documentation was good, although a bit obtuse at times, and the documentation was backed up by a ton of whitepapers that are available on their website. Their Coral8 Studio gives you a source code view of development, and the GUI part of the Studio is updated after every compile of the source. The CEO of the company was the person who created Crystal Reports, and knows what it takes to build a software company. But, most of all, their CTO and President interacted with us all of the time, and was extremely receptive to our ideas on improving his product. I like when a CTO and the pre-sales engineer send me mail on a Sunday morning!
Streambase was a strong contender. They had some great features in the product that Coral8 has only just come out with (ie: windows that are bucketed by column value). Their GUI is very strong, and their documentation and tutorials are first class. However, I have to say that the interest that Coral8 has shown in our success, and the availability of their CTO was what tipped the odds in Coral8’s favor.
By now, you must be saying to yourself “Where’s the meat?” Didn’t we try to soak and stress the various engines? Didn’t we have an OPRA feed running into the engines in an effort to break them? Didn’t we monitor the use of the CPU and other computing resources? Well … no. To tell you the truth, we were relying on STAC Research to try to do that job for us. STAC is only now just starting to get up to speed in the CEP world, and we will be monitoring their efforts in this space. The general feeling is that most of these CEP engines perform in roughly the same manner, and if one of the CEP engines is 5% faster than Coral8, it is not going to sway our decision, since we are not that concerned right now with super low latency. We are more concerned with the intangibles; responsiveness of the support organization, evolution of the product (and our input into the roadmap), support for .NET as a first-class citizen, stability of the company, etc.
Despite choosing Coral8, we have been careful in our architecture to abstract the specific CEP engine, and no external system will know that Coral8 is driving our CEP system. In the same way that CEP engines have abstracted datasources by using pluggable adapters, we have abstracted the CEP engine. Yes, we have chosen Coral8, and so far, we are satisfied by our choice. But, we are also keeping our eyes open for how the other products evolve in the world of Complex Event Processing.
I haven't blogged in about a month ... I have a ton to say, but I have just been too busy with the new employees who have joined my Complex Event Processing team (welcome Hanno, Scott and Feng). But this great news from Tenerife Joel is just too good to pass up.
On this blog a few month ago, we got on the case of KX Systems for making it virtually impossible to download, evaluate and learn if you were not a member of a large financial institution. Several people commented here that they really wanted to check out KDB, but they did not know how to get access to it.
I guess that Simon and Niall have seen the light, and are now making KDB available for personal use.
I would like to take all of the credit for this revelation, but I suspect that one of the motivating factors for KX Systems was the free, unrestricted availability of Coral8 for personal use. Any random person can download a personal-use, single-CPU version of the Coral8 server and development studio for no charge .... this version does not have any restrictions nor any time limits (are you listening, Aleri?).
During our evaluation of CEP systems, we found that Coral8 was the only company that did not put up any barriers to evaluation. Aleri had an annoying time limit. Apama wouldn't even let us download anything without having to run through their marketing gauntlet. Streambase directly gave us a copy of their product, so I am not sure if Streambase has any barriers to evaluation.
All of the CEP vendors face the conundrum of giving out unrestricted copies of their software vs fully qualifying prospective customers. I can understand this from the standpoint of support ... your technical support staff has a limited amount of time, and it usually has to be spent supporting the larger institutions, the ones who will most likely be dishing out several hundred thousand dollars for a license. But, today's independent hacker might be tomorrow's corporate developer .... and goodwill goes a long way.
So, congrats to Simon and Niall from KX. Here's hoping that a new generation of KDB developers will come out of this effort, and as a result, a wider set of tools is available for KDB.
This is from the Coral8 website. You may recognize some of these enhancements as things that may have been deficiencies when I detailed my use case a few months. Some of the enhancements catch Coral8 up to some of the features that its competitors have. (ie: Bucket Windows)
After vowing to bypass Streambase in my CEP engine evaluation, I may be forced to eat crow. I agreed to let Streambase into the evaluation process because I need to have two CEP engines in my project ... one as primary and one as "cold backup". And, for various reasons, Aleri and Esper did not pan out for me.
The new CEO of Streambase, Chris Ridley, came down to New York to meet with me, with his chief architect, Richard Tibbetts, in tow. They acknowledged some of the errors of their over-aggressive marketing, and told me about their sharpened focus on the financial industry marketplace.
They also let me have an eval version of Streambase that is not constrained by any license key, and in the interests of expediency, they graciously allowed me to bypass their eval agreement (which would have taken weeks to make it through my company's legal processes at this time of the year).
I installed Streambase on my laptop. My first impressions are ..... "slick". In other words, all the superficial, glossy stuff that gives the initial impression to a prospective customer is all there. Nice documentation with plenty of graphics, a great interactive tutorial, etc. I was "warned" that Streambase puts a lot of time into their studio and help system, and I can definitely concur. Nice job, guys.
I am going through the tutorials now. Several things jump out at me right away:
1) They use Eclipse as the foundation of their Streambase Studio. I am quickly becoming a fan of Eclipse, especially the way that you can automatically update Eclipse plugins.
2) The development methodology is more "file-based" than the other products. A familiar paradigm to Java/C# developers.
3) There are two ways to develop apps. The Event Flow method uses a GUI-based method. You can also program in StreamSQL. Unfortunately, there is no tie-in between the Event Flow and the Stream SQL files. In other words, unlike Coral8, if you make a change in the Event Flow, it does not get reflected in the StreamSQL file. In your project, you can have multiple Event Flow files and multiple StreamSQL files. I would love to be able to develop in either system, and have them automatically translated to the other system.
4) There are certain things that you need to do in the Event Flow system that you cannot do in StreamSQL. There are comments in their demo programs to this effect. I would welcome a document that outlined these differences.
5) I noticed that the icons used in the tool palette are identical to the ones that Aleri uses. Interesting. Someone looked at the other company's product.
6) Richard Tibbetts and Mark Tzimelson are very respectful to each other's work. Nice to see that kind of respect at the technical level.
I just tried to get some info on the Apama Event Processing solution (not their Algo Trading platform,just the simple ESP platform). I filled out a form, and now I have to wait for a Progress sales rep to call to arrange a demo. Even if I want to see an Apama webcast, I need to fill out a form.
Let's contrast this what Coral8 has to offer. Coral8 lets you download the entire developer platform, with all of the documentation included. Everything is included .... there are no important packages that are missing with the eval version. There is no 30-day license key that you have to get. There is no waiting for a salesperson to get in touch. As far as I know, you get everything is ready to go from the time you download the package.
I fail to understand why certain vendors make it so difficult to evaluate a package. In a big financial institution like the one I work for, if you use software in production and this software is not properly licensed and paid for, it is grounds for termination of your job.
Coral8 has the right attitude. Just get it into the hands of as many people as possible as spread the word around.
We are finishing up the first phase of the Coral8 evaluation. This week, I met with Terry Cunningham (who flew his Falcon 10 out to meet us), and I had a great session with Henry, their pre-sales engineer. Terry was the creator of Crystal Reports, and later, the head of Seagate Software. I always have a soft spot in my heart for a fellow pilot .... even if his plane can go faster and higher than mine!
We validated that Coral8 was putting out the same output as our custom app, and I was enlightened on some of Coral8's capabilities that were not so easy to find in their wads of documentation. Although there is much good in Coral8, there were also some gotchas.
- Documentation needs to be consolidated a bit. There are a lot of separate manuals plus technical articles. There needs to be a "cookbook" on their CCL language.
- You cannot test a simple user-defined function without writing an intermediate stream. There is no simple way to dump a variable to the console. In other words, I would like to do this simple thing:
CREATE VARIABLE commission; SET commission = CalculateCommission(); -- this is my user-defined function CONSOLE.WRITE(commission);
- We managed to get the Coral8 Studio to freeze consistently. Luckily, no work was lost. The Coral8 Studio is written using wxWidgets, so I wonder how they do unit-testing on the studio.
My opinion is that, although it is great to have the advanced features, you still need to pay attention on the everyday, little tasks that developers need to do. Henry tells me that, in the future, Coral8 will move to a more Visual Studio, file-based way of developing. I certainly welcome this. Henry spent 8 hours watching me drive. When I was having problems, I verbalized the issues so that Henry could see what I was going through and he could bring the issues back to his management.
On the plus side :
- I have been reading about Coral8's pattern matching capabilities. We will definitely be exploring this.
- Coral8 has a relatively inexpensive barrier to entry. If we have a production, development, and COB servers (all dual or quad-core machines), then it won't break our budget.
- Their software does have any time limits on the evaluation versions. One thing that I do not like is a license key that is only good for 30 days. Given the nature of financial companies, we often get pulled into a lot of side projects. I don't want my time to be in the "thick of things", only to find out that the license key elapsed. Coral8 is very friendly to the evaluator.
Now, on to Aleri. I will be using their new 2.4 release.
