Matt seems to have read my mind. I have just started a dive into Cloud Computing, and today, my questions to various CEP vendors were "Can I use you in the Cloud? Have you deployed to Amazon EC2 yet?"
There are various reasons why, at this time, I feel that we should start looking at the Cloud. I will be posting more on this in the future. But, for apps that are not latency sensitive, and if we can obfuscate our data and get the blessings of our Compliance Department (not an easy thing to do), I would like to think about off-loading some of our CEP processing (as well as some other trading functions) to a public Cloud.
I will be catching up with the Microsoft Orinoco team soon. And, it will be interesting to hear about their plans for Azure.
Meanwhile, this looks like a pretty good blog. Especially this post.....
©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Showing posts with label CEP. Show all posts
Showing posts with label CEP. Show all posts
Wednesday, June 24, 2009
Wednesday, March 11, 2009
Streambase and Quant Fund Corrections in WS&T
Thanks to Penny Crossman and her editor, Greg MacSweeny, for doing some due diligence and subsequent corrections on their article about PhaseCapital's evaluation process in which they chose Streambase. The link is here.
Here is the important paragraph from the revised article:
Last summer, the quant fund began evaluating major complex event processing venders by attending trade shows, checking general market acceptance and investigating research on CEP. "We don't claim we've seen every CEP vendor," Goodell notes. During the fall, the firm drilled down on CEP vendors by meeting with representatives, checking product references (blind and vendor provided), downloading publicly available products, and reading white papers. "Some products appeared to be more flexible than others, and that was a large part of our evaluation process," Goodell notes. "While we wanted to make sure the product would integrate well with our environment and that it was rich enough to allow us the control we needed to implement our algorithms, we also wanted to make sure it was enough of a high-level platform that we did not need to worry about data management issues."
And, the money shot is in the sentence: "We don't claim we've seen every CEP vendor"
This is quite a difference from the assertion of the Streambase press release in which they say:
"PhaseCapital chose StreamBase after a comprehensive evaluation of the leading CEP products. "
Kudos to Penny and Greg for having the integrity to drill down into the original questionable assertions from PhaseCapital and Streambase in their original article.
©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Here is the important paragraph from the revised article:
Last summer, the quant fund began evaluating major complex event processing venders by attending trade shows, checking general market acceptance and investigating research on CEP. "We don't claim we've seen every CEP vendor," Goodell notes. During the fall, the firm drilled down on CEP vendors by meeting with representatives, checking product references (blind and vendor provided), downloading publicly available products, and reading white papers. "Some products appeared to be more flexible than others, and that was a large part of our evaluation process," Goodell notes. "While we wanted to make sure the product would integrate well with our environment and that it was rich enough to allow us the control we needed to implement our algorithms, we also wanted to make sure it was enough of a high-level platform that we did not need to worry about data management issues."
And, the money shot is in the sentence: "We don't claim we've seen every CEP vendor"
This is quite a difference from the assertion of the Streambase press release in which they say:
"PhaseCapital chose StreamBase after a comprehensive evaluation of the leading CEP products. "
Kudos to Penny and Greg for having the integrity to drill down into the original questionable assertions from PhaseCapital and Streambase in their original article.
©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Wednesday, February 04, 2009
Streambase On The Move?
Mark Palmer seems to be turning Streambase in a positive direction since he joined .... either that, or the Streambase marketing machine is in full throttle.
Remarkably, Mark has managed to get testimonials from a number of financial services customers. Usually we are a secretive bunch, and many of the larger financial institutions do not like to have themselves used as a reference customer. Which makes it all the more interesting that Streambase has been able to persuade these customers to go public. Not only that, but Streambase has managed to convince Penny Crossman of Wall Street and Technology to rewrite their press release into an article.
Say what you will about Streambase's sales and marketing staff .... they have proven themselves to be amazingly effective. They have been able to trot out customer references, whereas Coral8 and Aleri have not. This gives the perception that Streambase is eating the competition, whether that perception is true or not.
Mark posted an interesting paragraph:
As Dr. Goodell described, Apama didn't even make the first cut in their evaluation. On the other hand, Aleri, the other final CEP product in PhaseCapital's evaluation, fared well with its high-performance, multi-threaded server architecture.
And, Penny follows it up with:
PhaseCapital evaluated a number of CEP vendors in the trading space, including Progress Apama, Coral8, Aleri and StreamBase.
It's very interesting that Phase Capital found that Aleri was the highest performing CEP engine. This has confirmed something that I have suspected. Aleri has always been touting themselves as the most performant of the CEP engines, and has welcomed the other CEP vendors to take the Aleri Challenge. However, it may have been the lack of spit-and-polish in Aleri's development environment that may have sunk it. I have previously blogged here that I thought that Streambase had the best development experience and documentation out of all of the other products.
I am wondering what factors caused Phase Capital to not choose Coral8. One thing that Streambase has that most of the others do not is built-in adapters for trading companies. As Penny writes:
StreamBase already had an adaptor for the QuickFIX engine as well as adaptors for the data streams provide by Lime Brokerage, PhaseCapital's execution broker. "The building blocks were very complementary to what we're ultimately trying to deploy," Pritchett says.
I have been saying this for a long long time on this blog --- the quality of the CEP engine does not matter as much as the entire ecosystem surrounding the CEP engine. You can have the best, fastest, most expressive CEP engine and language, but if you don't surround it with a compelling ecosystem, then you are sunk. And, Mark Palmer understands this.
©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Remarkably, Mark has managed to get testimonials from a number of financial services customers. Usually we are a secretive bunch, and many of the larger financial institutions do not like to have themselves used as a reference customer. Which makes it all the more interesting that Streambase has been able to persuade these customers to go public. Not only that, but Streambase has managed to convince Penny Crossman of Wall Street and Technology to rewrite their press release into an article.
Say what you will about Streambase's sales and marketing staff .... they have proven themselves to be amazingly effective. They have been able to trot out customer references, whereas Coral8 and Aleri have not. This gives the perception that Streambase is eating the competition, whether that perception is true or not.
Mark posted an interesting paragraph:
As Dr. Goodell described, Apama didn't even make the first cut in their evaluation. On the other hand, Aleri, the other final CEP product in PhaseCapital's evaluation, fared well with its high-performance, multi-threaded server architecture.
And, Penny follows it up with:
PhaseCapital evaluated a number of CEP vendors in the trading space, including Progress Apama, Coral8, Aleri and StreamBase.
It's very interesting that Phase Capital found that Aleri was the highest performing CEP engine. This has confirmed something that I have suspected. Aleri has always been touting themselves as the most performant of the CEP engines, and has welcomed the other CEP vendors to take the Aleri Challenge. However, it may have been the lack of spit-and-polish in Aleri's development environment that may have sunk it. I have previously blogged here that I thought that Streambase had the best development experience and documentation out of all of the other products.
I am wondering what factors caused Phase Capital to not choose Coral8. One thing that Streambase has that most of the others do not is built-in adapters for trading companies. As Penny writes:
StreamBase already had an adaptor for the QuickFIX engine as well as adaptors for the data streams provide by Lime Brokerage, PhaseCapital's execution broker. "The building blocks were very complementary to what we're ultimately trying to deploy," Pritchett says.
I have been saying this for a long long time on this blog --- the quality of the CEP engine does not matter as much as the entire ecosystem surrounding the CEP engine. You can have the best, fastest, most expressive CEP engine and language, but if you don't surround it with a compelling ecosystem, then you are sunk. And, Mark Palmer understands this.
©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Sunday, June 01, 2008
Saturday, February 23, 2008
Of Pipes, CEP, and Traders
In our Complex Event Processing system, we will eventually want to be able to give a front-end to our traders which will let the trader develop ad-hoc streaming queries on real-time streams. We also want to allow the traders to implement their own analytics by doing back-testing on several years’ worth of data.
I have been looking at Yahoo Pipes a bit recently, and it is conceptually the same thing as the kind of GUI that I have been envisioning for the traders. In Yahoo Pipes, you can take the output of one or more high-level objects and pipe it into the inputs of another high-level object.
Here is an example of a Yahoo Pipes gadget:
In our simple use case, let’s say that we define high-level objects that are of interest to traders. Such objects could be
a real-time market data stream,
a historical market data stream,
a real time order flow stream and cache,
a historical order flow stream,
a ticker-to-sector mapper and filter, and
some calculation modules.
These are the kinds of “queries” that we built in the Complex Event Processing engines (Streambase, Coral8, Aleri, NEsper) during our evaluation process. However, just like Domain-Specific Languages (DSL) can be created to make developing applications in a certain domain easier, we can make a domain-specific version of Yahoo Pipes that is geared for analyzing Equities data.
Once the trader has put together his Pipes, the result can be translated into one of the languages that the CEP engine supports. If we encapsulate the code generation module, then we can have code generators for Coral8, Streambase, Aleri, NEsper, and others. I am pretty sure that all of these engines allow you to dynamically register and unregister queries. You can dynamically read the status and output of the CEP engine and feed it back into the GUI.
Aleri and Streambase have GUI builders for their developers that look a lot like Yahoo Pipes. However, the GUIs are not packaged as toolkits; in other words, you could not take the Streambase GUI and adapt it for your own needs. Coral8 does not really have a fancy GUI builder; as mentioned here before, their GUI is fairly spartan. NEsper/Esper does not have any kind of GUI to speak of, as their model of embedding the engine inside of an application is different that the model used by the other vendors.
I have been looking at Yahoo Pipes a bit recently, and it is conceptually the same thing as the kind of GUI that I have been envisioning for the traders. In Yahoo Pipes, you can take the output of one or more high-level objects and pipe it into the inputs of another high-level object.
Here is an example of a Yahoo Pipes gadget:
In our simple use case, let’s say that we define high-level objects that are of interest to traders. Such objects could be
a real-time market data stream,
a historical market data stream,
a real time order flow stream and cache,
a historical order flow stream,
a ticker-to-sector mapper and filter, and
some calculation modules.
These are the kinds of “queries” that we built in the Complex Event Processing engines (Streambase, Coral8, Aleri, NEsper) during our evaluation process. However, just like Domain-Specific Languages (DSL) can be created to make developing applications in a certain domain easier, we can make a domain-specific version of Yahoo Pipes that is geared for analyzing Equities data.
Once the trader has put together his Pipes, the result can be translated into one of the languages that the CEP engine supports. If we encapsulate the code generation module, then we can have code generators for Coral8, Streambase, Aleri, NEsper, and others. I am pretty sure that all of these engines allow you to dynamically register and unregister queries. You can dynamically read the status and output of the CEP engine and feed it back into the GUI.
Aleri and Streambase have GUI builders for their developers that look a lot like Yahoo Pipes. However, the GUIs are not packaged as toolkits; in other words, you could not take the Streambase GUI and adapt it for your own needs. Coral8 does not really have a fancy GUI builder; as mentioned here before, their GUI is fairly spartan. NEsper/Esper does not have any kind of GUI to speak of, as their model of embedding the engine inside of an application is different that the model used by the other vendors.
©2008 Marc Adler - All Rights Reserved
Wednesday, December 26, 2007
CEP Vendors and the Ecosystem
While my wife and son cavort around Australia and New Zealand for the next few weeks (I get to stay home and watch my daughter, who only has one week off from high school), I hope to be able to catch up on some of the blog posts that I owe people.
One of the things that is most important for me in choosing a CEP vendor is the ecosystem that surrounds the CEP engine. In a company such as mine, we need to interface with many different legacy systems. These legacy systems can hold crucial data, such as historical orders, customer trades, market data, volatility curves, customer and security reference data, etc. This data may reside statically in a database, be published out as flow over some kind of middleware, or interfaced with an object cache or data fabric. We have every color and shape of database technology in our firm, whether it be more traditional relational databases like Oracle, SQL Server, and Sybase, or newer tick databases like KDB+.
From the input and output points of the CEP engine, we need seamless integration with all sorts of systems. Most CEP engines have the concept of in-process and out-of-process adapters. In-process adapters are more performant that out-of-process adapters. We would love to see as many in-process adapters delivered out-of-the-box by our CEP vendor. We do not want to spend time writing our own in-process adapters.
So far, none of the CEP vendors support KDB+ as an out-of-the-box solution. In fact, many of the CEP vendors did not even know what KDB+ was. (Is the same true for Vhayu as well?) My feeling is that, if a CEP vendor is going to be successful on Wall Street, then they must support KDB+. Is it even feasible for the CEP vendors to provide an abstraction layer around KDB+, and let the CEP developer write all queries in SQL instead of writing them in K or Q?
One of the most important things that I would like to see from the CEP vendors are tools to enable the analysis of all of the data that pass through the CEP engine. Many groups might not have the budget to hire a specialized mathematician or quant to perform time-series analysis on the data. Learning specialized languages like R or SPlus might not be possible for smaller groups that do not have a mathematical bent. The same goes for packages like Mathematica and Matlab.
Would it be worth it for the CEP vendors to come out with a pre-packaged "stack" for various financial verticals that incorporates analysis tools? Or, would writing a detailed cookbook be better? And, where does the responsibility of the CEP vendor end? Should we expect the CEP vendor to provide a one-stop shop for all of our needs, or should be just expect the CEP vendors to provide strong integration points?
Better yet, does this open up an opportunity for a third party company to provide this service? Like the many laptop vendors who buy a motherboard (the CEP engine), and slap together a disk drive, CD drive, screen and keyboard to make a complete system?
In examining the various CEP vendors, I have come to the conclusion that the offerings from Streambase, Coral8 and Aleri are very similar. Given another year, I might expect each vendor to fill in the gaps with regards to their competitors' offerings, and at that point, we might have practically identical technologies from 3 different vendors. In my opinion, the real win for these CEP vendors will come in the analysis tools they provide.
©2007 Marc Adler - All Rights Reserved
One of the things that is most important for me in choosing a CEP vendor is the ecosystem that surrounds the CEP engine. In a company such as mine, we need to interface with many different legacy systems. These legacy systems can hold crucial data, such as historical orders, customer trades, market data, volatility curves, customer and security reference data, etc. This data may reside statically in a database, be published out as flow over some kind of middleware, or interfaced with an object cache or data fabric. We have every color and shape of database technology in our firm, whether it be more traditional relational databases like Oracle, SQL Server, and Sybase, or newer tick databases like KDB+.
From the input and output points of the CEP engine, we need seamless integration with all sorts of systems. Most CEP engines have the concept of in-process and out-of-process adapters. In-process adapters are more performant that out-of-process adapters. We would love to see as many in-process adapters delivered out-of-the-box by our CEP vendor. We do not want to spend time writing our own in-process adapters.
So far, none of the CEP vendors support KDB+ as an out-of-the-box solution. In fact, many of the CEP vendors did not even know what KDB+ was. (Is the same true for Vhayu as well?) My feeling is that, if a CEP vendor is going to be successful on Wall Street, then they must support KDB+. Is it even feasible for the CEP vendors to provide an abstraction layer around KDB+, and let the CEP developer write all queries in SQL instead of writing them in K or Q?
One of the most important things that I would like to see from the CEP vendors are tools to enable the analysis of all of the data that pass through the CEP engine. Many groups might not have the budget to hire a specialized mathematician or quant to perform time-series analysis on the data. Learning specialized languages like R or SPlus might not be possible for smaller groups that do not have a mathematical bent. The same goes for packages like Mathematica and Matlab.
Would it be worth it for the CEP vendors to come out with a pre-packaged "stack" for various financial verticals that incorporates analysis tools? Or, would writing a detailed cookbook be better? And, where does the responsibility of the CEP vendor end? Should we expect the CEP vendor to provide a one-stop shop for all of our needs, or should be just expect the CEP vendors to provide strong integration points?
Better yet, does this open up an opportunity for a third party company to provide this service? Like the many laptop vendors who buy a motherboard (the CEP engine), and slap together a disk drive, CD drive, screen and keyboard to make a complete system?
In examining the various CEP vendors, I have come to the conclusion that the offerings from Streambase, Coral8 and Aleri are very similar. Given another year, I might expect each vendor to fill in the gaps with regards to their competitors' offerings, and at that point, we might have practically identical technologies from 3 different vendors. In my opinion, the real win for these CEP vendors will come in the analysis tools they provide.
©2007 Marc Adler - All Rights Reserved
Tuesday, December 18, 2007
Farewell Kaskad
A player in the CEP space, Kaskad, is no more. Colin Clark writes in to say that, since the Boston Stock Exchange (BSX) has ceased to function, Colin had to disband Kaskad. I assume that since Kaskad did the work for the BSX as consultants, the BSX maintained all or most of the IP rights. Or, perhaps, Kaskad felt that since their only client was no longer funding their development, they could not gain further VC money in this financial environment.
According to the Aite report on CEP vendors, Kaskad had 16 employees. Since they are based up in Boston, I wonder if Streambase is looking at adding some additional talent.
Colin is looking for new opportunities in the CEP space, so contact him if you have anything that might interest him. Colin was involved in the old NEON (New Era for Networks) back in the dotcom boom, so he has the entrepreneural streak running through him.
©2007 Marc Adler - All Rights Reserved
According to the Aite report on CEP vendors, Kaskad had 16 employees. Since they are based up in Boston, I wonder if Streambase is looking at adding some additional talent.
Colin is looking for new opportunities in the CEP space, so contact him if you have anything that might interest him. Colin was involved in the old NEON (New Era for Networks) back in the dotcom boom, so he has the entrepreneural streak running through him.
©2007 Marc Adler - All Rights Reserved
Friday, December 14, 2007
Streambase (yet again ...)
After vowing to bypass Streambase in my CEP engine evaluation, I may be forced to eat crow. I agreed to let Streambase into the evaluation process because I need to have two CEP engines in my project ... one as primary and one as "cold backup". And, for various reasons, Aleri and Esper did not pan out for me.
The new CEO of Streambase, Chris Ridley, came down to New York to meet with me, with his chief architect, Richard Tibbetts, in tow. They acknowledged some of the errors of their over-aggressive marketing, and told me about their sharpened focus on the financial industry marketplace.
They also let me have an eval version of Streambase that is not constrained by any license key, and in the interests of expediency, they graciously allowed me to bypass their eval agreement (which would have taken weeks to make it through my company's legal processes at this time of the year).
I installed Streambase on my laptop. My first impressions are ..... "slick". In other words, all the superficial, glossy stuff that gives the initial impression to a prospective customer is all there. Nice documentation with plenty of graphics, a great interactive tutorial, etc. I was "warned" that Streambase puts a lot of time into their studio and help system, and I can definitely concur. Nice job, guys.
I am going through the tutorials now. Several things jump out at me right away:
1) They use Eclipse as the foundation of their Streambase Studio. I am quickly becoming a fan of Eclipse, especially the way that you can automatically update Eclipse plugins.
2) The development methodology is more "file-based" than the other products. A familiar paradigm to Java/C# developers.
3) There are two ways to develop apps. The Event Flow method uses a GUI-based method. You can also program in StreamSQL. Unfortunately, there is no tie-in between the Event Flow and the Stream SQL files. In other words, unlike Coral8, if you make a change in the Event Flow, it does not get reflected in the StreamSQL file. In your project, you can have multiple Event Flow files and multiple StreamSQL files. I would love to be able to develop in either system, and have them automatically translated to the other system.
4) There are certain things that you need to do in the Event Flow system that you cannot do in StreamSQL. There are comments in their demo programs to this effect. I would welcome a document that outlined these differences.
5) I noticed that the icons used in the tool palette are identical to the ones that Aleri uses. Interesting. Someone looked at the other company's product.
6) Richard Tibbetts and Mark Tzimelson are very respectful to each other's work. Nice to see that kind of respect at the technical level.
©2007 Marc Adler - All Rights Reserved
The new CEO of Streambase, Chris Ridley, came down to New York to meet with me, with his chief architect, Richard Tibbetts, in tow. They acknowledged some of the errors of their over-aggressive marketing, and told me about their sharpened focus on the financial industry marketplace.
They also let me have an eval version of Streambase that is not constrained by any license key, and in the interests of expediency, they graciously allowed me to bypass their eval agreement (which would have taken weeks to make it through my company's legal processes at this time of the year).
I installed Streambase on my laptop. My first impressions are ..... "slick". In other words, all the superficial, glossy stuff that gives the initial impression to a prospective customer is all there. Nice documentation with plenty of graphics, a great interactive tutorial, etc. I was "warned" that Streambase puts a lot of time into their studio and help system, and I can definitely concur. Nice job, guys.
I am going through the tutorials now. Several things jump out at me right away:
1) They use Eclipse as the foundation of their Streambase Studio. I am quickly becoming a fan of Eclipse, especially the way that you can automatically update Eclipse plugins.
2) The development methodology is more "file-based" than the other products. A familiar paradigm to Java/C# developers.
3) There are two ways to develop apps. The Event Flow method uses a GUI-based method. You can also program in StreamSQL. Unfortunately, there is no tie-in between the Event Flow and the Stream SQL files. In other words, unlike Coral8, if you make a change in the Event Flow, it does not get reflected in the StreamSQL file. In your project, you can have multiple Event Flow files and multiple StreamSQL files. I would love to be able to develop in either system, and have them automatically translated to the other system.
4) There are certain things that you need to do in the Event Flow system that you cannot do in StreamSQL. There are comments in their demo programs to this effect. I would welcome a document that outlined these differences.
5) I noticed that the icons used in the tool palette are identical to the ones that Aleri uses. Interesting. Someone looked at the other company's product.
6) Richard Tibbetts and Mark Tzimelson are very respectful to each other's work. Nice to see that kind of respect at the technical level.
©2007 Marc Adler - All Rights Reserved
Friday, December 07, 2007
On to Esper/NEsper
I have had to expand the CEP evaluation process. I am going to start looking at NEsper, and maybe, Apama (after some recommendations by some of our Asia folks who seemed pleased at Apama's performance over heavy loads).
I just downloaded NEsper and I am starting to go over some of the docs. I am sure that Thomas and Aaron will correct me if I say anything incorrect about Esper. Two things stand out about the Esper offering:
1) No GUI Builder or visual tools, probably because .....
2) Esper/Nesper is a component designed to be incorporated into your application ... in other words, it is treated as a third-party .NET assembly, just like things like Syncfusion, Log4Net, etc.
So, unlike Coral8, where you run a separate Coral8 server process, you need to write an application that "contains" Esper/NEsper. While this solution does not favor quick, out-of-the-box prototyping as Coral8 does, it gives you more control over the CEP actions. Everything with Esper/Nesper is "in-process" to your application.
Also, Esper/Nesper is Open Source. I downloaded it, and I have the full source code sitting on my hard drive. I have to talk to the financial guys at my company, but I don't think that we would have to do the amount of financial due dilligence with an Open Source effort as we would with a company who does not follow the Open Source model. Maybe we will have to count the number of moths that fly out of Thomas' wallet.
©2007 Marc Adler - All Rights Reserved
I just downloaded NEsper and I am starting to go over some of the docs. I am sure that Thomas and Aaron will correct me if I say anything incorrect about Esper. Two things stand out about the Esper offering:
1) No GUI Builder or visual tools, probably because .....
2) Esper/Nesper is a component designed to be incorporated into your application ... in other words, it is treated as a third-party .NET assembly, just like things like Syncfusion, Log4Net, etc.
So, unlike Coral8, where you run a separate Coral8 server process, you need to write an application that "contains" Esper/NEsper. While this solution does not favor quick, out-of-the-box prototyping as Coral8 does, it gives you more control over the CEP actions. Everything with Esper/Nesper is "in-process" to your application.
Also, Esper/Nesper is Open Source. I downloaded it, and I have the full source code sitting on my hard drive. I have to talk to the financial guys at my company, but I don't think that we would have to do the amount of financial due dilligence with an Open Source effort as we would with a company who does not follow the Open Source model. Maybe we will have to count the number of moths that fly out of Thomas' wallet.
©2007 Marc Adler - All Rights Reserved
Friday, November 30, 2007
Financial Due Diligence
In my super-mega Investment Bank, we work with all kinds of vendors. In fact, in our old group, one of the things that we were charged with was investigating all kinds of esoteric technologies that could give us an edge in the trading world.
If you use a vendor for a "bet the farm"-type application, then you want to make sure that the vendor behind the product is rock solid. My boss, who is the Global Head of Equities Technology for the firm, asked me who the "800 lbs gorilla" is in the CEP space. He wanted to make sure that we were safe in our choice, and that no matter how good the technology is, we did not put all of our eggs into a guy working nights in his basement (although, those kinds of companies usually make the best software!).
There is really no 800-lbs gorilla is the "pure" CEP space. By "pure" CEP players, I am talking about guys like Aleri, Coral8, Streambase, Esper, Apama, Truviso, Kaskad, etc. In this space, most of these companies number their customers in the dozens rather than the thousands. These companies are all competing for that big reference customer, the one that companies can stand back and say "This investment bank doubled their trading revenues because of our product."
Compounding this fact is the whole credit crunch and subprime mess. The New York Times had an article yesterday that described the effects that the credit crunch is starting to have on all sorts of companies ... even softeware companies and web-design shops were mentioned. So, we need to make sure that the CEP vendor that we choose is not affected, nor will be affected by the credit crunch. Coincidentally, one ex-employee of a CEP vendor sent me private email that mentioned that his company was undergoing a round of layoffs.
No matter which CEP vendor you choose, you should always have one backup. This is standard practice when dealing with small vendors. You should have the vendor's source code in escrow. You should have your financial people do a deep dive on a vendor's financials. Is the vendor self-funded or are they VC funded? Does the VC have the appetite to wait 5 to 7 years for a good return on their investment? What is the past behavior of the VC firms with regards to startups? What happens if the chief architect of the product leaves the company? Is the product written in a mainstream language, in case you need to take possession of the source code that's in escrow?
These are all questions that you should ask before making the final selection of your vendor.
©2007 Marc Adler - All Rights Reserved
If you use a vendor for a "bet the farm"-type application, then you want to make sure that the vendor behind the product is rock solid. My boss, who is the Global Head of Equities Technology for the firm, asked me who the "800 lbs gorilla" is in the CEP space. He wanted to make sure that we were safe in our choice, and that no matter how good the technology is, we did not put all of our eggs into a guy working nights in his basement (although, those kinds of companies usually make the best software!).
There is really no 800-lbs gorilla is the "pure" CEP space. By "pure" CEP players, I am talking about guys like Aleri, Coral8, Streambase, Esper, Apama, Truviso, Kaskad, etc. In this space, most of these companies number their customers in the dozens rather than the thousands. These companies are all competing for that big reference customer, the one that companies can stand back and say "This investment bank doubled their trading revenues because of our product."
Compounding this fact is the whole credit crunch and subprime mess. The New York Times had an article yesterday that described the effects that the credit crunch is starting to have on all sorts of companies ... even softeware companies and web-design shops were mentioned. So, we need to make sure that the CEP vendor that we choose is not affected, nor will be affected by the credit crunch. Coincidentally, one ex-employee of a CEP vendor sent me private email that mentioned that his company was undergoing a round of layoffs.
No matter which CEP vendor you choose, you should always have one backup. This is standard practice when dealing with small vendors. You should have the vendor's source code in escrow. You should have your financial people do a deep dive on a vendor's financials. Is the vendor self-funded or are they VC funded? Does the VC have the appetite to wait 5 to 7 years for a good return on their investment? What is the past behavior of the VC firms with regards to startups? What happens if the chief architect of the product leaves the company? Is the product written in a mainstream language, in case you need to take possession of the source code that's in escrow?
These are all questions that you should ask before making the final selection of your vendor.
©2007 Marc Adler - All Rights Reserved
Saturday, November 24, 2007
First Use Case Done with Coral8
I now have Coral8 detecting when a sector has abnormal activity, and I have a Coral8 Output Stream publishing into a .NET application for visualization. If I want to, I can take the data from the Coral8 alert, transform it into a JMS message, and publish it out on the Tibco EMS bus for other applications in the organization to consume. Or, I can publish it out to another Coral8 stream.
Well done, Coral8 team!
Now, it's on to the Aleri evaluation. The good people at Aleri's sales engineering team have done most of this first use case, but now that I am armed with more Coral8 knowledge, I need to try to rebuild the Aleri use case from scratch by myself.
©2007 Marc Adler - All Rights Reserved
Well done, Coral8 team!
Now, it's on to the Aleri evaluation. The good people at Aleri's sales engineering team have done most of this first use case, but now that I am armed with more Coral8 knowledge, I need to try to rebuild the Aleri use case from scratch by myself.
©2007 Marc Adler - All Rights Reserved
Thursday, November 22, 2007
More on the first CEP Use Case
Yesterday, I had a great two-hour session with Henry and Bob from Coral8 in which most of the use case was done.
Henry is our designated pre-sales engineer. His job is to do what it takes to make sure that the prospective customer is happy with the product before making a decision to purchase the product. Bob is the head architect of Coral8, and his job (as he described it) is to make sure that the product is as easy to use as possible.
Between Henry and Bob, two solutions were offered. I will go into the first solution in this blog entry. The second solution revolves around custom timestamping of messages by the input adapter, and this topic deserves a blog entry of its own.
The main problem was to analyze the order flow for each sector over a one minute timeslice, and determine if any sectors showed abnormal activity. The problem that I was faced with was that the concept of “time” was determined by the TransactTime field in the FIX message, and not by the “clock on the wall”. So, if for some reason, I received two FIX messages in a row, one whose TransactTime field was 14:24:57 and one whose TransactTime field was 14:25:01, then the receipt of the second FIX message should cause a new timeslice, regardless of what the wall clock said.
The solution that Henry came up with was to use a pulse in a stream. Although the concept of raising an event is very common is programming, it is not really something that you tend to do in SQL stored procedure. The thing is that programming in Coral8’s CCL (as well as the SQL-like dialects that many of the CEP vendors have) is a combination of procedural and SQL programming, and the trick is to find the correct “pattern” to solve your problem. This is where many of the CEP vendors can improve; they can publish a listing of patterns, they can come up with FAQs, etc. I mentioned this to Bob of Coral8, so expect to see some movement on this front from the Coral8 folks.
Here is what the pulse stream looks like in Coral8’s CCL:
----------------------------------------------------------------------------------
-- LastTimeSlice holds the maximum timeslice (0 to 389) of the order stream.
-- When we see an order with a TransactTime greater than the current max timeslice,
-- then we set the new max timeslice. We also use this as a signal (pulse)
-- to one of the streams below.
----------------------------------------------------------------------------------
CREATE VARIABLE INTEGER LastTimeslice = -1;
CREATE LOCAL STREAM stream_Pulse;
INSERT INTO stream_Pulse
SELECT
TimeToTimeBucket(FlattenNewOrder.TransactTime) AS epoch
FROM
FlattenNewOrder
WHERE
TimeToTimeBucket(FlattenNewOrder.TransactTime) > LastTimeSlice;
-- When we insert a new timeslice into the stream_Pulse stream, we also
-- set the new maxmimum timeslice.
ON stream_Pulse
SET LastTimeSlice = stream_Pulse.epoch;
We have a global variable that keep the maximum timeslice that is flowing through our system. Since there are 6.5 hours in the trading day, there are 390 minute-sized timeslices that we want to consider.
In the INSERT statement, if the timeslice from the incoming FIX message is greater than the current maximum timeslice, then we insert a new record into the pulse stream.
The ON statement functions like a trigger. When a new record is inserted into a stream, you can have one or more ON statements that react to the event of inserting a record into the stream. Here, we set the new maximum timeslice.
We need to maintain a Window that contains all of the orders for the current timeslice. The order information includes the stock ticker, the sector that the stock belongs to, the number of shares in the order, and the current timeslice. In Coral8, a Window provides retention of records. You can specify a retention policy on a Window, whether it been a time-based retention policy (keep records in the window for 5 minutes) or a row-based retention policy (keep only the last 100 rows). What is missing here is a retention policy based on a boolean expression or on a certain column value changing. Streambase has this, and Coral8 knows that this feature should be implemented down the road.
----------------------------------------------------------------------------------
-- The TickerAndSector window holds all FIX orders for the current timeslice.
-- Each row of the window contains the FIX order and the sector information.
-- When we see a new timeslice, the TickerAndSelector window is cleared
-- using a DELETE statement.
----------------------------------------------------------------------------------
CREATE WINDOW TickerAndSector
SCHEMA (Ticker STRING, SectorName STRING, SectorId INTEGER, Shares INTEGER, TransactTimeBucket INTEGER)
KEEP ALL;
INSERT INTO TickerAndSector
SELECT
FlattenNewOrder.Ticker,
TickerToSectorMap.SectorName,
TickerToSectorMap.SectorId,
TO_INTEGER(FlattenNewOrder.Qty),
TimeToTimeBucket(FlattenNewOrder.TransactTime)
FROM
FlattenNewOrder,
TickerToSectorMap
WHERE
TickerToSectorMap.Ticker = FlattenNewOrder.Ticker
AND TimeToTimeBucket(FlattenNewOrder.TransactTime) >= LastTimeSlice;
Now that we have a list of orders that occur for the current timeslice, we need to know when a new timeslice occurs. At this point, we need to analyze the orders for the current timeslice, find out which sectors are showing abnormal activity, and clear out the TickerAndSector window so that new orders can be accumulated for the new timeslice.
----------------------------------------------------------------------------------
-- The OrdersPerSectorPerMinute window contains the aggregated totals
-- for each sector for the previous timeslice. The aggregated totals include
-- the number of orders for each sector and the total number of shares for each sector.
--
-- The interesting part of this is the join between the TickerAndSector window
-- and the stream_Pulse. The stream_Pulse will be triggered when we see a new
-- timeslice.
--
-- When we insert rows into the OrdersPerSectorPerMinute window, we will trigger
-- a deletion of the old info in the TickerAndSector window.
----------------------------------------------------------------------------------
CREATE WINDOW OrdersPerSectorPerMinute
SCHEMA (SectorName STRING, SectorId INTEGER, OrderCount INTEGER, TotalShares INTEGER, Timeslice INTEGER)
KEEP 2 MINUTES;
INSERT INTO OrdersPerSectorPerMinute
SELECT
tas.SectorName, tas.SectorId, COUNT(*), SUM(tas.Shares), stream_Pulse.epoch
FROM
TickerAndSector tas, stream_Pulse
GROUP BY
tas.SectorId;
ON OrdersPerSectorPerMinute
DELETE FROM TickerAndSector
WHERE TransactTimeBucket < LastTimeSlice;
As you can see from the above code, when a new timeslice appears, we aggregate the number of orders and the total number of shares that are in the TickerAndSector window. The interesting thing here, and the thing that I might not have figured out on my own, was that we need to join with the pulse stream that we talked about before. The pulse stream here is being used to “kick start” the calculating and dumping of the records in the current timeslice.
Finally, since we have aggregated the information for each sector for the current timeslice, we want to see if any sector exceeded the maximum “normal” number of orders.
----------------------------------------------------------------------------------
-- This output stream will alert the user when a sector exceeds the
-- max orders for that timeslice.
----------------------------------------------------------------------------------
INSERT INTO AlertStream
SELECT
R.SectorId, R.SectorName, R.OrderCount, R.TotalShares
FROM
OrdersPerSectorPerMinute AS R, NormalOrdersPerSectorPerTimeslice AS H
WHERE
R.SectorId = H.SectorId AND R.Timeslice = H.Timeslice AND R.OrderCount > H.MaxOrders;
And, that’s it! If we attach a JMS output adapter to the AlertStream, we can generate a new, derived event, put that event back on the EMS bus (or we can send it into another Coral8 stream), and alert some kind of monitoring application.
Thanks to the Coral8 guys for helping me slog my way through the learning process.
©2007 Marc Adler - All Rights Reserved
Henry is our designated pre-sales engineer. His job is to do what it takes to make sure that the prospective customer is happy with the product before making a decision to purchase the product. Bob is the head architect of Coral8, and his job (as he described it) is to make sure that the product is as easy to use as possible.
Between Henry and Bob, two solutions were offered. I will go into the first solution in this blog entry. The second solution revolves around custom timestamping of messages by the input adapter, and this topic deserves a blog entry of its own.
The main problem was to analyze the order flow for each sector over a one minute timeslice, and determine if any sectors showed abnormal activity. The problem that I was faced with was that the concept of “time” was determined by the TransactTime field in the FIX message, and not by the “clock on the wall”. So, if for some reason, I received two FIX messages in a row, one whose TransactTime field was 14:24:57 and one whose TransactTime field was 14:25:01, then the receipt of the second FIX message should cause a new timeslice, regardless of what the wall clock said.
The solution that Henry came up with was to use a pulse in a stream. Although the concept of raising an event is very common is programming, it is not really something that you tend to do in SQL stored procedure. The thing is that programming in Coral8’s CCL (as well as the SQL-like dialects that many of the CEP vendors have) is a combination of procedural and SQL programming, and the trick is to find the correct “pattern” to solve your problem. This is where many of the CEP vendors can improve; they can publish a listing of patterns, they can come up with FAQs, etc. I mentioned this to Bob of Coral8, so expect to see some movement on this front from the Coral8 folks.
Here is what the pulse stream looks like in Coral8’s CCL:
----------------------------------------------------------------------------------
-- LastTimeSlice holds the maximum timeslice (0 to 389) of the order stream.
-- When we see an order with a TransactTime greater than the current max timeslice,
-- then we set the new max timeslice. We also use this as a signal (pulse)
-- to one of the streams below.
----------------------------------------------------------------------------------
CREATE VARIABLE INTEGER LastTimeslice = -1;
CREATE LOCAL STREAM stream_Pulse;
INSERT INTO stream_Pulse
SELECT
TimeToTimeBucket(FlattenNewOrder.TransactTime) AS epoch
FROM
FlattenNewOrder
WHERE
TimeToTimeBucket(FlattenNewOrder.TransactTime) > LastTimeSlice;
-- When we insert a new timeslice into the stream_Pulse stream, we also
-- set the new maxmimum timeslice.
ON stream_Pulse
SET LastTimeSlice = stream_Pulse.epoch;
We have a global variable that keep the maximum timeslice that is flowing through our system. Since there are 6.5 hours in the trading day, there are 390 minute-sized timeslices that we want to consider.
In the INSERT statement, if the timeslice from the incoming FIX message is greater than the current maximum timeslice, then we insert a new record into the pulse stream.
The ON statement functions like a trigger. When a new record is inserted into a stream, you can have one or more ON statements that react to the event of inserting a record into the stream. Here, we set the new maximum timeslice.
We need to maintain a Window that contains all of the orders for the current timeslice. The order information includes the stock ticker, the sector that the stock belongs to, the number of shares in the order, and the current timeslice. In Coral8, a Window provides retention of records. You can specify a retention policy on a Window, whether it been a time-based retention policy (keep records in the window for 5 minutes) or a row-based retention policy (keep only the last 100 rows). What is missing here is a retention policy based on a boolean expression or on a certain column value changing. Streambase has this, and Coral8 knows that this feature should be implemented down the road.
----------------------------------------------------------------------------------
-- The TickerAndSector window holds all FIX orders for the current timeslice.
-- Each row of the window contains the FIX order and the sector information.
-- When we see a new timeslice, the TickerAndSelector window is cleared
-- using a DELETE statement.
----------------------------------------------------------------------------------
CREATE WINDOW TickerAndSector
SCHEMA (Ticker STRING, SectorName STRING, SectorId INTEGER, Shares INTEGER, TransactTimeBucket INTEGER)
KEEP ALL;
INSERT INTO TickerAndSector
SELECT
FlattenNewOrder.Ticker,
TickerToSectorMap.SectorName,
TickerToSectorMap.SectorId,
TO_INTEGER(FlattenNewOrder.Qty),
TimeToTimeBucket(FlattenNewOrder.TransactTime)
FROM
FlattenNewOrder,
TickerToSectorMap
WHERE
TickerToSectorMap.Ticker = FlattenNewOrder.Ticker
AND TimeToTimeBucket(FlattenNewOrder.TransactTime) >= LastTimeSlice;
Now that we have a list of orders that occur for the current timeslice, we need to know when a new timeslice occurs. At this point, we need to analyze the orders for the current timeslice, find out which sectors are showing abnormal activity, and clear out the TickerAndSector window so that new orders can be accumulated for the new timeslice.
----------------------------------------------------------------------------------
-- The OrdersPerSectorPerMinute window contains the aggregated totals
-- for each sector for the previous timeslice. The aggregated totals include
-- the number of orders for each sector and the total number of shares for each sector.
--
-- The interesting part of this is the join between the TickerAndSector window
-- and the stream_Pulse. The stream_Pulse will be triggered when we see a new
-- timeslice.
--
-- When we insert rows into the OrdersPerSectorPerMinute window, we will trigger
-- a deletion of the old info in the TickerAndSector window.
----------------------------------------------------------------------------------
CREATE WINDOW OrdersPerSectorPerMinute
SCHEMA (SectorName STRING, SectorId INTEGER, OrderCount INTEGER, TotalShares INTEGER, Timeslice INTEGER)
KEEP 2 MINUTES;
INSERT INTO OrdersPerSectorPerMinute
SELECT
tas.SectorName, tas.SectorId, COUNT(*), SUM(tas.Shares), stream_Pulse.epoch
FROM
TickerAndSector tas, stream_Pulse
GROUP BY
tas.SectorId;
ON OrdersPerSectorPerMinute
DELETE FROM TickerAndSector
WHERE TransactTimeBucket < LastTimeSlice;
As you can see from the above code, when a new timeslice appears, we aggregate the number of orders and the total number of shares that are in the TickerAndSector window. The interesting thing here, and the thing that I might not have figured out on my own, was that we need to join with the pulse stream that we talked about before. The pulse stream here is being used to “kick start” the calculating and dumping of the records in the current timeslice.
Finally, since we have aggregated the information for each sector for the current timeslice, we want to see if any sector exceeded the maximum “normal” number of orders.
----------------------------------------------------------------------------------
-- This output stream will alert the user when a sector exceeds the
-- max orders for that timeslice.
----------------------------------------------------------------------------------
INSERT INTO AlertStream
SELECT
R.SectorId, R.SectorName, R.OrderCount, R.TotalShares
FROM
OrdersPerSectorPerMinute AS R, NormalOrdersPerSectorPerTimeslice AS H
WHERE
R.SectorId = H.SectorId AND R.Timeslice = H.Timeslice AND R.OrderCount > H.MaxOrders;
And, that’s it! If we attach a JMS output adapter to the AlertStream, we can generate a new, derived event, put that event back on the EMS bus (or we can send it into another Coral8 stream), and alert some kind of monitoring application.
Thanks to the Coral8 guys for helping me slog my way through the learning process.
©2007 Marc Adler - All Rights Reserved
Saturday, November 17, 2007
CEP Vendor Thoughts
Recently, I came across an article on Streambase in Windows in Financial Services magazine. One of the questions to the head of Streambase went like this:
WFS: Does StreamBase have any competitors?
BM: The major players have not yet delivered anything in this space. IBM, for example, does not have a project to build a technology like this. We are IBM’s solution in this space.
In my opinion, this answer totally evades the question. What happened to companies like Aleri, Coral8, Esper, Apama, Skyler, Truviso, Kaskad, etc? How about the IBM offering that Opher is working on? Alll of these companies freely acknowledge Streambase as a worthy competitor, and rightly so. It would be nice to see Streambase acknowledge the same. Brown University certainly was not the only university doing CEP research and not the only one to commercialize their offerings.
And shame on Microsoft and Windows in Financial Services magazine for letting this slip by. Are you a journalistic effort or a fluff rag?
In our evaluation of CEP vendors, we chose not to evaluate Streambase for various reasons. Streambase might have the best technology of all of the CEP vendors (for example, look at Tibbets comment from a few weeks ago on a question about cancelling events), but we will never get to find out. The people who I feel badly for at Streambase are the dedicated development and support staff who have probably come up with a really good product.
(In the interest of fairness, Bill from Streambase told me recently that they had reduced the price of their offering, which was one of our concerns.)
And, if anybody from Streambase reads this blog ---- doing an end-run around me and trying to market directly to the business will not earn you any points. The business people rely on me to make the right decision, and all of your email to the business side (as is any email from information technology vendors to the business side) gets forwarded directly to me. And, I guess that we will end up paying real dollars to your imaginary competitors.
Meanwhile, let's take the attitudes of Coral8 and Aleri. One of these companies JUST hired its first salesperson. Their mantra was that the product should be the best that it can be before it was pushed by a salesforce. The other company has a low-key sales approach too. They have gone beyond the call of duty to incorporate our suggestions into their product and to come up with a POC that really impresses us.
Both vendors have come up with FIX input adapters at our behest. Aleri has incorporated some of our suggestions into their FlexStreams, and has cleaned up some of their visual development studio. (With FlexStreams, you can use a procedural programming language to create custom processing for streams). I am impressed in what these companies have done to earn our business. I feel that, in exchange for these companies doing some of what we want, they get to expand their offerings for the capital markets communities, and bring themselves out of the narrow focus of algorithmic trading and pricing engines.
Kudos to Mark, John, Henry and Gary of Coral8, and to Don, John, Jerry, Jon, David, etc of Aleri. All very nice people, and all trying compete honestly for a piece of the pie.
In my opinion, the Coral8 and Aleri offerings are so close that we will eventually be choosing one vendor as primary and the other as hot backup. What needs to be done is performance evaluation. Pushing multiple streams of fast moving data into the CEP engine and seeing their performance under heavy load. Let's see if they can handle the data rates that come at 2:15 PM on a Fed decision day.
One message that we have been hearing from the CEP and messaging vendors is that they perform better under Linux than Windows Server 2003. This is probably not a surprise to most people on Wall Street. But, I wonder what Windows Server 2008 has to offer in comparison to Linux. The November 8, 2007 article at Enhyper has some interesting things to say about Microsoft's marketing of the London Stock Exchange deal. We will most likely be running our CEP engine on Linux unless Microsoft comes up with a real compelling reason to the contrary.
©2007 Marc Adler - All Rights Reserved
WFS: Does StreamBase have any competitors?
BM: The major players have not yet delivered anything in this space. IBM, for example, does not have a project to build a technology like this. We are IBM’s solution in this space.
In my opinion, this answer totally evades the question. What happened to companies like Aleri, Coral8, Esper, Apama, Skyler, Truviso, Kaskad, etc? How about the IBM offering that Opher is working on? Alll of these companies freely acknowledge Streambase as a worthy competitor, and rightly so. It would be nice to see Streambase acknowledge the same. Brown University certainly was not the only university doing CEP research and not the only one to commercialize their offerings.
And shame on Microsoft and Windows in Financial Services magazine for letting this slip by. Are you a journalistic effort or a fluff rag?
In our evaluation of CEP vendors, we chose not to evaluate Streambase for various reasons. Streambase might have the best technology of all of the CEP vendors (for example, look at Tibbets comment from a few weeks ago on a question about cancelling events), but we will never get to find out. The people who I feel badly for at Streambase are the dedicated development and support staff who have probably come up with a really good product.
(In the interest of fairness, Bill from Streambase told me recently that they had reduced the price of their offering, which was one of our concerns.)
And, if anybody from Streambase reads this blog ---- doing an end-run around me and trying to market directly to the business will not earn you any points. The business people rely on me to make the right decision, and all of your email to the business side (as is any email from information technology vendors to the business side) gets forwarded directly to me. And, I guess that we will end up paying real dollars to your imaginary competitors.
Meanwhile, let's take the attitudes of Coral8 and Aleri. One of these companies JUST hired its first salesperson. Their mantra was that the product should be the best that it can be before it was pushed by a salesforce. The other company has a low-key sales approach too. They have gone beyond the call of duty to incorporate our suggestions into their product and to come up with a POC that really impresses us.
Both vendors have come up with FIX input adapters at our behest. Aleri has incorporated some of our suggestions into their FlexStreams, and has cleaned up some of their visual development studio. (With FlexStreams, you can use a procedural programming language to create custom processing for streams). I am impressed in what these companies have done to earn our business. I feel that, in exchange for these companies doing some of what we want, they get to expand their offerings for the capital markets communities, and bring themselves out of the narrow focus of algorithmic trading and pricing engines.
Kudos to Mark, John, Henry and Gary of Coral8, and to Don, John, Jerry, Jon, David, etc of Aleri. All very nice people, and all trying compete honestly for a piece of the pie.
In my opinion, the Coral8 and Aleri offerings are so close that we will eventually be choosing one vendor as primary and the other as hot backup. What needs to be done is performance evaluation. Pushing multiple streams of fast moving data into the CEP engine and seeing their performance under heavy load. Let's see if they can handle the data rates that come at 2:15 PM on a Fed decision day.
One message that we have been hearing from the CEP and messaging vendors is that they perform better under Linux than Windows Server 2003. This is probably not a surprise to most people on Wall Street. But, I wonder what Windows Server 2008 has to offer in comparison to Linux. The November 8, 2007 article at Enhyper has some interesting things to say about Microsoft's marketing of the London Stock Exchange deal. We will most likely be running our CEP engine on Linux unless Microsoft comes up with a real compelling reason to the contrary.
©2007 Marc Adler - All Rights Reserved
Sunday, November 04, 2007
Help Wanted for the Complex Event Processing Project
I have open headcount for about 4 or 5 people for 2008 for the Complex Event Processing project that I am running.
I realize that it is foolhardy to advertise for people who have prior experience in CEP. What I am looking for are smart developers who have a great passion to learn a new, interesting technology. The team that I envision will consist of:
1) Visualization developer - come up with new, interesting ways to visual events and data. The work may entail working with the .NET framework that my team has built, integrating visualizations with existing Java/Swing-based trader GUIs, or even exploring WPF (as the company gradually embraced .NET 3.x). You could be investigating visualization tools like heatmaps and you will definitely be evaluating third-party tools (both commercial and open-source). You will be involved in OLAP to some extent. There will be involvement in the building out of a notification and alerting framework.
2) CEP developer. You will be building out the analysis part inside the CEP engine. Most of the CEP engines use a variant of SQL, so you should be fairly comfortable with SQL concepts. It would be nice if you had previous experience with tools like Coral8, Aleri, Streambase, Esper, etc, but even if you haven't, you should be willing to learn these tools. You may also be interacting with consultants from these companies.
3) Networking, messaging, and market data specialist. Help us decide if we should migrate to a new messaging infrastructure (like RTI or 29West). Experience with Tibco EMS is a big plus, as well as experience with working with high volumes of data and low latency. Interact with Reuters and Wombat infrastructures, as well as internally-built market data infrastructures.
4) Data specialist. You will be the person who is responsible for breaking down silos and getting good data into the CEP engine. Experience with SQL Server 2005 and Sybase are important. Experience with tick databases like KDB+ and Vhayu would be nice to have.
Everyone will be doing a bit of everything, so everyone on this team will be intimately aware of what everyone else is doing.
This is a highly-visible position in an investment bank that has promised me that they will reward good talent who comes to us from the outside.
In addition to the positions mentioned above, I have two or three open headcount for people who want to work on the Ventana team. Ventana is the client-side .NET framework that is being used by various groups in our Investment Bank.
©2007 Marc Adler - All Rights Reserved
I realize that it is foolhardy to advertise for people who have prior experience in CEP. What I am looking for are smart developers who have a great passion to learn a new, interesting technology. The team that I envision will consist of:
1) Visualization developer - come up with new, interesting ways to visual events and data. The work may entail working with the .NET framework that my team has built, integrating visualizations with existing Java/Swing-based trader GUIs, or even exploring WPF (as the company gradually embraced .NET 3.x). You could be investigating visualization tools like heatmaps and you will definitely be evaluating third-party tools (both commercial and open-source). You will be involved in OLAP to some extent. There will be involvement in the building out of a notification and alerting framework.
2) CEP developer. You will be building out the analysis part inside the CEP engine. Most of the CEP engines use a variant of SQL, so you should be fairly comfortable with SQL concepts. It would be nice if you had previous experience with tools like Coral8, Aleri, Streambase, Esper, etc, but even if you haven't, you should be willing to learn these tools. You may also be interacting with consultants from these companies.
3) Networking, messaging, and market data specialist. Help us decide if we should migrate to a new messaging infrastructure (like RTI or 29West). Experience with Tibco EMS is a big plus, as well as experience with working with high volumes of data and low latency. Interact with Reuters and Wombat infrastructures, as well as internally-built market data infrastructures.
4) Data specialist. You will be the person who is responsible for breaking down silos and getting good data into the CEP engine. Experience with SQL Server 2005 and Sybase are important. Experience with tick databases like KDB+ and Vhayu would be nice to have.
Everyone will be doing a bit of everything, so everyone on this team will be intimately aware of what everyone else is doing.
This is a highly-visible position in an investment bank that has promised me that they will reward good talent who comes to us from the outside.
In addition to the positions mentioned above, I have two or three open headcount for people who want to work on the Ventana team. Ventana is the client-side .NET framework that is being used by various groups in our Investment Bank.
©2007 Marc Adler - All Rights Reserved
Saturday, September 22, 2007
CEP Conference Report
I thought that the CEP conference would be a sleepy little symposium, but I was wrong.
Gartner ran their conference on Business Process Monitoring on the Monday-Wednesday morning, and it dovetailed exactly into the CEP conference. There was also an academic conference on CEP that ran concurrently with the BPM conference, so you had a lot of academics and curiosity-seekers at the CEP conference.
I would estimate that there were about 200 attendees at the CEP conference, but probably half were from the vendor community. I counted about 50-60 attendees in the financial services track at the conference, and again, about 40-50% were vendors. So, there were probably about 30 folks from the financial services industry who were really curious about CEP.
I give these numbers to illustrate that, in financial services, CEP is still in the curiousity stage. I am happy to say that most of the people there were just as confused as we were (ie: what is the difference between a CEP engine and a rules engine, what is the best messaging middleware to use, do we really need a visual designer, etc). Most of the people who were using CEP were using it in a simple stream case ... pricing, algos, fraud detection, etc. I did not see any use cases that approach the magnitude of what we have to do, but then again, I don't expect organizations like Goldman Sachs to advertise what they are doing.
It was nice to see some of the vendors trotting out their successes in financial services (Aleri with HSBC, CommerzBank and Barcap, Streambase with BofA, Coral8 with Wombat), and start to establish some meaningful partnerships. Nice to see that Coral8 is working closely with RTI.
Mary Knox of Gartner did a nice presentation on what it takes to get CEP adopted by a large financial organization, but she compensated for this nice presentation by monopolizing the Q&A session with Robert Almgren of BofA. I was a bit puzzled by IBM's message about what parts of their CEP framework were already available in Message Broker, who was supporting it in IBM, who we get services from, etc.
I was impressed with Mark from Coral8, who seems to bring that Russian mad scientist mentality to his company. Also encouraging to see that Terry Cunningham is at the helm at Coral8, fresh from his successes at Crystal and other companies. Terry knows what it takes to make a successful software company. They also seem to be on the top of their game with financial services, without restricting themselves solely to algo trading.
Aleri seems like they have some raw power in their platform, and over time, they will be polishing their message more. They win the award for the most genial vendors of the conference.
The key takeaways were:
1) The CEP industry is still at its infancy.
2) It's a fairly hot buzzword right now.
3) The complete stack is important if you do large deployments. We need to understand all of the pieces surrounding the CEP engine.
4) Message volumes are increasing at an exponential rate in financial services.
5) It will be a long battle to break all of the silos in our company.
©2007 Marc Adler - All Rights Reserved
Gartner ran their conference on Business Process Monitoring on the Monday-Wednesday morning, and it dovetailed exactly into the CEP conference. There was also an academic conference on CEP that ran concurrently with the BPM conference, so you had a lot of academics and curiosity-seekers at the CEP conference.
I would estimate that there were about 200 attendees at the CEP conference, but probably half were from the vendor community. I counted about 50-60 attendees in the financial services track at the conference, and again, about 40-50% were vendors. So, there were probably about 30 folks from the financial services industry who were really curious about CEP.
I give these numbers to illustrate that, in financial services, CEP is still in the curiousity stage. I am happy to say that most of the people there were just as confused as we were (ie: what is the difference between a CEP engine and a rules engine, what is the best messaging middleware to use, do we really need a visual designer, etc). Most of the people who were using CEP were using it in a simple stream case ... pricing, algos, fraud detection, etc. I did not see any use cases that approach the magnitude of what we have to do, but then again, I don't expect organizations like Goldman Sachs to advertise what they are doing.
It was nice to see some of the vendors trotting out their successes in financial services (Aleri with HSBC, CommerzBank and Barcap, Streambase with BofA, Coral8 with Wombat), and start to establish some meaningful partnerships. Nice to see that Coral8 is working closely with RTI.
Mary Knox of Gartner did a nice presentation on what it takes to get CEP adopted by a large financial organization, but she compensated for this nice presentation by monopolizing the Q&A session with Robert Almgren of BofA. I was a bit puzzled by IBM's message about what parts of their CEP framework were already available in Message Broker, who was supporting it in IBM, who we get services from, etc.
I was impressed with Mark from Coral8, who seems to bring that Russian mad scientist mentality to his company. Also encouraging to see that Terry Cunningham is at the helm at Coral8, fresh from his successes at Crystal and other companies. Terry knows what it takes to make a successful software company. They also seem to be on the top of their game with financial services, without restricting themselves solely to algo trading.
Aleri seems like they have some raw power in their platform, and over time, they will be polishing their message more. They win the award for the most genial vendors of the conference.
The key takeaways were:
1) The CEP industry is still at its infancy.
2) It's a fairly hot buzzword right now.
3) The complete stack is important if you do large deployments. We need to understand all of the pieces surrounding the CEP engine.
4) Message volumes are increasing at an exponential rate in financial services.
5) It will be a long battle to break all of the silos in our company.
©2007 Marc Adler - All Rights Reserved
Tuesday, September 18, 2007
CEP Vendors, DBMS's and Defense Industry
I made a comment in my last posting about RTI, a small messaging and CEP vendor. It seems that RTI got its start in the defense industry. Coincidentally, IBM is allowed to mention the fact that its System S arose out of a 4-year defense contract.
CEP vendors who grew out of the DBMS sector might have certain advantages, but ones who grew out of defense have other advantages. First, you would expect that anything that had to pass Military testing and certification would be industrial strength. These systems would have to be able to handle a huge throughput of signals and would have to be able to react correctly to the input .. after all, you don't want our ships firing a missle at a seagull. Second, the analysis tools have to be top-rate ... detecting a signal and firing a missle is much like pulling the trigger on a trade. Both have enormous consequences, and once out there, cannot be retracted.
I am not completely positive, but I can imagine that messaging systems in the defense industry might not have to worry about guaranteed delivery. It seems to be OK to drop a few signals (events messages) that the radars send out, and eventually (a few milli-seconds later), the signals would resume.
©2007 Marc Adler - All Rights Reserved
CEP vendors who grew out of the DBMS sector might have certain advantages, but ones who grew out of defense have other advantages. First, you would expect that anything that had to pass Military testing and certification would be industrial strength. These systems would have to be able to handle a huge throughput of signals and would have to be able to react correctly to the input .. after all, you don't want our ships firing a missle at a seagull. Second, the analysis tools have to be top-rate ... detecting a signal and firing a missle is much like pulling the trigger on a trade. Both have enormous consequences, and once out there, cannot be retracted.
I am not completely positive, but I can imagine that messaging systems in the defense industry might not have to worry about guaranteed delivery. It seems to be OK to drop a few signals (events messages) that the radars send out, and eventually (a few milli-seconds later), the signals would resume.
©2007 Marc Adler - All Rights Reserved
Saturday, September 15, 2007
Another New Position (again)
Last week, I was about to finalize my transfer to the Derivatives Analytics desk when an amazing opportunity was offered to me by my company. I cannot blog too much about this new opportunity other to say that it involves Complex Event Processing and transformational new directions for trading and decision making in Equities. This is a huge initiative for my company, is being sponsored by the top people on the business side in Equities, and goes all the way up to the CEO of the company. I will be leading this new project on the technical end, and I am working with one of the movers-and-shakers on the business end.
This is not the standard algorithmic trading scenario that is the standard use case for vendors like Apama, Skyler, etc. We will be pushing the limits of the CEP and messaging vendors. We have the mandate to have all of our Equities systems interact with our. And, if everything goes right, it will transform the way that systems are written within our company.
The nice thing is that the Business is giving us several months of pure R&D time. Reading papers, attending conferences, meeting vendors, etc. For me, it is coming out of a silo and being given the opportunity to learn about every phase of our systems for trading equities. And, it will be building upon all of the things that I have been investigating for the past year in my Architecture role.
©2007 Marc Adler - All Rights Reserved
This is not the standard algorithmic trading scenario that is the standard use case for vendors like Apama, Skyler, etc. We will be pushing the limits of the CEP and messaging vendors. We have the mandate to have all of our Equities systems interact with our. And, if everything goes right, it will transform the way that systems are written within our company.
The nice thing is that the Business is giving us several months of pure R&D time. Reading papers, attending conferences, meeting vendors, etc. For me, it is coming out of a silo and being given the opportunity to learn about every phase of our systems for trading equities. And, it will be building upon all of the things that I have been investigating for the past year in my Architecture role.
©2007 Marc Adler - All Rights Reserved
Subscribe to:
Posts (Atom)
