Almost every day, we ask ourselves if making the decision to use a commercial CEP system was worth it. If you have been following my blog for the past 14 months, you have traced my thought processes around choosing whether to build vs. buy, and my choice of vendor. And, almost every day, when doing my 30 minutes of elliptical, my mind wanders and I start reflecting on my decision and wondering if it was correct to buy a commercial CEP system.
Our system does not do any sophisticated pattern matching yet. Right now, we are doing a lot of aggregations, groupings, rollups, and simple comparison testing in order to generate alerts and real-time dashboard information.
As long-time blog readers know, we were able to get a working prototype of a very early version of our system up and running in a few days with hand-crafted C# code. But, that’s just us. We had a lot of the elements already in place that the commercial CEP systems offer. But, what if we didn’t have that framework to leverage?
Let’s say that I recently got laid off from my Wall Street company (a very real situation nowadays), and I and a few friends decided to start a small prop trading shop out of my basement. We are a 3 traders and a developer. We need to build ourselves a small algo or high-frequency trading system. What do we need?
We need real-time market data from Reuters or Bloomberg, or if I am interested in something really fast, I might try Activ. We need to persist ticks and results into a database, so we will need both a tick database (something like KDB or Vhayu) and a standard database like SQL Server or MySQL. We need feed handlers that can talk to the engine that maintains our models. We need a GUI so the traders can monitor the market and the orders. We need a FIX engine so that we can send orders out to the markets.
I give my developer explicit orders to get it done as fast as possible. I don’t want him spending time writing a beautifully crafted framework that solves a general problem. I want him to write the simplest thing possible. If we are using Bloomberg, I want him to use the BBAPI to get the market data and put it into the trading engine in the fastest possible way and with the lowest latency. If we have to aggregate something, I want him to take the data, sum it up, and stick it into some hash table that is indexed by symbol.
With all of my requirements in place, I know that it would take my tech guy 6 months to get us up and trading on this new system. It would cost us roughly $100K in development costs. However, our opportunity cost is huge, as we have not been able to put on some of the trades that we would have liked to.
What does it cost to purchase a commercial CEP system? Coral8 charges $15K per core for the primary machine. There are discounts for the HA (high availability) machines. Aleri charges $26K per core. When we met with Streambase in early 2007, they told us that is would take $500K to get us up and running (I don’t know if they have reduced their price.) If I have a 4 CPU dual-core machine, then it would cost $120K for Coral8 and $208K for Aleri. Plus, most companies charge a 15-20% annual maintenance fee.
What do I get for that amount?
I get basically a real-time matching engine. I get a high-level language that, if my traders bother to learn and don’t get themselves into trouble with, will let my traders “tweak” their models. I get a bunch of input and output adapters that will let me get information in and out of the engine. (I might have to pay a lot more for a Reuters or Bloomberg adapter, and that is only if my vendor has one.) I get an IDE that, along with the high-level language, I can use to develop the models.
What do I give up if I buy a commercial CEP engine? I give up the right to fix the internals of the engine if I find a bug. I am at the mercy of the vendor’s release cycle. If I find a bug, and the vendor tells me that it will be fixed in 3 months, when the next release comes out, I won’t be happy. 3 months is really 4 months, because initial major releases always have problems with them. I give up the right to add new features when I need them. I am living in fear that, in this fiscal environment, the VC’s might decide to pull all funding out from under the vendor’s feet.
On the plus side, I get to tap into the vendor’s customer support staff, thus freeing my programmer to do other things. I might get to interface with other users of the CEP system so I could learn best practices.
Now that I have my commercial CEP system, what now? I have to get my tech guy to learn a new language (the Streaming SQL variants that are offered by the CEP vendors are easy to learn, but as we have come to discover, if you don’t use it properly, you can get yourself into a big mess). If we don’t have a specific input adapter (Bloomberg) or output adapter (FIX) that the vendor provides, then my developer has to write it himself. My developer still has to program the models, but now we have to worry if the vendor’s language is powerful and flexible enough to support the programming of the model.
If we find a blocking bug in the CEP engine, or if we find a language feature that we desperately need, then we are at the mercy of the vendor.
On the other hand, the vendor has a highly tuned multi-threaded engine, so that’s a big chunk of development that we don’t have to do. The CEP engine has cool things that we need, like sliding windows and nested pattern matching.
It’s a big decision to make. If I work at a large company, and I have several different frameworks available to me, and I have 5 full-time developers, then I could see writing everything from scratch. If I was a small startup, and I could not afford the market data fees AND the CEP engine, then I might look to a free solution like Esper. If I was a 5-20 person shop, then my decision is tougher.
What would make the decision easier would be to see if a commercial CEP vendor has a vertical application that solved our needs. A trading system in a box. That's the direction that Apama has taken, and that Mark Palmer is trying to pursue with Streambase. Coral8 will follow that path with the hiring of Colin, and Aleri has its eyes on that direction too.,
Subscribe to:
Post Comments (Atom)
4 comments:
Hi Marc,
Well written and spot on, as usual. I have responded with, Will Commercial CEP Engines Replace Algorithmic Trading Platforms?
http://www.thecepblog.com/2008/11/15/will-commercial-cep-engines-replace-algorithmic-trading-platforms/
I don't think my post is as insightful as yours; however, I offer a slightly different perspective, which I hope is complimentary.
Yours sincerely, Tim
www.thecepblog.com
I'll just say that if you ask yourself this question every day, then something is wrong. I'll bet that it's daily wondering if the language itself is a powerful time saver or a time wasting pain in the a$$.
Marc
You seem to discuss build bs buy and there are a few things I would like to add to your perspective:
- Open source such as Esper blurs the border. Some call that build vs buy vs borrow. I call this build vs buy vs blend.
Open source is one way of neutralizing vendor lockin of all kind especially when there are no widespread standards. This said we usually observe that applying your own fixes or features atop an open source core engine (say Esper) is actually real non-trivial work. I believe that is why Esper is applying a commercial open source model - you are free to take a ride but can also call for help if needed.
- Several CEP platforms allow you to extend the event processing language (custom view, patterns, guards, functions etc). They also often allow custom code in the middle of an event processing pipeline if you want to do some custom processing that the platform does not implement. Sounds like a key criteria to me. Platforms that have an event processing network (EPN) of some kind and come from a middleware angle do bring value there.
- I think quoting (innacurate?) raw vendor license cost is far too simplistic. This has to be put in perspective to your expected ROI, the features and your expected use, and we also all know every vendor will have discount policies, various add-ons, different pricing unit etc (core vs server vs subscription; unlimited license of some form; non-production pricing terms etc).
I get a bit emotional when I consider the huge costs involved in setting up a modern-day electronic trading platform. Everyone starts from scratch every time, and when we are in small companies we tend to overpay just to get some leverage over the crushing development cycles needed just to integrate someone else's overpriced systems, much less build it out ourselves.
This is the 21st century equivalent of paper tickets swamping the street in the 50's and 60's -- we all think we are getting things done but I know many seriously talented individuals who all spend most of their time just evaluating, negotiating licenses, pieceing systems together, and have precious little time left over to do whatever it is that we are supposed to be doing.
I rant, but perhaps you can see my point. How many teams for example have written their own "out of process" quickfix adapter? All the gains of a great OSS project like quickfix and then everyone either spends $100k/year too much on integration or spends time building their own.
I hope that Esper solidifies like quickfix did as an OSS project that can make some consultants some honest money. It seems narrowly focused enough and it solves a core problem.
Post a Comment