I have posted this information before in previous blog entries, but I think that, since I have some new readers here, I should post a list of some of the things that are important to me in a CEP system.
Input Adapters
-------------------
Market Data - Reuters, Bloomberg, and to a lesser extent, Wombat and Activ
FIX - 4.2, 4.4. Predefined schemas for the most popular FIX messages would be great.
SWIFT
Message Buses - Tibco EMS, RTI, LBM
Tick Databases - KDB+ (read and write capabilities, don't make us code in Q), Vhayu
Object Caches - Velocity, Gemfire, Coherence
Output Adapters
---------------------
Various chat/messenger mechanisms
Message Buses - Tibco EMS, RTI, LBM
KDB+, Vhayu
Object Caches - Velocity, Gemfire, Coherence
Development Environment
----------------------------------
Strong support for debugging
Profiling and performance monitoring
Hooks into Performance Counters
Ability to easily add C#-based custom functions
Analysis
-----------
Integration with tools like R, SPlus, Matlab
Real-time OLAP
Strong support for real-time dashboards
Miscellaneous
------------------
Workflow
Entitlements on output events
Integration with Active Directory
Priority input queues into the CEP engine
©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Subscribe to:
Post Comments (Atom)
12 comments:
Someone else had a comment to this effect, but I also find it telling that you don't include many EP language features here. Logic testing...
Is that because you're happy with the language features you have, or because you write so much in C#, or some other reason?
--H
Can you expound on what you mean by "logic testing"?
So far, the Coral8 CCL language has been satisfactory for what we need.
I just trying to describe some of the functionality OUTSIDE the CEP engine that would make a compelling ecosystem for us.
What do you mean by "Entitlements on output events"?
"don't make us code in Q"
please can you elaborate?
Sorry, didn't get that you were talking only about ecosystem type stuff.
Logic testing: Unit testing, for starters. Maybe this is more important in my mind than in reality. But when a project gets to thousands of lines of code without a single unit test, I worry. Not to say that every window-group-by event summary needs a unit test. But when you get into producing complex events, you find edge cases and you write lots of logic to match events together. And that's where I would want unit testing.
I coded up a very simple scheme where you have a file of event inputs and a file of what you are expecting. Then a script runs each file through the engine and compares the results to your expected results. Not the best, but a start. SB included a feature like this into the latest version of their IDE. And testing could get so much more sophisticated.
Thinking about how big a CEP application could get... Let's say you are looking to build a really big and important new system. Now you could write it in C# and in this case, you would set up not only unit testing at the class level, but also at the component level. I mean the level of automated testing these days is extremely high between testing frameworks, mock objects, dependency injection and scripted functional tests.
Over the life of a big project, the automated testing eventually pays off in a huge way. Now you can't currently get this level of testing with a CEP product. So which wins, the faster development time (cost savings) with CEP or the long term cost savings with intensive automated testing?
Hans, take a look at http://sourceforge.net/projects/pysys
This is an auto-test framework written based on experience with testing non-trivial Apama-based apps. We provide Apama-specific plugins to script our engine, adapter framework etc to provide auto-tests for correctness, robustness and performance - though the framework itself is generic (hence open-sourced).
We use this framework for creating tests for our client projects, and also provide the tests to clients along with the Apama plugins so they can run the tests and modify/develop their own as their code-base evolves.
Richard, this is great and something to expand upon.
Hopefully you will also add features for testing that is deeper than the functional/system level.
Hans, yes, we've got some stuff going on with a partner right now extending pysys with direct support for unit level testing - will make that visible when it's warmed through.
Marc, you missed the most important element in CEP - Detection. This motivated me to write:
Repeat the Term CEP Eleven Times and Click Your Heels
http://www.thecepblog.com/2009/02/06/repeat-the-term-cep-11-times-and-click-your-heals/
Basically, you are seemingly redefining any application that works with streaming market data as CEP.
This is not correct. CEP is about detecting complex events and detection theory is about false positives, false negatives, true positives, true negative, type I errors, type II errors and similar detection principles.
When you start discussing these topics, you will be a CEP blogger. :-)
Now, you are just a OMS, SOR, algo trading type of guy :-)
Yours sincerely, Tim
Tim,
To tell you the honest truth, I never set out to be a CEP blogger and I don't care if I am perceived as one or not.
I blog on what is important to me and my world. I don't get caught up in the arguments around semantics that most of the CEP bloggers engage in.
Call me a CEP blogger, call me an ESP blogger, call me an SOR blogger ... I really don't care. I blog on what I think is important to capital markets companies and to the people who do technology in these companies. I am all about delivering and practicality.
-m
>> don't make us code in Q
IMHO, Q is a wonderful language: concise, expressive and all-around excellent. Of course, the KDB environment would benefit from having more/better/cheaper tools or IDEs. I think what Marc is asking is better host language bindings and type mappings. Hiding the language control structures is not going to work, as demonstrated by Hibernate.
@marc,
Then please be technically correct, follow the wisdom at STAC and not contribute to misinformation. You should have named this post:
"What is Important to us in a EPP System"
FYI: Just because you are into bring "practical and delivery" does not justify miscategorizing EPP applications as CEP, repeating the inaccurate marketing hype :-)
Post a Comment