Here are two comments that I received yesterday, and my answers:
1) You bought an CEP Engine that doesn't support event clouds?
We feel that Coral8 does support event clouds, but we are looking for the best pattern to implement it. Mark, who is the CTO of Coral8, doesn't quite agree with the term "event cloud". His posting here highlights his argument. According to Mark, an "event cloud" can be represented as multiple event streams, something that Coral8 supports.
2) How and why did you abstract the CEP engine in your system?
First, the why. We want to insulate ourselves from any uncertainties concern the CEP engine, both in terms of the product itself and of the company. In this economic environment, we are concerned that some of these smallish CEP companies might be strained. Ones who are backed by Venture Capital might find their VC's getting worried and thinking that we are reliving those inglorious times from 2002 to 2003. Ones who are privately financed might find that the backers want to move into other areas. It is no secret that most firms are cutting back or delaying their software purchases, and the ones who get impacted first are the smaller niche companies.
We also want to have some flexibility in case the CEP engine itself does not function as advertised. Coral8 has given us great support, but we have not stressed it yet. We know other companies who have evaluated Coral8 who have foudn some shortcomings, things that the Coral8 staff have addressed. However, it is perfectly within the realm of possibility that we may need to consider another CEP engine should Coral8 fall on its face.
Now, the how ....
We are not using Coral8's native input and output adapters. We are not even reading databases using Coral8's PollFromDatabase and ReadFromDatabase adapters. We have an input server that is used to read static and real-time data and marshall that data into coral8 tuples. On the other side, we have an output server that takes the derived event tuples from Coral8, marshalls them into a common format, and does various kinds of alerting and visualizations.
From the days that we did evaluations of other CEP vendors, we have layers in our input and output servers that deal with Aleri and Streambase. In other words, we have our own adapters! Changing from Coral8 to Streambase or Aleri involves a simple edit to our Spring-like configuration files.
In addition, we have the ability to farm out work to other engines, such as KDB+. We can then read the derived events that are generated by other systems (as long as they are in our common format) and put them into the CEP engine's "event cloud".
In our architecture, we have introduced extra hops in order to abstract the CEP engine. But, we are not that concerned, since we are dealing with analysis and alerting rather than trading.
©2008 Marc Adler - All Rights Reserved
Subscribe to:
Post Comments (Atom)
13 comments:
It's really nice to hear to you too are building your own adapters. We hear this a lot and this is good news as we have decided that adapters are someone elses concern and we should not put too much effort into them.
The application integration world have already a solution for this. There's just no point, from my viewpoint, to duplicate all the efforts in EAI, business integration, ETL or other related areas.
We just provide a few in and out ports into ruleCore and assume that there is a nice integration infrastructure that we can use. And we don't build this ports ourselves either (Mule ESB).
The biggest problem with adapters (having run a business integration company in the past) is that you can't build enough of them. If you have 100 different adapters in your initial launch there's some kind of annoying law that says that your first customer will need an adapter not on that list.
So we just rely on others to build adapters. We are not in the adapter-business anyhow. One hour spent in adding CEP functionality is better than one hour coding adapters.
I'm sure that most CEP vendor do not agree me on this one. So it's nice to know that you as a representative of the customer segment in the CEP world seems to think this is a good idea.
Marc - I couldn't agree more on this cloud vs stream thing. I thought Mark T's post back in October summed it up quite well. To reinforce Mark's assertion that a cloud is an abstract concept - if you think about it, events will always arrives as streams. They have to arrive via channels, whether it's via a message bus, a socket, etc - which are effectively streams (after all, current computing technology can't just distill them from the ether). Streams can further serve to "group" events of similar type/form - even when arriving via a common channel. Some insist that streams are always ordered. While it's true there is an implicit arrival order within a stream, there's nothing to say that the CEP engine has to treat the arrival order as significant or meaningful.
Jeff
Hi Marc,
Arrival order, pointed out by Jeff in the comment above, has zero do to with the "cloud v. stream" debate. This argument has been, and continues to be, a complete red herring.
Why?
Because a linearly order set is still a linearly ordered set even when the elements of the set are observed out of sequence. The relationship of the elements in the set are still linear :-)
In addition, Mark's (and Jeff's) argument that a "cloud" can be represented as many "streams" does not pass the math test - both Jeff and Mark's comments are marketing speak, not scientific speak.
There is no scientific theory that POSETs can be represented by multiple TOSETS. TOSETS are subsets of POSETS. David Luckham has made the same comment, repeatedly, BTW.
You have purchased a (nice) product which is good for processing streams of data using forward chaining in defined slideing time windows. It is that simple, really. It is not a NN, a BN, or a backwards chaining inference engine. It cannot chain backwards and look for seemingly unrelated causals relationships.
When you purchase a truck (for exampe), it is a perfectly good truck, not a jet air plane. If all you need is a truck, why foster a truck v. jet plane dedate?
Honestly, you can't possibility believe that one stream-oriented software product can solve all classes of detection-oriented event processing problems do you?
Yours sincerely, Tim
PS: When folks post (blog)misinformation, that does not sync with the established math and science, and then reference it, there are problems.
Tim, you should consult a mathematician on your POSET ideas before you accuse other people of abusing math or science. The fact is that every POSET P can be divided into sequences such that every element of P belongs to one or more sequence. In other words, it is perfectly natural to divide a POSET into TOSETs and it was proven hundreds of years ago that every POSET can be divided as such.
You constantly confuse "being able to process POSETS" with "is the best choice for every class of event processing problem."
Rather than making spurious accusations and condescending and flip remarks, why don't you buy yourself a good abstract algebra book and come back to discuss POSETs when you have read it?
See this
post for some comments on how David Luckham's book actually uses the term POSET and TOSET. This also explains why it doesn't matter whether you can divide a "cloud" into "streams".
Hi Hans,
If you notice, neither David Luckham nor myself reply to your blog posts on this topic.
I have had many discussions with David; including his review of my keynote at DEBS on this topic.
David and I agree on the topic.
You disagree with both of us.
Neither of us post on your blog; for obvious reasons.
HINT: Please review detection theory.
Tim, that's fine. If I'm wrong about something, I would rather have a specific discussion about why. But on this topic, your comments always seem to boil down to how much you agree with Dr Luckham. I'm afraid that this wouldn't be much help to me in understanding my mistake anyway.
My posts are out there for anyone to read and if they are wrong, I look forward to specific comments on why. Until I find some specific reasons to think that I've been wrong, you'll have to pardon me if I am not very impressed by how much you and Dr. Luckham agree.
HINT: Please review Group Think
Hi Hans,
I think David and I, both, have explained and presented, more than should be necessary (this is my opinion) on this topic.
The folks who argue against it (often led by you, Hans), are folks who have little to no experience in complex detection where forward chaining over related events does not work.
You don't have this background, as you have mentioned before, so you think in terms of forward chaining with event that are related and ordered when created.
You demand we explain concepts to you, repeatedly, that you do not understand.
You challenge our education and use similar negatively aggressive debate points.
As soon as I can post on David's site, I will elaborate on this, hopefully, for the last time because I find it unfortunate you and a few others who do not have experience with complex detection theory continue to challenge (and sometimes insult) those who do.
Hans,
I do not agree with David because of "group think".... but it is very kind of you to continue your insulting remarks.
In fact, David and I have been know to diagree on a number of topics, mostly related to blogs and wikis (Web 2.0) topics, LOL.
I agree with David on the scope of CEP because our background and experience is similar complex detection oriented solutions.
We both have our roots (university degrees, experience) in Electrical Engineering, as I recall; and both of us have worked on complex detection problems for the US Department of Defense.
Tim, again rather than respond to the issues you try to attack the credibility of the person posing the challenge. And now you sneak in the insinuation that I'm insulting Dr. Luckham, when all of my comments have been directed at things that you said and never at anything he's said. Is that your idea of good debating?
Is it not rude to accuse others of posting misinformation? Is it not rude to insinuate that I lack the necessary experience to understand this issue? These are the tactics that you use. At least when I debate, I have the good grace to discuss the points that you have made.
I have no idea where you get your ideas on my experience, but you are wrong. In any case, this is not about my experience versus yours. It about ideas versus ideas. Would you like to discuss ideas?
You post about others misusing math and science and in that same post, you base your argument on a totally mistaken mathematical idea. So you got busted for it. All your mysterious "experience" can't cover the fact that you are personally abusing the very math and science that you claim to support. When you go out and attack others as spreading misinformation, you can't be surprised when the flaws in your own argument are exposed.
If you really have a good point to make, you should be able to answer debate with debate. But I've never seen you answer any point of the criticism leveled against your ideas. Instead you rely on either a connection to Dr. Luckham or your supposedly vast amounts of experience. Neither of these will impress me much, given the fact that every time I read your posts on this topic, I can not help but notice the same huge holes appearing again and again.
Hans,
The only huge hole in this debate is your lack of knowledge in detection theory.
You seem to think that your lack of knowledge and experience places a burden on others to explain to you what you do not understand.
Actually, going around saying "you and David do not impress me" is really a great debate point, LOL.
Thanks for being our most vocal CEP cyberstalker and event cloud critic.
Tim, great attempt to talk your way out of answering any criticism. You should have been a politician.
For the record, you do not explain your theories. All you do is repeat the same four ideas over and over and when you get any questions, you call the other guy some names and fail to answer them.
And I have never said anything like "you and David do not impress me." Why would you twist my words like that? If you've got such good points, you should be able to say in one or two sentences why I'm wrong. Rather than constantly twisting my words so that it sounds like I'm attacking Dr Luckham.
The bottom line is that earlier you used some math terms incorrectly and you got busted. Now you don't want to admit that, so you'll say anything to make it seem like there's something that I don't understand, or to distract the conversation by making it seem like I'm attacking Dr. Luckham. Your tactics are transparent.
Post a Comment