http://blogs.streamsql.org/streamsql/2006/08/handling_revisi.html
Mitch Cherniak poses a question that I posed a few weeks ago about "compensating events".
Mitch seems to be involved in that same project that produced Streambase. I am not sure if Cherniak is involved with Streambase in any way, but it would be interesting in the Streambase folks have thought about this same topic.
©2007 Marc Adler - All Rights Reserved
Saturday, September 29, 2007
Wednesday, September 26, 2007
Tuesday, September 25, 2007
Francis and Karen
My colleague Francis and his young wife are involved now in the battle of a lifetime. It makes the daily bullcrap that we put up with at work seem very trivial, and makes decisions like "should we use DDS or JMS" seem inconsequential.
My prayers and thoughts are with them daily.
Francis has written a WROX book that is available from Amazon.
©2007 Marc Adler - All Rights Reserved
My prayers and thoughts are with them daily.
Francis has written a WROX book that is available from Amazon.
©2007 Marc Adler - All Rights Reserved
DDS vs JMS
Data Distribution Service (DDS) is an alternative to JMS. This spec was designed and is supported by OMG (Object Management Group). The concepts are fairly familiar to anyone who has done JMS.
RTI implements DDS in their message bus.
My feeling is that getting a large enterprise to use DDS is akin to rewiring a battleship. However, it seems like DDS has some possible performance improvements over JMS.
Anybody out there using DDS?
©2007 Marc Adler - All Rights Reserved
RTI implements DDS in their message bus.
My feeling is that getting a large enterprise to use DDS is akin to rewiring a battleship. However, it seems like DDS has some possible performance improvements over JMS.
Anybody out there using DDS?
©2007 Marc Adler - All Rights Reserved
Saturday, September 22, 2007
Nancy
I will away the week of October 7th. I will be travelling to Paris and then to Nancy (France) for the annual Nancy Jazz Pulsations festival. The purpose of my trip is to see a 2-night concert by my favorite band, Magma. The famous French bassist, Jannick Top, will be performing with Magma, and for me, this recreation of their famous 1974 line-up is reason enough for me to dip into my AMEX points....
©2007 Marc Adler - All Rights Reserved
©2007 Marc Adler - All Rights Reserved
Coral8 vs Streambase
http://www.dbms2.com/2007/08/10/coral8-versus-streambase/
Interesting reply by Bill of Streambase .....
However, I think that the only way that this will be solved publically is by STAC and their independent benchmarks.
©2007 Marc Adler - All Rights Reserved
Interesting reply by Bill of Streambase .....
However, I think that the only way that this will be solved publically is by STAC and their independent benchmarks.
©2007 Marc Adler - All Rights Reserved
CEP Conference Report
I thought that the CEP conference would be a sleepy little symposium, but I was wrong.
Gartner ran their conference on Business Process Monitoring on the Monday-Wednesday morning, and it dovetailed exactly into the CEP conference. There was also an academic conference on CEP that ran concurrently with the BPM conference, so you had a lot of academics and curiosity-seekers at the CEP conference.
I would estimate that there were about 200 attendees at the CEP conference, but probably half were from the vendor community. I counted about 50-60 attendees in the financial services track at the conference, and again, about 40-50% were vendors. So, there were probably about 30 folks from the financial services industry who were really curious about CEP.
I give these numbers to illustrate that, in financial services, CEP is still in the curiousity stage. I am happy to say that most of the people there were just as confused as we were (ie: what is the difference between a CEP engine and a rules engine, what is the best messaging middleware to use, do we really need a visual designer, etc). Most of the people who were using CEP were using it in a simple stream case ... pricing, algos, fraud detection, etc. I did not see any use cases that approach the magnitude of what we have to do, but then again, I don't expect organizations like Goldman Sachs to advertise what they are doing.
It was nice to see some of the vendors trotting out their successes in financial services (Aleri with HSBC, CommerzBank and Barcap, Streambase with BofA, Coral8 with Wombat), and start to establish some meaningful partnerships. Nice to see that Coral8 is working closely with RTI.
Mary Knox of Gartner did a nice presentation on what it takes to get CEP adopted by a large financial organization, but she compensated for this nice presentation by monopolizing the Q&A session with Robert Almgren of BofA. I was a bit puzzled by IBM's message about what parts of their CEP framework were already available in Message Broker, who was supporting it in IBM, who we get services from, etc.
I was impressed with Mark from Coral8, who seems to bring that Russian mad scientist mentality to his company. Also encouraging to see that Terry Cunningham is at the helm at Coral8, fresh from his successes at Crystal and other companies. Terry knows what it takes to make a successful software company. They also seem to be on the top of their game with financial services, without restricting themselves solely to algo trading.
Aleri seems like they have some raw power in their platform, and over time, they will be polishing their message more. They win the award for the most genial vendors of the conference.
The key takeaways were:
1) The CEP industry is still at its infancy.
2) It's a fairly hot buzzword right now.
3) The complete stack is important if you do large deployments. We need to understand all of the pieces surrounding the CEP engine.
4) Message volumes are increasing at an exponential rate in financial services.
5) It will be a long battle to break all of the silos in our company.
©2007 Marc Adler - All Rights Reserved
Gartner ran their conference on Business Process Monitoring on the Monday-Wednesday morning, and it dovetailed exactly into the CEP conference. There was also an academic conference on CEP that ran concurrently with the BPM conference, so you had a lot of academics and curiosity-seekers at the CEP conference.
I would estimate that there were about 200 attendees at the CEP conference, but probably half were from the vendor community. I counted about 50-60 attendees in the financial services track at the conference, and again, about 40-50% were vendors. So, there were probably about 30 folks from the financial services industry who were really curious about CEP.
I give these numbers to illustrate that, in financial services, CEP is still in the curiousity stage. I am happy to say that most of the people there were just as confused as we were (ie: what is the difference between a CEP engine and a rules engine, what is the best messaging middleware to use, do we really need a visual designer, etc). Most of the people who were using CEP were using it in a simple stream case ... pricing, algos, fraud detection, etc. I did not see any use cases that approach the magnitude of what we have to do, but then again, I don't expect organizations like Goldman Sachs to advertise what they are doing.
It was nice to see some of the vendors trotting out their successes in financial services (Aleri with HSBC, CommerzBank and Barcap, Streambase with BofA, Coral8 with Wombat), and start to establish some meaningful partnerships. Nice to see that Coral8 is working closely with RTI.
Mary Knox of Gartner did a nice presentation on what it takes to get CEP adopted by a large financial organization, but she compensated for this nice presentation by monopolizing the Q&A session with Robert Almgren of BofA. I was a bit puzzled by IBM's message about what parts of their CEP framework were already available in Message Broker, who was supporting it in IBM, who we get services from, etc.
I was impressed with Mark from Coral8, who seems to bring that Russian mad scientist mentality to his company. Also encouraging to see that Terry Cunningham is at the helm at Coral8, fresh from his successes at Crystal and other companies. Terry knows what it takes to make a successful software company. They also seem to be on the top of their game with financial services, without restricting themselves solely to algo trading.
Aleri seems like they have some raw power in their platform, and over time, they will be polishing their message more. They win the award for the most genial vendors of the conference.
The key takeaways were:
1) The CEP industry is still at its infancy.
2) It's a fairly hot buzzword right now.
3) The complete stack is important if you do large deployments. We need to understand all of the pieces surrounding the CEP engine.
4) Message volumes are increasing at an exponential rate in financial services.
5) It will be a long battle to break all of the silos in our company.
©2007 Marc Adler - All Rights Reserved
Tuesday, September 18, 2007
CEP Vendors, DBMS's and Defense Industry
I made a comment in my last posting about RTI, a small messaging and CEP vendor. It seems that RTI got its start in the defense industry. Coincidentally, IBM is allowed to mention the fact that its System S arose out of a 4-year defense contract.
CEP vendors who grew out of the DBMS sector might have certain advantages, but ones who grew out of defense have other advantages. First, you would expect that anything that had to pass Military testing and certification would be industrial strength. These systems would have to be able to handle a huge throughput of signals and would have to be able to react correctly to the input .. after all, you don't want our ships firing a missle at a seagull. Second, the analysis tools have to be top-rate ... detecting a signal and firing a missle is much like pulling the trigger on a trade. Both have enormous consequences, and once out there, cannot be retracted.
I am not completely positive, but I can imagine that messaging systems in the defense industry might not have to worry about guaranteed delivery. It seems to be OK to drop a few signals (events messages) that the radars send out, and eventually (a few milli-seconds later), the signals would resume.
©2007 Marc Adler - All Rights Reserved
CEP vendors who grew out of the DBMS sector might have certain advantages, but ones who grew out of defense have other advantages. First, you would expect that anything that had to pass Military testing and certification would be industrial strength. These systems would have to be able to handle a huge throughput of signals and would have to be able to react correctly to the input .. after all, you don't want our ships firing a missle at a seagull. Second, the analysis tools have to be top-rate ... detecting a signal and firing a missle is much like pulling the trigger on a trade. Both have enormous consequences, and once out there, cannot be retracted.
I am not completely positive, but I can imagine that messaging systems in the defense industry might not have to worry about guaranteed delivery. It seems to be OK to drop a few signals (events messages) that the radars send out, and eventually (a few milli-seconds later), the signals would resume.
©2007 Marc Adler - All Rights Reserved
Thoughts about CEP Patterns
One of the things that I have been wondering about:
How do you do transactions in the context of event processing? For example, if I get an event, we might trigger some units-of-work and generate some more derived events from this initial event. These derived events might be put into the event cloud.
If one of the units-of-work fail, how do we rollback the "transaction"? Do we need to define compensating events? Do we need to suck the derived events out of the event cloud? Do we need to attach "state" to the derived events which signal whether they are part of a transaction?
We run into a lot of the same issues here that DBMS vendors have run into. How can the transactions patterns in the DBMS world be applied to CEP?
Another thing I have been interested in is the generation of recursive events. If I get an event and generate a derived event, how do I know that this derived event will not result in recursion? Again, DBMS vendors have run into the same situation with database triggers, and some lessons can be learned.
So, does this mean that we are better off considering a CEP vendor who has roots in the DBMS world?
(As you can see, I am just using the blog as scatchpad for some thoughts .... but it gives you an idea about the things that any of you who are implementing CEP might have to consider...)
©2007 Marc Adler - All Rights Reserved
How do you do transactions in the context of event processing? For example, if I get an event, we might trigger some units-of-work and generate some more derived events from this initial event. These derived events might be put into the event cloud.
If one of the units-of-work fail, how do we rollback the "transaction"? Do we need to define compensating events? Do we need to suck the derived events out of the event cloud? Do we need to attach "state" to the derived events which signal whether they are part of a transaction?
We run into a lot of the same issues here that DBMS vendors have run into. How can the transactions patterns in the DBMS world be applied to CEP?
Another thing I have been interested in is the generation of recursive events. If I get an event and generate a derived event, how do I know that this derived event will not result in recursion? Again, DBMS vendors have run into the same situation with database triggers, and some lessons can be learned.
So, does this mean that we are better off considering a CEP vendor who has roots in the DBMS world?
(As you can see, I am just using the blog as scatchpad for some thoughts .... but it gives you an idea about the things that any of you who are implementing CEP might have to consider...)
©2007 Marc Adler - All Rights Reserved
HPC Conference Thoughts
Yesterday's HPC On Wall Street conference could be summed up in one word .... PACKED! I thought that this would be a sleepy little conference, with a few crazy architects from a few IBs and hedge funds walking around. When I got there, I could not even move through the (very narrow) aisles that the vendors were set up in. I would guess that there must have been 700-800 people at this conference, and it is at these times that I am thankful that I am not a lightweight, as I merrily tossed my compatriots all over the Roosevelt Hotel in my quest to get to the conference rooms.
The two panels that I went to were, unfortunately, complete wastes of time. Even some of the panelists that I spoke to afterwards were half-embarassed to be on the folically-challenged panels. (I am still wondering what Gideon Low from Gemstone was doing up there in the session on Multicores and Market Data .. although he DID manage to come up with better answers than the Chief Architect of Fidelity!). It only reinforces my opinion that if you are doing something unique and important, your company will never let you speak about it on a panel!
And, thanks to the people from Bear Stearns, who during our lunch with a vendor, planted themselves at our lunch table and managed to kill our conversations.
I was heartened to see a constant flow of traffic at the Digipede booth. In several square meters, you could talk to Microsoft (CCS), Digipede, and Platform (who were displaying at the IBM booth). I was disappointed to see one CEP vendor there (Aleri).
The most interesting new player was STAC, who purports to be an "independent" kind of lab, where both vendors and customers can go to in order to get evaluations and benchmarking done .... sort of like a Consumer Reports magazine. One interesting comment from them was that, in their early experience, some of the claims about feature sets that various vendors make do not actually hold up when it came time to implement the vendors' products. An independent lab for the securities industry is long overdue, and it is heartening to see vendors contribute hardware to STAC in order to "bootstrap" their operations. The two main guys from STAC are ex-Reuters guys, and so you know that they have good experience in market data systems.
There were some real small players (like RTI in the messaging space) who might be ripe for joint ventures or investment. Seeing these guys work the floor reminded me of my days as a software vendors going exhibiting at shows ... an absolutely exhausting experience.
My colleagues have told me that the HPC show has grown exponentially over the past four years .... if I grows any more, then the Roosevelt Hotel is going to have to abandon the cattle car approach, or else, I am going to have to put on a few more pounds!
©2007 Marc Adler - All Rights Reserved
The two panels that I went to were, unfortunately, complete wastes of time. Even some of the panelists that I spoke to afterwards were half-embarassed to be on the folically-challenged panels. (I am still wondering what Gideon Low from Gemstone was doing up there in the session on Multicores and Market Data .. although he DID manage to come up with better answers than the Chief Architect of Fidelity!). It only reinforces my opinion that if you are doing something unique and important, your company will never let you speak about it on a panel!
And, thanks to the people from Bear Stearns, who during our lunch with a vendor, planted themselves at our lunch table and managed to kill our conversations.
I was heartened to see a constant flow of traffic at the Digipede booth. In several square meters, you could talk to Microsoft (CCS), Digipede, and Platform (who were displaying at the IBM booth). I was disappointed to see one CEP vendor there (Aleri).
The most interesting new player was STAC, who purports to be an "independent" kind of lab, where both vendors and customers can go to in order to get evaluations and benchmarking done .... sort of like a Consumer Reports magazine. One interesting comment from them was that, in their early experience, some of the claims about feature sets that various vendors make do not actually hold up when it came time to implement the vendors' products. An independent lab for the securities industry is long overdue, and it is heartening to see vendors contribute hardware to STAC in order to "bootstrap" their operations. The two main guys from STAC are ex-Reuters guys, and so you know that they have good experience in market data systems.
There were some real small players (like RTI in the messaging space) who might be ripe for joint ventures or investment. Seeing these guys work the floor reminded me of my days as a software vendors going exhibiting at shows ... an absolutely exhausting experience.
My colleagues have told me that the HPC show has grown exponentially over the past four years .... if I grows any more, then the Roosevelt Hotel is going to have to abandon the cattle car approach, or else, I am going to have to put on a few more pounds!
©2007 Marc Adler - All Rights Reserved
Saturday, September 15, 2007
Another New Position (again)
Last week, I was about to finalize my transfer to the Derivatives Analytics desk when an amazing opportunity was offered to me by my company. I cannot blog too much about this new opportunity other to say that it involves Complex Event Processing and transformational new directions for trading and decision making in Equities. This is a huge initiative for my company, is being sponsored by the top people on the business side in Equities, and goes all the way up to the CEO of the company. I will be leading this new project on the technical end, and I am working with one of the movers-and-shakers on the business end.
This is not the standard algorithmic trading scenario that is the standard use case for vendors like Apama, Skyler, etc. We will be pushing the limits of the CEP and messaging vendors. We have the mandate to have all of our Equities systems interact with our. And, if everything goes right, it will transform the way that systems are written within our company.
The nice thing is that the Business is giving us several months of pure R&D time. Reading papers, attending conferences, meeting vendors, etc. For me, it is coming out of a silo and being given the opportunity to learn about every phase of our systems for trading equities. And, it will be building upon all of the things that I have been investigating for the past year in my Architecture role.
©2007 Marc Adler - All Rights Reserved
This is not the standard algorithmic trading scenario that is the standard use case for vendors like Apama, Skyler, etc. We will be pushing the limits of the CEP and messaging vendors. We have the mandate to have all of our Equities systems interact with our. And, if everything goes right, it will transform the way that systems are written within our company.
The nice thing is that the Business is giving us several months of pure R&D time. Reading papers, attending conferences, meeting vendors, etc. For me, it is coming out of a silo and being given the opportunity to learn about every phase of our systems for trading equities. And, it will be building upon all of the things that I have been investigating for the past year in my Architecture role.
©2007 Marc Adler - All Rights Reserved
CEP and HPC Conferences
I will be stopping by the High Performance Computing on Wall Street conference on Monday, Sept 17, and catch up with the new Digipede release.
Wednesday through Friday, I will be at the Gartner Complex Event Processing conference in Orlando.
I am starting to feel like a vendor. However, in my new role, I am basically playing software entrepreneur again, this time in the context of a multi-billion company.
©2007 Marc Adler - All Rights Reserved
Wednesday through Friday, I will be at the Gartner Complex Event Processing conference in Orlando.
I am starting to feel like a vendor. However, in my new role, I am basically playing software entrepreneur again, this time in the context of a multi-billion company.
©2007 Marc Adler - All Rights Reserved
Thursday, September 06, 2007
.NET Winforms Expert Wanted
We would like to hire a true .NET/Winforms/C# expert (must be Winforms, not ASP.NET). A substantial part of the job would be to take over the .NET Client-side framework that Equities is using (ie: the thing I developed). You must be proficient in real-time front-ends, multithreading, understanding of messaging systems like Tibco EMS, etc.
You would be sitting in beautiful Warren, New Jersey, which is about 45 minutes west of New York City. However, you would have substantial opportunities to work with the business, which is located both in New York City and in Jersey City, NJ.
We would strongly prefer a full-time employee, although we might make an exception for an indie consultant.
This is an on-site job. So, please don't send me resumes from the Ukraine, asking if you can work remotely. Also, no calls from recruiters and consulting companies for now. I want to pick from people who are truly interested in financial services and .NET ... and these people probably read this blog already.
Since the position is in the Equities Advanced Architecture group, you must be a really strong communicator. If I ask you to get up to a whiteboard and explain something, I should be able to understand you without straining.
©2007 Marc Adler - All Rights Reserved
You would be sitting in beautiful Warren, New Jersey, which is about 45 minutes west of New York City. However, you would have substantial opportunities to work with the business, which is located both in New York City and in Jersey City, NJ.
We would strongly prefer a full-time employee, although we might make an exception for an indie consultant.
This is an on-site job. So, please don't send me resumes from the Ukraine, asking if you can work remotely. Also, no calls from recruiters and consulting companies for now. I want to pick from people who are truly interested in financial services and .NET ... and these people probably read this blog already.
Since the position is in the Equities Advanced Architecture group, you must be a really strong communicator. If I ask you to get up to a whiteboard and explain something, I should be able to understand you without straining.
©2007 Marc Adler - All Rights Reserved
Subscribe to:
Posts (Atom)
