A Day in the Life of a Chief Architect

Three people won last night's $225M MegaMillions drawing, and none of the three was me. So, I guess that I have to go back to my day job on Monday!

As I mentioned in my last blog posting, I have become so screaming busy that I have not had as much of a chance to blog as I would like. The increased workload has been mainly due to my new title of Chief Architect of Equities, and the fact that I now manage a few other teams besides the Complex Event Processing team. I "back into" this position after the last reorganization at my company. When I first joined my company in August 2006, I was hired to work for the Chief Architect of Equities. He "left" the company in January 2008 (I was already out of his group by then), and another person took over the position, albeit in a less "pompous" capacity. This person only lasted one year, and in January of 2009, I was asked if I would like the position. Unfortunately, I am never one to say no, and here I sit as Chief Architect, waiting for a similar fate to befall me.

In a nutshell, the Chief Architect position is 1/3 paperwork, 1/3 visionary, and 1/3 committees. What I do is most likely done by all of the Chief Architects at all of the various investment banks. There is no secret sauce in what I am about to write. We all have the same kinds of paperwork, we all sit on the same kinds of technical committees, we all meet with the same vendors, and we all read the same magazines and blogs.

The Process

In a company as big as mine, there is an incredible amount of process. Everything has to be approved in triplicate. Every request for a "restricted" piece of software has to be overridden by me. Every request for a server has to be approved by me. I have to eyeball every new purchase, and I have to track which projects have end-of-life components. I also have a dotted line into the CTO of Capital Markets, so Equities has to make sure that they align themselves with whatever roadmap the CTO is working on.

I wake up at 5:30 in the morning, turn on my computer, log in to my work email, and I spend the next half-hour doing approvals. First comes the server approvals, which are easy. Then come the requests for restricted software. Most companies have a Buy/Sell/Hold rating on every piece of software, and we are no different. We have Approved/Restricted/Prohibited. For some reason, the people who rate the software have decided that some essential pieces of software that people need to do their jobs are in the Restricted category, which means that I need to approve their usage. So, everyone who needs this software has to write a rationale for why they need it, which I need to read, and then either approve or ask questions about.

I also need to conduct Architectural Reviews on the major Equities projects. When you take TARP money, your company is under increased scrutiny. You are audited internally by the Audit team, who like to pick projects and start asking tough questions, mostly around security and high-availability. This is supposed to prepare you for audits by the government, who want to make sure that their TARP investments are well safeguarded. As part of the audit process, you have to show them that the projects have undergone extensive architectural reviews. So, this is another effort that I head up.

As part of the review process, I am constantly looking for duplication of effort, and if I find duplication, I try to push for consolidation. Not surprisingly, there are lots and lots of silos in our division. These silos arise because of the competitive nature between groups, because of acquisitions that we make where the acquired company has their own technology stack, and because of people who we hire from other companies who bring their own ideas as to how things should be done. So, part of the Architect's job is to have a broad vision over both the technology stack and the business flow, and push for consolidation.


The Committees

There are a couple of tech councils that I sit on, some which meet every week and some which meet every month. Most of these tech councils are comprised of the senior technologists and the senior management across the Capital Markets division. What we do is try to tackle a certain topic for a few weeks, like Messaging and Caching. We try to see what is being done now in the firm, what needs we will have, and we try to come up with the ideal future state. From time to time, we also bring in certain vendors to give us a level-setting in our knowledge of a certain technology stack. Committees often involve creating written reports and nice Powerpoints, and yours truly has his share of that kind of work to do.

When the committees make a decision, there is usually some degree of backlash. There are groups who have been invested in certain technologies, and those technologies have just been declared as non-strategic. There are groups who need to support users who are still running Windows 2000, and certain technologies just won't work on older machines and ten year old operating systems. And, there are people who just will not listen to any decision that is made by committee (with good reason). So, part of the committees' jobs are to go out and try to sell the decisions to the populace at large. Which involves more meetings and more Powerpoints.


The Visionary Stuff

This is the fun part of the job. We have to drive the company forward with our technology stack. So, I get to read a lot of blogs, read a lot of magazines like Wall Street and Technology, Waters Magazine, Automated Trader, and the A-Team reports, and I get to see what new technology is out there and what other companies are doing. If something looks interesting, I try to invite the vendors in for a talk. Sometimes these talks result in a POC with the vendor, and I have to coordinate the POC and make sure that other interested people in Equities can take advantage of the POCs. We are interested in all of the usual stuff .... Complex Event Processing, Messaging, Caching, Hardware Acceleration, better and faster ways to create models, faster ways to compute prices, ways of monitoring and improving latency and throughput, etc.

I also get asked for a lot of advice from newly-formed companies. There are a number of start-up companies (and very big companies) who are targetting products towards Capital Markets, and from time to time, I receive requests from these companies for a meeting in order to help validate their ideas. Some people might say that we are spending our valuable time giving free consulting to a small company, but if we really like something, and we are the first ones on the block to see their technology, then we might decide to make a principal investment in the company.


So, please bear with the sparseness of my blogging activities while I try to do things that will make my company a bit more competitive in the market .....



©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Things on the Blogging Queue

I have been super busy lately, and have not had much time to blog.

I have a post coming up to talk about the role of Chief Architect and what I do from day to day. Not giving away any secret sauce ... just to explain what the role entails, after I have received a bunch of emails asking me exactly what architects do on Wall Street.

I also want to bitch and moan a bit about Java. I recently had to do some coding of a C# framework, and I needed to deliver a Java-based SDK to some clients. I will dive into this a bit more in a future blog post, but let me tell you that I have a new appreciation for C# 3.0 and Visual Studio.

This article has been among the most widely-read articles on the Wall street & Technology website over the past two weeks.

I am also about to install db4o for the first time and play around with it. Equity-Linked Technology at Merrill (or what's left of it) has made good use of it, so I want to see what they see.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

FXAll and Complex Event Processing

Article on FXAll's use of the Oracle CEP Platform

"The FXall Trading solution relies on CEP technology to compute algorithmic trading strategies that are then sent off to more than 60 liquidity providers worldwide..."

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

IBM System S and Options Trading

Press release here

"IBM unveiled a prototype of an automated options trading system. During the project, scientists at IBM Research collaborated with TD Securities to achieve a 21 times performance improvement on the volume of data consumed by financial trading systems."

Glad to see that System S is finally seeing the light of day.

Opher, any comments?


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

How I helped build the bomb that blew up Wall Street

The subprime mess, this time from a Wall Street programmer's point of view:

http://nymag.com/news/business/55687/

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Yet Another Wall Street Tax

Yet another attempt to kill off the talent on Wall Street .... and this one isn't limited to banks who accepted TARP.

http://www.nytimes.com/2009/03/29/nyregion/29tax.html

New York's temporary 3-year tax kicks in for people earning over $300,000, with the highest rate imposed on people earning $500,000.

I am watching an entire industry being decimated in slow motion .... much like the melting glaciers due to global warming.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Some More Thoughts on Sybase RAP and Coraleri

First, some brief tidbits....

Congrats to Terry Cunningham, the former CEO of Coral8. It looks like he is now a Senior VP over at i365, which is a Seagate company. I guess that Terry will be wearing a dual hat, as he is also the Chairman of Aleri. One wonders if Seagate will eventually go into the CEP business :-)

Also, good luck to Mark Tsimelzon. A few people at Coral8 have remarked that Mark is no longer with the combined entity. With Mark's track record, I am sure that he will be involved with something new and interesting.

It looks like Coraleri has chosen Coral8's CCL to be the SQL-based language of choice for developing Coraleri applications. This is the first piece that is falling into place, and for me, it's good news. Now, we have to wait to for the Splash integration to happen, and then the second piece of the puzzle will be solved.

The latest Coral8 5.6.1 has lived up to its claims of major performance enhancements. My CEP team reports that CPU usage is down significantly in our tests on our most complex queries.

With Coral8 making big strides in performance, the case to keep the Aleri CEP engine around is weakening. Don DeLoach is adamant that none of their customers will need to rewrite a single line of code, and that Aleri has even put it in writing. I know that Don believes this with all of his heart, but I am still a bit skeptical. But I am willing to cut them a big piece of slack, as long it does not affect what we are doing with Coral8. We are continuing on as if nothing happened, and I am trusting in Don and the Coraleri team to make the transition completely transparent to us.


Which brings me to my next topic ... Sybase RAP and Coral8.

For this topic, I am putting on the other hat that I wear, which is Chief Architect of Equities. One of the jobs of the Architect is to keep abreast of the current technologies that we use to see if any of them will approach an "end-of-life" situation. Chief Architects needs to make sure that the company does not invest in any technology that might be risky in any way.

Almost every large investment bank has one of every technology. I am not giving away any of our secret sauce when I say that we have a number of applications that use Sybase ASE for the database. So, when Sybase comes knocking on my door, asking me if I am interested in their new CEP product, what do I think?

First, I think it is a bit curious that Sybase did not publicly mention the use of Coral8 as their CEP engine. I can now understand why. I can speculate that Sybase knew of the pending acquisition of Coral8 by Aleri, and wanted to avoid some of the confusion that goes along with a merger like this. If I knew that a company's product was based on another vendor's product, and that other vendor had just gotten acquired by another company, I would need to be absolutely sure that all of the moving parts were going to have a long life, and that we would not be faced with an end-of-life situation. I could imagine more that a few sweaty brows at Sybase when Coral8 said that they were selling to Aleri. I am sure that Sybase did not want this consternation to leak out to their potential customers. So, better not to mention the association between Coral8 and Sybase RAP. If this was the primary reason for the silence, then I can certainly appreciate that.

But, as the Architect, I would also want to know what Sybase's plans are for the future of their Coral8 engine. Sybase bought the source code to Coral8. Does Sybase plan on getting continuous enhancements from Coral8 as the Coral8 engine and language evolves? Does Sybase plan a separate fork of the source code so that, eventually, the Coral8 engine and language and the Sybase RAP engine and language are two different things? Will Sybase RAP eventually incorporate Aleri's Splash language? These are things that are very important to us. We would need to see Sybase's roadmap as well as Coraleri's roadmap.

In a posting on Marco's blog, Era asks some very important questions:

Are there other details on the deal? Is this a one time source code tree fork? Will Sybase get new versions from Aleri? It would be interesting to hear if Sybase they will develop it itself further or if bugfixes, patches and new features will come from Aleri’s developers?

John Morrell from Coraleri responds with the very mysterious:

Sorry, we are not releasing more details on the terms of the deal. Those are business terms between Sybase and Aleri.

That's a bit scary. I can understand the financial terms not being released (although it would be interesting to see if Coraleri gets a piece of every Sybase RAP sale, as that would help with Coraleri's financials). But not giving transparency into Sybase's plans is not good news for Sybase, although it is a positive for Coraleri. I have no transparency into whether Sybase will be upgrading their engine based on the upcoming work of Coraleri, and that does not sit well with me.

Sybase RAP can be used without the CEP portion. However, I do not know what other parts of RAP are dependent on other third-party vendors, and whether these other parts are subject to any risk. And, it looks like it has some overlap with KDB+ and with Vhayu. (I wonder if there is a third-party comparison between the three.)

I will not be recommending Sybase RAP for consideration until some of these underlying questions are answered. The current situation just leaves too many questions unanswered. I would welcome a bit more transparency from Sybase.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

90% Bonus Tax Article

This is the most lucid assessment of the madness that has engulfed the US Congress and the unintended consequences of the 90% bonus tax:

http://news.hereisthecity.com/news/business_news/8876.cntns



©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Sybase and Coral8

A very curious situation between Sybase RAP and Coral8.....

I ran into Ivy Schmerken the other day. She has been an editor and writer at Wall Street and Technology for over 20 years, and I used to read here articles when I started on Wall Street back in 1986.

She told me that she had just written an article on the new Sybase RAP "CEP" product, and asked me what I thought of it. I said, "I know all about it. They licensed the Coral8 engine, and that's how they are doing the CEP part of things." I thought it was public knowledge on Wall Street that Sybase RAP was using Coral8, but I guess that Sybase didn't think so, as they never disclosed this fact to Ivy in her interview.

Ivy followed up on this fact with the people from Sybase, and then it came out that Sybase was using Coral8 under the hood. Ivy's news story was then picked up by other news sources like Finextra, and all of a sudden, it was out that Sybase was using Coral8.

This prompted an interesting post from John Morrell of Coraleri that was posted on the Coraleri blog.

I am frankly puzzled why Sybase did not disclose the Coral8 partnership up front. Surely they realize that most of Wall Street (at least the people who care about CEP) knew about the partnership. I was told about this by Sybase themselves. Wombat used Coral8 under the hood in their Acumen product, and in fact, it was Wombat's strong endorsement of Coral8 that helped us in our choice of CEP engines.

I would have thought that it would be to Sybase's benefit that they tout the fact that a tried and tested CEP engine was at the heart of their new offering. Can anyone shed some insight on the thought process here?


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Ovum article on Coraleri

http://www.ovum.com/news/euronews.asp?id=7745

(Basically parroting what was said here before)


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

The Streambase Amnesty Program

Every since this morning, I have have gotten a number of emails telling me of a new Streambase press release and asking me what I thought of it. It seems that Streambase is trying to take advantage of the Coraleri merger in their "Streambase Amnesty Program".

And Coraleri may be making a mistake by publicizing Streambase's program.

When I first was alerted to Streambase's program by various people, I thought that it was yet one more instance of Streambase's questionable marketing practices.

But then I thought more and more about it, and now, I am not entire sure if I would categorize it as something totally sleazy.

When Don and Terry told me of their plans to have two engines, the first thing that I told them was that it would result in confusion, and that in order to avoid confusion, NEW customers and prospects might shy away from Coraleri and go to another vendor like Streambase or Apama. They might even want to avoid smaller companies totally, and go with something safe like IBM or Oracle. And, I even stated this fact in my earlier blog posting on the Coraleri merger.

I know that Streambase are big readers of my blog, and they certainly picked up on this fact right away. And, they are using this to take advantage of the period of haziness that surrounds any merger.

I question the use of the word "Amnesty". Amnesty usually implies forgiveness for someone who has committed a transgression. What transgression has a Coral8 or Aleri customer committed except not use Streambase? I'm sorry Streambase, but I don't that anyone needs your forgiveness. But it is very gracious of you to offer your beneficence to the Coraleri customers.

Streambase has detected a weakness here in the marketplace, and is positioning itself to exploit it. Is this sleazy or is this simply an aggressive but honest business practice? Competitive upgrades happen all of the time in the business world. Companies are constantly trying to poach customers from each other by offering various kinds of incentives.

Let's examine the Streambase press release.

(Bile On)

The Streambase press release says:

“StreamBase received several enquiries on Monday from concerned customers."

Really? The combined Coraleri only has about 80 customers. There were actually customers who called you on Monday, the day of the merger? Everyone was that disturbed by the merger?

"StreamBase has designed a Best Practices Workshop for those investigating the migration process."

Wow! An entire workshop was designed and completed in the 2 days since the merger was announced. Pretty quick work there!

“Migration Best Practices: Moving from Aleri SPLASH or Coral8 CCL to StreamBase StreamSQL” is a workshop designed by Richard Tibbetts, StreamBase’s Chief Technology Officer and led by StreamBase’s senior field architects."

All of your senior field architects became experts in CCL and Splash in two days? Pretty impressive!

(Bile Off)


I am sure that Streambase has performed competitive analyses and gap analyses on both Aleri and Coral8 over the years. Most vendors know what their competitors are capable of. So, I am sure, at some level, there are people at Streambase who know what CCL and Splash do. I am sure that there are customers who evaluate Streambase, Aleri and Coral8 side by side, just like we did. A good field engineer like Streambase's Steve Barber probably gets questions from prospects all of the time like "I did this in Coral8. How do I do the same in Streambase?" I would question whether Streambase could handle the load of even 10 Coraleri customers asking for conversions at once. One migration would probably take Steve Barber several weeks to do.

As much as I dislike Streambase's sales and marketing organization, and as much as I personally think that they sometimes play fast and easy with the facts, I think that Streambase is probably doing right by their VC's by taking advantage of some of the uncertainty that surrounds a merger of complimentary products. And, I am sure that most majors companies would do the same.

I would like to think that Don and Terry and John and Jeff had completely thought this out, but until I see a complete roadmap from Coraleri, I will continue to think that there will be some degree of risk that arises from the integration. And, I cannot fault Mark Palmer from looking at this situation and thinking to himself that there's money to be made from this. I might do the same....


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Streambase and Quant Fund Corrections in WS&T

Thanks to Penny Crossman and her editor, Greg MacSweeny, for doing some due diligence and subsequent corrections on their article about PhaseCapital's evaluation process in which they chose Streambase. The link is here.

Here is the important paragraph from the revised article:

Last summer, the quant fund began evaluating major complex event processing venders by attending trade shows, checking general market acceptance and investigating research on CEP. "We don't claim we've seen every CEP vendor," Goodell notes. During the fall, the firm drilled down on CEP vendors by meeting with representatives, checking product references (blind and vendor provided), downloading publicly available products, and reading white papers. "Some products appeared to be more flexible than others, and that was a large part of our evaluation process," Goodell notes. "While we wanted to make sure the product would integrate well with our environment and that it was rich enough to allow us the control we needed to implement our algorithms, we also wanted to make sure it was enough of a high-level platform that we did not need to worry about data management issues."

And, the money shot is in the sentence: "We don't claim we've seen every CEP vendor"

This is quite a difference from the assertion of the Streambase press release in which they say:
"PhaseCapital chose StreamBase after a comprehensive evaluation of the leading CEP products. "

Kudos to Penny and Greg for having the integrity to drill down into the original questionable assertions from PhaseCapital and Streambase in their original article.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

The Birth of CorAleri

It is now March 9^th, 2009 somewhere in the world. And, the atomic bomb news that I have been mysteriously referring to for the past two weeks is ….

ALERI BUYS CORAL8

Not quaking in your boots? Well, this IS earth-shattering news in the small community of Complex Event Processing. But, you guys were good guessers. No, I am not leaving my company (yet). No, I did not take the CTO position at a Complex Event Processing vendor (none has been offered yet). No, I did not get escorted out off of the trading floor by a gaggle of security guards (yet). No, I didn’t run off to play marimba with Magma.

However, it confirms the predictions of many people in the blogosphere that the consolidation amongst CEP vendors is starting to happen. And, I am sure that this consolidation is not over, both in the CEP space and in the infrastructure space. Already, I am hearing of financial troubles among several of the infrastructure players, and it would be really wise of everyone out there to re-engage your vendors in a little bit of financial due diligence. Make sure that their source code is in escrow …. not that you would ever do anything with it.

I had been predicting that Coral8 would be acquired for quite some time. Terry Cunningham had been personally bank-rolling the company since its inception, and unless Terry went short on the market, I am sure that his portfolio has suffered like the rest of ours. I can only speculate whether financial considerations inspired Terry to sell the company, or whether Terry just got the itch to move on to his next adventure (although he did take the role of Chairman of the new company). To be honest, when I heard that a large database company had purchased a source code license for Coral8, I thought that the end was near for our intrepid band of CEP’ers from Mountain View.

My curiosity was aroused when I received a dinner invitation from Terry about two weeks ago. Since I have been burning the midnight oil due to my new role and responsibilities, I politely declined, saying that I was just two exhausted to go out at night and play. However, Terry knew how to appeal to the closet “foodie” in me, and picked a very nice restaurant at Columbus Circle that I have only been able to dream about. Imagine my surprise when I got to the restaurant, and found Terry locked arm-in-arm with my other favorite CEO, Don DeLoach of Aleri. I thought that they had united to jointly serve me with the lawsuit that I deserved for publicly taking their products to task on my blog.

(I am taking a bit of literary license here … I actually found out about the merger the night before from another source, so I was not too surprised to see Don there.)

Why did they single me out for such attention? I guess that having a big mouth and a title pays off sometimes. But as a customer of the newly combined company, they wanted to know how a typical customer would react, and what effect the merger might have on us. Would we continue to buy licenses, or would we go running to a competitor? Or, did they want me to give them love and kisses on my blog?

So, in between lot of glasses of wine and a lot of arm-waving on my part, I expressed a number of concerns. But, also, there is lots of goodness in this merger.

There have always been some issues with Coral8. First and foremost, they had no financial domain expertise. They had attempted to rectify that recently by hiring Colin Clark, but I had my doubts whether that strategy would result in anything concrete. Second, they had no true presence in New York, and if you are going to market to the financial services industry, you have to at least have an office in New York City. Third, they had no European penetration, and I could imagine that Apama, Streambase and Aleri were cleaning Coral8’s clock in EMEA. Fourth, they were one of the only CEP vendors who did not have a vertical product for trading or risk, and they really had no intentions of writing one.

Here was this nice 40-person company, sitting in their nice, sunny offices in Mountain View, 3000 miles away from their main customer base, without a clue of what the capital markets companies really needed. However, Coral8 is an extremely nice company, very friendly to developers, writing lots of nice white papers, and putting out a good product that was less expensive than all of the other products (except Esper).

Then, there was Aleri. Lots of financial domain expertise, with a few products that were geared towards trading firms. Good European presence. Ex-Bell Labs rocket scientists working in a basement office in Mountainside, right across the street from their old Bell Labs haunts. Not a very developer-friendly company. Very hard to configure the product. Documentation that was not as good as Coral8's (although my last experience with their docs was the old 2.4 version). A better IDE than Coral8, but that’s not saying much as Coral8 has a minimalist IDE. Not very good at testing and QA (based on 2.4 experiences, and some recent feedback from some customers in London). Some sort of real-time OLAP strategy. More adapters geared towards financial services companies.

Now, Aleri has bought all of the assets of Coral8. What does the combined company have to offer us? It has a European presence, it has financial expertise, it has a good documentation writer, it has a person who knows .NET, it has people who are concerned about compelling visualizations, it has a procedural language called Splash, and it has two engines and two SQL-ish languages. And it has two pricing models, with Coral8 coming in at $15K per core and Aleri costing about $27K per core.

What did you say? Did you say two engines and two languages? What’s a customer to think?

First, let’s tackle the issues of the engineers. You have two very smart teams located on different sides of the USA. Each team is justifiably proud of the CEP engines that they wrote. One engine came out of a Stanford University research group, and the other engine came out of a commercial effort from Bell Labs. (Correction: the engine is not the same one that they developed at Bell Labs.) Eventually, the combined CorAleri is going to have to choose one language and one engine. So, you will have two sets of development teams positioning themselves as the winner of the language and engine competition.

Every customer of Coral8 MUST program the engine in CCL, which is Coral8’s SQL-like language. However, even though Aleri supports an SQL language, almost nobody uses it, as the Aleri IDE does not support it. (The Aleri IDE likes to generate XML-based code.) Let’s assume that Coral8 wins the language and engine competition. However, in my vision, Aleri gets their licks in when CCL is supplemented with the Splash language.

So, in my ideal world, the language of CorAleri would be CCL with some Splash. And, since the language and the engine are linked tightly, Coral8 keeps the engine as well. However, Aleri would help the Coral8 engine team to make their engine even faster.

At first, CorAleri will have to support two engines and two languages. But, the longer you postpone the single-engine strategy, the more time you give the Coral8 and Aleri engineers to improve their individual engines in order to win the engine competition. Then, it becomes more difficult to have a single-engine strategy.

Don and Terry talked about a single product that includes BOTH engines. The Aleri engine would be the “performance engine” and the Coral8 engine would be the “deterministic engine”. If you check a certain checkbox, “stuff” would be sent to the Coral8 engine and if you check another checkbox, “stuff” would be sent to the Aleri engine.

You cannot have two engines running simultaneously on a single server. They would be competing for resources. We have to worry about administering another process on our servers. We wouldn’t be sure which engine was handling what. It makes monitoring more difficult.

If you give a customer too many choices, then the customer might get confused. Customers want transparency and simplicity. If you confuse a customer, then that customer might change over to a CEP vendor who offers a simpler solution. This means that a confused customer might run over to Apama or Streambase.

CorAleri must come up with a solid roadmap about the merging of the languages and engines, and they must make this roadmap available to customers.

We do not want to take 25% of each sprint to play catch-up on a moving target. The worst thing for us is to be surprised. With each release of the new CorAleri engine and language, we want the migration effort to be painless. We want to change as little of our existing CCL scripts as possible, and we don’t want to find that queries that previously worked do not work anymore.

To me, this is the greatest challenge to CorAleri and the greatest risk to us.

I would like to see the engine and language and the documentation stay with the Mountain View folks. What would the Aleri folks do? They could concentrate on writing more adapters, work on integration with other partners (ie: Netezza, Gemfire, RTI, etc), come up with a better IDE with debugging a monitoring support, come up with more Jack Rusher-type visualizations, real-time OLAP and analytics, and develop some more financial apps. This would give a clear line of separation of duties between the two development teams.

In summation, there is a lot to be excited about, but there is also risk that is associated with confusion.

I wish CorAleri good luck. They have some really good people. They have a sense of integrity. They have a new sense of purpose. And, if they could come up with a unified roadmap that makes sense, then it would be a company to be reckoned with in the CEP marketplace.

A Response from Penny Crossman

Hi Marc,

I just came across this blog today — I guess I should check your blog more often! Thank you for reading my article, here's the self-serving link to it: Quant Fund Uses CEP for Smart Order Routing, Data Feeds

I've placed calls to set up a follow-up interview with Phase Capital and StreamBase to clarify this, but my article was based on an interview with Phase Capital's CEO Eric Pritchett and director of technology Geoff Goodell.

Goodell said, "We did a comprehensive evaluation of different OMS/EMS solution providers in the CEP space. Our goal was first to discern if a CEP solution was appropriate for our firm, then also determine which of these CEP solution providers were most well suited to our development paradigm and the kinds of analysis we’d need to be able to do to execute our strategies in the market. We looked at a number of providers, including Progress Apama, Coral8, Aleri and StreamBase, and we drilled down on these and tried to figure out what aspects of these different solutions were interesting to us. We’re a very heavily technology focused firm, not just for performance reasons but also because the algorithm, the process is our strategy, this is a fully automated process at its core, the code we developed to build our process is the product itself, and as such we need tools that enable us to build our product that will allow us to focus on the business problems without having to worry so much about the implementation details of the internals of databases. At the same time, we’re concerned to make sure we have tools we need.

Some of these CEP solutions seem like completely packaged solutions that strive to deliver complete trading algos against the market that will achieve low market impact etc., we’re not looking for that sort of thing because we are in the business of designing algorithms that get the best execution. So those are right out of the box. Streambase’s support paradigm we considered appropriate because Streambase is suited to working with developers, people who are actually able to build systems and algos that will achieve the kinds of detailed and specific execution properties we want. They have the infrastructure in place to work with us to make sure that what we’re trying to achieve is feasible given the internals of their database. This is one of the important characteristics that made Streambase win for us. The extractions were clean, it didn’t seem like they built their system around satisfying the needs of a small number of clients, the Streambase system is a platform for developing solutions, it’s not a solution itself. The idea of using a platform was extremely appropriate for our needs, because our goal is not to buy a packaged solution, our goal is to develop something using well-designed, modular tools and Streambase’s well designed, modular tools were appropriate."

Eric Pritchett said, "The process we went through was we selected a universe of potentials, we collected information about them the best we could with phone calls, researching websites, we went to conferences in New York, we did a little narrowing at that stage, we had detailed demos and interaction with the teams from the companies Jeff mentioned, we then narrowed that down to our final two: Streambase and Aleri. We had extensive, on-site working sessions with both of those teams and we did a thorough evaluation of the support paradigm and got under the covers with developers from both companies, that took about two months."

I have no reason to believe Eric or Geoff were lying, nor do I have a way of checking whether they downloaded other CEP vendors' software. They didn't actually claim that they downloaded every CEP solution in their evaluation. The CEP vendors who are attacking this story would be better served by getting Wall Street customer wins and getting those customers to speak on the record so that I can write case studies about them.

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

It Has Begun !!!!!!

I'm the First !

:-)

Watch the news over the next few days and you will see what I mean.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Job Openings with my Group

I have one full-time opening in my group, and possibly two consulting positions. The full-time job will be located on our Equities trading floor in the TriBeCa section of New York City. The consulting positions will be either in NY or in Jersey City.

For the full-time position, I am looking for a great .NET developer. The ideal candidate should be familiar with Team Foundation Server and issues around build and deployment. The candidate should also be familiar with WPF and WCF; we would ideally welcome someone who is proficient with one of the two and would be willing to become expert in the other.

For the consulting positions, I am looking for great .NET developers. WPF and WCF are highly desirable, as there will be definite UI work.

The consulting positions are on-site. Do not email me with requests to work remotely. They will be routed directly to /dev/null.

Because of the current fiscal environment, I cannot use external headhunters. Sorry.

You can send email to me at magmasystems at yahoo dot com.



©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Magma in Paris, February 12, 13, 14 2009 - A Recap

-->

I arrived in Paris at 11:00 AM on Wednesday, February 11 2009 for 3 concerts given by my favorite group of all time, Magma. The concerts were in celebration of their 40^th anniversary as a group (even though they had their 35^th anniversary concerts in 2005). The concerts were to take place at Le Casino de Paris, a theater in the 9^th Arrondissement, which is a section of Paris more known for its sex shops than anything else.

In addition to Magma, the opening act for all three nights was Jannick Top’s Infernal Machina. Top was the bassist for Magma during their glory years of 1973 and 1974 (and also, 1976), and is a personal hero of mine. His new album, also called Infernal Machina, is a throwback to the glory days of Magma, and to tell you the truth, I wanted to see Jannick just as much as I wanted to see Magma.

After taking the RER B train from the airport to Gare de Nord, and taking the number 2 train to the Pigalle stop, I managed to find my hotel. I used my time-honored technique of sleeping for a few hours immediately after arriving in Europe so that I would have energy for the night’s festivities. I picked a 3-star hotel that I found on Expedia that was only about a ten minute walk from Le Casino. Typical 3-star Paris hotel … clean, sparse (not even a clock in the room), and cable TV.

That night, there was a pre-festivities dinner at a Southwestern French restaurant that was organized by Yann Richard (yannzeuhlwortz). Besides Yann and me, in attendance that night were photographer Fabrice Journo (Riah), Fanny Layani (A-Fiieh) and Muriel Huster (a photographer who is not on the Magma forum). Later, we were joined by Marcello and Tathiane, two Brazilian Magma fans who live in Los Angeles, and whom I met in 2007 in Nancy, France.

Thankfully, between the courses of escargot and Magret de Canard, everyone mostly spoke English to me, and thus spared me the pain of inflicting my self-taught French upon the fair people of Paris.

Yann had a friend who had seen Magma two weeks before in Toulouse. Yann relayed to me a mini-review that his friend had given. The review consisted of the single sentence “Magma is Dead, Long Live Magma.” This review turned out to be very prophetic.

Then I was told that Magma was supposed to play the same exact set every night, and they had no special guests scheduled. Fuck!!!! I thought that, for their 40^th anniversary, they would trot out some of their masterpieces. But, this was going to be 3 nights of new or unrecorded compositions.

Everyone had planned to meet up at the brasserie across the street from Le Casino at 5:00 PM the next day. I spent most of Thursday sleeping, as I went to sleep around 4 in the morning. At 5:00, a small contingent of Magma fans had started to gather at the brasserie. I met my old pal Robert Guillerault from Bordeaux, who runs the Magma Web Pressbook. Robert speaks no English and I cannot understand his spoken French, so we spent a half hour gesturing wildly in the air. We were soon joined by some of the other Zeuhl Kombattahnts that I only knew from the French Magma forums. Fabrice, Yann, Fanny, and Muriel soon arrived. Marino (hpl) came from Venice, and brought me some DVDs of some live shows. Maxence (Zwenskaia) from Montpelier came by, as well as photographer-supreme Madi, Francis Lecointe (who has graciously bought the tickets for me), old friend Fabrice Bellearbe (Barfix), ace photographer Marco Tchamp, Roman (Peaudhorse), and more. Magma themselves came by for their pre-concert drinks. Robert introduced me to Magma bassist Philippe Bussonet, who seemed thoroughly unimpressed that I came from New York to see them. I also got to shake hands with Stella Vander and Herve Aknin. I usually try to avoid contact with bands that I like; I am a firm believer in the old adage “Never meet your heroes because you will inevitably be disappointed.”

At 7:30, we all staggered across the street to the venue. Inside, I met old friend Gaeten Villard (Zess), and I finally met British magma fanatics Steve Ashworth and Steve Davis. Steve Davis has one of the few English websites devoted to Magma. Steve is well-known in England for his exploits in the world of Snookers, which is much like the American game of Pool. In fact, he was 6 or 7-time world snookers champion, and now, devotes a substantial amount of his time playing professional poker. Steve also brought the Magma offshoot group, Offering, to the UK in the mid 1980’s. Steve and I have been corresponding at infrequent intervals over the years, so it was great to put a face to the name. Both Steves had brought their girlfriends to the Magma show, which I thought was a very dicey proposition.

The seats in Le Casino were uncomfortable, to say the least. The amount of space between the rows of seats was designed for little Frenchmen, not for big Americans.

At about 8:15, the first group, Jannick Top’s Infernal Machina, came on. The group consisted of two drummers (one playing a large Japanese Koto drum), bass, guitar, electric piano, and two vocalists. One of the vocalists was Klaus Blasquiz, who was Magma’s vocalist during the glory years. The other vocalist was a tall, blond woman who was the subject of much conversation over the next three days.

After a soft opening section of cello and percussion, Top’s band ripped into the full force of the composition. Most of the hour-long piece consisted of the group jamming out on various old Magma themes, including parts of MDK, Kohntarkosz (the ending section), and De Futura. I loved it.

There were various issues with the sound system on the first night. From where we were sitting in the third row of the mezzanine, the sound was too muddy. Luckily, over the next two nights, the sound improved a great deal.

In a portent of things to come, Magma also had technical issues at the start. They opened the concert with a brand new piece, only to have Bussonet’s amplifier shut off after the first few chords. Vander signaled for the band to stop, and Bussonet took a few minutes to rectify the situation.

The first piece is unnamed as of now. To my ears, it sounds as if Christian Vander has been listening to a lot of Steve Reich. Two main themes play off of each other in this composition. First, there is a Reich-like ostinato pattern that is played by the vibes, guitar, and electric piano. This is overlaid by a more Magma-like vocal line that is sung by Herve Aknin, the new vocalist who recently replaced Antoine Paganotti. As these two themes move in and out of phase, the drums and bass punctuate the music with a series of intermittent crashes. This goes on for about ten minutes, building up to a climax.

The second piece was a real snoozer for me. It is a 25 minute piece named Felicite Thosz, and is reminiscent of the vocal-intensive Magma offshoot called Offering. To my ears, it sounded like a combination of Magma playing Christmas songs with Magma doing a mouthwash commercial. There were lots of vocal solos and harmonies and sleigh bells … and for me, a reason to catch up on my sleep. A bit of promise in the last section with a Russian-dance theme, ala Wurdah Itah, but no real punch. Polite applause from the audience at the end.

The third piece was Emehnteht-Re, which is third piece of the Magma trilogy that includes Kohntarkosz and Theusz Hamtaahk. It is an hour-long piece that unites several of Magma’s previously-recorded themes. I first saw Magma perform this piece in Brooklyn at Club Europa in June of 2007. Since then, Vander has reworked the piece in various minor ways. The piece now seems more coherent to me. The piece starts out with Rinde from the Attahk album, transitioning into Hhai, transitioning into Zombies, and ending with a dirge that represents the funeral of an Egyptian king.

The Zombies section is supposed to be the atom bomb in the piece. About 10 minutes of unadulterated Zeuhl madness, and is supposed to be the centerpiece for Vander’s remarkable drumming. For the first night, Vander seemed rather subdued. I am not sure if it is his age catching up to him or the tendonitis that has been bothering him for the past few years.

E-R was followed by the cacophonous De Futura. This is a Jannick Top composition that appeared on the Magma album, Udu Wudu, in 1976. When it was recorded, it was a sparse but rhythmically driven piece that only included bass, drums, occasional synthesizer, and growling vocals. However, for the 3 concerts, it was played by the total combined forces of Magma and Infernal Machina. 15 people on the stage, including 3 drummers, two guitars, two basses, two pianists, and 5 vocalists. All I can say is “What a mess!” Remember the blond vocalist from Infernal Machina? The running joke during the 3 nights of the show was “What will she be wearing during De Futura?” Each night, she seemed to come out with less and less clothes on, and during the afternoon, she must have been shopping for her next costume at the shops next to the Moulon Rouge. The performance of De Futura was extremely sloppy, as confirmed by a lot of the opinions I heard the next day at lunch.

As an encore, Magma finished with Ballade, which is Vander’s moving tribute to his idol, John Coltrane. A beautiful piece with lots of rich harmonies, and featuring Vander doing his famous scat-singing in Kobaian.

After the concert, a lot of the Magma fans gravitated to another brasserie that was a few blocks up from the concert hall. This was to be a nightly occurrence that kept me out until 2:00 AM. I spent most of the time with the only other native English speakers, Steve Davis and his girlfriend, Jeannie, talking about Magma, snookers, and poker. You can actually watch videos on YouTube of Steve playing snookers at the various stages of his career. Steve was also able to satisfy some of my curiosity about the Women’s Pro Pool circuit in the US.

The next day, we had a wonderful Zeuhl lunch at Le Triton, a jazz club in Paris that Magma has played in many times. The talk around the table was about the collective disappointment of the previous night’s concert. There was consensus about the lack of “punch” in the songs and about the poor sound quality. If you ever find yourself in Paris to see a show at Le Triton, I hardily recommend eating there before the show … the food does not disappoint.

After a brief siesta, it was back to the brasserie for some beers, and onto the show. Friday night’s concert was noticeably better. The word must have gotten to the bands about the poor performance on Thursday, and there were great improvements in the sound and in the quality of playing. It was the same set list, but it seemed that Infernal Machina really jammed out on the De Futura-sounding section, and Magma did the same on the Zombies section of E-R. And, the blond vocalist did not disappoint, as she seemed to outdo her previous night’s antics. Would someone tell Jannick that her presence on stage seriously detracts from his music!

I also met some other fans from the French Magma forum. People like Eric (RI Kobaia), Maahnt (name unknown), Bruno (Nohb), Fatcat (name unknown), JC Alluin, Allain Julliac (Chicago), Bernard (Wurd), and others. I was surprised to see Steve Ashworth’s girlfriend return for the last two nights, as she did not seem happy after the first night. Marcello and Tathiane were so bummed out by Thursday night’s performance that they did not attend on Friday.

The next day, Guy and Bernadette Mas organized a nice lunch across the street from Les Halles. Lots of wine, laughter, broken French and broken English. Fanny and Fab Journo could not attend the show on Saturday, so we said goodbye to them.

At the pre-concert meet-up, I had the chance to meet and talk to Klaus Blasquiz, who is friends with Marcello and Tathianne. Very nice guy and speaks impeccable English. I did not have the nerve to ask Klaus about all of the old Magma tapes that he has stored in his basement. (He was Magma’s archivist during the early years, and I cannot imagine the riches that he has hidden away.)

Saturday night’s show might have been the best. Even the super-group that played De Futura seemed to be in sync. I congratulate Jon Grandcamp, the drummer for Infernal Machina, for having so much restraint and letting Vander do most of the drumming.

After the show on Saturday night, we tried to go to the usual brasserie again, but for some reason, it was closed, as were some of the other brasseries around. New York City still retains the title of the “City that Never Sleeps”. I said my goodbyes to Madi and Maxence, and even said goodbye to Jean-Claude Buire.

Why did I go to Paris for these three days? Christian Vander is 61 years old, and Magma won’t be around forever. I need to be able to see Magma while Vander is still able to play. For me, it’s like being able to see the last days of Michael Jordan or Babe Ruth. But the biggest reason for me to have gone to Paris was to see all of the great friends that I have made through Magma, and those are the memories that I like to keep.

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

I am in Paris in Magma Heaven

Leaving in an hour to go to Paris for the 40th anniversary series of concerts by Magma at Le Casino de Paris. I will be back on Sunday night.

Since the comments are moderated here, if you post a comment, I will not be able to approve it until Sunday evening. So please don't keep resubmitting the same comment over and over ... the approval pixies are also in Paris.

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Request: Streaming R or R-like Functions in CEP Systems

I have been looking a bit into R lately. I have written here before about being able to transform R into something that can take real-time streams of data, or integrating R (or SPlus) into a CEP system. Here is a small example. Let's say that you want to take a sample of a "window" of streaming data every few seconds and compute some function on that sample.

Let's see how we can do this in R using static data.

In R, we can define a range of stock prices that range from $10.00 to $100.00, in increments of 25 cents.

> StockPrices<-seq(10.0, 100.0, 0.25)
> StockPrices
[1] 10.00 10.25 10.50 10.75 11.00 11.25 11.50 11.75 12.00 12.25 12.50 12.75 13.00 13.25 13.50 13.75 14.00
......
[358] 99.25 99.50 99.75 100.00


We can take a sample of 10 random stock prices like this:

> sample(StockPrices, 10, rep=F)
[1] 20.50 19.00 78.75 44.75 62.75 45.25 66.25 70.00 71.00 91.00

We can compute the average price of a sample of 10 random prices like this:

> mean(sample(StockPrices, 10, rep=F))
[1] 52.775


Using the Coral8 term of "window", let's say that we have a window called StockPrices that contains one minute's worth of prices. Every 5 seconds, we want to take a random sampling of 10 prices in that window and compute the average price of that random sample.

How can we do it in your CEP system? I would welcome examples in Coral8, Apama, Streambase, Aleri, Esper, StreamSQL, and any other CEP system.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

IBM MQ Series, XMS, and .NET

We have to be able to read in some information that another system is putting out over IBM MQ. There is a messaging implementation from IBM called XMS that seems like it's JMS over MQ. This is great for us because this means that we can (hopefully) clone our existing Tibco EMS adapters, and with a few little changes, be able to read off of MQ.

I am not sure if the original publisher of the data also has to use XMS ... maybe someone here can tell me. I know for a fact that the publishing system does not use XMS. If the publisher does have to use XMS in order for us to consume through XMS, then we will need to come up with an input adapter that uses the native MQ API.

The information about the IBM Messaging Client for .NET is here.


Update: A great comment by Richard Brown of IBM WebSphere pre-sales tells me that there is nothing to worry about if we are just a consumer ... which we are.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Follow-ups to Some Comments

Marco says:
>> What do you mean by "Entitlements on output events"?

I have blogged about this here before. When our CEP engine generates alerts, we do not want everyone to see them. There might be very few people who can see alerts on orders. Prop traders should not see many of the the alerts that sales traders see. One trading desk might not be able to receive the alerts that another trading desk should see.

We have the concept of coarse-grained and fine-grained alerts. A coarse-grained alert might be filtered out by role. For example, a particular alert should only be seen by a North American Cash Trader, and not by derivative traders. In the case of fine-grained entitlements, we might have to filter by certain data that is within the alert's payload. For example, someone might not be able to see an alert if the symbol is IBM and the volume is greater that 1000 shares.

There is also the matter of entitlements on real-time aggregations, but that is another long topic.


Charlie Skelton from KX says:
>> "don't make us code in Q" Please can you elaborate?

Charlie, I have written ad nauseum here about Kdb and Q. I actually started to learn Q a few months ago in order to be able to read some of the code that was thrown our way. I have a greater appreciation of Q than I did a few months ago, but I would not want to ask all of my developers to learn Q.

My opinion is that, if you have a facade (input adapter) to a database, then you should be able to code in some database-agnostic way and leave the db-specific language access to the drivers. Look what you can do with O/R frameworks like Hibernate and iBatis.


Hans asked about Unit Testing

When we first started to use Coral8, we actually wrote a C# application that did the exact same thing as our Coral8 application and we did reconciliations between the two. We had found an issue or two with Coral8, which they were able to fix fairly rapidly. We also found some issues with our Coral8 coding, which helped us learn Coral8 better.

Unit testing is still something that is not mature in our Coral8 processing, although we do have unit tests for our internal framework. We have fabricated data in flat files that we can pump into our C8 engine, and we can do regression testing against a known set of outputs. However, we know that we can do a better job.

I would like to hear more about Streambase's unit testing capabilities.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Streambase Again!!!! Arghhhhh !

It seems that my posting on Streambase's recent customer win has gotten some people pretty ticked off. I have received some private messages from a few of the CEP vendors who are crying foul. Nobody wants to appear that they have sour grapes against the Streambase win, but of course by contacting me, they know that I will pursue the issue in the interest of fairness and transparency.

The issue revolves around the joint claims by Mark Palmer and Penny Crossman that Phase Capital had evaluated all of the major CEP vendors. Penny had written the following:

PhaseCapital evaluated a number of CEP vendors in the trading space, including Progress Apama, Coral8, Aleri and StreamBase.

What some of the CEP vendors are claiming is that Phase Capital had never downloaded copies of their products for evaluation, nor contacted the CEP vendor with any questions.

In my experience, before you can download a copy of a CEP product for evaluation, you need to fill out a registration form. Some of the CEP vendors won't even let you download a copy from their website ... they want to pre-certify you by paying you a personal visit.

So, if Phase Capital did not download the evaluation copies, how did they evaluate some of the products. Did they do part of their evaluation from going over a publically available list of features? Did they get information from blogs such as this one? Did they download the product under another name? Did they have a third-party consultant do the evaluations?

Update: They could have gotten the copies from a reseller. Sybase resells a number of CEP products.

In the interest of transparency, I would ask Greg McSweeny from Wall Street and Technology to ask his writer, Penny Crossman, to substantiate what she wrote in her article concerning the competitive evaluation that Phase Capital did between the various CEP products. Penny is the designated beat reporter for the CEP industry, so I would like to see a bit of investigative journalism from her on this matter. Otherwise, serious doubts would still remain as to Phase Capital's evaluation process.

This is why organizations like Stac Research are so important to our industry.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

What is Important to us in a CEP System

I have posted this information before in previous blog entries, but I think that, since I have some new readers here, I should post a list of some of the things that are important to me in a CEP system.


Input Adapters
-------------------
Market Data - Reuters, Bloomberg, and to a lesser extent, Wombat and Activ
FIX - 4.2, 4.4. Predefined schemas for the most popular FIX messages would be great.
SWIFT
Message Buses - Tibco EMS, RTI, LBM
Tick Databases - KDB+ (read and write capabilities, don't make us code in Q), Vhayu
Object Caches - Velocity, Gemfire, Coherence


Output Adapters
---------------------
Various chat/messenger mechanisms
Message Buses - Tibco EMS, RTI, LBM
KDB+, Vhayu
Object Caches - Velocity, Gemfire, Coherence

Development Environment
----------------------------------
Strong support for debugging
Profiling and performance monitoring
Hooks into Performance Counters
Ability to easily add C#-based custom functions


Analysis
-----------
Integration with tools like R, SPlus, Matlab
Real-time OLAP
Strong support for real-time dashboards


Miscellaneous
------------------
Workflow
Entitlements on output events
Integration with Active Directory
Priority input queues into the CEP engine


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Streambase On The Move?

Mark Palmer seems to be turning Streambase in a positive direction since he joined .... either that, or the Streambase marketing machine is in full throttle.

Remarkably, Mark has managed to get testimonials from a number of financial services customers. Usually we are a secretive bunch, and many of the larger financial institutions do not like to have themselves used as a reference customer. Which makes it all the more interesting that Streambase has been able to persuade these customers to go public. Not only that, but Streambase has managed to convince Penny Crossman of Wall Street and Technology to rewrite their press release into an article.

Say what you will about Streambase's sales and marketing staff .... they have proven themselves to be amazingly effective. They have been able to trot out customer references, whereas Coral8 and Aleri have not. This gives the perception that Streambase is eating the competition, whether that perception is true or not.

Mark posted an interesting paragraph:

As Dr. Goodell described, Apama didn't even make the first cut in their evaluation. On the other hand, Aleri, the other final CEP product in PhaseCapital's evaluation, fared well with its high-performance, multi-threaded server architecture.

And, Penny follows it up with:

PhaseCapital evaluated a number of CEP vendors in the trading space, including Progress Apama, Coral8, Aleri and StreamBase.

It's very interesting that Phase Capital found that Aleri was the highest performing CEP engine. This has confirmed something that I have suspected. Aleri has always been touting themselves as the most performant of the CEP engines, and has welcomed the other CEP vendors to take the Aleri Challenge. However, it may have been the lack of spit-and-polish in Aleri's development environment that may have sunk it. I have previously blogged here that I thought that Streambase had the best development experience and documentation out of all of the other products.

I am wondering what factors caused Phase Capital to not choose Coral8. One thing that Streambase has that most of the others do not is built-in adapters for trading companies. As Penny writes:

StreamBase already had an adaptor for the QuickFIX engine as well as adaptors for the data streams provide by Lime Brokerage, PhaseCapital's execution broker. "The building blocks were very complementary to what we're ultimately trying to deploy," Pritchett says.


I have been saying this for a long long time on this blog --- the quality of the CEP engine does not matter as much as the entire ecosystem surrounding the CEP engine. You can have the best, fastest, most expressive CEP engine and language, but if you don't surround it with a compelling ecosystem, then you are sunk. And, Mark Palmer understands this.



©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Real-time Messaging in Sybase 15

If your organization is migrating from Sybase ASE 12.5.4 to Sybase 15, then you might want to take a look at these slides.

One slide caught my attention. It was a slide on the migration effort done by eSpeed. One of the things that eSpeed did in their Sybase migration was:

Replace "Polling-Broadcasting”legacy solution with integrated real-time messaging (RTDS 4.0) in ASE 15

I need to find out more about this. Sybase has some slides here.

Many financial companies who are using Sybase have applications that actually do database polling every 'n' minutes in order to find out what rows in a table have changed. If eSpeed did what I think they did, then this presents itself as a compelling reason to move to Sybase 15. (Of course, this assumes that you are not goign to be moving eventually to SQL Server 2008/2010.)

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Back from Seattle

I just returned home from a two day trip to Microsoft. I cannot say anything specific about this trip because of NDAs and my promises to the Microsoft team. But, there is GOODNESS coming out of Redmond soon, and I am totally pumped !!! And, the people there "get it". And, the people there are listening.

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

I Have Succumbed to the Facebook Phenomenon

I always said that I would never join a social network where my kids were active. The kids have a word called "teenile", which refers to an adult who tried to act like a teenager. I vowed that nobody woul dever describe me a teenile.

I establish a token account on Facebook a few years ago in order to try to see what my kids were up to. However, since I had to "friend" my kids in order to see what they were doing, and since any sensible kid will never accept a friend request from his folks, then I just let my Facebook account lie dormant. Over the years, I have received friend requests from various tangential business associates, but I have always ignored these.

Over the past few weeks, the pressure just became too much to resist. First, my high school graduating class (and the class directly before and after mine) established Facebook groups, complete with photos. Then, a few old girlfriends managed to track me down and sent me Facebook links to photo albums of me in somewhat embarassing situations. Then, a few of my college buddies. who I still see, got themselves up on Facebook.

So, with a beet-red face, I admit that I am now a Facebooker.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Some travel plans

I'll be at Microsoft in Redmond on January 25, 26 and 27. If any of the Microsofties who read this blog would like to talk to me about capital markets, please send me an email.

From February 11 to 15th, I will be in Paris to see the 3-nights of concerts by Magma to celebrate their 40th anniversary. I don't even want to think about work and capital markets during those 5 days ... I will be searching for a bar that serves Absynthe, so if you have any recommendations, please let me know.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Followup on Horizon Graphs

(From Stephen Few)

This paper confirms the usefulness of Horizon Graphs.

Congrats to Panopticon and Stefan Odelik.

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Reuters Feeds and Market Cap (Grrrr....)

We need to find out the market cap of a company that we are receiving quotes on from Reuters using RFA 6.x. We see a field called "MKT_CAP" coming through on the initial quotes. Looking at the Reuters documentation, this should contain the market capitalization of the company.

However, every MKT_CAP field has the value of zero.

Reuters support has confirmed that the MKT_CAP field will ALWAYS be 0 for every quote.

So, Reuters ... why are you sending this field with every quote if you are never populating it? And, how many other fields that come with each quote are not being populated with valid values? Don't you realize that low-latency means that you should be sending the minimum of data for a quote .... and we expect each and every field to contain something useful?

Brian ... please clean this up if you can.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Blogs Influence Purchasing

References here to a study by Jupiter Research

I get a lot of private email with questions like this: "I read your blog. I want a buy system X. what do you think of it?" So, I can personally attest that people who want to purchase trading/CEP/algo systems do a lot of research on blogs. I think that it would be irresponsible to make a major purchase without researching every piece of information that is publicly available, and that include what people say on blogs.

I think that a blog, if written responsibly, can positively influence the development and quality of a product. If I was running a software company, I would certainly try to keep any negative publicity off of the blogosphere, and if that means doing a little bit of extra QA, then I would do it ... and that benefits all customers.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Two interesting posts from Scott

Scott is the main UI and build/deployment guy on the CEP team. He is also the one who has pushed us into the world of WCF and LINQ.

He just came out with two interesting posts that have ramifications for our development team.

We communicate with many of our external systems using Tibco EMS. However, between our CEP GUI and our one of our .NET-based CEP servers, we use WCF. We were experiencing some performance issues when a lot of users started to connect to our servers, and the issues were so bad that I had asked Scott to start to consider abandoning WCF and going to Tibco for communication between the GUI and server. Not even the Microsoft PSS guy that Microsoft sent us was able to help.

In his blog post, Scott documents all of the steps he took in order to solve the performance issues. I advance the notion that some of these problems could have been avoided by documentation and patterns from Microsoft, either from the WCF group or from the Patterns and Practices group. Also, I would like tools like Resharper and Coderush to be able to identify possible performance problems in code like this:

[OperationContract]
MyDataTable GetData(...);

In another interesting post, Scott shows what it would take to use LINQ with KDB, and validates the notion that we can probably replace CodeRush templates with T4. (In this economic environment, if we can get away with a free templating tool that is just as good as CodeRush, then it is our fiduciary duty to use it.) Even though several people on my CEP team have started to learn Q on their own (including myself), it is more natural if we use LINQ to query our KDB database. (This is the approach that we would have liked for Coral8 to take when they wrote a KDB adapter ... why query KDB in Q when you could have queried it in CCL?)




©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

CEP 2009 Predictions

David Luckham has asked the CEP crowd to weigh in with their predictions for the CEP marketplace for 2009. While this is an interesting effort in mental masturbation, I think that you can only look foolish if you set yourself up as a prognosticator in an economic environment like the world is experiencing right now.

Instead, I would point you to certain facts that I have observed in 2008, and you can draw your own trendlines from these data points.

1) There has been a massive consolidation in the bulge-bracket financial firms. I know for a fact that most of the CEP companies had spent a lot of time and resources on POCs at some of the now-deceased firms. If a CEP vendor spent most of their time pursuing these IBs, then you can be assured that they lost some amount of money on pursuing these deals. Most of the acquirers of the struggling IBs (Wells Fargo, Bank of America) are not known for their pursuit of advanced technologies.

2) There has been a consolidation in hedge funds. Some are no longer with us. Most of the others have lost money and will have to start consolidating expenses.

3) The bulge-bracket IBs that are still around are consolidating expenses, and taking a very hard look at discretionary technology spending. In other words, it will be very difficult to convince the business to spend $250,000 on CEP technology because it looks "interesting". Technologies that allow machines to do what humans do now will be looked at favorable. Green technologies and data center consolodation (ie: expense reduction) are also looked at favorably.

4) The Gartner CEP conference was poorly attended by customers.

5) Cloud Computing is now the buzzword of the day. People do not go into a cold faint anymore when you mention that you are involved with CEP. Blog traffic is down in the CEP space. The bloom is off the rose.

6) Venture Capitalists are re-examining their investments. There is pressure on the CEP vendors to generate revenue. Several of these vendors had approached me with very interesting and peculiar financing deals in 2008. Some of the CEP vendors have had reorganizations and layoffs.

7) Several of the horizontal CEP vendors are now hotly developing for the financial vertical. Will these new products be used for alpha generation, risk reduction, or cost savings?

8) I get email frequently from people in the financial industry who have heard the term CEP, but have no idea how to apply it to their business. The interest is there. The ready-made solutions are not.


What does all of this mean in 2009? Although these are not predictions, things that I would not be surprised to see in 2009 are:

1) Consolidation of the CEP marketplace. It would be difficult to merge different CEP product lines. However, a larger company would be able to buy a smaller CEP vendor for a cheap price in order to get rid of some competition and to pick up some very good talent. (Hans and Tim predicted this.)

2) A push towards delivering realtime OLAP and integrated analytical tools. The CEP vendors will have to make it more difficult for IT staff to justify the development of their own CEP technologies. Ready-made stacks of CEP processing need to pushed out there. (This is analagous to Hans Gilde's "patterns".)

3) A greater push towards using LINQ as a streaming query tool.

4) The emergence of a new major player in the CEP marketplace, perhaps with CEP technology as a free add-on to their existing technology stack. This would make stream-based processing more mainstream, but would set a new (very low) price point that other CEP vendors would feel compelled to match.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Are You All Getting Ready for the New Options Symbology?

http://www.theocc.com/initiatives/symbology/default.jsp

We are getting ready. Are you?

The new symbology (21 bytes per symbol)
Symbol – 6 bytes
Year – 2 bytes
Month – 2 bytes
Day – 2 bytes
Call/Put Indicator – 1 byte
Strike Dollar – 5 bytes
Strike Decimal – 3 bytes


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.