Thursday, July 31, 2008

Berkeley Visualization Labs

The Berkeley Visualization Labs has a nice Java (grrr..) tool kit for visualization here.

In particular, their FlowMap visualization is a nice way to see order flow, as orders come in and get split up amongst different systems.

©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Sunday, July 27, 2008

Lock-Free Queues and Market Data

Courtesy of Craig ....

Lock Free Queues

And, if you scroll down to the end of the blog posting, the author has links for other lock-free data structures.

According to Craig, this is ideal for implementing the market-data pattern.

(The author of the blog, Julian Bucknall, is the CTO of DevExpress. This might become my new favorite component vendor. Their Xtra PivotGrid is a pretty nice grid for viewing OLAP cubes.)

©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Squiggles vs Resharper

Squiggles for C# will be available in Service Pack 1 of Visual Studio 2008.

The big question for me is how it will cooperate with Resharper. Will be have a race condition where Resharper and Squiggles do simultaneous error checking and fight to be the first one to notify us? How about the possible overlap of rules?

I hope that the VS team is aware of tools like Resharper, and is not working in the typical isolationist-mode that is prevalent in Microsoft.

©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Saturday, July 26, 2008

New York Times Article on Wall St. Pay Drop

It's sobering to think that we are down to 6 major investment banks in New York City : JP Morgan, Morgan Stanley, Lehman, Citigroup, Goldman Sachs, and Merrill Lynch. You also have Bank of America, Wachovia, DeutscheBank, Barcap, and UBS among the secondary employers (ie: banks whose primary headquarters are not in New York). If Lehman goes down, then it leave 5 major IB's.

Back in the day, you had many many more firms that Wall Street IT professionals could hop between. The old adage was that you had to change companies every two years, with each move resulting in a salary increase and a bump in title. You had companies like Chase, Banker's Trust, Manny Hanny, Kidder Peabody, Drexel Burnham, Chemical Bank, Republic Bank, and many more. Then, you had all of the consulting firms like PWC, Ernst and Young, Anderson, Coopers and Lybrand, etc.

With all of consolidation that has gone on, plus companies who were forced out of business by the regulators, you are now more limited in terms of companies to go to. There are countless hedge funds in Connecticut, but that means you have to commute to Connecticut.

©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Tuesday, July 22, 2008

Welcome, Patric

My CEP team is happy to welcome Patric, a very experienced developer who specializes in UI design.

Our GUI is shaping up very nicely. We made the decision to go to WPF, and under Scott's leadership, the GUI is starting to come together nicely. We have been getting a lot of oohs and aahs on the trading floor, and the buzz that the GUI is generating will hopefully give us space on the traders' screens.

We have just about reached the point where people stopped wondering what our row on the trading floor has been up to. We are now starting to get requests from different groups, which is a position that we were hoping that we would be in. As many people have said about CEP, once you show people what you can get from a CEP infrastructure, your next set of problems is how to balance all of the requests that you will get.

©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Saturday, July 19, 2008

The 2008 Waters Rankings

(If you are getting this post at the same time as you are getting my post about the Fluffies, read the Fluffies first.)

Coincidentaly, right after I posted the entry about the Jolt awards, I received my hardcopy of Waters Magazine in which they posted articles about their award winners.

What is the difference between the Jolt Awards and the Waters Rankings?

First, I trust the process that Waters has. In order to vote in the Waters Rankings, you need to be a subscriber to Waters. Most people in financial institutions can qualify for a free subscription to Waters, and you can fill out a form to apply. Waters will go through the trouble to qualify you for a subscription, and will make sure that you work for a financial firm. Unlike a magazine like Dr Dobbs, buying a copy of Waters magazine is difficult to do, and individual copies are quite expensive. Therefore, my conclusion is that Waters has a subscription base of highly-qualified financial professionals who have experience with at least some of the products that they vote for.

Second, in the Waters process, the readers themselves vote on the products. Waters received 600 votes this year. In fact, I am constantly receiving emails from Waters, encouraging me to vote on this and that. So, unlike a secretive panel of "highly renowned judges", the users of the products themselves are the ones who vote in the categories. (For the record, I do not participate in the voting.) Waters also goes through a process to weed out votes from the vendors themselves.

So, whereas the Jolt Awards have a lot of question marks associated with them, I trust the Waters Rankings more because I expect that the voters have actually evaluated or used the products that they vote for.

Just like Tim, I find the distinction between the Streaming Data Management category and the Complex Event Processing category to be a confusing one. Nevertheless, congratulations to Streambase's marketing machine. When I was evaluating Streambase, I found many good things about their product, and if their marketing and sales organization did not turn me off so much at the beginning, we might very well be using Streambase instead of Coral8.

It was also good to see Kx Systems in there as the number 3 solution. Kx certainly does not have the marketing oomph that Streambase has, relying on the mystique of Arthur Whitney and its Q and K languages instead. I would consider KDB to be more of a pure streaming data management solution (along with Vhayu) than a CEP solution.

©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

The First Annual Fluffies for CEP

Tim Bass wrote an excellent post the other day in which he questioned the categorization process around “Streaming Databases” and “Complex Event Processing” products.

Tim’s post inspired me to write something that I have been meaning to write for a very long time. This has to do with how some CEP products have won "Fluff" awards over the past few years.

In this ( press release, Coral8 announced that they have won the 2007 Jolt Award. According to the press release, “The Coral8 Engine was selected from among 26 product nominations in the Database Engines and Data Tools category by a renowned team of judges that select Jolt award winners based on how nominees have made it faster and easier to create data-centric applications.”

In this ( press release, Aleri trots out the fact that they won the 2008 Jolt Award, which I assume tells people that Aleri is now better than the 2007 Jolt winner.

“Being recognized by Jolt’s renowned panel of judges consisting of industry insiders, columnist, and technology leaders demonstrates the impact that our complex event processing software has had on the industry,” said Don DeLoach, CEO of Aleri.

Both Aleri and Coral8 were graded by a “renowned panel of judges”. I guess that Terry and Don both drank the Jolt Kool-Aid (sic) that day :-)

Another award that was just won by Coral8 was from Windows in Financial Services magazine, which is a free magazine given out to people in the financial community. This magazine has absolutely no technical content, and reads like one large advertisement for Microsoft.

“Coral8 was a clear choice for the 2008 WFS Innovator Award because of their demonstrated leadership in providing solutions that deliver results,” said Windows in Financial Services publisher Joe Viviani. “We evaluated companies of all sizes and the companies selected delivered ground-breaking applications and platforms that better serve the securities industry.”

Who did the judging for this award? What were the criteria? It was interesting to see that Coral8 was the only CEP vendor that was showing off their wares at the Microsoft booth at SIFMA. Did WFS magazine happen to consider Apama? How about Streambase? (Oops … sorry WFS readers ... Streambase is written in Java)

Just who are these judges? According to Don, they are columnists and technology leaders. Wow! With respect to my journalistic colleagues like Greg McSweeny, when was the last time a columnist cracked open Coral8, wrote some CCL, and forced market data feeds into the pipe. Have any of these “technology leaders” taken Aleri and built and end-to-end application with it? Most of the columnists I know have trouble understanding the things that they write about.

What were the criteria that these renowned judges used to benchmark the systems? They should let us know, because this might mean that we don’t need the Stac Council anymore. The CEP working group of the Stac Council spends a few hours a few trying to work out comprehensive and objective testing specs for CEP products. The Stac Council is comprised of the chief architects of most of the CEP vendors, plus several customers from Capital Markets firms.

I can imagine the reaction of people like Jon/Jack/Scott from Aleri, Mark/Bob from Coral8, and Thomas from Esper when they read about these Jolt awards. These people have worked hard to provide compelling architectures, and awards like these Jolt Awards tend to marginalize their work.

I am not picking on Coral8 and Aleri here. I love both companies, and both have great products, great technical staffs, and CEO’s who are passionate about their products. What I find disturbing is the fact that they are giving credence to awards that, to me, seem like pure fluff. I talk to my colleagues at our competitors about their evaluation processes, and believe me when I tell you that the Jolt or WFS awards have zero impact on our evaluation process. These awards also marginalize the hard work of Stac to provide real benchmarks that companies can use when evaluating these products.

Awards are fine when the judging process is transparent to all. Until then, CEP vendors should not insult their customers by crowing about fluff. It works to your detriment rather than to your advantage.

©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Saturday, July 12, 2008

Eclipse Open Market Financial Platform


©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Tuesday, July 08, 2008

A First Hand Look at Wall Street Layoffs

The old saying goes that you never feel a recession until it hits you .....

I haven't seen this much good talent that has "become available" on the Street in quite a while. I am humbled by the level of talent that I have interviewed lately. I have recently had the privilege of having a number of "heavyweights" sitting across the interviewing table from me, including a number of people from competing firms who never thought that they would be ever be interviewing at my company.

If you are reading this blog, and you have recently been let go from your firm, then you may take some small solice in knowing that there are a lot of fantastic people out there in your position ... so, don't take the fact that you have been separated from your company as an insult on your level of talent and expertise. Unfortunately, this is the nature of Wall Street. We over-hire in good times and over-fire in bad times. If you have great technical skills, then there are many opportunities for you outside of the financial sector. Look at the list of interesting positions over at the Joel On Software site.

I have been through a number of these cycles.

I was developing trader workstations at Goldman Sachs in 1987 when the "Crash" hit on October 19, 1987. A large number of us were gathered around the Quotron terminal, laughing hysterically as the Dow dropped 500 points, watching our holdings evaporate. Little did we know that Oct 19 was one of the best buying opportunities in history.

My software company, Magma Systems, was going full steam in 1992 when that recession hit. To tell you the truth, I did not even notice the recession, because we were selling a niche product that did extremely well, but the economy recovered from these hiccups.

I was with my wife in Brazil and Argentina in 1998, at the height of various currency crises, and we recovered from this.

I was in New York on September 11, 2001, and this unfortunate day happened right in the middle of the DotCom bust. 2002 and 2003 were some of the worst years I could remember on Wall Street. We over-fired, and then starting in 2004, we were going like gangbusters again.

I view this as an opportunity. This is a period in which IT departments on Wall Street can assert themselves, and come up with really innovative products to support the business. From adversity comes opportunity. I am ever the optimist.

©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Wednesday, July 02, 2008

SPlus, Adhoc Queries, and Coral8 Public Windows

I am reading a Powerpoint presentation by Prof. Eric Zivot on SPlus, and I was intrigued by the ad-hoc query facility that SPlus offers. For example, if you have TAQ tick data loaded into SPlus, you can use the following SPlus query to get the average price of Microsoft over a non-overlapping 5-minute window:

mean.5min = aggregateSeries(msftt.ts[,"Price"], + by="minutes",,FUN=mean)

Coral8 recently implemented queryable "public windows" as a way to do ad-hoc queries. In order to get this feature out the door quickly, Coral8 embedded a version of SQLite into their system. So, you cannot use the Coral8 streaming query language, CCL, to do ad-hoc queries of windows. Instead, you need to use a dialect of SQL-92.

(Coral8 tells me that eventually. you will be able to use CCL for ad-hoc queries)

SPlus looks like it supports a lot of built-in functions for time-series financial data. This is something I would also like to see in Coral8. (I guess there is an opportunity for a little cottage industry to be built around enhancing Coral8 for capital markets firms...)

I wonder if any of the CEP vendors considered SPlus/R as a query language before deciding on Streaming SQL? (I know that JOS has been on the R bandwagon for a while at Barcap. He hasn't blogged too much about it recently though...)

©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

First experiments with Velocity

I played around with Microsoft Velocity this past weekend, and I put together a little Order Cache. This application reads FIX messages off of a Tibco EMS bus, turns the messages into a C# object, and stores the object in the Velocity cache.

The primary key of the Order Cache is the ClOrdId, and this value should be unique for each trading day. Every time we see a FIX message, we check to see if an object with the same ClOrdId is in the cache. If it is, the Order object is updated, and store the object back in the cache. The order and execution state is kept, the quantities are updated, and Cancels and Cancel/Replaces are accounted for.

Admittedly, it is an extremely simple application, but I did it just so I could get familiar with the first CTP of Velocity. One cache holds all order objects. I did not try to alpha-partition the orders by symbols amongst different caches, nor did I experiment with multiple regions yet. Just simple Gets, Puts and Adds. I used Reflector a bit to peer into the Velocity code, and I could see that there is a lot more hidden under the surface of Velocity that has not been exposed yet.

Where can we use an object cache with a Complex Event Processing system?

First, we can use it to support resiliency of the CEP system. Let’s say that our CEP engine crashes in the middle of the day. We want to be able to retrieve the current state of our orders and executions, but we don’t want to have to go back to some sort of FIX repository, and play back each FIX message in order to re-create the state of the CEP engine. We can use the object cache to aid in recovering the state.

Now, most CEP engines have a facility where they can save state. For instance, with Coral8, you can choose to save all windows out to a database every n seconds or minutes. But, there is a certain tension with persistence of a CEP engine that is processing fast streams of real-time data. If you persist too frequently, you can slow down the CEP engine. If you persist infrequently, then you can lose a significant amount of your state if the CEP engine fails.

Most of the CEP engines will use an external database like SleepyCat, BerkeleyDB, and SQL Server to store state. But, depending on the size and shape of your saved data, recovery might be slow. So, why not save to an object cache? Reading and writing to an object cache would be extremely fast, and if the object cache supports asynchronous storage to a back-end database (or Microsoft’s SSDS), then you can have the best of both worlds.

Among the various CEP engines, I have not seen any built-in adapter support for object caches, like Gemfire, Tangasol, GigaSpaces, and Velocity. It would be fairly easy to write one, but it takes time to marshal between a native C# object and something like a Coral8 tuple.

Wish #1: The CEP vendors to explore object caches as a way to do persistence.

Another use of an object cache is to assist with a query façade. CEP engines like Coral8 support the notion of ad-hoc queries of the windows. If the CEP engine is aggregating data like orders and executions inside a publicly queryable window, any application should be able to query the CEP engine for something like “What are the top ten stocks that have open orders between 10:15 and 10:30”.

We don’t want external applications connecting directly to the CEP engine. There are compliance and security issues involved, and we don’t want any application tied to a particular variant of Streaming SQL. (Note: Coral8 has SQLite embedded in it, and that is what you use to query the public windows.) Therefore, we want to provide a “query façade”. All applications have to connect to the query façade in order to do ad-hoc querying of the CEP engine.

There is a cost to performing ad-hoc queries on the CEP engine while it is trying to process tons of streaming data. And, unless you index your public window correctly, the ad-hoc query can be very slow.

So, why not be able to query the object cache instead? If we are storing aggregated orders and execution data inside of Velocity, why can’t we query Velocity to find the top ten stocks that have open orders between 10:15 and 10:30? In other words, querying an object cache should be almost the same as querying an SQL Server database. I don’t care if the query language is SQL 92, Streaming SQL, LINQ, or the LINQ-ish thing that SSDS has…. As long as it’s not Q :-)

Wish #2: Microsoft will get together with the object cache and CEP vendors and come up with a common method to query database, caches, and CEP engines.

©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Some Summer Reading

A great read on the collapse of Bear Stearns in this month's issue of Vanity Fair.

More on the Vanity Fair article at the New York Times.

Especially interesting is the part of the Vanity Fair article where they talk about dealing with CNBC. There is really too much power given to the CNBC correspondents. Give one correspondent an interview, and the other correspondents will snipe at your company on the air.

©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.