Orinoco vs Reactive Framework

I have been doing a bit of playing around with the new Reactive Framework (Rx) lately. And, I like what I see so far.

The first thing that came to mind was how the Orinoco team was going to resolve the fact that another LINQ-based streaming API was coming out of a "competitive" group in Microsoft.

A little birdie has told me that recently, Microsoft merged several of the competitive division, and that Orinoco product was going to use Rx as the LINQ-based surface API.

If this is true, will this mean that we all have to wait until .NET 4.0 in order to use Orinoco? Or, will there be a .NET 3.5 SP2 that contains the necessary Systems DLLs? If .NET 4.0 is required, it would be an interesting situation for a lot of the very large customers (like us) who are quite a ways away from even looking at .NET 4.0. Not to mention the certification process that most companies have to go through before adopting a major .NET release.

By the way ... Too bad that I had to hear this news from external sources, as the Orinoco team has gone very quiet lately. Let's just hope that it's because of the summer vacation, or because their developers are heads down in code. And, like Matt Davey pondered ... How can they be releasing CTP 2 when CTP 1 has not even gone out yet?

(Note : All references to Orinoco should be changed mentally to StreamInsight. Too bad, because I like the name Orinoco much better.)

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

New Magma Blog

For those very few people who come here to read about Magma, the French Rock band ...

I have just started a new blog just for Magma and Zeuhl music. Please come and visit, and remember to add the new blog to your blog reading software.



©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Drools 5.0 for CEP

The new 5.0 version of Drools seems to have CEP capabilities.

Has anyone here tried it out yet? Do you think that it can handle about 10,000 events per second? Has anyone pumped market data through it?

I am going to start reading up on Drools Fusion and see how it contrasts with the upcoming Microsoft Orinoco. Fusion is here right now. Orinoco .... well .... we'll see.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

High Frequency Trading

I am sitting in my hotel lobby in Santiago de Compostelo, Spain, and trying to put in a half hour of laptop time while my wife and daughter are still sleeping. I am reading all of the hub-bub surrounding HFT and algo trading, all brought on by the arrest of the Goldman Sachs employee. This morning, Paul Wilmott has an editorial in the New York Times.

What is going through my mind right now is the fact that, if HFT was regulated or banned, it would cost the jobs of a good number of very smart people in the Capital Markets industry. It would cause irreparable harm to some of the companies that support low-latency infrastructure. And, it would transform the Capital Markets industry in ways that I could not even imagine.

More thoughts on this later.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

New York Times HFT Article

New York Times article and opinions on High Frequency Trading here. It also has a good explanation of what is called "gaming" in algo trading.

(Sorry for the lack of posts ... I've been traveling in Europe for the past week and will return to New York in the first week in August).

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Nice Wall Street & Tech Article on Citi

Nice article here.

Very interesting quote on Page 2, second paragraph.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Goldman's "Stolen" Program Trading code

(I am sitting here with a HP dv9700t laptop with two broken hinges, trying to cross my knees in such a way that the screen will be propped up in a way that I can see it. I am going to Spain for the last two weeks in July, so right before I go, I will send the laptop off for repair. I estimate the repair cost to be about $300. I cannot believe that HP makes a complete line of laptops whose hinges crack after a few months, and are so arrogant and uncompromising when it comes to addressing this issue.)


I spent an hour on Monday morning reading the great post on ZeroHedge about the Goldman developer who zipped up the code to a High Frequency Trading (HFT) system. I am not going to rehash what has been written in the various news outlets. But, it's always interesting to read Lars Toomre's opinions on the subject. (Update: here's another one from Hack The Market)

I am not going to weigh in on my own opinions, as I feel that it's not appropriate to comment on the public issues that one of our competitors is going through. But, here are some things that I have seen in my time on Wall Street:

1) Many developers seem to come to a new firm with some old code in hand. I have interviewed candidates in the past where, in the middle of an interview, they pull out a stack of printed source code and show it to me as proof of their coding ability.

2) Many coders think of themselves as authors and artists. When they write something, they want to keep a copy for posterity sake. They might not ever want to use it again. They just want a memento of what they have written. There is no malfeasance intended.

3) Companies are cracking down. Look at what went on with Morgan Stanley a few years ago when a consultant sent client data to his personal email account. You always have to assume that every keystroke, every out-going email, every website you browse to, is being monitored by your company.

In my opinion, you can always write the same application again, but even better. And, if you bring your old source to a new company, you are putting your new company in a precarious situation. And, if you decide to work on any other projects in your spare time, make sure that you check your employment contract very carefully ... some companies say that anything you write during your term of employment belongs to them, even if you write it in your off-hours at home.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

SIFMA 2009

Not much to write about SIFMA this year. I was warned by a friend that it was a ghost-town on the first day. I went to SIFMA on the second day, which is usually the most attended day, and unfortunately, my friend was correct.

In 2008, SIFMA has 350 vendors, but this year, they had 201 vendors, which is a decrease of about 42%. The side corridors used to be jammed with small vendors, but they were totally barren this year. Many large vendors were conspicuously absent, like Sungard and Reuters and Intel. Cust0mer traffic was way down, and in most booths, vendors were conversing with other vendors.

Sorry folks ... not much to write about here .....

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Cloud Computing

Matt seems to have read my mind. I have just started a dive into Cloud Computing, and today, my questions to various CEP vendors were "Can I use you in the Cloud? Have you deployed to Amazon EC2 yet?"

There are various reasons why, at this time, I feel that we should start looking at the Cloud. I will be posting more on this in the future. But, for apps that are not latency sensitive, and if we can obfuscate our data and get the blessings of our Compliance Department (not an easy thing to do), I would like to think about off-loading some of our CEP processing (as well as some other trading functions) to a public Cloud.

I will be catching up with the Microsoft Orinoco team soon. And, it will be interesting to hear about their plans for Azure.

Meanwhile, this looks like a pretty good blog. Especially this post.....


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Karen's Cancer is Gone

Congratulations to Karen Shanahan for beating the odds.

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

DSLs for Trading using Microsoft Oslo

Video on using Oslo and DSL's for testing trading systems.

(Thanks to Terry for the link)

I would love to explore using a DSL to have our traders register lightweight queries with our CEP system.

Notable about this use case is their use of Microsoft's CCR.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Some Small Ideas on Google Wave

I went through the painful 1 hour and 20 minute bad-joke-athon that was the Google Wave video on YouTube. Nevertheless, I see some immediate things that I would love to deliver to our traders.

The obvious thing would be the ability to drag and drop some of the interesting graphs that we produce from our real-time WPF-based dashboard onto a Wave, have the traders discuss the graphs in real-time, and embed the graphs and discussions into a blog for dissemination to other departments. (Do we even need a blog anymore if everyone has Wave?)

We might also have a Wave Robot tied into our CEP engine or our delivery mechanism so that, upon seeing certain interesting events, the Robot automatically publishes the data into the Wave. This means that we would need to write a Wave Output Adapter for Coraleri or Orinoco.

The final thing I would hope that we could do with Wave is "bookmark" a certain point in time. When traders interact with this "temporal bookmark", the traders can use the "playback" feature of Wave to replay the events leading up to this bookmark. The playback feature would interact with our CEP engine or our Query Facade in order to replay the state.

Our business sponsor is someone who is known for thinking outside the box, so I am hoping that he comes up with similar ideas of how CEP can be married with Google Wave.

One thing that I do know ..... Microsoft has its work cut out for them if they ever want to compete with Google Wave and some of the other innovative products that are coming out of Google. I am hoping that the thinkers behind Orinoco are thinking the same thing, and will start thinking outside the box too!


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Two New User Groups on LinkedIn

I just created two new User Groups on LinkedIn.

1) Microsoft Complex Event Processing
This is for discussions surrounding the new Microsoft CEP (aka Orinoco) product.

2) Aleri
This is for the newly combined CorAleri product. We can discuss the legacy Coral8 and Aleri products, and discuss the parts of the CorAleri roadmap that are not under NDA.

If these groups interest you, please feel free to join.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

The First First Derivatives/KDB User's Group

First Derivatives held it's first New York KDB User's Group meeting at LiquidNet in new York city on May 28th, 2009. About 100 people showed up, and it seemed that there was no more than one degree of separation between everyone in the room. Jacob Loveless did a great presentation on the things that he is doing at BGC/Cantor, and Dan Nugent showed a glimpse of a KDB unit testing framework.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

One Trader's Opinion about Algo Trading and CEP

I was recently talking to a very technical algo trader from another company about CEP. I asked him why he did not use a product like Apama or Truviso for his algo trading needs. His reply was that it did not give him a competitive advantage if he used the same technology as other competitors used. He has the budget and the resources to build out a custom platform, one which he thought would give him a distinct advantage.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

A Brief Look at LINQ and Microsoft Orinoco

This was taken from the Microsoft whitepaper on Orinoco. Thanks for the Orinoco team for providing this to me.

Let's assume that you have a number of power meters that you need to monitor. The "InputStream" contains a stream of tuples where each tuple contains a power reading from an individual meter. The reading happens to come in as the wattage times 10.

We can capture each reading a project a new stream that has the normalized wattage. We can also create a sliding 3-second window of meter readings that we can do calculations against.

var realValueStream =

from e in InputStream

select new MeterWattage((double)e.Consumption / 10);

var windowedStream =

from e in realValueStream.SlidingWindow(TimeSpan.FromSeconds(3))

select new MeterWattage(e.wattage);

We can then create a stream that has the average power reading over a sliding 3 second period.

var averagedStream =

from e in windowedStream

select new OneMeterAverage(CEPAggregates.Average(e.wattage));

(I think that this next query may be incorrect, as the ID field was not captured in the windowedStream, but I will continue with the example anyway).

If you want to get the 3-second sliding average for each individual meter, you can do the following 2 queries:

var meterStreamGroups =

from e in windowedStream

group e by e.id;

var groupedAveragedStream =

from eb in meterStreamGroups

from e in eb

select new MeterAverage(eb.Key, CEPAggregates.Avg(e.wattage));

Now, we want to compute the ratio of consumption for each meter over the 3-second window.

var sumStream =

from e in groupedAveragedStream

select new MeterSum(CEPAggregates.Sum(e.wattage));

var ratioStream =

from e1 in groupedAveragedStream

join e2 in sumStream on true equals true

select new

MeterRatio(e1.Id, e1.Wattage, e1.Wattage / e2.TotalWattage);

Finally, we just want to capture the ratio of power consumption from meter #2.

var filteredStream =

from e in ratioStream

where e.id == 2

select e;

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

A Strange Dream

I had a very strange dream last night. I dreamed that I left my current job and went to work for Google. When I got there, I found myself in a big loft, surrounded by 20-something year old geniuses. I asked them where my desk and computer was. They looked at me with puzzlement. I asked then where all of the computer monitors were, and they didn't know what a computer monitor was. I started describing how, on the trading floor where I worked, the traders has 6 or 8 monitors stacked on their desks. Then everyone started to laugh at me.

In seemed that everyone at Google worked on laptops. They had no conception of what a desktop system was, and the fact that a trader had multiple monitors seemed absurd. The Google-ites didn't even have desks. They just sat on the floor wherever they wanted with their laptops.

I think that the dream was inspired by an article that a friend just sent me.

And, it got me wondering if we at the large investment banks could be doing things in a better way .....


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Orinoco Pricing vs IBM System S ???

The New York Times had a nice article on IBM's System S. You know that it's been a good week for CEP when the Times starts covering it.

IBM has really been rolling out the PR machine on this one. They have been going around to the various investment banks and talking a bit about the pilot that they did with TD Securities, which looks very much like an options market making app.

Jerome wants to know what I think of IBM System S. I hope to have a future blog post devoted to System S and their SPADE language.

Another reader pointed to a quote at the end of the Times article, and wanted to know why I thought that Microsoft should be "giving out Orinoco for free". The quote from the Times article is:

"Buying such an advantage from I.B.M. has its price. The company will charge at least hundreds of thousands of dollars for the software, Mr. Mills said."

This should not be such a shocking quote. Most CEP vendors charge several hundred thousand dollars for their software. But, IBM has been using System S somewhat as a showcase for their Blue Gene supercomputer, and IBM has not made a secret of the fact that System S works great on the grid computing infrastructure offered by Blue Gene. And, if you want to marry System S with Blue Gene, you will find yourself few million dollars lighter in the wallet. But, then again, a trading firm can make that money up with one good day of trading.

I just cannot picture Microsoft moving into the same pricing model as IBM yet. Orinoco has just started to explore the area of HPC. Orinoco hopes to be able to "process 100,000 events per second" while IBM is claiming to process several million per second. (Of course, what do these people consider to be an event? And, what do they mean by "process an event"? Does this mean complex calculations, outer joins, or does it mean just putting an event into the memory space of the CEP engine? We need Stac!!!)

Orinoco can be best positioned as an add-on to SQL Server, something that Microsoft has to do in order to compete with the new CEP offerings by Sybase and Oracle. Microsoft should be capturing the mindset of every .NET developer, working its way into a company starting at the ground level and permeating the organization upwards.

I can imagine that IBM does not want System S to appeal to the everyday developer ... just to the ones that work for companies and government agencies that have multi-million dollar budgets for real-time analysis. IBM says "If you have to ask about the price, then System S is probably too expensive for you." The actual cost of System S will be a fraction of what it will cost to purchase and deploy the proper IBM hardware and hire the reams of IBM consultants to set everything up and customize System S.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Wolfram Alpha - Maybe Overloaded?

Entering "IBM June Calls" into Wolfram Alpha freezes my Firefox browser.

Their system must be truly overloaded beyond their wildest expectations.

Update: I tried the "IBM June Calls" in IE8, and it did not hang the browser. Alpha did not completely understand what the query meant. I then typed in "IBM June 100 calls", and at least Alpha understood that I was talking about an option. I further refined the query by entering "IBM June 2009 calls".

Alpha tried to bring up a graph of the Option Payoff Function, with the price of the payout on the Y axis and the underlying price on the X axis, but the graph contained no data.

Looks like Alpha needs to be trained a bit more ....




©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Experiment with Coral8 and T4

Scott was a big believer in a tool called CodeSmith to automatically generate code. In our group, he used it to take a Coral8 schema and generate serializers, deserializers, and something that we call "ServiceAgents". The problem was that Scott had the only CodeSmith license, so only Scott's machine was able to do the code generation. A few months ago, I had stumbled upon the T4 tool in a blog post, and I had asked Scott to switch over from CodeSmith to T4. Now, the code generation is done by a free tool that Microsoft includes in Visual Studio, so now everyone can experiment with automatic code generation.

A few weeks ago, I found an article in Visual Studio magazine about T4, and then I saw that Chris Donnan was experimenting with T4.

So, I decided to spend an hour last weekend and code up my own T4 Coral8 class generator. I did not bother to read the T4 manual, but I was impressed with what I could rig up in a very short time. This generates a class for every Coral8 CCS file in a certain directory.

This isn't world-class code .... I generated nullable types and I did not bother to generate null-checking code on the nullable fields. But, feel free to modify it.

I will certainly use T4 to generate classes and serializers/derserializers for Orinoco.

<#@ template language="C#" #>
<#@ assembly name="System.Xml" #>
<#@ import namespace="System.IO" #>
<#@ import namespace="System.Xml" #>
<#@ import namespace="System.Collections.Generic" #>


using System;
using Coral8.Transport;
using Coral8.Value;

<# 
    DirectoryInfo dir = new DirectoryInfo(@"v:\dev\CEP\C8Projects\Schemas");
    foreach (FileInfo file in dir.GetFiles("*.ccs"))
    {
        string sSchemaFile = file.Name.Replace(".ccs", "").Replace("-", "");
        List<Info> colInfo = new List<Info>();
        using (StreamReader reader = new StreamReader(file.FullName))
        {
            XmlDocument xmldoc = new XmlDocument();
            xmldoc.Load(reader);
            XmlNodeList columns = xmldoc.GetElementsByTagName("Column");
            foreach (XmlNode node in columns)
            {
                string colName = node.Attributes["Name"].Value;
                string type = node.Attributes["xsi:type"].Value;
                string netType = "unknown";
                string c8FieldType = "unknown";
                
                if (type.StartsWith("String"))
                {
                    netType = "string";
                    c8FieldType = "StringFieldValue";
                }
                else if (type.StartsWith("Int"))
                {
                    netType = "int?";
                    c8FieldType = "IntegerFieldValue";
                }
                else if (type.StartsWith("Long"))
                {
                    netType = "long?";
                    c8FieldType = "LongFieldValue";
                }    
                else if (type.StartsWith("Time"))
                {
                    netType = "DateTime?";
                    c8FieldType = "TimeFieldValue";
                }    
                else if (type.StartsWith("Float"))
                {
                    netType = "double?";
                    c8FieldType = "FloatFieldValue";
                }    
                else if (type.StartsWith("Bool"))
                {
                    netType = "bool?";
                    c8FieldType = "BooleanFieldValue";
                }    
                else if (type.StartsWith("Xml"))
                {
                    netType = "string";
                    c8FieldType = "XmlFieldValue";
                }
                
                colInfo.Add(new Info(colName, netType, c8FieldType));
            } // end foreach node
        } // end using
#>

#region <#= sSchemaFile #> Serializer
public class <#= sSchemaFile #>
{
<# 
        foreach (Info info in colInfo)
        {
#>
    public <#= info.NetType #> <#= info.Name #>;
<#
        }
#>        
    public void ToObject(Tuple tuple)
    {
<# 
        foreach (Info info in colInfo)
        {
            if (info.C8FieldValue == "TimeFieldValue")
            {
#>        
        this.<#= info.Name #> = new DateTime(((<#= info.C8FieldValue #>) tuple.GetFieldValue("<#= info.Name #>")).Value.Value);
<#
            }
            else
            {
#>        
        this.<#= info.Name #> = ((<#= info.C8FieldValue #>) tuple.GetFieldValue("<#= info.Name #>")).Value;
<#
            }
        }
#>        
    }
    
    public Tuple ToTuple()
    {
        Tuple tuple = new Tuple();
<# 
        foreach (Info info in colInfo)
        {
            if (info.C8FieldValue == "TimeFieldValue")
            {
#>        
        tuple.SetFieldValue("<#= info.Name #>", new <#= info.C8FieldValue #>(this.<#= info.Name #>.Value.Ticks));
<#
            }
            else if (info.C8FieldValue == "StringFieldValue" || info.C8FieldValue == "XmlFieldValue")
            {
#>
        tuple.SetFieldValue("<#= info.Name #>", new <#= info.C8FieldValue #>(this.<#= info.Name #>));
<#            
            }
            else
            {
#>        
        tuple.SetFieldValue("<#= info.Name #>", new <#= info.C8FieldValue #>(this.<#= info.Name #>.Value));
<#
            }
        }
#>

        return tuple;
    }
}
#endregion
<#
    }
#>
<#+
    public class Info
    {
        public string Name;
        public string NetType;
        public string C8FieldValue;
        
        public Info(string name, string type, string val)
        {
            this.Name = name;
            this.NetType = type;
            this.C8FieldValue = val;
        }
    }
#>

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.