A First Experiment with the RX Framework

A first experiment with the RX framework. This generates 5 stock prices for 4 different stocks.

using System;
using System.Collections.Generic;
using System.Linq;
 
namespace RXStockGenerator
{
    class TestRX
    {
        public TestRX()
        {
            using (new StockPriceGenerator().Subscribe(q => Console.WriteLine("Quote: " + q)))
            {
                Console.WriteLine("Press any key to unsubscribe");
                Console.ReadKey();
            }
        }
    }
 
    class StockPriceGenerator : IObservable<StockQuote>
    {
        private readonly Random m_random;
 
        public StockPriceGenerator()
        {
            this.m_random = new Random();
        }
 
        public IDisposable Subscribe(IObserver<StockQuote> observer)
        {
            List<IObservable<StockQuote>> generators = new List<IObservable<StockQuote>>
            {
                this.Generate("AAPL", 200.00, 100),
                this.Generate("IBM", 120.00, 150),
                this.Generate("MSFT", 30.00, 125),
                this.Generate("GE", 15.00, 175),
            };
 
            return generators.Merge().Subscribe(observer);
        }
 
        public IObservable<StockQuote> Generate(string symbol, double startPrice, int delay)
        {
            return Observable.Generate(
                0,
                i => i < 5,
                _ => new StockQuote(symbol, startPrice + this.m_random.NextDouble()),
                _ => TimeSpan.FromMilliseconds(delay),
                i => i + 1);
        }
    }
 
    class StockQuote
    {
        public string Symbol { get; set; }
        public double Price { get; set; }
 
        public StockQuote(string symbol, double price)
        {
            this.Symbol = symbol;
            this.Price = price;
        }
 
        public override string ToString()
        {
            return string.Format("{0}/{1:##.#0}", this.Symbol, this.Price);
        }
    }
}

LinkedIn Groups Created By Headhunters

I have noticed that Headhunters who operate in a particular space in Capital Markets or have a job opening in a certain area will create LinkedIn groups for the sole purpose of identifying candidates. I never join groups that have been created by a headhunter unless they are personally recommended to me.

For me, LinkedIn is becoming a bit like the Wild West, and its usefulness will be diluted by recruiters who misuse the service.

Thoughts?


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Anything Happening in the World of CEP?

Since starting my new job, I have not heard very much about the world of Complex Event Processing. Has the buzz disappeared?


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

New York Word - Advertisement

I was speaking to a very old friend and colleague from 1984, when I worked at Computer Associates. His start-up company, Scientia, was bought out by Sorcim, the makers of SuperCalc. In turn, Sorcim was gobbled up by CA.

Out of curiosity, I Googled "Scientia", and I found out that Google has archived past issues of InfoWorld magazine.

I came across this little ditty .....




©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Some Reading and Thoughts

Smart Guys on Wall Street from the New York Times

Rico Mariani looks back on his personal history of Visual Studio while working at Microsoft.

Matt Davey is having some fun with StreamInsight (aka Orinoco). When I was at the Microsoft Design Review last January, I was jumping up and down about some of the interesting design decisions that the Orinoco team make concerning their adapter model. Hopefully, with more people chiming in, the StreamInsight guys will refactor their model while they are in CTP mode. Or, at least, before I have to start using it again.

Meanwhile, I am down to less than 2 weeks left before my Gardening Leave is over. It's great to get a paycheck while I am sitting at home, but I REALLY need to get back to work. I have been occupied with my other blog that is devoted to Magma, the French band that I have been addicted to for the past 35 years. If any of you Capital Markets guys are fans of progressive music, I would love to hear about it.

Rodrick Brown just left the Munis Tech area of my former firm and has struck out on his own with his own shop, Orvid Technologies.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Take a look at this Tibco/C# Consultant Solicitation

This evening, I received the following message from a recruiter who must have me in his database:

Hi Marc,

C# Tibco contract, 40-60/hr, 12 months in NYC. Would you be interested in this?

Thanks and have a nice evening,

Mr. XXX YYYY
Managing Partner
(Contracting Company)



Even though I am not in the market, I paused to consider this solicitation, and I found this rate to be somewhat insulting and way below what the market would bear. As a hiring manager, I would be highly suspicious of any Tibco consultant who would be getting this rate.

One nice thing that my former employer did was to make sure that the rate that an approved vendor (ie: body shop) pays to a consultant is no less that a certain percentage of the billing rate. For example, if a body shop was billing me $1000 a day for a consultant on a 1099, then we insured that the consultant would get no less than 85% of that billing rate (usually higher).

This would be more than enough profit for the vendor, and we would know that our consultant was not getting taken to the cleaners. A happy consultant is a productive consultant.

By the way, I would consider $100 an hour to be a fair rate for a really good Tibco/C# consultant. Why the low rate for the job that this recruiter is offering? It's not the days of 2002-2004, where great developers were getting $35/hr. The consulting market is very strong right now, and the big companies (ie: the users of things like Tibco) are holding their consulting rates steady.


Update - The recruiter contacted me and mentioned that this position was for a JUNIOR Tibco/C# consultant, hence the lower rate. I feel that Tibco is tricky enough (especially for lower latency apps) where I would only be comfortable with a senior-level Tibco consultant. However, as always, this is my opinion. Your mileage may vary.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Review of the Rock In Opposition Music Festival

For those wondering what I am doing on my gardening leave, I am putting some effort into my other blog. Just returned from 5 days in Europe. Now I have to concentrate on reworking the lighting in my house.

Thanks to all of the people who have written to me since I announced my resignation.

Over the next few days, I actually might write a little blog post here on some of the lessons that were learned in running the Complex Event Processing group at Citi. I have some pretty strong opinions about the state of commercial CEP technology, but would you expect anything else????

Meanwhile, I am so stoked to be joining my new company. I kept myself busy for a little while with the constant twiddling and refactoring of my Facebook Notifier. My son tells me that I can make a fortune by selling it, as it seems to be something that is badly needed by PC users. But, with my new job, I will not have any time to enhance it and support it. I ran a software company for 10 years, and I don't know if I will ever be releasing another piece of commercial software again. Just too much hassle right now.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Facebook Notifier

OK, I've become hopelessly addicted to Facebook ... I admit it.

Last night, with the rain pouring outside and the US Open tennis matches postponed, I decided to write something that I've wanted for a long time. An application that sits in the Windows System Tray and notifies me when I get a new Facebook notification. By "notification", I am referring to the red balloon with the number '4':

By having this, I don't have to constantly surf over to the Facebook homepage in order to check my activity.

Luckily, there is a .NET Toolkit for Facebook here. Using this toolkit, it only took me about two hours from start to finish to complete my notifier. I spent another hour cleaning up the code and parameterizing some of the hard-coded things like my user ID, my application's secret codes, and the stuff around the playing of the notification sound.

I found that I had to resort to FQL (which is the SQL-like query facility that Facebook provides) to properly retrieve the state of my Facebook notifications. The function that the Facebook Toolkit gave me always told me that I had no notifications.

I also added the ability to pump in my own notifications for testing purposes.

This application is just for me, and I do not think that I will be releasing it. If I really wanted to do something serious, I would create a general-purpose notifier toolkit that is able to work over Facebook, GMail, Yahoo Mail, etc. Each of these services would have its own "input adapter" as part of the notification framework. And, I would probably integrate this with some sort of CEP engine so that alerts could be sent to a Facebook account.

Maybe one day in the future .....



©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Update on the Job Market

(As I was in the middle of answering a question from someone during my hand-off, my Blackberry reset itself. When it came back on, I found my account terminated from the Citigroup Blackberry server. Oh well ... let them figure out the answers for themselves!)

Some people have asked me what I think of the job market right now in Capital Markets IT.

From my vantage point, the job market for good IT people is exploding right now. There are a few reasons for this:

1) Traders who were displaced from companies like Lehman, Bear, and Merrill are finding themselves in demand, and they are starting to land at various Investment Banks and Hedge Funds. As these people get settled in their new positions, they are asking for systems that are similar to what they had before and what they used to generate profits.

2) Good IT people are leaving the existing companies in droves and going to the non-TARP (ie: non-US) banks. For example, the entire CEP team from Merrill left to go to Deutsche Bank. In particular, Deutsche has been hiring like mad. Hedge Funds who had a bad year last year have recovered, and along with that recovery, they are starting to fill some of the positions that they had eliminated.

3) Upper-level IT management are changing firms, and bringing along their best people with them. This leaves a void at their old firms.

4) More demand for risk management, WPF-based GUIs, equity derivatives.

What is also interesting is that companies are starting to pay up. Even the companies who have been known as notoriously low payers realize that they need to compete, and they are paying above-market rates for new talent, much to the chagrin of the legacy people at the firms.

Is the market completely back to normal? I don't think so. But if you are a really good IT person with some Capital Markets experience, you should have problem landing a position right now.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Leaving Citi - Part 2

I have been overwhelmed by the amount of good wishes and calls that I have received since I announced my resignation.

I am in a period right now that is affectionately known as "Gardening Leave". In most companies, you give a two-week notice when you resign, and it is up to the management as to whether or not you serve out that full two weeks. In most companies on Wall Street (and especially in The City), the gardening leave time is more extensive. I have 50 calendar days in which I must sit in a kind of limbo before I join my new company. People at higher levels have longer periods to sit out. We all get paid, and theoretically, we have to be "on call". But, in reality, the old management would prefer if we don't speak to anyone from the old firm for the obvious reasons. And, I will certainly expect that from my firm.

So, tomorrow, I go to HR, hand in my Blackberry, my corporate credit card, and my ID card, and then I go out for leaving-drinks. And then I bid adieu to Citigroup.

I have been happy to give people a glimpse into what it is like working in Capital Markets IT. As Jeff Wooten from Coraleri says, my feedback on this blog has (hopefully) resulted in some better products out there. I hope that someone else will be able to create a "bully pulpit" on the blogosphere that will keep some of the vendors honest. Even the vendors that I ended up not using have become important allies for me, and who knows what may happen in the future ....

The blog has been going since the fall of 2005. I will try to find something innocuous to blog about from time to time. But, as I said in my previous posting, my professional blogging will undergo a temporary hiatus. When I joined Citigroup in 2006, the hiring manager was a reader of my blog, so I was sort of grandfathered in as far as the blogging goes. However, I enjoy no such luck in my new company, and I have to take my new role very seriously.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Leaving Citigroup

You may have noticed that this blog has been very quiet lately. The reason is because I have resigned from Citigroup (this is most likely the first and only time that I will mention my employer by name on this blog). I will be joining one of the premier names in Capital Markets, helping them bootstrap new businesses, build technology infrastructure, and building great teams.

I am grateful for the opportunities that the people at Citigroup have afforded me over the past three years.

The new company that I am joining is not very permissive of blogging. As a result, my blogging here will be cut back dramatically. I don't want to give anyone any hints about the technologies that we are using or the technologies that we are considering. However, my pursuit of new and interesting technologies to solve business problems will continue.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Gardening

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Orinoco vs Reactive Framework

I have been doing a bit of playing around with the new Reactive Framework (Rx) lately. And, I like what I see so far.

The first thing that came to mind was how the Orinoco team was going to resolve the fact that another LINQ-based streaming API was coming out of a "competitive" group in Microsoft.

A little birdie has told me that recently, Microsoft merged several of the competitive division, and that Orinoco product was going to use Rx as the LINQ-based surface API.

If this is true, will this mean that we all have to wait until .NET 4.0 in order to use Orinoco? Or, will there be a .NET 3.5 SP2 that contains the necessary Systems DLLs? If .NET 4.0 is required, it would be an interesting situation for a lot of the very large customers (like us) who are quite a ways away from even looking at .NET 4.0. Not to mention the certification process that most companies have to go through before adopting a major .NET release.

By the way ... Too bad that I had to hear this news from external sources, as the Orinoco team has gone very quiet lately. Let's just hope that it's because of the summer vacation, or because their developers are heads down in code. And, like Matt Davey pondered ... How can they be releasing CTP 2 when CTP 1 has not even gone out yet?

(Note : All references to Orinoco should be changed mentally to StreamInsight. Too bad, because I like the name Orinoco much better.)

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

New Magma Blog

For those very few people who come here to read about Magma, the French Rock band ...

I have just started a new blog just for Magma and Zeuhl music. Please come and visit, and remember to add the new blog to your blog reading software.



©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Drools 5.0 for CEP

The new 5.0 version of Drools seems to have CEP capabilities.

Has anyone here tried it out yet? Do you think that it can handle about 10,000 events per second? Has anyone pumped market data through it?

I am going to start reading up on Drools Fusion and see how it contrasts with the upcoming Microsoft Orinoco. Fusion is here right now. Orinoco .... well .... we'll see.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

High Frequency Trading

I am sitting in my hotel lobby in Santiago de Compostelo, Spain, and trying to put in a half hour of laptop time while my wife and daughter are still sleeping. I am reading all of the hub-bub surrounding HFT and algo trading, all brought on by the arrest of the Goldman Sachs employee. This morning, Paul Wilmott has an editorial in the New York Times.

What is going through my mind right now is the fact that, if HFT was regulated or banned, it would cost the jobs of a good number of very smart people in the Capital Markets industry. It would cause irreparable harm to some of the companies that support low-latency infrastructure. And, it would transform the Capital Markets industry in ways that I could not even imagine.

More thoughts on this later.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

New York Times HFT Article

New York Times article and opinions on High Frequency Trading here. It also has a good explanation of what is called "gaming" in algo trading.

(Sorry for the lack of posts ... I've been traveling in Europe for the past week and will return to New York in the first week in August).

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Nice Wall Street & Tech Article on Citi

Nice article here.

Very interesting quote on Page 2, second paragraph.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Goldman's "Stolen" Program Trading code

(I am sitting here with a HP dv9700t laptop with two broken hinges, trying to cross my knees in such a way that the screen will be propped up in a way that I can see it. I am going to Spain for the last two weeks in July, so right before I go, I will send the laptop off for repair. I estimate the repair cost to be about $300. I cannot believe that HP makes a complete line of laptops whose hinges crack after a few months, and are so arrogant and uncompromising when it comes to addressing this issue.)


I spent an hour on Monday morning reading the great post on ZeroHedge about the Goldman developer who zipped up the code to a High Frequency Trading (HFT) system. I am not going to rehash what has been written in the various news outlets. But, it's always interesting to read Lars Toomre's opinions on the subject. (Update: here's another one from Hack The Market)

I am not going to weigh in on my own opinions, as I feel that it's not appropriate to comment on the public issues that one of our competitors is going through. But, here are some things that I have seen in my time on Wall Street:

1) Many developers seem to come to a new firm with some old code in hand. I have interviewed candidates in the past where, in the middle of an interview, they pull out a stack of printed source code and show it to me as proof of their coding ability.

2) Many coders think of themselves as authors and artists. When they write something, they want to keep a copy for posterity sake. They might not ever want to use it again. They just want a memento of what they have written. There is no malfeasance intended.

3) Companies are cracking down. Look at what went on with Morgan Stanley a few years ago when a consultant sent client data to his personal email account. You always have to assume that every keystroke, every out-going email, every website you browse to, is being monitored by your company.

In my opinion, you can always write the same application again, but even better. And, if you bring your old source to a new company, you are putting your new company in a precarious situation. And, if you decide to work on any other projects in your spare time, make sure that you check your employment contract very carefully ... some companies say that anything you write during your term of employment belongs to them, even if you write it in your off-hours at home.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

SIFMA 2009

Not much to write about SIFMA this year. I was warned by a friend that it was a ghost-town on the first day. I went to SIFMA on the second day, which is usually the most attended day, and unfortunately, my friend was correct.

In 2008, SIFMA has 350 vendors, but this year, they had 201 vendors, which is a decrease of about 42%. The side corridors used to be jammed with small vendors, but they were totally barren this year. Many large vendors were conspicuously absent, like Sungard and Reuters and Intel. Cust0mer traffic was way down, and in most booths, vendors were conversing with other vendors.

Sorry folks ... not much to write about here .....

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Cloud Computing

Matt seems to have read my mind. I have just started a dive into Cloud Computing, and today, my questions to various CEP vendors were "Can I use you in the Cloud? Have you deployed to Amazon EC2 yet?"

There are various reasons why, at this time, I feel that we should start looking at the Cloud. I will be posting more on this in the future. But, for apps that are not latency sensitive, and if we can obfuscate our data and get the blessings of our Compliance Department (not an easy thing to do), I would like to think about off-loading some of our CEP processing (as well as some other trading functions) to a public Cloud.

I will be catching up with the Microsoft Orinoco team soon. And, it will be interesting to hear about their plans for Azure.

Meanwhile, this looks like a pretty good blog. Especially this post.....


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Karen's Cancer is Gone

Congratulations to Karen Shanahan for beating the odds.

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

DSLs for Trading using Microsoft Oslo

Video on using Oslo and DSL's for testing trading systems.

(Thanks to Terry for the link)

I would love to explore using a DSL to have our traders register lightweight queries with our CEP system.

Notable about this use case is their use of Microsoft's CCR.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Some Small Ideas on Google Wave

I went through the painful 1 hour and 20 minute bad-joke-athon that was the Google Wave video on YouTube. Nevertheless, I see some immediate things that I would love to deliver to our traders.

The obvious thing would be the ability to drag and drop some of the interesting graphs that we produce from our real-time WPF-based dashboard onto a Wave, have the traders discuss the graphs in real-time, and embed the graphs and discussions into a blog for dissemination to other departments. (Do we even need a blog anymore if everyone has Wave?)

We might also have a Wave Robot tied into our CEP engine or our delivery mechanism so that, upon seeing certain interesting events, the Robot automatically publishes the data into the Wave. This means that we would need to write a Wave Output Adapter for Coraleri or Orinoco.

The final thing I would hope that we could do with Wave is "bookmark" a certain point in time. When traders interact with this "temporal bookmark", the traders can use the "playback" feature of Wave to replay the events leading up to this bookmark. The playback feature would interact with our CEP engine or our Query Facade in order to replay the state.

Our business sponsor is someone who is known for thinking outside the box, so I am hoping that he comes up with similar ideas of how CEP can be married with Google Wave.

One thing that I do know ..... Microsoft has its work cut out for them if they ever want to compete with Google Wave and some of the other innovative products that are coming out of Google. I am hoping that the thinkers behind Orinoco are thinking the same thing, and will start thinking outside the box too!


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Two New User Groups on LinkedIn

I just created two new User Groups on LinkedIn.

1) Microsoft Complex Event Processing
This is for discussions surrounding the new Microsoft CEP (aka Orinoco) product.

2) Aleri
This is for the newly combined CorAleri product. We can discuss the legacy Coral8 and Aleri products, and discuss the parts of the CorAleri roadmap that are not under NDA.

If these groups interest you, please feel free to join.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

The First First Derivatives/KDB User's Group

First Derivatives held it's first New York KDB User's Group meeting at LiquidNet in new York city on May 28th, 2009. About 100 people showed up, and it seemed that there was no more than one degree of separation between everyone in the room. Jacob Loveless did a great presentation on the things that he is doing at BGC/Cantor, and Dan Nugent showed a glimpse of a KDB unit testing framework.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

One Trader's Opinion about Algo Trading and CEP

I was recently talking to a very technical algo trader from another company about CEP. I asked him why he did not use a product like Apama or Truviso for his algo trading needs. His reply was that it did not give him a competitive advantage if he used the same technology as other competitors used. He has the budget and the resources to build out a custom platform, one which he thought would give him a distinct advantage.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

A Brief Look at LINQ and Microsoft Orinoco

This was taken from the Microsoft whitepaper on Orinoco. Thanks for the Orinoco team for providing this to me.

Let's assume that you have a number of power meters that you need to monitor. The "InputStream" contains a stream of tuples where each tuple contains a power reading from an individual meter. The reading happens to come in as the wattage times 10.

We can capture each reading a project a new stream that has the normalized wattage. We can also create a sliding 3-second window of meter readings that we can do calculations against.

var realValueStream =

from e in InputStream

select new MeterWattage((double)e.Consumption / 10);

var windowedStream =

from e in realValueStream.SlidingWindow(TimeSpan.FromSeconds(3))

select new MeterWattage(e.wattage);

We can then create a stream that has the average power reading over a sliding 3 second period.

var averagedStream =

from e in windowedStream

select new OneMeterAverage(CEPAggregates.Average(e.wattage));

(I think that this next query may be incorrect, as the ID field was not captured in the windowedStream, but I will continue with the example anyway).

If you want to get the 3-second sliding average for each individual meter, you can do the following 2 queries:

var meterStreamGroups =

from e in windowedStream

group e by e.id;

var groupedAveragedStream =

from eb in meterStreamGroups

from e in eb

select new MeterAverage(eb.Key, CEPAggregates.Avg(e.wattage));

Now, we want to compute the ratio of consumption for each meter over the 3-second window.

var sumStream =

from e in groupedAveragedStream

select new MeterSum(CEPAggregates.Sum(e.wattage));

var ratioStream =

from e1 in groupedAveragedStream

join e2 in sumStream on true equals true

select new

MeterRatio(e1.Id, e1.Wattage, e1.Wattage / e2.TotalWattage);

Finally, we just want to capture the ratio of power consumption from meter #2.

var filteredStream =

from e in ratioStream

where e.id == 2

select e;

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

A Strange Dream

I had a very strange dream last night. I dreamed that I left my current job and went to work for Google. When I got there, I found myself in a big loft, surrounded by 20-something year old geniuses. I asked them where my desk and computer was. They looked at me with puzzlement. I asked then where all of the computer monitors were, and they didn't know what a computer monitor was. I started describing how, on the trading floor where I worked, the traders has 6 or 8 monitors stacked on their desks. Then everyone started to laugh at me.

In seemed that everyone at Google worked on laptops. They had no conception of what a desktop system was, and the fact that a trader had multiple monitors seemed absurd. The Google-ites didn't even have desks. They just sat on the floor wherever they wanted with their laptops.

I think that the dream was inspired by an article that a friend just sent me.

And, it got me wondering if we at the large investment banks could be doing things in a better way .....


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Orinoco Pricing vs IBM System S ???

The New York Times had a nice article on IBM's System S. You know that it's been a good week for CEP when the Times starts covering it.

IBM has really been rolling out the PR machine on this one. They have been going around to the various investment banks and talking a bit about the pilot that they did with TD Securities, which looks very much like an options market making app.

Jerome wants to know what I think of IBM System S. I hope to have a future blog post devoted to System S and their SPADE language.

Another reader pointed to a quote at the end of the Times article, and wanted to know why I thought that Microsoft should be "giving out Orinoco for free". The quote from the Times article is:

"Buying such an advantage from I.B.M. has its price. The company will charge at least hundreds of thousands of dollars for the software, Mr. Mills said."

This should not be such a shocking quote. Most CEP vendors charge several hundred thousand dollars for their software. But, IBM has been using System S somewhat as a showcase for their Blue Gene supercomputer, and IBM has not made a secret of the fact that System S works great on the grid computing infrastructure offered by Blue Gene. And, if you want to marry System S with Blue Gene, you will find yourself few million dollars lighter in the wallet. But, then again, a trading firm can make that money up with one good day of trading.

I just cannot picture Microsoft moving into the same pricing model as IBM yet. Orinoco has just started to explore the area of HPC. Orinoco hopes to be able to "process 100,000 events per second" while IBM is claiming to process several million per second. (Of course, what do these people consider to be an event? And, what do they mean by "process an event"? Does this mean complex calculations, outer joins, or does it mean just putting an event into the memory space of the CEP engine? We need Stac!!!)

Orinoco can be best positioned as an add-on to SQL Server, something that Microsoft has to do in order to compete with the new CEP offerings by Sybase and Oracle. Microsoft should be capturing the mindset of every .NET developer, working its way into a company starting at the ground level and permeating the organization upwards.

I can imagine that IBM does not want System S to appeal to the everyday developer ... just to the ones that work for companies and government agencies that have multi-million dollar budgets for real-time analysis. IBM says "If you have to ask about the price, then System S is probably too expensive for you." The actual cost of System S will be a fraction of what it will cost to purchase and deploy the proper IBM hardware and hire the reams of IBM consultants to set everything up and customize System S.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Wolfram Alpha - Maybe Overloaded?

Entering "IBM June Calls" into Wolfram Alpha freezes my Firefox browser.

Their system must be truly overloaded beyond their wildest expectations.

Update: I tried the "IBM June Calls" in IE8, and it did not hang the browser. Alpha did not completely understand what the query meant. I then typed in "IBM June 100 calls", and at least Alpha understood that I was talking about an option. I further refined the query by entering "IBM June 2009 calls".

Alpha tried to bring up a graph of the Option Payoff Function, with the price of the payout on the Y axis and the underlying price on the X axis, but the graph contained no data.

Looks like Alpha needs to be trained a bit more ....




©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Experiment with Coral8 and T4

Scott was a big believer in a tool called CodeSmith to automatically generate code. In our group, he used it to take a Coral8 schema and generate serializers, deserializers, and something that we call "ServiceAgents". The problem was that Scott had the only CodeSmith license, so only Scott's machine was able to do the code generation. A few months ago, I had stumbled upon the T4 tool in a blog post, and I had asked Scott to switch over from CodeSmith to T4. Now, the code generation is done by a free tool that Microsoft includes in Visual Studio, so now everyone can experiment with automatic code generation.

A few weeks ago, I found an article in Visual Studio magazine about T4, and then I saw that Chris Donnan was experimenting with T4.

So, I decided to spend an hour last weekend and code up my own T4 Coral8 class generator. I did not bother to read the T4 manual, but I was impressed with what I could rig up in a very short time. This generates a class for every Coral8 CCS file in a certain directory.

This isn't world-class code .... I generated nullable types and I did not bother to generate null-checking code on the nullable fields. But, feel free to modify it.

I will certainly use T4 to generate classes and serializers/derserializers for Orinoco.

<#@ template language="C#" #>
<#@ assembly name="System.Xml" #>
<#@ import namespace="System.IO" #>
<#@ import namespace="System.Xml" #>
<#@ import namespace="System.Collections.Generic" #>


using System;
using Coral8.Transport;
using Coral8.Value;

<# 
    DirectoryInfo dir = new DirectoryInfo(@"v:\dev\CEP\C8Projects\Schemas");
    foreach (FileInfo file in dir.GetFiles("*.ccs"))
    {
        string sSchemaFile = file.Name.Replace(".ccs", "").Replace("-", "");
        List<Info> colInfo = new List<Info>();
        using (StreamReader reader = new StreamReader(file.FullName))
        {
            XmlDocument xmldoc = new XmlDocument();
            xmldoc.Load(reader);
            XmlNodeList columns = xmldoc.GetElementsByTagName("Column");
            foreach (XmlNode node in columns)
            {
                string colName = node.Attributes["Name"].Value;
                string type = node.Attributes["xsi:type"].Value;
                string netType = "unknown";
                string c8FieldType = "unknown";
                
                if (type.StartsWith("String"))
                {
                    netType = "string";
                    c8FieldType = "StringFieldValue";
                }
                else if (type.StartsWith("Int"))
                {
                    netType = "int?";
                    c8FieldType = "IntegerFieldValue";
                }
                else if (type.StartsWith("Long"))
                {
                    netType = "long?";
                    c8FieldType = "LongFieldValue";
                }    
                else if (type.StartsWith("Time"))
                {
                    netType = "DateTime?";
                    c8FieldType = "TimeFieldValue";
                }    
                else if (type.StartsWith("Float"))
                {
                    netType = "double?";
                    c8FieldType = "FloatFieldValue";
                }    
                else if (type.StartsWith("Bool"))
                {
                    netType = "bool?";
                    c8FieldType = "BooleanFieldValue";
                }    
                else if (type.StartsWith("Xml"))
                {
                    netType = "string";
                    c8FieldType = "XmlFieldValue";
                }
                
                colInfo.Add(new Info(colName, netType, c8FieldType));
            } // end foreach node
        } // end using
#>

#region <#= sSchemaFile #> Serializer
public class <#= sSchemaFile #>
{
<# 
        foreach (Info info in colInfo)
        {
#>
    public <#= info.NetType #> <#= info.Name #>;
<#
        }
#>        
    public void ToObject(Tuple tuple)
    {
<# 
        foreach (Info info in colInfo)
        {
            if (info.C8FieldValue == "TimeFieldValue")
            {
#>        
        this.<#= info.Name #> = new DateTime(((<#= info.C8FieldValue #>) tuple.GetFieldValue("<#= info.Name #>")).Value.Value);
<#
            }
            else
            {
#>        
        this.<#= info.Name #> = ((<#= info.C8FieldValue #>) tuple.GetFieldValue("<#= info.Name #>")).Value;
<#
            }
        }
#>        
    }
    
    public Tuple ToTuple()
    {
        Tuple tuple = new Tuple();
<# 
        foreach (Info info in colInfo)
        {
            if (info.C8FieldValue == "TimeFieldValue")
            {
#>        
        tuple.SetFieldValue("<#= info.Name #>", new <#= info.C8FieldValue #>(this.<#= info.Name #>.Value.Ticks));
<#
            }
            else if (info.C8FieldValue == "StringFieldValue" || info.C8FieldValue == "XmlFieldValue")
            {
#>
        tuple.SetFieldValue("<#= info.Name #>", new <#= info.C8FieldValue #>(this.<#= info.Name #>));
<#            
            }
            else
            {
#>        
        tuple.SetFieldValue("<#= info.Name #>", new <#= info.C8FieldValue #>(this.<#= info.Name #>.Value));
<#
            }
        }
#>

        return tuple;
    }
}
#endregion
<#
    }
#>
<#+
    public class Info
    {
        public string Name;
        public string NetType;
        public string C8FieldValue;
        
        public Info(string name, string type, string val)
        {
            this.Name = name;
            this.NetType = type;
            this.C8FieldValue = val;
        }
    }
#>

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Wolfram Alpha

Just saw a screencast for Wolfram Alpha. This blows me away. Tons of ideas. How do we make it real-time?

If it can answer questions like "Show me all options whose buy price is under the fair value", then most of us are out of a job.


©2009 Marc Adler - All \Rights Reserved.
All opinions here are personal, and have no relation to my employer.

First Experiences with Java

I thankfully got the chance to do a little bit of coding over the past few weeks, working on an alerting services framework that is an extension of our CEP system.

Because we have a lot of systems that are in Java, I needed to deliver both a C# and a Java SDK. I used to do a tiny bit of Java back in the late 1990's, but I haven't touched it in over ten years.

So, I had a completely finished C# SDK, and I needed to port it over to Java. The C# SDK lets a system publish alerts over Tibco EMS and over WCF using the webHttpBinding. I knew that I was going to have an issue doing the Java interop to WCF. and, I wanted to do this entire effort without reading a lot of documentation.

I started off by downloading Eclipse. I created an empty Java project and started to port my C# code over. Immediately, I started to miss some of the features of C# and Visual Studio that I was used to.

1) Properties. Where are they in Java? Especially the automatic properties of C# 3.0. I don't want to have to write separate get and set methods for every property.

2) Events. That's a big miss for me. I had to end up using the Observable and Observer classes/interfaces. By the way, I used to the names of interfaces starting with the letter "I". It's not obvious in Java what is an interface and what is a class.

3) The ability to easily create Web Service proxy code. This is a no-brainer in Visual Studio. For the Java project, I ended up having to download and use the NetBeans 6.1 IDE (which is a lot slower than Eclipse). NetBeans was a nicer IDE. It understands Eclipse projects. It has a very easy way to create a Web Services client. I found out through a bit of pain and a lot of blogs postings that it is best for interoperability if your WCF service uses the basicHttpBinding instead of the wsHttpBinding or webHttpBinding.

So now, after all of these years, I can say that I have coded in Java. And, I can say that I have a greater appreciation of C# and Visual Studio.





©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

HP Notebook Crack - Part 2

The original HP case worker never called me back during the 24 to 72 hour time period that they promised.

After calling back again and going through two more levels of the "gatekeepers", I was transferred to India and assigned a new case worker, who promptly informed me that HP had not yet determined if my laptop fell into the same category as the other 200 models of HP laptops that had the hinge crack problems.

I ended up ordering a new front bezel and some other spare parts from the HP website. I will try to do the repairs myself.

The hinge crack issue, combined with the fact that I have an unbearably noisy fan (with no easy access), makes me think that I will never purchase an HP Laptop again.

I may go back to Dell after a hiatus of several years. The Dell laptops that I have had have been built like tanks.



©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

2 Papers on Microsoft CEDR

Consistency Sensitive Operators in CEDR

Jonathan Goldstein, Mingsheng Hong, Mohamed Ali, Roger Barga
December 2007

This technical report consists of two concatenated papers. The first is a theoretical paper describing interesting tradeoffs when designing speculative streaming operators. In addition, it describes specific algorithms with specific tradeoffs and the associated complexity analysis for the aggregate operator The second paper describes the other operator algorithms in the overall CEDR context and their associated complexity analyses. In all, the papers describe the necessary operator algorithms for building an efficient, computationally rich, speculative stream processing engine.

Event Correlation and Pattern Detection in CEDR
Roger S. Barga1 and Hillary Caituiro-Monge2
1 Microsoft Research, Microsoft Corporation,
One Microsoft Way, Redmond WA 98052
barga@microsoft.com
2 UC Santa Barbara, Computer Science Department,
Santa Barbara, CA
hcaituiro@cs.ucsb.edu



©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

CorAleri Update

We met with the good folks from CorAleri yesterday in order to get an update on their post-merger roadmap. Some of the presentation was under NDA, but all in all, things seem to be falling in a fairly predictable path.

Here are the highlights:

1) Merger going well. The dev teams from New Jersey and California have drank the mutual KoolAid. Nice to see Jon and Bob sitting next to each other, doing the photo-op. Seriously, its great to see two of the nicest companies join forces.

2) CCL with Splash is the new language.

3) Work is going on towards a new Eclipse-based IDE.

4) There will be a new engine that incorporates the best of both engines. Aleri is very committed to keeping low latency a big priority.

5) Support for debugging and profiling.

6) Most likely will use the Aleri adapters.

7) The Coral8 engine can now use Aleri's LiveOLAP, and the Aleri engine can now use the Coral8 Portal.


Some of this was announced publicly already by CorAleri, some of these are things that were requested by customers. Basically, no surprises ... just a clear roadmap for consolidation and evolution.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Microsoft CEP - Part 2

Over the next few days, I hope to blog a bit about the new Microsoft CEP product, code-named "Orinoco".

To get yourself a bit more familiar with what was said at the announcement, you may want to go over to Richard Seroter's blog for a great post that summarizes the session at TechEd in which the Microsoft CEP product was announced.

I first got wind of Orinoco in the Spring of 2008 during conversations with some members of the SQL Server group at Microsoft. The Orinoco group maintained radio silence until January of this year, when I got an invitation to come out to Redmond for a Design Review Session. I was joined there by several internal and external Microsoft customers, and we spent 2 days going over Orinoco from front to back.

I will blog a bit more about the technology of Orinoco in some future blog posts. But, I would like to say that Orinoco has the potential to radically change the CEP marketplace.

Orinoco will use LINQ as the "surface language". This builds upon the good work of people like Kevin Hoffman, whose CLINQ is used heavily at Liquidnet. The ability to embed CEP operations within a normal C# application is extremely powerful. You can take advantage of the power of the C# language and you can go back to using mechanisms like inheritance and generics in your CEP applications. This approach combines the expressive power of something like Streaming SQL with the flexibility of something like Apama's MonitorScript.

I don't think that Microsoft has defined their final marketing strategy around Orinoco (although Charles mentions that it will be part of SS 2008 R2) . When I met with the Orinoco team in April, they were still gathering feedback from potential customers as to how to market the CEP technology. But, I can imagine that for the immediate future, Orinoco will be free to users of SQL Server. I can also imagine that some part of Orinoco will be free to Visual Studio users (like SQL Server Express) and that an Enterprise version will be licensed like SQL Server. Perhaps a free developer version will come with MSDN? Perhaps it will be a free add-on to SQL Server in the same way that Reporting Services is free. This can be used to drive more SQL Server sales, and put a damper on competitive products like Sybase RAP. It can also be a compelling factor for developers to choose Windows instead of Linux for their CEP servers.

This means that every .Net developer will have access to CEP technology at a price point that will make it extremely attractive.

CEP vendors like CorAleri, Streambase, and Apama will inevitably find themselves under a degree of pricing pressure. When the 2-ton gorilla bursts into the room with a free or low-cost offering, everyone has to sit up and take notice.

Orinoco is still a while away from being as polished as some of the commercial CEP products. But, the prospect of using LINQ, the integration with Microsoft HPC, and the ability to use Visual Studio as the IDE have gotten us pretty excited.

More later....


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.