Alex Baldwin's most memorable bit of acting comes in the great movie, Glengarry Glen Ross. In giving a motivational speech to a crew of "salesmen", Baldwin invokes the basic tenet of sales, "ABC".
A - Always
B - Be
C - Closing
My adaptation of this is:
A - Always
B - Be
C - Coding
Even though I have managed teams from 1 to 30 people, I never gave up coding. People are amazed when I tell them that I still code.
The ability to write code is a skill like plumbing or being an electrician .... someone will always need coding done. No matter what industry you are in ... finance, healthcare, media, entertainment, e-commerce .... someone always needs you to write a new system or to enhance an old one.
No matter where I go with management, there are reasons why I still like to code:
1) It is creative and relaxing. There is no feeling like being "in the zone" while coding up a new idea. It is left-brain activity which calms the right side.
2) You produce an objective piece of work. At year's end, you can always print out your reams of source code and show others that you actually accomplished something. You have the feeling that you have created something concrete, like a piece of art.
3) You need to have something to fall back on if things ever get rough. This is important for us folks who have 1 kids in college and another kid two years away from entering college.
The other day, I was standing by the large windows of our trading floor, talking with the head of our Stat Arb department, looking at all of the sailboats going up and down the Hudson River. The head of Stat Arb still codes as well. So does the head of our Derivatives Analytics department. These are both Managing Directors and they are still hacking away.
The head of Stat Arb and I were talking about the recent financial crisis and the ongoing wave of mergers. We were both saying that, if our company ever gets acquired, or if we acquired another company which would make our positions redundant, we could still go off and make a living as coders. A great Java coder (with a bit of KDB knowledge) like Mr. Stat Arb, who has deep financial domain knowledge, might be able to make $1500 a day ... I am not sure if consulting rates have collapsed recently, but I assume that some financial institution would be able to bring him on for a short-term contract. (Maybe someone can let me know what the consulting market is doing these days.) I still see plenty of jobs out there for C# hackers in all sorts of industries ... not only UI jobs, but C# server-side jobs as well.
It rained this entire weekend, as Tropical Storm Kyle passed off the East Coast of the United States. My wife was sleeping, as she was recovering from jet lag, and my daughter was busy doing homework and studying for various tests. So, I decided to do something very foolish, and try to teach myself some WPF (Windows Presentation Foundation).
Our Complex Event Processing system uses WPF for the GUI, and so far, I have left my UI developers alone while they coded the GUI. However, I feel that I need to be able to understand their XAML code at some level. So, I opened Visual Studio 2008 and started coding up some simple screens, experimenting mostly with XAML.
I felt a bit silly in trying to learn a new UI paradigm as my management responsibilities increase at work. However, as I started mucking around with XAML, I started to feel myself get "into the zone" again.
Visual Studio 2008 (without SP1) seems to be a pretty crappy environment to develop WPF apps in. I don't use any other tools, like Expression and Blend. This is a really vanilla WPF development environment. (I hear that SP1 is supposed to enhance the WPF design experience, but I am a bit afraid to install it after reading some of the blogs that detailed some horror stores that occured post installation.)
But, give me a few weeks to explore more WPF. I want to get to the point where I can make some simple changes and do a bit of debugging of our GUI if I have to.
Always Be Coding.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Sunday, September 28, 2008
Sunday, September 21, 2008
Colin is Building a CEP App
Colin Clark, is starting to blog about OMICRON 5000, a CEP-based order router/liquidity finder that he wants to build. I am not sure if Colin is going to just sketch the system out on his blog, or if he is going to get down and dirty and start coding this thing. But, I will follow his upcoming exploits very closely, as Colin implies that he will keep us informed during every step of the development process.
I don't know what technology platform Colin is going to choose ... Windows or Linux, C# or C++ or Java. If Colin is very public about this new effort, then I might expect some vendors to throw him some complimentary software, as it would be a tremendous marketing coup for the CEP vendor who eventually is chosen.
Colin's evaluation will take place a year after we did our initial evaluation, and I am interested to see how he ends up choosing the CEP engine. Colin is a company of one. *IF* Colin can get his hands on some CEP software, he will not have the same luxury that we did --- 4 months devoted to research --- to make a prolonged evaluation.
Let's see what a company of one can do to get bootstrapped into the world of CEP.
How will Colin get his hands on the CEP software to evaluate? Coral8 is free to download, and it is very easy to get started with their developer version. You can download Aleri, but from what I remember, it comes with a 30-day license. Progress Apama would not let me download their software without requiring a dog-and-pony show. Streambase gave us their software, so I don't know anything about their evaluation process. Esper is totally free (at least, their non-HA version is). I wonder if Paul Vincent will throw Colin a copy of Business Events to play with.
Once Colin chooses a CEP engine, how much will he have to spend to get a production license for the software? Esper is free. The others require a minimum outlay of $60,000, not including yearly maintenance fees. You can play with your downloaded version of Coral8 all you want, but you cannot deploy it into a production app without purchasing a license.
Colin says that OMICRON needs a database. I assume that MySQL is still free, even after the Sun purchase. You need to find a CEP engine that has MySQL adapters.
OMICRON needs a FIX engine. The only free one that I know of is QuickFix. Will Colin build an in-process or out-of-process FIX adapter for his CEP engine?
OMICRON needs market data. OpenTick is free to use, and all you need to do is pay the exchange fees. Worse comes to worse, you can write a service that scrapes Yahoo Finance, but I am sure that will not satisfy Colin (although it is useful for early-stage development).
Finally, Colin needs a GUI. If Colin uses C#/.NET, there are plenty of controls that come with Visual Studio, and there are tons of others that you can find on sites like CodeProject. I am assuming that all OMICRON needs is a simple grid, with perhaps some real-time updating of the cells.
It seems that a single developer can get away with totally free, Open Source software if he was going to choose Esper. There might be CEP engines systems from universities that you can use.
I will eagerly follow Colin's blog and kibbitz on his every move....
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
I don't know what technology platform Colin is going to choose ... Windows or Linux, C# or C++ or Java. If Colin is very public about this new effort, then I might expect some vendors to throw him some complimentary software, as it would be a tremendous marketing coup for the CEP vendor who eventually is chosen.
Colin's evaluation will take place a year after we did our initial evaluation, and I am interested to see how he ends up choosing the CEP engine. Colin is a company of one. *IF* Colin can get his hands on some CEP software, he will not have the same luxury that we did --- 4 months devoted to research --- to make a prolonged evaluation.
Let's see what a company of one can do to get bootstrapped into the world of CEP.
How will Colin get his hands on the CEP software to evaluate? Coral8 is free to download, and it is very easy to get started with their developer version. You can download Aleri, but from what I remember, it comes with a 30-day license. Progress Apama would not let me download their software without requiring a dog-and-pony show. Streambase gave us their software, so I don't know anything about their evaluation process. Esper is totally free (at least, their non-HA version is). I wonder if Paul Vincent will throw Colin a copy of Business Events to play with.
Once Colin chooses a CEP engine, how much will he have to spend to get a production license for the software? Esper is free. The others require a minimum outlay of $60,000, not including yearly maintenance fees. You can play with your downloaded version of Coral8 all you want, but you cannot deploy it into a production app without purchasing a license.
Colin says that OMICRON needs a database. I assume that MySQL is still free, even after the Sun purchase. You need to find a CEP engine that has MySQL adapters.
OMICRON needs a FIX engine. The only free one that I know of is QuickFix. Will Colin build an in-process or out-of-process FIX adapter for his CEP engine?
OMICRON needs market data. OpenTick is free to use, and all you need to do is pay the exchange fees. Worse comes to worse, you can write a service that scrapes Yahoo Finance, but I am sure that will not satisfy Colin (although it is useful for early-stage development).
Finally, Colin needs a GUI. If Colin uses C#/.NET, there are plenty of controls that come with Visual Studio, and there are tons of others that you can find on sites like CodeProject. I am assuming that all OMICRON needs is a simple grid, with perhaps some real-time updating of the cells.
It seems that a single developer can get away with totally free, Open Source software if he was going to choose Esper. There might be CEP engines systems from universities that you can use.
I will eagerly follow Colin's blog and kibbitz on his every move....
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
The SQLDependency Object and Entitlements
Our CEP Servers have to run 24x6. When our server starts up, it reads some tables in our SQL Server 2005 database that contains information about users and the alerts that they are entitled to see. We don't want most of our users to be able to see alerts that occur because of order messages, but we do want our users to see alerts that occur from execution reports.
If we had to query the database for the entitlements on every single alert that flows through our system, we would take a tremendous hit in performance. Therefore, we cache the entitlements in memory and cache the result sets.
We want to be able to change our entitlements on-the-fly, while the server is running. We want to add and delete users, change the method of notification for a user (Tibco, SMS, email, messenger, etc), and add new alerts to our system. We could restrict all changes to the underlying entitlements tables so that the changes would have to made made through an Administrative GUI, but because of the environment inside our company, we cannot. Someone might add a user through the Admin GUI, and others might decide to go right into our SQL Server database and add some new rows to our tables.
We need a way to detect when the entitlement tables have changed within our SQL Server database while our server in running, and have those changes reflected in the operation of our system. We cannot afford to bring our system down, make the changes to the entitlements, and restart our system. We also don't want to wait until the end of the week for the "green zone" period, where we can safely bring down our system, make the changes, and restart the system.
SQL Server 2005 has a class called SqlDependency. This class can be used as an interface between SQL Service Broker and a .NET application to inform the .NET application when a certain query's result set has changed.
Most of the literature on the SqlDependency class revolves around using it in an ASP.NET application. There are also a bunch of things that you need to be aware of, such as restrictions on the syntax of the query that you give to the SqlDependency class. It took a little bit of digging and experimentation for me to get this stuff to work, but after a little while, we found that it did the job nicely.
I have enclosed the source code to a simple EntitlementTableWatcher class that shows you how to use SqlDependency. I have also include some links to URLs that I found useful. If you have any comments about the code, or if you find any bugs, please let me know and I will correct them.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
If we had to query the database for the entitlements on every single alert that flows through our system, we would take a tremendous hit in performance. Therefore, we cache the entitlements in memory and cache the result sets.
We want to be able to change our entitlements on-the-fly, while the server is running. We want to add and delete users, change the method of notification for a user (Tibco, SMS, email, messenger, etc), and add new alerts to our system. We could restrict all changes to the underlying entitlements tables so that the changes would have to made made through an Administrative GUI, but because of the environment inside our company, we cannot. Someone might add a user through the Admin GUI, and others might decide to go right into our SQL Server database and add some new rows to our tables.
We need a way to detect when the entitlement tables have changed within our SQL Server database while our server in running, and have those changes reflected in the operation of our system. We cannot afford to bring our system down, make the changes to the entitlements, and restart our system. We also don't want to wait until the end of the week for the "green zone" period, where we can safely bring down our system, make the changes, and restart the system.
SQL Server 2005 has a class called SqlDependency. This class can be used as an interface between SQL Service Broker and a .NET application to inform the .NET application when a certain query's result set has changed.
Most of the literature on the SqlDependency class revolves around using it in an ASP.NET application. There are also a bunch of things that you need to be aware of, such as restrictions on the syntax of the query that you give to the SqlDependency class. It took a little bit of digging and experimentation for me to get this stuff to work, but after a little while, we found that it did the job nicely.
I have enclosed the source code to a simple EntitlementTableWatcher class that shows you how to use SqlDependency. I have also include some links to URLs that I found useful. If you have any comments about the code, or if you find any bugs, please let me know and I will correct them.
#region Entitlements Table Watcher
internal class EntitlementTableChangedWatcher : DisposableObject
{
// http://msdn.microsoft.com/en-us/library/ms379594.aspx is a good article by Developmentor
private readonly EntitlementModel m_entitlementModel;
private SqlConnection m_connection;
private SqlDependency m_dependency;
private SqlCommand m_command;
private SqlDataReader m_reader;
public EntitlementTableChangedWatcher(EntitlementModel entitlementModel)
{
SqlDependency.Stop(ConnectionString);
SqlDependency.Start(ConnectionString);
this.m_connection = new SqlConnection(ConnectionString);
this.m_connection.Open();
this.m_entitlementModel = entitlementModel;
}
protected override void Free(bool disposedByUser)
{
if (disposedByUser)
{
this.CleanupCommand();
if (this.m_connection != null)
{
this.m_connection.Dispose();
this.m_connection = null;
}
SqlDependency.Stop(ConnectionString);
}
base.Free(disposedByUser);
}
private void CleanupCommand()
{
// In case we entered here from the callback, we need to clean up the last command so that
// we can issue a new command.
if (this.m_reader != null)
{
this.m_reader.Dispose();
this.m_reader = null;
}
if (this.m_command != null)
{
this.m_command.Notification = null;
this.m_command.Dispose();
}
}
public void PollForNotifications()
{
try
{
// In case we entered here from the callback, we need to clean up the last command so that
// we can issue a new command.
this.CleanupCommand();
// Create a new SqlCommand object.
// http://msdn.microsoft.com/en-us/library/aewzkxxh.aspx contains rules for the syntax of the query.
// Make sure that the command has no notification object associated with it
this.m_command = new SqlCommand("SELECT Column1, Column2 FROM dbo.Subscription", this.m_connection)
{Notification = null};
// Create a dependency and associate it with the SqlCommand.
this.m_dependency = new SqlDependency(this.m_command);
this.m_dependency.OnChange += this.OnDependencyChange;
// We want to send a fresh SELECT command to SQL Server on startup and after we receive a notification.
// We use the DataReader to drain the result set resulting from the query. SQL Server will execute
// this command, and set up the entire Query Notification plumbing. SQL Server will execute
// this command internally every once in a while, and will send a one-time event to us when it detects that the
// query has changed. It is up to us to manually subscribe for further events.
this.m_reader = this.m_command.ExecuteReader();
while (this.m_reader.Read())
;
}
catch (Exception exc)
{
Logger.Log.Error("Problem in the Server's entitlement table watcher", exc);
}
}
void OnDependencyChange(object sender, SqlNotificationEventArgs e)
{
// This event firing is a one-time thing ... we have to manually
// reset everything in order to receive another event
SqlDependency dependency = (SqlDependency) sender;
dependency.OnChange -= this.OnDependencyChange;
// Process the event
if (e.Type == SqlNotificationType.Change)
{
this.m_entitlementModel.ClearEntitlementsCache();
}
// Go back and wait for another event
this.PollForNotifications();
}
}
#endregion
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Saturday, September 20, 2008
Colin Clark has a Blog
Colin is the former CEO of Kaskad Technology, and is very involved int he Event Processing world. His new blog can be found at http://colinclarkeventprocessing.com
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Friday, September 19, 2008
Wow! What a Week!
It's finally the Friday morning of one of the most amazing weeks of my life. There are so, so many things to write about. I often use this blog as a "diary" for myself, so that I can look back 10 years from now and see the state of the world and my career at that time. And, this has to rank as a life-changing week.
Yesterday may have been the most thrilling ride in the stock market that I have ever seen. The day played out like a football game. I arrived in the morning, and watched the market try to mount a comeback after a nearly 500 point drop the day before. The Dow was up about 150 points, but all of a sudden, the rally started to lose steam. What was happening? Oh no! Several huge money market funds said that they broke the buck. The Dow plunged from 150 to about -100. I have watching Goldman and Morgan trade like penny stocks, as Goldman trades below 90 and Morgan trades below 13.
But wait, the Dow is mounting a huge rally, back into positive territory. What is happening now? It's our brethren in Britain banning short sales. Huge rally under way.
Now, another jump in the Dow. What's going on? It's the Yanks' turn now. CALPERS and a few other huge retirement funds coming out and saying that they won't lend out their shares of financial companies for the shorts. CNBC is saying that it's an all-out war against the shorts. God bless Apple Pie and the USA!
I go into a meeting with the Dow up about 100, and a half hour later, I come out and the Dow is up by 400. It's Hank Paulson and the new Resolution Trust Company! Not to be outdone, the SEC now wants to ban shorting of about 800 financial stocks.
As I write this, the FTSE is up around 8%.
My CEP team sits on the Equity Trading Floor, right below the big board in the first row of the floor. Sitting next to me are the two "brokers of the stars" .... the two guys who are the brokers for the traders themselves. Traders are screaming all around me. Applause breaks out spontaneously.
I may end up penniless, but at least, you cannot ever say that it has been boring!
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Yesterday may have been the most thrilling ride in the stock market that I have ever seen. The day played out like a football game. I arrived in the morning, and watched the market try to mount a comeback after a nearly 500 point drop the day before. The Dow was up about 150 points, but all of a sudden, the rally started to lose steam. What was happening? Oh no! Several huge money market funds said that they broke the buck. The Dow plunged from 150 to about -100. I have watching Goldman and Morgan trade like penny stocks, as Goldman trades below 90 and Morgan trades below 13.
But wait, the Dow is mounting a huge rally, back into positive territory. What is happening now? It's our brethren in Britain banning short sales. Huge rally under way.
Now, another jump in the Dow. What's going on? It's the Yanks' turn now. CALPERS and a few other huge retirement funds coming out and saying that they won't lend out their shares of financial companies for the shorts. CNBC is saying that it's an all-out war against the shorts. God bless Apple Pie and the USA!
I go into a meeting with the Dow up about 100, and a half hour later, I come out and the Dow is up by 400. It's Hank Paulson and the new Resolution Trust Company! Not to be outdone, the SEC now wants to ban shorting of about 800 financial stocks.
As I write this, the FTSE is up around 8%.
My CEP team sits on the Equity Trading Floor, right below the big board in the first row of the floor. Sitting next to me are the two "brokers of the stars" .... the two guys who are the brokers for the traders themselves. Traders are screaming all around me. Applause breaks out spontaneously.
I may end up penniless, but at least, you cannot ever say that it has been boring!
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Thursday, September 18, 2008
Morgan Stanley and Citi/Wachovia ???
As I drove home from work yesterday, I listened to the reports on Bloomberg Radio about a possible new company called MorganStanleyWachovia or MorganStanleyCiti.
Most people think about the financial ramifications of such a marriage. I think of the ramifications for the IT departments, especially focusing on trading technology.
I think that both of the combinations would be extremely difficult to integrate as far as IT goes. The IT culture of Morgan Stanley is so much different that any other place (with maybe the exception of Goldman). Morgan is very much of a build-it-yourself culture. They have their own messaging system (CPS) and their own ticker plant (Filter). The mad Russian scientists who populate the various IT departments pride themselves on the code that they right. It is much much different than the IT culture that you find in large banks like Citi, Wachovia, Well Fargo, etc.
It is public information that a lot of Morgan Stanley talent has recently migrated to Citi. Vikram Pandit is ex-Morgan, and you see many more ex-Morganites starting to populate the executive ranks. So, the "pump is primed" at Citi for a Morgan Stanley merger, at least at the executive levels. But the IT cultures are so radically different, both in terms of culture and technology, that I think that it will take a very long time to integrate.
Wachovia always had the relatively laid-back Charlotte way of doing things. Lots of legacy technology swimming around their halls. A few years ago, Wachovia established a base in New York, and was able to lure a lot of good Goldman people away. But, there is still that back-and-forth between New York and Charlotte, and most of the Wachovia people that I know in New York have to fly to Charlotte on a regular basis to get their marching directions. This is not the Morgan way of doing things.
BankOfAmericaMerrill is going to be another fun ride on the IT-integration-ferris wheel. Merrill has some really good technology. Some is very new and some is really old (the CICS-based systems that were written 20 years ago are still fairly important).
This is a time when Integration Architecture might be the hottest skill set on Wall Street. I expect boom times for companies like Accenture, IBM, Capco, and the other large consulting firms that specialize in technology integration.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Most people think about the financial ramifications of such a marriage. I think of the ramifications for the IT departments, especially focusing on trading technology.
I think that both of the combinations would be extremely difficult to integrate as far as IT goes. The IT culture of Morgan Stanley is so much different that any other place (with maybe the exception of Goldman). Morgan is very much of a build-it-yourself culture. They have their own messaging system (CPS) and their own ticker plant (Filter). The mad Russian scientists who populate the various IT departments pride themselves on the code that they right. It is much much different than the IT culture that you find in large banks like Citi, Wachovia, Well Fargo, etc.
It is public information that a lot of Morgan Stanley talent has recently migrated to Citi. Vikram Pandit is ex-Morgan, and you see many more ex-Morganites starting to populate the executive ranks. So, the "pump is primed" at Citi for a Morgan Stanley merger, at least at the executive levels. But the IT cultures are so radically different, both in terms of culture and technology, that I think that it will take a very long time to integrate.
Wachovia always had the relatively laid-back Charlotte way of doing things. Lots of legacy technology swimming around their halls. A few years ago, Wachovia established a base in New York, and was able to lure a lot of good Goldman people away. But, there is still that back-and-forth between New York and Charlotte, and most of the Wachovia people that I know in New York have to fly to Charlotte on a regular basis to get their marching directions. This is not the Morgan way of doing things.
BankOfAmericaMerrill is going to be another fun ride on the IT-integration-ferris wheel. Merrill has some really good technology. Some is very new and some is really old (the CICS-based systems that were written 20 years ago are still fairly important).
This is a time when Integration Architecture might be the hottest skill set on Wall Street. I expect boom times for companies like Accenture, IBM, Capco, and the other large consulting firms that specialize in technology integration.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Wednesday, September 17, 2008
We are Hiring
I am going to indulge in what some of you might think is boorish and vulture-like behavior. My heart bleeds for the people at Lehman, Bear, and Merrill who have been displaced, either recently or in the near future. However, people have to live and they need to support their families. So, here I go ....
My company has openings in most of the departments within Equities .... market making, trading systems, risk, etc. We are interested in all skill sets, most notably Java and C++. We need great, hard-core developers. These are not management positions.
If you are interested, please send me a resume. I do not profit personally from this. I am just interested in seeing us with the best talent out there, and catching up to our remaining competitors.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
My company has openings in most of the departments within Equities .... market making, trading systems, risk, etc. We are interested in all skill sets, most notably Java and C++. We need great, hard-core developers. These are not management positions.
If you are interested, please send me a resume. I do not profit personally from this. I am just interested in seeing us with the best talent out there, and catching up to our remaining competitors.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Review of the Gartner 2008 Conference on Event Processing
I spent an interesting two days in scenic Stamford, Connecticut on September 15 and 16, 2008. It was my second venture to a Gartner conference solely devoted to event processing, the first being in Orlando in 2007 when we were CEP neophytes. This time, I was not in learning mode .... the sessions that I attended were designed to give me an insight into what other financial companies are doing with CEP, and what the future direction of CEP might be.
The conference seemed a bit smaller than last year's conference, maybe because this year, it was not preceded directly by the Gartner conference on Business Intelligence. Some of the vendors were lamenting the lack of prospects. Indeed, the conference seemed much more heavily stacked with vendor participants than with potential customers. This speculation was confirmed when I saw a list of the attendees. I would estimate that, out of about 200 attendees, maybe 2o-25% were customers. I was one of 9 attendees from financial services firms, with most of the other attendees coming from fairly small companies. Does this mean that financial companies have already made their decisions concerning CEP, or are the financial companies being more prudent with their spending on conferences and travel? The next year should be a very interesting one for the vendors of CEP products, especially ones that are targeting financial firms.
Mary Knox started off with her usually overview of adopting CEP in financial services. A fairly good overview, with many valid points. I also notice that Mary has been reading certain blogs, as some of this information was included in her presentation. I questioned Mary's numbers on CEP spending, as it showed an extremely rosy outlook for the CEP marketplace, an outlook which may have been valid at last year's conference, but may no longer be valid in these times.
Mark Palmer of Streambase gave a great presentation on Smart Order Routing. I feared that it was going to be an informercial for Streambase, especially since, as I walked into the conference room, a Streambase rep thrust a USB flash drive at me which contained the latest incarnation of the Streambase platform. However, Mark presented a very good presentation without really harping on Streambase, and to tell you the truth, I like the direction where they are heading. It's pretty obvious that Mark is bringing a lot of the direction from Apama to Streambase in terms of developing financial verticals. I also hope that Mark will have a positive effect on the Streambase culture, and that their sales and marketing team will adopt a lower-key approach.
Mark's presentation was followed by Robert Almgren, who used to be with Bank of America before striking out on his own (???). Last year, Robert gave a fascinating intro to Algo trading. This year, he gave some in-depth looks at the architecture of an algo trading system, something that he was probably not allowed to do last year. Streambase is sponsoring some work that he and Bill Harts are doing in Smart Order Routing, and I am sure that the results will make their way back into the Streambase platform. Streambase's direction of totally embracing the financial services industry may or may not pay off in the future; from what a few of the vendors have told me, the interest in CEP in the hedge fund sector has cooled off somewhat.
Louis Morgan from HG Trading pretty much rehashed last year's presentation. HG Trading specializes in High Frequency Trading. Their average holding time for a stock is 7 minutes, and they never hold a position overnight. He mentioned that trading volume have gone up substantially, while the latency of his system has gone from 20ms to under 2ms. HG trading is a three person shop .... Lou, his developer, and a quant. They are extremely agile, and not afraid to spend money where it is needed. For instance, they spend money on co-location for their servers. They have 6 Apama correlators running. It was interesting that Lou gave his talk around the time the market opened on Monday (the morning of the Merrill and Lehman announcements), and he would have been totally forgiven if he has to cancel his presentation and attend to business.
Another good talk with given by Albert Doolittle of George Weiss, a small trading firm in Hartford, Connecticut. I saw many parallels between Albert's experience and my own .... he just jumped into CEP last year, did the trade show circuit, picked Aleri, did a POC, and now he has a CEP system up and running.
I think that my own talk went fairly well, and that I save the attendees some things to ponder.
It was nice to see some of the luminaries of the CEP world, including Opher, and to reconnect with many of the CEP vendors. Ron from Aleri introduced me to the Black Martini (Vodka, Blue Curacao, and Chamborde), and I know that he was moving a few steps slower the following morning.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
The conference seemed a bit smaller than last year's conference, maybe because this year, it was not preceded directly by the Gartner conference on Business Intelligence. Some of the vendors were lamenting the lack of prospects. Indeed, the conference seemed much more heavily stacked with vendor participants than with potential customers. This speculation was confirmed when I saw a list of the attendees. I would estimate that, out of about 200 attendees, maybe 2o-25% were customers. I was one of 9 attendees from financial services firms, with most of the other attendees coming from fairly small companies. Does this mean that financial companies have already made their decisions concerning CEP, or are the financial companies being more prudent with their spending on conferences and travel? The next year should be a very interesting one for the vendors of CEP products, especially ones that are targeting financial firms.
Mary Knox started off with her usually overview of adopting CEP in financial services. A fairly good overview, with many valid points. I also notice that Mary has been reading certain blogs, as some of this information was included in her presentation. I questioned Mary's numbers on CEP spending, as it showed an extremely rosy outlook for the CEP marketplace, an outlook which may have been valid at last year's conference, but may no longer be valid in these times.
Mark Palmer of Streambase gave a great presentation on Smart Order Routing. I feared that it was going to be an informercial for Streambase, especially since, as I walked into the conference room, a Streambase rep thrust a USB flash drive at me which contained the latest incarnation of the Streambase platform. However, Mark presented a very good presentation without really harping on Streambase, and to tell you the truth, I like the direction where they are heading. It's pretty obvious that Mark is bringing a lot of the direction from Apama to Streambase in terms of developing financial verticals. I also hope that Mark will have a positive effect on the Streambase culture, and that their sales and marketing team will adopt a lower-key approach.
Mark's presentation was followed by Robert Almgren, who used to be with Bank of America before striking out on his own (???). Last year, Robert gave a fascinating intro to Algo trading. This year, he gave some in-depth looks at the architecture of an algo trading system, something that he was probably not allowed to do last year. Streambase is sponsoring some work that he and Bill Harts are doing in Smart Order Routing, and I am sure that the results will make their way back into the Streambase platform. Streambase's direction of totally embracing the financial services industry may or may not pay off in the future; from what a few of the vendors have told me, the interest in CEP in the hedge fund sector has cooled off somewhat.
Louis Morgan from HG Trading pretty much rehashed last year's presentation. HG Trading specializes in High Frequency Trading. Their average holding time for a stock is 7 minutes, and they never hold a position overnight. He mentioned that trading volume have gone up substantially, while the latency of his system has gone from 20ms to under 2ms. HG trading is a three person shop .... Lou, his developer, and a quant. They are extremely agile, and not afraid to spend money where it is needed. For instance, they spend money on co-location for their servers. They have 6 Apama correlators running. It was interesting that Lou gave his talk around the time the market opened on Monday (the morning of the Merrill and Lehman announcements), and he would have been totally forgiven if he has to cancel his presentation and attend to business.
Another good talk with given by Albert Doolittle of George Weiss, a small trading firm in Hartford, Connecticut. I saw many parallels between Albert's experience and my own .... he just jumped into CEP last year, did the trade show circuit, picked Aleri, did a POC, and now he has a CEP system up and running.
I think that my own talk went fairly well, and that I save the attendees some things to ponder.
It was nice to see some of the luminaries of the CEP world, including Opher, and to reconnect with many of the CEP vendors. Ron from Aleri introduced me to the Black Martini (Vodka, Blue Curacao, and Chamborde), and I know that he was moving a few steps slower the following morning.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Goodbye Lehman and Merrill
I could not sleep last Sunday night, the night before the Gartner CEP conference was to start. I was transfixed to CNBC for the entire night. And, I was really depressed ....
Why? Merrill Lynch was disappearing as a firm. One of the most storied financial firms in the world, and a widely recognized brand name.
My father was a broker for Merrill Lynch for 25 years, first starting in a little branch office on Austin Street in Forest Hills, Queens. One of his friends at the office, a guy who used to come over our house and play Pinochle with my dad, was a guy named Dave Kamamsky. Big Dave eventually rose through the ranks at Merrill to become the CEO of Merrill during the go-go years of the 1990's.
My wife worked for Merrill Lynch for about 10 years, starting off as a CICS/Cobol developer and rising to run several of the Equity trading systems there.
Seeing Merrill being swallowed up by Bank of America is like losing an arm. In a year, we have lost three of the major financial institutions in New York. Who knows how long Morgan Stanley will be around ... as I write this, I am watching Charles Gasparino on CNBC saying that John Mack is weighing whether or not to keep Morgan Stanley independent.
These are life-changing times we are in ....
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Why? Merrill Lynch was disappearing as a firm. One of the most storied financial firms in the world, and a widely recognized brand name.
My father was a broker for Merrill Lynch for 25 years, first starting in a little branch office on Austin Street in Forest Hills, Queens. One of his friends at the office, a guy who used to come over our house and play Pinochle with my dad, was a guy named Dave Kamamsky. Big Dave eventually rose through the ranks at Merrill to become the CEO of Merrill during the go-go years of the 1990's.
My wife worked for Merrill Lynch for about 10 years, starting off as a CICS/Cobol developer and rising to run several of the Equity trading systems there.
Seeing Merrill being swallowed up by Bank of America is like losing an arm. In a year, we have lost three of the major financial institutions in New York. Who knows how long Morgan Stanley will be around ... as I write this, I am watching Charles Gasparino on CNBC saying that John Mack is weighing whether or not to keep Morgan Stanley independent.
These are life-changing times we are in ....
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Saturday, September 13, 2008
Optimizing Performance in a CEP System
Here are some lessons learned that helped us reduce message backup from our Tibco EMS broker into our CEP Input Server.
1) Make sure that you have fast networking between the Tibco broker and your servers.
We just happened to find out that the hardware folks gave us legacy 100MB connections between our Tibco EMS broker and our CEP Input Server. We asked them to upgrade us to GigE immediately. Unless you have personally installed your hardware, never assume anything in your infrastructure.
2) It might be helpful to turn the "AutoAcknowledge" flag off in your Tibco session.
To do this, change
this.m_connection.CreateSession(false, SessionMode.AutoAcknowledge);
to
this.m_connection.CreateSession(false, SessionMode.NoAcknowledge);
Be careful about using No Acks. What is the impact to your CEP system if you happen to drop a message?
3) Implement threaded queues on the input from Tibco to your input server, and on the output from your input server to the CEP system.
If you need more performance, change the single threaded queue that reads messages from Tibco to a load-balanced multiple queues, each with its own thread. The load balancer can partition by Tibco topic, by ticker, or by OrderId.
Of course, you can always use the built-in adapters from your CEP vendor. But that may come at an extra monetary cost, so check with your CEP vendor to see if a certain input adapter comes free of charge. Also, the more of the CEP vendor's infrastructure you use, the more it ties you into the CEP vendor.
4) Try to do some pre-filtering of messages in your input server before passing them on to the CEP engine.
Recently, we were able to reduce the CPU usage of Coral8 by about 50% (according to our Coral8 guy) by filtering out FIX messages that we knew that Coral8 was not interested in.
5) Continually optimize your code. Just this morning, I did a code review of the input server, and I was able to suggest a change which reduced one hashtable lookup on each FIX message. I know that there are tons of other places in our code that can do with some optimization. If you need to parse FIX messages into C# objects, then do some investigation and benchmarking of FIX engines.
6) Use the Server Garbage collector
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
1) Make sure that you have fast networking between the Tibco broker and your servers.
We just happened to find out that the hardware folks gave us legacy 100MB connections between our Tibco EMS broker and our CEP Input Server. We asked them to upgrade us to GigE immediately. Unless you have personally installed your hardware, never assume anything in your infrastructure.
2) It might be helpful to turn the "AutoAcknowledge" flag off in your Tibco session.
To do this, change
this.m_connection.CreateSession(false, SessionMode.AutoAcknowledge);
to
this.m_connection.CreateSession(false, SessionMode.NoAcknowledge);
Be careful about using No Acks. What is the impact to your CEP system if you happen to drop a message?
3) Implement threaded queues on the input from Tibco to your input server, and on the output from your input server to the CEP system.
If you need more performance, change the single threaded queue that reads messages from Tibco to a load-balanced multiple queues, each with its own thread. The load balancer can partition by Tibco topic, by ticker, or by OrderId.
Of course, you can always use the built-in adapters from your CEP vendor. But that may come at an extra monetary cost, so check with your CEP vendor to see if a certain input adapter comes free of charge. Also, the more of the CEP vendor's infrastructure you use, the more it ties you into the CEP vendor.
4) Try to do some pre-filtering of messages in your input server before passing them on to the CEP engine.
Recently, we were able to reduce the CPU usage of Coral8 by about 50% (according to our Coral8 guy) by filtering out FIX messages that we knew that Coral8 was not interested in.
5) Continually optimize your code. Just this morning, I did a code review of the input server, and I was able to suggest a change which reduced one hashtable lookup on each FIX message. I know that there are tons of other places in our code that can do with some optimization. If you need to parse FIX messages into C# objects, then do some investigation and benchmarking of FIX engines.
6) Use the Server Garbage collector
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
The London stock Exchange Crash and .NET/Windows
I have been on the receiving side of some good-natured ribbing inside of my company in the wake of the LSE crash and the suspected involvement of .NET and Windows. Our Complex Event Processing System, one of the most strategic projects in our company, runs totally on Windows 2003/64-bit and .NET 3.5. It is one of the highest-profile usages of .NET in one of the largest financial institutions in the world.
Microsoft has been very public is touting the fact that the LSE is a heavy user of .NET. But, from what I can tell, there hasn't been a peep out of Microsoft in defending their platforms in the wake of the LSE debacle. Speculation has pointed to problems with Cisco, problems with SQL Server 2000, problems with the code written by Accenture consultants, and problems with Sarah Palin.
I have been vocal on my blog in the past about the viability of Windows and .NET as a platform for real-time trading applications. I have never seen Microsoft come out with a definitive roadmap for using its platforms for realtime trading apps. No benchmarks against Realtime Linux. No attempts to through OPRA feeds at a Microsoft Server.
Sure, there is some good stuff going on at Microsoft with their grid efforts, but that effort will be the most beneficial to risk and analytics. Things like Velocity Object Caches and SSDS are a lot of fun. Excel is everywhere. But, the core operating system is Microsoft's bread and butter.
We need Microsoft to defend the LSE's use of Windows and .NET, and we need Microsoft to address viability of using its platform for low-latency, high-bandwidth trading apps.
At the Stac Council meeting in July, the companies who gave presentations about using their platforms for trading apps included IBM, Sun, and HP. Where was Microsoft? Why aren't they involved in the Stac Council?
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Microsoft has been very public is touting the fact that the LSE is a heavy user of .NET. But, from what I can tell, there hasn't been a peep out of Microsoft in defending their platforms in the wake of the LSE debacle. Speculation has pointed to problems with Cisco, problems with SQL Server 2000, problems with the code written by Accenture consultants, and problems with Sarah Palin.
I have been vocal on my blog in the past about the viability of Windows and .NET as a platform for real-time trading applications. I have never seen Microsoft come out with a definitive roadmap for using its platforms for realtime trading apps. No benchmarks against Realtime Linux. No attempts to through OPRA feeds at a Microsoft Server.
Sure, there is some good stuff going on at Microsoft with their grid efforts, but that effort will be the most beneficial to risk and analytics. Things like Velocity Object Caches and SSDS are a lot of fun. Excel is everywhere. But, the core operating system is Microsoft's bread and butter.
We need Microsoft to defend the LSE's use of Windows and .NET, and we need Microsoft to address viability of using its platform for low-latency, high-bandwidth trading apps.
At the Stac Council meeting in July, the companies who gave presentations about using their platforms for trading apps included IBM, Sun, and HP. Where was Microsoft? Why aren't they involved in the Stac Council?
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Wednesday, September 10, 2008
Automated Trading and CEP on News Feeds
Something for us all to think about if we ever use automated news in systematic trading or CEP.
Wall Street Journal article here.
“The damage was exacerbated by the growing use on Wall Street of automated programs that trigger stock trades without any human interaction,”
This caused a drop of $1B in the market share of United Airlines.
I wonder if the sentiment engines (like Reuters Sentiment, SemLab, Corpora) would have caught this?
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Wall Street Journal article here.
“The damage was exacerbated by the growing use on Wall Street of automated programs that trigger stock trades without any human interaction,”
This caused a drop of $1B in the market share of United Airlines.
I wonder if the sentiment engines (like Reuters Sentiment, SemLab, Corpora) would have caught this?
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Saturday, September 06, 2008
CEP and Shoplifting
Yesterday, I decided to drive the long way for a lunch with an ex-colleague of mine from Morgan Stanley. The local NPR public radio station (WNYC) has replaced the heavy metal station (WSOU) as my primary station in my car (sigh ... the joys of getting older ...), and I caught the last part of the Leonard Lopate show, a very popular news-talk show in New York.
Leonard was interviewing John Colapinto, who had written an article for The New Yorker on loss prevention in stores ... commonly known as shoplifting.
What was fascinating was that, as John was talking about how store detectives monitor customer patterns for shoplifting, it seemed like the perfect thing for Complex Event Processing (I hope that Tim will agree that this is truly Complex).
Now, I am sure that this is not news to the CEP vendors out there. Fraud detection and surveillance is one of the big applications of CEP. But, after each detection pattern Colapinto described, I said to myself "How would we do this in Coral8?".
Some of the patterns Colapinto mentioned included:
1) Customers examining clothing without looking at the price tag.
2) Customers moving randomly from table to table. There seems to be a certain pattern of movement that is common amongst shoplifters.
In the "Shoplifter Alert System" that we would build, there would be multiple levels of complex events (Tim can correct me on the proper terminology to use) :
Level 1 Event
"This person is a shoplifter"
Level 2 Events
"This person is moving too randomly through the store"
"This person does not seem interested in the price"
Level 3 Events
The amount of time a person is spending at each "table" in the store
The amount of time a person handles each garment
A sensor on the price tag to indicate whether the person has looked at the price
Level 4 Events
A person has entered the store
A person has left the store
A person is moving through the store
I would love to hear any anecdotes about how CEP is actually being used in detecting shoplifting, so if you know of anything, please comment.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Leonard was interviewing John Colapinto, who had written an article for The New Yorker on loss prevention in stores ... commonly known as shoplifting.
What was fascinating was that, as John was talking about how store detectives monitor customer patterns for shoplifting, it seemed like the perfect thing for Complex Event Processing (I hope that Tim will agree that this is truly Complex).
Now, I am sure that this is not news to the CEP vendors out there. Fraud detection and surveillance is one of the big applications of CEP. But, after each detection pattern Colapinto described, I said to myself "How would we do this in Coral8?".
Some of the patterns Colapinto mentioned included:
1) Customers examining clothing without looking at the price tag.
2) Customers moving randomly from table to table. There seems to be a certain pattern of movement that is common amongst shoplifters.
In the "Shoplifter Alert System" that we would build, there would be multiple levels of complex events (Tim can correct me on the proper terminology to use) :
Level 1 Event
"This person is a shoplifter"
Level 2 Events
"This person is moving too randomly through the store"
"This person does not seem interested in the price"
Level 3 Events
The amount of time a person is spending at each "table" in the store
The amount of time a person handles each garment
A sensor on the price tag to indicate whether the person has looked at the price
Level 4 Events
A person has entered the store
A person has left the store
A person is moving through the store
I would love to hear any anecdotes about how CEP is actually being used in detecting shoplifting, so if you know of anything, please comment.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Monday, September 01, 2008
Two strong .NET/C# Developers Wanted
We have openings for two .NET developers. One is for our options market-making operating in Jersey City. The other is for an FX trading system in New York.
1) No consultants. These are full-time positions.
2) Must have very strong experience in WinForms.
3) Real-time messaging experience needed. Sockets, Tibco (RV or EMS), MSMQ, MQ are all nice to haves.
4) Multi-threading experience is a must.
If you are interested, please send me your resume.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
1) No consultants. These are full-time positions.
2) Must have very strong experience in WinForms.
3) Real-time messaging experience needed. Sockets, Tibco (RV or EMS), MSMQ, MQ are all nice to haves.
4) Multi-threading experience is a must.
If you are interested, please send me your resume.
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Subscribe to:
Posts (Atom)
