HPC on Wall Street - Hardware Acceleration

Sam Brown of the University of Utah just came out with some slides on his initial forays into hardware acceleration. He contrasts the use of Cell Processors, FPGAs, and Multi-core processors. He was trying to speed up 2-Way Wave Equation Modeling, which is not a financial application, but the lessons that he learned can be applied to those of use who are doing (or exploring) HPC in Wall Street.

The big effort is in unrolling loops and parallelizing your code. Using the POSIX PThread API and multi-core cpus is the easiest thing to do. With 4 threads (2 dual-core 2.2 Ghz Opterons, 4 GB RAM), he got a 11x improvement on one set of benchmarks, and a 2x improvement on the big application.

The FPGA system used the Altera Stratix II FPGA. He got about a 14x improvement on the initial set of benchmarks, but the development effort was a lot harder. It took 12 hours to get the design onto the FPGA chip.

The Cell systems were a Sony Playstation 3 (!!!) and a IBM Cell Blade with 2 Cell Processors with 8 SPEs each (a Cell processor has 8 SPEs, which function as coprocessors.) The Cell results were generally disappointing, with the author wondering if he programmed the Cell correctly. One thing that I have heard about the Cell is that it is very difficult to program. Perhaps companies like PeakStream can help you with this effort.

(For instant gratification, the money shots are slides 28, 29, and 30)

What can hardware acceleration do for you? Faster pricing of derivatives. Faster generation of risk values, especially for exotics. Better overnight stressing.

The problem is in acquiring the skill set to parallelize algorithms, deal with the development environments (especially debugging), learning new variants of C, and getting buy-in from the business.



©2007 Marc Adler - All Rights Reserved

My New Favorite Place

The Brandy Library is in the TriBeCa section of New York City. Plush furniture, wood-paneled walls that contain top-shelf liquor, amazing scotches and brandies, interesting tapas to knosh on.

My scotches for the evening were Ardbeg Uigeadail and Caol Ila (18 years). Jerry was sipping their Hot Apple Cider all night. To top it off, a great "hot chocolate".

(I am not affliliated with the Brandy Library in any way ... just a satisfied customer.)









©2007 Marc Adler - All Rights Reserved

Sharing XSDs between .NET and C++/Win32

Free products for generating C++ code for XSD's :

http://www.codesynthesis.com/products/xsd/
http://freshmeat.net/projects/xsd/

A further list of products can be found at http://www.w3.org/XML/Schema

I am in the market for a FREE XSD->C++ code generation tool that I can use to share schemas between .NET GUIs and C++/Win32 servers. Matt pointed me to these two.

What is the best JaxB/Castor equivalent for the C++/Win32 world? I'll let you know when I have finished my evaluation.

(Morgan Stanley makes it easy with CPS and their SOAP libs... every firm should have one of these)


©2007 Marc Adler - All Rights Reserved

Just returned from London, where 2 inches of snow managed to shut down most of the city. Even the newscasters on Sky News were lamenting the fact that London (with one week of advance warning) cannot seem to deal with white powder.

Got together with some ex-colleagues, who were marvelling at the Lodnon consulting market.
The hot areas are Grid Computing, with the prevelant stack being DataSynapse and Tangasol. Also demand is picking up for WPF, with Morgan Stanley leading the way. The daily rates for qualified individuals are about 1000 pounds per day, which at the current exchange rates, is about $2000.

There is a well-known financial consulting firm who shut down its London offices last year, telling employees that they were losing money and that there was no money to be made in the London market. Hmmmmmm.....

It will be interesting to see if Microsoft's Compute Cluster Server and Digipede can make any inroads intot his market. There seems to be a very strong bias against using .NET for a grid infrastructure, something which I hope to see turned around in 2007.

I got a chance to walk around Canary Wharf a bit more this time. I hope that JL gets to go to the Gaucho Grill (Argentinian Steaks) and Nobu (Japanese) on his upcoming trip. But I need to have TS take JL out for a night of karaoke, and take some pictures to prove it! (Our karaoke outing got cancelled because of the snow .... to the relief of the collective ears of the London Equities Architecture group!)

©2007 Marc Adler - All Rights Reserved

Welcome Back Crispin

Crispin has a new blog ... but you'll have to find it for yourselves!

©2007 Marc Adler - All Rights Reserved

Syncfusion GridControl vs GroupingGrid

I am starting to write the grid abstraction for my client-side framework. Developers will code to a "virtual grid" API, and there will be adapters for the actual implementations of the grids (Syncfusion, Infragistics, WinForms, DevExpress, etc).

Because most of Wall Street uses Syncfusion grids for trading apps, this is the first adapter that I need to write.

Syncfusion has 3 different grids: the regular old GridControl, the DataBoundGrid, and the GroupingGrid. The grouping grid is used a lot in my place to provides rollups of data. You would think that the GroupingGrid would inherit from the GridControl, but unbelievably, it does not. The API sets are not even consistent.

I have spent my time using Lutz Roeder's very excellent Reflector to try to hunt and peck through the Syncfusion class hierarchy in search of the various classes and functions in the GroupingGrid that implement the same functionality as the GridControl. The fact that I am dealing with two incompatible API sets has increased my work by 100%.

C'mon Syncfusion ... you should know better than that!


©2007 Marc Adler - All Rights Reserved

New York Code Camp

DonXml is organizing a Code Camp on Saturday, March 3, in New York City. Last year's camp sold out in 24 hours and they had to max out at 150 attendees. This year, they expanded the number to 400, so there should be plenty of room.

Don is looking for speakers, especially from the financial sector. So, all of you people from Goldman, Morgan, JPMC, Citi, DB, Barcap, Lehman, hedge funds, etc .... get ready to spill the beans! Come tell us all how you are using .NET at your jobs.

Don is also trying to organize a Java/.NET Code Camp in New Jersey, and is looking for some company to donate space. Get in touch with Don if you are a North/NorthCentral-based organization that can donate some conference facility for one Saturday.

©2007 Marc Adler - All Rights Reserved

Microsoft MVP Summit

Although I am not a Microsoft MVP, I have been invited to attend this year's MVP Summit in Seattle as one of the 20 "Influencers" of .NET technology. I hope to see some of the readers of this blog over there and share in some good Washington State microbrews. The summit runs the week of March 12. (Big shout out to Joe Rubino for this event...)

Coming from the capital markets sector, I will be most interested in the following:

Windows Compute Cluster
- how can Microsoft compete with establish grid players like Platform
- how do they cooperate with other grid vendors (ie: Digipede) that we might want to use for some lighter projects
- tie-ins with Excel Services

Analysis Services
- how can we use Analysis Services to perform risk analysis
- how can we use AS in a quasi real-time mode to perform real-time risk analysis and performance attribution

WPF
- what new compelling visualizations for trading systems can WPF give us
- performance aspects of WPF for trading systems

WCF
- would love to see WCF expanded to Tibco RV and EMS. That wouold give it more bang in the Capital Markets area.

SQL Server
- what is coming down the pike for SS that would be most applicable to our systems (time-series, complex event processing, etc). Will they be doing anything to get into Vhayu's and Streambase's space?

Capital Markets Strategy
- I have been very vocal with Microsoft about their (lack of) coherent strategy for Capital Markets. I want to see what plans they have for the entire stack. Let them demonstrate to me that they are think about coming out with best-of-breed components along the entire stack, and that people will take Microsoft seriously for server-side components.


©2007 Marc Adler - All Rights Reserved

Off to London Again

I'll be in London again on the week of February 5, so I will be moderating the comments sporadically during that week. A week of hard work, and a kareoke session on Thursday night. The last time I was there, I found that the Brits could not handle their tequila shots very well (The Aussies and Kiwis were only slightly better). They have promised to practice while I was away .... we'll see if their hard work paid off. Maybe Terry will take a few photos...

©2007 Marc Adler - All Rights Reserved

Dissecting a Wall Street Job Posting

A lot of you have emailed me privately with questions about working on Wall Street, and how to land a job there. (I assume that the questions are coming by email because everyone is afraid of their bosses tracing the IP addresses.)

I would like to take a recent job description that was on the 96Bosses.com site and dissect it.

Here it is:

=======================================================

Job Description
Our client, a top-tier global brokerage firm, is seeking multiple C++/Java Developers to work on an a real-time algo trading platform for their US Equitites trading area. This is a fulltime opportunity located in New York City, NY.

Qualifications
Experience in designing and implementing high speed trading system that can be used for algorithmic automated trading and smart order routing.

Prior experience in US equities trading environment is essential. Prior experience in US options/futures trading environment a plus. Prior experience in UK/Asia markets a plus. Knowledge of building automated trading systems and/or exchange connectivity is a big plus.

• Strong technical programming skills is a key requirement. The Object Oriented Design and Programming is essential. Knowledge of design patterns is also key. Hands on expertise in programming in C++ and Java (server side programming) is necessary. In depth knowledge of multithreading, exceptions, template etc.

• Needs to have strong expertise in inter process communication and event driven programming
Projects will include:
• Algorithmic Trading
• Automated Market Making
• Smart Order Routing
• Trade Analysis for Equities, Options and Futures in US Domestic and International Markets

Experience with the following products:
• TIBCO Messaging Services
• Ticker Plant
• Times Ten
• GemFire
• Transact Tools (FIX Gateway)
• Web servers (Tomcat/Weblogic)

Compensation
$150 -$300k

========================================================

The work is for an Algorithmic Trading platform. This is a good thing. Wall Street is becoming more and more automated, with the NYSE reducing floor space and a number of firms paring their trading staffs.

Algo trading is a very hot area. Firms want to trade large blocks of stocks without affecting the marketplace. The trading has to be broken up into a lot of small blocks, according to different strategies. The firm also has to be able to switch strategies in mid-stream. This requires monitoring the marketplace in real time, and keeping an intraday history of the stocks that you are interested in.

The firm is looking for experience in high-speed, low latency trading. Latency is everything. You want to get yourselves out there faster than all of your competitors. Latency can be affected by many factors, from speed of the networks to inefficient locking algorithms in your code. There is also a need for experience in smart order routing, which means that you might get to work with some kind of rules engine, like ILog Rules.

Let's look at the technology stack.

- Tibco is being used (either Rendevous or EMS), probably to get the spot prices from the ticker plants into the algo trading server. (Watch out for those multicast storms...)

- "Ticker Plant" probably means something like Reuters or Wombat. If the job is for Merrill, Bear Stearns or RBC, then most likely that Wombat is being used,otherwise most likely Reuters.

- Times Ten is an in-memory database (now owned by Oracle) that is probably being used to hold tick history. Other solutions in this area include KDB+, Vhayu, StreamBase, and some of the newer players that I have talked about on this blog in the past. These newer time-series databases can be used to perform complex event processing on the incoming tick stream.

- Gemfire (by Gemstone) is a distributed cache. This is probably being used to cache queries to the reference data. The use of Times Ten may mean that you will have an Oracle database.

- Transact Tools TTConnect will be used to send and receive FIX messages between your Algo server and the various exchanges. FIX is the standard message format that is used to send equity orders to exchanges and to receive updates about the status of your orders. Latency is extremely important here too. How fast can you get your messages to and from the exchanges.

- Web Servers. Perhaps for some web apps that will be used to monitor status. I would expect to see a .NET front-end here. The fact that .NET is not on the stack makes me believe that the job is not for Goldman nor Morgan Stanley, where .NET front-ends are "de riguer".

Now, onto the compensation. 150-300K ... That's a pretty wide range. The 300K might lead me to believe that this position was for a hedge fund, but the first line tells me that it is a top-tier global brokerage firm. If you take the word "Brokerage" literally, then this would eliminate Lehman, Goldman, and Bear Stearns. Base salaries go from about a low of 130K to about 160K, with 150K being the norm. So, if the total comp was 150K, then you are not getting much of a bonus, are you? Let's take 150K as being the base. Lehman had about 60-80% bonuses this year and Morgan Stanley had about 45-55%. Except for Goldman, you are probably not going to get 100% bonus for a Java/C++ developer role from one of the other firms. So, I would take the 300K figure with a HUGE grain of salt, and look more at 210-225K as total comp.

With all of this said, my firm is also looking for top talent. If you have experience in the above areas, and if you want to explore some opportunities with us, please send your resume to me.


©2007 Marc Adler - All Rights Reserved

96Bosses

While searching for some good interviewing questions for some candidates we have coming in (you are rolling your eyes at that one, but I swear that it's true!), I came upon this site, presumably created by the guy who wrote So You Want To Be a Wall Street Programmer .

Will all people who are applying to my group please *not* read this site, as I don't want you to see the questions that I am going to be asking you....

©2007 Marc Adler - All Rights Reserved

Python and FP on Wall Street???

One of the heads of a group that I deal with asked me about the use of Python on Wall Street. This group is strictly Windows.

Anyone reading this blog using IronPython? Any opinions? What are you doing with it?

Looks like Jane Street is using OCaml. Perhaps we should also be keeping our eyes on Microsoft's F#? Credit Suisse is investigating FP in one of their derivatives groups (FP through Excel).

©2007 Marc Adler - All Rights Reserved

How was your bonus this year?

Bonus season is upon us, and for the first time in my life, I have been part of it as a full-time employee.

When you are independent consultant, you view bonus season as some sort of other-worldly event, a day where the only people who are expected to work are the consultants. Now, when you are part of that world, you have to have to act as father, brother, priest, rabbi, drinking buddy, etc to fellow employees at various times throughout the day.

When I went into work that day, everyone was just standing around eachother's cubicles, nobody able to concentrate on anything. One division actually cancelled all meetings for the entire day. And, despite the exhortations to the employees to NOT discuss their compensation, you know that everyone knows what everyone else got.

Now comes the post-bonus season where the shuffling of chairs begins. I have already gotten calls from ex-colleagues inquiring into positions in my company. Let me give you some advice:

1) The grass always seems greener on the other side, but often isn't.

2) Did you deserve your bonus? Be your harshest critic.

3) Calm down for a day or two, and don't panic the second that you leave your manager's office.

4) Do you really want to work for Goldman? At this time of the year, there is a black hole at GS that resumes flow into. Look at quality of life. Do you want a job where your wife has to remind your kids of your name?

5) The Street is very very very small. Do not burn bridges!


So, how was your bonus?

©2007 Marc Adler - All Rights Reserved

Blood on the Streets

The New York Post reports layoffs on the trading floor.

©2007 Marc Adler - All Rights Reserved

New Visualizations for Trading Systems

I am doing some thinking about new kinds of visual interfaces for trading systems. Whenever you walk through the trading floor, you see rows and rows of traders, each one having 6 monitors, each monitor containing a multiple array of grids of cells flashing in different colors. It takes a special kind of mentality to be able to navigate this extremely hypnotic array of lights 8 hours a day, 5 days a week, 52 weeks a year.

I came upon this thesis, written by one Pasha Roberts of MIT. This is probably the best summary of different visualization techniques being used for financial industries. Pasha also formed his own company, LinePlot, to take advantage of his thesis work.

The use of WPF for new kinds of visualizations is upon us. I have heard that several of the big investment banks are dipping their toes cautiously into the waters. The trick will be to come up with compelling visualizations without overpowering the user. Word is also that several IBs are recruiting game developers. In fact, the other day, there was a job posting on CraigsList for a hedge fund seeking to recruit a game developer.

I wonder if any of the CEP companies like StreamBase, Vhayu, Kaskad, etc have considered visualization schemes as output for the data it processes. It would be interesting for one of these engines to do some CEP and eventual transformation into a stream of XAML. I bet that Matt is thinking along these lines, maybe as an offshoot of his patented work at JPMC.


©2007 Marc Adler - All Rights Reserved

New Book on Building Automated Trading Systems

I'm looking forward to this one ....


©2007 Marc Adler - All Rights Reserved

This is Patently Mad

Financial Services firms rushing to the patent office.

Details of a Goldman patent for reducing volatility. And this is the patent that Goldman just got for diddling.

What does this mean for small companies developing financial software? I know a few people developing new trading packages which they hope to sell. Hope you guys are engaging a patent attorney.

Whenever I came out with a new product through Magma Systems, the first thing I would do was to copyright the code. At that time, you needed to send the "first 100 and last 100 pages" of the souce code to the copyright office.

(thanks to Terry for the first link)


©2007 Marc Adler - All Rights Reserved

Coffee

Random thoughts ....

I have always wondered if there is any correlation between the coffee service that a company provides to the IT staff and the perceived quality of the company.

Barclays had the best coffee service. Being a British bank, it paid a lot of attention to the variety of teas it made available. It also had one of those big machines that made cappucino, expresso, or hot chocolate. I was at that machine five times a day.

BusinessEdge and Morgan Stanley have the little cups of Flavia coffee and hot chocolate ... totally free.

Wachovia in Charlotte ... never tried their coffee service ...in Charlotte, there are three Starbucks in every office building ... I never saw so many Starbucks per square meter.

My current company has free Martinson's coffee, but charges 50 or 75 cents for Flavia. After years of drinking Starbucks, Martinson's seems very bitter and flavorless. I guess that I am too used to Starbucks.

Do you bring in your own coffee machine? Watch it... many companies are cracking down on that practice.

How does your company's coffee culture stack up?


©2007 Marc Adler - All Rights Reserved

I Have Been BlogTagged

The newest fad in the blogsphere is "tag, you're it". DonXml tagged me, and now I have to tell you all five things about myself that you did not already know. I also have to tag five other people.

Ok, let me think ....

1) In high school, I played in a Progressive Rock copy band called Heresy. We performed songs by Gentle Giant, King Crimson, ELP, and others. The keyboard player was a rich kid from Baldwin, Long Island, who had every keyboard ever made.

2) I was a huge pro wrestling fan when I was a kid. I used to get ringside passes to the old Sunnyside Gardens in Queens, and take photos of wrestlers like Pedro Morales, Chief Jay Strongbow, Haystacks Calhoun, etc. I was also a writer for a little wrestling newsletter put out by a guy named David Meltzer ... now Meltzer is probably the best known journalist in pro wrestling.

3) In college, I was the winner of the annual concerto competition, and I played the Mayazumi Conceto for Marimba and Orchestra with the SUNY Albany Symphony Orchestra.

4) My father was good friends with Dave Kamansky. Dave was the ex-CEO of Merrill Lynch, and presided over Merrill in the go-go 90's. My father and Dave worked together in the Forest Hills branch of Merrill Lynch in the 70's, and Dave was part of my dad's pinochle group. So, every few weeks, Dave would come over to our apartment for the big pinochle game.

5) I was classmates with the actor, Tim Robbins. A better of acquaintance of mine was Stacey Nelkin, who was the female lead in Halloween III, and was a former girlfriend of Woody Allen.

Now, I need to tag 5 people. They will be:

Francis Shanahan

Chris Donnan

Matt Davey

John O'Sullivan

David Lattimore-Gay


©2007 Marc Adler - All Rights Reserved

Congrats to Chris

Chris is the new columnist for Automated Trader Magazine.

C'mon Chris .... when are you going to get that venture funding!

©2006 Marc Adler - All Rights Reserved

Penn-Lehman Automated Trading Project

Here

Plus, an internship at the Prop Trading group at Lehman.

Another reason to make sure that your kids study hard for the SATs.

©2006 Marc Adler - All Rights Reserved

News Mining Agent for Automated Stock Trading

Here is a thesis on semantic recognition of financial news items.

... as we slowly move towards phasing out traders (as per IBM's prediction).

©2006 Marc Adler - All Rights Reserved

Coding for Humanity

At this time of year, one's thoughts sometimes turn to the larger things in life. You might ask yourself what your legacy is going to be. Does your coding and architecture skills somehow contribute to the greater good of the world, and will humanity benefit from your efforts?

I never put this on my resume, but I consulted part-time for 4 1/2 years for a company called Classroom Inc. From January 1997 to October 2001, I would devote part of my time to writing "simulations" for CRI, and probably over one million school children have used my programs.

Classroom Inc (CRI) was originally formed as a non-profit partnership between IBM, JP Morgan, and Bear Stearns. Lewis Bernard, who was very high up at JPMC, was the CEO of Classroom Inc. The mission was to provide education computer software to inner-city and rural schools where the children could benefit from an "alternative learning experience".

Each one of the "simulations" was an interactive game, where the student was put in a certain life situation. For example, one simulation put you in the role of a bank manaer, while another one put you in the role as the CEO of a paper company. Each simulation consisted of 12 or 15 "chapters", where each chapter was devoted to a certain issue.

When I started consulting for CRI, each simulation went out to over 100,000 students, and by the time I finished up, each simulation reached more than 250,000 students.

The entire framework was written in MFC/C++. A lot of the internal design came from the old Macromedia Director, which was very popular at the time for creating interactive storyboards. A typical simulation took 6 to 8 months to write. The team consisted of a producer, 2 writers, 2 artists, a QA tester, and myself. Every few weeks, I would get a ZIP file in the mail consisting of the script for a new chapter, and all of the graphics, plus some haphazard directions for one or more "activities" that the kids would have to do in the chapter.

My involvement with CRI ended in 2001 when IBM, who was one of the partners, decided that they wanted to move to a new, internal Flash-based system, and wanted to end all C++ development. It took the IBM consultants quite a while to get the first simulation out there in Flash, but they finally managed to recreate all of the simulations and re-release them.

I was proud to have been a part of this effort for such a long time.

I was inspired to write this by a post that was on Joel on Software a few weeks ago. Joel talked about some cool new jobs that his jobsite was advertising, and one of the jobs dealt with medical technology, which is one of the noblest enterprises. Another worthy position is at DonorsChoice, which reminds me a bit of CRI.


©2006 Marc Adler - All Rights Reserved

Automatic Resource Refactoring Tool

Here

Move all of your hard-coded strings automatically into resource files.

©2006 Marc Adler - All Rights Reserved

Skyler Technology

Yet another entry in the time-series, object-cache, feed handler world ... Skyler Technology's C3 Database. This is definitely a hot area to be in right now, as Skyler, Vhayu, Streambase, etc all seem to be vying for a piece of the pie.

Skyler has a nice use case for order book management.



©2006 Marc Adler - All Rights Reserved

DevExpress XtraPivotGridSuite

Recommended by a colleague who is very interested in OLAP tools....

How long before Infragistics comes up with something similar?


©2006 Marc Adler - All Rights Reserved

OLAP/Analysis Services

Had a very interesting presentation from Microsoft on Analysis Services and OLAP.

What are traders and risk managers using OLAP for at your bank?

Any experience using OLAP is a real-time scenario to perform up-to-the-second reporting? Our feeling is that OLAP cannot be used successfully in a real-time environment unless you have very small cubes.

©2006 Marc Adler - All Rights Reserved

FPG911

Brilliant quote from a colleague:

If you need time off from work to test-drive a Porsche, just tell your boss that you are investigating hardware acceleration.

©2006 Marc Adler - All Rights Reserved

Less Reliance on Vendors

Even though there are a lot of things to be desired about working at Morgan Stanley, I must say that they have got the right idea about lessening their reliance on vendors. Their EIA group has their own XML-based pub/sub message bus (CPS), their own market data infrastructure (Filter), their own .NET client-side framework, and more. What they have done is cut companies like Tibco out of the loop, and are no longer beholden to vendor release cycles, upgrade fees, and huge licensing costs. Morgan owns the source, and has the staff to maintain and enhace their IP. In fact, a friend of mine at Morgan told me that, if they wanted to, Morgan could take CPS and give Tibco a run for their money.

©2006 Marc Adler - All Rights Reserved

CAB, EventBroker, and Wildcards

I will be blogging about the CAB EventBroker soon. But, I think that I like mine (previously published here) better. I would like to see wildcard support in the EventBroker's subscription strings.

In CAB, you can define an event to be published like this:

[EventPublication("event://Trade/Update", PublicationScope.Global)]
public event EventHandler TradeUpdated;

........

public void TradeIsUpdated(Trade trade)
{
if (this.TradeUpdated != null)
{
this.TradeUpdated(this, new InstrumentUpdatedEventArgs(trade));
}
}

In some other module, you can subscribe to an event like this:

[EventSubscription("event://Trade/Update")
public void OnTradeUpdated(object sender, InstrumentUpdatedEventArgs e)
{
}

I need wildcards. I might like to have a function that gets called when any operation happens to a Trade object. So, I would like to see subscription topics like these:

"event://Trade/*"

or

"event://Trade

Both of these would cover the case when any operation happens to a trade. The subscription string would catch the following topic:

event://Trade/Updated
event://Trade/Deleted
event://Trade/Created

I need Tibco-lite as my internal message bus.

©2006 Marc Adler - All Rights Reserved

CAB and Status Bars?

I have not gotten into the Smart Client Factory yet (preferring to learn the underlying CAB framework), so this question might be answered by the SCSF .... but has anyone made a UIExtensionSite for a statusbar object yet?

©2006 Marc Adler - All Rights Reserved

Grid Computing and UBS

Grid computing webcast here, featuring people from UBS, Microsoft and Digipede.

Our head quant tells us that UBS has an internal website that can price exotics amazingly fast. Something that all of us can strive for...

©2006 Marc Adler - All Rights Reserved

Microsoft Single Point of Contact

Microsoft DPE and MCS need to read this

©2006 Marc Adler - All Rights Reserved

CAB and WorkItems

WorkItems

A WorkItem is considered to represent a “use case” in CAB terminology. Ignore this. It is really just a container of other kinds of objects along with some state information.

A CAB application has a tree of WorkItems. The main CabApplication class contains a reference to the root WorkItem, which is referred to by the RootWorkItem property. Given a WorkItem, you can go up one level to its ParentWorkItem, or down to the next level by accessing the workItem.WorkItems collection.

Recall that the main application class is defined like this:

class CABQuoteViewerApplication : FormShellApplication

The first argument in the generic’s argument list is the type of WorkItem that will be the RootWorkItem of our entire application. All other WorkItems will be descendants of this RootWorkItem. A WorkItem has access to any of its descendant’s properties; however, sibling WorkItems cannot access eachother’s properties directly.

A WorkItem also contains various other collections. It has collections of:

• Workspaces
  
• SmartParts
  
• Commands
  
• EventTopics
  
• Services
  
• Items (you can stick any object in this collection, including state, views, etc)
  

A WorkItem can contain a State. This is a named collection of name/value pairs. Events are fired when a value in the State changes. When a WorkItem is saved, its State is saved along with it.

You can Activate/Deactivate a WorkItem, Terminate it, and persist it (Save and Load).

There is a virtual function called OnRunStarted() that you can override in order to create views, read data, etc.



WorkItemExtensions

A WorkItemExtension is a way of extending the behavior of a WorkItem without having to change the WorkItem’s code nor resorting to subclassing the WorkItem. It is a class that just receives certain events that happen to the associated WorkItem. These events are:

Initialized
RunStarted
Activated
Deactivated
Terminated

To extend a WorkItem, you need to create a new subclass of WorkItemExtension. Then you need to create an object of that class; this object is associated with an underlying WorkItem. Then you just handle certain events that happen to the WorkItem.

public class QuoteViewWorkItemExtension : WorkItemExtension
{
  public QuoteViewWorkItemExtension() {}

  protected override OnActivated()
  {
    PerformanceTimer.StartPerformanceTiming();
  }

  protected override OnDeactivated()
  {
    PerformanceTimer.StopPerformanceTiming();
  }
}

And, to use this new WorkItemExtension, you would do the following:

QuoteViewWorkItemExtension wix = new QuoteViewWorkItemExtension();
Wix.Initialize(myWorkItem);

This concept is very much akin to the “advice” that AOP containers like Spring.Net provide for you, except that you do not have any runtime code injection.

©2006 Marc Adler - All Rights Reserved

CAB and Workspaces

Workspaces

You might be familiar with various kinds of layout managers that automatically arrange the windows that the manager contains. If you are a Java developer, you might be used to layout managers like the FlowLayout manager and the GridLayout manager. The layout manager works in conjunction with a container. The container holds the controls, and the layout manager positions and sizes the controls as they are added to the container.

In CAB, we have Workspaces and SmartParts. A Workspace is a container for holding SmartParts. A WorkItem contains a list of zero or more Workspaces, so you can have workspaces within workspaces.

Most CAB applications will need to create a root Workspace within the main form.

The different kinds of Workspaces in CAB are:

• WindowWorkspace
  Vanilla area for holding SmartParts
  For an MdiWorkspace, will automatically create a Form to hold a SmartPart
  
• DeckWorkspace
  Stacks SmartParts in an overlapping manner
  
• MdiWorkspace
  Regular MDI container, derives from WindowWorkspace
  
• TabWorkspace
  Tabbed Windows
  
• ZoneWorkspace
  Allows tiling of window areas, good for implementing an Outlook type layout
  

There are two ways of adding zones. One is to use the Visual Studio .Net designer, and drag a workspace from the Toolbox onto a form.




The other way is to dynamically create the workspace in the FormShellApplication’s AfterShellCreate() override.

Here is an example of creating various types of workspaces using the second method (the code is “unwound” for the sake of this article):

using System;
using System.Windows.Forms;
using CABQuoteViewer.WorkItems;
using Microsoft.Practices.CompositeUI.SmartParts;
using Microsoft.Practices.CompositeUI.WinForms;

namespace CABQuoteViewer
{
    class CABQuoteViewerApplication : FormShellApplication<QuoteViewerWorkItem, MainForm>
    {
        private IWorkspace m_workspace;
        private QuoteViewerWorkItemExtension m_quoteWorkItemExt;

        /// <summary>
        /// The main entry point for the application.
        /// </summary>
        [STAThread]
        static void Main()
        {
            new CABQuoteViewerApplication().Run();
        }

        protected override void BeforeShellCreated()
        {
            base.BeforeShellCreated();
            
            if (this.RootWorkItem != null)
            {
                this.m_quoteWorkItemExt = new QuoteViewerWorkItemExtension();
                this.m_quoteWorkItemExt.Initialize(this.RootWorkItem);
            }
        }
        
        protected override void AfterShellCreated()
        {
            base.AfterShellCreated();
            this.CreateWorkspace("Deck");
        }
        
        private void CreateWorkspace(string wsTypeName)
        {
            if (wsTypeName == "Mdi")
            {
                this.m_workspace = new MdiWorkspace(this.Shell);
                this.RootWorkItem.Workspaces.Add(this.m_workspace, "ClientWorkspace");
            }
            else if (wsTypeName == "Tab")
            {
                this.m_workspace = this.RootWorkItem.Workspaces.AddNew<TabWorkspace>("ClientWorkspace");

                TabWorkspace tabWorkspace = this.m_workspace as TabWorkspace;
                tabWorkspace.Dock = DockStyle.Fill;
                this.Shell.Controls.Add(tabWorkspace);
            }
            else if (wsTypeName == "Deck")
            {
                this.m_workspace = this.RootWorkItem.Workspaces.AddNew<DeckWorkspace>("ClientWorkspace");

                DeckWorkspace deckWorkspace = this.m_workspace as DeckWorkspace;
                deckWorkspace.Dock = DockStyle.Fill;
                this.Shell.Controls.Add(deckWorkspace);
            }
            else if (wsTypeName == "Zone")
            {
                this.m_workspace = this.RootWorkItem.Workspaces.AddNew<ZoneWorkspace>("ClientWorkspace");

                ZoneWorkspace zoneWorkspace = this.m_workspace as ZoneWorkspace;
                zoneWorkspace.Dock = DockStyle.Fill;
                this.Shell.Controls.Add(zoneWorkspace);
            }
            else
            {
                throw new Exception("Cannot create workspace");
            }
        }
    }
}

When you add a SmartPart to a Workspace, you can pass along hints that tell the Workspace how to layout and decorate the SmartPart. The Workspace class has functions for

• Showing a SmartPart (which also adds the SmartPart to the Workspace as well)
• Hiding a SmartPart
• Activating a SmartPart
• Closing a SmartPart

There are events that get fired when a SmartPart is activated within a Workspace, and when a SmartPart is closing within a Workspace.

You can create new, custom workspaces in CAB, and a later article will cover this.

©2006 Marc Adler - All Rights Reserved

The CAB Application Class Hierarchy

CAB Application Classes

The hierarchy of the CABApplication classes is shown in the diagram below. The top three classes should never be derived from. In almost all classes, your WinForms-based application will derive from FormShellApplication.





A CAB Application will be driven by the base CabApplication class. This class does most of the work to get the CAB application started.

The only field that the CabApplication class has is the root WorkItem, which as the name implies, is the root node of the entire application’s WorkItem tree. I will discuss WorkItems later.

The constructor of CabApplication does nothing; it is the Run() method (which is called by your application’s Main() function) that bootstraps the entire application. The Run() method will do the following:

• Create the pre-defined services
  
• Authenticate the user.
  
• Reads the config file (ProfileCatalog.xml) that contains a list all of the plug-in modules that the app wants to load.
  Each module can have an optional list of Roles. The module will be loaded only if the current user belongs to that role. If a module has no role information associated with it, then the module will be loaded.
  
• Creates the Shell (ie: the MainForm)
  
• Loads all of the modules that are permitted to be loaded.
  In each class in the module that derived from the ModuleInit class, all fields that are tagged with the [ServiceDependency] attribute are created.
  All classes tagged with the [Service] attribute are added to the application’s list of services.
  
• RootWorkItem.Run() is called for the applications root WorkItem.
  
• In the FormShellApplication class, the WinForm’s method Application.Run() is called to start the WinForm app.
  

The Derived Application Classes

The CabShellApplication class extends the CabApplication by

• Keeping a reference to the Shell (our MainForm) by adding it to the RootWorkItem’s list of items.
  
• Adding BeforeShellCreated and AfterShellCreated virtual methods. These methods can be overridden in your application class.
  

The WindowsFormApplication extends the CabShellApplication class by

• Using a WindowsForm Visualizer


• Adding some UI and command-routine services

The FormShellApplication class merely overrides the Start() method and calls the familiar WinForms method Application.Run(mainform).


©2006 Marc Adler - All Rights Reserved

A Minimal CAB Application

First CAB Application

1) Create a new Windows Form Application solution named CABQuoteViewer.
a. Change the name of the form to MainForm.

2) Add the existing CAB projects to the solution.
a. Browse to C:\Program Files\Microsoft Composite UI App Block\CSharp\Source

b. Add the existing projects for ObjectBuilder, CompositeUI, and CompositeUI.WinForms.

c. In the CABQuoteViewer project, add references to the above 3 projects.

3) Rename the file Program.cs to CABQuoteViewerApplication.cs

4) In the file CABQuoteViewerApplication.cs,
a. Add references to the CAB namespace
using Microsoft.Practices.CompositeUI;
using Microsoft.Practices.CompositeUI.WinForms;

b. Change the definition of the class from
static class Program
to
class QuoteViewerApplication : FormShellApplication

c. The body of the Main() function should just be
new QuoteViewerApplication().Run();


The final version of CABQuoteViewerApplication.cs is:

using System;
using Microsoft.Practices.CompositeUI;
using Microsoft.Practices.CompositeUI.WinForms;

namespace CABQuoteViewer
{
class CABQuoteViewerApplication : FormShellApplication
{
[STAThread]
static void Main()
{
new CABQuoteViewerApplication().Run();
}
}
}

After compiling and running this application, you will see the empty MainForm appear.

©2006 Marc Adler - All Rights Reserved

Getting Started with CAB

It looks like, for various reasons that shall accompany me to the grave, we will be bootstrapping the "top part" of our client-side framework with Microsoft's Composite Application Block. I am excited to finally be given the chance to learn CAB, and to lead the team doing the new client-side framework for my investment bank.

I will try to document my learning process with CAB so that I can save my successors some pain.


Installation of CAB and Accessories

Download and install the following files in order (make sure to install GAX before installing GAT):

1) Download Composite Application Block for C# (CAB)
2) Download Enterprise Library 2006 (EntLib)
3) Download the Guidance Automation Extensions (GAX)
4) Download the Guidance Automation Toolkit (GAT)
(At this point, before installing the Smart Client Software Factory, you must build CAB using the CompositeUI.sln solution file located in the directory C:\Program Files\Microsoft Composite UI App Block\CSharp. You must also close Visual Studio before installing SCSF.)
5) Download the Smart Client Software Factory (June 2006 version) (SCSF)
6) Download the CAB Hands-on Lab
7) Download the Intro to CAB document and the CAB help files (you might want to create shortcuts on the desktop for these files.)
8) Download the Sample Visualizations

If you have problems uninstalling GAX or GAT, read this.


Resources

· The CAB home on Microsoft Patterns and Practices area on GotDotNet
· CabPedia
· MSDN Magazine Article
· Getting Started with CAB on the Fear and Loathing blog
· Understanding CAB series on Szymon’s blog


Class Hierarchy

Here is a diagram of the major classes in CAB:




©2006 Marc Adler - All Rights Reserved

Free Market Data

Free ECN real-time data from OpenTick.

Very cheap ($1) monthly fees for some other data.

They have APIs in various languages, including C# for .Net 2.0. This seems like a good way to test that async data handling part of a framework. Here is some example docs for one of their calbacks:

static void onQuote(OTQuote quote)
Description
Provides real-time and historical quotes.
Provides
OTQuote


©2006 Marc Adler - All Rights Reserved

Eclipse Trader

A colleague pointed out the open-source Eclipse Trader, built upon the Eclipse Rich Client Platform (RCP).

There exists people in my company whose number > 1 who would be most pleased if I would fully embrace Eclipse RCP.

©2006 Marc Adler - All Rights Reserved

Downloading Yahoo Quotes

Here is a great page for composing URLs to get all sort of delayed quote data from Yahoo

©2006 Marc Adler - All Rights Reserved

Coderific

Ratings for Coders, by the Coders.

©2006 Marc Adler - All Rights Reserved

Windows Shutdown Crapfest

Via Joel, there is required reading on Microsoft here.

I am currently involved with Microsoft's MCS and DPE in my job. I know that many of the people that I interact with are fervent readers of both Joel and of Mini-Microsoft, and that they will eventually read the above-mentioned article.

What has happened to my beloved Microsoft? Thank the lord that we are engaging some of the most talented Microsoft partners on the planet.

©2006 Marc Adler - All Rights Reserved

Object Cache Considerations (Part 3)

Distribution and Subscriptions

An out-of-process object cache should not only have a storage component, but a messaging system as well. One of the architects in my group, who is a well-known messaging guru, told me that the ideal object cache should have a state-of-the-art messaging system attached to it.

Our object caches should be distributed and subscribable.

A logical cache can be distributed amongst several different servers. We can do this for load balancing and for failover. Applications also have local caches that communicate with the master, distributed cache(s).

Let's say that we are storing information about each position that our company maintains. We might want to have 3 distributed caches, one that stores positions for our customers in the US, one for customers in Europe and the Middle East/Africa, and one for Asia. Upon startup, the master cache loader will read all of the positions from the database and will populate each of the three caches.

This is an example of a very simple load balancer for the distributed caches. Other load balancing schemes include partitioning the positions by the first digit of the position id, a date range, etc.

Each application that uses positions will have its own local cache. These local caches will usually contain a subset of the data that is in the master caches. For example, the US Derivatives Desk might just need to cache positions from US portfolios that have been active in the last 30 days.

When an application updates or deletes a position in one of the master caches, we need to update all of the other master caches that we are using for failover purposes, and any other master caches that contain that particular position. Similarly, when we create a new position, we need to propagate that new position to any redundant caches or any caches that might be interested in the new position.

We might need to push the new or updated position to any of the local caches that are interested in that position. We have a choice of architecture for distributing updates to local caches.
1) We do not distribute the updated object at all. An application won’t know that there is new data in the master cache until it retrieves that object again.
2) We push the update to the local caches right away. We can push out full objects or just the deltas (changes to the object).

There are disadvantages of both schemes. Under scheme (1), the application could be working with old data. Under scheme (2), we could be updating an object in the local cache while the application is working on that same object. Also, under scheme (2), we now have to worry about messaging more.

The master caches have to have some way of communicating with the local caches. We can communicate with each application by one of the familiar messaging mechanisms; Tibco EMS, Tibco RV, LBM, Sockets, etc.

We need to make sure that the messaging is reliable. Each subscriber must receive the update of the object from the master cache. There is no tolerance for dropped messages. Otherwise, different applications might be working with different versions of an object.

We do not have to make the message durable. In other words, if a client goes offline for a while, then the messaging part of the cache does not have to save the update until a time where that client decided to reconnect. So, this saves us the need of storing out-of-date messages.

Using a JMS-based messaging scheme also means that we can use JMS Selectors to filter out objects that an application is not interested in. Selectors have overhead with them, but it is easy to set up a filter-based pub/sub mechanism between the master caches and any local caches. For example, one application might only be interested in updates to position objects whose position id starts with the prefix “A23”. It is easy to set up a JMS selector that has the pattern “positioned LIKE ‘A23%’”.

©2006 Marc Adler - All Rights Reserved

So long Mike

Our friend Mike was a 25-year veteran of the Financial Service practice of a very large consulting firm. He was just informed that he was to be dismissed, on the basis of not making his (unrealistic) sales quota. The new management of the division and Mike did not agree on things, and the easy way to get rid of a person is to set him up for failure.

Mike will easily find a new position, as he is known and respected in the financial services area. And, opportunity is also borne out of adversity.

Know the warning signs when you are being set up to fail. Miike saw them a mile away, and was prepared.

©2006 Marc Adler - All Rights Reserved

SQl Server 2005 Performance Tips

Here

Good performance hints if you are doing a lot of OLTP processing. Most of the tips will work for Sybase as well.

It is remarkable what performance improvements you can make in an OLTP system if you hire a true database tuning expert to go over your systems with a fine-tooted comb. Unfortunately, the tuning does not usually extend into refactoring the data model. This is because a lot of apps touch the databases, and you would then have to go in and start refactoring app code.

SQL Server 2005 comes with new, easy-to-use profiling tools, and I encourage you to take advantage of them as you are developing new apps. If you are a dev manager, try to get funding for two weeks of a SQL Server 2005 expert's time once your data model is written.

In addition, try to get your SANs tuned correctly, and optimize the interaction between frequenty-used database indexes and the spindles.

How about hardware acceleration, like Solid State Disks?

Kudos go out to this site, completely devoted to SQL Server performance tuning.

Final word - if you have a mission-critical application that involves heavy database access, spend the bucks and hire a DB architect who knows how to tune databases like a 1964 Fender Strat.


©2006 Marc Adler - All Rights Reserved

Saudi Stock Market Collapse

I had no idea that this was going on....

©2006 Marc Adler - All Rights Reserved

Object Cache Considerations (Part 2)

Object Versioning

You can consider the ‘version’ of an object to be two different things.

In the first case, the ‘version’ of an object could represent the number of times a particular object has been written to. When an object is first created, its version number is set to 1, and then each time a client updates the object, the version number is increased. So, we can have an API call is our object cache that tests to see if we are holding on to the most recent version:

if (!ObjectCache.IsCurrent(object))
object = ObjectCache.Get(object.Key);

or, if we are using a object that has a proxy to the cache, we can do something like this:

if (!object.IsCurrent())
object.Refresh();


In the second (and substantially more complex) case, the ‘version’ can represent the actual layout or shape of an object’s class. Consider a Trade object:

public class Trade : CachedObject
{
public int SecurityId;
public double Price;
}

This would be version 1.0 of the class.

Let’s say that we have a trading system that has to run 24x7, as is the current rage. Systems that run 24x7 theoretically have no chance to be bounced. Even a system that runs 24x6.5 has a window for maintenance. Our trading system has version 1.0 of the Trade object.

Now let’s say that we have a request to add sales attribution to the trading system, so now we need to add the id of the sales trader that took the trade request.

public class Trade : CachedObject
{
public int SecurityId;
public double Price;
public int BrokerId;
}

This is now version 1.1 of the class.

Our object cache holds version 1.0 objects, and all of our subscribers also hold version 1.0 objects. But, now let’s say that the system that writes new trades into the object cache now has to write version 1.1 objects. What do we do?

There are several things to consider here. How do we represent the object in the cache? Because we are using name/value pairs, all new objects will just have the BrokerId/ field added. The old 1.0 objects that are in the cache do not have to change.

The object cache might want to broadcast a message to all subscribers, telling them that the version number of the Trade object has changed. Since the subscribers may be systems that must run 24x7, then the systems might not be able to be bounced in order to rebuild their trade caches. The systems must be able to read and write the new version 1.1 objects as well as continue to support the older 1.0 objects. But, we cannot reconfigure the layout of the objects dynamically, can we?

Instead of using C# objects, we might consider using dictionaries of dictionaries to represent an app’s object cache. But this is a different kind of programming model. Instead of coding:

Trade obj = ObjectCache.Get(“102374”);
int broker = obj.BrokerId;

We might have to do the following (taking advantage of C# 2.0’s nullable types) :

TradeDictionaryObject obj = ObjectCache.Get(“1002374”);
int? broker = obj.GetInt(“BrokerId”);

What a mess!

What does this tell us? When using an object cache for a 24x7 system, make sure you get your class definition right the first time, and avoid object versioning!

©2006 Marc Adler - All Rights Reserved

Considerations for an Object Cache (Part 1)

Let’s say that we wanted to write our own multi-platform, distributed, subscription-based object cache. What would we need to do to write the ultimate object cache?

Let’s consider a variety of issues that we would have to consider when writing caching middleware. I am sure that vendors like Gemstone have gone through this exercise already, but why not go through it again!

Multiplatform support

Most Investment Banks have a combination of C++ (both Win32 and Unix), C#/.Net and Java (both Win32 and Unix) applications. It is common to have a .Net front-end talking to a Java server, which in turn, communicates to a C++-based pricing engine. We need to be able to represent the object data is some sort of form that can be easily accessed by applications in all of the various platforms.

The most universal representation would be to represent the object as pure text, and to send it across the wire as text. What kind of text representation would we use?

XML – quasi-universal. We would have to ensure that XML written by one system is readable by other systems. XML serialization is well-known between Java and C# apps, but what about older C++ apps. For C++, would we use Xerces? Also, there is the cost of serialization and deserialization, not to mention the amount of data that is sent over the wire.

Name/Values Pairs – easy to generate and parse. Same costs as XML. We would have to write our own serialization and deserialization schemes for each system. How about complex, hierarchical data? Can simple name/value pairs represent complex data efficiently? Or would we just end up rewriting the XML spec?

Instead of text, we can store objects as true binary objects. What kind of binary object do we store? Native or system-agnostic? If you have a variety of platforms writing into the object cache, do we store the object in the binary format of the system that created the object, or do we pick one platform and use that as master?

Master Format – We pick one format, either C++, C#, or Java binary objects. We would need a series of adapters to transform between binary formats. We would also need an indication of the platform that is doing the reading or writing. Let’s say that we were to store all objects as binary Java objects. If a Java app reads an object, then there would be no costs associated with object transformation, so we can just send a binary Java object down the wire (although we may have to worry about differences between the various versions of Java … can a Java 1.5 object with Java 1.5-specific types or classes be read by a Java 1.4 app?). If a C# app wants to read the Java object, then we must perform some translation. (Do we use something like CodeMesh to do this?) We also need to ensure that the adaptors can support all of the features of the various languages. For example, let’s say that Java came up with a new data type that C# did not support … would we try to find some representation of that type in C#, or would we just not translate that particular data type?

Native Format – We store pure binary objects, without regards to the system that is reading or writing the object. There is no translation layer. Apps are responsible for doing translation themselves. This is the fastest, most efficient way of storing objects. However, different teams might end up writing their own versions of the translation layer.

What other factors might we consider when choosing a native object format?

How about deltas in subscriptions? If we are storing large objects, then we might only want to broadcast changes to the object instead of resending the entire object. Delta transmission favors sending the changes out in text, and we can save the cost of translating the binary into text if we were just to store the objects as text. And, in this case, name/value pairs are favored.

Large sets of name/value pairs can be compressed if necessary, but we have to consider the time needed to compress and decompress.

Can our object cache store both text and binary? Sure, why not. We can tag a cache region as supporting binary or text, and have appropriate plugins for various operations on each.

As always, comments are welcome.

©2006 Marc Adler - All Rights Reserved

Tourist Warning : City Pride

While staying in Canary Wharf last week, a colleague from Microsoft and I went to the City Pride pub, which is one of the few pubs around the area, and close to the Hilton.

After sitting at a table for 15 minutes, and not being served, some kind soul told us that you actually has to go up to the bar to order your beers. (Warning #1). We Americans like to be coddled, and demand waitress service.

Then I ordered a Black and Tan (Guiness and Bass, standard fare at any Irish pub in New York). The bartender had no idea what I was talking about (Warning #2).

To top it off, the bartender gave me the bill, and said that we could feel free to add a tip onto it ... which I did (a 10%, 2 Pound tip) (Warning #3 ... I was told the next day never to tip the bartender).

I guess this was my Lost In Translation moment that every tourist has when visiting a foreign country .... even though Bush treats Britain as our 51st state. (Yo Blair!)

©2006 Marc Adler - All Rights Reserved

Wanted : JMX to .Net Bridge

We need a way for a .Net GUI to speak JMX to a Java server. Anyone come up with anything yet?

Doing a Google search, it looks like we are not the only ones with that need.

Has anyone checked out the WS-JMX Connector?

©2006 Marc Adler - All Rights Reserved

A Typical Interaction Between Me and My Colleagues

©2006 Marc Adler - All Rights Reserved

Office 2007 Compatibility Pack is available

If you are like me, and you keep getting these Office 2007 files sent to you by your local Microsoft reps, but you are only running Office 2003, then you need this

©2006 Marc Adler - All Rights Reserved

What does a Database Architect do?

Even though I was the very first developer on the Microsoft SQL Server Team, I have to admit that databases don't thrill me .... You have to have a special mindset to deal with databases all day, and to tell you the truth, my interests lie elsewhere. In fact, the sure way to get me to fail an interview is to ask me to write any moderately-complicated SQL query.

I firmly believe that, for major systems, the developers should not be allowed to design the data model, set up the databases, nor write the DDL. I have seen a number of instances in the past where systems that have had their database components designed by non-database experts have performed very poorly. Slow queries, no indexes, lock contention, etc. The best projects that I have been involved in have had a separate person just devoted to the database. If I am leading a major project, I will always have a dedicated db expert as part of the effort.

Let's say that we want to hire a database expert for our Architecture Team. What duties would they have?

1) Advise teams on best practices.
2) Come up with DDL coding standards.
3) Review existing systems and provide guidance on performance improvements.
4) Know the competitve landscape (Sybase vs SQL Server vs Oracle) and affect corporate standards.
5) Be expert at OLAP, MDX, Analysis Services, etc.
6) Know how to tune databases and hardware in order to provide optimal performance.
7) Advise all of the DBAs.
8) Monitor upcoming technologies, like Complex Event Processing, time-series databases, etc. Be familiar with KDB+, StreamBase, Vhayu, etc.

A Database Architect is a full-time job that I think that all Architecture groups should have a slot for.

Know anyone who wants to join us?


©2006 Marc Adler - All Rights Reserved

DrKW and Cross-Asset Trading

Dresdner folded its much-hyped Digital Markets division, which they liked to view as their "Bell Labs" of DrKW. All of the major players involved in the Digital Markets group have left or are in the process of leaving.

According to the DWT newsletter, there were several charters to the Digital Markets group:

1) Provide synergies across all lines-of-business at DrKW, and stop the siloing.
2) Provide a system for cross-asset class trading.

Eugene Grygo, the editor of the DWT newsletter, devoted his "Before the Spin" column to this news, and brought up some questions with regards to the future of DrKW. In particular, what will happen to the dream of cross-asset class trading? Grygo mentions that HSBC is actively exploring this space, and I know a few other IBs doing the same. Is it impossible to coalesce the silos and provide true cross-asset class trading? If it is technically feasible, then is it politically feasible?

In these times where everyone is predicting the reduction of traders due to automation, will cross-asset trading be the last field of battle as the silos struggle to maintain their autonomy?

I also wonder what becomes of DrKW's grid project. Maybe Matt or Deglan can illuminate us...

©2006 Marc Adler - All Rights Reserved

Grid in Financial Markets

A presentation by JP Morgan on their use of Grid Computing.

Here is a page of PDF's from a February 2006 conference in Italy on Grid Computing in financial markets. There is even a paper on using grid for semantic analysis of financial news feeds. I need to get our London team to read some of this stuff.

There must be synergies between Complex Event Processing and Grids. Anyone looking at this space?


©2006 Marc Adler - All Rights Reserved

Mini-Guide to .Net/Java Interop

Here

Terry is sure to have this stuff in our Wiki before I get to London :-)

©2006 Marc Adler - All Rights Reserved

Pricing Turbo Warrants

Here

Today was the first I have ever heard about Turbo Warrants.

A Turbo Warrant call is:
- a barrier knock-out option paying a ...
- small rebate to the holder if the barrier is hit and
- with the barrier typically in-the-money.

©2006 Marc Adler - All Rights Reserved

The Bile Blog

The Bile Blog is even most cynical and caustic than I am.

Thanks to the ThoughtWorkers at my office for pointing this out. Especially funny are the posts that harpoon the Fowlbots. And, I think that he has even biled yours truly.

This blog is going to take me quite a while to go through ... would be great if Virgin Atlantic had internet connectivity ... I think reading this blog would occupy my entire flight.

By the way ... kudos to our own Fowlbots ... James, Chris, Dave and Alistair. Job well done, boys!

©2006 Marc Adler - All Rights Reserved

C++/CLI Opposition

C++/CLI == C++ Divided By CLI

Here

Interesting reading, especially in the light that investment banks have a ton of old Visual C++/6, non-MFC code. The choice is to go to C++/CLI, or start fresh with C#.

©2006 Marc Adler - All Rights Reserved

I am scared of The Wharf

Earlier this year, my former consulting company closed its London office without warning. In a mailing to the staff, the partners said that there was just no business to be had for very smart .Net and Java consultants in Canary Wharf.

I am imaging what kind of place this Wharf could be. When I get there, will I see a lone seaman, in a yellow raincoat, yelling "Ahoy Matey" to me from a distant pier? Are there thugs and holligans behind every dark corner, waiting to roll me for my wallet? Will I see row of cars, inhabited by hormonal teenagers, "watching the submarine races" (a popular saying in the 1950's)?

I am very afraid of this Wharf place. Perhaps JohnOS, Matt, Deglan and Pin can organize a bile-night to keep me off the streets? Perhaps Rut the Nut is reading this blog and will get some of our old Citicorp/EBS gang together for a slosh-up?



©2006 Marc Adler - All Rights Reserved

Funny Code

What is the funniest line of code you ever saw? One of the developers at Barcap wrote the following: line in a C# function

if (this == null)
{
}

It absolutely cracked me up (well... I guess you had to be there).


©2006 Marc Adler - All Rights Reserved

London Bound

I will be in London all next week, checking out the happenings in Canary Wharf. I have not been to London for a long time, so it will be interesting to see how built up the Wharf is. More interesting will be to see if they built any quality pubs. Last time I was in London, all of the pubs closed at 11 at night.

One nice thing will be to test our Virgin Atlantic's Business Class. I have been hearing tales of free massages .... hmmm ...

©2006 Marc Adler - All Rights Reserved

Separated at Birth

Chris and I came from the same company, and we were both equally dissatisfied with the kind of body shop work that we were doing at our last client. We both struck out simultaneously to find a place where we could do meaningful work. We landed in very similar positions at two of the largest investment banks in the world.

Funnier still is that we share almost the same technology stack, from front to back. I am sure that right after the conference call that we have with our market data infrastructure vendor, Chris is on the same call an hour later.

Craig mentions that there are only about 200 market data specialists in the whole world. Everyone probably knows what eachother is doing. The same thing goes on in Wall Street, and by extension, the City. The after-bonus-shuffle will take place in another 4 months, and it's an opportunity for every company to find out what every other company is doing.

When you come down to it, we all have very similar technology stacks. We all know which vendors are out there, and we all have done the same kind of performance and stressing comparisons between similar vendors. The goal is to squeeze that one extra millisecond of performance so that your order is hit before your competitor's order. It could be the difference of one extra 'lock' in a piece of code. It will be a race to recruit the Joe Duffy's and Rico Mariani's of the world. All of the IBs will need to recognize the need for these kind of people, and adapt themselves so that these people will not feel stiffled within a Wall Street environment.

©2006 Marc Adler - All Rights Reserved

NCache and db4o

Matt is looking at NCache and Chris recommends db4o. Plus Geva is telling us to look at Gigaspaces.

Has anyone does an in-depoth comparison between NCache, db4o, Giga, Gemfire, Tangasol, and any others? Roll-your-owns are also welcome in the comparison.

Hey, hey Microsoft ... what 'ya got cookin'? How 'bout cooking something up for me? Here is the JCP-107 spec. Don't deviate too far from it.

©2006 Marc Adler - All Rights Reserved

.Net - Java Integration

http://www.infoq.com/articles/java-dotnet-integration-intro

Especially interesting are the comments from Roger Voss, formerely of the Aldus Pagemaker portability team. He has chosen .Net for the front end and Tibco EMS as the messaging layer between his Java and .Net tiers.

©2006 Marc Adler - All Rights Reserved

Free .NET Object Cache

Free .NET object cache at CodeProject.

Wonder if the code has been touch for a while, or whether anyone is using this cache...

©2006 Marc Adler - All Rights Reserved

Concurrency, Joe Duffy and Wall Street

Joe Duffy is soliciting ideas for a new book on concurrency.

Joe's name comes up a lot my talks with Microsoft team that supports me. He and Rico Mariani are two resources from Redmond that I would love to get on an advisory basis. My vision is that they would help us out with optimal .Net architectures for low-latency, high-performance systems.

The elephant in the room that everyone is worried about is the projected Opra feeds of 456,000 messages per second. As I am the Global Architect for Equity Derivatives for one of the largest investment banks in the world, this is something that I am responsible for. As such,the things I need for our stack include high-speed market data feeds with caching and conflation, complex event processing, hardware acceleration, object caches, threading models, efficient GUIs and client-side frameworks, hardware acceleration, super-tight code generation, efficient messaging between components, (and did I mention hardware acceleration?).

From a concurrency point of view, I would like to know what Microsoft considers to be best practices to implement quasi-real-time data processing. Also, what anti-patterns exist and what to avoid in the .Net framework. How to use PerfMon and other third-party performance tools to get the most out of my systems. Coding rules that maximize performance. Things that Microsoft will be releasing (or thinking about) five years down the road.

And, for my own selfish reasons, I would love to see Joe implement a trading system to test out his ideas!


©2006 Marc Adler - All Rights Reserved

Financial Systems Books

I just got the new Third Edition of the classic After the Trade is Made. At a first glance, it looks like they have fattened up the book a bunch, and added a chapter on trading systems. I will report later on the quality of this chapter if I can get a block of free time to read it.

A decent companion to this book is Practical .Net for Financial Markets. I ordered this book after reading my colleague Ted's glowing review of the book on Amazon. I have to admit that, for me, the book was a bit on the disappointing side. The book has a little bit of everything ... a simple crossing engine, some messaging (designed to illustrate how STP works), some encryption, some good example of networking, etc.

However, what seems to be missing from the marketplace is a great book that deals with the entire spectrum of financial instruments from a developer's point of view. Something that will not only discuss the business domain and underlying technology, but will also point to real products that implement the various systems. A deep dive for the developer to become completly immersed in Wall Street systems.

For example, let's take Equities. From a systems standpoint, I want to know what goes on from the time that a trade is entered until someone receives confirmation through snailmail. I want to know what an order entry system does, how trades are routed to the exchanges, how FIX messaging is used, how crossing engines and auto-execution engines work, how stat arb and algorithmic trading factors in, how market-making functions, how settlment is done, how positions are maintained and how P&L are calculated, how market data gets into a system, how risk is calculated, etc. I want to know how systems and vendors like Bloomberg, Fidessa, Reuters, Wombat, Vhayu, etc fit into this space.

In addition, I would like to see topics that are tangental in nature, but geared towards the financial systems developer. How to develop low-latency systems. How to write UI's with fast-updating grids of market data. How to use complex event processing to implement stat arb trading. How to do order routing efficiently using rules engines.

I want a complete end-to-end picture. I want FinancialSystemsPedia.

My former colleague Matt was thinking about writing this kind of book several years ago. I think that there is a real need for this kind of knowledge to be put on paper.

©2006 Marc Adler - All Rights Reserved

Good Luck to Matt Devlin

One of our favorite poachers, Matt Devlin, has just left Finetix to strike out on his own.

Good luck Matt (but stay away from my team!). Poaching is one of the toughest jobs on The Street, especially when it is so close to bonus season.

©2006 Marc Adler - All Rights Reserved

Shortage of Indian Engineers

From today's New York Times:

As its technology companies soar to the outsourcing skies, India is bumping up against an improbable challenge. In a country once regarded as a bottomless well of low-cost, ready-to-work, English-speaking engineers, a shortage looms.

India still produces plenty of engineers, nearly 400,000 a year at last count. But their competence has become the issue.

...found only one in four engineering graduates to be employable. The rest were deficient in the required technical skills, fluency in English or ability to work in a team or deliver basic oral presentations.



©2006 Marc Adler - All Rights Reserved

.NET/C# Trading System

RightEdge is a trading platform. Lets you construct C#/.NET based trading systems. Full object hierarchy here.

©2006 Marc Adler - All Rights Reserved

Amazing Aeronautical Charts Site

http://skyvector.com/

If you are a real pilot or a virtual one, then this is the site you want to go to in order to see charts.


©2006 Marc Adler - All Rights Reserved

Assorted Blog News

A better Perfmon here

Free .Net 3.0 training here


©2006 Marc Adler - All Rights Reserved

Wall Street meets Hollywood

From the New York Times article here.

Hedge funds and Wall Street investment banks are plowing money into Hollywood films, paying producers like Joel Silver and Ivan Reitman to produce hits.

I guess that we have to write a market data source that monitors the turnstiles of each of the movies that we invest in. Craig, can you come up with a feed handler in a week?

(I wonder how we will hedge this .... Long Ivan Reitman and short Delta House?)

©2006 Marc Adler - All Rights Reserved

Wanted: Easy way to set multiple breakpoints in VS 2005

I am tracing through some legacy code where a certain class has about 20 different constructors. I would like some way where I can tell VS.NET 2005 to set a breakpoint on each of the constructors.

Likewise, given the name of a method, I would like a way to set a breakpoint at the entry to all of the variants of the method. For instance, if I have a method called GetValue(), and we have a large number of polymophic methods such as

void GetValue(ref bool b)
void GetValue(ref int i)
void GetValue(ref double d)
void GetValue(ref string s)

there should be a way to set a breakpoint at the entry point to each one the GetValue() methods with one one right-click of the mouse.

Sounds like a possible enhancement for Resharper? Or maybe, someone has written a VS macro to do this?


©2006 Marc Adler - All Rights Reserved

Fall Foliage

I flew up to Bennington, Vermont yesterday. The fall foliage is absolutely magnificent. There is no place in the United States like Vermont.

It's about a 1-1/4 hour flight from Morristown (MMU) Airport to Stephen Morse Airport in Bennington. Fly over the Catskills to Albany, then make a right turn at the 083 radial of the Albany VOR. It's about another 10 minutes.

©2006 Marc Adler - All Rights Reserved

What are you doing with your MFC Apps?

An awful lot of IBs still have a lot of front ends written in Microsoft Foundation Classes (MFC).

1) Are you considering MFC to be an end-of-life product?

2) What are your plans for migrating the front ends to other frameworks? Are you going to .NET/C#, .NET/C++, or Java?

3) How will you be doing the migration? Are you devoting a year's worth of time to rewritting the app from scratch? And, while you are doing that, will you be refactoring all of the business logic that you wished that you ha not embedded in the GUI into a new middle tier?

4) Are you going straight for Vista/WPF for your rewrite?

5) Do you wish that Microsoft had a strategy for upgrading your massive MFC codebase to .NET?

Update: John responds here

©2006 Marc Adler - All Rights Reserved

Microsoft ESB News

http://www.intelligententerprise.com/showArticle.jhtml?articleID=193104592

©2006 Marc Adler - All Rights Reserved

Wanna put your quants out of business?

Courtesy of the new Google Code Search is ....

QuantLib.Net



©2006 Marc Adler - All Rights Reserved

Sungard's Grid for Analytics

It was brought to my attention today that Sungard has Adaptiv Analytics, a calculation framework that is grid enabled. The marketing literature says that it has pricing and risk simulations right out of the box.

What might be attractive about this is :

1) Sungard has a long history of dealing with financial companies

2) The product seems to be tuned for a specific vertical industry and a specific purpose .... to speed up pricing and risk.

3) They claim to be based totally on .NET !!!!!!!!!!

Anybody using this thing yet? Wonder what the other grid vendors have to say about this?


©2006 Marc Adler - All Rights Reserved

Thanks, Mark Pollack

The co-head of the Spring.Net consortium gave very generously of his time and gave a 2-hour lecture on Spring to my company. There are interesting things coming down the pike for Spring.Net, especially along the areas of messaging.

An ex-colleague of mine was developing a GUI framework for Spring.Net, but now that he is gainfully employed by a major IB, I wonder what will become of that effort. I thought that it could give CAB some real competition. Matt is also delving deep into the CAB world. Wonder if he will take over the framework.

©2006 Marc Adler - All Rights Reserved

Newest Joshi Paper

Thanks to Mark Joshi for pointing out that he has an updated version of his paper on "A Day in the Life of a Quant".

http://www.markjoshi.com/downloads/advice.pdf

Now maybe I'll be able to talk to Ryan! (Inside joke)

©2006 Marc Adler - All Rights Reserved