Monday, August 29, 2011

Git and Jenkins working

This blog post helped me get running with Git and Jenkins:

The key was finding the Plugins Management page within the local Jenkins monitoring page.

So, my build environment is now set up, and consists of

  • Eclipse Helios (IDE)
  • Maven (Build and dependencies)
  • Git (Source control)
  • Jenkins (Continuous Integration)
Next step is to install JUnit and write a few unit tests.

©2011 Marc Adler - All Rights Reserved. All opinions here are personal, and have no relation to my employer.

Maven and m2Eclipse

The continuing saga of the ".NET Guy" exploring the world of Java....

After doing some cleanup from Hurricane Irene (my 2 300-foot trees remained rock-steady during the storm), I sat down with my task for the day, and that was to get familiar with Maven. In order to shield myself from command-line arguments and XML configuration files, I downloaded a very wonderful called m2eclipse. This tools integrates Maven and Eclipse pretty nicely, and it looks like that I can do all of the basic dependency management through this tool.

Maven and m2Eclipse are two more things that a .NET developer has to get use to, since Visual Studio seems to manage most of the dependency and build process. However, one nice thing about the entire Open Source movement is that you can create a maven dependency, automatically download the source code for the dependency, and download the source for all of the sub-dependencies. Also, the Maven Repositories provides a nice, centralized way of locating frameworks and tools from within Eclipse.

I also have Git working nicely, and the Eclipse integration is being provided by the eGit tool. Much nicer than Clearcase.

Now, I need to download and learn Jenkins for my Continuous Integration needs.

Are there any other Eclipse-based tools that you recommend? I would love to hear about them.

©2011 Marc Adler - All Rights Reserved. All opinions here are personal, and have no relation to my employer.

Sunday, August 28, 2011

Excursions Back into the World of Java

I did some Java development when it first came out. Honest I did. I even taught Java classes in the late 1990's at Bellcore, which was a division of AT&T.

I have managed teams that have Java-based systems. And, as part of the CEP system that I did, I had to go and write Java-based web services and deliver a Java API so that other groups could hook into the Notification Services.

Despite that, and because of my long history with Microsoft technologies, I am often viewed as "The Dot Net Guy".

So, this week, I broke out the Java manuals again, and I decided to write the Exchange Simulator portion of my little project in Java 1.6. Writing the simulator was fairly easy, and in little time, I had FIX messages being passed back and forth between my Java-based Exchange Simulator and my .NET-based OMS.

Java is not that much different than C#. They even have Reflection, something that I sorely missed when I had to do C++ in the past year. I wish that Java supported C# auto properties and C# events (is this stuff coming in Java 1.8?), but other than that, it feels like programming in C# 1.0.

However, the Java language is only one small part about being a true Java developer and architect. There is an entire ecosystem that surrounds Java development. I got a list of tools and frameworks to learn from one of my friends, and there is a lot of stuff in there. This is a list of the basic tools that one needs:

  • JDK 1.6
  • Eclipse 3.6 (Java EE Edition)
  • JTA/JTS (transaction services)
  • Spring 3.0 or 3.1
  • Hibernate 3.6
  • Open JMS or ActiveMQ
  • JBoss, Jetty, or Apache Geronimo
  • Subversion
  • Jenkins or Hudson
For the past year, I have been using Eclipse for my C++ development, editing on Windows and compiling on Linux. Editing, building and debugging Java apps using Eclipse seems to be a breeze. Eclipse has a lot of the real-time error detection built into it that I am used to with Resharper in Visual Studio. The incremental compilation helps, and it takes almost no time to go from editing to running an application.

I have experimented with Spring in various forms over the years (mostly Spring.Net), in addition to writing my own dependency injection framework. Plus, my recent experiences with Prism has gotten me back into the world of IOC (although I am annoyed that Prism/UnityContainer does not support any kind of "IDisposable" pattern when your modules are unloaded when an app closes). I cannot integrate Spring 3.1 into my app just yet because of some logging dependencies that are documented (QuickFix/J uses SLF4J, which is not compatible with Spring).

(N)Hibernate is also another ORM that I have looked at over the years.

As far as source control, I have always been a TFS/Perforce user. The past year and a half was spent in Clearcase, which seems a bit ancient, although it seems to work. Many people seem to prefer Subversion, so this is a good opportunity for me to learn it.

The same goes for Continuous Integration servers. I have been using Cruise Control .NET for all of my CI needs, and Hudson/Jenkins is something that I have heard mentioned for a while now.

The only piece of this stack that is totally alien to me is the Java Application Server. The whole concept of a Java Application Server is a bit new to me, as we really don't have an equivalent in the .NET world. I am not interested in the web server capability of these containers, so I need to find out the other capabilities that I can leverage. This morning, I am going to download JBoss as a first step.

(Update - It looks like you need to purchase a $99 subscription for JBoss. So, I will download Jetty, even though I am not sure that Jetty is a complete Java Application Server.)

(From the Stack Overflow post, where is a great summary of what a JAS is:
A Java EE app server is a transaction monitor for distributed components. It provides a number of abstractions (e.g., naming, pooling, component lifecycle, persistence, messaging, etc.) to help accomplish this. Lots of these services are part of the Windows operating system. Java EE needs the abstraction because it's independent of operating system.)

I will keep everyone posted on my progress.

©2011 Marc Adler - All Rights Reserved. All opinions here are personal, and have no relation to my employer.

Monday, August 15, 2011

First Steps in WPF and Prism

As I mentioned in my last posting, I want to spend a little time to better educate myself on WPF and Microsoft Prism. I see both of these being used more and more in Capital Markets applications, and I want to see if these two will truly allow me to build better applications faster.

I spent the last few days on a little sprint to come up with a prototype of a "Quote Workbench" that I could eventually expand into an algo trading platform or a backtesting platform. All of this code was written from scratch. My only tool was Visual Studio 2010. There are no third-party commercial controls involved, so that if you wanted to download this code and build it yourself, you would not have to purchase any additional software.

I ended up using two small DLLs that I found on CodeProject and CodePlex. One was a set of themes for the Microsoft WPF DataGrid, and the other was a color picker control that I needed to enhance in order to support WPF Databinding.

I am also curious to see if the Microsoft DataGrid can keep up with a high volume of real-time quotes. Most of the time, there is some sort of quote throttling that goes on before the quotes ever reach the grid. The general rule of thumb is that human eyes can only follow two updates per second. And, unless your GUI has some sort of CEP system or business rules embedded in it, you can usually get away of throttling quotes to the GUI. Nevertheless, I am curious to see if WPF-based grids can keep up with a high volume of quotes. From what I have heard, some of the commercial grids (DevExpress, Syncfusion, Infragistics, Xceed) are more performant that the out-of-the-box Microsoft DataGrid, but I do not have these commercial UI toolkits at my disposal right now.

From my recent experience, it seems that most developers are wrapping Winforms grids in a WindowsFormsHost control, and are not going to native WPF grids. If you have any opinions on the performance of Winforms vs WPF grids, please post a comment here.

The idea for this sprint was to write two applications, a quote server and a quote client. The quote server spits out two types of quotes, Level 1 and Level 2 equity quotes. The quote client consumes these quotes and displays the Level 1 quotes in a WPF DataGrid and displays the Level 2 quotes in a standard Depth-of-Book control.

Since the overall theme of this exercise was to learn as much WPF and Prism as possible in the shortest amount of time, I did not pay attention to the way that the GUI looks. Nor did I try to optimize the internal plumbing. A lot of the application was designed to explore as much of WPF as possible, especially aspects surrounding Data Binding. My goal for Prism was to have as much loose coupling as possible.

The first task was to write a Level 1 Equity Quote simulator. This is always a useful thing to have in your bag of tools. I preloaded a quote cache with a few symbols, and implemented a few strategies with which I could pump out quotes. Right now the strategies are limited to Round Robin and Random. A next step would be to download a list of the S&P 500 and corresponding Average Daily Volumes (ADV) for each instrument. A new strategy would be to generate quotes based on the weighted ADVs.

I also put GUIs in both the quote server and quote client so that I can see if there is any noticeable latency in delivering quotes from the server to a GUI. Of course, since I am delivering it between processes on my laptop, there shouldn't be any latency. But, if I decide to experiment with Comet and deliver the quotes across the Internet to a web-based GUI, then it's useful to have a monitoring GUI attached to the quote server.

The Level 1 quote simulator is fairly dumb right now. All it does is push out a lot of quotes at a user-specified rate. Additional improvements to the quote simulator would be

  • Let the client subscribe and unsubscribe to specific quotes
    • Use the FIX QuoteRequest  message for this
  • When no clients are interested in a symbol, automatically stop publishing
  • Use something like Google Finance or Yahoo Finance to seed the initial quotes with realistic values

Once I got the quote simulator working, and the quotes updating the server-side DataGrid in real-time, I wanted to send the quotes from the server to the client. For that, I decided to publish the quotes using the FIX Protocol. I downloaded a copy of QuickFIX.Net, and created a Prism module that would take a stream of internally generated quote objects and push them across the wire to a FIX subscriber. Generally, you would not use FIX to push across a high volume of equity quotes. But, since this is a toolbox for experimentation, and since we want to stick with some free tools that implement a quasi-standard, let's just use FIX.

Note - The QuickFIX assembly is built with .NET 2.0, so we need to add  useLegacyV2RuntimeActivationPolicy="true" to the app.config file for any application that uses QuickFix.NET.

  <startup uselegacyv2runtimeactivationpolicy="true">
    <supportedruntime sku=".NETFramework,Version=v4.0" version="v4.0"/>

Now I had Level 1 equity quotes flowing between the client and server, I wanted to start publishing Level 2 quotes. I wrote a simple Book Generator that, given a certain symbol, generates an entire new book at a specified interval. Of course, this is not how the real world works. Generally, when you receive a book, you receive the initial image of the book and then incremental updates. The job of the book builder module would be to build up a complete order book for all of the subscribed instruments, and ship out deltas to the GUI. The GUI would then need to insert new rows in the book, delete rows in the book, or modify rows in the book. I will leave this for a later exercise.

How can I transmit the book from the server to the GUI? I coded up a quick WCF PubSub mechanism using the NetTCP binding. The book was transmitted across the wire as a SOAP message. Not very good for performance, but OK for interoperability in case I want to write an iOS-based or web-based GUI.

Improvements for the Level 2 Simulator are similar to the improvements I need to make to the Level 1 simulator, and include:

  • Improve performance - use binary encoding where possible, and investigate WCF-specific improvements
    • Investigate UDP
  • When a client subscribes to a symbol, send the full image and deltas, not the entire book.
  • The pub-sub "topics" should contain information about the specific security
  • Let the client subscribe and unsubscribe to specific quotes
  • When no clients are interested in a symbol, automatically stop publishing
  • Use something like Google Finance or Yahoo Finance to seed the initial quotes with realistic values
  • In the GUI, let the user switch between aggregated and non-aggregated views of the book. In an aggregated view, we aggregate the volume at each price point.

A screenshot of version 0.0001 is included below. Just a reminder that the GUI was coded so that I could explore different aspects of WPF and Prism, and is intended to be a test bench for learning.

What's next?

Probably a very light exchange simulator. I don't have access to a high-quality commercial product like the Aegis Exchange Simulator, but several years ago, a reader of this blog sent me the binary of a small exchange simulator that he wrote.

To keep on the track of investigating new technology, I would like to explore Microsoft-specific frameworks like TPL and perhaps Rx. I need to go over Matt Davey's blog and get some ideas :-)

The depth-of-book view can use some new kind of visualization in aggregated mode.

There is some unfinished business that I have with Microsoft StreamInsight. I would like to provide a GUI that allows the user to create CEP queries, and have StreamInsight monitor the quote stream and provide alerting. I haven't looked at SI in two years, and I don't even know if I have access to it if I don't have a MSDN subscription. I can always used NEsper as a fallback (is Esper still around?)

©2011 Marc Adler - All Rights Reserved. All opinions here are personal, and have no relation to my employer.

Sunday, August 14, 2011

On WPF, Prism, Grids, and Wall Street

I am taking this time off to try to learn some technology that I have been planning on learning, but have not had much time to dive into. Two of these technologies are WPF and Prism. For the past few years, I have been managing teams that have been using WPF and Prism, but when I have had to write GUIs myself, I have fallen back to the old comfort zone of Winforms.

When I was Chief Architect for Equities at Citi, I was asked to make a call on the future GUI technology of the Citigroup Investment Bank. After a lot of thought, I recommended the use of WPF. The recommendation was certainly not made for performance reasons, and it was not made because of any extra features that WPF had over Winforms. It was made so that Citi would be able to retain its developers. As techies, we want to be involved in the latest and greatest of technologies. We don’t want to be perceived as dinosaurs when it comes time to changing jobs. I have seen the effects on developers of companies who do not move to the latest technologies. Developers will naturally migrate to companies that will challenge them technically. So, in order to preserve the “forward moving outlook” that I was trying to promote in Equities, I chose WPF for the future direction. But I wasn’t convinced.

However, now I see that almost everyone in Capital Markets has moved to WPF. A lot of the time, you have a WPF shell that contains Winforms-based grids. WPF-based grids just can’t match the performance of Winforms grids. And, grids are everything.

I have never been totally comfortable with the thought of marrying WPF and trader workstations. Most trader workstations that I have seen are just a bunch of grids. Traders use the grids to monitor the market and put in the trades. Research and analytics are mostly done on Bloomberg. If you want to look at some charts, you will use your Bloomberg workstation.

The main difficulty in writing quote blotters is how to handle a large number of quotes. If you have a trader who is dealing with ETFs, the trader needs to be able to monitor the ETFs as well as the components in the underlying basket. This means that a quote blotter could potentially have hundreds of quotes in each tab, and several thousand quotes being monitored simultaneously. It would be interesting to see if a totally WPF-based workstation could be written that would keep up with this kind of flow.

In order to learn WPF and Prism, I have started to write a POC from scratch. I wrote a level 1 quote simulator that can push quotes out at varying speeds, and I have the quote collection data-bound to a standard Microsoft WPF DataGrid control. Over a small range of quotes, the speed seems acceptable. I have not yet explored ways to speed up the grid and to handle a high volume of quotes. I will need to start using unbound columns, implement a ring buffer of last-value caches, use more intelligent updating, use virtual grids, and start examining the use of small objects and garbage collection. All of these improvements are standard when writing GUIs to process real-time data. But I want to dive deep into WPF before going horizontally and writing these optimization.

Next up is to write a Level 2 quote simulator and to write a depth-of-book control.

Should Wall Street give up on WPF? No, not at all. I have seen some beautiful applications written totally in WPF. The old Lighthouse CEP system at Citi is one example of a beautiful and compelling GUI that is totally WPF-based. And, there were enough challenges overcome in writing the GUI where I feel that the GUI developers on this Lighthouse team are some of the very best WPF developers on Wall Street.

I would love to find out what everyone’s experiences with WPF have been in your work in Capital Markets. Please take the time to send some comments here.

©2011 Marc Adler - All Rights Reserved. All opinions here are personal, and have no relation to my employer.

Monday, August 08, 2011

New Blog - Bootstrapping a Low Latency Trading Firm

Is this the same blog that was created and quickly shut down last year?

©2011 Marc Adler - All Rights Reserved. All opinions here are personal, and have no relation to my employer.

Thursday, August 04, 2011

I am Free to Blog Again

Hello???? Is there anybody out there??? Does anyone still have my blog on their RSS reader?

Yesterday was my last day at Citadel. As everyone knows, when you join a secretive company such as Citadel, you are not free to blog and disclose any inkling of what you were working on. This is what I agreed to when joining Citadel. Now that these constraints have been removed, I will hopefully be resurrecting this blog and will start to talk about some topics which may be of interest to the readers here.

I am appreciative to Citadel for the opportunities that they gave me, and as always, you come out of a situation knowing more than what you came in with. I ran the desktop apps team there, and morphed into leading the effort to build a new web-based real-time multi-asset trading system. And we did great work there, and came up with a beautiful system. Even though I ran the team and, as always, did the management-BA-architecture roles, I had to do a lot of the coding on the server side. After about 10 years, I got back into the C++/Boost/Linux world, and it was a lot of fun. Unix hasn't changed very much in 30 years. People still use vi, and the best way to debug is to use lots of logging statements. This gives me an even greater appreciation of the entire Microsoft Visual Studio development stack.

So, now I will embark on new adventures. It looks like the entire CEP industry disappeared while I've been gone, and people do not write too much about CEP anymore. My old CEP team at Citigroup seems to still be going strong, and keeps on delivering new functionality all the time. My friend HH did a great job in taking over the team and taking the architecture to the next level, pushing Coral8 into a background role and implementing a custom server in C# (remember that I theorized a long time ago that the best CEP engines were the bespoke ones, written for a custom purpose?).

I have been intrigued by the iPad. I was never an Apple person. I bought an iPad because we were strongly considering coming up with some products on a mobile platform, and in order to get in front of the technology, I borrowed my wife's MacBook Pro, installed XCode, and starting plugging away on Objective C.

All of the banks seem to be rapidly moving to develop mobile platforms. Are they targeting just the iPad/iPhone, or are Android and WindowsPhone in their gunsights as well? Are they doing Objective C development for the iPad versions, or is everyone moving towards HTML5, using technologies like PhoneGap where necessary? Can the new iCloud compete in any way against Azure? Can iCloud be leveraged in trading apps (ie: can you use it for backtesting services?)

If anyone would like to contact me about opportunities,  I can be reached by my Gmail address (magmasystems). Meanwhile, I am going to crack open some books, do a lot of puttering around different technologies, and try learn some of the things that I have been too busy to learn.

©2011 Marc Adler - All Rights Reserved. All opinions here are personal, and have no relation to my employer.