It Has Begun !!!!!!

I'm the First !

:-)

Watch the news over the next few days and you will see what I mean.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Job Openings with my Group

I have one full-time opening in my group, and possibly two consulting positions. The full-time job will be located on our Equities trading floor in the TriBeCa section of New York City. The consulting positions will be either in NY or in Jersey City.

For the full-time position, I am looking for a great .NET developer. The ideal candidate should be familiar with Team Foundation Server and issues around build and deployment. The candidate should also be familiar with WPF and WCF; we would ideally welcome someone who is proficient with one of the two and would be willing to become expert in the other.

For the consulting positions, I am looking for great .NET developers. WPF and WCF are highly desirable, as there will be definite UI work.

The consulting positions are on-site. Do not email me with requests to work remotely. They will be routed directly to /dev/null.

Because of the current fiscal environment, I cannot use external headhunters. Sorry.

You can send email to me at magmasystems at yahoo dot com.



©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Magma in Paris, February 12, 13, 14 2009 - A Recap

I arrived in Paris at 11:00 AM on Wednesday, February 11 2009 for 3 concerts given by my favorite group of all time, Magma. The concerts were in celebration of their 40^th anniversary as a group (even though they had their 35^th anniversary concerts in 2005). The concerts were to take place at Le Casino de Paris, a theater in the 9^th Arrondissement, which is a section of Paris more known for its sex shops than anything else.

In addition to Magma, the opening act for all three nights was Jannick Top’s Infernal Machina. Top was the bassist for Magma during their glory years of 1973 and 1974 (and also, 1976), and is a personal hero of mine. His new album, also called Infernal Machina, is a throwback to the glory days of Magma, and to tell you the truth, I wanted to see Jannick just as much as I wanted to see Magma.

After taking the RER B train from the airport to Gare de Nord, and taking the number 2 train to the Pigalle stop, I managed to find my hotel. I used my time-honored technique of sleeping for a few hours immediately after arriving in Europe so that I would have energy for the night’s festivities. I picked a 3-star hotel that I found on Expedia that was only about a ten minute walk from Le Casino. Typical 3-star Paris hotel … clean, sparse (not even a clock in the room), and cable TV.

That night, there was a pre-festivities dinner at a Southwestern French restaurant that was organized by Yann Richard (yannzeuhlwortz). Besides Yann and me, in attendance that night were photographer Fabrice Journo (Riah), Fanny Layani (A-Fiieh) and Muriel Huster (a photographer who is not on the Magma forum). Later, we were joined by Marcello and Tathiane, two Brazilian Magma fans who live in Los Angeles, and whom I met in 2007 in Nancy, France.

Thankfully, between the courses of escargot and Magret de Canard, everyone mostly spoke English to me, and thus spared me the pain of inflicting my self-taught French upon the fair people of Paris.

Yann had a friend who had seen Magma two weeks before in Toulouse. Yann relayed to me a mini-review that his friend had given. The review consisted of the single sentence “Magma is Dead, Long Live Magma.” This review turned out to be very prophetic.

Then I was told that Magma was supposed to play the same exact set every night, and they had no special guests scheduled. Fuck!!!! I thought that, for their 40^th anniversary, they would trot out some of their masterpieces. But, this was going to be 3 nights of new or unrecorded compositions.

Everyone had planned to meet up at the brasserie across the street from Le Casino at 5:00 PM the next day. I spent most of Thursday sleeping, as I went to sleep around 4 in the morning. At 5:00, a small contingent of Magma fans had started to gather at the brasserie. I met my old pal Robert Guillerault from Bordeaux, who runs the Magma Web Pressbook. Robert speaks no English and I cannot understand his spoken French, so we spent a half hour gesturing wildly in the air. We were soon joined by some of the other Zeuhl Kombattahnts that I only knew from the French Magma forums. Fabrice, Yann, Fanny, and Muriel soon arrived. Marino (hpl) came from Venice, and brought me some DVDs of some live shows. Maxence (Zwenskaia) from Montpelier came by, as well as photographer-supreme Madi, Francis Lecointe (who has graciously bought the tickets for me), old friend Fabrice Bellearbe (Barfix), ace photographer Marco Tchamp, Roman (Peaudhorse), and more. Magma themselves came by for their pre-concert drinks. Robert introduced me to Magma bassist Philippe Bussonet, who seemed thoroughly unimpressed that I came from New York to see them. I also got to shake hands with Stella Vander and Herve Aknin. I usually try to avoid contact with bands that I like; I am a firm believer in the old adage “Never meet your heroes because you will inevitably be disappointed.”

At 7:30, we all staggered across the street to the venue. Inside, I met old friend Gaeten Villard (Zess), and I finally met British magma fanatics Steve Ashworth and Steve Davis. Steve Davis has one of the few English websites devoted to Magma. Steve is well-known in England for his exploits in the world of Snookers, which is much like the American game of Pool. In fact, he was 6 or 7-time world snookers champion, and now, devotes a substantial amount of his time playing professional poker. Steve also brought the Magma offshoot group, Offering, to the UK in the mid 1980’s. Steve and I have been corresponding at infrequent intervals over the years, so it was great to put a face to the name. Both Steves had brought their girlfriends to the Magma show, which I thought was a very dicey proposition.

The seats in Le Casino were uncomfortable, to say the least. The amount of space between the rows of seats was designed for little Frenchmen, not for big Americans.

At about 8:15, the first group, Jannick Top’s Infernal Machina, came on. The group consisted of two drummers (one playing a large Japanese Koto drum), bass, guitar, electric piano, and two vocalists. One of the vocalists was Klaus Blasquiz, who was Magma’s vocalist during the glory years. The other vocalist was a tall, blond woman who was the subject of much conversation over the next three days.

After a soft opening section of cello and percussion, Top’s band ripped into the full force of the composition. Most of the hour-long piece consisted of the group jamming out on various old Magma themes, including parts of MDK, Kohntarkosz (the ending section), and De Futura. I loved it.

There were various issues with the sound system on the first night. From where we were sitting in the third row of the mezzanine, the sound was too muddy. Luckily, over the next two nights, the sound improved a great deal.

In a portent of things to come, Magma also had technical issues at the start. They opened the concert with a brand new piece, only to have Bussonet’s amplifier shut off after the first few chords. Vander signaled for the band to stop, and Bussonet took a few minutes to rectify the situation.

The first piece is unnamed as of now. To my ears, it sounds as if Christian Vander has been listening to a lot of Steve Reich. Two main themes play off of each other in this composition. First, there is a Reich-like ostinato pattern that is played by the vibes, guitar, and electric piano. This is overlaid by a more Magma-like vocal line that is sung by Herve Aknin, the new vocalist who recently replaced Antoine Paganotti. As these two themes move in and out of phase, the drums and bass punctuate the music with a series of intermittent crashes. This goes on for about ten minutes, building up to a climax.

The second piece was a real snoozer for me. It is a 25 minute piece named Felicite Thosz, and is reminiscent of the vocal-intensive Magma offshoot called Offering. To my ears, it sounded like a combination of Magma playing Christmas songs with Magma doing a mouthwash commercial. There were lots of vocal solos and harmonies and sleigh bells … and for me, a reason to catch up on my sleep. A bit of promise in the last section with a Russian-dance theme, ala Wurdah Itah, but no real punch. Polite applause from the audience at the end.

The third piece was Emehnteht-Re, which is third piece of the Magma trilogy that includes Kohntarkosz and Theusz Hamtaahk. It is an hour-long piece that unites several of Magma’s previously-recorded themes. I first saw Magma perform this piece in Brooklyn at Club Europa in June of 2007. Since then, Vander has reworked the piece in various minor ways. The piece now seems more coherent to me. The piece starts out with Rinde from the Attahk album, transitioning into Hhai, transitioning into Zombies, and ending with a dirge that represents the funeral of an Egyptian king.

The Zombies section is supposed to be the atom bomb in the piece. About 10 minutes of unadulterated Zeuhl madness, and is supposed to be the centerpiece for Vander’s remarkable drumming. For the first night, Vander seemed rather subdued. I am not sure if it is his age catching up to him or the tendonitis that has been bothering him for the past few years.

E-R was followed by the cacophonous De Futura. This is a Jannick Top composition that appeared on the Magma album, Udu Wudu, in 1976. When it was recorded, it was a sparse but rhythmically driven piece that only included bass, drums, occasional synthesizer, and growling vocals. However, for the 3 concerts, it was played by the total combined forces of Magma and Infernal Machina. 15 people on the stage, including 3 drummers, two guitars, two basses, two pianists, and 5 vocalists. All I can say is “What a mess!” Remember the blond vocalist from Infernal Machina? The running joke during the 3 nights of the show was “What will she be wearing during De Futura?” Each night, she seemed to come out with less and less clothes on, and during the afternoon, she must have been shopping for her next costume at the shops next to the Moulon Rouge. The performance of De Futura was extremely sloppy, as confirmed by a lot of the opinions I heard the next day at lunch.

As an encore, Magma finished with Ballade, which is Vander’s moving tribute to his idol, John Coltrane. A beautiful piece with lots of rich harmonies, and featuring Vander doing his famous scat-singing in Kobaian.

After the concert, a lot of the Magma fans gravitated to another brasserie that was a few blocks up from the concert hall. This was to be a nightly occurrence that kept me out until 2:00 AM. I spent most of the time with the only other native English speakers, Steve Davis and his girlfriend, Jeannie, talking about Magma, snookers, and poker. You can actually watch videos on YouTube of Steve playing snookers at the various stages of his career. Steve was also able to satisfy some of my curiosity about the Women’s Pro Pool circuit in the US.

The next day, we had a wonderful Zeuhl lunch at Le Triton, a jazz club in Paris that Magma has played in many times. The talk around the table was about the collective disappointment of the previous night’s concert. There was consensus about the lack of “punch” in the songs and about the poor sound quality. If you ever find yourself in Paris to see a show at Le Triton, I hardily recommend eating there before the show … the food does not disappoint.

After a brief siesta, it was back to the brasserie for some beers, and onto the show. Friday night’s concert was noticeably better. The word must have gotten to the bands about the poor performance on Thursday, and there were great improvements in the sound and in the quality of playing. It was the same set list, but it seemed that Infernal Machina really jammed out on the De Futura-sounding section, and Magma did the same on the Zombies section of E-R. And, the blond vocalist did not disappoint, as she seemed to outdo her previous night’s antics. Would someone tell Jannick that her presence on stage seriously detracts from his music!

I also met some other fans from the French Magma forum. People like Eric (RI Kobaia), Maahnt (name unknown), Bruno (Nohb), Fatcat (name unknown), JC Alluin, Allain Julliac (Chicago), Bernard (Wurd), and others. I was surprised to see Steve Ashworth’s girlfriend return for the last two nights, as she did not seem happy after the first night. Marcello and Tathiane were so bummed out by Thursday night’s performance that they did not attend on Friday.

The next day, Guy and Bernadette Mas organized a nice lunch across the street from Les Halles. Lots of wine, laughter, broken French and broken English. Fanny and Fab Journo could not attend the show on Saturday, so we said goodbye to them.

At the pre-concert meet-up, I had the chance to meet and talk to Klaus Blasquiz, who is friends with Marcello and Tathianne. Very nice guy and speaks impeccable English. I did not have the nerve to ask Klaus about all of the old Magma tapes that he has stored in his basement. (He was Magma’s archivist during the early years, and I cannot imagine the riches that he has hidden away.)

At the bar, we ran into Offering’s old drummer, Jean-Claud Buire. He must have been drinking for the entire day. He sat behind us during the first half of the show on Saturday night, and made a total ass of himself, shouting, hissing and whooping every few seconds. I had to restrain myself from throwing him off of the balcony … I might have if I had been able to move my knees (if you recall, my knees felt like they were in a vice because of the little amount of room between the rows of seats).

Saturday night’s show might have been the best. Even the super-group that played De Futura seemed to be in sync. I congratulate Jon Grandcamp, the drummer for Infernal Machina, for having so much restraint and letting Vander do most of the drumming.

After the show on Saturday night, we tried to go to the usual brasserie again, but for some reason, it was closed, as were some of the other brasseries around. New York City still retains the title of the “City that Never Sleeps”. I said my goodbyes to Madi and Maxence, and even said goodbye to Jean-Claude Buire, who had stumbled into the only open bar in town.

Why did I go to Paris for these three days? Christian Vander is 61 years old, and Magma won’t be around forever. I need to be able to see Magma while Vander is still able to play. For me, it’s like being able to see the last days of Michael Jordan or Babe Ruth. But the biggest reason for me to have gone to Paris was to see all of the great friends that I have made through Magma, and those are the memories that I like to keep.

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

I am in Paris in Magma Heaven

Leaving in an hour to go to Paris for the 40th anniversary series of concerts by Magma at Le Casino de Paris. I will be back on Sunday night.

Since the comments are moderated here, if you post a comment, I will not be able to approve it until Sunday evening. So please don't keep resubmitting the same comment over and over ... the approval pixies are also in Paris.

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Request: Streaming R or R-like Functions in CEP Systems

I have been looking a bit into R lately. I have written here before about being able to transform R into something that can take real-time streams of data, or integrating R (or SPlus) into a CEP system. Here is a small example. Let's say that you want to take a sample of a "window" of streaming data every few seconds and compute some function on that sample.

Let's see how we can do this in R using static data.

In R, we can define a range of stock prices that range from $10.00 to $100.00, in increments of 25 cents.

> StockPrices<-seq(10.0, 100.0, 0.25)
> StockPrices
[1] 10.00 10.25 10.50 10.75 11.00 11.25 11.50 11.75 12.00 12.25 12.50 12.75 13.00 13.25 13.50 13.75 14.00
......
[358] 99.25 99.50 99.75 100.00


We can take a sample of 10 random stock prices like this:

> sample(StockPrices, 10, rep=F)
[1] 20.50 19.00 78.75 44.75 62.75 45.25 66.25 70.00 71.00 91.00

We can compute the average price of a sample of 10 random prices like this:

> mean(sample(StockPrices, 10, rep=F))
[1] 52.775


Using the Coral8 term of "window", let's say that we have a window called StockPrices that contains one minute's worth of prices. Every 5 seconds, we want to take a random sampling of 10 prices in that window and compute the average price of that random sample.

How can we do it in your CEP system? I would welcome examples in Coral8, Apama, Streambase, Aleri, Esper, StreamSQL, and any other CEP system.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

IBM MQ Series, XMS, and .NET

We have to be able to read in some information that another system is putting out over IBM MQ. There is a messaging implementation from IBM called XMS that seems like it's JMS over MQ. This is great for us because this means that we can (hopefully) clone our existing Tibco EMS adapters, and with a few little changes, be able to read off of MQ.

I am not sure if the original publisher of the data also has to use XMS ... maybe someone here can tell me. I know for a fact that the publishing system does not use XMS. If the publisher does have to use XMS in order for us to consume through XMS, then we will need to come up with an input adapter that uses the native MQ API.

The information about the IBM Messaging Client for .NET is here.


Update: A great comment by Richard Brown of IBM WebSphere pre-sales tells me that there is nothing to worry about if we are just a consumer ... which we are.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Follow-ups to Some Comments

Marco says:
>> What do you mean by "Entitlements on output events"?

I have blogged about this here before. When our CEP engine generates alerts, we do not want everyone to see them. There might be very few people who can see alerts on orders. Prop traders should not see many of the the alerts that sales traders see. One trading desk might not be able to receive the alerts that another trading desk should see.

We have the concept of coarse-grained and fine-grained alerts. A coarse-grained alert might be filtered out by role. For example, a particular alert should only be seen by a North American Cash Trader, and not by derivative traders. In the case of fine-grained entitlements, we might have to filter by certain data that is within the alert's payload. For example, someone might not be able to see an alert if the symbol is IBM and the volume is greater that 1000 shares.

There is also the matter of entitlements on real-time aggregations, but that is another long topic.


Charlie Skelton from KX says:
>> "don't make us code in Q" Please can you elaborate?

Charlie, I have written ad nauseum here about Kdb and Q. I actually started to learn Q a few months ago in order to be able to read some of the code that was thrown our way. I have a greater appreciation of Q than I did a few months ago, but I would not want to ask all of my developers to learn Q.

My opinion is that, if you have a facade (input adapter) to a database, then you should be able to code in some database-agnostic way and leave the db-specific language access to the drivers. Look what you can do with O/R frameworks like Hibernate and iBatis.


Hans asked about Unit Testing

When we first started to use Coral8, we actually wrote a C# application that did the exact same thing as our Coral8 application and we did reconciliations between the two. We had found an issue or two with Coral8, which they were able to fix fairly rapidly. We also found some issues with our Coral8 coding, which helped us learn Coral8 better.

Unit testing is still something that is not mature in our Coral8 processing, although we do have unit tests for our internal framework. We have fabricated data in flat files that we can pump into our C8 engine, and we can do regression testing against a known set of outputs. However, we know that we can do a better job.

I would like to hear more about Streambase's unit testing capabilities.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Streambase Again!!!! Arghhhhh !

It seems that my posting on Streambase's recent customer win has gotten some people pretty ticked off. I have received some private messages from a few of the CEP vendors who are crying foul. Nobody wants to appear that they have sour grapes against the Streambase win, but of course by contacting me, they know that I will pursue the issue in the interest of fairness and transparency.

The issue revolves around the joint claims by Mark Palmer and Penny Crossman that Phase Capital had evaluated all of the major CEP vendors. Penny had written the following:

PhaseCapital evaluated a number of CEP vendors in the trading space, including Progress Apama, Coral8, Aleri and StreamBase.

What some of the CEP vendors are claiming is that Phase Capital had never downloaded copies of their products for evaluation, nor contacted the CEP vendor with any questions.

In my experience, before you can download a copy of a CEP product for evaluation, you need to fill out a registration form. Some of the CEP vendors won't even let you download a copy from their website ... they want to pre-certify you by paying you a personal visit.

So, if Phase Capital did not download the evaluation copies, how did they evaluate some of the products. Did they do part of their evaluation from going over a publically available list of features? Did they get information from blogs such as this one? Did they download the product under another name? Did they have a third-party consultant do the evaluations?

Update: They could have gotten the copies from a reseller. Sybase resells a number of CEP products.

In the interest of transparency, I would ask Greg McSweeny from Wall Street and Technology to ask his writer, Penny Crossman, to substantiate what she wrote in her article concerning the competitive evaluation that Phase Capital did between the various CEP products. Penny is the designated beat reporter for the CEP industry, so I would like to see a bit of investigative journalism from her on this matter. Otherwise, serious doubts would still remain as to Phase Capital's evaluation process.

This is why organizations like Stac Research are so important to our industry.


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

What is Important to us in a CEP System

I have posted this information before in previous blog entries, but I think that, since I have some new readers here, I should post a list of some of the things that are important to me in a CEP system.


Input Adapters
-------------------
Market Data - Reuters, Bloomberg, and to a lesser extent, Wombat and Activ
FIX - 4.2, 4.4. Predefined schemas for the most popular FIX messages would be great.
SWIFT
Message Buses - Tibco EMS, RTI, LBM
Tick Databases - KDB+ (read and write capabilities, don't make us code in Q), Vhayu
Object Caches - Velocity, Gemfire, Coherence


Output Adapters
---------------------
Various chat/messenger mechanisms
Message Buses - Tibco EMS, RTI, LBM
KDB+, Vhayu
Object Caches - Velocity, Gemfire, Coherence

Development Environment
----------------------------------
Strong support for debugging
Profiling and performance monitoring
Hooks into Performance Counters
Ability to easily add C#-based custom functions


Analysis
-----------
Integration with tools like R, SPlus, Matlab
Real-time OLAP
Strong support for real-time dashboards


Miscellaneous
------------------
Workflow
Entitlements on output events
Integration with Active Directory
Priority input queues into the CEP engine


©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Streambase On The Move?

Mark Palmer seems to be turning Streambase in a positive direction since he joined .... either that, or the Streambase marketing machine is in full throttle.

Remarkably, Mark has managed to get testimonials from a number of financial services customers. Usually we are a secretive bunch, and many of the larger financial institutions do not like to have themselves used as a reference customer. Which makes it all the more interesting that Streambase has been able to persuade these customers to go public. Not only that, but Streambase has managed to convince Penny Crossman of Wall Street and Technology to rewrite their press release into an article.

Say what you will about Streambase's sales and marketing staff .... they have proven themselves to be amazingly effective. They have been able to trot out customer references, whereas Coral8 and Aleri have not. This gives the perception that Streambase is eating the competition, whether that perception is true or not.

Mark posted an interesting paragraph:

As Dr. Goodell described, Apama didn't even make the first cut in their evaluation. On the other hand, Aleri, the other final CEP product in PhaseCapital's evaluation, fared well with its high-performance, multi-threaded server architecture.

And, Penny follows it up with:

PhaseCapital evaluated a number of CEP vendors in the trading space, including Progress Apama, Coral8, Aleri and StreamBase.

It's very interesting that Phase Capital found that Aleri was the highest performing CEP engine. This has confirmed something that I have suspected. Aleri has always been touting themselves as the most performant of the CEP engines, and has welcomed the other CEP vendors to take the Aleri Challenge. However, it may have been the lack of spit-and-polish in Aleri's development environment that may have sunk it. I have previously blogged here that I thought that Streambase had the best development experience and documentation out of all of the other products.

I am wondering what factors caused Phase Capital to not choose Coral8. One thing that Streambase has that most of the others do not is built-in adapters for trading companies. As Penny writes:

StreamBase already had an adaptor for the QuickFIX engine as well as adaptors for the data streams provide by Lime Brokerage, PhaseCapital's execution broker. "The building blocks were very complementary to what we're ultimately trying to deploy," Pritchett says.


I have been saying this for a long long time on this blog --- the quality of the CEP engine does not matter as much as the entire ecosystem surrounding the CEP engine. You can have the best, fastest, most expressive CEP engine and language, but if you don't surround it with a compelling ecosystem, then you are sunk. And, Mark Palmer understands this.



©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.

Real-time Messaging in Sybase 15

If your organization is migrating from Sybase ASE 12.5.4 to Sybase 15, then you might want to take a look at these slides.

One slide caught my attention. It was a slide on the migration effort done by eSpeed. One of the things that eSpeed did in their Sybase migration was:

Replace "Polling-Broadcasting”legacy solution with integrated real-time messaging (RTDS 4.0) in ASE 15

I need to find out more about this. Sybase has some slides here.

Many financial companies who are using Sybase have applications that actually do database polling every 'n' minutes in order to find out what rows in a table have changed. If eSpeed did what I think they did, then this presents itself as a compelling reason to move to Sybase 15. (Of course, this assumes that you are not goign to be moving eventually to SQL Server 2008/2010.)

©2009 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.