Tuesday, August 01, 2006


The word conflation sounds like a condition that requires you to take a few spoonfuls of Pepto Bizmol. But, it is a term that is used when discussion efficient market data distribution.

The technique has been around for a while. I have always referred to it as aggregation. The term conflation means the combining of data in messages to form one message. You use conflation of messages to overcome slow consumers, limited message queues, and any situation where a message publisher has to throttle the outgoing stream of messages.

Let's say that you are a consumer of stock quotes. The market data distributor receives 1000 quotes for MSFT in one second, but you are only interested in the latest price every second. So, the market data distributor will conflate all of the MSFT quote messages into a single message that gets broadcast to you every second. This means that the server will only send you the last bid/ask price.

You can choose various parameters for the conflation. For instance, as mentioned above, you might want time-based conflation (every 2 seconds, every 30 seconds, etc), you may want conflation based on the size of your message queue, or you might choose your own custom way of doing conflation.

©2006 Marc Adler - All Rights Reserved

1 comment:

Anonymous said...

Wombat dont currently support out of the box conflation today on either TCP or multicast based transports.

Im not sure what you mean by 'one gobal conflation setting' for multicast transports like RV? Any one->many transport that dynamically degraded quality of service to the level of the slowest client would be pretty useless and RV doesnt have any capability to transport moer than one stream of data on a single subject.

I think you are referring to providing an alternative service that operates at a lower message rate. This rate cannot be dynamically affected by the client(s) so its really just an intervalized scheme. i.e. If the interval was 1sec you would be offering a 1sec time series service.

Today Wombat dont offer either capability although you can emulate streaming time series using Kx kdb with the Wombat DBAC adapter, but its up to the client what it subscribes to.

The only widely used market data platform that supports dynamic per cient conflation today is Reuters RMDS via their P2PS server.

- guess who :-)