Vhayu's Velocity (too many Velocity's out there!) now has data compression performed by FPGAs. This looks like pretty impressive stuff, and sets the bar a bit higher for KDB.
With on-the-fly data compression, you can have more tick and trade data available for backtesting, and since the compression is done in hardware, this should reduce the latency compared with software-based on-the-fly compression.
Even though this is pretty good news, I wonder if Compliance Departments might have an issue with storing zipped data, as it is not readily available for inspection and analysis by non-Vhayu applications. But that is a trade-off that you will need to make. I am sure Ross Dubin can further enlighten us on this...
I would rename the product from Velocity to ZIP-ON-A-CHIP.
:-)
©2008 Marc Adler - All Rights Reserved.
All opinions here are personal, and have no relation to my employer.
Subscribe to:
Post Comments (Atom)
4 comments:
The compression is transparent to end users and applications. All data is readily query-able through our standard interfaces: ODBC/JDBC, Java, .NET, http, Excel, C++, TCL etc. It is decompressed on the fly with no performance impact.
These look interesting:
AHA363-PCIe
Would be interesting if you could get it to accelerate e2compr then you could have transparent hardware accelerated disks and ramdisks that would work for all software (on Linux anyway).
Probably moot point tho - there is little to be gained by hardware accelerated compression given the capabilities of modern processors - one core should be more than enough.
Hardware accelerated compression tends to focus on classical compression algorithms which is not optimal for market data. Gzip and variants will typically compress 2:1 or 3:1 (if you are lucky) perhaps a little more on large datasets. Good proprietary market data compression routines (not junk like FAST) will compress market data ticks ~14:1 but these are unlikely to ever be popular enough to be hardware accelerated.
-c
Hiding behind an anonymous post hardly makes you credible and in fact raises more questions about who you are, your level of expertise (or lack there of) and your motives.
In response to your “moot point” comment, I’d like to set the record straight. Based on extensive testing during our 18 month R&D process, we can factually state that software-only compression saturates one processor (core) and runs around 6 times slower than FPGA. We've already proved this to existing customers who agree with our findings. If you have any doubts, you're welcome to reveal your identity and come see the results yourself (Palo Alto and Emeryville aren't too far from Los Gatos, however Romania could be a bit of a trip). Our customers have welcomed this announcement and fully understand the benefits of a hybrid hardware/software solution for 4:1 data compression with no performance penalty and enterprise-level fault tolerance.
Update - kdb+ v2.7 has built-in file compression and is also compatible with the AHA367-PCIe, achieving approx. a 10x speed up over software compression.
Post a Comment