Low-latency Hadoop for Risk Analytics

Most Platform Symphony customers today are running monte-carlo simulations in support of various market and credit risk applications.

Some customers are deploying or considering the deployment of Hadoop to incorporate unstructured data into their risk models.

Recently I had the opportunity to download and play with the latest version of IBM InfoSphere BigInsights which now includes a run-time version of Platform Symphony called Adaptive MapReduce.

I setup two small clusters comprised of identical hardware resources in a local VMware environment. The top window in the demonstration is logged into a cluster master running standard Hadoop (Hadoop 1.1.1 in this case). The lower window is running IBM BigInsights 2.1, using the Platform Symphony scheduling infrastructure instead of open source Hadoop.

The performance differences are dramatic.

Clearly not every application will see this kind of acceleration, but for applications that are latency sensitive (as many in financial services are) the advantages of using the Symphony scheduler are compelling. Not only do workloads run faster as can be seen here – customers can share the same infrastructure between multiple Hadoop and non-Hadoop applications.

You can learn more about IBM Platform Symphony at http://www.ibm.com/systems/technicalcomputing/platformcomputing/products/symphony/