Celebrating Open Source Successes

osi_symbol

2014 was a fantastic year for the Valuation Services team as we rolled out a ton of technology including ~40 micro services written in Java and Scala, and effectively built-out the full stack of systems used by an investment bank from the ground up. We covered a broad set of business processes across North America, Europe, and Asia including but not limited to:

  • Integration into the firm’s book of record
  • e-Trading and quoting for LCH/CME cleared trades
  • Market Data, Reference Data, Derived Market Data, Trades
  • Real-time Risk, Scenario Risk, Attribution

All of this was built primarily on Open Source technologies and in 2015 we have already started to significantly expand our Open Source footprint by delivering:

Geographical expansion of our data storage layer built on Apache Cassandra

In mid 2014 we migrated the storage layer of a number of our services to an Apache Cassandra grid running in North America. As global data usage increased in Europe and Asia, we are now expanding the Cassandra grid to Europe in order to reduce latency for clients in Europe (by 10x!) and to also cut down on the trans-atlantic WAN traffic.

While we’ve learned to not be on the bleeding-edge of Cassandra releases, running Cassandra in production has been everything we’ve hoped it to be and we’re quite pleased with it’s abilities to traverse the WAN.

Data distribution with Apache Kafka

We have started to distribute a number of our streaming data sources using Apache Kafka and have been quite happy with it’s stability and usability. We believe that our Apache Kafka footprint will expand in 2015 as a number of other streaming data sources will be onboarded.

Unlike traditional message brokers, it’s nice to see that scalability and fault-tolerance have been addressed from the ground up and implemented in fundamentally correct ways.

Risk reporting on Hadoop and Spark

Finally, we are currently testing a prototype for scalable risk reporting running on HDFS, YARN, Spark, and Scala Parser Combinators. We were able to fairly quickly deliver a very capable initial prototype running on a 48 core Spark grid which has been showing a lot of promise.

In addition, we’ve built a number of UI prototypes using AngularJS on top of this Hadoop/Spark grid and have been very happy with the capabilities that Angular provides us. While Angular puts the traditional ways of rendering dynamic HTML content on its head and has a bit of a learning curve, it enables us to very sensibly compartmentalize JS code and to build out dynamic single-page applications in very modular ways.


We are very grateful for all the high quality software that we’ve been able to use for free and are always looking for ways to give back to many of these fantastic projects and their talented contributors.

Obviously none of this would be possible without the hard work of the extremely talented Valuation Services Engineering team, our highly knowledgeable Solutions team, our fantastic Dev Ops and Support teams, and our very capable senior management team (this is reading like an award show acceptance speech, but hey, it’s true!).

If you are passionate about Software Engineering in Financial Services or any of these excellent Open Source projects, please reach out!

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s