BK is Clear Street’s proprietary transaction processing system, used to process more than $3 billion in daily trading volume. BK is the newer, faster version of our legacy Book Keeping system, Bank. Where Bank had issues with low throughput, data modeling, and messy points of extension, BK is more agile and scalable, to support our growing business.
In part 1 of this series, I outlined how we identified areas for improvement within Bank and used a data validation system to ensure we were solving those issues when we built BK. In part 2, I’ll discuss how we approached compatibility layers and data migration.
After thorough testing, our next challenge was to integrate existing services with BK. Initially, we aimed to have all clients of the system migrate to BK pre-release. As we began to understand the complexity of the release, it became apparent that this would create significant risk and take a lot of time.
We chose to build a Façade service to release BK without needing to update multiple services in different programming languages, while simultaneously ensuring every single service kept the same behavior between the old system and the new one.
This service implements the APIs of Bank, fully backed by BK, and concentrates all the necessary adapter logic into a single service. The process was not simple. It required a lot of testing and some trade-offs, but in the end, it enabled us to test other services using BK without even touching them.
To make the switch as seamless as possible, we used a few interesting approaches:
The configuration of Façade supported multiple modes:
We orchestrated multiple deployments with several milestones until the final release, when Bank was no longer needed. These milestones were:
At this point, I’ve only discussed the RPC compatibility layer, but it wasn’t this simple. Bank also contained an event layer, which hydrates our Data Warehouse and other downstream services. For this use case, we introduced a service called the “Façade Stream,” a Kafka consumer and producer, which translated the BK event stream into the Bank equivalent. Introducing the Façade Stream was difficult for a few reasons, but primarily because it required us to rehydrate data in different ways because of critical differences in the Bank and BK data models.
The output of BK’s façade stream and Bank’s event stream are meant to match very closely and the output of both systems landed in our common data warehouse, Snowflake, which enabled us to perform validation against these data streams at scale through SQL.
A critical factor I have not yet mentioned is that all of the data in Bank also needed to be available in BK. To complicate things further, Bank and BK have fairly different data models. Well, that’s a little gremlin that reared its ugly head multiple times as we worked towards migrating the data from Bank and loading it into BK.
We wrote an application that we called the “Bootstrapper” to fetch data from Bank, map it into the BK equivalent, and load it into BK’s datastore. The process of “bootstrapping” is fairly complex because we were mapping Bank’s data model again into BK’s and it required handling many edge cases within that translation and re-enriching data to make it “whole” for BK to function properly.
BK’s core database is AWS Aurora (Postgres flavor). One of the many wonderful things about it is that it supports loading and unloading data to and from S3. The Bootstrapper leveraged this to perform large data operations within its migration workflows.
To put it simply, this process was extremely complex. We were overjoyed when we got to delete the code for this application. When migrating data, it’s critical to plan ahead, be ready to continuously discover edge cases, and factor in additional time, because it will take much longer than expected.
The Bootstrapper lived for a whole year, and we were making adjustments to get all the data correctly loaded into BK until the very end. Once we handled compatibility and proper data migration, we were ready for deployment, which I’ll cover in part 3.
Clear Street does not provide investment, legal, regulatory, tax, or compliance advice. Consult professionals in these fields to address your specific circumstances. These materials are: (i) solely an overview of Clear Street’s products and services; (ii) provided for informational purposes only; and (iii) subject to change without notice or obligation to replace any information contained therein.
Products and services offered by Clear Street LLC, member FINRA and SIPC. Additional information about Clear Street is available on FINRA BrokerCheck, including its Customer Relationship Summary.
Copyright © 2023 Clear Street LLC. All rights reserved. Clear Street and the Shield Logo are Registered Trademarks of Clear Street LLC