Transaction processing is at the very core of Clear Street’s business. It’s what hydrates our ledger and manages the lifecycle of trades that go through our system. It’s the obvious choke point for all data, and for this reason it needs to be scalable, resilient, and extensible.
Over the past year and a half, Clear Street’s engineering teams took on the daunting task of replacing our core transaction processing engine, Bank, and changing the many moving parts that were necessary to ensure a successful migration to our new system, BK.
We cover BK’s design in another engineering blog, “Scaling Clear Street’s Trade Capture System”, but here I want to focus on our mission to replace our legacy system with the new, shiny BK. In this three part series, I’ll discuss how we pulled off this complex system change through validation, planning, testing, and re-testing, and collaboration within our engineering team and across the organization.
In part one, we’ll dive into why we decided to switch from our legacy transaction processing system to a newer — and shinier — version, and how we approached understanding the problem and validating the solution.
Bank is Clear Street’s original proprietary transaction processing system. For Clear Street’s first three years as a company, Bank’s simple transactional model supported our business requirements and worked well to get our company started.
As we scaled, we began to identify Bank’s shortcomings and our need for a newer, more scalable transaction processing engine to support our growth. Write throughput was low, the data model was less than ideal, and the points of extension became messy as we grew. These flaws limited our ability to scale our transaction ingestion and accommodate growing demands from the business.
BK (Book Keeper, or Burger King, depending who you ask) is Clear Street’s new core transaction processing system. It aims to correct the sins of the previous system with an updated data model that incorporates and builds upon learnings from Bank. It optimizes for writes and shuffles the unit of scale from the database to our message bus (Kafka), where we have gained effectively unlimited horizontal scale. BK also moves transaction processing to an eventually consistent system, thus opening up possibilities for scalability and future needs.
Most of my team joined Clear Street without a prime brokerage background. Some, like myself, had absolutely no finance experience. Our only trading knowledge came from our own personal finances. This posed a unique challenge because we were unfamiliar with the business, and our requirements were effectively reversed engineered from the previous version of the system. So, how did we know what we built would work?
Very early on, we identified issues that were obvious to our resident post-trade expert Mo Arshad, but less obvious to us. For example, we have a simple type of entity called a journal. We needed to support the scheduling of journals so they can post at some time in the future. That wasn’t particularly obvious to us at the beginning, but Mo quickly identified that it was necessary for many use cases within trade capture and settlement processing.
At first, manually testing the system seemed like the only option. There were many moving parts, like reference data, different account types, and a large number of combinations of possible inputs. We didn’t have the domain experience on the Engineering team to know how the data should look. We worked with our Operations team to distill their domain experience into concrete requirements and, more importantly, into repeatable test cases.
After some hair pulling, I decided to start working on a little side project that would help test this complex system. I built a BDD framework on top of Behave that would allow us to express test cases in plain English and exercise different functionality, with full reproducibility. This framework has the core primitives to describe common items requirements such as, “Creating a trade that settles in T+2”.
BDD is a very useful tool. Our main goal for adopting it in this case was enabling our SMEs to provide us with test cases and review the behaviors of the system without necessarily stepping through the services’ Go code. We will discuss how we use BDD to test our microservices in a later blog post.
While our BDD test cases would be able to determine that our requirements were being met, we needed a way to identify how the new system differed from the old system, capture meaningful differences, and help to identify bugs. To solve this problem, we built the Validator. Upon the closing of the day, this service would fetch all of the data from BK and all of the data from Bank, then perform a diff algorithm to find any differences between the two data sets and report on the outcome.
Here’s a screenshot of one of the validation reports, just before BK went live. It tells us that our ledgers, trades, settlements, and journals were completely matching between the legacy system and BK, and that a number of “extra” transactions were found in BK. The “extra” transactions were expected due to differences in the data models. It’s important to note that the full data backing this report is available for download, and we used it to understand and account for differences.
After some time, we realized that finding a 100% match between the two systems was going to be close to impossible, or at least so difficult that the ends would not justify the means. The systems behave differently, but a combination of automated and manual verification over the validation period helped eliminate discrepancies and gave us the confidence that the two systems were producing correct data.
The validator was an instrumental part of building the system. It was the first method we used to ensure the system was on the right track, so we could continue to develop and eventually release it.
This summarizes how we approached testing and planning to develop our new system. In part 2, I’ll elaborate on how we approached BK’s integration and data migration.
Clear Street does not provide investment, legal, regulatory, tax, or compliance advice. Consult professionals in these fields to address your specific circumstances. These materials are: (i) solely an overview of Clear Street’s products and services; (ii) provided for informational purposes only; and (iii) subject to change without notice or obligation to replace any information contained therein.
Products and services are offered by Clear Street LLC as a Broker Dealer member FINRA and SIPC and a Futures Commission Merchant registered with the CFTC and member of NFA. Additional information about Clear Street is available on FINRA BrokerCheck, including its Customer Relationship Summary and NFA BASIC | NFA (futures.org).
Copyright © 2024 Clear Street LLC. All rights reserved. Clear Street and the Shield Logo are Registered Trademarks of Clear Street LLC