BK is Clear Street’s proprietary transaction processing system, used to process more than $3 billion in daily trading volume. BK is the newer, faster version of our legacy Book Keeping system, Bank. Where Bank had issues with low throughput, data modeling, and messy points of extension, BK is more agile and scalable, to support our growing business.
In part 1 of this series, I outlined how we identified areas for improvement within Bank and used a data validation system to ensure we were solving those issues when we built BK. In part 2, I covered how we approached compatibility layers and data migration. After multiple rounds of validating, integrating and migrating, we were finally ready to deploy our new core transaction processing system. In the third and final part of this series, I’ll walk through the final release and the lessons we learned along the way.
Before deployment day, we fully scripted what needed to happen upon BK’s release and thoroughly documented each step in a single Release Script wiki page. Clear Street’s engineering and operations teams reviewed this document end-to-end and held multiple dry runs in our pre-production environment.
The Release Script included commands that must be run, services that needed to be scaled down, links to code changes that had to be merged at certain points, SQL scripts to verify things like replication lag and resetting of database sequences, and a laundry list of other items to make sure deployment would go off without a hitch.
The script also included a point of no return, when BK would be released and we had to commit to it being live. We triple-checked everything to avoid any last-minute changes. It was critical that by the time we reached the point of no return, we were comfortable that the system was healthy and in its correct state. Specifically, we had to verify that ledgers were matching and that open trades and settlements were sitting in the right places.
I wish I had some screenshots or recordings of the deployment. We had ~20 people on a Google Meet call, sharing their screens and confirming the state before and after executing each step of the script. It felt like we were landing on the Moon! It was a sight to be seen.
Releasing software this big, complicated, and intertwined with the whole business is… a big deal, so how did Clear Street get it done? We executed multiple rounds of testing with the different tools described, we worked hand-in-hand with our wonderful Operations team, and at the end of the day, we had a LOT of support across the business to work through the release script.
Major lessons from the this journey were:
Success in performing large scale migrations is dependent on having a well thought-out plan and being incredibly thorough. Hot swapping Clear Street’s core system with a completely different one was a long and challenging journey, but it was absolutely worth it. We are now in a much better position to continue rapidly scaling our business with a much more solid foundation.
If solving complex, highly rewarding problems interests you, check out our careers page!
Clear Street does not provide investment, legal, regulatory, tax, or compliance advice. Consult professionals in these fields to address your specific circumstances. These materials are: (i) solely an overview of Clear Street’s products and services; (ii) provided for informational purposes only; and (iii) subject to change without notice or obligation to replace any information contained therein.
Products and services offered by Clear Street LLC, member FINRA and SIPC. Additional information about Clear Street is available on FINRA BrokerCheck, including its Customer Relationship Summary.
Copyright © 2023 Clear Street LLC. All rights reserved. Clear Street and the Shield Logo are Registered Trademarks of Clear Street LLC