Despite our best intentions, sometimes existing architecture won’t scale to meet the needs of a rapidly growing business. We recently ported our analytics pipeline to a new database, and learning some practical lessons along the way…
Think Again: Do you REALLY need this project?
It can be very tempting to blame difficulties on underlying infrastructure. “This would work smoothly if only we used XYZ new technology,” you think. Porting projects are a major undertaking, and they won’t necessarily solve underlying problems with the way your system is designed.
With this in mind, Jana’s infrastructure team carefully chose a new database system that can scale rapidly going forward. As our user base reached millions of users, the old database was groaning under the weight of terabytes of data. We definitely needed to be on a technology that could dynamically add storage, so we switched to a new flavor of SQL – Snowflake.
Scope the project first
What features/systems are being included in the port? Does this new system touch other pieces of your architecture? You should also consider the port MVP. Do you get quick wins by completing a certain portion of the porting project first?
Most of the time, you project will be more modular (and more manageable) if you plan for a direct port first. Then, iterate by making product changes. Don’t forget that you’ll need to port your testing framework, too.
Balance re-architecting with direct porting
The blessing and the curse of porting is that we get to look at the whole system. With iterative software development. Porting can be an opportunity to establish more consistent patterns or implement more comprehensive testing.
On the other hand, it’s easy to get lost in the weeds, trying to tidy everything. One of Jana’s core values is Get Shit Done. Remember, done is better than perfect.
Chances are, your legacy architecture was built over a long period of time by a diverse team of engineers. When porting, any accrued technical debt will come to light. An optimistic team will always assume they can move swiftly
Where possible, perform your port iteratively. Get what pieces you can running in the new system quickly to knock out easy wins.
The team should be in communication with other stakeholders within the company. When delays occur, it’s better to communicate them early rather than drop a bomb.
Whether we’re building features, busting bugs or porting databases, Jana teams meet for a brief standup each day to report what each team member has done, what’s she’s going to do, and if there is anything blocking that work.
Celebrate your wins
Porting projects are typically under-the-hood jobs. If all goes well, the product users can’t easily tell that the underlying architecture has changed.
Make sure you and your team celebrate big wins amongst each other. And shout your accomplishments from the (metaphorical) rooftops! Let other teams know what you’ve accomplished, and share your learning with other engineers who might follow your footsteps.