Ten Databases Into One

The problem

The client is an established online marketplace with years of history behind it, many services, and ten production databases that had accumulated one at a time, each for a reason that made sense when it was made.

I led the modernisation programme, covering tech lead, backend architecture, reliability and DevOps as a single role.

The consolidation was the centre of it. Not because merging schemas is hard, because the marketplace could not stop. There was no maintenance window to ask for, and a marketplace that loses a record loses a customer’s trust permanently. “Zero data loss” wasn’t an aspiration in the plan; it was the constraint the plan had to be built around.

How the cutover worked

The approach was to make the migration boring by making it reversible at every step.

I built the continuous data-movement layer that kept both sides in sync while traffic stayed on the old topology. That turns a cutover from an event into a switch: the target database is already correct and already current, so the decision to move is a routing change rather than a data operation. If the switch is wrong, it goes back, because nothing was destroyed to get there.

I also built the dashboard the team watched it on. That mattered more than it sounds. A migration where only one engineer can tell whether it’s healthy is a migration that runs at the speed of that engineer’s attention. Making replication lag, row counts and divergence visible to everyone turned a held breath into something the whole team could read.

Ten databases became one. No data was lost.

The deployment system

The platform runs as a Laravel API across Compute Engine managed instance groups on GCP. The incumbent deploy tool was PHP-based and struggled with a fleet that changed shape underneath it. I replaced it with a single static Go binary:

• Fleet auto-discovery through the GCP Compute API, so the deployer finds the MIG instances rather than being told about them, plus a static mode for standalone VMs like the queue workers.
• Leader election by lowest internal IP, so exactly one instance runs migrations and the rest wait. Concurrent migrations on a fleet are the failure mode nobody catches in staging, because staging has one box.
• A distributed deployment lock in Cloud Storage using generation-match preconditions, with a five-minute auto-expiry. Correct under concurrent deploys, and it cannot wedge the pipeline if a deploy dies holding it.
• Artifact-based releases, build once from Git, upload to GCS, download and swap on each node. Parallel SSH across the fleet, .env synced from Secret Manager.
• Automatic rollback on partial failure, which is the property that actually matters. A deploy that half-succeeds across a fleet is worse than one that fails cleanly.

Zero-downtime releases, and a deploy became something anyone on the team could run.

Reliability and cost

I carried reliability for the programme alone, alerting, cost control and production incident response, with no rotation behind me. That shapes what you build: anything that needs a human at 3am to interpret it is a design defect, not an operational reality to accept.

Alongside it, two pieces of cost work. I designed a cloud consolidation projected to reduce infrastructure spend by roughly 63%, and replaced the incumbent CDN with an in-house solution that removed the vendor entirely. Both were costed and phased before either was approved, the consolidation’s value was in the sequencing, not the end state, because a migration that can’t be paused halfway is a migration nobody will sign off.

I also wrote the architecture documentation the programme ran on, and built the operational dashboards the Ops and Product teams use day to day.

What I’d take from it

The technical core of this work wasn’t schema design or Go. It was recognising that on a live system, the reversibility of each step is the feature, and that a migration becomes safe at the point where everyone can see its state, not at the point where one person is confident.