Building a Platform We Weren't Allowed to Run

Over the last few years a good chunk of my time at Okta has gone on compliance: PCI, GDPR and HIPAA, and then the US government programmes, FedRAMP Moderate, FedRAMP High and DoD IL4.

The lower levels are mostly about showing you do what you say you do. The higher levels also restrict who is allowed to do it. For the most sensitive environments, anyone operating the platform had to be a US citizen working on US soil.

My teams were in the UK. We could plan, design and build the platform, but once it was live we weren’t allowed to operate it. A US team would be running it, so we had to build it for them.

That had a few knock-on effects on how we worked.

Everything had to be code. We couldn’t log in and fix things by hand, so every change went through the pipeline. That also meant the operators could see exactly what had changed and reproduce it.

The documentation was the handover. Back in 2014 I wrote about the operations documentation problem. Building something we’d never touch in production was a good fix for it. Runbooks, alerts and dashboards all had to make sense to people who hadn’t been involved in the design.

Audit evidence needed to come from the automation. The people who could collect evidence weren’t the people who built the system. Having the automation record what it did as it ran was much easier than hunting for screenshots at audit time.

The operators were our users. We weren’t on call for it, so the alerts and tooling had to be good enough that we’d have been happy to be.

It was an odd way to work and it could be frustrating, but it made us much more disciplined. Building something as if you’ll never be allowed to touch it after go-live is a useful exercise even when there’s no regulation making you do it.