Enterprise Data Integration Platform

What it was

FlowGenX sold a data platform to teams who wanted production pipelines without hiring a data-engineering department. It had two halves that had to behave like one product: a control plane that stood up an isolated Kubernetes environment for each tenant, and a low-code editor where someone who was not an engineer could wire a pipeline out of pre-built components and run it against real data.

What made it hard

Standing up a tenant environment was a six-to-eight hour job of a human reading a runbook and pasting YAML. That was the headline problem, but the harder one sat underneath it: every tenant wanted isolation guarantees that imply a dedicated cluster, and nobody wanted to pay for a dedicated cluster. On top of that, the pipelines were long-running. A job that takes four hours will meet a node eviction, a rate limit, and a malformed record, and it has to survive all three without a human babysitting it. And because customers were pushing customer data through it, PII handling could not be a feature added in the second year.

What I owned

I architected and wrote the provisioning service in Go, including the resource-templating layer, RBAC wiring, and the tenant quota model. I led the workflow editor and the execution engine behind it, and built out the component library the editor draws from. I designed the state management for long-running runs and implemented the privacy and anonymization components. Infrastructure delivery — the CI/CD and monitoring plumbing — was a joint effort with DevOps rather than something I owned alone.

The decision I would defend

Namespace-per-tenant, not cluster-per-tenant. Every tenant gets a namespace with hard resource quotas, network policies, and its own RBAC boundary, all reconciled by ArgoCD from a declarative spec.

The cost is real and I would not pretend otherwise. A shared control plane is a shared blast radius: a kubelet CVE is everyone’s problem, and a tenant that saturates a node degrades its neighbours in ways quotas do not fully prevent. What it bought was the entire premise of the product. Provisioning became a reconcile loop instead of a cluster build, which is the difference between minutes and hours, and the infrastructure bill stayed proportional to actual usage rather than to tenant count.

Where it landed

Provisioning went from most of a working day to minutes. The component library reached 40+ pieces and the platform sustained 2M+ records daily through the execution engine. The privacy layer covered PII across 100k+ customer records with configurable anonymization strategies.

What I would do differently

I picked the wrong granularity for the components. One component to one operation felt clean to me because that is how an engineer decomposes a problem, but the people actually using the editor were analysts, and they ended up building sprawling graphs to express what a single SQL statement says in four lines. Those graphs were genuinely harder to read and debug than the code they replaced, which defeats the point. I would ship fewer, fatter, task-shaped components and put a raw SQL and Python escape hatch in front of users from the first release rather than treating it as an admission of defeat.

The second thing is checkpointing. I designed it as an internal reliability mechanism, so when a run resumed after a failure, operators had no way to see what the earlier attempt had already committed. That turned every recovery into a support conversation. The checkpoint log should have been a user-facing artifact on day one, not an implementation detail.