A new dbt project at Moeve used to take days to get running. Somebody had to build a pipeline for it, stand up infrastructure for it, and decide, again, how it should retry when a run failed.
Now it takes about 15 minutes. According to the team's write-up on the AWS Big Data Blog, onboarding is "a merge of the dbt models and one configuration entry." No Terraform. No new pipeline.
The post is full of service names: Step Functions, Fargate, Athena, DynamoDB, EventBridge. It would be easy to read it as an architecture story and go shopping.
That is the wrong lesson.
Read the before list again
Here is what the team says was broken before:
- Run logic embedded in each team's pipelines
- Different compute engines chosen for similar workloads
- No consistent standards for run parameters and retries
- Cost visible only as an aggregate per account, never per project
None of those is a technology gap. Every one of them is a decision nobody had made once, so every team made it for themselves.
How long should a run wait before it times out? How many times should it retry? Which engine does this kind of work? Who pays for it? Those are policy questions. At Moeve they were being answered in code, inside each pipeline, by whoever happened to be building it that week.
That is what sprawl actually is....
Not too many tools...
Too many private answers to shared questions.
The Monday move
Pick one transformation workload your organization runs in more than one place. Then ask the three teams that run it the same question: what happens when a run fails?
Write down the answers side by side. If they differ, you have not found a tooling problem. You have found a decision that nobody owns, being made quietly in code, over and over.
Make it once. Write it where anyone can read it. Put a name next to it.
The platform can come later, and it will be easier to build, because it will finally have something to enforce.
So: how many answers does your organization currently have to the question "how many times do we retry?"
Sources
- How Moeve standardized dbt runs across data lakes with Amazon Athena, AWS Big Data Blog