Posts on Ghost in the data

The Style Isn't the Skill

Chris Hillman — Mon, 31 Aug 2026 09:00:00 +1000

I was a kid when I first watched Bruce Lee, soon after and my friend signed up for kung fu lessons.

What hooked me wasn’t the fighting. It was the speed at which Bruce moved. The more you watched, though, the more you noticed something sitting behind the speed. He was obviously smart, fast, cool, and clearly thinking several moves ahead of everyone else in the room. Even as a kid you could tell the fighting was the visible bit of something much bigger.

Looking back, the thing I admired was the confidence. Not swagger. This was different. It was the kind of confidence that comes off someone who has poured a frightening amount of themselves into one craft. You can see it on people in all areas of life. I’m sure you’ve seen it in your industry. It could be a senior data engineer walking a junior through a query plan without looking at the screen. A modeller who hears the grain of a new fact table and already knows which join is going to hurt. It radiates, and it can’t be faked for long.

Then people will say…

Yeah, but he’s Bruce Lee.

You’ve probably said the same about someone like him. It sounds like humility, but it isn’t. It’s a permission slip — if he’s a different species, then the work he did to become that good is not available to me, and I’m excused from trying.

The problem with this is that it never was easy for Bruce Lee, and I’m sure it’s the same for others as well. The three moments that made Bruce Lee happened before anyone outside a few gyms and a cancelled TV show knew who he was. He wasn’t Bruce Lee yet. He was a Wing Chun kid from Hong Kong teaching out of a friend’s house in Oakland, then a sidekick on a show that lasted one season, then a man in a back brace. Enter the Dragon came out weeks after he died. Everything that matters here is from before.

1964: the messy win

In late 1964 Lee fought Wong Jack Man in Oakland, behind closed doors, in front of a handful of people. There are disagreements as to how it went. Lee’s side says he finished it in about three minutes. Wong’s side says it dragged past twenty and ended with both men exhausted. What everyone agrees on is that it was ugly, and that Lee came out of it winded and rattled. For a man who had spent the previous year telling anyone who’d listen that his technique was superior to the “classical” styles, it was a humiliating result. He’d won, by his own account. It didn’t feel like winning.

What he did next is the interesting part. He didn’t defend the technique. He tore it up, and hit the reset switch. That fight is the generally accepted origin of Jeet Kune Do, and the first thing he changed wasn’t his kung fu. It was his conditioning. He went looking for far more extreme fitness training than any traditional school would have asked of him.

The line he used to explain the whole shift:

“The proper method is the one that works.”

As with most things that Bruce Lee said, there is so much in those few words. It’s easy to read it as “method doesn’t matter,” and that’s the opposite of what it says.

It says there is a proper method. It says the test of whether a method is proper is a single question: does it work? And it says, quietly, that the answer can change. A method that worked last year against last year’s opponent gets re-tested, and if it fails, it isn’t proper anymore. Loyalty to it is not a virtue.

He made the point a second time, in a way only Bruce Lee would. He put a mock tombstone in his studio. The inscription:

“In memory of a once fluid man, crammed and distorted by the classical mess.”

He wasn’t mourning kung fu. He was mourning himself. The “classical mess” was the ceremony of the art mistaken for the art: the numbered forms, the fixed stances, the correct sequence of steps that everyone agreed was correct because everyone had always agreed. He’d drilled it for years. And in a real fight, against an opponent who didn’t know the choreography, it had crammed and distorted him.

On one side of the spectrum there might be a template or process that was set up. Does it work? Or is the only answer “it’s how we’ve always done it”? Sometimes we don’t spend the time reflecting — it might have worked perfectly initially, but is it as effective now? There’s also the other side, where we make exceptions to the process far too often, to the point it becomes a mess and we no longer know what the technique even is.

Bruce was ruthless about fixed form. He was obsessive about fundamentals. His response to a bad fight was more discipline, not less. The freedom he’s famous for was built on top of conditioning most people wouldn’t tolerate.

The style isn’t the skill. That’s the whole thing.

One kick, ten thousand times

One of Bruce’s quotes was:

“I fear not the man who has practiced ten thousand kicks once, but I fear the man who has practiced one kick ten thousand times.”

Ten thousand kicks once is a career spent collecting — a new tool every quarter, every certification going, every architecture pattern from every conference talk. You can talk about all of them. You’ve done each of them once, in a tutorial, on a dataset that was designed to make the tutorial work.

One kick ten thousand times is mastery of the skill. It isn’t knowing the technique. It’s having met every way the technique fails. I see this a lot meeting people with various titles: have they actually dealt with quality incidents, and how did they protect the data from it? Have they worked with a source system that offers deltas, and tested them thoroughly? The engineer who’s built them fifty times knows what happens when the source system back-dates a change, when two updates land in the same batch with the same timestamp, when the current flag and the high date disagree, and when a backfill quietly rewrites six months of history because someone ran it day by day. (I wrote a whole post about that last one, because it took me an embarrassing number of reps to see it coming.)

So pick the kick. Dimensional modelling. Streaming ingestion. Query performance on the warehouse you actually run. Pipeline testing. Not all of them. One, then maybe a second. The industry will keep offering you ten thousand shiny new ones, and most of them will be gone in three years. However, honing these skills is what shines out in interviews I conduct — I can see the difference between someone who knows how to kick, versus someone who has done that kick thousands of times.

Absorb, discard, add

The quote that describes how Lee built Jeet Kune Do out of Wing Chun, Western boxing, fencing, and whatever else he could get his hands on:

“Absorb what is useful, discard what is not, add what is uniquely your own.”

Absorb what is useful. Not “learn everything.” Absorb implies digestion. You take the thing in and it becomes part of how you move, without needing to consult it. Kimball’s grain-first modelling is worth absorbing. So are idempotent loads, and the habit of writing the test before the transformation. Once absorbed, you stop noticing you’re doing them.

Discard what is not. Harder than it sounds, because discarding means admitting you spent time on something that didn’t pay off. The pattern that made sense at your last company and doesn’t fit this one. The tool you got certified in that your current platform doesn’t run. Lee walked away from the style his own master taught him. Most of us can’t walk away from a Confluence page.

Add what is uniquely your own. What have you built that isn’t in any documentation? The reconciliation check you wrote because the standard tests kept missing one specific failure. The way you explain a lineage problem to finance so they stop asking the same question every month. That’s yours. It’s the part nobody can hire off the shelf. If your self-evaluation lists things you absorbed and nothing you added, you’ve described a good student, not a good engineer.

Water and the shape of the cup

“You put water in a cup, it becomes the cup.”

Water has no form of its own; it takes the form of whatever holds it. Bruce also points out that water can flow or it can crash. Formless isn’t the same as weak.

For a data modeller this is close to a job description. The model takes the shape of the question, not the shape of the framework diagram. A star schema is right until the business hands you a many-to-many relationship that every consumer needs, and then it’s wrong, and the modeller who is loyal to the star instead of the question will build something the analysts route around within a month. I’ve argued before that there’s no single right way to do data engineering, only the right way for the world you’re in. Lee got there fifty years earlier with the motto he put on the Jeet Kune Do emblem:

“Using no way as way, having no limitation as limitation.”

No way as way doesn’t mean no method. It means the method is chosen fresh for each fight. Kimball for the reporting layer, something flatter and wider for the feature store, streaming where latency is the product and batch everywhere else, with no shame in any of it. The limitation you refuse to accept is the one that says “we’re a Kimball shop” or “we don’t do streaming here” before anyone has looked at the problem.

1969: the letter

January 1969. Lee is twenty-eight. The Green Hornet had come and gone after one season. He’s picking up bit parts and teaching private lessons to actors to pay the bills. Linda is pregnant with their second child. Bruce’s Hollywood career has stalled.

He sits down and writes himself a letter. It’s on a sheet of notepaper headed “Secret,” in red and blue ink, and he titles it “My Definite Chief Aim.” The whole thing is four sentences. In the first he declares that he will be the first highest paid Oriental superstar in the United States (his words, and the era’s). The last two set a dated target and a personal one: world fame from 1970, ten million dollars by the end of 1980, and a life lived the way he pleases.

The sentence in the middle is the one that matters most:

“In return I will give the most exciting performances…”

This isn’t a wish list. It’s a contract — here’s what I will be, here’s what I will give in return.

I’ve read enough self-evaluations to notice a pattern. Most of them are the first sentence without the second. “I want to grow into a senior role.” “I’d like more exposure to the streaming work.” “I’m keen to lead a project next year.” All fine. All wishes. The “in return” clause is missing, and without it there’s nothing for a manager to hold onto and nothing for you to hold yourself to.

Try writing it Lee’s way. Something like: By the end of June I will be the person this team hands streaming ingestion to without a second thought. In return I will have built late-arrival handling on both event pipelines, documented the watermark strategy so someone else can run it, and carried the on-call for it the first three times it breaks.

That’s a sentence a manager can act on. It’s also a sentence that scares you slightly to write, which is how you know it’s the real one. (He achieved everything in the letter, incidentally, and then died at thirty-two.)

In the review room

When I sit with someone to go through their self-evaluation, I’ve started asking myself a slightly ridiculous question: what would Bruce Lee do with this? Ridiculous or not, it changes what I look for.

If you’re writing the evaluation.

Write the “in return” clause. Every aim gets a consideration.

Name your kick. What’s the one thing you’ve done enough times that you know how it fails? If you can’t name one, that’s the finding, and it’s a useful one.

Include something you added, not just something you absorbed. The thing you built that wasn’t in the docs.

If you’re reading it.

Judge what works, not adherence to form. The engineer who follows every ceremony and ships fragile pipelines has the classical mess. Say so, kindly and specifically.

But check the reps. “Judge outcomes” drifts very easily into “reward whoever got lucky.” Someone who has practised one kick ten thousand times has a track record of predictable outcomes, and that’s the difference between skill and a good quarter.

Look for the gap between knowing and doing. Lee was fond of a line he borrowed from Goethe: knowing is not enough, we must apply; willing is not enough, we must do. Most people in a review know exactly what they should be doing. The evaluation is a measure of the distance between that and what happened.

If you want a structured way to run that kind of exercise with your own team, I’ve written before about the Reflective Best Self process.

The tradeoff

Judging by what works is harder than judging by ceremony. Ceremony is easy to audit: the template was filled in, the points were estimated, the meeting was attended. Skill is hard to audit, because you have to understand the work well enough to know whether the outcome was earned or lucky, and that takes time managers don’t have and technical depth some of them stopped maintaining years ago. Take the Lee approach and you sign up for that cost.

There’s a cost on the other side too. A self-evaluation written without the mask, with a real “in return” clause and a named weakness, is a sharper tool for your own growth and a sharper tool in the wrong hands. In a healthy team it gets you coached. In an unhealthy one it gets quoted back at you.

AWS MWAA: The Practitioner's Unvarnished Guide

Chris Hillman — Sat, 18 Jul 2026 09:00:00 +1000

I remember the first time I sat down properly with the Step Functions architecture our team inherited. Someone had designed it thoughtfully — real engineering thought had gone into it when it was built.

But what I was looking at in the console was a map of state machines and Lambda functions that had grown well past the point where anyone could hold it in their head.

The first production issue we had to debug on that setup took most of a working day. Not because the problem was complicated — it wasn’t — but because finding it meant jumping between CloudWatch log groups, correlating timestamps, building a mental picture of which Lambda had received what input and where the chain had broken. Every time I thought I had it, another log group. Another timestamp comparison. Another dead end.

That’s when I suggested to the team that we test out Airflow. I remember the first time something failed on the new setup. I clicked into the task, read the log attached to that specific run, saw the error, fixed it, and restarted the DAG. Four minutes. The CDK stacks for the old architecture ran to thousands of lines. The equivalent DAG was fifty to two hundred. Same logic. Made legible.

The difference wasn’t performance — Lambda and Airflow are comparable where it counts. It was visibility. One place to look. That’s what orchestration is actually about: not execution speed, not infrastructure elegance, but whether you can understand what’s happening in your data platform — at 9am in a post-mortem, or at 2am when something breaks and you need to know why.

AWS MWAA — Managed Workflows for Apache Airflow — is Amazon’s answer to this problem. And it’s genuinely good at some things. But the gap between what the marketing page promises and what you’ll actually experience in production is wider than most managed services I’ve worked with. The $365/month sticker price? That’s just the beginning. The “managed” part? It manages less than you’d think.

This guide covers everything I wish someone had told me before committing to MWAA — the real architecture, the actual costs (including the ones AWS conveniently leaves off the pricing page), the operational gotchas that’ll bite you in production, and honest head-to-head comparisons with every serious alternative: Astronomer Astro, Dagster Cloud, Prefect Cloud, Step Functions, and the self-hosted approach that might be smarter than you’d expect.

How MWAA Actually Works Under the Hood

MWAA runs on a dual-VPC architecture that’s important to understand because it explains most of the service’s constraints.

AWS provisions Fargate containers for your schedulers, workers, and web server inside an AWS-managed ECS cluster. These containers connect to your VPC through Elastic Network Interfaces injected into your private subnets. A single-tenant Aurora PostgreSQL instance stores the Airflow metadata — fully managed by AWS, but completely inaccessible to you. No connection string. No direct SQL queries. No custom dashboards built against the metadata database. If you’ve ever relied on querying task_instance or dag_run tables directly for custom monitoring, MWAA takes that away.

The networking requirements are specific and unforgiving. You need two private subnets in different Availability Zones, a NAT gateway (or VPC endpoints) for internet access, and a self-referencing security group. The VPC you choose at creation is permanent — change your mind later and you’re building a new environment from scratch. Choose private webserver mode (which most production teams do for security), and accessing the Airflow UI requires a VPN or bastion host setup.

DAGs live in an S3 bucket with versioning enabled. MWAA syncs files to the workers every 30 seconds, though new DAG files take roughly five minutes to be recognised (controlled by dag_dir_list_interval). Updates to requirements.txt or plugins.zip trigger a full environment update — and that means 20 to 40 minutes of waiting while MWAA reprovisions everything. More on that particular joy later.

A few setup details that’ll save you time: the S3 bucket must be in the same region as your MWAA environment and must have block public access enabled. Your dags/ folder sits at the root of the bucket, with requirements.txt alongside it (not inside the dags/ folder). The plugins.zip file follows a strict structure — if your custom operator lives at plugins/operators/my_operator.py, the zip must mirror that path exactly. Get the structure wrong and the import will silently fail with no useful error message until you dig into the DAGProcessing logs.

For IAM, MWAA needs an execution role with access to your S3 bucket, CloudWatch Logs, SQS (for the Celery broker), and whatever AWS services your DAGs interact with — Glue, EMR, Redshift, Lambda, whatever. The principle of least privilege matters here, but AWS’s own documentation starts with fairly broad permissions and leaves tightening to you. In practice, most teams start broad and restrict after stabilising.

Environment classes span six tiers since the 2024 additions, from micro through 2xlarge:

Class	vCPU / Memory per Worker	Concurrent Tasks/Worker	Approx. DAG Capacity	Base Cost/hr
mw1.micro	Combined scheduler/worker: 1 vCPU, 3 GB	3	~25	~$0.05
mw1.small	1 vCPU, 2 GB	5	~50	$0.49
mw1.medium	2 vCPU, 4 GB	10	~250	~$0.74
mw1.large	4 vCPU, 8 GB	20	~1,000	$0.99
mw1.xlarge	8 vCPU, 24 GB	40	~2,000	~$1.49
mw1.2xlarge	16 vCPU, 48 GB	80	~4,000	~$1.99

The micro class (November 2024) is a genuine game-changer for dev/test — at roughly $37/month, you can finally have a development MWAA environment without the $365+ price floor. The catch: it combines the scheduler and worker into a single container with no autoscaling. Fine for testing DAGs, not for production.

Worker autoscaling follows a straightforward formula: (running + queued tasks) / (tasks per worker) = required workers. Scale-down triggers after running and queued tasks hit zero for more than two minutes, and takes two to five minutes to complete. A significant autoscaling bug — where tasks could be assigned to workers being decommissioned — persisted until May 2025. If you’re running a version from before that fix, you’ll want to upgrade.

The True Cost of MWAA (It’s Not $365 a Month)

The AWS pricing page says mw1.small costs $0.49/hr, which works out to $364.56/month. That number is misleading. Every production MWAA deployment carries infrastructure costs that the headline figure conveniently excludes.

NAT gateways are the number one hidden cost. The standard architecture requires two NAT gateways (one per AZ), costing $0.045/hr each — that’s $65.70/month before a single byte of data moves through them. Add $0.045/GB for data processing. One team documented a $12K monthly bill where 87% was unused NAT gateway costs from misconfigured routing. This is not an MWAA-specific problem, but MWAA forces you into it because of the VPC requirement.

CloudWatch metrics are the second surprise. MWAA auto-publishes per-DAG and per-task metrics, and one practitioner documented $272/month for custom metrics alone — 907 metrics at $0.30 per metric. Unless you actively restrict what gets published using metrics.statsd_allow_list, you’ll see this line item growing steadily as you add more DAGs.

CloudWatch Logs at INFO level with active workflows adds $50–300/month in ingestion and storage charges, depending on how chatty your DAGs are.

Here’s what a small team running a production mw1.small environment actually pays:

Component	Monthly Cost
mw1.small environment (24/7)	$364.56
2 NAT gateways (hourly)	$65.70
NAT data processing (~20 GB)	$0.90
CloudWatch metrics	$30–100
CloudWatch Logs (~2 GB)	$1.00
Elastic IPs (2)	$7.30
S3 + metadata storage	$1.50
Total	$470–540

Scale up to mw1.large with additional workers running six hours a day, extra schedulers, and full logging, and you’re looking at $2,150–2,300/month all-in. AWS’s own pricing example for a comparable setup quotes $1,047/month — but conveniently excludes NAT, CloudWatch, and networking overhead.

Here’s that breakdown for a heavier workload — say a mid-size data team running 200+ DAGs with burst periods:

Component	Monthly Cost
mw1.large environment (24/7)	$733.68
19 additional workers (6 hrs/day avg)	$672.00
5 schedulers	$164.00
2 NAT gateways (hourly)	$65.70
NAT data processing (~100 GB)	$4.50
CloudWatch metrics (400+)	$120.00
CloudWatch Logs (~10 GB)	$5.03
Elastic IPs, S3, misc.	$10.00
Total	~$1,775–2,300

The range depends heavily on how well you control CloudWatch metric sprawl and how long your burst periods actually last. The point is: you need to model the real infrastructure cost, not just the MWAA line item.

Cost optimisation levers do exist. Replace NAT gateways with VPC endpoints — the S3 Gateway Endpoint is free, and interface endpoints process data at $0.01/GB versus NAT’s $0.045/GB. Restrict CloudWatch metrics with metrics.statsd_allow_list = scheduler,executor or disable custom metrics entirely. Set log retention to 30–90 days instead of indefinite. For dev/test environments, implement pause/resume schedules — one team reported 70% savings by running MWAA only during business hours.

The November 2025 launch of MWAA Serverless fundamentally changes the cost equation for intermittent workloads. At $0.08/hr per task (billed per-second, minimum one minute), 2,000 tasks averaging one to two minutes costs roughly $4/month — versus $365+ for the always-on equivalent. The trade-off is substantial though: no Airflow UI, YAML-based workflow definitions, and only 80-odd AWS operators supported (no PythonOperator, no BashOperator). It’s better understood as a different product targeting a different use case than as a cheaper MWAA.

The Gotchas

Practitioners consistently flag the same pain points, and understanding them before committing is worth more than any architecture diagram.

Dependency management is the highest-risk operational area. A bad requirements.txt can crash-loop your entire environment for hours. Not minutes — hours. The constraint statement is mandatory since Airflow 2.7.2, and omitting it risks catastrophic dependency conflicts. If pip install exceeds ten minutes, the Fargate task times out and rolls back — leaving your environment stuck in an update cycle. Certain packages can break MWAA’s CloudWatch logging by overriding the watchtower library. The only safe approach: always test with the aws-mwaa-local-runner Docker tool before deploying dependencies to production.

Environment update times frustrate everyone. Creating an environment takes 25–30 minutes. Updating requirements or plugins: 20–40 minutes. Version upgrades: up to two hours with the environment unavailable during the process. The May 2025 “graceful updates” feature helps — it replaces components without interrupting running tasks — but the wall-clock time for the update itself hasn’t changed meaningfully.

Observability is CloudWatch-only, and navigating it is painful. MWAA creates five separate CloudWatch log groups per environment: DAGProcessing, Scheduler, Task, WebServer, and Worker. Correlating a single pipeline failure requires jumping between log groups, and CloudWatch Logs Insights queries aren’t intuitive for Airflow-style troubleshooting. You can’t redirect StatsD metrics to external tools like Datadog or Prometheus — MWAA overrides the statsd_host configuration. External monitoring integration requires CloudWatch Logs subscriptions via Kinesis Data Firehose, adding both complexity and cost.

If you’ve spent time in the trenches of AWS logs — and maybe this is just me — it can be a genuine nightmare to piece together what happened across services. MWAA should improve this, and in some ways it does (the Airflow UI’s task logs are excellent). But the moment you need infrastructure-level debugging, you’re back in CloudWatch hell.

The 12-hour task execution limit is a hard wall. Tasks exceeding 12 hours are killed and get stuck in the SQS queue for another 12 hours due to SQS timeout settings. The CeleryExecutor-only constraint means no per-task Docker images and no pod-level resource isolation. If you need tasks that run longer than 12 hours, you’ll need to offload compute to ECS Fargate or Lambda, using MWAA purely for orchestration.

Secrets Manager integration works but generates surprising API volume. Without configuring connections_lookup_pattern, Airflow attempts to look up every connection in Secrets Manager — including non-existent ones. One practitioner reported 100,000+ error API calls per day per MWAA instance. Secrets stored in Secrets Manager don’t appear in the Airflow UI either, which only shows connections from its own backend.

Other gotchas worth noting: you can’t remove plugins.zip or requirements.txt once added (you can only point to empty files), the VPC can’t be changed after creation, frequent DAG updates to the S3 folder can break the MWAA installation, and if VPC endpoints are accidentally deleted the environment is broken and must be entirely recreated.

One more that catches people off guard: Airflow configuration overrides. MWAA lets you set most Airflow configuration options through the console or API, but several are explicitly blocked — statsd_host, statsd_port, broker_url, result_backend, and anything related to the metadata database connection. These are locked down because MWAA manages those components. If your Airflow experience includes tuning Celery broker settings or metadata database connection pools, you’ll need to adjust your expectations about what “managed” means here. It means AWS decides those values, and some of them (particularly the default Celery broker configuration) aren’t optimal for all workload patterns.

The practical impact: if you’ve tuned self-hosted Airflow for specific performance characteristics — aggressive task polling, custom broker acknowledgement settings, metadata database query timeouts — MWAA won’t let you replicate that tuning. For most teams this doesn’t matter. For teams at scale with specific performance requirements, it can be a dealbreaker.

Head-to-Head: Astronomer Astro

Astronomer’s Astro platform runs Apache Airflow underneath, so your DAGs, operators, and muscle memory all transfer. The differences are in the operational layer — and they’re significant.

Workers scale to zero. When no tasks are running, you pay nothing for compute. MWAA charges for the environment 24/7 regardless. Astro’s base pricing starts at $0.35/hr for the control plane, with worker costs at $0.13/hr that only accrue when tasks actually execute.

The developer experience gap is substantial. Astro CLI provides a full local Airflow environment with astro dev start that mirrors production exactly — same Airflow version, same Python version, same provider packages. MWAA has no equivalent. The aws-mwaa-local-runner Docker tool is helpful but doesn’t perfectly replicate the MWAA environment. DAG deployment via astro deploy takes 5–10 minutes through a proper CI/CD pipeline; MWAA’s S3 upload plus sync cycle is slower and less integrated into developer workflows.

Astro supports the KubernetesExecutor natively, meaning you get per-task Docker images, custom resource limits, and the thin orchestrator pattern described below — without managing Kubernetes yourself. MWAA is locked to CeleryExecutor with no workaround.

Campspot, which migrated from MWAA to Astro, reported that a critical nightly job went from over two hours to two to three minutes. Another company, Black Crow AI, recouped roughly 20% more engineering time after leaving MWAA, citing zombie task issues and the absence of Airflow-specific support.

Astro’s Observe feature provides built-in pipeline lineage, SLA tracking, and data quality monitoring — capabilities absent from MWAA. Support comes from actual Airflow core committers, versus MWAA’s generic AWS support channels.

The migration story from MWAA to Astro is worth paying attention to because it reveals what teams actually struggle with. Campspot completed their migration in a two-week sprint — their DAGs transferred with minimal modification because both platforms run Airflow. The performance improvement came not from different code, but from better infrastructure: KubernetesExecutor allowing parallel task execution in isolated pods versus CeleryExecutor bottlenecking through shared workers. Black Crow AI’s experience was similar — the DAGs were the same, but the operational layer (scaling, monitoring, deployment speed) was categorically better.

That said, Astronomer is a startup, not AWS. If your organisation’s procurement process requires a vendor that’s been around for 20+ years, has SOC 2 Type II (Astronomer does have this, to be fair), and won’t disappear if funding dries up — that’s a valid concern. Astronomer raised significant funding and is growing, but the risk calculus is different from buying from AWS.

The trade-offs: Astro’s dedicated clusters cost $2.40/hr ($1,782/month at the cluster level), and networking costs (NAT, PrivateLink) are passed through from your cloud provider. For teams deeply embedded in AWS wanting consolidated billing and native IAM integration without multi-cloud requirements, MWAA’s tight ecosystem coupling has genuine value. But if developer experience and operational flexibility are priorities, Astro is meaningfully ahead.

Head-to-Head: Dagster Cloud

Dagster represents the most fundamental philosophical departure from the Airflow model — and by extension, from MWAA.

Where Airflow (and MWAA) asks “which tasks ran successfully?”, Dagster asks “which data assets are fresh, and why?” The difference sounds academic until you’re debugging a pipeline at 7am and you want to know not just which task failed, but which downstream datasets are now stale and which business reports can’t be trusted.

Software-Defined Assets are Dagster’s core abstraction. Each asset declares its dependencies, its materialisation logic, and its freshness expectations. The framework automatically builds a dependency graph, tracks lineage at the column level, and can tell you in real time which assets are stale, which are being materialised, and which are healthy. This is observability that MWAA simply can’t match — you’d need to bolt on separate lineage tools, metadata platforms, and custom monitoring to approximate what Dagster provides natively.

The developer experience is excellent. dagster dev starts a local instance immediately without Docker. Branch deployments create isolated staging environments from pull requests — push a feature branch, get a dedicated Dagster instance for testing. dbt integration maps models directly into the asset graph, so your dbt transformations are first-class citizens alongside your Python pipelines.

Dagster+ pricing uses a credit-based model: one credit equals one asset materialisation. The Solo tier starts at $10/month, Starter at $100/month. For infrequent batch workloads under 30,000 materialisations per month, Dagster is dramatically cheaper than MWAA. But credits can scale unpredictably — one practitioner calculated that an 8-operation job running every five minutes consumed 69,120 credits per month, costing $2,464 at Solo rates. For high-frequency, multi-step jobs, MWAA’s predictable hourly pricing may actually be more economical.

The ecosystem trade-off is real. Airflow has 1,600+ operators and a community of 80,000+ organisations built over a decade. Dagster’s integration library is growing rapidly but isn’t as broad. If you need a provider for an obscure SaaS API or legacy system, Airflow almost certainly has one; Dagster might not. For teams starting fresh without Airflow baggage, Dagster’s asset-centric model is compelling. For teams with hundreds of existing DAGs, the migration cost is substantial.

There’s also a philosophical difference worth naming. Airflow thinks in terms of schedules and tasks — “run this DAG at 6am, execute these tasks in order.” Dagster thinks in terms of data freshness and assets — “these datasets should be no more than 2 hours stale, materialise them as needed.” Both models work, but they lead to fundamentally different approaches to monitoring, alerting, and debugging. If your team’s primary question is “did the 6am job run?” then Airflow’s model fits naturally. If your question is “is the executive dashboard showing current data, and if not, what’s stale?” then Dagster’s model is more direct.

The deployment model also differs meaningfully. Dagster+ offers both Serverless (fully managed, AWS-hosted) and Hybrid (Dagster manages the control plane, your infrastructure runs the compute). The Hybrid model gives you the thin orchestrator pattern with professional management of the scheduling layer — a compelling middle ground between fully managed and fully self-hosted.

Head-to-Head: Prefect Cloud

Prefect takes the most Python-native approach to orchestration. Add @flow and @task decorators to regular Python functions. No DSL, no custom operators, no XCom for passing data between tasks — just Python.

This matters most for teams where the data engineers are primarily Python developers who find Airflow’s operator model cumbersome. Dynamic workflows, event-driven triggers, and runtime conditional logic are all first-class features in Prefect. Need to loop over a list of files and process each one as a separate task? That’s a Python for-loop, not an Airflow dynamic task mapping exercise.

Prefect Cloud’s free Hobby tier (2 users, 5 deployments) lets small teams start at $0 — versus MWAA’s $365+ minimum. The Starter tier at $100/month includes bring-your-own-compute, meaning you still pay separately for the infrastructure where your flows actually run. Prefect Cloud is a control plane, not a compute platform — similar to the thin orchestrator pattern, but with Prefect managing the scheduling and monitoring layer instead of Airflow.

The honest limitation: Prefect’s community is a fraction of Airflow’s. Airflow pulls 30 million monthly downloads; Prefect sits around 1.8 million weekly. That gap means fewer Stack Overflow answers, fewer blog posts solving your exact problem, and fewer engineers who already know the tool when you’re hiring. For teams where hiring velocity matters, Airflow’s ubiquity is a genuine competitive advantage — and MWAA inherits that.

Where Prefect genuinely shines is the rapid-prototyping-to-production pipeline. You can convert an existing Python script into an orchestrated workflow by adding decorators — no rewriting into operator patterns, no separating logic from configuration, no learning a new abstraction layer. One engineering team compared this directly to their Airflow experience and found that Prefect flows took roughly half the time to develop and deploy. The trade-off is less structure — Airflow’s opinionated DAG model forces a discipline that Prefect’s flexibility doesn’t enforce.

Prefect Cloud’s Pro tier ($500/month) adds RBAC, audit logs, and custom retention policies. The Enterprise tier (custom pricing) adds SSO, dedicated infrastructure, and priority support. For context, an MWAA mw1.small environment costs $470–540/month all-in — roughly equivalent to Prefect Pro before you add your own compute infrastructure. The total cost comparison depends entirely on how much compute your flows need and where it runs.

One more thing worth noting: Prefect 2 (the current generation) was a ground-up rewrite that broke compatibility with Prefect 1. If you’re evaluating Prefect, you’re evaluating a framework that made the decision to start over once already. That took conviction, and the result is a significantly better product — but it’s worth understanding that the ecosystem is younger than the company’s founding date suggests.

Head-to-Head: AWS Step Functions

Step Functions is the right answer more often than most Airflow advocates want to admit.

For a 10-step workflow running once daily (~300 state transitions per month), Step Functions costs nothing within the free tier. MWAA’s minimum is $365+. Even at medium scale — 100 executions per day — costs stay under $1/month for standard workflows. The Distributed Map mode can process up to 10,000 parallel S3 objects; one AWS demo processed 560,000 CSV files in 100 seconds.

Step Functions also runs natively serverless with zero infrastructure management. No VPC, no NAT gateways, no requirements.txt crashes, no 20-minute environment updates. For teams building AWS-native event-driven architectures — S3 events triggering Glue jobs, Lambda processing, DynamoDB writes — Step Functions integrates with over 220 AWS services directly in the workflow definition.

The limitations are real though. The 256KB payload limit between states constrains data-heavy handoffs. JSON-based Amazon States Language for workflow definitions is verbose and hard to debug. There’s no built-in scheduling (you need EventBridge), no backfill capability, no equivalent of Airflow’s catchup=True for replaying historical runs. The 25,000-event history limit per standard execution bites data engineering workloads — though Express Workflows (5-minute maximum duration, $1 per million requests) eliminate that constraint for short-lived jobs.

For complex data pipelines with interdependencies, branching logic, retries, and backfill requirements, MWAA is genuinely the better tool. For linear workflows, event-driven processing, and microservice orchestration, Step Functions wins on cost, simplicity, and operational overhead.

Beyond Named Platforms: A Pattern Worth Considering

Beyond named platforms, there’s an architectural pattern that deserves consideration alongside them — one that doesn’t come with a vendor attached.

The idea is simple: run Airflow as a lightweight scheduler and UI on minimal infrastructure, and delegate all actual compute to Kubernetes pods or ECS tasks that spin up on demand.

In this model, your Airflow installation is deliberately “dumb.” A small EC2 instance or a modest EKS deployment runs just the webserver, scheduler, and metadata database. Workers don’t exist in the traditional sense. Instead, every task in your DAG uses the KubernetesExecutor (or EcsRunTaskOperator) to launch a purpose-built container that does the actual work, then terminates.

Here’s why this matters:

Per-task isolation. Each task runs in its own container with its own Docker image, its own dependencies, its own resource limits. Your dbt task runs a slim Python image with just dbt-core. Your Spark submission task runs an image with PySpark and your JAR files. Your ML training task gets a GPU-enabled image with PyTorch. No dependency conflicts. No shared memory pressure. No crash-loop risk from a bad requirements.txt — because each task manages its own dependencies independently.

Scale to zero. When nothing is running, you’re paying for the scheduler and webserver — maybe $50–80/month on a small EC2 instance or ECS task. When a hundred tasks fire simultaneously, Kubernetes spins up a hundred pods, each with exactly the resources that task needs. When they finish, those pods terminate and you stop paying. MWAA, by contrast, charges for workers 24/7 regardless of whether tasks are running.

No CeleryExecutor limitations. The KubernetesExecutor gives you everything MWAA’s CeleryExecutor can’t: custom Docker images per task, no 12-hour execution limit (Kubernetes pods can run indefinitely), pod-level resource requests and limits, and the ability to use Spot instances for worker nodes at 50–70% cost savings.

The cold-start trade-off. Spinning up a Kubernetes pod takes 15–30 seconds. For batch workloads that run hourly or daily, this is negligible. For real-time or sub-minute scheduling, it’s not ideal — but those workloads probably shouldn’t be in Airflow anyway.

What does this actually look like in practice? Your DAG file stays clean and declarative. Instead of using PythonOperator to run transformations inside the Airflow worker, you use KubernetesPodOperator pointing at a Docker image that contains your transformation logic and its dependencies:

transform_sales = KubernetesPodOperator(
 task_id="transform_sales_data",
 image="your-ecr-repo/sales-transform:v2.3",
 namespace="airflow-tasks",
 resources={"request_cpu": "2", "request_memory": "4Gi"},
 node_selector={"lifecycle": "spot"}, # 50-70% cost savings
 is_delete_operator_pod=True,
)

Each task declares exactly what it needs. Your dbt task runs a slim image. Your Spark submission gets a heavier one. Your ML training task gets GPU resources. And because each pod is ephemeral, a dependency conflict in one task can never affect another — the isolation is complete.

So is this just a slower version of what MWAA does? No — it’s architecturally different in important ways.

MWAA uses CeleryExecutor with always-on Fargate workers. Tasks run inside those workers, sharing the same Python environment, the same dependencies, the same memory. The workers exist whether tasks are queued or not. When you update requirements.txt, every worker gets rebuilt (hence the 20–40 minute update times).

The thin orchestrator pattern puts compute outside the scheduler entirely. Airflow becomes a pure control plane. The data plane — where actual work happens — is ephemeral and independently deployable. You can update a task’s Docker image in seconds without touching the Airflow environment at all.

The self-hosted overhead is real though. You’re managing the Airflow installation yourself: upgrades, database administration (or using Amazon RDS for the metadata database), security patching, and monitoring setup. Budget 0.25–0.5 FTE for a small-to-medium deployment. At $150K/year fully loaded, even 0.25 FTE equals $3,125/month in engineering time — which can exceed MWAA’s premium for teams that don’t already have Kubernetes expertise.

This pattern works best when your team already runs Kubernetes for other workloads (so the infrastructure cost is shared), you need per-task dependency isolation, or you’re running compute-heavy tasks where Spot instance pricing makes a material difference.

One team running 80 pipelines on self-hosted ECS reported $400–500/month total compute costs. A comparable MWAA setup would run $600–900/month. But the engineering time to maintain the self-hosted setup is the real cost comparison — and it varies enormously depending on team capability.

When MWAA Is — and Isn’t — the Right Choice

MWAA earns its place when three conditions align: deep AWS commitment (your stack is S3, Redshift, Glue, EMR), existing Airflow investment (DAGs, team expertise, operator familiarity), and a preference for managed infrastructure over platform engineering. Teams running 50–500 DAGs on AWS with two or more data engineers and no multi-cloud requirements will find MWAA productive and cost-effective relative to the operational burden of self-hosting.

Here’s where it falls short:

Intermittent or light workloads — unless you’re willing to accept MWAA Serverless’s operator restrictions, the always-on cost floor is too high. Step Functions or Prefect’s free tier are dramatically cheaper.

Multi-cloud requirements — MWAA is AWS-only. Astronomer or self-hosted Airflow provides portability across clouds.

Asset-centric observability — if data lineage, freshness tracking, and column-level quality monitoring are priorities, Dagster Cloud is purpose-built for this. Bolting these capabilities onto MWAA requires multiple additional tools and significant integration work.

KubernetesExecutor or per-task isolation — MWAA is locked to CeleryExecutor. Teams needing custom Docker images per task or pod-level resource isolation should look at Astronomer, the thin orchestrator pattern on self-hosted, or Dagster Cloud.

Rapid Airflow version adoption — MWAA lags upstream releases by 2–6 months. Astronomer typically offers day-zero support for new versions.

Cost sensitivity at scale with multiple environments — per-team MWAA environments compound quickly. Three teams each needing their own environment is $1,100+/month before any work happens. Astronomer’s scale-to-zero workers or self-hosted with Spot instances become significantly cheaper at this scale.

For teams genuinely evaluating from scratch, here’s the landscape at a glance:

Factor	MWAA	Astronomer Astro	Dagster Cloud	Prefect Cloud	Step Functions	Self-Hosted + K8s
Minimum monthly cost	~$470	~$260 (scale-to-zero)	$10	$0 (Hobby)	$0 (free tier)	~$80 (scheduler only)
Heavy workload cost	$1,800–2,300	$1,200–2,000	Variable (credits)	$500+ plus compute	<$50	$400–600 plus eng. time
Setup complexity	Medium	Low	Low	Low	Low	High
Ongoing ops burden	Low	Low	Low	Low–Medium	None	High
Airflow compatibility	Full	Full	None (different paradigm)	None (different paradigm)	N/A	Full
KubernetesExecutor	No (Celery only)	Yes	N/A	N/A	N/A	Yes
Observability	CloudWatch only	Built-in Observe	Native asset tracking	Built-in dashboard	CloudWatch/X-Ray	Whatever you build
Multi-cloud	No	Yes	Yes	Yes	No	Yes
Ecosystem breadth	Largest (Airflow)	Largest (Airflow)	Growing	Moderate	AWS-native (220+ services)	Largest (Airflow)

Here’s how I’d frame the decision:

Choose MWAA if you’re on AWS, your team knows Airflow, you want AWS to handle the scheduler and metadata database, and your workloads are substantial enough to justify the cost floor. It’s a solid, mature service that has improved dramatically in 2024–2025.

Choose Astronomer Astro if developer experience, KubernetesExecutor support, and fast Airflow upgrades matter more than AWS-native billing integration. It’s MWAA but better in almost every operational dimension — at a comparable or lower cost for active workloads.

Choose Dagster Cloud if you’re starting fresh, prioritise observability and data quality, and your team is comfortable learning a new paradigm. The asset-centric model is genuinely superior for understanding the health of your data platform.

Choose Prefect Cloud if your team is Python-first, dislikes Airflow’s operator model, and values dynamic workflows with minimal boilerplate. The free tier makes experimentation trivial.

Choose the thin orchestrator pattern (self-hosted Airflow + Kubernetes/ECS compute) if you already run Kubernetes, need per-task isolation, want maximum cost control, and have the engineering capacity to maintain the Airflow installation.

Choose Step Functions if your workflows are linear, event-driven, AWS-native, and don’t require backfill or complex scheduling. For many data engineering use cases, Step Functions plus EventBridge is simpler, cheaper, and less to maintain than any Airflow deployment.

The 2024–2025 Updates That Changed the Equation

MWAA’s pace of development accelerated dramatically in this period, and several updates addressed the community’s loudest complaints.

Environment class expansion in April 2024 (mw1.xlarge and mw1.2xlarge) and November 2024 (mw1.micro) gave teams both the headroom for heavy workloads and the affordable entry point for dev/test that had been missing since launch. The micro class at ~$37/month is particularly valuable — before its introduction, every dev environment cost the same $365+ as production.

Airflow versions progressed from 2.8 through 2.9 and 2.10, with Airflow 3.0 landing on MWAA in October 2025. The 3.0 release brought a redesigned React UI, event-driven scheduling via Assets (Airflow’s answer to Dagster’s Software-Defined Assets), a Task SDK with least-privilege execution, and Python 3.12 support. The Asset concept is significant — it means Airflow is converging toward the same asset-centric thinking that Dagster pioneered.

MWAA Serverless (November 2025) was the most consequential launch — true pay-per-execution pricing that directly addresses the always-on cost complaint. But as mentioned earlier, the limitations (no Airflow UI, YAML-only definitions, no PythonOperator) make it a different product for a different use case, not a cheaper version of MWAA.

Graceful updates (May 2025) eliminated the other major pain point: environments can now be updated without interrupting running tasks. The SigV4 REST API (October 2024) simplified programmatic access by supporting standard AWS credentials instead of login token management.

Community sentiment sits at mixed-positive. One engineering team’s May 2025 evaluation captured the tension well: Airflow remains the industry standard with a proven track record, but the improvements in Airflow 3 are following features that Dagster and Prefect already have. Apache Airflow pulled 320 million downloads in 2024, dwarfing competitors — but Dagster’s growth trajectory and commit activity signal a genuine shift in the market.

Closing Thoughts

The orchestration landscape is converging. Airflow 3.0 adopted asset-aware scheduling — a concept Dagster pioneered. Prefect’s event-driven model influenced Airflow’s trigger and asset-watcher features. Dagster is building broader operator support. They’re all moving toward the same destination from different starting points.

That convergence means the “right” choice increasingly depends not on feature parity but on team expertise, existing investment, and cloud strategy. And honestly? The worst decision you can make is choosing MWAA by default because you’re on AWS. Run the numbers. Evaluate the developer experience gaps. Try spinning up a Dagster Cloud or Prefect Cloud instance alongside your MWAA evaluation — both have free tiers, and the comparison will be illuminating.

I think about that first debugging session on the inherited Step Functions setup often. Hours to find something that should have taken minutes. The platform we eventually chose didn’t change the underlying compute or the logic of the pipelines — it changed whether we could see what was happening. Pick the orchestrator that gives your team that visibility. The one they’ll trust when something breaks at an inconvenient hour. That’s the one that wins.

Data Engineers: Just Do Code Reviews

Chris Hillman — Sat, 11 Jul 2026 09:00:00 +1000

In Code Complete, Steve McConnell writes:

Software testing alone has limited effectiveness — the average defect detection rate is only 25 percent for unit testing, 35 percent for function testing, and 45 percent for integration testing. In contrast, the average effectiveness of design and code inspections are 55 and 60 percent.

That was published in 1993. The evidence has only accumulated since. Peer code review is the single most effective defect-removal technique available to software teams. Nothing else — not tests, not CI pipelines, not observability tooling — comes close on a per-hour basis.

Software engineers have known this for fifty years. Data engineers mostly pretend it doesn’t apply to them.

It does. And the stakes, if anything, are higher.

Why the excuses don’t survive contact with the data

The objections are familiar. We’re not real software engineers. We’re too small a team. We need to move fast. Our tools don’t support it. None of them hold up.

The evidence on review effectiveness is overwhelming. Michael Fagan’s original 1976 IBM study found formal inspections caught 82% of errors before unit testing even began. NASA’s Software Engineering Laboratory found code reading detected roughly twice as many defects per hour as testing. Russell’s 1991 AT&T study found that each hour spent in review avoided 33 hours of downstream maintenance — making inspections up to 20× more cost-effective than testing per defect found.

Modern numbers are just as clear. The SmartBear/Cisco study of 2,500 reviews across 3.2 million lines of code established that reviews conducted within sensible limits — under 400 lines per session, under 500 lines per hour, sessions capped at 60 minutes — yield 70–90% defect removal. Google’s case study of 9 million reviewed changes across 25,000 engineers proved the practice scales to 20,000 commits a day, with median review latency under four hours. The DORA 2023 State of DevOps report, surveying nearly 3,000 practitioners, found that teams with faster code reviews had 50% better software delivery performance. Elite DORA performers ship with a 7× lower change failure rate than low performers.

SmartBear’s State of Code Review asked 800 respondents what most improves quality. Code review ranked first, with unit testing a distant second. Among teams satisfied with their software quality, 80% used tool-based peer review and 82% had documented review guidelines.

This is not a close debate in software engineering. It shouldn’t be one in data engineering either.

Data pipelines fail differently — and that makes reviews more important, not less

Here’s the thing that makes the data engineering situation more precarious: application bugs are loud. They throw exceptions, crash processes, page on-call at 2am. The feedback loop that keeps application engineers honest — run it, watch it break — works quickly.

Data bugs are quiet. The pipeline is green. Row counts look reasonable. The dashboard renders. And the number is wrong.

Benn Stancil describes it precisely: silent data errors “don’t look suspicious and they trigger no warnings, in observability tools or in our manual spot checks. Instead, they often linger undetected, slowly and silently pushing the analytical assets that use them further and further from reality — until, during a board meeting…”

Barr Moses of Monte Carlo has noted that time-to-detection for silent data errors can be measured in months, not minutes. Her 2023 survey of 200 data leaders found the average organisation deals with 67 data incidents per month, with average resolution time up 166% year over year to 15 hours per incident, and 68% of teams taking four or more hours just to detect a problem. In the 2024 follow-up, two-thirds of data leaders had experienced a data incident costing over $100,000 in the previous six months.

The “run it and see if it breaks” safety net that keeps application code marginally honest doesn’t exist for pipelines. The only reliable substitute is another human reading the code before it runs.

The public case studies have already priced in what happens when you skip it

These are not theoretical risks.

Unity Software told investors in Q1 2022 that ingesting bad training data from a large customer cost it roughly $110 million in 2022 revenue, erasing approximately $5 billion in market capitalisation in a day. A data pipeline integrity problem, undetected until it had been compounding for quarters.

Citigroup paid a $400 million OCC fine in 2020 for long-standing failure to establish effective risk management and data governance programs — followed by another $135.6 million in 2024 for insufficient progress on remediation. Data quality and governance failures, sustained over years.

Equifax sent incorrect credit scores to lenders for three weeks in 2022, shifting scores by 25 points or more for around 300,000 consumers. The downstream decisions those scores informed — loan approvals, interest rates, credit limits — cannot be unwound.

Public Health England lost roughly 16,000 positive COVID-19 test results in October 2020 because an .xls pipeline hit Excel’s row limit. Those contacts were not traced.

Every one of these is, at root, a data-pipeline change that no sufficiently rigorous second set of eyes reviewed before it reached production. The review wouldn’t have guaranteed they caught it. The absence of a review guaranteed they had no chance to.

The broader cost is already priced in — you just don’t see the bill

Gartner’s running estimate is that poor data quality costs the average organisation $12.9 million per year. Thomas Redman, writing in Harvard Business Review, put the drag on the US economy at $3.1 trillion annually. His most uncomfortable finding, from measuring 75 executives’ own data: only 3% of company data meets basic quality standards, and 47% of newly-created records contain at least one critical error.

The practitioner surveys agree. Great Expectations’ 2022 study found 77% of data practitioners report quality issues, with 91% saying those issues hurt company performance. dbt Labs’ State of Analytics Engineering 2024 reported 57% of data professionals cite poor data quality as their top challenge — up from 41% in 2022. The problem is getting worse. Monte Carlo’s telemetry puts it at roughly one incident per fifteen tables per year in production. Forrester’s 2023 Data Culture and Literacy Survey found more than a quarter of respondents lose over $5 million a year to poor data quality, with 7% losing more than $25 million.

The worst number: Monte Carlo’s 2022 data suggests data engineers spend roughly 40% of their working week — two full days — firefighting quality issues. Every hour of that is an hour a peer review before merge might have prevented.

What the data community has actually said about this

Michael Kaminsky at Locally Optimistic is clear: “Code review for analytics is often substantively different from code review for software engineering because reviewers need to check the business logic and the analytical methods as well as the code.” Different, meaning harder and more consequential — not optional.

Tristan Handy’s argument that mart models should be treated as stable interfaces, with versions and deprecation windows like APIs, is now the foundation of dbt’s contracts feature. It is also the right frame for what a reviewer is protecting: not just the code, but the downstream trust placed in it.

Emilie Schario’s observation remains accurate: “Data is behind software development when it comes to learning and implementing the best practices of DevOps.” The review is how you close the gap.

And Maxime Beauchemin, who built Airflow and Superset, has documented the cultural cost of data engineering’s second-class status. When the team is small, when velocity is the priority, when “we’re not really software engineers” is the ambient assumption, reviews are the first thing to go. And the work becomes fragile, tribal, and unreviewable — not because of the tools, but because of the habits.

Datafold’s write-up of analytics-engineer review culture names the failure mode precisely: “Faced with the prospect of move fast and break stuff vs. move slow, carefully review, and maybe break less stuff, most reviewers will default to the first option, and ‘LGTM’ their way through the day.”

That is not a review culture. That is a rubber-stamp culture with extra steps.

What a data engineering review should actually cover

The community consensus — distilled from dbt Labs’ PR guidance, the GitLab Data Team Handbook, Locally Optimistic, Datafold, and the dbt Discourse — converges on a specific checklist. One focused on the failure modes unique to data, not imported wholesale from software engineering.

Area	What the reviewer checks
SQL logic and grain	No silent fan-outs from joins; stated grain matches actual grain; CTEs are readable; row counts make sense against production
dbt design	`ref()` used throughout; sources declared; correct materialisation type; incremental models have `unique_key` and a sane predicate
Schema and breaking changes	Contracts enforced for public models; version bumps with a deprecation window; data-diff output reviewed
Tests	New models carry `unique` and `not_null` at the grain; grain-changing transforms have data tests; business rules encoded as tests, not comments
Docs and metadata	Model and column descriptions present; source freshness configured; PII flags set
Idempotency and performance	Re-runs produce identical results; no pinned `CURRENT_TIMESTAMP`; partition and cluster keys considered; warehouse cost estimated

This is not a burdensome list. It’s the checklist a thorough engineer would run through mentally anyway — just written down and shared, so someone else can run through it too.

Just do it

The case for code review in software engineering was settled by Fagan, IBM, NASA, Cisco, Microsoft, Google, and DORA — across decades and millions of commits. Reading code finds roughly 70% of defects before they ship. Teams that review fast deliver 50% better. Silent errors cost enterprises tens of millions per year and mid-market teams their credibility with the business.

The case for code review in data engineering is not weaker than in software engineering. It’s stronger, because pipelines fail silently, errors compound downstream before anyone notices, and the financial consequences show up on earnings calls rather than error logs.

The practitioners who have thought hardest about this — Kaminsky, Stancil, Moses, Sanderson, Beauchemin, Handy — all land in the same place. Every SQL change, every dbt model, every DAG, every schema migration deserves a second set of eyes before it touches production. Not because data engineers are software engineers, but because the cost of pretending the practice doesn’t apply has become impossible to afford.

The tools exist. The checklists exist. The statistics exist. The only thing missing is the decision to stop skipping it.

You Can't Incentivise a Pipeline That Doesn't Break

Chris Hillman — Sat, 04 Jul 2026 08:00:00 +1000

I worked alongside a data engineer who was, by every formal measure, the best performer on the team. He was also quietly destroying the platform.

Not maliciously. He was optimising for the thing being measured. His work shipped fast because he skipped the edge case analysis. He closed tickets at first resolution without ever checking whether the underlying pattern would recur. He didn’t review anyone else’s PRs (not his KPIs, so why would he?).

One of his teammates was the person actually holding the platform together. Quieter, slower to release, never at the top of any metric. She questioned everything, dug into the data, and worked with stakeholders to write elaborate testing scenarios. She was the one who caught a new source system sending nulls where the schema expected integers, and had been doing so for three weeks. She paired with the juniors. She wrote the runbook nobody had written.

Her performance review that year was average. His was excellent.

If you’ve worked in data engineering for any length of time, you’ve seen a version of this. If you’ve moved into leadership, you may have been the one signing off on it.

That’s the incentive pay problem. Not that it doesn’t motivate people. It motivates them to do the wrong things.

The Measurement Trap

There’s a name for this: Goodhart’s Law. When a measure becomes a target, it ceases to be a good measure. The moment you attach money to a metric, you’ve changed what people optimise for, and nearly every metric you can pull from the tooling is the wrong target.

Pipelines deployed rewards volume and punishes care. An engineer who ships fifteen small, well-tested, well-documented pipelines is worth more than one who ships thirty fragile ones. No spreadsheet you build will ever agree.

Tickets closed incentivises decomposition. Want a higher count? Chop big problems into small ones and resolve at the surface. The underlying pattern that keeps generating tickets isn’t your problem. That’s a different ticket, a different quarter, someone else’s metric.

Incident count punishes the team for things outside their control. An upstream system starts sending malformed payloads at 2am and your bonus goes down. (The vendor who caused it hits their SLA and gets paid in full, naturally.)

I’ve watched other leads volunteer for these measures because they’re easy to extract from Jira and GitHub. Easy to extract is not the same as meaningful. In my experience it’s usually the opposite.

The Invisible Work Problem

The deeper issue is that the most valuable work a data engineer does often leaves no trace.

The schema that doesn’t need refactoring in six months because someone thought carefully about how it would evolve. The data contract that caught a breaking change. Good engineering actively suppresses the numbers that look like productivity, because fewer incidents means fewer tickets, and fewer tickets means a lower “work completed” count.

Then there’s the social work. The engineer who reviews PRs with care, who explains why and not just what, who flags when a colleague’s approach will cause trouble downstream, is compressing the learning curve for the whole team. They prevent incidents that never appear in any log, because they never happened.

I wrote about this from a different angle in The Invisible PR You’re Building Right Now — the reputation built through small interactions that compound over years. Incentive pay is the structural version of the same idea. Most schemes don’t just miss this work, they penalise it, because a careful PR review is time not spent on your own ticket count.

The Lopsided Ledger

Even a perfect set of metrics — which you can’t build, but suppose — runs into some uncomfortable arithmetic. A strong review doesn’t make a good engineer better; at most it confirms what they already believed. A review that misses what someone actually did makes them recalibrate, and next quarter they do the work that gets measured instead of the work that matters. The downside dwarfs the upside. And since most people believe they’re above average and most systems grade on a curve, disappointment is the default output. You’re spending management time and team morale on a mechanism that runs at a loss.

So What Actually Works

I’m not arguing that data engineers don’t care about money. They do, and they should.

Dan Pink covered this territory in Drive, and his central finding is facinating: if-then rewards work well for mechanical tasks and reliably degrade performance on cognitive ones. Data engineering is nothing but cognitive work. His answer, and mine after years of watching teams, is to pay people well enough that money stops being the conversation — then use the levers that actually move things.

Autonomy. Engineers who are told what outcome is needed, and trusted to work out how, consistently produce better work than those handed a specification. The person who understands the problem deeply enough to design the solution will always see things the specification missed. Give people the problem, not the method.

Mastery. The most engaged engineers on my teams are the ones getting meaningfully better at something. Not completing training modules — deepening their craft. Conference time, learning budgets that actually get spent, a problem slightly beyond their current reach with a safety net underneath.

Purpose. Data engineers rarely see the downstream consequence of what they build. Closing that loop matters enormously. When an engineer learns that the student outcome data they built powers the retention intervention that changed whether someone finished their degree, no quarterly bonus comes close.

And recognition, the specific and timely kind. Not plaques, not a Teams message with too many emojis. The manager who says quietly in a 1:1: “The way you caught that schema drift before it hit the mart prevented an incident nobody will ever know about, and I noticed.” That costs nothing, and it does more for morale than any incentive scheme I’ve witnessed.

The Pipeline That Doesn’t Break

The best data engineering work is characterised by what doesn’t happen.

The data that arrives without error. The migration that completes cleanly. The breaking change caught in review. The junior who is now autonomous on a system they couldn’t have touched six months ago.

None of this is countable. None of it generates a closed ticket or increments a deployment counter. Which means it will always be invisible to any incentive scheme complex enough to automate and simple enough to implement.

There is an honest cost on the other side. Drop the metrics and you lose the comfort of a ranking you can defend in a calibration meeting. Judging invisible work means knowing your team’s work well enough to see it, and that’s harder than reading a dashboard.

Pay people fairly. Pay them enough that salary isn’t what they’re thinking about while debugging a nightly job at 7am. Then build the conditions where the work itself is worth doing.

The engineers who hold your platform together aren’t doing it for the bonus. And the moment you start paying them as if they were, you’ll start losing the ones who weren’t.

Your Team Already Has Patterns. They Just Don't Know It.

Chris Hillman — Sat, 27 Jun 2026 09:00:00 +1000

When I started a new role, one of the first things I did was try to understand how data moved through the system. Not the dashboards, not the data models — the pipes. Where did things come from? How did they get in? What happened to them along the way?

There were somewhere between twenty and thirty source systems feeding the platform. Not a massive number, but enough to tell a story when you looked at the ingestion layer all at once. What I found was that all the pipelines had originated from two base templates. A sensible starting point. The kind of thing a small team puts in place early to stop complete chaos.

But over time, as new sources were onboarded by different engineers under different pressures, the implementations had drifted from those templates in ways that were hard to track. Some pipelines were applying data transformations before data even landed in Snowflake — so the raw layer wasn’t actually raw. Some were enforcing data contracts at the landing stage, before the data had been typed, which created problems downstream when contract violations turned up in places the pipeline wasn’t designed to handle them. And there was no consistent approach to manifest file checking — verifying that the file you received matched what the source said it sent — which is exactly the kind of gap that sits quietly until a source delivers a partial file on a bad day and nobody notices until a stakeholder does.

None of this was catastrophic. I want to be clear about that. I’ve seen far worse — environments where the ingestion layer is a genuine free-for-all with no common ancestry at all. By comparison, this was a B+. The bones were solid. There was a lineage to the templates you could trace, and the engineers who’d built on them had generally been trying to do the right thing. But there was drift, and where there’s drift there are gaps, and gaps have a way of becoming expensive at the worst possible moments.

The flip side was interesting. The transformation layer and the serving patterns were genuinely robust. Whoever had shaped those had been thinking in terms of reusable approaches from the beginning. SCD2 loads were SCD2 loads. Reporting tables followed a consistent structure. That part of the platform had a grammar to it, even if nobody had written it down.

The contrast between the two layers was instructive. And it surfaced a question I keep coming back to: what would it look like if the whole team had a shared vocabulary for the work they were doing — not just in transformation, but across the whole pipeline lifecycle? Not a rigid rulebook. Not a standards document that nobody reads. Just a common language, agreed on together, that made the implicit explicit.

That’s what a pattern bank is. And this article is about how to build one.

The problem that nobody names

Before we get to the solution, it’s worth naming what actually breaks when there’s no shared pattern vocabulary on a data engineering team.

The most obvious symptom is inconsistency. When every engineer approaches recurring problems from first principles, you end up with a codebase that looks like it was built by ten different people — because it was. Code reviews become debates about form rather than substance. A new team member joins and spends their first month just trying to understand why the SFTP ingestion from System A looks nothing like the SFTP ingestion from System B, even though they’re structurally the same thing. The answer — “that’s just how it grew” — isn’t satisfying, and it doesn’t help them build.

But inconsistency isn’t the thing that hurts most. The thing that hurts most is estimation.

Oe of the early challenges was quoting projects to internal customers — duration, cost, team size. The kind of thing stakeholders reasonably want to know before they commit. And without a clear picture of which parts of a solution were known quantities versus genuinely novel territory, those conversations were uncomfortable. You could make an educated guess, but you couldn’t anchor it to anything concrete. Each project felt like a bespoke exercise, even when large parts of it were things the team had done before.

The problem has a name in project planning circles: unknown unknowns. The things you don’t know you don’t know. The hidden surprises that don’t appear in any upfront estimate because you haven’t recognised them yet. Pattern reuse directly attacks this. When you’ve built the same ingestion type before, you know roughly how long it takes, where the edge cases live, and what tends to go wrong. You’ve converted some of the unknown unknowns into known quantities — and that changes everything about how you plan.

There was one more piece that crystallised why the pattern bank mattered. The technical architects were responsible for designing how source systems should serve data to the platform. They sat upstream of the data team, making decisions about formats, frequency, delivery mechanisms. But they didn’t have a clear picture of what happened once data arrived — what the ingestion layer expected, what would make the data engineer’s life easy versus hard, what patterns the team was already working with.

The pattern bank, as it developed, became a communication artefact as much as an engineering one. It made the platform legible to people who sat outside it. Here is how we ingest data. Here is what we need from you when you’re designing a new source feed. Here is why the way you’re proposing to serve this data is going to create problems downstream.

That transparency was worth as much as the internal consistency.

What a pattern bank actually is

The simplest definition: a pattern bank is a catalogue of reusable solution approaches that your team draws from when designing new work.

It’s not a code library — though patterns might eventually generate template code. It’s not a runbook — those are step-by-step guides for specific incidents. It’s not a playbook — those describe how to handle particular scenarios. A pattern bank operates at a higher level of abstraction than all of those. It describes the shape of a solution, not the substance of a specific implementation.

Think of it like a recipe catalogue. When you’re planning a dinner party, you choose recipes based on what you’re cooking for and what ingredients you have. You don’t invent a new cooking method each time — you draw from a repertoire. The pattern bank is the repertoire. Each new project is the dinner.

For a data engineering team, the pattern bank tends to organise naturally into three layers that follow how data moves through a platform:

Ingestion patterns cover how data gets from a source into your environment. This might be a direct file transfer from SFTP, an API pull, an event stream, a database replication — the mechanics differ, but each is a recognisable type. At the ingestion layer, the goal is usually to get data as close to your transformation environment as possible in its raw form, before worrying about types or structure. Staging — the step where you impose data types and light transformations — often lives here too, and it’s worth treating it as a distinct sub-pattern.

Transformation patterns cover what happens to data once it’s in. SCD1 for current-state overwrite, SCD2 for full history with row versioning, transactional loads for append-only fact tables, reference table management, bi-temporal designs for systems that need to track both valid time and transaction time. These patterns are where most of the design complexity lives, and they’re often the most stable part of a mature pattern bank because the underlying approaches have been well-understood for decades.

Delivery patterns cover how processed data gets to consumers. A flat file export to a third-party system, a reporting aggregate table consumed by a BI tool, a real-time serving layer for an operational dashboard — each is a pattern with its own shape, its own testing considerations, its own failure modes.

The power of this structure is in what it enables at design time. When a new project lands, you’re no longer starting with a blank page. You’re asking: which ingestion pattern fits this source? Which transformation pattern fits this business requirement? Which delivery pattern fits this consumer? You pick from the catalogue, combine them into a solution design, and — critically — you know what you’re working with because you’ve worked with it before.

One recipe, one pattern from each stage. The amber path is a single solution design; the grey fan into the shared sub-pattern is the factoring at work.

One instance of an approach is a one-off solution. Two instances is a coincidence. Three instances is a pattern worth documenting. That’s the threshold I’d suggest for adding something to the bank — build it twice, recognise it on the third, name it and write it down.

What it should not be

The counter-example that keeps me honest about what a pattern bank should not be.

I’ve seen pattern banks that went too far. Not too far in terms of coverage — too far in terms of specificity. Every combination of source type and transformation type and delivery type had its own entry. Every edge case was enumerated. Every variant had its own named pattern. The catalogue was exhaustive in the worst sense of the word.

The result was a pattern bank that engineers couldn’t use without first spending an hour navigating it. The patterns were so granular that applying one felt like following a script rather than making a design decision. Worse — they were so specific that anything slightly different from the documented pattern triggered a governance conversation about whether to create a new entry or adapt an existing one. The bank had become bureaucracy wearing an engineering costume.

A pattern bank should describe the shape of a solution clearly enough that a mid-level engineer can use it to make a design decision without asking five follow-up questions. If your patterns require a full implementation guide to be useful, they’re not patterns — they’re procedures. Keep them abstract enough to cover genuine variants. Keep them concrete enough to actually guide work. The test is whether someone can look at a pattern and understand what kind of problem it solves within about thirty seconds.

The three-layer structure helps with this. When you keep ingestion patterns, transformation patterns, and delivery patterns separate, each pattern is already scoped to a defined part of the problem. That’s enough structure. Within each pattern, describe what problem it solves, when to use it, when not to use it, and roughly what it involves. That’s the minimum viable pattern documentation. A short note on known effort range doesn’t hurt either.

Resist the urge to enumerate every combination. The recipe catalogue analogy holds: a good recipe book teaches you to cook, not to follow instructions. The patterns should develop your team’s design intuition, not replace it.

There’s also a structural cure for the combinatorial urge, and it took me longer than it should have to see it: factor the convergence points out. When several patterns funnel into the same mechanism — say, five different ingestion types that all end in the same landing-and-staging steps — that shared tail isn’t a reason to write five longer entries, and it’s certainly not a reason to write twenty-five combined ones. It’s a sub-pattern. Give it one entry of its own and have the parent patterns reference it.

On the platform I described at the start, every ingestion pattern — file drops, API pulls, database replication, direct pushes — converged on a single landing mechanism into the warehouse. Once we pulled that out as its own sub-pattern, each parent entry lost about a third of its length and gained a crisp ending: “hands over to the landing sub-pattern.” And when the landing mechanism later needed a change, it was one edit instead of five. Factoring is how you keep the bank small while the platform grows.

You cannot build this alone

Most writing about pattern banks covers what they should contain and then leaves you to figure out how to get your team to actually care about it.

The truth is that a pattern bank built by one person — even a thoughtful, experienced person — will struggle to gain traction. Not because the patterns are wrong, but because nobody else owns them. When something doesn’t quite fit a pattern from the bank, the engineer’s instinct is to work around it rather than raise the question, because the bank feels like someone else’s thing. The patterns were handed down; they weren’t grown.

The irony is that the teams who need a pattern bank most urgently — teams with inconsistent approaches, estimation problems, high bus-factor risk — are also the teams most likely to resist a top-down version of one. Engineers who’ve been doing the work their way for two years don’t want to be told their approach is being standardised out of existence.

So you have to build it differently.

The starting point is an audit, not a design. Before you introduce a pattern bank as a concept, spend time with your team’s existing work. Look at what’s been built over the last six to twelve months. Look for the shapes in it. Which approaches keep appearing? Where did engineers independently converge on similar solutions? Where did they diverge on problems that are structurally the same?

This is the part where you’re listening, not speaking. You’re mapping the territory.

When you bring the team together for the first conversation about patterns, the framing matters enormously. The worst version of that conversation starts with: “I’ve been looking at how we do things, and I think we should standardise on some approaches.” Even if that’s true and well-intentioned, it positions you as the author and everyone else as the audience.

The better version starts with a question: “What do we keep building over and over?” Give people a chance to name the recurring shapes in their own work. You’ll find that engineers who’ve been on the team for a while can articulate these patterns clearly — they just haven’t had a reason to before. When they name a pattern, write it down. Give it back to them as documentation. That’s their knowledge, made visible.

When you name what already exists rather than imposing what should exist, the dynamic shifts. You’re not standardising their work. You’re recognising it.

This is the buy-in mechanism. The process of building the pattern bank together is the adoption mechanism. When an engineer contributed to the definition of a pattern, they’ll defend it, refine it, and reach for it naturally in design conversations. When they received it from above, they’ll comply with it when they remember to — and quietly work around it when something doesn’t quite fit.

Making the first session work

The first time you sit down as a team to build the pattern bank, keep it light. You’re not trying to produce a finished document. You’re trying to start a conversation that will take months to reach maturity.

A good opening question: “If you were explaining to a new team member what kinds of pipelines we build, how would you describe them?” This is deliberately informal. You’re not asking for a taxonomy — you’re asking for the words people already use.

Let the conversation happen. Take notes. You’ll start to hear natural groupings emerge. Someone will say “well, most of our ingestion is basically file drops or API calls.” Someone else will add “plus the two database replications, but those work the same way as the file drops once they’re in landing.” That’s a pattern taking shape.

When objections come up — “but the System A feed is completely different because…” — treat them as contributions, not resistance. Either the system A feed is genuinely a different pattern (name it separately), or the difference is a variant within the existing pattern (document it as a variant). Both outcomes are useful. The engineer who raised the objection has just enriched the catalogue.

There’s an important interpersonal dynamic to watch in these early sessions. Engineers who’ve been on the team a long time have often built strong opinions about how certain things should be done — and those opinions are usually right, because they’re grounded in hard experience. If you’re newer to the team, or coming in with a perspective shaped by a different environment, the temptation is to steer the conversation toward what you think the patterns should look like. Resist that. The first job is to surface what exists, not to redesign it.

Even if you can see clearly that a particular approach has flaws — that two of the team’s existing ingestion patterns are more similar than they appear and probably belong under one umbrella — raise it as a question, not a conclusion. “I’m noticing these two look structurally similar — do you see them as the same pattern, or are there meaningful differences I’m missing?” That’s an invitation to examine, not an instruction to comply. And the answer might surprise you. The engineer who built both of them might have a very clear reason why they’re distinct that isn’t obvious from the outside.

The sceptics in the room deserve more than a paragraph, because for a long time I misread what was actually happening for them. The standard advice — don’t argue them into participation, ask them what they’d add or change, give them an editing role rather than an audience role — is right as far as it goes. But it treats scepticism as a mood to be managed, and it’s usually something more legitimate than that.

The engineers most resistant to a pattern bank are very often the ones whose expertise is the current way of working. The person who can untangle the gnarliest legacy pipeline from memory, who knows every undocumented quirk of every source feed, who gets pulled into every incident because nobody else holds the map — that person has real, earned status, and it rests on knowledge that lives in their head. A pattern bank threatens to convert that hard-won knowledge into a page anyone can read. That’s a genuine loss, and you cannot argue someone out of feeling it.

What works is role transfer, not persuasion. The new way of working needs heroes too: someone has to lead the audit of existing work, someone has to own the pattern standards, someone has to write the entries for the genuinely tricky patterns that nobody else fully understands. The people who hold the deepest knowledge of the current state should get first claim on those roles. You’re not asking them to surrender their expertise — you’re asking them to encode it, with their name on it.

And there’s one question that does more work than any amount of advocacy: “what would you need to see before you’d trust this?” Ask it sincerely and write the answers down, because the answers are the acceptance criteria for the pattern bank — co-authored by the people most likely to find its weaknesses. The sceptic who told you exactly what would convince them has just committed, in public, to being convincible. The person most likely to undermine a pattern bank built without them is the same person who becomes its most vocal defender when they’ve shaped it.

At the end of the session, you should have a rough list of candidate patterns and a few volunteers to draft the first write-ups. Keep the documentation minimal — one page per pattern is plenty to start. The goal is a living document, not a finished artefact.

The question of where to keep the pattern bank is worth getting right early. It should live somewhere the team already goes — a Teams wiki, a shared OneNote, a Confluence space, a structured markdown repo. Wherever your existing documentation lives is the right place. Don’t create a new system that requires a new habit. The pattern bank should be the path of least resistance in a design conversation, not a side trip.

Don’t wait until the bank is “complete” to start using it. Start referencing patterns in design conversations almost immediately. When a new project comes in, run the question out loud: “Which ingestion pattern are we looking at here? Which transformation pattern?” Even before the bank is formally documented, the vocabulary starts to take hold. The naming matters as much as the documentation — maybe more. Once a team has words for what it’s doing, it starts to think in those words.

What to put in a pattern entry

The minimum useful documentation for a pattern:

Name. Short, memorable, specific to your team’s context. “SFTP file ingestion” is fine. “Direct file transfer pattern” is fine. Resist the urge to make it sound architectural.

What problem it solves. One sentence. What kind of situation does this pattern address?

When to use it. What conditions make this the right choice?

When not to use it. This is the part most pattern documentation skips, and it’s often the most valuable. What situations look like they’d fit this pattern but actually don’t?

The basic shape. Three to six steps describing the high-level approach. Not a full implementation guide. Think “what are the stages?” not “what does the code look like?”

Known variants. What changes in different situations? An SFTP pull from an external system and an S3-to-S3 transfer might share a parent pattern but have meaningfully different considerations. Document the variants without creating a separate entry for each.

What it hands over, and what it needs to start. Patterns don’t run in isolation — each one ends where another begins, and the boundary is where work stalls. Two short lists fix most of that: the things this pattern produces that the next one depends on (the handover), and the things that must already exist before work on this pattern can sensibly start (the pickup criteria). If a transformation pattern needs the source data landed, typed, and described before it can begin, say so in the entry — because that single line is the difference between an engineer starting tomorrow and an engineer discovering, two days in, that they’re blocked.

How it’s validated. What kind of testing closes this pattern out, and with what kind of data? A useful rule of thumb that took us a while to articulate: fabricated rows belong in unit tests and nowhere else; real production history is the default for business validation; and if you genuinely need a synthetic scenario, create it through the source system’s own test environment so it arrives via the real pipeline — never hand-inserted into the warehouse. A pattern entry doesn’t need the whole standard. It just needs to say which of those applies, so validation is part of the design conversation rather than an afterthought at the end.

Effort signal. Is this a well-understood pattern with a reasonable effort history? Or does it have significant unknown elements? Even a simple “known / known with caveats / novel” classification is enough. It gives you something to point to when you’re estimating.

To make this concrete, here’s roughly what a pattern entry might look like for a direct file ingestion pattern — pared back, readable, just enough to guide a design decision:

Pattern: Direct File Ingestion

What it solves: Landing structured or semi-structured files from an external source into the data platform’s landing zone.

Use when: Source delivers files on a schedule (push or pull), and transformation happens separately once the file is confirmed complete.

Don’t use when: Source delivers events in real-time that need immediate processing — that’s a streaming pattern. Also not appropriate when source volume is large enough that full file delivery creates latency problems.

Basic shape:

File arrives in agreed drop location (SFTP, S3, or internal file transfer)
Arrival trigger fires (event-driven or scheduled poll)
File is validated for completeness (presence check, row count if available)
File is copied to landing zone in raw form — no type conversion at this stage
Staging job applies data types, basic cleansing, and audit columns
Downstream transformation pattern picks up from staging

Known variants: SFTP pull (we poll the source), S3 push (source lands in our bucket), internal transfer from another system on the platform. Error handling and retry logic differ slightly for each, but the core shape is the same.

Hands over: A landed, typed staging table; a manifest reconciliation result; a note of any rejected or quarantined rows. Needs to start: Agreed drop location and credentials; a sample file; a named contact at the source who can answer format questions.

Validated by: Completeness check against the manifest; row counts reconciled to source; staging types asserted against the agreed contract.

Effort signal: Well-known. Delivery estimate should be grounded in actuals from previous builds. Flag if source has no delivery receipt mechanism — that adds complexity and uncertainty.

That’s one page. Maybe two if the variant notes are detailed. It’s not comprehensive documentation — it’s a design guide. The engineer using it brings the implementation knowledge; the pattern provides the frame.

The test I mentioned earlier is worth applying to every entry: can a mid-level engineer use this to make a design decision within about thirty seconds of reading it? If the answer is no, trim it or elevate the abstraction level. If someone needs a wall of text to understand what kind of problem the pattern addresses, the problem hasn’t been stated clearly enough.

Patterns in use: from blank page to solution design

The recipe analogy only works if you can actually see someone cooking. So here’s what using a pattern bank looks like in practice, when a project lands on your desk.

Say your team is onboarding a new source system — a third-party vendor who delivers customer transaction data daily via SFTP. The commercial team wants it available in the reporting environment within forty-eight hours of the file landing. There are some business rules around transaction categorisation that need to be applied, and the output needs to land in an existing aggregate table that feeds a Power BI dashboard.

Without a pattern bank, this is a design conversation that starts with “so how do we want to approach this?” With a pattern bank, it’s a pattern-matching exercise.

Ingestion: The source is delivering files via SFTP on a schedule. That’s the direct file ingestion pattern. You’ve done it before; you know the shape. The main variant question is whether you’re pulling from their server or they’re pushing to yours — that changes the authentication setup but not the core pattern. You note that their SFTP has had reliability issues in the past (the commercial team mentioned it), so you flag the retry handling variant in your notes.

Transformation: The business rules around transaction categorisation sound like a reference table lookup — you’re mapping raw values from the source to agreed internal categories. That’s a known pattern. The historical record requirement is the key question: does the business need to see how a transaction was categorised at the time it was processed, even if the categorisation rules change later? If yes, that’s a bi-temporal concern and it changes the transformation pattern significantly. If no — if current categorisation is all that matters — it’s a much simpler SCD1 or SCD2 load depending on whether you need row history.

Delivery: The output goes into an existing aggregate table feeding a Power BI dashboard. Depending on how that table is structured, you’re either appending new rows, overwriting a date partition, or recalculating an aggregate. Each is a recognisable delivery pattern with known characteristics.

By the time that design conversation is twenty minutes in, you have a solution sketch: direct file ingestion (SFTP pull variant), SCD1 transformation with reference table lookup, aggregate table delivery. Three patterns, all known, all with effort history. The only real design question that needs time is the bi-temporal one — you need to clarify the business requirement before you can confirm the transformation pattern.

And before the estimate goes anywhere, you walk the boundaries: what does ingestion hand to transformation, what does transformation hand to delivery, and who is waiting on whom at each point. Which brings me to the part of this picture I got wrong for the longest time.

The seams between patterns

Everything I’ve described so far treats patterns as blocks: pick one per stage, snap them together, estimate from history. That’s true, and it’s also where the picture quietly lies to you — because in practice, the blocks are not where delivery goes wrong. The boundaries between them are.

I learned this when we took our pattern bank a step further and turned it into a delivery plan: each pattern decomposed into roughly day-sized pieces of work, each piece with an exit criterion. The decomposition surfaced something the catalogue view had hidden. The estimates inside patterns were fine — we had effort history, the day-sized pieces held up. The overruns lived between patterns: the ingestion work finished on Tuesday and the transformation work started the following Thursday, and nobody could quite say where the week went.

The week went into the seam. A review that sat in someone’s queue. A handover that turned out to be missing the one thing the next engineer needed, triggering a conversation, then a clarification, then a small rework. A wait for a batch window, or for another team, or for a sign-off from someone who didn’t know they were on the critical path.

This gives you a distinction worth building into how you plan: effort versus elapsed time. Effort lives inside patterns — it’s the day-sized, estimable work the bank already describes. Elapsed time accumulates at seams, and it follows completely different rules. You can’t reduce a review queue by working harder, and you can’t estimate a sign-off from effort history. If your plans only count effort, the seams are invisible right up until the deadline isn’t.

The same work, on a calendar. The solid segments are effort — the patterns. The gaps are the seams.

So once the patterns are named, name the seams too. Every place where work passes between patterns — or between people — gets the same lightweight treatment a pattern gets: what crosses the boundary (the handover pack), and what the receiving side needs before they can start (the pickup criteria). The handover from ingestion to transformation, for instance, might be: the staging table, the agreed contract it conforms to, a note of known quirks, and confirmation the data is actually flowing. Write it down once, and every future handover at that seam stops being a negotiation.

The quality bar for a handover is a test I now apply to everything: a piece of work is done when a different engineer could pick it up tomorrow with no conversation. Not “done pending a chat,” not “done but ask me about the weird bit.” No conversation. It sounds strict, and it is — but every conversation a handover requires is elapsed time hiding in plain sight, and elapsed time at seams is precisely the thing your effort-based estimates can’t see.

The pattern bank describes the blocks. The seam map describes the gaps between the blocks. You need both, and the second one is the one nobody writes down.

The estimation conversation

There’s a distinction in planning that doesn’t get talked about enough: the difference between things you don’t know, and things you don’t know you don’t know.

The first kind — known unknowns — you can account for in estimates. You know they’re there, you can add buffer, you can plan a spike to resolve them early.

The second kind — unknown unknowns — are the ones that blow up timelines. They surface mid-project, when you’re already committed to a delivery date. They’re the integration behaviour you didn’t anticipate, the edge case in the source system nobody mentioned, the schema change that came through without notice.

Pattern reuse doesn’t eliminate unknown unknowns. But it converts some of them into known unknowns, and some known unknowns into known quantities. When you’ve built the same ingestion type four times, you know where the surprises usually come from. You’ve already met most of the ways that type of pipeline can misbehave.

Each repetition of a pattern moves surprises to the right — out of the category that blows up timelines and into the category you can plan.

That conversion is where a pattern bank earns its keep. It’s not a tidiness exercise. It changes the quality of your estimates, and it changes how you talk about risk with internal customers.

At the organisation I mentioned earlier, once we had a clearer picture of which project components were known patterns versus genuinely novel territory, the project conversations changed character. Instead of quoting a number and hoping for the best, we could say: “This solution uses three patterns from our standard set — the effort estimate for those is grounded in what we’ve delivered before. This fourth component is new for us — we’re treating it as discovery work and we’ll re-estimate once we’ve built a proof of concept.” That’s a different kind of conversation. It’s a more honest one, and paradoxically, it inspires more confidence.

There’s a maturation point worth knowing about in advance. Once a pattern has genuine effort history — three or four builds behind it — you can decompose it into day-sized pieces of work, each with its own exit criterion. That’s the moment the bank stops being a catalogue and starts being a delivery playbook: a new project isn’t just “three known patterns,” it’s a sequence of named, day-sized stories you can lay against a calendar. And when you do lay it against a calendar, estimate the seams separately, in elapsed time rather than effort — because the question at a seam isn’t “how long will this take to do” but “how long will this take to happen.”

Stakeholders aren’t usually uncomfortable with uncertainty. They’re uncomfortable with surprises. Pattern banking reduces surprises by making the known quantities explicit — which in turn makes the uncertain parts easier to name, plan around, and manage.

From documentation to machinery

Remember the opening of this article: a team with two sensible base templates, and twenty implementations that had drifted from them in ways nobody could track. Here’s the uncomfortable implication for everything I’ve said so far — a documented pattern bank is still just documentation, and documentation drifts. The patterns describe what the team agreed to do; nothing stops the codebase from quietly doing something else, one reasonable-seeming exception at a time. The bank I’ve described prevents the team from forgetting the patterns. It doesn’t prevent them from departing from them.

So it’s worth knowing that a pattern bank has a maturity path, and documentation is the middle of it, not the end.

The first stage is vocabulary: the team has names for the work, spoken but not written. This is further than most teams ever get, and it’s where the adoption battle is won. The second stage is documentation: the one-page entries, the seam map, the effort signals. The third stage is machinery: the pattern’s invariants are enforced by your delivery tooling, so that violating them doesn’t produce a governance conversation — it produces a failed build.

Not everything in a pattern can or should be mechanised. But some pattern elements are invariants — things that must hold for the platform to stay trustworthy — and invariants are exactly the things that drift erodes first. A sequencing rule is a good example. We had one that said, in effect, data ships before code: the ingestion side of a new source goes to production first, and transformation work doesn’t start until the data it depends on actually exists where production builds can see it. As documentation, that rule held about as well as documented rules ever do. Then we wired it into CI — the transformation build simply fails if the upstream data isn’t there — and the rule stopped being a rule. It became a property of the system. The pattern cannot drift on that invariant, because the machinery won’t let it.

Encoding invariants has a sharp corollary that I didn’t anticipate: optional elements and mechanical gates don’t mix. We’d had data contracts in our patterns as “preferred but optional” — encouraged, written up, mostly adopted. The moment we wanted CI to validate new sources against their contract, “optional” stopped being coherent. A gate can’t check a thing that may or may not exist. So the gate forced the decision we’d been politely deferring: contracts became required for new sources, with existing ones grandfathered. That’s the general lesson — every time you automate the enforcement of a pattern element, you’ll be forced to decide whether the things it depends on are truly required. The machinery is honest in a way the documentation never has to be.

Don’t read this as “mechanise everything.” Pick the invariants whose violation is expensive — the raw layer staying raw, the sequencing between stages, the contract a source promised — and let the rest stay as guidance. The thirty-second test still applies to the documentation; the machinery is just there to hold the lines that matter most while humans exercise judgement on everything else.

A fourth job: scoping change

I’ve described three jobs for a pattern bank so far: a shared design vocabulary, grounded estimation, and legibility to the people around the team. There’s a fourth, and I only discovered it later, when the bank was already in place and we needed it for something it was never designed to do.

We were proposing a significant change to how the team delivered — the kind of change that touches process people have used for years, with passionate, technically senior advocates on multiple sides. The instinctive reaction to a proposal like that is that everything is changing: the way I build, the way I test, the way my work gets to production, the way I’m on the hook when it breaks. When change feels total, people defend totally.

The pattern bank changed the shape of that conversation entirely. Because the team’s whole way of working was laid out as a catalogue of named patterns, we could put the proposal against it and show precisely where the change landed: of the entire bank, exactly one pattern’s promotion steps were being rewritten, plus one new gate at a seam between two stages. Every ingestion pattern: untouched. Every delivery pattern: untouched. The transformation logic itself: untouched. We could literally point at the diagram and say — the workshop is deciding the contents of one box.

The most useful slide in the whole proposal wasn’t about the change. It was about everything that wasn’t changing.

That framing did more to lower the temperature than any argument about the merits of the change itself. Not because it dodged the hard conversation — the contents of that one box still had to be debated, properly and at length — but because it bounded the conversation. People could engage with the actual proposal instead of defending against an imagined one.

This is, I think, a structural property rather than a one-off trick. Without a pattern bank, a process change has no edges: nobody can say with confidence what it touches and what it doesn’t, so everyone reasonably assumes it touches them. With a pattern bank, a change is a diff. You can enumerate exactly which entries are modified, which seams gain or lose a gate, and which patterns are provably unaffected. The same catalogue that scopes new work scopes change to the work itself.

If you’re a lead who expects to steer your team through a significant shift in the next year or two — a new deployment model, a platform migration, a rework of how releases happen — that alone might justify building the bank now. It’s much easier to show people that only one box is changing if the boxes already exist.

Keeping it alive without making it a burden

The biggest risk to a pattern bank isn’t that it will be rejected. It’s that it will be adopted once and then quietly ignored as the team moves on to delivery pressures.

The solution is governance that’s light enough to actually happen.

Pattern review doesn’t need its own meeting. Fold it into your existing team rhythms — a retrospective, a sprint review, a regular team catch-up. Set aside fifteen minutes every month or two to ask: have any of our recently built solutions introduced a new pattern we should document? Has anything we’ve built repeatedly over the past quarter highlighted that one of our existing patterns needs refinement?

The trigger for adding a new pattern remains the same: you’ve built the same thing three times. Before that, document it as a one-off or a variant. After that, give it its own entry. The three-instance rule keeps the bank from growing with speculative patterns that haven’t proven their recurring value.

Retiring patterns is just as important as adding them. A pattern that hasn’t been used in twelve months is probably no longer part of how your team works. Don’t delete it — move it to a legacy section. Old pipelines built on deprecated patterns still exist, and future engineers will want context when they encounter them.

The most common way a pattern bank dies is through drift: the documented patterns diverge from what the team actually does, and nobody updates the documentation because the overhead isn’t worth it. The way to avoid this is to make update rituals small. One sentence changed, one variant added, one effort note updated after a project completes. It takes five minutes if it’s part of the project close-out conversation. It takes months if it’s treated as a separate documentation effort.

One small artefact worth keeping alongside the bank: a glossary. A dozen or so terms of art, one line each — what “landing” means here, what “staging” means here, what “the contract” refers to. It sounds trivial until you’ve watched two documents written three months apart quietly disagree about what a word means, or a design review burn twenty minutes discovering that two people were using “raw” differently. The patterns give the team nouns for solutions; the glossary keeps the rest of the vocabulary honest. It costs a page, and it makes everything else the team writes — pattern entries, design docs, handover notes — interoperable.

One practical approach: put the pattern bank in a place the team already touches regularly. A Teams wiki, a Confluence space, a shared document with a clear structure — wherever the team already goes for reference material. Don’t create a dedicated system that requires a new habit to use. The path of least resistance should lead to the pattern bank, not away from it.

What it looks like when it’s working

The sign that a pattern bank has taken hold isn’t that engineers consult it before starting every project. It’s that the vocabulary from it enters the team’s natural speech.

You’ll hear it in design conversations: “this looks like an SCD2 load with a late-arriving records wrinkle” rather than “so this is kind of like what we did for System B but slightly different.” You’ll hear it in planning: “the ingestion here is a known pattern, it’s the delivery piece that’s new territory.” You’ll hear it at the seams: “what’s in the handover pack for this one — could someone pick it up tomorrow without talking to you?” You’ll hear it when a new team member asks how things work, and an experienced engineer can answer them in fifteen minutes using pattern language rather than two weeks of reading old code.

You’ll also hear it in conversations with people outside the data team — architects, project managers, internal customers — who start to understand the platform not as a black box but as a system with legible parts. That transparency matters. When the architects designing how source systems should serve data understand what an ingestion pattern expects, they make better design decisions. When an internal customer understands the difference between a project that uses known patterns and one that introduces a new one, they understand why the effort estimates are different.

The pattern bank doesn’t make the work easier. It makes the work legible. And when the work is legible — to the engineers doing it, to the people managing it, and to the people upstream of it — everything else gets a little simpler.

Going back to that organisation where I started: the ingestion layer that had twenty different approaches to twenty similar problems didn’t get rebuilt. That wasn’t the point. But over time, as new source systems were onboarded, they were designed against the patterns the team had agreed on. The sprawl stopped sprawling. The conversations about effort estimates became more grounded. The invariants that mattered most stopped relying on memory and started failing builds instead. And when the day came that we needed to change how the team delivered — not just what it delivered — the bank turned a frightening proposal into a bounded one.

When a new engineer joined the team, there was something to hand them that explained not just what the platform did, but why things were built the way they were.

That’s what the pattern bank was, at its core. Not a governance document. Not a bureaucratic exercise. A way of saying: here is what we know, here is how we think about this, and here is how we’ve agreed to do it together.

Start there. Build it with your team. Refine it every time you learn something new.

Keep Moving

Chris Hillman — Tue, 23 Jun 2026 08:00:00 +1000

Some days I open my laptop and by 5pm I genuinely cannot tell you what I did.

Not because it was complicated. Not because there were emergencies. The stand-up happened. A few Teams messages were sent. A ticket was groomed. A document was “reviewed”. A meeting was attended where everyone agreed something was important and then the meeting ended and nothing changed.

And then somehow it was evening and the pipeline I meant to fix was exactly as broken as it was in the morning.

I used to think this was a time management problem. I’ve since come to believe it’s a physics problem.

An object at rest

Actual productive work — the kind that moves a platform forward — probably occupies about two or three hours of a given workday. Maybe four if you’re lucky. The rest is coordination, context-switching, Teams, and the ritual of almost starting.

I’ve seen senior engineers arrive late and leave early and outship everyone else on the team. I’ve seen people in meetings from 9 to 5 who are technically “at work” but haven’t touched a pipeline in three weeks. Hours in office — or in Jira — are a spectacularly poor proxy for output.

The hard part isn’t the work itself. Once you’re in it — once the editor is open, the query is running, the dbt model is taking shape — it flows. The painful part is the gap between intending to start and actually starting. That gap has a gravitational pull all its own.

I have a version of this routine I’m not proud of: open laptop, check email, check Teams, open the ticket I meant to work on, re-read my own notes from last week, decide I need more context, open Confluence, fall into a 20-minute rabbit hole of documentation I wrote six months ago, circle back to Teams, check if anyone replied to that thread, think about getting coffee, get coffee, and then, somewhere around 11am, finally open the editor and actually write the thing that takes forty-five minutes.

The work was never the problem. It was everything I let stand between me and starting.

This isn’t a character flaw. It’s inertia. Physics applies to engineers too.

The only way to build a platform

There’s a way I’ve come to think about what separates data platforms that actually mature from the ones that are permanently “almost there”.

It’s not the architecture. It’s not the tooling. It’s not the quality of the backlog or the rigour of the sprint ceremonies or the neatness of the Confluence hierarchy.

It’s whether the team is moving forward — every day, in some small way — or whether it’s standing still.

That sounds obvious until you try to do it. Because there are a hundred forces, every single day, designed to stop you from moving.

Some of them are internal. The engineer who won’t write a single line until the data model is “finalised”. The team that spends three sprints documenting the current state before touching anything. The migration project that requires a complete source system audit before the first table lands in Snowflake. All reasonable-sounding activities. All forms of not moving.

Some of them are external. That’s where it gets more interesting.

Cover fire

The data tooling industry has discovered something effective: if you can keep teams busy reacting to you, they don’t have time to ship.

Not consciously, probably. But the effect is the same.

Joel Spolsky wrote about this pattern more than twenty years ago in Fire and Motion. In infantry terms: you fire at the enemy not because you expect to hit much, but because it forces them to keep their heads down while you advance. In software terms, the big players generate a constant barrage of new technologies and standards so that everyone else burns their days keeping up instead of building. He was writing about Microsoft in 2002. The names have changed. The strategy hasn’t.

Every six months there is a new architectural paradigm your team is supposed to evaluate. The lakehouse. The semantic layer. Data contracts. The medallion architecture (deprecated). The medallion architecture (rehabilitated). Streaming-first. The modern data stack (thriving). The modern data stack (dead). AI-native pipelines. Agents that write your dbt models for you.

The vendor ecosystem has learned to generate enough noise that a conscientious engineering team can spend most of its time just keeping up: reading the announcements, watching the conference talks, debating whether to adopt the new pattern, setting up a proof-of-concept, discovering the PoC is more complex than advertised, abandoning it, and then starting the cycle again when the next announcement drops.

Meanwhile your users still can’t get a reliable enrolment count. The Oracle migration is in month fourteen. The dashboard that takes eleven seconds to load still takes eleven seconds to load.

The Snowflake vs Databricks wars, dbt Fusion vs dbt Core, Iceberg vs Delta vs everything else: some of it is genuinely important and you should stay informed. But a lot of it is just noise that benefits the people generating it, not the people consuming it. While your team is deep in an evaluation of a new ingestion paradigm, the basic pipelines you already have aren’t getting more reliable. The stakeholders who’ve been asking for a new data source for four months are still waiting.

They don’t care which open table format you’re using. They care whether their data is there in the morning.

When a sales engineer from a major vendor asks whether your platform supports the latest hot acronym — and they will, and it will be in a room full of your stakeholders — that question is doing work. It’s designed to make you feel behind. To make you wonder if you should be rebuilding rather than shipping. To redirect your energy toward something that benefits them.

The right response is rarely to immediately build the thing they’re asking about. The right response is usually to keep moving on the things that actually matter to the people you serve.

What moving forward looks like

I want to be specific about this, because “keep moving” is easy to say and genuinely hard to operationalise.

It doesn’t mean rushing. It doesn’t mean skipping the thinking or writing code you’ll hate in six months. It means that every sprint, every week, every day — something is better than it was. A pipeline is more reliable. A model is documented. A test passes that didn’t pass before. A stakeholder gets a data source they couldn’t access last month.

Small, consistent, compounding. The platform that looks dramatically better in twelve months is almost always the one that had someone quietly improving things every single day, not the one that attempted a big-bang rewrite.

On the personal side: the most useful thing I’ve found is to remove the decision of whether to start. If you have to decide to begin every morning, you’re giving inertia a fighting chance. The editor should be open before you check Teams. The model you’re working on should be the first tab, not the fifth. Pair programming works partly because it eliminates optionality. You can’t not start when someone is sitting there waiting.

On the team side: the most dangerous status update is “working on it”. It sounds like movement. It often isn’t. The rituals that actually correlate with forward progress are the boring ones: small PRs merged frequently, tickets moving from in-progress to done in days not weeks, users getting things they asked for.

On the strategic side: have an honest answer for what you’re choosing not to evaluate right now, and why. “We’re not looking at that this quarter because we’re migrating the Oracle GL tables and that’s more valuable” is a complete sentence. You don’t need to defend it at length.

The platform that keeps moving wins

I don’t think data engineering is particularly hard compared to what else people do with computers. The craft is learnable. The patterns are well documented. The tooling is genuinely better than it was five years ago.

What’s hard is sustaining forward motion through the accumulated weight of meetings and tool evaluations and perfectly reasonable reasons not to ship today.

The teams I’ve seen build platforms that people actually trust and use are almost never the ones with the most sophisticated architecture. They’re the ones that kept moving — through organisational changes, through vendor noise, through the days when it felt like nothing was getting done.

There’s a cost to this approach, and it’s worth naming. If you treat most of the noise as noise, occasionally you’ll be wrong. Something you dismissed will turn out to be the real thing, and you’ll arrive eighteen months after the early adopters, reading their write-ups instead of writing your own. That’s the trade, and I’d take it. Being late to a paradigm is recoverable. A platform nobody trusts is not.

The job, at every level, is to move forward a little every day. Fix the pipeline. Write the test. Ship the model. Even when it’s small. Even when it’s imperfect.

The platform gets better a little at a time, or not at all.

Somehow, launch the editor.

Ghost Skills: Teaching AI Agents to Think Like Data Engineers

Chris Hillman — Sun, 14 Jun 2026 09:00:00 +1000

Another week, another skills repo on the GitHub trending page. I know. There are roughly seventeen of them now, all promising to turn your AI coding agent from a confident intern into a slightly-less-confident intern. Most of them are great. Most of them are also built by solo devs, for solo devs, on solo-dev codebases that fit comfortably in a context window.

Which is fine, if that’s your world.

Less fine if your world involves a Snowflake warehouse with four tables that could be the source of truth for “customer”, an SCD2 someone half-built in 2021 and quietly walked away from, and a dbt project where stg_users_final_v3_actually_use_this is, somehow, the one you’re meant to use. (Don’t laugh. You’ve seen worse.)

I’ve been using AI agents day-to-day for a while now — mostly as a peer reviewer, a second pair of eyes, a thinking partner that doesn’t sigh when I ask it to look at the same query three times. They’re genuinely useful.

So I built something Ghost Skills — a collection of data-engineering methodology skills for AI coding agents. Before I get into what’s in it, let me explain why I bothered.

The problem isn’t the model. It’s that data work has rules nobody told it about.

Your AI agent is, by default, an extremely bright graduate on their first day. It knows Python. It knows SQL. It can write a beautifully-commented for loop and explain window functions to you in five different ways. It will happily generate a fact table, a dbt model, an Airflow DAG, all of it looking very professional.

What it doesn’t know is everything that actually matters.

It doesn’t know that the customer_id in that source isn’t really unique because of the 2022 CRM migration. It doesn’t know that Roger pushed a release on Friday afternoon (because of course he did) and the platform’s been wheezing through a seven-year backfill ever since. It doesn’t know that this dimension changes slowly and the audit team will lose their minds if you flatten the history. It doesn’t know that the business definition of “active customer” changed in March, but only in the marketing data mart, and only sometimes.

These aren’t model intelligence problems. The model is plenty smart. They’re context problems — and the context is exactly the stuff that takes a data engineer two years to learn and roughly thirty minutes to forget when they leave the company.

What agents are missing isn’t capability. It’s methodology. It’s the durable craft of data engineering — the part that doesn’t change when you swap Snowflake for BigQuery, or dbt for SQLMesh, or your warehouse for a “lakehouse” that’s somehow priced like a warehouse anyway.

That’s the gap this is attempting to fill.

Skills, briefly, for anyone who hasn’t drunk this particular Kool-Aid yet

If you’ve not been living inside Claude Code or Cursor for the last six months: a skill is a folder of instructions you give an AI coding agent. “When you build a fact table, declare the grain first.” “When you write tests, follow this severity model.” “When something breaks, here’s the comms workflow.” You version it, you commit it, you keep it in the repo. The agent reads it before doing the thing, and re-reads it next session because agents have the long-term memory of a goldfish.

It’s a deceptively powerful pattern. Instead of pasting the same prompt every session (and forgetting half of it, and writing it slightly differently this time), the standard lives in the repo. New team member joins? They get the skills. Agent updates next month? It still reads the skills.

The catch — and we’ll come back to this — is that skills tell an agent how you do something. They don’t tell it what you’ve built, why you built it that way, or which of your seven utility tables is the one that’s still maintained.

For solo devs, that’s fine. There is no “seven utility tables”. For data teams at any kind of scale, it’s a real limit. More on that at the end.

Introducing Ghost Skills

Right. The repo: github.com/ghostinthedata-info/skills. Built mostly for myself, polished up for sharing.

30-second setup:

npx skills@latest add ghostinthedata-info/skills

Pick the skills you want, pick your agent (Claude Code, Codex, Cursor — whatever you’re using), and run /setup-ghost-skills. It’ll ask you three questions: warehouse dialect, transform tooling, and how your domain docs are laid out. Then it writes itself a configuration block into your CLAUDE.md or AGENTS.md and gets out of your way.

The agent reads it next session. And the one after that. And the one after that. The same standards, every time, without you re-explaining them.

That’s it. That’s the whole thing.

What’s in the catalogue

The skills group into four areas, mapped to roughly how data work actually happens.

Discovery is everything you should do before you build anything, and frequently don’t. profile-data runs the baseline checks on a new dataset — row counts, cardinality, null analysis, key uniqueness — so the agent isn’t generating models against assumptions that fall apart in week two. gather-requirements pins down grain, sources, consumers, freshness SLAs, and acceptance criteria one question at a time, instead of letting the agent guess. refine-context stress-tests a plan against your documented domain model and writes the decisions back to CONTEXT.md and ADRs as they crystallise.

Modeling is the methodology your agent should already be applying and usually isn’t. dimensional-modeling walks the four-step process. fact-table-design forces grain declaration and measure classification before a single line of SQL gets generated. keys covers business, natural, surrogate, composite, and durable keys — and the anti-patterns that bite you eighteen months later when the source system gets re-platformed. slowly-changing-dimensions covers SCD types 0 through 7, including my Healing Tables approach to deterministic, path-independent SCD2.

Quality is where defensive engineering lives. test-data produces a test plan you can actually defend in a code review — uniqueness, referential integrity, nulls, accepted values, freshness, volume variance — and maps each test to a severity level. performance-tuning follows the measure-first philosophy: find the critical path, then fix partition pruning, incremental processing, and the phantom dependencies that are silently doubling your costs. spark-performance handles the distributed case — partition counts, shuffle minimisation, skew handling, broadcasting small tables.

Operations is what should happen when things go pop. incident-comms gives the agent the severity classification, notification workflow, update cadence, and post-incident review template — the “Don’t Go Dark” pattern in skill form. pipeline-design encodes idempotency, reproducibility, and defensive engineering as defaults instead of afterthoughts. data-as-a-product brings data mesh thinking into the agent — domain ownership, discoverability, SLAs, federated governance. data-security-classification covers the four A’s, PII/PHI handling, and least privilege.

There’s a setup skill (setup-ghost-skills) that runs once per repo and handles the configuration. Everything else, you opt into.

Why tool-agnostic, even though tool-specific would sell better

The temptation when building something like this is to specialise. Snowflake-only. dbt-only. Make every skill assume your exact stack and produce immediately runnable code.

I deliberately didn’t do that, and the tradeoff is worth naming honestly. Generic skills are less immediately useful than highly specific ones. A skill that knows your exact dbt project structure, your column naming conventions, your team’s snake_case-vs-camelCase argument that’s been going on since 2022 — that’s more powerful, day one.

But it’s also yours. It’s not shareable. And the bits that are genuinely universal — declare the grain before you build the fact, a business key isn’t a real business key until you’ve checked it’s unique, test before publish — those don’t change between platforms. They didn’t change when we moved from Teradata to Snowflake. They won’t change when we move from Snowflake to whatever everyone’s furiously rebranding next year.

Ghost Skills is the base layer. Your repo’s CONTEXT.md is your specialisation layer. They’re meant to work together, not replace each other. Fork it. Add a snowflake-cost-optimisation skill on top. Open a PR if you want to share it back. (Or don’t. Do what you like.)

Skills help. They don’t solve everything. Let’s not pretend.

The honest thing to say at this point is that skills are not a silver bullet, and anyone telling you otherwise is selling you a course.

Skills tell an agent how to do things. They don’t tell it what’s there or why. An agent that knows the Kimball four-step process will still make the wrong grain declaration if it doesn’t understand the business process it’s modelling. A skill can encode what SCD2 is; it can’t replace a conversation with the stakeholder who actually needs the history. A skill can encode the Write-Audit-Publish pattern; it can’t tell the agent which of your fourteen “users” tables is the one to run it against.

The other thing skills can’t do is fix a stale skill. The conventions you encode today are the conventions you knew today. If your team’s standards evolve — and they should — your skills need to evolve with them. A stale skill is worse than no skill, because it gives the agent a confident wrong answer instead of asking a question.

So treat them as a starting point. Use them. Fork them. Argue with them in a code review. Update them when your standards shift. The agent will keep reading whatever’s in the repo, faithfully, every session — which means the skills are only ever as good as the last person who maintained them.

The repo is at github.com/ghostinthedata-info/skills. Fork it for your cloud-specific layer. Open issues if something’s wrong. Open a PR if you’ve got methodology worth encoding and you don’t want to keep it to yourself.

And if you build something better on top of it, please tell me.

The Competitive Moat That AI Can't Replicate

Chris Hillman — Sat, 13 Jun 2026 09:00:00 +1000

The Restaurant That Refused to Take Bookings Online

Let me tell you a story about a restaurant owner who became obsessed with human connection.

He didn’t want people booking online. He wanted them to call. He wanted the ritual of a human voice, the small exchange about an anniversary or a first date, the warmth of being recognised. His team thought he was losing his mind. Online bookings were standard. Everyone did it. Why make customers work harder?

One of the managers finally asked him a pointed question: Have you ever actually made a reservation with us?

He hadn’t. So he did.

He called the restaurant. He was put on hold for thirty minutes. When someone finally answered, they were apologetic but firm — the restaurant was fully booked. No warmth. No conversation. Just a long wait and a closed door.

In trying to humanise the process, he’d made it worse.

Here’s where most stories would end with “so they moved everything online and fired the reservation staff.” That’s not what happened. They did move to online booking, but they kept the entire reservation team and repurposed them. These people now spent their days learning about the customers coming in that night. Who was celebrating a birthday? Who was on a first date? What had a regular not finished on their plate six months ago?

The team became mini-concierges. Every guest walked in to find someone who knew them — not in a creepy, surveillance-state way, but in the way a good friend remembers what you’re going through. The technology handled the transactional layer so the humans could focus on the relational layer.

I’m telling you this story because it captures something I think most organisations have forgotten in the rush to automate everything: the work of connection isn’t overhead. It’s the point. And as AI swallows the transactional layer of every industry at once, that relational layer is about to become the only point of difference left.

Trust Is Built One Marble at a Time

Brené Brown tells a story about her daughter Ellen that I realate to. Ellen came home from third grade devastated. She’d shared something vulnerable with a friend at recess, and by the time class started again, half the school knew. She told her mum, through tears, that she would never trust anyone again.

Brown’s first instinct, honestly, was to agree. Tell no one anything, ever. But she caught herself and offered a different idea — the marble jar. There’s a jar in Ellen’s classroom where good choices add marbles and bad ones remove them. When the jar fills up, the class celebrates. Trust, Brown told her, works the same way. You share the hard things with people who have filled up your marble jar over time. Marble by marble. Small moment by small moment.

Ellen immediately named two friends whose jars were full.

What stopped Brown cold was what counted as marbles. One friend had scooted over at the lunch table to make “half a heinie seat” when Ellen had nowhere to sit. Another had remembered the names of Ellen’s grandparents at a soccer game. Brown couldn’t believe it. Surely trust required grander gestures than this.

She went back to her research data and found exactly the opposite. The single highest-ranked trust-building behaviour in her entire dataset? People who attended funerals. Not the people who made big speeches or grand gestures. The people who showed up in small moments that mattered.

Trust isn’t a transaction. You can’t purchase it, automate it, or accelerate it with a clever marketing campaign. It accumulates in tiny, unremarkable moments that look like nothing from the outside — and everything to the person on the receiving end.

And here’s what I find most interesting about the marble jar: once it’s full, it’s incredibly hard to empty. You can weather bad days, miscommunications, even genuine mistakes. But if you’ve never bothered to fill it, a single bad interaction is all it takes. There’s nothing to cushion the fall.

Every organisation has customers whose jars are full and customers whose jars are empty. The ones with full jars forgive outages, laugh off a late delivery, stay through a price increase. The ones with empty jars churn the first time anything goes wrong. Most organisations don’t realise which is which until it’s too late.

Hospitality Is a Dialogue, Service Is a Monologue

Danny Meyer, the restaurateur behind Union Square Hospitality Group and Shake Shack, draws a great distinction. Service is the technical delivery of a product — the food arrives hot, the bill is accurate, the room is clean. Hospitality is how the delivery of that product makes its recipient feel.

Service is a monologue. You decide your standards and you deliver against them. Hospitality is a dialogue. You notice, you adjust, you respond.

Meyer puts it even more sharply: hospitality exists when you believe the other person is on your side. It’s present when something happens for you. It’s absent when something happens to you. Those two little prepositions contain the whole difference. You feel it instantly when it’s there. You feel it even more instantly when it’s not.

Meyer isn’t anti-technology, by the way. But he draws a firm line: the technology he’s interested in is technology that reinforces hospitality, not technology that replaces it. He gives sommeliers Apple Watches for real-time information. He doesn’t give them to waiters, because waiters need to maintain eye contact with guests.

Thats super important. A restaurateur whose restaurants have earned basically every award that exists decided his waiters should not be looking at screens because eye contact is more important than information. When was the last time you saw a decision like that made inside a large organisation?

The best story I know in this vein is from Will Guidara, who ran Eleven Madison Park when it was named the best restaurant in the world. One night, Guidara overheard a table of European food tourists lamenting that they’d eaten at Per Se, Le Bernardin, and Daniel — but they were flying home the next day and had never tried a proper New York City hot dog.

Guidara ran outside, bought a $2 hot dog from a street cart, convinced his chef to plate it with “swooshes of ketchup and a quenelle of relish,” and had it served as a surprise mid-course. Every person at the table said it was the highlight of not just the meal, but their entire trip to New York.

A $2 hot dog in a room full of the most expensive food in America. That’s the gap between service and hospitality. You cannot design an algorithm that eavesdrops on dinner conversation and dispatches someone to buy a street hot dog, because the person on the receiving end would immediately sense the machinery of it. The magic is precisely that a human heard, a human decided, a human cared.

Guidara was so moved by what that moment revealed that he created a dedicated role called the Dreamweaver — someone whose entire job was to make these moments happen. Researching guests ahead of visits, building cheat sheets so staff could greet people by name, listening for conversational cues during dinner. This is exactly what the restaurant in the opening did. The technology handles the boring parts so the humans can do the thing only humans can do.

The Things We Destroy Because We Can’t Measure Them

Here’s where I get frustrated, and this is the part of the article I’ve rewritten three times.

I’ve spent a lot of time in and around banking over the past few years, and I’ve watched a particular kind of decision get made over and over. It always starts the same way. Someone pulls a report showing branch transaction volumes declining. Someone else pulls a report showing the cost per square metre of the branch network. A third person models what happens if you close the bottom quartile. The numbers are clean. The deck practically writes itself. And the decision gets made in a room where nobody has ever stood behind the counter and watched what actually happens on a Tuesday morning at half past ten.

What you see when you actually stand there is not transactions. You see a retired schoolteacher who comes in every week not because she can’t use the app — she can — but because the teller knows her by name and asks about her grandson. You see a small business owner dropping off the weekly takings and mentioning, almost offhandedly, that he’s thinking about expanding into the next suburb, and the branch manager quietly filing that away because she knows a commercial lender who should probably call him. You see the elderly man whose wife just died, standing at the counter trying to change the joint account, and the staff member who recognises what’s actually happening and gently walks him through it for forty-five minutes without once looking at a clock.

None of that shows up in the transaction volume report. None of it shows up in the cost-per-square-metre model. But it’s where the loyalty comes from. It’s the reason the schoolteacher’s kids bank with the same bank and her business-owner neighbour refinanced there and the grieving husband never even considered moving his accounts when the rate went up.

When the branch closes, all of that evaporates. Not dramatically. Just quietly. The schoolteacher switches to the credit union down the road because they still have tellers. The business owner realises nobody at the new centralised lending hub remembers him, and starts taking meetings with competitors. The grieving husband’s children, watching their dad struggle with the phone banking menu, consolidate everything with a different bank when they inherit. Three years later the spreadsheet shows a modest increase in customer churn and nobody connects it to the branch closure, because the causal chain is too long and the data was never collected in the first place.

The broader pattern is consistent everywhere I look. Australian banks have closed something like 2,500 branches since 2017. The justification was always the same: digital adoption, customer preference, efficiency. But when whistleblowers from the Finance Sector Union testified at the Senate inquiry, a darker story emerged. Branch staff had been performance-managed to suppress in-branch activity — pushed to redirect customers to ATMs, pressured to sign them up for digital banking. The data that justified the closures had been partly manufactured by the closures themselves.

And what the banks never properly measured was any of this. Every one of those moments is share of wallet. Every one of them is retention. And none of it appeared on the spreadsheet that closed the branch.

The tragic irony is what came next. The same banks that couldn’t justify keeping branches open are now spending billions on AI personalisation engines designed to replicate the exact relationships they dismantled. JPMorgan Chase budgeted $18 billion for technology spending in 2025. Bank of America’s AI assistant has surpassed two billion client interactions. And yet only about a quarter of consumers say their bank provides tailored financial advice. They destroyed the organic version and are now paying orders of magnitude more trying to rebuild a synthetic one.

There’s a concept I keep coming back to called the McNamara Fallacy, named after Robert McNamara, the US Defense Secretary who tried to run the Vietnam War like a business. McNamara was brilliant with metrics. A general once told him he needed to add an “x-factor” to his measurement list — the feelings of the rural Vietnamese people. McNamara wrote it down, asked what it meant, and then sarcastically told the general he couldn’t measure it, so he erased it.

Daniel Yankelovich described the fallacy as a four-step descent. First, you measure whatever can be easily measured. Second, you disregard what can’t be easily measured. Third, you presume that what can’t be measured easily isn’t important. And finally — this is the fatal step — you presume that what can’t be easily measured doesn’t exist. Yankelovich called this last step “suicide.”

Every organisation I’ve worked with has done this at some level. Dashboards capture the measurable. But the most valuable things — trust, loyalty, the feeling a customer gets when someone remembers their name, the way an employee feels when their manager notices they’re having a hard week — live in the unmeasured space between the data points. And when we optimise away the things we can’t count, we destroy the foundation that made the things we can count actually work.

If you work in data, as most people reading this do. The reports that justified those branch closures were technically accurate. Well-modelled, properly sourced, beautifully visualised to requirements. Someone like us built every one of them.

Wells Fargo is the textbook case. Leadership set a target of eight accounts per customer. “Eight is Great” became the mantra. Over fourteen years, employees created 3.5 million fraudulent accounts to hit the number. They optimised the metric so aggressively they destroyed the thing — trust — that made banking relationships worth anything in the first place. The fake accounts generated about two million dollars in fees. The fallout cost them more than three billion.

Charles Goodhart’s law captures this beautifully: “When a measure becomes a target, it ceases to be a good measure.” The moment you start optimising for NPS, NPS stops telling you anything true about your customer relationships. The moment you start optimising for branch foot traffic, foot traffic stops telling you anything true about community banking. You end up managing the shadow and losing the substance.

Why the Apple Store Is Always Full and the Carrier Store Is Always Empty

Walk into any shopping centre and do a little experiment. Find the Apple Store. It’ll be packed. People learning photography, kids in coding workshops, someone at the Genius Bar getting help with an old laptop, others just hanging out because the store feels like a place you want to be. Then walk to the Telstra store, or Optus, or whichever carrier is nearest. You’ll find two or three bored staff members, maybe one customer, and the unmistakable atmosphere of a waiting room at a tax office.

They sell, to a remarkable degree, the same products. iPhones, accessories, plans. So why does one feel like a community space and the other feel like a place you’re only in because you have to be?

The answer starts with Ron Johnson, who built Apple retail alongside Steve Jobs in 2000. Johnson made a decision most retailers would never make: a store needed to be much more than a place to acquire merchandise. It needed to help people enrich their lives. Any website could transact. A store had to do something a website couldn’t.

He and Jobs asked themselves a question I still think about: what would the Four Seasons do? The Four Seasons doesn’t have cashiers. It has a concierge. So Apple introduced concierge greeters. The Four Seasons has a bar — a place where you can sit, ask questions, and get real help from a knowledgeable person. Apple created the Genius Bar. It dispensed advice instead of alcohol, but the idea was the same: a place to be cared for, not a place to be processed.

Then they did something that, from a commercial perspective, looks insane. They removed commissions. Apple Store employees earn zero percent on sales. No upsell quotas, no commission targets, no pressure scripts. Johnson’s framing was that Apple wanted to reach your heart instead of your wallet — and staff whose paycheque depends on closing a sale cannot be fully on the customer’s side.

Compare this to how a carrier store operates. Employees have sales targets. They’re incentivised to push you toward plans and devices that pay them more. Former carrier staff in Australia and the US tell the same story: focus on customer service and you get coached on pushing harder instead. The misalignment between what the employee needs and what the customer needs sits in the air. Customers feel it as distrust, even if they can’t name it.

Apple Stores generate roughly $5,500 in revenue per square foot — about twice the next-highest retailer on earth. They get over a million customer visits a day. But only about one in a hundred visitors actually buys something. Ninety-nine percent walk in, spend time, and walk out without making a purchase. And that’s the whole point. Apple built the most profitable retail operation in history by investing in the ninety-nine people who aren’t buying today, because every visit is a marble in the jar.

The carrier stores are trying to convert everyone who walks in. They fail. Apple is trying to make everyone who walks in feel good. They succeed, and they also happen to sell more per square foot than any retailer in history. The causation is not subtle.

AI Raises the Floor. Humans Raise the Ceiling.

This isn’t an anti-AI article. I work in data. I use AI every day. I think it’s genuinely transformative and most organisations should be using it more aggressively than they currently are.

But the conversation about AI has got the direction wrong. The question isn’t whether AI will replace human connection. The question is what AI should free us up to do. If the answer is “more automation, more optimisation, more throughput” — we’ve missed the plot entirely.

There’s a concept emerging in the marketing and design literature that I find genuinely useful: AI raises the floor, but it lowers the ceiling. A Fortune 500 study of customer-support agents found that AI boosted the average agent’s productivity by 14%, but boosted the least-skilled agents by 34%. That’s the floor rising. But when researchers tested AI on creative writing, they found something unsettling. Individual AI-assisted stories were rated higher than purely human stories. Read a whole collection of them together, though, and they were rated lower. They all felt the same. The ceiling dropped because the variance dropped.

Here’s what that means for organisations competing on anything other than price. If every company uses the same AI tools to optimise the same metrics, the optimisation itself becomes commodity. Everyone has the same chatbot, the same personalisation engine, the same churn predictor. The floor rises for everyone simultaneously, which means relative advantage disappears. The only remaining differentiator — the actual moat — becomes the irreducibly human: genuine empathy in a crisis, a person who actually cares, a brand that feels run by real people who listen.

John Naisbitt, the futurist who coined “high-tech, high-touch,” made an observation back in 1999 that feels almost prophetic now: the two biggest markets in America are consumer technology and escape from consumer technology. We’re in the same dynamic today, just at a higher frequency. The more AI we get, the more valuable the unmediated human encounter becomes.

I’m seeing this play out in hospitality right now. Mid-range hotels are racing toward automation — kiosk check-in, chatbot concierges, app-based everything. But luxury hotels are moving in the opposite direction. They’re bringing back butlers, personalised greetings, tailored in-person experiences. In an automated world, human service becomes the luxury.

Simon Sinek put it well: AI is a tool, not a replacement. The struggles and imperfections in human interaction are often where the most meaningful growth and strongest bonds happen. Those imperfections are not bugs to be optimised away. They’re the texture of real connection.

Your Employees Are the Moat. The Compounding Is Invisible.

There’s a piece of this conversation that most leaders miss, and it’s the part I care about most. You cannot deliver genuine human connection to customers if you haven’t first delivered it to your employees. The service-profit chain — a Harvard Business School framework that’s been around for decades — is unambiguous: internal service quality drives employee satisfaction, which drives retention, which drives external service value, which drives customer loyalty, which drives revenue. It’s a chain. You cannot cheat the order.

Herb Kelleher ran Southwest Airlines on this principle for decades. Employees come first; treat them right and they treat your customers right, and the customers come back. The things you can’t buy, he said, are dedication, devotion, loyalty — the feeling of participating in a crusade. Competitors can buy the physical stuff. They cannot buy the feeling. The results: number one in US Department of Transportation customer service rankings twenty-six times in thirty-four years, and forty-seven consecutive years of profitability in an industry famous for burning money.

His successor as president, Colleen Barrett, started at Southwest as his legal secretary. She said he treated her as an equal from the day they met, and he treated every employee the same way. That’s not a culture program. That’s a human being who decided to care about other human beings and then built a company around the decision.

This matters for the AI conversation because there’s a temptation, especially under cost pressure, to look at AI and think “great, we can run leaner.” That’s the trap. Use AI to squeeze more out of fewer people and you’ll hit short-term margins while quietly draining the moat. Use it to free people up to do the things only people can do — notice, listen, care, remember — and you’ll build something that compounds for years.

And compounding is exactly the right word: trust compounds, but the compounding is invisible for a long time. James Clear’s ice cube analogy applies. Heat a frozen room from twenty-five degrees to thirty-one (Fahrenheit — stay with me) and nothing appears to happen. Nothing, nothing, nothing — then at thirty-two degrees, everything happens at once. All the action is at the threshold, and all of it is invisible until it isn’t. Most organisations quit before the threshold arrives, because the accounting cycle is quarterly and the compounding cycle is years.

Seth Godin says: “You must build trust before you need it. Building trust right when you want to make a sale is just too late.” You get there drip by drip by drip, until people would miss you if you were gone.

So What Do You Actually Do With This?

Ask yourself which of your customer-facing moments are dialogue and which are monologue. If every interaction is something you’ve decided to deliver to the customer rather than for them, you don’t have hospitality. You have service. Service is fine. Service is table stakes. But it’s not a moat.

Ask yourself what your employees would say if someone asked them whether they were cared for. Not in a survey. In an honest conversation over coffee. That answer is the ceiling on what your customers will ever feel, no matter how much you spend on customer experience.

Ask yourself which of your metrics you’re managing and which ones you’re serving. If your teams are optimising for NPS, you’ve lost NPS as a signal. The moment a metric becomes a target, it stops telling you the truth. You need to look at the behaviour underneath the number — the actual quality of the human moments — and measure those, even if the measurement is messy.

Ask yourself where you’re using technology to replace human moments rather than amplify them. The restaurant in the opening didn’t remove the reservation staff. They redeployed them to do the thing only humans can do. That’s the move. Automate the transaction. Invest the savings in the relationship.

Remember that the dashboards capture the measurable but the valuable things live in the space between the measurements. The best data work I’ve ever seen hasn’t been about building better dashboards. It’s been about giving humans better context so they can have better conversations with other humans. That’s the quiet revolution most data teams are missing. We keep building tools to replace the human layer when we should be building tools to enrich it.

The moat isn’t your data pipeline. It isn’t your AI model. It isn’t your CRM. The moat is what your people do with the insights — the handshake, the eye contact, the moment of genuine care that no algorithm can fake and no competitor can copy. You can’t buy it and you can’t rush it. You can only show up, drip by drip, marble by marble, for longer than everyone else is willing to.

SQL Tells You What. Comments Tell You Why.

Chris Hillman — Sat, 06 Jun 2026 09:00:00 +1000

The best SQL doesn’t need comments. Write meaningful CTE names, descriptive aliases, clear column labels — and a skilled reader will follow your logic without a single annotation. That’s the right instinct.

It’s also only half right.

SQL is a declarative language. You’re not writing how the database retrieves your data; you’re writing what you want. That’s a useful distinction, because “what” and “why” are very different questions, and SQL can answer exactly one of them.

A query can be perfectly, elegantly readable and still be completely opaque about the reason it exists. The name of your CTE, no matter how well chosen, cannot explain the business decision that gave birth to it.

The self-documenting ceiling

Consider this query:

SELECT
 customer_id,
 SUM(order_total) AS lifetime_value,
 DATEDIFF(day, MIN(order_date), CURRENT_DATE) AS customer_age_days
FROM
 orders
WHERE
 order_date >= DATEADD(day, -90, CURRENT_DATE)
 AND status_code NOT IN (1, 2, 9)
GROUP BY
 customer_id

Clean SQL. No ambiguity about what it does. But try answering these questions from the code alone:

Why 90 days? Why not 60, or 180, or the full history? Is 90 the standard customer lifetime window, a finance reporting period, the SLA in a merchant agreement, or the number someone picked in a meeting four years ago and nobody has questioned since?

What are status codes 1, 2, and 9? You might guess from context — pending, draft, cancelled — but you’re guessing. More importantly: why are they excluded? Is this a business rule about what counts as “real” revenue, or a data quality workaround because those status codes appear when an upstream webhook fires twice?

These are not trivial questions. The 90-day cutoff defines what “active customer” means across your entire reporting layer. The excluded status codes determine what gets counted as revenue. Change either without understanding the original intent, and you’re not refactoring — you’re silently redefining business logic that stakeholders are relying on.

Better SQL helps, but only so far:

SELECT
 customer_id,
 SUM(order_total) AS lifetime_value,
 DATEDIFF(day, MIN(order_date), CURRENT_DATE) AS customer_age_days
FROM
 orders
WHERE
 order_date >= DATEADD(day, -90, CURRENT_DATE)
 AND status_code NOT IN (
 1, -- pending
 2, -- draft
 9 -- cancelled
 )
GROUP BY
 customer_id

This is better. You’ve replaced the mystery with labels. But inline labels aren’t explanations — they’re just naming the what more explicitly. You still don’t know why.

What SQL literally cannot tell you

SQL cannot explain why the program was written the way it was. It cannot discuss the reasons certain alternative approaches were taken. It cannot tell you that the business definition changed, that the upstream system is broken, or that this logic exists specifically to handle a problem that was supposed to be fixed in Q2 and wasn’t.

Here’s the kind of context that belongs in a comment:

-- Customer lifetime window: 90-day lookback aligns with the SLA in the Merchant Agreement (v3.2).
-- Reviewed and confirmed with Finance, Dec 2024. Don't change without checking with that team first.
-- status_codes 1, 2, 9 excluded per the order lifecycle defined in the legacy Salesforce migration.
-- These codes appear as artefacts when the Salesforce sync fires before the order is confirmed.
-- The upstream fix was deprioritised (JIRA: DATA-3841). Removing this filter will cause double-counting.
SELECT
 customer_id,
 SUM(order_total) AS lifetime_value,
 DATEDIFF(day, MIN(order_date), CURRENT_DATE) AS customer_age_days
FROM
 orders
WHERE
 order_date >= DATEADD(day, -90, CURRENT_DATE)
 AND status_code NOT IN (1, 2, 9)
GROUP BY
 customer_id

That comment block is five lines. It contains: the business rationale, the stakeholder who owns the decision, the date it was confirmed, the upstream system causing the problem, the known issue reference, and the consequence of removing the filter. None of that information is recoverable from the SQL. All of it is load-bearing.

The deduplication that nobody remembers adding

Have you seen this line before:

QUALIFY ROW_NUMBER() OVER (
 PARTITION BY customer_id, order_id
 ORDER BY ingested_at DESC
) = 1

The SQL is completely transparent about what it does: take the most recently ingested row for each customer/order combination. What it cannot tell you is why duplicates exist in the first place, whether the source of those duplicates has been fixed, whether this deduplication logic is still necessary, or whether “most recently ingested” is actually the right tiebreaker for your use case.

-- Deduplicating on customer_id + order_id to handle duplicate webhook events from Eftpos.
-- Eftpos retries failed webhooks, which can fire the same order_created event multiple times.
-- Taking the latest ingested_at gives us the most recent event state (status, amount, etc.)
-- Revisit if we adopt Eftpos's idempotency keys at the integration layer — this dedup may become redundant.
-- Background: this logic was added after the October 2024 incident (post-mortem in Confluence).
QUALIFY ROW_NUMBER() OVER (
 PARTITION BY customer_id, order_id
 ORDER BY ingested_at DESC
) = 1

The second version tells the reader everything they need to know to maintain this safely: the upstream cause, the logic behind the tiebreaker, when to reconsider it, and where to find the history. Without that comment, every future engineer who touches this code has to reverse-engineer context that no longer exists anywhere.

When two models define the same word differently

This one doesn’t generate an error. It generates confusion at the worst possible moment.

-- IMPORTANT: 'churned' here means >90 days since last order.
-- This differs from mkt_customer_segments, which uses >60 days for winback campaign targeting.
-- Both definitions are intentional. Customer Success uses 90 days because cohort analysis
-- showed that 30% of "60-day churned" customers reorder organically within the next 30 days
-- and shouldn't receive a discount. Marketing uses the stricter threshold to maximise winback opportunities.
-- The discrepancy is documented and known. Do not "fix" one to match the other.
CASE
 WHEN days_since_last_order <= 30 THEN 'active'
 WHEN days_since_last_order <= 90 THEN 'at_risk'
 ELSE 'churned'
END AS customer_segment

Without that comment, someone eventually “cleans up” one of the two definitions because having different churn thresholds in two models looks like a bug. They’re wrong, and they’re about to break a winback campaign that’s been performing well for six months.

The SQL cannot warn them. The comment can.

dbt descriptions aren’t exempt

dbt gives you a proper home for the “why”: model descriptions and column-level descriptions in your YAML.

Bad:

models:
 - name: fct_orders
 description: "The orders fact table. Contains order data."
 columns:
 - name: customer_segment
 description: "The customer segment based on days since last order."

That description restates what the model name already says. It is a comment-shaped void.

Good:

models:
 - name: fct_orders
 description: >
 Order-level fact table used as the single source of truth for revenue reporting.
 Revenue excludes test accounts (prefixed 'INTERNAL_') and gift card redemptions,
 per the Finance revenue recognition policy agreed January 2025.
 See Confluence: Revenue Recognition Standards v3.
 Do not use this model for marketing attribution — use mkt_attributed_revenue instead,
 which applies different channel logic.
 columns:
 - name: customer_segment
 description: >
 Lifecycle segment defined by the Customer Success team, Q1 2025.
 Thresholds (30/90 days) were derived from cohort analysis showing a median
 reorder window of 28 days. Intentionally differs from the Marketing segment
 definition in mkt_customer_segments, which uses a 60-day churn threshold
 for winback campaign purposes. Both are correct for their respective use cases.

That description does something a column name never can: it tells you who owns the definition, when it was set, what analysis underpins it, and — critically — where the definition intentionally diverges from a similar field elsewhere in the warehouse. That last part is the thing that saves someone three hours of confused Teams messages.

The archaeology problem

Data teams inherit codebases. It happens constantly, and it will keep happening. Engineers leave. Consultants finish their contracts. Reorgs shuffle ownership. What gets left behind is the SQL, and whatever context wasn’t written down is gone.

Good SQL survives the handover. It’s readable, consistent, and correct. But “readable” in the sense of mechanically parseable is not the same as “understandable” in the sense of knowing what business problem the code was solving, whose decision it was, and what you’d need to change if that decision changed.

Comments are how you write for the engineer who inherits this in two years. They’re how you write for yourself in six months. They’re how you ensure that a working pipeline stays working through the context collapse that happens every time anyone leaves a team.

SQL can only tell you what. Try not to shortchange the people reading it in either direction.

Don't Go Dark: Visibility Is a Data Engineering Skill

Chris Hillman — Sat, 23 May 2026 09:00:00 +1000

There’s a specific kind of silence in data engineering that I’ve learned to fear.

Not the silence of a system that’s working well. Not the comfortable quiet of a team in flow. I mean the silence of a project that’s been running for three weeks and you still can’t point to a single visible thing it has produced. The kind of silence where, if your manager stopped you in the hallway and asked “how’s that migration going?”, you’d say “fine” because saying anything more accurate would require explaining things you haven’t fully articulated yet — even to yourself.

I watched a talented engineer do this once. Let’s call her Mia. She’d been handed a significant refactor: six months of accumulated technical debt in a set of dbt models that sat at the centre of our revenue reporting. The kind of work that is genuinely unglamorous, invisible by nature, and deeply important. She approached it the way a lot of good engineers approach hard problems — she went quiet and started digging.

For three weeks, she showed up, she worked hard, and she said almost nothing. Her pull requests were sparse. Her standups were “still investigating.” Her Teams messages were infrequent. She wasn’t slacking off. She was grinding through some of the most complex lineage work I’d ever seen someone untangle.

Then a finance stakeholder messaged me — not her, me. Someone had noticed a spike in our Snowflake credit usage. They were nervous. They’d been burned by silent changes before.

I didn’t have an answer. Mia had one, but nobody knew to ask her.

I went to her and said, “walk me through what you’ve been doing.” What she showed me was extraordinary. She’d traced dependency chains through twelve models. She’d found three bugs that had been silently wrong for months. She’d drafted a migration plan that would cut query time in half. It was genuinely impressive work — and it had been completely invisible for twenty-one days.

That’s the moment I understood something that Jeff Atwood wrote in 2008, in a post called Don’t Go Dark, that I wish someone had made me read before I managed my first data team. Atwood’s essay was aimed at software engineers. It quoted a Microsoft engineering rule that said, in essence: three weeks is going dark. The rule wasn’t about laziness. It wasn’t about bad engineers. It was about a failure mode so common in technical work that it needed a name.

Mia wasn’t lazy. She was going dark.

What Atwood Actually Meant

Atwood’s post is short — shorter than most people expect from the weight it carries. He builds on two sources: a software engineering principle from Jim McCarthy’s Dynamics of Software Development, and an observation from open-source contributor Ben Collins-Sussman. Together they identify a pattern that every engineering lead will recognise the moment they read it.

The pattern is this: programmers don’t want to show work in progress. They want to disappear into the problem, solve it completely, and emerge holding something finished. Collins-Sussman pinned it to fear — not arrogance — arguing that most developers treat coding as an arena for personal heroics rather than an inherently social activity. They’ll share code once it’s polished. They’ll share failures once they’re resolved. But the messy in-between, the half-formed thinking, the models with todos and the branches with four commits and no description? That stays private.

McCarthy’s rule was direct: manage your tasks in short enough increments that you always have a visible deliverable at the end of each one. Three weeks without something to show is too long. The early warning of slippage — one day, discovered this week — is worth ten times more than six months of slippage discovered at deadline.

Atwood’s additional observation is the one that has aged best: agile development structures made going dark mechanically difficult. If you’re running sprints, you’re producing something reviewable every two weeks regardless. The iteration boundary forces surfacing. Joel Spolsky, Atwood’s co-founder at Stack Overflow and another writer in this intellectual lineage, made the same argument about daily builds a few years earlier: a team that builds its software every day has, at minimum, a daily signal that things are either working or not. The build is the forcing function. It converts invisible progress into visible evidence.

Open source projects, Atwood observed, face the highest risk of going dark — because there’s no manager, no sprint ceremony, no daily build process unless someone chooses to implement one. The discipline has to be entirely internal. The only currency that matters on an open source project is visible, reviewable work.

Data engineering, as a discipline, has no such forcing function. Our work doesn’t end at a sprint boundary with a demo. Our pipelines run on schedules that have nothing to do with how long the underlying thinking took. Our models transform quietly, in the background, producing outputs that are visible only when something breaks — or worse, when something breaks subtly and doesn’t look broken at all.

That’s the gap Atwood’s advice never closed. He fixed going dark for software teams. Nobody translated it for us.

Why Data Engineering Goes Dark More Than Anyone Else

There’s a structural reason data work is so prone to invisibility.

When a front-end engineer ships a feature, there’s an artifact: a button, a page, a flow you can click through. When a backend engineer merges a PR, there’s a diff: before and after, visible in the tool, reviewable by anyone. The work and the evidence of work are coupled. They arrive together.

Data work decouples them. Benn Stancil, writing about the data industry, put it well when he described the fuzzy boundary between production and everything else in data teams — there’s often no clean staging/prod line, no clear “shipped” moment, no obvious artifact to point to. A dbt model runs. A dashboard refreshes. A pipeline completes. Did anything change? Did anything improve? Is this better than it was last week? The answers require knowing the context that lives in the engineer’s head, which is precisely the context that isn’t written down.

This gets worse the longer the project. Migrations are the canonical example. A warehouse migration — say, moving from Redshift to Snowflake, or rebuilding a legacy reporting layer in dbt — might take three months of continuous work before a stakeholder sees a single dashboard change. During those three months, the data engineer is doing real, difficult, valuable work: tracing lineage, resolving naming conflicts, negotiating data contracts with upstream producers, writing tests that have never existed before. All of it is invisible. All of it looks like nothing to anyone watching from the outside.

There’s also the problem that Locally Optimistic describes as the distinction between linear and circular projects. Linear projects have a known path and a clear endpoint. Circular projects — most of the genuinely interesting data work — have outcomes that depend heavily on what you discover along the way. The best-laid migration plan doesn’t survive contact with a production schema that hasn’t been properly documented since 2019. Exploration takes time that doesn’t map cleanly to sprint cards or progress percentages.

When you combine those two things — structural invisibility and genuinely unpredictable progress — you get a discipline that almost naturally produces going-dark behaviour, even among engineers who are working diligently and in good faith. Mia wasn’t hiding. The nature of her work was hiding it for her.

The IC Chapter: Staying Visible Without Performing Busyness

Staying visible doesn’t mean performing busyness. It doesn’t mean posting standup updates about tasks you haven’t started. It doesn’t mean narrating every dead end like it’s a breakthrough. That’s noise, and experienced leads learn to tune it out.

What it does mean is producing artifacts — real, durable, reviewable traces of your thinking — on a cadence that keeps your work legible to the people around you. The goal is to make silence meaningful. When there’s nothing from you for three days, the people you work with should know that’s fine because you’ve pre-declared it and pre-committed to what you’ll surface when you come up for air. When you’re grinding through something genuinely hard, the work itself should leave evidence that can be reviewed asynchronously, without requiring a meeting to explain it.

Draft PRs as the lowest-friction visibility tool you already have.

Open a pull request on day one of a significant piece of work. Not when it’s ready for review — when you have a scaffold and an intent. GitHub’s draft PR feature exists exactly for this: your code can’t be merged, reviewers aren’t auto-requested, and everyone in the team can see that the work exists, what it’s targeting, and roughly how it’s going. Update it with a comment as you progress. “Blocked on upstream source delay — will resume Thursday.” “Discovered three extra joins needed — revised estimate to end of week.” “Models passing, running full test suite now.”

This pattern turns the PR into the canonical artifact of the work — a living document that replaces scattered messages, status check-ins, and “hey, how’s that thing going?” conversations. The PR description, written well, tells the story of what changed, why it changed, and how it was tested. Six months from now, when a model breaks at 2am and the engineer who built it is on leave, that description is the thing that saves the on-call.

Commit messages as team communication.

There’s a widely-cited piece by Chris Beams on how to write a good git commit message that’s been floating around engineering circles since 2014. The core argument is simple: the subject line tells you what changed; the body tells you why. A commit message that says fix bug costs the next engineer ten minutes of archaeology. A commit message that says Fix null handling in fct_orders when currency column is missing and then explains that a vendor renamed a field and the defensive .get() was silently returning zero for three months — that commit message is worth something.

The Conventional Commits specification formalises this further: feat:, fix:, refactor:, docs:, chore:, each with optional scope. It sounds bureaucratic until you try to understand a six-month-old change at 9pm during an incident, and someone had the discipline to write refactor(revenue): consolidate gross margin calculation into shared macro in the subject line. Then you understand immediately.

dbt documentation as a visibility discipline.

Every model in a dbt project can carry descriptions at the model level and at the column level, written in the same schema.yml file that defines your tests. Most teams don’t write them. Or they write them at launch and never update them. Or they live in a separate Confluence page that nobody visits.

Here’s the thing: schema.yml descriptions review in the same PR as the code they describe. They can’t silently drift from the implementation because they’re changed in the same commit. A model description that says “Revenue aggregated by market, excluding internal transfers — see ADR-004 for the exclusion logic” is a piece of team communication that persists across every engineer who will ever touch that model.

Go further. Use dbt Exposures to document which dashboards and downstream artifacts depend on your models. A handful of YAML lines, committing which report or ML model consumes each table, converts “will this refactor break anything?” from a Slack archaeology project into something you can run a query against. When a finance stakeholder asks if their dashboard will be affected by a model change, you can show them the dependency chain. That’s a conversation that previously required guesswork. Now it requires a terminal command.

Architecture Decision Records for the decisions people will question.

ADRs — short markdown files committed to docs/adr/ in your repository — document the decisions that will otherwise live only in one engineer’s memory. Why did you choose Snowflake over BigQuery? Why does the revenue model exclude returns that arrived after 30 days? Why is this model incremental rather than table? These questions come up six months later, in a meeting, when the person who made the decision has moved on or forgotten the reasoning. An ADR is three paragraphs and a status field. It takes fifteen minutes to write. It has saved me more than one painful meeting where the alternative was reconstructing a decision from Slack history.

Async standups that actually work.

The daily standup ritual made sense when software teams sat together and needed a quick synchronising pulse. For data teams — often distributed across timezones, frequently doing work that looks the same from the outside regardless of whether it’s going well or terribly — the fifteen-minute morning video call has become a source of collective fiction. Everyone says “still on X, should be done soon” and nobody learns anything.

Async standup tools like Geekbot (for Slack) or equivalent bots for Teams send each person the standup questions by DM on a schedule and post responses to a shared channel. The format that consistently performs best isn’t the classic three questions — it’s a single question: “What would you like the team to know today?” That question has enough latitude to surface a blocker, a discovery, a risk, or a win. It invites signal rather than mandating a ritual. Responses posted to a shared channel are searchable, linkable, and skimmable — a week of them gives a manager more information than five daily standups would.

The written standup format also decouples availability from communication. A data engineer in Melbourne doesn’t need to be online at the same time as her lead in London for him to understand what’s happening. The post is there when he arrives. She’s not interrupted in her maker’s block to attend a meeting she’ll spend eight minutes waiting through before she has sixty seconds to talk.

One discipline worth enforcing: when you name a blocker in a written standup, link to it. Don’t write “blocked on upstream data delay” — write “blocked on upstream raw_orders freshness, opened Issue #247 to track.” The standup post becomes navigable rather than descriptive. Anyone who wants to understand the blocker can follow the link. Anyone who just needs the pulse can read the post and move on.

Data diffs in CI — making the invisible visible before it ships.

The traditional CI check for a dbt project tells you whether models compile and whether your schema-level tests pass. Unique keys, not-null constraints, accepted values — these are important. They catch structural problems before they reach production.

What they don’t catch is when a model that was returning 65.00 starts returning 65.10, or when a currency denomination silently shifts from whole dollars to cents because an upstream source changed its representation without changing its schema. Those changes are invisible to schema tests. They’re invisible to the log output. They’re invisible right up until a finance stakeholder notices that last month’s revenue looks 100× higher than expected in a board deck.

Data diff tooling — whether through Datafold integrated into GitHub CI, or the open-source audit_helper package in dbt, or a homegrown comparison query — runs value-level comparisons between the current state of a model and a reference point. Merge a PR that refactors how gross margin is calculated, and the CI comment shows you a row-level diff of what changed. Every affected row. Every affected column. Before the code reaches production.

This changes the nature of what a PR review means. Instead of reviewing whether the SQL is logically correct — a genuinely hard thing to assess from reading code alone — reviewers can look at the data that would result. A refactor that should be semantically identical shows zero diff. One that inadvertently changes three percent of records in a specific market shows exactly which records and by how much. The artifact created by data diff in CI is one of the most powerful anti-going-dark tools available to a data team, because it makes the consequence of code changes legible to anyone who can read a table — not just the engineers who wrote the SQL.

Async documentation that travels with the work.

One pattern that consistently separates data teams that are legible from those that aren’t: Loom or short screen recordings attached to PR descriptions on significant changes. Not for every fix. Not as a replacement for the written description. But for a model redesign, a migration milestone, or a new data product, a two-minute screen recording walking through the dbt DAG and explaining the decisions is worth more than three paragraphs of prose that nobody will read.

The video doesn’t have to be polished. It has to exist. It converts work that is legible only to the person who did it into work that is legible to a finance analyst, a product manager, or a new data engineer joining in six months. It replaces the meeting where you’d have explained this anyway, except now it’s available asynchronously, can be rewatched, and is linked from the artifact it describes.

The Distributed Team Problem

Remote and distributed data teams face a compounded version of the going-dark problem. The structural invisibility of data work — no natural demo format, no clean shipped/not-shipped boundary — gets worse when your team spans timezones and your primary communication channel is a text-based messaging app with a scroll-back limit.

The teams that navigate this best tend to share a principle: optimize for knowledge retrieval, not knowledge transfer. The goal isn’t to ensure that information passes from person A to person B in a meeting. The goal is to ensure that the information exists in a form that person B can find when they need it, regardless of whether person A is online.

GitLab, operating as a fully remote company since its founding, formalised this as handbook-first communication: any decision, policy, or significant piece of information gets documented before it gets communicated or implemented. Not as bureaucracy, but because documentation creates the retrievable artifact. A verbal decision made in a meeting exists only in the attendees’ memories. A written decision linked from a PR exists for the lifetime of the repository.

For data teams, this translates practically into a few habits. Meeting decisions get a one-paragraph summary posted to a Teams thread, linked from the relevant GitHub issue. A verbal conversation about model design gets a brief written follow-up: “Following on from our chat — going with incremental strategy for fct_orders because full refresh at scale was timing out in staging. ADR drafted at docs/adr/0012-incremental-fct-orders.md.” The conversation happened. Now it has an artifact that travels.

Long-running projects — migrations, platform changes, major refactors — benefit from a written brief at the start that defines the scope, the success criteria, and the milestone cadence. Not a Jira epic with fifty tasks. A two-page document that answers: what are we doing, why are we doing it, how will we know when we’re done, and what are the milestones where we’ll regroup. The brief is the thing you hand to a new stakeholder who asks what’s happening with the platform migration. The brief is the thing you update when the scope changes. The brief is what prevents the three-week silence.

Locally Optimistic describes a useful scripting pattern for circular projects — the kind where the outcomes depend heavily on what you discover along the way. Rather than promising a deliverable, you promise a process: “We will investigate X, Y, and Z over the next two weeks. Those might not give us a conclusive answer, but we’ll regroup to discuss the findings and determine next steps.” That framing is honest. It sets appropriate expectations. And critically, it includes a defined moment of resurfacing — the regroup — which prevents the silence from becoming open-ended.

Julia Evans has written persuasively about the value of maintaining a running document of your own accomplishments — not as a vanity project but as a memory aid. The problem she identifies is real: at review time, you’ve forgotten half of what you did in the last six months, and your manager has forgotten more than that. A brag document is a log of things that mattered: bugs found before they became incidents, migrations completed, models refactored, stakeholders unblocked, junior engineers mentored.

This isn’t about inflating your achievements. It’s about making them visible in a context where the work itself produces no natural evidence trail.

Equally useful: weekly notes. A short message to your team lead, every Friday, that says “this week I did X, I’m picking up Y on Monday, and I want to flag Z as a potential blocker.” It doesn’t have to be long. It doesn’t have to be polished. It just has to exist. Will Larson calls this the drip — communication on cadence, regardless of whether there’s exciting news. The drip is what makes silence informative. When you’ve been sending a Friday note every week for three months and then you stop, the absence is a signal. When you’ve never sent one, silence tells nobody anything.

Tanya Reilly’s glue trap — for senior ICs especially.

If you’re a staff or principal data engineer, you’re probably doing a lot of work that doesn’t look like work from a promotion perspective: reviewing designs, unblocking others, noticing what’s slipping, maintaining relationships with stakeholders. Reilly calls this glue work — the coordination and communication and knowledge-transfer that holds projects together. It’s real work. It’s often the most valuable work on the team. And it’s completely invisible without a deliberate effort to create artifacts.

Design proposals, written-up meeting decisions, documented onboarding pathways, group emails that summarise a decision made in a verbal conversation — these are the artifacts that make glue work visible. Without them, you’re building organisational infrastructure that has no evidence it exists.

The Fear Dimension: What’s Really Happening When Engineers Go Dark

Here’s something I think we don’t talk about honestly enough — and I’m including myself in this.

I’ve been on both sides of it. I’ve gone dark on work I wasn’t confident about, telling myself I’d surface when I had something worth showing. I’ve also been the lead who didn’t notice, or didn’t ask, until a stakeholder beat me to it. Neither version felt like failure at the time. Both were.

Most going-dark behaviour isn’t strategic. It isn’t laziness. It’s fear — specifically, the fear that surfacing in-progress work will reveal that you don’t have it as together as you implied you did.

Collins-Sussman put this precisely in the piece Atwood quoted: developers don’t want their peers to see mistakes or failures. They want to present themselves as infallible. They’re comfortable sharing code once it’s polished. The messy middle, the exploratory branches, the models with big question marks in the comments — that stays private.

This fear is rational in the short term. Showing a half-finished thing invites feedback that can derail you. Showing a broken thing invites questions you can’t answer yet. Staying silent avoids both. The problem is that in the medium term, silence accumulates into something much worse: an engineer who surfaces three weeks into a problem with nothing to show, facing a stakeholder conversation where the first question is “why didn’t you just tell us?”

Paloma Medina’s BICEPS framework identifies six core needs that, when threatened, produce withdrawal: Belonging, Improvement, Choice, Equality, Predictability, Significance. (Yes, it spells BICEPS. Yes, it will appear on a workshop slide near you. The underlying insight is still useful.) Lara Hogan, who has done more than anyone I know of to translate this framework into engineering management practice, makes the observation that going dark is often a flight response to one of these needs being threatened. A project pivot that makes the last three months of work feel wasted threatens Significance — and the engineer who feels that their work no longer matters is likely to disengage before they articulate why. An abrupt change in priorities threatens Predictability and Choice simultaneously. A team reorganisation threatens Belonging.

The leader’s job, when they notice going-dark behaviour, isn’t to increase accountability pressure. It’s to identify which need has been threatened and address it directly. Fournier and others recommend this as the actual content of the 1:1 when something feels off: not “how’s the project going?” but “how are you doing?” The project status comes second. The human state comes first.

Hogan’s Red/Yellow/Green check-in is a lightweight mechanism for this. At the start of a 1:1, an engineer can say “red” without having to explain why — just to signal that something is wrong. The permission to say “red” without an explanation is itself the thing that makes the saying possible. When explanation is required before disclosure, disclosure gets deferred until the explanation is formed — and by then, the problem has grown. Hogan’s observation: only having to say red, and not having to explain the why, is huge. The absence of the explanation requirement is the mechanism. It lowers the cost of disclosure below the threshold where people start editing themselves.

Amy Edmondson’s research on psychological safety is directly relevant here. Her finding, counterintuitively, was that better hospital units didn’t make fewer medication errors — they reported more of them. Her reframe: the better units weren’t more error-prone, they were more willing to surface problems early. The error rate looked higher because the environment was safe enough that people actually said what was happening.

Google’s Project Aristotle, which studied 180 teams over several years, found the same pattern at scale: the single strongest predictor of team performance was whether team members felt safe to take interpersonal risks. Safe to say “I’m stuck.” Safe to say “I found a bug.” Safe to say “I don’t know how long this will take.”

The data engineering implication is uncomfortable but important: if your team never has visible incidents, never surfaces broken assumptions, never flags a model that’s been silently wrong — that’s probably not a sign of excellence. It’s probably a sign that the environment doesn’t feel safe enough to tell the truth. The team that surfaces the most broken pipelines, the most failed tests, the most “we got this wrong” disclosures might be your best team. They’re the ones telling you what’s actually happening.

The “I’ll tell them when it’s fixed” trap is the going-dark failure mode in its purest form. An engineer discovers a problem — a data quality issue, a budget overrun, a model that produces subtly wrong results. Rather than surface it, they decide to fix it first. Each day that passes raises the psychological cost of disclosure. The problem grows. The fix gets harder. Eventually, either the engineer surfaces in a state of exhaustion with a fix and hopes nobody asks hard questions, or a stakeholder finds the problem externally — at which point the disclosure isn’t just about the bug. It’s about the three weeks of silence.

Fixing the number is not the same as fixing what the number did to people who trusted it.

The Lead Chapter: Building Conditions Where Your Team Doesn’t Go Dark

If you lead a data team, the going-dark problem is partly yours to own — not because your engineers are your responsibility in a parental sense, but because the conditions that make going dark feel safe are conditions you set.

Camille Fournier’s framing, in The Manager’s Path, is that 1:1s are like oil changes: if you skip them, plan to get stranded. The 1:1 is the primary diagnostic for whether someone is going dark, and it works only if it creates genuine psychological safety — which means it has to be something other than a status update. If every 1:1 is “what are you working on this week?”, you’ll get accurate project status and zero signal about whether someone is actually stuck, scared, or burning out.

Lara Hogan recommends opening the relationship with simple questions: what makes you grumpy? How will I know when you’re grumpy? What do you need from me when you’re struggling? These aren’t soft questions — they’re diagnostic instruments. They establish that the 1:1 is a place where the real state of things can be named.

Her Red/Yellow/Green check-in is a lightweight version of the same tool. At the start of a 1:1, an engineer can say “red” without having to explain why — just to signal that something is wrong. The permission to say “red” without an explanation is itself the thing that makes the saying possible. When explanation is required before disclosure, disclosure gets deferred until the explanation is formed — and by then, the problem has grown.

Will Larson frames visibility for leads as a mechanical practice rather than an art form. The goal isn’t to inspire your team to communicate — it’s to build systems that make communication the path of least resistance. A weekly update template, committed to as a team norm. An async standup channel where the format is so lightweight that posting takes two minutes. A retrospective cadence where surfacing what went wrong is the expected behaviour, not the exception.

Warning signs to watch for.

Going dark rarely announces itself. The early signals are easy to miss.

Standups that shift from specific to vague — “still on X” across multiple days with no detail — are an indicator. PRs that open in draft and stay there, accumulating commits but never progressing to review. Estimates that consistently slip by a day or two, week after week, without the engineer naming the slippage explicitly. A previously engaged person who starts attending meetings on camera-off, contributing less, leaving threads unresponded to.

The most useful signal is often external. When a stakeholder or adjacent team member asks you what’s happening with a project before the engineer working on it has raised anything — that’s the canary. The information is flowing around the engineer rather than through them. That’s a conversation to have in the next 1:1, and it’s worth naming directly: “I heard from finance about the revenue models. Tell me where things actually are.”

Deep work is not going dark — and the distinction matters.

There’s a real risk of over-correcting. Data engineering requires extended, cognitively-demanding focus. Paul Graham’s framing of the maker’s schedule versus the manager’s schedule is directly applicable: a data engineer tracing lineage through a DAG, rebuilding a model from first principles, or debugging an intermittent pipeline failure needs hours of uninterrupted context, not a constant stream of check-in messages.

Deep work and going dark are not the same thing. The difference is legibility.

Deep work has a pre-declared output and timebox. Going dark does not. Deep work is legible at a coarse grain — “I’m heads-down on X until Thursday, update Friday” — even when it’s illegible at a fine grain. Going dark is illegible at every grain.

The mechanism that makes the distinction: the cadence contract. Before going heads-down, the engineer declares: here’s what I’m working on, here’s when you’ll hear from me, here’s what would make me surface early. “If I’m stuck for more than a day, I’ll tell you. If I’m on track, Friday update. Ping me in #urgent if there’s a fire.” With that contract in place, silence during the week means “on track.” Without it, silence means nothing — or worse, triggers anxiety that prompts the exact interruptions that fragment deep work.

The goal is to make silence meaningful. That requires prior investment in communication, not more communication during the work itself.

The Visible Data Team Playbook

There are a handful of practices that the best data teams I’ve seen use consistently. None of them are revolutionary. Most of them are embarrassingly simple. They work because they’re habitual, not because they’re clever.

Weekly insight posts.

A practice described from data teams at companies like Netlify involves a dedicated Teams channel where each analyst posts one insight per week — not a status update, not a task list, but one thing they found interesting in the data. The format is designed to answer five questions: what am I looking at, why should you care, what caught my eye, where can you learn more, and who do you ask if you have questions.

The explicit benefit isn’t just visibility — it’s demonstrating the range of what the data team does. Stakeholders who only see the data team when something breaks, or when they submit a request, have no model of what the team actually knows. A weekly insight post gives them one. Over time, it becomes the evidence base for why the team should be resourced, why data work is complex, why the business should care about data quality. It’s not reporting. It’s positioning.

Parity dashboards during migrations.

One of the most effective anti-going-dark tools for long migrations is a parity dashboard: a side-by-side view of critical metrics computed in the old and new systems simultaneously. Before you’ve finished the migration, you build the parallel calculation and make it visible. Divergence between the two numbers becomes the artifact of progress — something demoable, something reviewable, something that gives a stakeholder a concrete thing to look at even when the underlying models are still being rebuilt.

The parity dashboard also builds confidence in a way that purely technical communication can’t. When a finance stakeholder sees that their ARR number matches between the legacy and new system for three weeks running, they start to trust the migration in a way that no amount of “we’ve tested this thoroughly” can produce. Evidence beats assertion.

Incident communication as a team norm.

The instinct to fix a data quality issue quietly and move on is almost universal among data engineers. The impulse is understandable — nobody wants to be the person who caused the problem, and surfacing it draws attention to the failure. But the silent fix has a consistent failure mode: it gets discovered later, retrospectively, in a context where the silence becomes the story rather than the fix.

Monte Carlo’s research showed that the data teams with the highest stakeholder trust tend to over-communicate around incidents — sending post-mortems after significant issues, flagging discoveries before fixes are complete, maintaining a clear status trail from “investigating” to “resolved.” What one team described as transformative wasn’t their mean time to resolution. It was the shift from reactive firefighting to proactive communication that demonstrated awareness and control. The communication is the product, as much as the fix.

A simple practice: any time you discover a data quality issue, the first act is naming it in writing — a Teams message, a GitHub issue, an incident channel post — before you start fixing it. Not to perform distress, but to start the paper trail. “Found an issue with the fct_orders null handling — investigating now, will update by EOD.” That message, and the chain of messages that follow it, is what turns an incident into evidence of a well-run team rather than evidence of a careless one.

ADRs for the decisions people will question.

The architecture decisions that feel obvious when you make them feel arbitrary when someone encounters them six months later without context. Why incremental and not table? Why Snowflake and not BigQuery? Why is revenue calculated at invoice date rather than payment receipt date? These questions have answers. The answers live in someone’s head, or in a Confluence page that nobody linked to the model.

ADRs — short markdown files in docs/adr/, committed to the repository — are the durable artifact of those decisions. They don’t need to be long. They need a title, a status, a description of the context, the decision, and the consequences. Once written and committed, they’re part of the repository’s history. They review in PRs. They can be linked from model descriptions. They don’t disappear when someone leaves the team.

dbt Exposures for impact legibility.

If you’re using dbt and you haven’t adopted Exposures, this is worth doing this week. Exposures are YAML-defined references to downstream artifacts — dashboards, ML models, reverse-ETL syncs — that extend the dbt DAG past the warehouse. A few lines in schema.yml that say “this model feeds the Weekly Revenue dashboard” means that when someone runs dbt ls --select +exposure:weekly_revenue, they get the complete list of upstream models that would need to change to affect that dashboard.

For a migration, this means you can answer “will this change break the CEO’s dashboard?” without archaeology. For a refactor, it means PR reviewers have a concrete list of stakeholders to notify. For a data quality incident, it means you know who to tell before they find out themselves.

Going Back to Mia

After my conversation with Mia — the one where she walked me through three weeks of extraordinary, invisible work — we had a more direct conversation about why none of it had been visible.

Her answer was the one Collins-Sussman identified in 2008: she didn’t want to show something until it was right. She was worried about raising a false alarm. She was concerned that surfacing work in progress would invite questions she couldn’t answer yet. There was also something she said that stuck with me: “I thought if I kept my head down long enough, I’d come up with something worth showing.”

All of those concerns were reasonable. None of them were wrong, exactly. What she’d underestimated was the cost of the silence — not to herself, but to the people around her. The finance stakeholder who’d gone to me with questions would have gone to Mia first if they’d had any signal that the work was in progress and legible. My not knowing, in a week where I’d been asked directly by leadership about the revenue model timeline, was a problem I hadn’t been equipped to solve.

What I should have done differently, and what I’ve done differently since: established the cadence contract at the start of the project, not after the silence had already built. Agreed, before the first commit, what “on track” communication would look like — what the update cadence would be, what would trigger an early surface, and what silence during a declared heads-down period meant. With that contract in place, three weeks of quiet is three weeks of “on track, as agreed.” Without it, three weeks of quiet is three weeks of “nobody knows what’s happening.”

The artifact we built together after that conversation was simple. For the remaining six weeks of the migration, Mia opened a draft PR on Monday morning with a brief plan for the week. She posted a comment each Friday with what had been done, what was pending, and any surprises. She added a parity dashboard that the finance lead could check at any time. She wrote three ADRs for the decisions that had taken the most deliberation.

The work didn’t change. The visibility changed. And the experience of the migration — for Mia, for the stakeholders, for me — changed entirely.

At the end of the project, the finance lead sent a message that I still think about. She said: “I’ve never felt so informed during a data migration. Usually I find out things are done when the thing I was waiting for just starts working.”

Mia had gone dark for three weeks. For the remaining six, she hadn’t. The difference wasn’t effort. It was artifacts.

The Closing Thought

Jeff Atwood closed his post with something direct: don’t go dark. Don’t be the developer in the room who hides their code until it’s done. Sharing work in progress feels riskier than it is. The feedback and connection it produces are worth more than the protection the silence provides.

He was right in 2008. The principle has only compounded since, as data teams have become more distributed, more remote, and more responsible for the kind of long-horizon work that naturally produces invisibility.

The tradeoff is real, though. Better visibility takes time that could go to the work itself. An ADR written at the end of a hard week is time you’re not spending fixing the problem. A parity dashboard adds engineering overhead before the migration is even done. Writing commit messages properly requires discipline you don’t always have at 5pm on a Friday when you just want to push and go home.

The case I’m making isn’t that artifacts are free. It’s that the alternative — silence — costs more than you can see while you’re inside it. The cost lands on the stakeholder who went to your manager instead of you. On the on-call engineer at 2am who has nothing to go on. On the project that looked fine until it suddenly didn’t.

Amy Edmondson’s hospital study keeps coming back to me. She’d expected to find that better units made fewer errors. Instead she found that they reported more. Her reframe: maybe the better units don’t have fewer problems. Maybe they’re just more willing to say so.

The best data teams are the same. Not the ones with the fewest broken pipelines — the ones where a broken pipeline doesn’t stay quiet for long.

Three weeks is going dark. One good artifact is enough. The rest follows from that.

The Broken Window in Your Data Pipeline

Chris Hillman — Sat, 09 May 2026 09:00:00 +1000

There’s a particular kind of data problem that doesn’t announce itself. It accumulates.

We were receiving Salesforce data through delta extraction — sensible in theory, because full snapshots can run to hundreds of terabytes and less than 1% of records change on any given day. The problem is that deltas require someone to know what “changed” means. In Salesforce, that’s less obvious than it sounds. Watch a last_modified column and you’ll miss objects that get updated when a related object changes, without their own timestamp reflecting it.

Over time: drift. Orphan records. Data that looks current because the record exists, but isn’t. The fix was documented — run a full snapshot periodically to correct the accumulated drift, painful as that was — and everyone with any context on the system knew it.

What happened was roughly this: the drift accumulated silently until something downstream looked wrong. An investigation traced it back to the delta logic. The full snapshot was run. The problem was resolved. Notes were written. The workaround was filed away.

And then, about twelve months later, the same conversation happened again.

What made that moment stick wasn’t the technical failure. It was the recognition that the window had been broken for a while. Everybody who walked past it knew it was broken. We’d even put a note next to it.

I’ve been that person — the one who wrote the documentation, filed the Jira ticket, and moved on. Which is probably why I remember the twelve-month cycle so clearly. And why I started paying attention to the pattern of it, across teams and companies, long after that particular incident.

We just hadn’t fixed it.

Bear with me here, because what I’m about to describe starts with an abandoned car in the Bronx in 1969 — and I promise it ends somewhere relevant to your dbt models.

A broken window in the Bronx, a broken window in your warehouse

A Stanford psychologist named Philip Zimbardo ran a strange experiment. He abandoned two cars — one in the Bronx, one in Palo Alto — and watched what happened.

The Bronx car was stripped within twenty-four hours. Within three days it was completely gutted. The Palo Alto car sat untouched for a week — until Zimbardo himself walked up with a sledgehammer and broke a window. Within hours, it had been stripped too.

Same car. Different signal.

Thirteen years later, criminologists James Q. Wilson and George Kelling built a theory on that experiment. If a window in a building is broken and left unrepaired, they argued, all the rest will soon follow. Not because there’s a particular breed of window-breaker lurking around, but because an unrepaired window sends a message: nobody here cares. And once that signal is broadcast, breaking more windows costs nothing.

The insight was semiotic, not structural. It wasn’t about windows. It was about what an unrepaired window communicates about the norms of a place.

The theory eventually found its way into software. Andrew Hunt and David Thomas put it into The Pragmatic Programmer almost verbatim: don’t leave broken windows — bad designs, wrong decisions, poor code — unrepaired. They’d watched clean systems deteriorate quickly once windows started breaking. The mechanism was the same. A developer looking at a messy codebase thinks: if someone else got away with being careless, maybe I can too. The norm shifts. The entropy accelerates.

Researchers at Empirical Software Engineering ran a controlled experiment — twenty-nine developers, codebases seeded with either high or low technical-debt density — and found exactly what Hunt and Thomas had intuited. Pre-existing debt measurably caused developers to introduce more debt. Non-descriptive variable names. Duplicated logic instead of reuse. Additional code smells. The broken window was statistically contagious.

It’s a compelling idea, and it maps well to software. But it doesn’t map perfectly to data engineering. And the gap between “maps well” and “maps perfectly” is where teams get into serious trouble.

The thing that’s different about data

When a broken window appears in a codebase, it’s local. A messy function, an undocumented module, a class that’s doing five things at once — these are ugly, they invite imitation, and they slow future development. But they stay where they are. They don’t go anywhere.

A broken window in a data pipeline doesn’t stay where it is.

It travels.

That’s the thing nobody really talks about when they apply broken-windows thinking to data work. In a pipeline, everything is connected. A schema drift in a source table doesn’t just make that table annoying to work with — it silently corrupts every model downstream that touches that field. Which means it corrupts every dashboard that uses those models. Which means it corrupts the metrics those dashboards expose. Which means it corrupts the business decisions made from those metrics.

The broken window is in row one. By the time someone notices, the damage is in the boardroom.

And unlike software, where the damage is visible — a stacktrace, a failing build, a crash — data damage is often invisible. The pipeline still runs. The dashboard still renders. The report still reconciles. The numbers just happen to be wrong, quietly, for reasons nobody can easily trace.

Software bugs produce noise. Bad data produces silence.

In software, the broken window signals disorder and invites imitation. In data pipelines, it does all of that and it propagates at machine speed through every downstream system that trusts the upstream to be clean. By the time the propagation is discovered, it’s usually been underway for a while.

This is the propagation problem. It’s not just that bad data begets more bad data. It’s that one bad window can contaminate an entire watershed.

What a contaminated watershed actually looks like

In May 2022, Unity Software — the game engine company — disclosed something remarkable in their SEC earnings filing. Their Audience Pinpointer ML model, the system that powered their ad targeting business, had ingested bad data from a large customer. That data had corrupted their training set.

CEO John Riccitiello on the earnings call: “we lost the value of a portion of our training data due in part to us ingesting bad data from a large customer.”

The bad data didn’t just produce one wrong prediction. It poisoned the model weights. Every subsequent retrain was built on a contaminated foundation. The model had to be taken offline, the bad data removed, and training restarted from scratch. The estimated impact was $110 million in revenue for 2022. The stock dropped 37% in a single day. Market cap losses in the billions.

The broken window wasn’t in Unity’s systems — it was in data they were ingesting. Once it crossed the boundary into their training pipeline, it propagated in the only direction data knows: forward and downstream, embedding itself into every layer of the system that touched it.

Riccitiello’s pledge after the fact was almost poignant: “We are deploying monitoring, alerting and recovery systems and processes to promptly mitigate future events.” The observability came after the disaster. The window had already broken every other window in the building.

These are the normal failure mode of connected data systems. The propagation isn’t a bug in the design — it’s inherent to the architecture. Data flows in one direction. Trust flows with it. The moment you have a pipeline, you have propagation risk.

The question isn’t whether your broken windows will propagate. It’s how far they’ll travel before anyone notices.

The tools we built to tolerate this

Modern data tooling has given us genuinely sophisticated ways to formally accept broken windows.

dbt — the transformation tool most data engineering teams live inside — has a severity configuration on data tests. You can set a test to warn instead of error. The test runs, detects a problem, and… doesn’t fail the pipeline. It records the warning and moves on.

The documentation is cheerful about it: “Maybe 1 duplicate record can count as a warning, but 10 duplicate records should count as an error.”

In principle, this is sensible. In practice, the warn threshold becomes a Schelling point. Teams configure tests to warn to avoid breaking CI. The warnings accumulate. And then — almost invariably — the warnings become wallpaper. (Go check your own test results right now. Count how many have been in warn state for more than two weeks. I’ll wait.)

dbt also has a feature called store_failures that saves failed records to an audit schema table. The test is doing its job. The failures are being recorded. They overwrite the previous run’s failures on the next run. If nobody is actively querying that audit table — and almost nobody is — the failures exist only to make the test feel like it’s being taken seriously.

It’s a passive graveyard. The window is monitored. Nobody fixes the glass.

Airflow has an equivalent pattern. The soft_fail parameter on sensors means that if an exception is raised — the source system is down, the file hasn’t arrived — the task is marked as skipped rather than failed. Downstream tasks, depending on their trigger rules, may also skip. An entire branch of your DAG quietly collapses to a skipped state, which most pipelines treat as benign, and your stakeholders get a dashboard with no data in it instead of an error message.

Retries do something similar. A flaky source that fails 30% of the time gets retries=3 configured. The task eventually succeeds on the third attempt. The 30% failure rate never surfaces as an anomaly in any meaningful way. Until the source dies entirely, at which point the symptom everyone responds to is not “this has been flaky for six months” but “this suddenly started failing today.”

None of this is the fault of the tools. dbt and Airflow are doing what they’re designed to do. The issue is that the default ergonomics of both make tolerating failure significantly easier than stopping propagation. “Don’t break the build” is a more convenient goal than “don’t ship broken data downstream.” The tools give you excellent knobs for the former and adequate knobs for the latter.

Chad Sanderson — formerly at Convoy, now building in the data contracts space — has a name for what this produces over time. He calls it the POSIWID principle: the Purpose Of a System Is What It Does. If your data pipelines are consistently producing low-quality data, and your team consistently tolerates that, then whatever you think the purpose of your data platform is, its actual purpose is to enable teams to move fast, ship without accountability, and tolerate breakages.

The broken windows aren’t exceptions. They’re the product.

We’ve built monitoring. Why isn’t it working?

Data observability emerged as a formal discipline. The concept — largely popularised by Barr Moses at Monte Carlo — drew directly from the software world’s SRE and distributed systems monitoring practices: freshness, distribution, volume, schema, lineage. Five pillars. Automated anomaly detection. Alerts in Slack when something looks wrong.

The framing was right. The problem is what happened next.

Monte Carlo’s own telemetry — across millions of monitored tables — shows that alert engagement drops 15% when a Slack channel exceeds 50 alerts per week, and a further 20% past 100 per week. The data observability system has a data quality problem: the signal-to-noise ratio degrades, and so does the human response.

This isn’t specific to data. The clinical literature on alarm fatigue is sobering. ICU environments average more than 150 alarms per bed per day. Studies consistently find that 72% to 99% of those alarms are non-actionable. The clinical community’s response to this was institutional — the Joint Commission made alarm safety a National Patient Safety Goal in 2014 — because they recognised that a monitoring system that produces more noise than signal doesn’t just fail to help. It actively worsens outcomes by training humans to stop responding.

Cybersecurity teams face the same thing. Security operations centres receive thousands of alerts daily; most go unaddressed, not because analysts are lazy, but because the ratio of genuine signals to false positives has degraded to the point where sustained attention is cognitively impossible.

The Google SRE book is direct on this: “Every page should be actionable. If a page merely merits a robotic response, it shouldn’t be a page.”

In data engineering, the equivalent of a “robotic response” is the acknowledged-and-unresolved alert. The monitor that fires every Tuesday morning, gets a thumbs up in Teams, gets added to the “known issues” document, and never gets fixed. The window is now monitored. The fact that it’s monitored makes the team feel responsible. The window stays broken.

There’s a term for this: observability theatre. The infrastructure of visibility exists. The dashboards are green. Nobody’s actually looking at the glass.

This is where the broken windows metaphor earns its keep most fully. Wilson and Kelling weren’t saying that disorder is bad because it looks bad. They were saying that an unrepaired broken window sends a signal that no one cares — and that signal is the actual mechanism of decay. Monitoring a broken window without repairing it sends exactly the same signal. Possibly a worse one, because now everyone knows the problem is being tracked and nobody’s acting on it. The norm becomes: acknowledged problems are not necessarily fixed problems.

The stakeholder sees it differently

Here is the asymmetry that data teams consistently underestimate.

When an engineer looks at a broken window in a data pipeline — a column with a known null rate, a test that’s been in warn state for three weeks, a DAG that soft-fails every second run — they see technical debt. A problem to eventually be addressed. Something that exists on a spectrum of severity, probably not at the top of the list.

When a business stakeholder encounters the downstream consequence of that broken window — a dashboard that contradicts a number they just presented to the CFO, a metric that moved in an unexplained direction, a report that doesn’t reconcile with another report — they don’t think “technical debt.” They think: can I trust the data team?

Benn Stancil — one of the clearest thinkers writing about this — put it well: “Trust is built, and blown up, by the outputs — and specifically, the consistency of those outputs.” He uses a vivid image that I keep coming back to: “Data and the dashboards that display it create a shared sense of reality. Looking at two dashboards that don’t match is like looking out two adjacent windows and not seeing the same thing.”

That’s the experience of broken-window propagation from the stakeholder’s side. Two adjacent windows. Different views. A reality that doesn’t cohere.

Monte Carlo’s 2023 State of Data Quality research put a number to the trust inversion that most data engineers already sense. In 2022, 47% of respondents reported that business stakeholders identified data issues first “all or most of the time” — more often than the data team itself. By 2023, that figure had risen to 74%. (Three in four. Let that land.)

Think about what that means. In three out of four data incidents, the people who rely on the data found the problem before the people who built it. The pipeline runs, the test passes, the dashboard renders, and somewhere downstream a business analyst is staring at a number that doesn’t look right and is about to send a message that starts with: “Quick question about this figure…”

The damage isn’t technical. It’s relational. And it compounds in a way that’s harder to reverse than any schema migration.

Thomas Redman — who has spent decades studying data quality — made this point in Harvard Business Review: “When data are unreliable, managers quickly lose faith in them and fall back on their intuition to make decisions.” Once that happens, you haven’t just produced bad data. You’ve trained decision-makers to ignore good data too, because they can no longer distinguish between the two.

The broken window didn’t just propagate through the pipeline. It propagated into the culture.

Nobody owns the broken window

There’s a specific pattern worth naming, because it’s where most data teams I’ve worked with have the most unacknowledged broken windows: the orphaned asset.

The pipeline that was built by an engineer who left eighteen months ago. The table in the warehouse that exists in the data catalogue but whose owner field says “Unknown” or, worse, the name of someone who’s no longer at the company. The dbt model that runs in production, that feeds two dashboards, that nobody on the current team can fully explain.

These aren’t broken in any obvious sense. They run. They load. The tests pass, or they’re set to warn. But they’re broken windows in the original sense: they signal that nobody cares. And because nobody owns them, nobody’s in a position to repair them even when something goes wrong.

What typically happens is this: a new engineer joins the team. They need to understand how the data flows. They look at the catalogue. They look at the undocumented table. They look at the model that references it. They look at the Jira ticket from fourteen months ago that says “Known issue — downstream teams aware.” And they make a rational decision: don’t touch it, build around it, replicate it if necessary.

The broken window has now inspired a new window. Same mechanism as the Bronx car. Different materials.

The organisational research on this is unambiguous. Ron Westrum’s typology of organisational cultures — validated empirically by the DORA research programme across thousands of software teams — found that information flow predicts safety and performance more reliably than almost any structural variable. In pathological cultures, information is hoarded or withheld for political reasons. In generative cultures, information flows freely, failures are shared, and problems get fixed because surfacing problems is rewarded rather than penalised.

Amy Edmondson’s research on psychological safety adds the other half: 85% of employees have withheld important information from their manager due to fear of speaking up. In data teams, this looks like: the junior engineer who noticed the null rate had been climbing for two weeks and didn’t raise it because they weren’t sure it was their call to make. The analyst who suspected the metric definition had drifted but didn’t want to slow down the dashboard delivery. The data engineer who knew the pipeline was flaky but figured someone more senior would have noticed if it really mattered.

The broken window gets left unrepaired not because nobody sees it, but because the culture hasn’t made repair feel safe or worthwhile.

Edmondson on this: “If there’s no bad news, remind yourself: It’s not that it’s not there. It’s that you’re not hearing about it.”

The original theory’s mistake — and ours

Before we get to what to actually do about this, it’s worth pausing on where the original theory went wrong. Because data engineering is at risk of making exactly the same mistake.

When Wilson and Kelling published their 1982 Atlantic article, the theory was nuanced. It was about signals and norms. It was explicitly a theory about community — about how residents and police could work together to maintain shared standards in public spaces. It was, in Kelling’s own framing, a theory of collective efficacy.

What cities did with it was zero tolerance. Mass arrests for minor offences. Stop-and-frisk. 685,000 stops in New York City in a single year, more than 85% of them finding nothing at all.

Kelling’s reaction, when he learned how comprehensively his theory had been misapplied: “Oh, shit.”

The software world made a version of the same mistake when it imported broken windows thinking. “Don’t live with broken windows” became, for some teams, a linting rule for everything, a zero-tolerance policy for code smells, a culture of perfectionism that burned people out and produced beautiful codebases that never shipped.

Adam Tornhill’s research at CodeScene offers the corrective: not all broken windows matter equally. In a 400,000-line codebase with 89 developers, his analysis found that 4% of the code was responsible for 72% of the defects. The broken windows that needed fixing weren’t distributed evenly. They clustered in hotspots — files that were both frequently changed and highly complex. Fix the hotspots. Let the cold, ugly, stable corner of the codebase sit.

The principle for data engineering is the same. Zero-tolerance data quality enforcement — failing every pipeline on every test at severity error — produces fragile systems and tired teams. It’s the data equivalent of arresting everyone for jaywalking. The signal gets lost in the noise.

What the original theory was actually pointing at was something more like: maintain the norm. Make it visible that people care. Ensure that the broken windows that matter — the ones that propagate, the ones in the high-traffic, high-trust parts of the system — get fixed promptly and publicly. Not every window. The ones that signal whether anyone’s paying attention.

What collective efficacy looks like in a data team

Sampson, Raudenbush, and Earls — the Chicago sociologists who ran the most rigorous empirical test of broken windows theory — found that the variable that actually predicted neighbourhood safety wasn’t the presence or absence of disorder. It was collective efficacy: the combination of social cohesion and shared willingness to intervene. Neighbours who knew each other, trusted each other, and were willing to act on behalf of each other’s wellbeing.

The policy prescription that emerges from their work is very different from zero tolerance. It’s community investment. Shared ownership. Making the costs of non-intervention visible.

The equivalent in data engineering starts with one thing, and if you do nothing else in this list, do this one:

Make propagation visible before anything else.

Column-level data lineage — now available in most modern data observability platforms and increasingly in the open-source OpenLineage standard — lets you answer the question: if this column is wrong, what does it break? That question should be answerable in seconds, not hours. Teams that can visualise propagation chains respond to broken windows faster because they can see the radius of the damage before they decide whether to act.

This matters beyond incident response. When you can show an engineer that the null column they’re tolerating in a staging model feeds seven downstream gold-layer tables, three dashboards, and a Snowflake share that two other teams consume — the calculus on whether to fix it changes. The broken window stops being an abstract code quality concern and becomes a blast radius. That’s a much more compelling argument for repair than “we should improve our data quality culture.”

Treat ownership as load-bearing infrastructure, not housekeeping.

Every pipeline, every table, every model should have a named owner. Not a team. A person. The Jira ticket that says “Known issue — downstream teams aware” with no assigned owner is the data equivalent of a broken window with an orange cone next to it. The cone acknowledges the hazard. Nobody’s fixing the glass.

The counterargument is always resourcing — “we don’t have time to own everything properly.” That’s true, and worth taking seriously. But the right response isn’t to pretend you own things you don’t. It’s to make orphaned assets visible and have an honest conversation about whether the organisation can afford to run production pipelines with no accountable maintainer. Most of the time, when the question is asked that directly, the answer is no.

Calibrate your tolerance patterns deliberately — and actually revisit them.

The dbt severity: warn setting and Airflow’s soft_fail are legitimate tools when they’re conscious decisions: “this condition is a signal worth tracking but not a pipeline-stopper, and here’s the threshold at which that changes.” The problem is that almost nobody uses them that way. They’re the path of least resistance to avoid a broken CI build — and six months later you audit your test results and discover forty tests set to warn that haven’t been at zero failures since the day they were written.

Treat warn-severity tests the way you treat Jira tickets that never get triaged. Set a review cadence. If a test has been consistently warning for more than two weeks without an associated investigation, it’s either a bug that needs fixing or a threshold that needs changing. “Known issue” is not a status. It’s an admission.

Fix alert fatigue before it hollows out your monitoring culture.

The Teams channel where every data quality alert lands is the digital equivalent of a neighbourhood where every broken window gets photographed and logged and nobody ever fixes one. The log is evidence that someone noticed. It’s not evidence that anyone will act.

Set alert thresholds that produce actionable signals. The Google SRE principle applies directly: if an alert merits a robotic response, it shouldn’t be an alert. If your first instinct on seeing a particular monitor fire is to click acknowledge and move on, that monitor is producing noise, not signal. Change it or delete it. Grouping alerts by lineage — “these five monitors fired because of one upstream schema change” rather than five separate pings — reduces volume while making propagation visible at the moment of failure, which is exactly when you want it.

Make broken windows visible to leadership, because right now the cost is invisible.

Data quality work is famously hard to demonstrate. Pipelines that run cleanly leave no artefact. A well-maintained model with a 0% null rate looks identical to a freshly built one. There’s no natural demo format for “nothing bad happened this week.”

This invisibility is part of why broken windows accumulate. The cost of prevention is hidden in maintenance time that doesn’t get counted. The cost of failure gets absorbed by analysts who spend their Tuesdays reconciling numbers instead of answering strategic questions, by data engineers who spend their Fridays investigating stakeholder tickets, by business decisions made on incorrect information that can’t be traced back to a specific incident.

Quantify the propagation radius when incidents do occur. How many downstream models were affected? How many stakeholders were exposed to incorrect data, and for how long? What was the resolution effort in hours? Those numbers, tracked consistently, build a case for investment that abstract arguments about data quality never will.

The window was broken before I noticed

What we ended up doing was building the fix into the operating rhythm. A scheduled data drift catch-up: a full snapshot weekly, or monthly, depending on the volatility of the Salesforce object — not to respond to an incident, but to systematically correct the accumulated drift before it became visible to anyone downstream. We stopped waiting for the escalation. We made the repair part of the architecture.

The broken window was still there, technically. The delta logic still had its blind spots around hidden relationships. But we stopped waiting for it to propagate before we addressed it. Some broken windows aren’t things you permanently seal — they’re things you build a maintenance routine around. Knowing that is the fix.

What stayed with me wasn’t the technical solution, which was straightforward enough once we committed to it. What stayed with me was the calendar predictability of the escalation cycle before the fix. The fact that we had a known issue, a known workaround, documentation of both — and still managed to have the same downstream discovery, the same investigation, the same remediation conversation, almost exactly twelve months apart.

The window wasn’t hidden. It was visible in the delta logs if you knew where to look. We just hadn’t made it someone’s job to look, and we hadn’t made repair part of the routine. We’d fixed the glass, filed the paperwork, and assumed the problem was solved. Until the same signal appeared in the same downstream reports, a year later.

What I keep coming back to, though, is something that was true of that experience and is true of most of the data quality failures I’ve seen since: the window wasn’t broken secretly. It wasn’t hidden. It was visible, it was acknowledged, and it was left.

We had the monitoring. We had the test. We had the Jira ticket.

We had, in other words, all the infrastructure of concern. What we didn’t have was the collective agreement that repair mattered — that the propagation radius of that one broken window was large enough, and trust-eroding enough, that it justified stopping what we were doing and fixing the glass.

That’s the thing broken windows theory is actually about, underneath all the criminology and the code smells and the schema drift. It’s about the signal that unrepaired damage sends. Not to the criminals or the developers or the data consumers. To the people who are supposed to care about the system.

When a broken window sits long enough in a data pipeline, it stops being a problem and starts being a norm. The new engineer doesn’t flag it — they work around it. The analyst doesn’t escalate it — they add a caveat to their report. The data engineer doesn’t fix it — they document it.

And somewhere downstream, a business decision gets made on numbers that were broken before anyone thought to check.

The window isn’t just in the pipeline. The window is in the standard you’re willing to keep.

Five Worlds of Data Engineering

Chris Hillman — Sat, 02 May 2026 09:00:00 +1000

You watch a conference talk about implementing data contracts, and nobody mentions that the advice assumes you have multiple teams producing data — which you don’t. You read a post declaring “if you’re still using stored procedures in 2026, you’re doing it wrong,” and the comments erupt. Half the people are nodding along. Half are furious. Both sides are right. They’re just living in different worlds and don’t realise it.

That mismatch — smart people giving each other advice that doesn’t apply — is the thing that almost never gets named. And the reason it doesn’t get named is that most public data engineering discourse is produced by and for one particular world, while pretending to speak for all of them.

I’ve worked across a few of these worlds myself — SQL Server migrations to Teradata, then Teradata to S3, regulatory reporting under APRA and ASIC, a data mesh initiative at scale, and now a university environment that runs closer to a startup than anything I expected. The advice that kept me out of trouble in one would have gotten me fired in the other. That experience is what’s behind this taxonomy.

I think there are five worlds here. Sometimes they intersect. Often they don’t.

World 1: The Modern Analytics Shop

This is the world most people picture when they hear “data engineering.” A team of two to ten engineers at a startup, scale-up, or mid-size company. Cloud-native from day one, or recently migrated. The stack reads like a vendor sponsor list: Snowflake or BigQuery or Databricks, dbt for transformations, Fivetran or Airbyte for ingestion, Looker or Metabase for dashboards. Everything managed. Everything SaaS. GitHub Actions wiring it together.

What matters here is speed. Getting value to stakeholders fast. Shipping a working dashboard before the quarterly review. Iterating on models without a change advisory board. The team is small enough that everyone knows the codebase, and requirements come from a Teams message, not a 40-page specification document.

I find myself in this world more than I expected right now. The current role is an institution that by any measure is large and complex — but the data team is small, the autonomy is real, and the energy genuinely feels like a startup. New tools, fast decisions, the ability to actually make change. It’s a reminder that World 1 isn’t only about company size. It’s about how the team operates.

This is also a great world to work in, especially early in your career. The tooling is mature, the feedback loops are tight, and the problems are tractable.

Here’s the thing, though. This is also the world that roughly 80% of public data engineering content is written for and about. Not because it’s the most common world — it isn’t — but because it’s the most fundable. The vendor-funded conference talks, the sponsored podcasts, the developer advocate blog posts, the LinkedIn hot takes — they overwhelmingly reflect this world. Developer advocates write about the stack their employer sells (this is not a dig — it’s the job). Conference sponsors want talks that showcase their tools in the most flattering light. The technical depth suffers because the incentive is awareness, not education.

None of that is wrong. But it creates a gravitational pull that distorts the whole discourse. When someone writes “the right way to do data engineering,” they almost always mean this world. And if you’re in a different one and don’t recognise the mismatch, you end up feeling like you’re doing it wrong when you’re actually just solving a different problem.

Advice that works here but rarely travels: “Just use dbt.” “Schema-on-read is fine for now.” “You don’t need a data catalog yet.” “Start with a star schema and iterate.”

World 2: The Enterprise Legacy Estate

This is the world of large organisations — banks, insurers, manufacturers, utilities, healthcare systems, government agencies — with data infrastructure that predates the cloud. Often predates the people currently maintaining it. Teams of twenty to two hundred data professionals spread across business units that don’t always talk to each other.

The stack tells a different story: Informatica, SSIS, Ab Initio, Teradata, Oracle, maybe an on-prem Hadoop cluster that someone championed in 2014 and nobody’s had the political capital to decommission. Perhaps some Snowflake or Databricks grafted on top, creating a hybrid that’s more complex than either system alone.

I lived this migration arc twice — once from SQL Server to Teradata, and again from Teradata to S3 and Starburst. Each time, the temptation was to treat the old system as a problem to escape rather than a body of knowledge to decode. Each time, the engineers who did the most damage were the ones who arrived with the new stack and immediately started designing around the old one rather than understanding it first.

What matters here is stability. Not breaking things. Migration plans that span years, not sprints. Governance and lineage aren’t aspirational — they’re audit requirements. And politics, because the data warehouse your predecessor built in 2011 is somebody’s empire (you know the one), and you can’t “just replace it” without navigating a web of organisational power dynamics that no architecture diagram captures.

This is the world where refactoring beats rebuilding every time. That fact table with 200 columns? Those bridge tables nobody understands? The slowly-changing-dimension-within-a-slowly-changing-dimension? They’re not bugs — they’re reality encoded. Every weird modelling choice represents a business rule someone fought to understand. The “clean” data vault remodel will eventually end up with the same complexity, just distributed across more tables with more confusing names.

Advice from World 1 can be actively dangerous here. “Just rewrite it” destroys institutional memory. “Adopt a lakehouse architecture” sounds great until you realise you have 4,000 stored procedures that encode fifteen years of business rules, and nobody documented them. The engineer in this world isn’t slow because they’re behind the curve. They’re careful because the cost of breaking something is measured in regulatory findings and executive phone calls, not a failed CI check.

Advice that works here but rarely travels: “Document everything before you touch it.” “Strangler fig, never big bang.” “Spend more time understanding why it was built this way than planning what to replace it with.” “The weird WHERE clause isn’t a bug — it’s institutional memory.”

World 3: The Product Engine

This is the world where data engineering directly powers an outcome the business delivers. That might be real-time personalisation, recommendation engines, or pricing algorithms. But it also includes building data products that drive campaigns, marketing attribution, and business decisions — cases where the engineer’s output flows into something customers or commercial teams act on directly, rather than landing in a dashboard that an analyst reviews on Monday morning.

The real-time variant of this world has its own stack: Kafka or Kinesis for streaming, Spark or Flink for processing, feature stores for serving machine learning models. Often custom infrastructure because off-the-shelf tools can’t meet the latency requirements.

But the defining characteristic isn’t milliseconds — it’s consequence. If a pipeline breaks in this world, someone notices immediately. A campaign fires with the wrong audience. A pricing decision gets made on stale data. A recommendation engine serves the same product to everyone because the features stopped updating overnight. The engineer is accountable to an outcome, not just a pipeline.

I’ve worked in this space building data products that powered business campaigns and marketing decisions. Not real-time serving in the p99 latency sense, but consequential enough that a broken pipeline meant a broken business process. That accountability changes how you think about testing, monitoring, and what “done” actually means.

The skills that matter here — operational thinking, system design, understanding how your data is consumed downstream — overlap more with software engineering and product thinking than with SQL and dbt. When someone says “data engineering is just SQL and orchestration,” someone in this world quietly closes the tab.

Advice that works here but rarely travels: “Treat pipelines like production services.” “Your tests need to run in CI, not in a notebook.” “Schema evolution is a deployment problem, not a modelling problem.” “The consumer of your data is your customer — know what breaks their day.”

World 4: The Regulated Pipeline

This is the world defined by compliance. In Australia: APRA for prudential standards, ASIC for financial services conduct, AFCA for dispute resolution obligations, and the Privacy Act for anything touching personal information. Internationally: HIPAA, SOX, MiFID, GDPR. The defining characteristic isn’t the technology — it’s that regulatory requirements shape every architectural decision before the first line of code is written.

The stack is whatever passed the security review. Often years behind the cutting edge because new tools need months of compliance evaluation before they’re approved. Data lineage tools aren’t nice-to-haves — they’re audit requirements. Encryption isn’t a best practice — it’s a legal mandate. Access controls aren’t “we should get around to that” — they’re the first thing you build.

What matters is auditability. Can you answer “who accessed what, when, and why” for any record in the system? Can you prove data lineage from source to report? Can you demonstrate that a deletion request was honoured within the legally mandated timeframe? Your data deletion pipeline is as important as your ingestion pipeline, and it needs the same rigour.

I spent significant time in this world — financial services, where regulatory reporting wasn’t a background concern but a core deliverable. APRA and ASIC don’t ask nicely. The audit wasn’t a hypothetical and the regulator’s question list arrived without warning. The engineers I worked alongside weren’t slow because they lacked ambition. They were deliberate because the cost of getting it wrong wasn’t a postmortem — it was a regulatory finding, a remediation programme, and occasionally a front-page story.

The trap for engineers entering this world from World 1 is assuming that governance is bureaucracy. It isn’t. Governance is architecture. The compliance requirements aren’t obstacles to good engineering — they’re constraints that shape what good engineering looks like. The best engineers I’ve worked with in regulated environments don’t fight the constraints. They design systems where compliance is a property of the architecture itself, not a layer bolted on top.

Advice that works here but rarely travels: “Governance is architecture, not bureaucracy.” “If you can’t prove lineage, you can’t ship it.” “The security review IS the sprint.” “Your data deletion pipeline is as important as your data ingestion pipeline.”

World 5: The Internal Platform

This is the world of organisations large enough that the data team has split into “platform” and “domain” functions. The platform team builds the infrastructure, tooling, and self-service capabilities that other data teams consume. Data mesh adopters. Companies with fifty or more data practitioners who realised that a centralised team can’t scale to serve every domain’s needs.

The stack is about abstraction and enablement: Kubernetes, self-hosted Airflow or Dagster or Prefect, internal developer portals, data contracts, schema registries, internal PyPI packages. Often custom abstractions layered on top of cloud services, designed to give domain teams guardrails without bottlenecks.

I experienced this world through ANZx — ANZ’s digital platform initiative — where data mesh concepts were applied at scale. Not as a theoretical framework on a conference slide, but as a real attempt to distribute data ownership across domains while maintaining coherence at the platform level. The challenge wasn’t the technology. It was getting domain teams to think like data product owners rather than data consumers. That shift is harder than any infrastructure problem.

What matters here is adoption. Not how many pipelines you build, but how many pipelines your consumers build without needing your help. Your success metric isn’t “pipelines shipped” — it’s “time to first pipeline for a new domain team.” Developer experience for internal consumers is your product, and if the experience is poor, your consumers will route around you. They’ll spin up their own Snowflake account, write their own ingestion scripts, and create exactly the kind of ungoverned sprawl your platform was supposed to prevent.

If adopting your platform requires a Jira ticket and a two-week wait, you’ve already lost. The shadow pipelines will multiply, and nobody will tell you until the audit.

Advice that works here but rarely travels: “Treat internal teams as customers with choices.” “Self-service is the goal, not centralised delivery.” “Your documentation IS the product.” “Measure adoption, not output.”

So What?

Most things in data engineering are the same no matter which world you’re in. Data quality matters everywhere. Testing matters everywhere. Documentation matters everywhere. Understanding the business context behind the data — that’s universal, and I’d argue it’s the most undervalued skill in the profession.

But not everything transfers. And when somebody tells you about methodology — at a conference, on LinkedIn, in a blog post (including mine) — it’s worth thinking about which world they’re coming from.

Here’s something I’ve noticed when hiring vetran data engineers: the ones who stand out aren’t necessarily the deepest specialists in any one world. They’re the ones who’ve visited enough worlds to know the laws. They don’t have to have lived in each one for years — but they’ve spent enough time in the regulated environment to understand why governance isn’t bureaucracy. Enough time in the enterprise to know that “just rewrite it” destroys things. Enough time in the product engine to feel what accountability to an outcome actually means. That layered experience is what hardens an engineer. It’s what lets them walk into an unfamiliar environment and read it quickly — recognising the constraints, the trade-offs, the things that will matter before anyone tells them.

There’s also something worth saying about what these worlds are not: interchangeable, or combinable. I’ve never seen an organisation that genuinely encompasses all five. You occasionally see attempts — usually during a data mesh initiative — and they tend to produce something more complex than any individual world, with the clarity of none of them. The worlds don’t collapse into a super-universe. They coexist, imperfectly, inside large organisations. A regulated pipeline team and a modern analytics shop can operate within the same company and have almost nothing in common in terms of how they work, what they value, and what good looks like.

The tradeoff is real and worth naming plainly: the content that’s most abundant — the conference talks, the vendor posts, the LinkedIn hot takes — is calibrated for a world that isn’t yours if you’re in World 2, 3, 4, or 5. There’s more of it than ever, and it’s easier than ever to access. What you give up is the ability to consume it passively. Filtering for relevance is now part of the job, and nobody puts that in the job description.

When you read advice — any advice, including everything on this blog — ask yourself which world it’s coming from. If it doesn’t apply to yours, that’s not a failing on your part or theirs. It just means they’re in a different world.

Now you know to notice.

Your Data Platform Costs More Than It Should

Chris Hillman — Sat, 25 Apr 2026 09:00:00 +1100

Let me tell you about the moment I stopped treating cloud costs as someone else’s problem.

We were three months into a Snowflake migration. Everything was humming. Pipelines were green, dashboards were fast, the analytics team was happier than I’d seen them before. I felt good about the work we’d done.

Then finance forwarded me the invoice.

The number wasn’t catastrophic. But it was significantly higher than what we’d budgeted, and when I started digging, I couldn’t explain where most of it was going. I knew we had warehouses running. I knew we had pipelines executing. But I couldn’t tell you which warehouse was responsible for what cost, which pipelines were the expensive ones, or whether the money was well spent. I had built a platform I was proud of — and I had no idea what it actually cost to operate.

That’s the moment that changed how I think about data engineering. Not because of the dollar amount, but because of the realisation underneath it: I had built something I couldn’t explain to the people paying for it. And if I couldn’t explain it, I couldn’t defend it. And if I couldn’t defend it, someone else would make the decisions for me — someone who didn’t understand why the platform mattered.

I’m telling you this because cost management is one of those things that sounds like a finance problem until you experience the consequences firsthand. It’s not about being cheap. It’s about being intentional. It’s about knowing that every credit you spend is buying something valuable — and being able to prove it when someone asks.

The data engineers who understand their costs don’t just save money. They earn trust. They get budget for the projects that matter. They sleep better because they’ve eliminated the waste that eventually becomes someone else’s excuse to cut headcount or freeze hiring.

This article is about building that understanding. Not with a vendor’s optimisation tool or a consultant’s audit — but with the habits, queries, and mental models that let you own your platform’s economics from the inside. Everything here is grounded in Snowflake and AWS, with specific code you can run today.

You can’t optimise what you can’t see

Before you touch a single warehouse configuration, you need to answer one question: where is the money going?

Most teams skip this step. They read a blog post about auto-suspend settings, change a few defaults, and call it optimisation. That’s like going on a diet by switching to diet soda while eating three pizzas a day. The soda wasn’t the problem.

Here’s the query I run first on every Snowflake environment I touch. It tells you which warehouses are consuming the most credits over the last 30 days:

SELECT
 warehouse_name,
 SUM(credits_used) AS total_credits,
 SUM(credits_used) * 3.00 AS estimated_cost_usd, -- adjust your credit price
 COUNT(DISTINCT DATE_TRUNC('day', start_time)) AS active_days,
 ROUND(SUM(credits_used) / COUNT(DISTINCT DATE_TRUNC('day', start_time)), 2) AS credits_per_day
FROM snowflake.account_usage.warehouse_metering_history
WHERE start_time >= DATEADD('day', -30, CURRENT_TIMESTAMP())
GROUP BY warehouse_name
ORDER BY total_credits DESC;

Run that. Right now. I’ll wait.

If you’re like most teams, 70–80% of your credits come from two or three warehouses. That’s your starting point. Not everything — just the expensive stuff.

Now do the same thing for queries. This one finds your top 20 most expensive queries by bytes scanned:

SELECT
 query_id,
 query_text,
 warehouse_name,
 user_name,
 total_elapsed_time / 1000 AS elapsed_seconds,
 bytes_scanned / POWER(1024, 3) AS gb_scanned,
 partitions_scanned,
 partitions_total,
 ROUND(partitions_scanned / NULLIF(partitions_total, 0) * 100, 1) AS pct_partitions_scanned
FROM
 snowflake.account_usage.query_history
WHERE
 start_time >= DATEADD('day', -7, CURRENT_TIMESTAMP())
 AND bytes_scanned > 0
ORDER BY
 bytes_scanned DESC
LIMIT 20;

Pay attention to that pct_partitions_scanned column. If you see queries scanning 90–100% of a table’s partitions, those queries aren’t benefiting from clustering or partition pruning. That’s where the big wins hide.

This 15-minute exercise — top warehouses, top queries — tells you more about your cost profile than any dashboard. It’s the equivalent of checking your bank statement before creating a budget. Obvious in hindsight. Almost nobody does it.

The idle tax is your single biggest waste

Here’s what nobody tells you about Snowflake billing: you pay for compute in 60-second minimums. Every time a warehouse resumes from suspension, you’re billed for at least one minute — regardless of whether the query takes 2 seconds or 58 seconds.

This matters more than it sounds. Picture a BI tool like Metabase or Tableau hitting your warehouse with 20 small metadata queries over 15 minutes. If your warehouse auto-suspends after 5 minutes (Snowflake’s default for many setups), it might suspend and resume multiple times during that window. Each resume triggers another 60-second charge.

Twenty queries that take 3 seconds each? That’s 60 seconds of actual compute. But if the warehouse suspends and resumes 4 times, you’re billed for 240 seconds. A 4x overhead.

The fix is straightforward but requires thought:

-- For transformation warehouses (predictable, bursty workloads)
ALTER WAREHOUSE transform_wh
 SET AUTO_SUSPEND = 60
 AUTO_RESUME = TRUE
 WAREHOUSE_SIZE = 'MEDIUM';

-- For BI/dashboard warehouses (frequent small queries)
ALTER WAREHOUSE bi_wh
 SET AUTO_SUSPEND = 300 -- 5 min keeps it warm between dashboard interactions
 AUTO_RESUME = TRUE
 WAREHOUSE_SIZE = 'SMALL';

-- For ad-hoc/analyst warehouses (unpredictable, intermittent)
ALTER WAREHOUSE adhoc_wh
 SET AUTO_SUSPEND = 60
 AUTO_RESUME = TRUE
 WAREHOUSE_SIZE = 'XSMALL';

The principle: transformation warehouses should suspend aggressively because they run in defined windows with gaps between runs. BI warehouses should stay warm a bit longer because dashboard users generate clusters of queries with short pauses in between. Ad-hoc warehouses should be small and aggressive — analysts can tolerate a 1–2 second resume delay.

Here’s how to find warehouses that are running but idle — the silent cost killer:

SELECT
 warehouse_name,
 SUM(credits_used) AS total_credits,
 SUM(credits_used_compute) AS compute_credits,
 SUM(credits_used_cloud_services) AS cloud_credits,
 ROUND(
 (SUM(credits_used) - SUM(credits_used_compute))
 / NULLIF(SUM(credits_used), 0) * 100, 1
 ) AS pct_idle_cost
FROM
 snowflake.account_usage.warehouse_metering_history
WHERE
 start_time >= DATEADD('day', -30, CURRENT_TIMESTAMP())
GROUP BY
 warehouse_name
HAVING
 pct_idle_cost > 20
ORDER BY
 total_credits DESC;

If pct_idle_cost is above 20% on any warehouse, you’re burning money on idle time. Tighten the auto-suspend, or investigate what’s keeping the warehouse awake between queries.

The most expensive query is the one that scans everything

Snowflake charges you for the compute time your queries consume, and nothing drives compute time like full table scans. The two most common culprits are queries that wrap filter columns in functions, and queries that select more columns than they need.

Here’s what I mean. This query looks innocent:

-- Expensive: function on the filter column disables pruning
SELECT
 order_id, customer_id, total_amount
FROM
 analytics.fct_orders
WHERE
 DATE(order_timestamp) = '2026-03-01';

But it’s quietly terrible. Wrapping order_timestamp in DATE() forces Snowflake to evaluate every row before filtering. The query planner can’t use micro-partition metadata to skip irrelevant partitions.

This version does the same thing, but lets Snowflake prune:

-- Cheap: range filter on raw column enables pruning
SELECT
 order_id, customer_id, total_amount
FROM
 analytics.fct_orders
WHERE
 order_timestamp >= '2026-03-01'
 AND order_timestamp < '2026-03-02';

The difference can be enormous on large tables — I’ve seen this single change reduce bytes scanned by 95% on billion-row fact tables.

The other silent killer is SELECT * in intermediate transformations. In a dbt project, I often find staging models that look like this:

-- stg_orders.sql (before)
SELECT *
FROM
 {{ source('raw', 'orders') }}
WHERE
 _loaded_at > (SELECT MAX(_loaded_at) FROM {{ this }})

That SELECT * pulls every column from the source — including columns nobody downstream ever uses. In a columnar store like Snowflake, you only pay to scan the columns you reference. Trimming to the columns you actually need is free performance:

-- stg_orders.sql (after)
SELECT
 order_id,
 customer_id,
 order_timestamp,
 total_amount,
 status,
 _loaded_at
FROM
 {{ source('raw', 'orders') }}
WHERE
 _loaded_at > (SELECT MAX(_loaded_at) FROM {{ this }})

This isn’t premature optimisation. It’s hygiene. Every SELECT * in your transformation layer is a small tax you pay on every single run.

Full refreshes are the most expensive default in data engineering

If you’re using dbt — and most teams on Snowflake are — the default materialisation is either view or table. Both have the same problem at scale: they rebuild everything, every time.

A table materialisation on a 500-million-row fact table means Snowflake reads, transforms, and writes 500 million rows every run. Even if only 50,000 rows changed since yesterday. That’s a 10,000x overhead.

Switching to incremental materialisation is the single highest-impact cost change most teams can make:

-- fct_orders.sql
{{
 config(
 materialized='incremental',
 unique_key='order_id',
 incremental_strategy='merge',
 on_schema_change='append_new_columns'
 )
}}

SELECT
 order_id,
 customer_id,
 order_timestamp,
 total_amount,
 status,
 updated_at
FROM
 {{ ref('stg_orders') }}

{% if is_incremental() %}
WHERE updated_at > (
 SELECT DATEADD('hour', -3, MAX(updated_at))
 FROM {{ this }}
)
{% endif %}

A few things worth noting here. The DATEADD('hour', -3, ...) creates a 3-hour lookback window. This catches late-arriving data and handles clock skew between source systems. Without it, you’ll miss rows that arrive slightly out of order — and your numbers will silently drift.

The unique_key with incremental_strategy='merge' means Snowflake will update existing rows and insert new ones. This is essential for tables where source records get modified after initial load (order status changes, for example).

The rule of thumb: if a table has more than a million rows and less than 20% changes per run, make it incremental. The cost savings are usually 80–95% on that model’s compute.

But — and this is important — schedule a periodic full refresh to correct any drift. I typically set up a weekly dbt build --full-refresh --select fct_orders via a separate Airflow DAG or GitHub Actions workflow. Belt and suspenders.

The hidden costs your Snowflake dashboard won’t show you

Most teams monitor warehouse credits because that’s what the Snowflake UI makes visible. But there are cost vectors that don’t show up in the obvious places.

Cloud services credits accrue when queries use Snowflake’s coordination layer — query compilation, metadata operations, result set caching. Normally this is covered by a 10% “free” adjustment against your compute credits. But if you have BI tools making thousands of small metadata queries, cloud services can exceed that adjustment and start costing real money.

Find them:

SELECT
 DATE_TRUNC('day', usage_date) AS day,
 SUM(credits_used) AS total_credits,
 SUM(credits_adjustment_cloud_services) AS cloud_services_adjustment,
 CASE
 WHEN SUM(credits_adjustment_cloud_services) < 0
 THEN ABS(SUM(credits_adjustment_cloud_services))
 ELSE 0
 END AS excess_cloud_services_cost
FROM
 snowflake.account_usage.metering_daily_history
WHERE
 usage_date >= DATEADD('day', -30, CURRENT_DATE())
GROUP BY
 day
ORDER BY
 day DESC;

Serverless feature credits — Snowpipe, automatic clustering, materialized view maintenance, search optimisation — all consume credits outside your warehouse billing. They don’t show up in warehouse_metering_history. You need to check separately:

-- Snowpipe costs
SELECT
 pipe_name,
 SUM(credits_used) AS total_credits
FROM
 snowflake.account_usage.pipe_usage_history
WHERE
 start_time >= DATEADD('day', -30, CURRENT_TIMESTAMP())
GROUP BY
 pipe_name
ORDER BY
 total_credits DESC;

-- Automatic clustering costs
SELECT
 table_name,
 SUM(credits_used) AS total_credits
FROM
 snowflake.account_usage.automatic_clustering_history
WHERE
 start_time >= DATEADD('day', -30, CURRENT_TIMESTAMP())
GROUP BY
 table_name
ORDER BY
 total_credits DESC;

On the AWS side, the sneaky costs tend to be data transfer. Moving data between S3 regions, or from S3 out to the internet, adds up quietly. If your Snowflake account is in us-east-1 but your S3 landing zone is in ap-southeast-2, every byte of ingestion carries a cross-region transfer charge. Check your AWS Cost Explorer with the “Data Transfer” service filter — the numbers are often surprising.

You cannot reduce costs you can’t attribute. The simplest and most underused tool in Snowflake is the query tag — a piece of metadata you attach to every query that tells you what generated it.

If you’re using dbt, this takes about five minutes to set up. Create a macro:

-- macros/set_query_tag.sql
{% macro set_query_tag() %}
 {% set query_tag = {
 "dbt_model": model.name,
 "dbt_schema": model.schema,
 "dbt_materialized": model.config.materialized,
 "dbt_invocation_id": invocation_id,
 "environment": target.name
 } %}
 {% do run_query("ALTER SESSION SET QUERY_TAG = '{}'".format(query_tag | tojson)) %}
{% endmacro %}

Then add it as a pre-hook in your dbt_project.yml:

# dbt_project.yml
models:
 your_project:
 +pre-hook: "{{ set_query_tag() }}"

Now every query dbt runs is tagged with the model name, materialisation type, and environment. You can query cost by model:

SELECT
 PARSE_JSON(query_tag):dbt_model::STRING AS model_name,
 PARSE_JSON(query_tag):dbt_materialized::STRING AS materialization,
 COUNT(*) AS query_count,
 SUM(total_elapsed_time) / 1000 AS total_seconds,
 SUM(bytes_scanned) / POWER(1024, 4) AS tb_scanned
FROM
 snowflake.account_usage.query_history
WHERE
 query_tag IS NOT NULL
 AND TRY_PARSE_JSON(query_tag) IS NOT NULL
 AND start_time >= DATEADD('day', -7, CURRENT_TIMESTAMP())
GROUP BY
 model_name, materialization
ORDER BY
 tb_scanned DESC
LIMIT 20;

This is how you find the one dbt model that’s responsible for 40% of your bill. Without tags, it’s a guessing game.

For non-dbt workloads — Airflow tasks, Lambda functions, BI tools — set query tags at the session level in your connection configuration. In Python:

import snowflake.connector
import json

conn = snowflake.connector.connect(
 account='your_account',
 user='your_user',
 password='your_password',
 warehouse='transform_wh',
 session_parameters={
 'QUERY_TAG': json.dumps({
 'pipeline': 'customer_ingestion',
 'task': 'load_raw_customers',
 'environment': 'production'
 })
 }
)

The goal is simple: every query that runs on your platform should be attributable to a team, a pipeline, or a tool. Start with the big consumers and work outward.

Kill the zombies

Every data platform accumulates dead weight. Tables nobody queries. Pipelines that run faithfully every morning, transforming data that no dashboard, no analyst, and no model has touched in months.

These zombies cost you twice: once in compute (the pipeline that refreshes them) and once in storage (the data that sits there). Finding them is straightforward:

-- Tables with zero reads in the last 90 days
SELECT
 t.table_schema,
 t.table_name,
 t.row_count,
 t.bytes / POWER(1024, 3) AS size_gb,
 t.last_altered,
 MAX(ah.query_start_time) AS last_queried
FROM
 snowflake.account_usage.tables t

 LEFT JOIN snowflake.account_usage.access_history ah
 ON ah.base_objects_accessed LIKE '%' || t.table_name || '%'
 AND ah.query_start_time >= DATEADD('day', -90, CURRENT_TIMESTAMP())
WHERE
 t.table_schema NOT IN ('INFORMATION_SCHEMA')
 AND t.deleted IS NULL
 AND t.row_count > 0
GROUP BY
 t.table_schema, t.table_name, t.row_count, t.bytes, t.last_altered
HAVING
 last_queried IS NULL
ORDER BY
 size_gb DESC;

On the AWS side, check your S3 storage for orphaned data. Landing zones accumulate raw files that were loaded months ago and never cleaned up. A simple lifecycle policy handles this:

{
 "Rules": [
 {
 "ID": "Archive raw data after 90 days",
 "Status": "Enabled",
 "Filter": {
 "Prefix": "raw-landing/"
 },
 "Transitions": [
 {
 "Days": 90,
 "StorageClass": "GLACIER_INSTANT_RETRIEVAL"
 }
 ],
 "Expiration": {
 "Days": 365
 }
 }
 ]
}

The cultural shift matters as much as the tooling. Make it a quarterly habit: pull the zombie table list, review it with the team, and deprecate what nobody uses. If someone screams, you can always restore from Time Travel. But in my experience, nobody screams. The data was already dead — you’re just acknowledging it.

Cost anomalies will find you — or you can find them first

The scariest cost event isn’t the gradual creep. It’s the single bad query or misconfigured pipeline that doubles your weekly bill overnight. I’ve seen a single Snowflake query with a missing WHERE clause scan an entire 2TB table repeatedly inside a loop — burning through hundreds of credits in an hour.

On Snowflake, set up resource monitors as a basic guardrail:

-- Create a resource monitor with alerts and hard stop
CREATE OR REPLACE RESOURCE MONITOR monthly_budget
 WITH CREDIT_QUOTA = 5000 -- adjust to your monthly budget
 FREQUENCY = MONTHLY
 START_TIMESTAMP = IMMEDIATELY
 TRIGGERS
 ON 75 PERCENT DO NOTIFY -- email alert at 75%
 ON 90 PERCENT DO NOTIFY -- email alert at 90%
 ON 100 PERCENT DO SUSPEND; -- hard stop at 100%

-- Apply it to a warehouse
ALTER WAREHOUSE transform_wh
 SET RESOURCE_MONITOR = monthly_budget;

On AWS, enable Cost Anomaly Detection in the AWS Cost Management console. It uses ML to detect unusual spending patterns and sends alerts via SNS. For Snowflake-specific monitoring, a lightweight approach is a scheduled task that checks daily credit consumption against a rolling average:

-- Create a simple anomaly detection view
CREATE OR REPLACE VIEW monitoring.daily_cost_anomalies AS
WITH daily_usage AS (
 SELECT
 DATE_TRUNC('day', start_time) AS usage_date,
 warehouse_name,
 SUM(credits_used) AS daily_credits
 FROM
 snowflake.account_usage.warehouse_metering_history
 WHERE
 start_time >= DATEADD('day', -60, CURRENT_TIMESTAMP())
 GROUP BY
 usage_date, warehouse_name
),
averages AS
(
 SELECT
 warehouse_name,
 AVG(daily_credits) AS avg_daily_credits,
 STDDEV(daily_credits) AS stddev_daily_credits
 FROM
 daily_usage
 WHERE
 usage_date < CURRENT_DATE()
 GROUP BY
 warehouse_name
)
SELECT
 du.usage_date,
 du.warehouse_name,
 du.daily_credits,
 a.avg_daily_credits,
 ROUND((du.daily_credits - a.avg_daily_credits) / NULLIF(a.stddev_daily_credits, 0), 2) AS z_score
FROM
 daily_usage du

 JOIN averages a
 ON du.warehouse_name = a.warehouse_name
WHERE
 du.usage_date = CURRENT_DATE()
 AND (du.daily_credits - a.avg_daily_credits) / NULLIF(a.stddev_daily_credits, 0) > 2
ORDER BY
 z_score DESC;

A z-score above 2 means today’s spend is more than two standard deviations above the 60-day average. That’s worth investigating. Pipe this into a Teams or Slack alert via an AWS Lambda function and you’ve got same-day cost anomaly detection for the price of a few lines of SQL.

What your data model costs you

This one’s subtle but powerful. The way you model your data directly affects how much compute you burn on every query.

In columnar warehouses like Snowflake, wide denormalised tables query faster than star schemas with multiple JOINs — because each JOIN has overhead, and Snowflake is optimised for scanning columns from flat structures. But wide tables cost more to maintain because updating a single dimension attribute means rewriting many rows in the fact table.

The practical approach is a layered architecture:

Staging models (views or ephemeral): Minimal transformation, column selection, type casting. Cheap to run, no storage cost.
Intermediate models (tables or incremental): Business logic, deduplication, SCD handling. Materialised because they’re referenced by multiple downstream models.
Mart models (incremental or table): Wide, denormalised, optimised for consumer queries. These are what analysts and BI tools actually hit.

The cost trap is materialising too much too early. Every table materialisation means Snowflake stores and maintains that data. Every dbt run rebuilds it. If an intermediate model is only referenced by one downstream model, make it ephemeral or a view — let Snowflake inline it at query time.

Check your dbt DAG for this pattern: a staging model materialised as a table, referenced by a single intermediate model, which is also materialised as a table, referenced by a single mart. That’s three materialisations where one (the mart) would suffice. The staging and intermediate models can be views or ephemeral — the compute happens once when the mart builds, not three separate times.

Coming back to why this matters

I started this article with a story about not being able to explain my own platform’s costs. That wasn’t a technical failure. It was a leadership failure. I’d built something valuable and then neglected the part that kept it funded.

The data engineers I respect most aren’t the ones who build the most elegant pipelines. They’re the ones who can walk into a budget conversation and say: “Here’s what we spend. Here’s what we get for it. Here’s what I’d cut, and here’s what I’d invest more in.” That’s the kind of clarity that earns trust — and trust is what keeps data teams alive when the inevitable cost-cutting conversations happen.

You don’t need a FinOps certification or an expensive monitoring tool to get there. You need the queries in this article, a quarterly habit of reviewing them, and the willingness to kill the zombies nobody wants to admit are dead.

Start with the 15-minute audit. Tag your queries. Set up a resource monitor. Do those three things this week and you’ll know more about your platform’s economics than most data teams learn in a year.

The cheapest query is the one you never need to run. But the most valuable skill is knowing which queries are worth paying for.

Why Your Pipeline Finishes Later Every Month

Chris Hillman — Sat, 18 Apr 2026 09:00:00 +1100

Let me tell you about a graph that changed how I think about data engineering.

A junior engineer on my team — let’s call her Priya — had been tracking something nobody asked her to track. Every morning for two months, she’d noted the timestamp when our main analytics pipeline completed. She wasn’t trying to make a point. She was just curious, because the finance team kept mentioning their dashboards weren’t ready when they arrived at 8 AM anymore.

One afternoon she pulled me aside and showed me a scatter plot on her laptop. Pipeline completion time, plotted daily over several months. The trend was unmistakable: a slow, steady drift to the right. What used to finish at 5:47 AM was now finishing at 7:23 AM. And the slope wasn’t flattening.

“If this keeps going,” she said, “we’ll miss the 9 AM SLA in about six weeks.”

She was right. And nobody else on the team — including me — had noticed. We were watching for failures. Green DAGs, clean logs, no alerts. But the pipeline wasn’t failing. It was slowing. And slow is harder to see than broken, because slow doesn’t trigger an alert. Slow just quietly erodes trust until one day someone in finance builds their own spreadsheet and stops asking you for anything.

That’s the moment I understood that pipeline health isn’t about pass/fail. It’s about trajectory. A pipeline that runs successfully but takes 5% longer every month is a ticking clock. And the people who notice first aren’t the engineers watching the DAG — they’re the consumers waiting for their data.

I’m telling you this because pipeline optimisation sounds like a performance engineering problem, and it is. But underneath the technical work, it’s really about a commitment: the commitment to deliver data when you said you would, every single day. That’s what builds trust between data teams and the rest of the organisation. Not fancy architectures. Not real-time everything. Just showing up on time, reliably.

This article is about diagnosing why your pipelines get slower, identifying the bottlenecks that actually matter, and fixing the patterns that cause data to arrive later than it should. Everything is grounded in Snowflake, AWS, Airflow, and dbt — with specific patterns you can apply immediately.

Measure the pipeline in stages, not as a single number

When a pipeline is slow, the instinct is to look at the longest-running task. That’s often the wrong place to start.

A pipeline is a chain of stages: extract, transform, load, test. The total runtime is the sum of all stages on the critical path — the longest chain of dependent tasks. But the bottleneck might not be the longest task. It might be a 30-second task that blocks five parallel branches from starting.

Before you optimise anything, instrument your pipeline to measure each stage independently. In Airflow, the task instance metadata already captures this.

Two numbers matter here: duration_seconds (how long the task actually ran) and queue_wait_seconds (how long it waited before it could run). If queue wait is high, your problem isn’t the task — it’s resource contention. Too many tasks competing for too few Airflow worker slots, or too many queries competing for the same Snowflake warehouse.

For dbt runs specifically, the run_results.json that dbt generates after every invocation is a goldmine:

import json

with open('target/run_results.json') as f:
 results = json.load(f)

# Find your slowest models
models = sorted(
 [r for r in results['results'] if r['status'] == 'success'],
 key=lambda x: x['execution_time'],
 reverse=True
)

for m in models[:10]:
 print(f"{m['unique_id']:60s} {m['execution_time']:8.1f}s")

Run that after your next dbt build. You’ll immediately see which models dominate your pipeline runtime. In my experience, 3–5 models account for 60–80% of total execution time. Those are the only models worth optimising.

Your DAG probably has dependencies it doesn’t need

This is the most common structural problem in data pipelines, and it comes in two forms. One is obvious once you know to look for it. The other is sneaky, because it starts as a completely reasonable engineering decision.

The first form: phantom dependencies.

An engineer builds Model C and declares it depends on Model B — because Model B produces a table that Model C reads. Fair enough. But six months later, someone refactors Model C and it no longer reads from Model B. The ref() gets removed from the SQL. But the dependency in the Airflow DAG? That stays. Or in dbt, someone adds a ref() to a model they don’t actually need, just to “make sure it runs first.”

The result is a DAG with phantom dependencies — tasks that wait for other tasks to complete even though they don’t use those tasks’ outputs. Every phantom dependency adds serial wait time to your pipeline.

The second form: dependency monsters.

This one is trickier, because it starts with good intentions.

The customer team needs three new enrichment attributes on dim_customers — regional segment codes sourced from a CRM export, tenure tier derived from a subscription history table, and a propensity score from a data science model. All reasonable requests. Each one approved and added without much ceremony.

But dim_customers is upstream of 40+ models across your DAG. Adding those three attributes means pulling in the CRM extract, joining the subscription history table — which has its own upstream dependencies — and waiting on the propensity score model to complete before any of those 40 downstream models can start. Your pipeline used to finish at 9 AM. Eighteen months and a dozen enrichment requests later, it finishes at 3 PM.

Each addition was individually justified. Nobody modelled what they’d cost collectively. That’s how you build a dependency monster.

The fix isn’t to refuse enrichment requests — it’s to stop embedding enrichment into your core spine model. Keep dim_customers lean: identifiers, names, status, the attributes that nearly every consumer genuinely needs. Build a separate dim_customers_extended model that joins in the expensive enrichment for the consumers who actually need it. Most of your downstream models will never touch the propensity score. There’s no reason to make them wait for it.

-- dim_customers.sql: lean spine — runs fast, unblocks the DAG
SELECT
 customer_id,
 customer_name,
 email,
 status,
 created_at
FROM
 {{ ref('stg_customers') }}

-- dim_customers_extended.sql: enrichment for consumers who need it
-- runs independently, doesn't block the main pipeline
SELECT
 c.customer_id,
 c.customer_name,
 c.status,
 crm.regional_segment,
 sub.tenure_tier,
 ps.propensity_score
FROM
 {{ ref('dim_customers') }} c

 LEFT JOIN {{ ref('stg_crm_segments') }} crm
 on crm.customer_id = c.customer_id

 LEFT JOIN {{ ref('int_subscription_tenure') }} sub
 on sub.customer_id = c.customer_id

 LEFT JOIN {{ ref('ml_propensity_scores') }} ps
 on ps.customer_id = c.customer_id

Now dim_customers_extended and its expensive upstream dependencies sit on their own branch of the DAG. Consumers who need the enrichment reference the extended model. The rest of the pipeline is unblocked.

A useful governance rule of thumb before adding a field to a core model: if fewer than half your consumers will ever query that attribute, it probably doesn’t belong there. The team who needs it should own the enrichment themselves, as a downstream model they maintain.

Auditing for both problems

In dbt, the dbt_project_evaluator package surfaces structural issues systematically. Install it:

# packages.yml
packages:
 - package: dbt-labs/dbt_project_evaluator
 version: ">=0.8.0 <1.0.0"

Then run:

dbt build --select package:dbt_project_evaluator

It will flag models with fan-out issues (one model referenced by too many downstream models), unnecessary dependencies, and models that could be parallelised but aren’t. It won’t catch dependency monsters directly — those require a human asking “does every consumer of this model actually use every attribute in it?” — but it’s a good starting point for the structural audit.

For Airflow DAGs, the visual DAG view tells you a lot at a glance. A healthy DAG looks like a tree with wide parallel branches. An unhealthy DAG looks like a chain — or worse, a funnel where everything converges through a single overloaded task before it can branch out again. That funnel shape is the visual signature of a dependency monster: many expensive upstreams flowing into one hub model, with dozens of downstream models queued behind it.

If your DAG looks like a chain, ask this question for every dependency edge: does downstream task B actually read data produced by upstream task A? If the answer is no, remove the dependency. Let them run in parallel.

I once audited a DAG with 47 tasks running in a strict serial chain. After removing phantom dependencies and restructuring, 31 of those tasks could run in parallel. The pipeline went from 2 hours 15 minutes to 38 minutes. Same tasks, same compute, same data. Just fewer unnecessary wait states.

The critical path is the only thing worth optimising

Here’s a concept from project management that applies directly to pipeline engineering: the critical path.

The critical path is the longest chain of dependent tasks through your DAG. It determines your pipeline’s minimum possible runtime. Every other path through the DAG has “float” — slack time where tasks can be delayed without affecting the overall completion time.

This means: optimising a task that isn’t on the critical path has zero impact on your pipeline’s end-to-end runtime. You could make a non-critical task 10x faster and your pipeline would finish at exactly the same time.

Finding the critical path requires knowing each task’s duration and dependency structure. For a dbt project, you can approximate it:

-- Using dbt's run results stored in Snowflake (if you log them)
-- Or query your orchestrator's task instance table
WITH task_durations AS (
 SELECT
 task_id AS model_name,
 AVG(EXTRACT(EPOCH FROM (end_date - start_date))) AS avg_duration
 FROM task_instance
 WHERE dag_id = 'dbt_daily'
 AND state = 'success'
 AND execution_date >= CURRENT_DATE - INTERVAL '7 days'
 GROUP BY task_id
)
SELECT
 model_name,
 ROUND(avg_duration, 1) AS avg_seconds,
 ROUND(avg_duration / 60, 1) AS avg_minutes
FROM task_durations
ORDER BY avg_duration DESC
LIMIT 20;

The longest tasks are candidates for the critical path, but only if they’re on the dependency chain that determines total runtime. A 20-minute model that runs in parallel with a 45-minute model isn’t the bottleneck — the 45-minute model is.

Once you’ve identified the critical path, you have three options for shortening it: make the slow tasks faster (query optimisation, incremental models), reduce the number of tasks on the path (remove unnecessary dependencies), or parallelise sequential tasks (split a monolithic model into independent pieces).

Shifting right: diagnose it before it breaks your SLA

Remember Priya’s scatter plot? That pattern — pipeline completion drifting later and later — has a name: shifting right. And it has predictable causes.

Track it with a simple query against your orchestrator’s metadata:

-- Airflow: track pipeline completion time drift
SELECT
 execution_date::DATE AS run_date,
 MAX(end_date)::TIME AS completion_time,
 EXTRACT(EPOCH FROM (MAX(end_date) - MIN(start_date))) / 60 AS total_runtime_minutes
FROM
 task_instance
WHERE
 dag_id = 'daily_analytics'
 AND state = 'success'
 AND execution_date >= CURRENT_DATE - INTERVAL '60 days'
GROUP BY
 run_date
ORDER BY
 run_date;

Plot completion_time over run_date. If it trends upward, you’re shifting right. The four root causes, in order of how often I see them:

1. Data volume growth. Your table had 10 million rows when you built the model. Now it has 200 million. That JOIN that took 8 seconds now takes 3 minutes. This is the most common cause and the easiest to fix — switch to incremental materialisation and the growth stops mattering.

2. Resource contention. You’ve added more pipelines and they all run in the same window, competing for the same Snowflake warehouse. In Snowflake, you can see this directly:

-- Find warehouse queueing (queries waiting for compute)
SELECT
 warehouse_name,
 DATE_TRUNC('hour', start_time) AS hour,
 COUNT(*) AS total_queries,
 SUM(CASE WHEN queued_overload_time > 0 THEN 1 ELSE 0 END) AS queued_queries,
 AVG(queued_overload_time) / 1000 AS avg_queue_seconds
FROM
 snowflake.account_usage.query_history
WHERE
 start_time >= DATEADD('day', -7, CURRENT_TIMESTAMP())
 AND warehouse_name = 'TRANSFORM_WH'
GROUP BY
 warehouse_name, hour
HAVING
 queued_queries > 0
ORDER BY
 hour;

If queued_queries is non-zero during your pipeline window, queries are waiting for warehouse compute. Either scale up the warehouse during that window, split workloads across dedicated warehouses, or stagger pipeline start times to reduce concurrency.

3. Dependency chain lengthening. Someone added a new intermediate model between your staging and mart layers. That model takes 4 minutes. But because it’s on the critical path, the entire pipeline now finishes 4 minutes later. This is death by a thousand cuts — each addition is small, but they accumulate.

4. Upstream source delays. Your pipeline starts at 4 AM because the source system’s extract used to land in S3 by 3:45 AM. But the source system has grown too, and now the extract doesn’t land until 4:30 AM. Your pipeline sensors wait, and everything shifts right.

For upstream delays in an AWS environment, replace time-based scheduling with event-driven triggers. Instead of scheduling your Airflow DAG at 4 AM and hoping the data is there, trigger it when the data actually arrives:

# Airflow DAG: trigger on S3 file landing using AWS sensor
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
from airflow import DAG
from datetime import datetime

with DAG(
 'daily_analytics',
 start_date=datetime(2026, 1, 1),
 schedule_interval=None, # triggered externally, not on a schedule
 catchup=False,
) as dag:

 wait_for_source = S3KeySensor(
 task_id='wait_for_orders_extract',
 bucket_name='your-data-lake',
 bucket_key='raw/orders/dt={{ ds }}/orders.parquet',
 aws_conn_id='aws_default',
 mode='reschedule', # frees the worker slot while waiting
 poke_interval=300, # check every 5 minutes
 timeout=7200, # fail after 2 hours if file never arrives
 )

The mode='reschedule' is critical. Without it, the sensor occupies a worker slot for the entire time it’s waiting. With reschedule, it checks, releases the slot, and checks again later. This prevents the classic deadlock where all your worker slots are consumed by sensors and no actual work can run.

Even better: use S3 event notifications to trigger a Lambda function that kicks off the DAG via Airflow’s REST API. Zero polling, zero wasted slots:

# Lambda function triggered by S3 PutObject event
import boto3
import requests
import os

def handler(event, context):
 bucket = event['Records'][0]['s3']['bucket']['name']
 key = event['Records'][0]['s3']['object']['key']

 # Trigger Airflow DAG via REST API
 airflow_url = os.environ['AIRFLOW_API_URL']
 response = requests.post(
 f"{airflow_url}/api/v1/dags/daily_analytics/dagRuns",
 json={
 "conf": {"source_bucket": bucket, "source_key": key}
 },
 auth=(os.environ['AIRFLOW_USER'], os.environ['AIRFLOW_PASSWORD']),
 headers={"Content-Type": "application/json"}
 )

 return {"statusCode": response.status_code}

This eliminates the “safety margin” scheduling pattern entirely. Your pipeline runs as soon as the data is available — not 30 minutes after you hope it will be available.

Partition and cluster to eliminate full table scans

The single most effective performance optimisation in Snowflake is making sure your queries only read the data they need. Snowflake’s micro-partition pruning does this automatically — but only if your data is physically organised in a way that aligns with your query patterns.

Clustering keys tell Snowflake how to organise data within micro-partitions. If you consistently filter on order_date, clustering on that column means queries with a WHERE order_date = '2026-03-01' clause scan a tiny fraction of the table instead of the whole thing.

-- Add clustering to a large fact table
ALTER TABLE analytics.fct_orders
 CLUSTER BY (order_date, customer_segment);

Choose clustering keys based on how the table is actually queried, not how it’s loaded. The best candidates are columns that appear in WHERE clauses, JOIN conditions, and range filters. Limit yourself to 2–3 keys — more than that and Snowflake can’t maintain effective clustering.

Check whether your existing tables benefit from clustering:

-- Check clustering depth (lower is better, 0 is perfect)
SELECT
 table_name,
 clustering_key,
 total_constant_partition_count,
 total_partition_count,
 average_overlaps,
 average_depth
FROM
 snowflake.account_usage.table_storage_metrics
WHERE
 table_catalog = 'YOUR_DATABASE'
 AND clustering_key IS NOT NULL
 AND active_bytes > 0
ORDER BY
 average_depth DESC;

An average_depth above 5 means your clustering isn’t effective — queries are still scanning more partitions than they should. Either the clustering key doesn’t match query patterns, or the table has had so many small writes that the clustering has degraded. Running ALTER TABLE ... RECLUSTER manually or relying on automatic clustering (which costs credits) addresses the latter.

On the AWS side, if you’re using S3 as a data lake with Parquet or Iceberg, the equivalent is partition layout. Partition your S3 data by the columns you filter most:

When Snowflake external tables or AWS Athena query this structure with a date filter, they skip entire directories. A query for March 15th reads two files instead of scanning the entire events/ prefix. The savings compound with data volume — at a billion rows, proper partitioning can reduce query times from minutes to seconds.

Make every task idempotent or debugging becomes a nightmare

Here’s a pattern I see constantly: a pipeline fails midway through, the engineer reruns it, and now there are duplicate rows in the target table. Because the first run inserted half the data before failing, and the rerun inserted all the data — including the half that already existed.

Idempotency means a task produces the same result whether it runs once or ten times. This isn’t just a nice-to-have — it’s the foundation that makes everything else in pipeline engineering possible. Without it, you can’t safely retry. You can’t backfill. You can’t debug.

In dbt, incremental models with a unique_key are idempotent by default — the MERGE statement handles duplicates:

{{
 config(
 materialized='incremental',
 unique_key='event_id',
 incremental_strategy='merge'
 )
}}

SELECT
 event_id,
 user_id,
 event_type,
 event_timestamp,
 properties
FROM
 {{ ref('stg_events') }}

{% if is_incremental() %}
WHERE
 event_timestamp >= (
 SELECT DATEADD('day', -3, MAX(event_timestamp))
 FROM {{ this }}
)
{% endif %}

For non-dbt loads — say, a Lambda function loading data from an API into Snowflake — use MERGE instead of INSERT:

MERGE INTO raw.api_customers AS target
USING (
 SELECT $1:id::STRING AS customer_id,
 $1:name::STRING AS customer_name,
 $1:email::STRING AS email,
 $1:updated_at::TIMESTAMP_NTZ AS updated_at
 FROM @raw.s3_stage/customers/
 (FILE_FORMAT => 'json_format')
) AS source
ON target.customer_id = source.customer_id

 WHEN MATCHED AND source.updated_at > target.updated_at THEN
 UPDATE SET
 customer_name = source.customer_name,
 email = source.email,
 updated_at = source.updated_at

 WHEN NOT MATCHED THEN
 INSERT (customer_id, customer_name, email, updated_at)
 VALUES (source.customer_id, source.customer_name, source.email, source.updated_at);

The WHEN MATCHED AND source.updated_at > target.updated_at condition prevents overwriting newer data with older data during reruns. This matters when you’re replaying historical loads or running overlapping backfills.

Most “we need real-time” requests actually need “we need faster batch”

Before you reach for Kafka, Kinesis, or any streaming infrastructure, have this conversation with your stakeholders: “When you say real-time, what do you actually mean?”

In my experience, the answers fall into three categories:

“I want data from today, not yesterday.” That’s daily batch with a morning refresh. You already have this. Maybe you need to move the refresh earlier.

“I want data that’s at most an hour old.” That’s micro-batch — running your existing pipeline every 15–60 minutes instead of once a day. No new infrastructure required. Your existing SQL, dbt, and Airflow tools work unchanged at shorter intervals.

“I need to see events within seconds of them happening.” This is actual real-time. And it’s genuinely rare. Fraud detection, safety monitoring, real-time bidding — these need streaming. Your internal sales dashboard almost certainly does not.

For the micro-batch pattern in Airflow, it’s a scheduling change:

with DAG(
 'micro_batch_analytics',
 schedule_interval='*/15 * * * *', # every 15 minutes
 catchup=False,
 max_active_runs=1, # prevent overlap
) as dag:
 # ...tasks here

The max_active_runs=1 is important. Without it, if a run takes longer than 15 minutes, the next run starts before the previous one finishes. They compete for the same warehouse, both run slower, and you’ve created the resource contention problem from the shifting-right section.

In Snowflake, pair this with Snowpipe for continuous S3 ingestion:

-- Create a pipe for continuous loading from S3
CREATE OR REPLACE PIPE raw.orders_pipe
 AUTO_INGEST = TRUE
 AS
 COPY INTO raw.orders_stream
 FROM @raw.s3_orders_stage
 FILE_FORMAT = (TYPE = 'PARQUET')
 MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;

Snowpipe loads files within minutes of them landing in S3. Your micro-batch dbt pipeline then transforms whatever arrived since the last run. The combination delivers data freshness measured in minutes — not seconds, but close enough for the vast majority of business use cases — without any streaming infrastructure.

Measure time-to-consumer, not just pipeline runtime

Pipeline runtime tells you how long your DAG takes. Time-to-consumer tells you how long a business event takes to reach a human decision-maker. These are different numbers, and the second one is what your stakeholders actually care about.

Time-to-consumer decomposes into stages:

Source extraction delay: Time between the event occurring and the data landing in your lake
Pipeline processing time: Your DAG runtime (the part you control most directly)
Warehouse serving time: Query execution time when a dashboard or report reads the data
Cache/refresh delay: How often the BI tool refreshes its cache

Track it by stamping timestamps at each handoff:

-- Build a freshness tracking model in dbt
-- mart_data_freshness.sql
SELECT
 'fct_orders' AS model_name,
 MAX(order_timestamp) AS latest_source_event,
 MAX(_loaded_at) AS latest_load_time,
 CURRENT_TIMESTAMP() AS measured_at,
 TIMESTAMPDIFF(
 'minute',
 MAX(order_timestamp),
 CURRENT_TIMESTAMP()
 ) AS minutes_since_latest_event,
 TIMESTAMPDIFF(
 'minute',
 MAX(_loaded_at),
 CURRENT_TIMESTAMP()
 ) AS minutes_since_latest_load
FROM
 {{ ref('fct_orders') }}

Schedule this model to run after every pipeline completion. Over time, you’ll see whether minutes_since_latest_event is stable, improving, or drifting. If it’s drifting, you can pinpoint which stage is responsible by comparing the event timestamp, load timestamp, and measurement timestamp.

In dbt, you can also use source freshness tests to alert when upstream data is stale:

# models/staging/_sources.yml
sources:
 - name: raw
 database: raw_db
 schema: public
 loaded_at_field: _loaded_at
 freshness:
 warn_after: { count: 2, period: hour }
 error_after: { count: 6, period: hour }
 tables:
 - name: orders
 - name: customers
 - name: events

Running dbt source freshness checks whether the upstream data is as fresh as you expect. If raw.orders hasn’t received new rows in 6 hours, the test fails — and that’s a signal that the source system’s extract is delayed, not that your pipeline is broken.

The view-on-view problem: the compound interest of bad performance

This is the architectural anti-pattern I see most often in Snowflake environments, and it’s one of those problems that starts small and compounds quietly.

A view in Snowflake doesn’t store any data — it re-executes its SQL every time it’s queried. That’s fine for a simple view. But when View C references View B, which references View A, which scans a large table — every query against View C triggers the entire chain from scratch.

I’ve seen environments where analysts querying a “simple” dashboard view were unknowingly triggering five layers of nested views, each with its own JOINs and aggregations. The query took 4 minutes. After materialising the two most expensive intermediate layers as tables (refreshed daily), the same query took 3 seconds.

The fix is to audit your materialisation strategy:

-- Find views that reference other views (potential nesting)
SELECT
 referencing_object_name AS downstream_view,
 referenced_object_name AS upstream_object,
 referenced_object_type
FROM
 snowflake.account_usage.object_dependencies
WHERE
 referencing_object_type = 'VIEW'
 AND referenced_object_type = 'VIEW'
ORDER BY
 downstream_view;

If this query returns results, you have view-on-view nesting. Not all of it is bad — a simple renaming view on top of a complex view is fine. But if you find three or more layers, or if any of the intermediate views involve heavy JOINs or aggregations, materialise the expensive layers as tables.

In dbt terms, the decision framework is:

Ephemeral: Simple CTEs, column renaming, type casting. Zero cost.
View: Light transformations queried infrequently by a few consumers.
Table: Complex transformations queried frequently. Rebuilt every run.
Incremental: Large tables where only a fraction of rows change. The right choice for most fact tables.

If a model takes more than 30 seconds to build and is queried more than once between builds, it should be a table or incremental — not a view.

Coming back to Priya’s scatter plot

Six weeks after Priya showed me that graph, we’d fixed the three root causes: an intermediate model that had grown from 5 million to 180 million rows without being switched to incremental, a phantom dependency chain that added 12 minutes of serial wait time, and a warehouse that was being shared between transformation and BI queries during the morning refresh window.

Pipeline completion moved from 7:23 AM back to 5:15 AM. The finance team got their dashboards before their morning coffee. And Priya? She built a monitoring dashboard that tracked completion time drift automatically — the thing I should have built from the start.

Here’s what I learned from that experience, and what I hope you take from this article: pipeline performance isn’t a problem you solve once. It’s a trajectory you manage. Data grows. Teams add models. Source systems change. The pipeline that’s fast today will be slow in six months unless someone is watching the trend line.

The engineers who build reliable data platforms aren’t the ones who build the fastest pipeline on day one. They’re the ones who notice when completion time drifts by 3 minutes per week — and fix it before anyone else has to ask.

Track your completion times. Audit your dependencies quarterly. Profile your critical path after every significant change. And teach your team that a green DAG doesn’t mean a healthy pipeline. Sometimes green just means it hasn’t failed yet.

Stop Building Salesforce Integrations From Scratch

Chris Hillman — Sat, 04 Apr 2026 09:00:00 +1100

Let me tell you about Marcus.

Marcus was on a team I led a few years back. Sharp, motivated, the kind of engineer who actually read documentation before writing code. When the business asked us to get Salesforce data into our warehouse, Marcus volunteered. He’d done API work before. He figured a few weeks, tops.

He scoped it carefully. Built a Python service that authenticated via OAuth, pulled Account, Contact, and Opportunity objects through the Bulk API, flattened the nested JSON into relational tables, handled pagination, managed rate limits. Wrote solid tests. Documented everything. The kind of work you’d point to in a code review and say this is how it’s done.

It worked beautifully. For about three months.

Then our Salesforce admin added a custom field to the Opportunity object — Renewal_Likelihood__c — and nobody told the data team. The pipeline didn’t fail. That’s the insidious part. It kept running, kept landing data. It just quietly dropped the new field on the floor.

When Marcus tracked it down, he added the field. Then he realised there were eleven other custom fields that had been added since go-live that the pipeline was silently ignoring. And the compound address fields — BillingAddress, ShippingAddress — had never worked with the Bulk API in the first place. He’d been extracting the component parts (BillingStreet, BillingCity) as a workaround, but the workaround had a bug that truncated postal codes for international addresses.

Marcus spent three weeks patching all of it. Three weeks of a talented engineer doing work that a managed connector handles automatically.

When I watched a good engineer spend the best part of a month on a problem that shouldn’t exist. Marcus wasn’t learning anything. He wasn’t growing. He was hand-coding schema detection logic for the fourth time because Salesforce ships three major releases a year and our sales ops team adds custom fields like they’re decorating a Christmas tree.

I’m telling you this because Snowflake OpenFlow is one of those things that sounds like “just another integration tool” until you’ve lived through the alternative. What I want to show you today is how to set it up with Salesforce — step by step, with enough detail that you could do it tomorrow — and more importantly, why the schema evolution feature alone justifies the switch from custom code.

If you’ve ever maintained a custom Salesforce integration, you already know why this matters.

The Problem Nobody Warns You About

Here’s what the “just build it yourself” crowd doesn’t tell you: the Salesforce API isn’t one API. It’s an ecosystem of overlapping interfaces, each with its own quirks, limits, and failure modes.

The REST API handles small, synchronous requests well — but try pulling a million Opportunity records through it and you’ll burn through your rate limits before lunch. The Bulk API 2.0 is designed for volume — it can handle up to 150 million records in a 24-hour window — but it doesn’t support compound fields like BillingAddress or MailingAddress. Those silently return nothing. Not an error. Nothing.

Then there’s SOQL, Salesforce’s proprietary query language, which looks enough like SQL to trick you into thinking you understand it. Until you hit the 2,000-record offset limit, or try to resolve a polymorphic relationship where a WhoId field on a Task could reference either a Contact or a Lead depending on the record. That’s not a data modelling problem — that’s a “your pipeline logic needs to branch based on the referenced object type” problem.

And rate limits. Enterprise Edition gives you 100,000 API calls per 24 hours, plus 1,000 per user licence. Sounds generous until you remember that your BI tool, marketing automation platform, customer support system, and your data pipeline are all drinking from the same well.

But schema evolution is the silent killer.

Every Salesforce org is different because every business is different. Your Salesforce admin creates custom objects and fields to match how your company actually works. That’s the point — it’s a customisation platform. But every custom field added to Salesforce is a potential break point for any integration that doesn’t handle schema changes automatically.

And those changes happen constantly. Research suggests schemas in enterprise SaaS tools change roughly every three days on average. In Salesforce specifically, between admin-driven customisation and Salesforce’s own three-times-a-year release cycle (Spring, Summer, Winter), the schema you built against last quarter may not be the schema you’re ingesting from today.

This is the trap. Building the initial integration isn’t hard. Any competent engineer can get Salesforce data into a warehouse. Keeping it working — across schema changes, API version deprecations, authentication token rotations, and rate limit adjustments — is where it eats your life.

I’ve seen teams eliminate two full days of monthly engineering maintenance by migrating off custom connectors. Two days, every month, spent on work that a managed tool does automatically. That’s not engineering — it’s janitorial work wearing an engineering costume.

What Is Snowflake OpenFlow?

OpenFlow is Snowflake’s native, fully managed data integration service. It’s not a partnership or a marketplace app — it’s a first-party feature built into the platform. If you’re already running Snowflake, OpenFlow runs inside your existing environment.

The backstory matters. In November 2024, Snowflake acquired a company called Datavolo for roughly $110 million. Datavolo was founded by Joe Witt, who co-created Apache NiFi back when it was an NSA project. That NiFi heritage is the foundation of OpenFlow — curated, versioned connector definitions powered by NiFi’s flow engine, wrapped in a managed Snowflake experience.

Architecturally, it splits into two pieces. The control plane lives inside Snowflake and gives you pipeline management, scheduling, and monitoring through the Snowsight UI. The data plane handles the actual work and can run in two modes: Snowflake Deployments (running on Snowpark Container Services within Snowflake’s own infrastructure) or BYOC (Bring Your Own Cloud, running as a Kubernetes cluster in your VPC). For most teams starting out, the SPCS option is simpler — it went generally available in November 2025 on AWS and Azure.

Data flows into Snowflake primarily through Snowpipe Streaming for the initial load, with merge queries handling incremental updates. The result is structured tables in your Snowflake database — not staged JSON files, not semi-structured VARIANT columns. Actual, queryable tables with proper column types.

The connector catalogue currently covers about 20 curated sources: databases (PostgreSQL, MySQL, SQL Server, Oracle), SaaS platforms (Salesforce, Workday, Jira, LinkedIn Ads, Meta Ads, Google Ads), cloud storage (Google Drive, SharePoint, Box), and streaming sources (Kafka, Kinesis). It’s not Fivetran’s 500+ connector library, and nobody’s pretending it is. But the sources it does cover are the ones that account for the vast majority of enterprise data movement.

One important caveat up front: the Salesforce Bulk API connector is still in preview as of March 2026. It works. Preview in Snowflake terms means the feature is functional but not yet covered by production support SLAs. Keep that in your mental model as we walk through the setup.

Setting Up the Salesforce Connector: A Walkthrough

This section is the step-by-step. I’m going to walk through the full setup — Salesforce configuration, Snowflake preparation, and connector deployment — with enough detail that you could follow along on your own org.

Phase 1: Salesforce Configuration

The connector uses JWT Bearer Flow for authentication, which means you need an RSA key pair and a Connected App configured in Salesforce.

Generate your RSA key pair. Open a terminal and run:

# Generate 2048-bit RSA private key
openssl genrsa -out salesforce_private_key.pem 2048

# Generate the corresponding public certificate (valid 1 year)
openssl req -new -x509 -key salesforce_private_key.pem \
 -out salesforce_certificate.pem -days 365

Keep that private key safe. You’ll need it when configuring the connector in Snowflake.

Create the External Client App in Salesforce. Navigate to Setup → Apps → App Manager → New External Client App. Enable OAuth and add two scopes: api and refresh_token (sometimes labelled offline_access). Enable the JWT Bearer Flow toggle and upload your salesforce_certificate.pem public certificate. Save the app and record the Consumer Key and Consumer Secret that Salesforce generates.

Configure access. On the app’s Manage settings, set the Permitted Users policy to “Admin approved users are pre-authorized.” Then assign the appropriate user profiles or permission sets. The user account the connector will authenticate as needs read access to every object you intend to sync.

Phase 2: Snowflake Preparation

Before deploying the connector, you need a database, schema, role, and warehouse ready for it.

-- Create a dedicated database for Salesforce data
CREATE DATABASE IF NOT EXISTS SALESFORCE_RAW;
CREATE SCHEMA IF NOT EXISTS SALESFORCE_RAW.OPENFLOW;

-- Create a connector-specific role
CREATE ROLE IF NOT EXISTS OPENFLOW_SF_CONNECTOR;

-- Grant the role permissions to write to the destination
GRANT USAGE ON DATABASE SALESFORCE_RAW TO ROLE OPENFLOW_SF_CONNECTOR;
GRANT USAGE ON SCHEMA SALESFORCE_RAW.OPENFLOW TO ROLE OPENFLOW_SF_CONNECTOR;
GRANT CREATE TABLE ON SCHEMA SALESFORCE_RAW.OPENFLOW TO ROLE OPENFLOW_SF_CONNECTOR;

-- Create a warehouse for merge operations
CREATE WAREHOUSE IF NOT EXISTS OPENFLOW_WH
 WAREHOUSE_SIZE = 'SMALL'
 AUTO_SUSPEND = 60
 AUTO_RESUME = TRUE;

GRANT USAGE ON WAREHOUSE OPENFLOW_WH TO ROLE OPENFLOW_SF_CONNECTOR;
GRANT OPERATE ON WAREHOUSE OPENFLOW_WH TO ROLE OPENFLOW_SF_CONNECTOR;

For SPCS deployments, you also need a network rule allowing the connector to reach your Salesforce instance. OpenFlow runs inside Snowflake’s container environment, and by default it can’t reach external endpoints without explicit permission:

-- Allow egress to your Salesforce instance
CREATE NETWORK RULE IF NOT EXISTS OPENFLOW_SF_EGRESS
 TYPE = HOST_PORT
 MODE = EGRESS
 VALUE_LIST = ('yourinstance.salesforce.com:443');

-- Wrap it in an external access integration
CREATE EXTERNAL ACCESS INTEGRATION IF NOT EXISTS OPENFLOW_SF_ACCESS
 ALLOWED_NETWORK_RULES = (OPENFLOW_SF_EGRESS)
 ENABLED = TRUE;

Replace yourinstance.salesforce.com with your actual Salesforce instance hostname. If you’re not sure what that is, log in to Salesforce and look at the URL bar — it’s the domain before .salesforce.com.

Phase 3: Deploying the Connector

Now the fun part. Navigate to Snowsight → Data → Openflow in the left sidebar. Find the “Openflow connector for Salesforce Bulk API” tile and add it to your runtime.

This opens the NiFi canvas — a visual flow editor where the connector’s pre-built process groups appear. Right-click the Salesforce process group and select Configure Parameters. Here’s where you’ll enter:

Salesforce Instance URL — e.g., https://yourinstance.salesforce.com
Consumer Key — from the Connected App you created
Consumer Secret — same source
Private Key — the contents of salesforce_private_key.pem (the full PEM text, headers included)
Username — the Salesforce user the connector authenticates as
Filter — the objects to sync, by API name

The Filter parameter is where you define what to pull. List standard objects by name (Account, Contact, Opportunity, Lead, Task, Event) and custom objects using their API name with the __c suffix:

There’s also a Special Objects Filter for objects that require different handling (like User or RecordType).

Once the parameters are configured, enable all controller services in the process group, then start the connector. The initial sync will pull a full snapshot of every object in your filter list. Subsequent runs perform incremental syncs using Salesforce’s SystemModstamp field to identify changed records.

What You Get

Once the connector runs, you’ll find one table per Salesforce object in SALESFORCE_RAW.OPENFLOW. The tables are flattened and structured — every Salesforce field becomes a column with an appropriate Snowflake data type. No VARIANT columns to parse. No nested JSON to unpack.

An Account table will have columns like ID, NAME, INDUSTRY, ANNUALREVENUE, BILLINGSTREET, BILLINGCITY, BILLINGSTATE, BILLINGPOSTALCODE, CREATEDDATE, LASTMODIFIEDDATE, and so on. Custom fields like Renewal_Likelihood__c appear as RENEWAL_LIKELIHOOD__C.

Two metadata columns are added automatically: ISDELETED (tracking Salesforce soft deletes) and the system timestamp fields. This is important — hard deletes in Salesforce are not reflected. If a record is permanently deleted in Salesforce, it won’t disappear from your Snowflake table. You’ll need to handle that in your downstream dbt models if it matters for your use case.

A Note on Multiple Sync Frequencies

Not every Salesforce object changes at the same pace. Your Opportunity table might need hourly syncs while Account is fine with daily. OpenFlow handles this by letting you deploy multiple connector instances in the same runtime — each with its own filter list and schedule — at no additional compute cost beyond the shared runtime.

Schema Evolution: Where OpenFlow Earns Its Keep

This is the feature that matters most, and the one that would have saved Marcus three weeks.

When someone adds a new custom field to a Salesforce object — say the sales ops team creates Deal_Confidence_Score__c on the Opportunity object — the OpenFlow connector automatically detects the new field and adds a corresponding column to the Snowflake destination table on the next sync. No configuration change. No redeployment. No Teams message at 7am asking the data team to “add that new field we made yesterday.”

The column appears, data starts flowing into it, and your dbt models can reference it whenever you’re ready.

For field renames, OpenFlow takes a conservative approach: the old column stays in place (with stale data from the last sync before the rename), and a new column is created under the new field name. This means you’ll have both Old_Field_Name__c and New_Field_Name__c in your table for a period. That’s actually useful for audit purposes — you can see exactly when the rename happened by comparing timestamps — but it does mean your downstream queries need updating.

Here’s the honest gap: new custom objects are not auto-discovered. If someone creates an entirely new custom object in Salesforce, you have to manually add it to the connector’s Filter parameter. Schema evolution handles field-level changes automatically, but object-level discovery is still a manual step.

And field type changes — say someone changes a text field to a picklist, or a number to a formula — aren’t comprehensively documented in the current OpenFlow materials. In practice, this is a rare enough occurrence that it hasn’t been a problem in my experience, but it’s worth knowing the edge case exists.

Compare this to what Marcus had to maintain. His custom integration used a hardcoded list of fields per object. Every time a field was added, renamed, or retyped, someone had to update the Python code, test it, deploy it, and verify the data. Usually that “someone” was Marcus, usually at an inconvenient time, and usually because nobody told him the change was coming.

The value of automatic schema evolution isn’t technical elegance. It’s that your engineers stop spending time on schema babysitting and start spending it on work that actually matters — building models, improving data quality, answering the questions the business is actually asking.

A Word About Fivetran

I’d be doing you a disservice if I wrote about managed Salesforce connectors without mentioning Fivetran. For most of the last decade, Fivetran has been the gold standard here, and their Salesforce connector is genuinely excellent.

Fivetran’s schema evolution handling is more mature and more configurable than OpenFlow’s current offering. They offer three schema change policies: ALLOW_ALL (sync everything automatically, including new tables), ALLOW_COLUMNS (auto-add new columns but not new tables), and BLOCK_ALL (manual opt-in for everything). They also generate a LOG table that records every schema change event — create_table, alter_table, drop_table — giving you an audit trail that’s useful for debugging and compliance. Field type widening (like INT to LONG) happens automatically; narrowing changes are blocked to prevent data loss.

Their pricing model is fundamentally different from OpenFlow. Fivetran charges per Monthly Active Row (MAR) — the count of distinct rows synced per connection per month. A unique primary key counts once regardless of how many times it’s updated. Approximate pricing clusters around $500 per million MAR on the Standard plan, with a $12,000 annual minimum commitment. Since early 2025, MAR is calculated per connection rather than account-wide, which eliminated the bulk discount that previously benefited multi-connector setups.

OpenFlow, by contrast, charges for compute — SPCS credits for the container runtime, Snowpipe Streaming costs for ingestion, and warehouse costs for merge operations. There’s no per-row fee and no separate licence. It’s included with Snowflake. The trade-off is that your management compute pool runs continuously while a deployment exists, which means you’re paying a base cost even when no data is flowing. One estimate I’ve seen puts idle costs at roughly $10 per day, though this varies with configuration.

For a single large Salesforce org, OpenFlow’s compute-based model may end up cheaper than Fivetran’s per-row pricing — especially if your Salesforce data doesn’t churn heavily. For environments where you need 30+ connectors across different source types, Fivetran’s breadth and maturity is hard to argue with. They have 500+ connectors. OpenFlow has about 20.

The honest framing: OpenFlow is the better choice if you’re already deeply invested in Snowflake, value platform consolidation, and your primary sources are in the current connector catalogue. Fivetran is the safer choice if you need broad connector coverage, battle-tested reliability, and you’d rather pay a premium for someone else to handle the ops.

They’re not mutually exclusive, either. I’ve seen teams run Fivetran for the long tail of small connectors while using OpenFlow for their highest-volume sources where compute-based pricing wins.

The Zero-Copy Alternative

There’s a third path I haven’t mentioned yet, and it’s worth understanding even if it’s not right for every team: zero-copy data federation.

Salesforce Data Cloud offers a zero-copy integration with Snowflake that flips the entire model on its head. Instead of extracting data from Salesforce and loading it into your warehouse, zero-copy gives you direct read access to Salesforce data without moving it at all. Salesforce’s query pushdown engine handles the work — when you query the federated data, Salesforce pushes the query to the source, filters and aggregates there, and returns only the results you need. No pipelines to build. No schemas to manage. No sync schedules to configure.

It works bidirectionally, too. You can share data from Snowflake back into Salesforce Data Cloud, letting sales reps see warehouse-enriched signals directly on Account and Opportunity pages without anyone exporting a CSV or building a reverse ETL pipeline. The integration runs on Apache Iceberg under the hood, which at least means the underlying format is open.

For handling new custom objects — the gap I flagged with OpenFlow — zero-copy sidesteps the problem entirely. There’s nothing to discover because there’s nothing to sync. The data stays where it is, and you access it in place. As your Salesforce org evolves, the federated view evolves with it.

That’s genuinely appealing. For proof-of-concept work, rapid prototyping, or use cases where you need real-time access to Salesforce data without building infrastructure, zero-copy is fast and effective.

But here’s where I get cautious.

It comes at a premium. Salesforce Data Cloud runs on a consumption credit model — credits are purchased in bundles (roughly $500 per 100,000 credits at list price), and every action burns them. Zero-copy federation queries consume about 70 credits per million records, which sounds reasonable until you’re running it at scale across multiple business units. The pricing model has simplified since late 2025 — Salesforce consolidated multiple credit types into a single fungible credit and made ingestion from core Salesforce products free — but it’s still a consumption model with real costs that compound quickly if you’re not watching.

More importantly, zero-copy creates a deep platform dependency. Your data access is mediated entirely through Salesforce and Snowflake’s partnership. If Salesforce changes their terms, adjusts their pricing (which they’ve done repeatedly), or if you decide to move off Snowflake to Databricks or BigQuery, that zero-copy integration doesn’t come with you. You’d need to build actual data movement infrastructure — the thing you avoided — under time pressure and with no existing pipeline to fall back on.

I value keeping infrastructure portable. Not because I’m paranoid about vendor lock-in — I’m pragmatic about it. I’ve been in this industry long enough to see what happens when a new CEO arrives and wants to renegotiate every vendor contract, or when a cloud provider changes their pricing structure overnight, or when the company you depend on gets acquired and the roadmap shifts. Having your data physically in your own warehouse, in formats you control, with pipelines you can redirect to a different destination — that’s insurance. Not glamorous insurance, but the kind you’re grateful for when you need it.

The balance is real, though. You always have to weigh portability against the time you spend building and maintaining pipelines. If your team is drowning in pipeline maintenance, zero-copy might buy you breathing room while you stand up proper infrastructure. Just go in with your eyes open about what you’re trading for that convenience.

There’s a more fundamental concern, though, and it’s the one that won’t appear in any vendor comparison chart.

Zero-copy makes it dangerously easy to skip the data warehouse entirely. When you give analysts and report builders direct read access to raw Salesforce data, the temptation to build dashboards straight from it is enormous. And for a quick proof of concept or a one-off analysis, fine. But the moment those PoC dashboards become “the dashboard the VP checks every Monday,” you’ve got a problem.

Raw source data doesn’t tell you which CustomerNo field to use — the one from the legacy migration, the one from the 2023 CRM consolidation, or the one from the current integration. It doesn’t encode the business rule that says “Closed Won” opportunities in APAC exclude training deals under $5,000 because those are tracked separately. It doesn’t flag that the Last_Activity_Date field is unreliable for accounts owned by the partner team because they log activities in a different system.

That knowledge lives in your data warehouse layer. It lives in the dbt models that your team built over months of conversations with stakeholders, through change management cycles, through debugging sessions where someone finally said “oh, we stopped using that field in Q3 because…” That’s not transformation logic — it’s institutional memory encoded as code. It connects raw data across domains and gives the business a full landscape picture instead of a narrow, single-source view. It catches the behavioural anomalies unique to your organisation — the sales rep who bulk-updates 500 records every Friday afternoon, the integration that occasionally double-fires during DST transitions, the custom object that gets repurposed every time the business restructures.

So by all means, evaluate zero-copy for the use cases where it shines. Real-time signals on Account pages for sales reps? Great fit. Federated access for a specific analytics team that knows the source data intimately? Reasonable. But don’t let it replace the warehouse layer. That layer is where the hard-won understanding of your data lives, and it’s harder to recreate than any pipeline.

The Honest Trade-Offs

I’ve been positive about OpenFlow because I think it solves a real problem well. But I’d be violating my own writing principles if I didn’t lay out the limitations clearly.

The Salesforce connector is still in preview. It works, but it’s not covered by production support SLAs. If you’re running a mission-critical pipeline that your CFO stares at every Monday morning, that matters.

Custom Salesforce domains aren’t supported. If your org uses a vanity URL (like mycompany.my.salesforce.com with a custom domain), check the documentation carefully before committing.

Hard deletes aren’t tracked. The ISDELETED column catches soft deletes, but records that are permanently purged from Salesforce will persist in your Snowflake tables indefinitely. You’ll need a reconciliation process if that matters for your use case.

Certain field types are silently dropped. location, address (compound), and base64 fields are not synced. The connector doesn’t error on these — it just ignores them. This is the kind of thing that bites you three months in when someone asks why geographic data isn’t in the warehouse.

Formula fields require full refresh. Because formula field values are computed server-side and don’t update SystemModstamp, they can’t be synced incrementally. You need a separate connector instance running full refreshes to capture formula field changes. That’s not a bug — it’s how Salesforce works — but it’s a gotcha if you’re not expecting it.

No relationship traversal. You can’t configure the connector to follow lookups and pull related objects automatically. Each object is pulled independently. Joining them is your job in the transformation layer, which is where it belongs anyway if you’re running dbt.

None of these are dealbreakers. But every one of them is the kind of thing that bites an engineer at 4pm on a Friday if they weren’t expecting it. Now you’re expecting it.

Getting Started

If you want to try this on your own Salesforce org, here’s my recommended order of operations:

Start with a non-production Salesforce sandbox and a Snowflake trial account if you don’t have a dev environment handy. Pick two or three standard objects — Account, Contact, Opportunity — and one custom object if your org has them. Run through the three-phase setup I described above. Get data flowing, verify the table structures, and then add a custom field to one of the objects in Salesforce. Wait for the next sync. Watch the column appear automatically in Snowflake.

That’s the moment it clicks. That’s the moment you stop thinking about managed connectors as a luxury and start thinking about custom API code as technical debt you’re choosing to carry.

Once you’ve validated the core flow, layer on your dbt models. OpenFlow lands raw data — it doesn’t transform it. Your staging models handle naming conventions, type casting, soft delete filtering, and the join logic that stitches related objects together. This is the same pattern you’d use with Fivetran or any other EL tool: land it raw, transform it in the warehouse.

For teams already running Fivetran for Salesforce, I’m not suggesting you rip it out tomorrow. If it’s working and the cost is acceptable, that’s a solved problem. But the next time you’re adding a new high-volume source that’s in OpenFlow’s catalogue — especially if you’re already paying for Snowflake — it’s worth running the numbers. You might find that native integration with compute-based pricing is the better deal.

The Real Why

I started this article with Marcus’s story because it illustrates something I feel strongly about: your engineers’ time is the most expensive resource in your data organisation, and spending it on solved problems is a leadership failure.

Building a custom Salesforce integration isn’t impressive. It was impressive in 2016, when the tooling didn’t exist. Today, it’s a choice to take on maintenance burden that a managed service handles automatically. It’s choosing to babysit schema changes instead of building the models and analyses that actually move the business forward.

Marcus is a senior engineer now. He builds dimensional models and designs data products that directly influence how the sales team allocates resources. He doesn’t write API integration code anymore. Not because he can’t — because his time is worth more than that.

But here’s what I keep coming back to. The tools keep getting better — OpenFlow, Fivetran, zero-copy, and yes, AI-assisted pipeline generation. Every year there’s a new thing that promises to automate another piece of the data engineering workflow. And many of them genuinely do.

What they don’t automate is the part that actually matters.

No tool — not OpenFlow, not Fivetran, not an AI agent — is going to sit in a room with your head of sales and ask “what keeps you up at night?” No connector is going to notice that the way your company defines “active customer” has quietly drifted from what the CRM tracks. No pipeline is going to push back and say “we could build that dashboard, but the metric you’re asking for doesn’t answer the question you actually have.”

That’s the connective tissue between raw data and business value. It’s a person who understands the domain, who’s been present through the change management cycles, who’s built relationships with the stakeholders and knows which CustomerNo field to use because they were in the room when the decision was made three years ago. It’s someone who asks clarifying questions instead of presuming they already know the answer.

AI will keep getting better at the mechanical parts of data engineering. Schema detection, pipeline code generation, anomaly detection — those are well-defined problems that automation is suited for. But the assumption that any tool can skip the human step — the clarification, the context, the judgment about what’s actually worth building — that’s where I’ve seen projects go sideways. Not because the technology failed, but because nobody asked the right questions before building.

OpenFlow isn’t perfect. The connector catalogue is young, the Salesforce connector is in preview, and Fivetran still wins on breadth and battle-hardened maturity. But OpenFlow represents something important: the data platform taking responsibility for data movement, not just data storage and compute. It’s Snowflake saying “we’ll handle getting the data in — you focus on making it useful.”

That last part — making it useful — is still your job. And it’s the part that no tool can do for you. The best thing a managed connector gives you isn’t fewer lines of code. It’s time back. Time to spend on the work that actually requires a human: understanding the business, building the right models, and making sure the data tells an accurate story.

The next time someone on your team says “I’ll just build a quick Salesforce integration” — send them this article. And the next time someone says “AI can just handle the data pipeline” — ask them who’s going to sit with the stakeholders and figure out what the pipeline should actually deliver.

That’s still you. Make sure you have the time for it.

Your Data Model Isn't Broken, Part II: The Refactoring Playbook

Chris Hillman — Sat, 28 Mar 2026 09:00:00 +1100

In [Part I], I made the case that your legacy data model isn’t the disaster it looks like. That the strange WHERE clauses, the bridge tables nobody can explain, and the slowly-changing-dimension-within-a-slowly-changing-dimension aren’t bugs — they’re business rules earned through years of production reality. I argued that big-bang rebuilds fail at alarming rates, that the complexity you’re fighting is mostly essential rather than accidental, and that the impulse to “start from scratch” is driven more by cognitive bias than by engineering judgment.

A few people messaged me afterward and said, roughly: “Okay, I’m convinced. But what am I supposed to do about it?”

Fair question. Let’s talk about the playbook.

The vine that eats the tree

In 2004, Martin Fowler was visiting a rainforest in Queensland when he noticed something remarkable about the strangler fig trees. These vines start as seeds deposited in the upper branches of a host tree by birds. They grow downward, wrapping around the trunk, eventually establishing their own root system in the soil. Over years — sometimes decades — the fig gradually replaces the host tree. The original tree decays inside the fig’s lattice until there’s nothing left of it. But at no point during this process does the canopy collapse. The forest continues to function the entire time.

Fowler looked at those trees and thought: that’s how you should replace software systems.

The Strangler Fig pattern is, in my experience, the single most useful idea in all of software architecture — and it maps onto data warehouse migrations with almost uncomfortable precision. The concept is deceptively simple: instead of replacing the old system all at once, you build the new system around it. You intercept one piece of functionality at a time, route it through the new implementation, verify it works, and move on. The old system slowly shrinks. The new system slowly grows. And at no point does the business lose access to anything.

Fowler updated his essay on this pattern as recently as August 2024, and his framing is worth internalising. He described the alternative — the big-bang rewrite — and noted that he’d watched it fail most of the time. The strangler fig approach, by contrast, gives value steadily and allows you to monitor progress more carefully through frequent releases.

Now, here’s where it gets interesting for data teams specifically. Application systems have clean interfaces — APIs, endpoints, well-defined contracts. You can intercept a request, route it to the new system, and nobody downstream knows the difference. Data systems are messier. Elliott Cordo, who documented migrating three legacy data warehouses using this pattern, identified the core challenge: analytics systems have porous boundaries and poorly defined interfaces with consumers. You don’t always know who’s querying what, or how.

Cordo’s solution — and this is the variant I’ve seen work best — is what he calls the Legacy Façade. You build a new interface that mimics the legacy system’s contract. Same table names, same column names, same output format. Behind the façade, you’ve plumbed in the new logic. Consumers don’t change anything. They don’t even need to know a migration is happening. And this, critically, eliminates the organisational bottleneck that kills most migrations: getting twenty different teams to simultaneously update their queries, dashboards, and downstream pipelines to point at the new system.

Cordo’s experience is particularly telling because one of the three warehouses he migrated existed precisely because a previous attempt to replace one of the others had failed — mainly due to change management. The strangler fig approach bypasses that failure mode entirely by asking consumers to do little or nothing to adopt the new system.

I think this matters so much: the majority of data warehouse migrations don’t fail for technical reasons. They fail because you can’t convince the entire organisation to cut over on the same Tuesday. The Legacy Façade removes that dependency. You migrate at your pace, not at the pace of every team that consumes your data.

Stop publishing before you’ve checked

I’m continually surprised by how few people have heard of this pattern. When Robin Moffatt polled data engineers on Reddit about Write-Audit-Publish, most hadn’t even encountered the term — let alone used it. That’s a problem, because WAP solves the single scariest moment in any refactoring effort: the moment you push changed logic to production and hope you didn’t break anything.

Netflix introduced WAP at the DataWorks Summit in 2017, and the concept is elegant in its simplicity. Instead of the typical pipeline flow — transform data, write it to the production table, then maybe run some checks afterward — you split the process into three distinct stages:

Write to an isolated staging area. Not production. Not anywhere consumers can see it.

Audit the output against your quality checks. Row counts. Schema validation. Business rule assertions. Whatever gives you confidence.

Publish only if the audit passes. If it doesn’t, production never sees the bad data. Nobody’s dashboard breaks. Nobody’s quarterly report includes garbage. The corrupted data dies quietly in staging where it belongs.

The reason this matters for refactoring specifically is that refactoring requires you to change transformation logic while producing identical output. You’re restructuring the internals — breaking a 400-line SQL query into modular CTEs, moving logic from stored procedures into dbt models, replacing hardcoded values with reference table lookups. Each of these changes is supposed to be behaviour-preserving. WAP gives you the safety net to verify that it actually was.

Julien Hurault described the common anti-pattern — what he calls Write-(Publish)-Audit — where if an error is detected during testing, it’s already too late because the corrupted data has already been released to downstream systems. I’ve seen this play out more times than I want to admit. Someone refactors a model, the tests pass in the development environment, the change gets deployed, and then a stakeholder notices that revenue is off by 12% in a dashboard three hours later. By that point, the bad data has propagated through six downstream models and someone’s already screenshot the dashboard for a presentation.

WAP prevents this entirely. And with Apache Iceberg now supporting it natively through branch-based isolation, the tooling has finally caught up with the idea. You write to an audit branch, run your validations, and only fast-forward merge to main when everything checks out. It’s the data engineering equivalent of a pull request with automated tests — and honestly, it’s bizarre that we’ve accepted as normal the practice of deploying data transformations straight to production without a verification step in between.

If you’re on Snowflake, the implementation is straightforward using CLONE and SWAP operations. Clone the production table, run your refactored pipeline against the clone, validate the output, and swap the clone into the production role. If validation fails, the clone gets dropped and production is untouched. I’ve written about this pattern in more detail here — but the key point for this article is simpler: WAP turns refactoring from a high-wire act into a methodical, reversible process.

The archaeology of SQL

How does refactoring a data model actually works in practice, because it’s less dramatic than people expect. That’s sort of the point.

dbt Labs published a refactoring course that formalises the approach, and having used variants of this workflow on multiple teams, I think they’ve got it right. The core philosophy is one sentence: while refactoring, you’ll be moving around a lot of logic, but ideally you won’t be changing the logic.

Read that again. You are not improving business rules during a refactor. You are not fixing data quality issues during a refactor. You are not adding new features during a refactor. You are rearranging the existing logic into a better structure while proving, at every step, that the output hasn’t changed. The improvements come later, once the structure is clean enough to support them safely.

The workflow goes something like this. First, you take the legacy SQL — the 800-line stored procedure, the five nested subqueries, the transformation that nobody wants to touch because the last person who edited it left the company — and you port it into a dbt model unchanged. You don’t clean it up. You don’t refactor it. You bring it over exactly as it is, warts and all, and you verify that it produces identical output.

This is the step most people want to skip, and it’s the most important one. That ugly SQL is your baseline. It’s your source of truth. Until you’ve proved your refactored version matches it row-for-row, column-for-column, you don’t have a refactor — you have a rewrite wearing a refactor’s clothing.

Once you’ve got your baseline, you start decomposing. Extract the source references into dbt sources. Break the monolithic query into CTEs, each one doing a single logical operation. Move shared logic into staging models. Separate business rules from data cleaning from aggregation. At each step — and this is non-negotiable — you use something like dbt’s audit_helper package to compare the output of your refactored model against the original. If the outputs match, you move on. If they don’t, you figure out what you changed and fix it before going further.

Tristan Handy talks about his formative experience at Casper in 2016, where his team refactored all of their existing pipelines and brought them over to dbt in a single week, delivering work at least ten times faster than they would have been able to do otherwise. That speed came not from heroics but from methodology: migrate unchanged, then decompose with verification at every step.

I’ve watched a team at refactor over 200 dbt models with complex logic in three days using audit_helper. Three days. Not three months. Not the eighteen-month rebuild timeline that some stakeholder approved in a steering committee. Three days, because the tooling verified every change automatically and the team could move fast with confidence.

That’s the real payoff of refactoring over rebuilding. Not that it’s safer — though it is. Not that it preserves business logic — though it does. The payoff is that it’s faster. Faster to start delivering value. Faster to reach a clean architecture. Faster to build the confidence you need to actually make meaningful improvements.

Expand, then contract

There’s a pattern for handling breaking schema changes that I don’t see data teams use nearly enough, and it deserves more attention. Fowler calls it Parallel Change. Others call it Expand and Contract. The name doesn’t matter.

Imagine you need to rename a column in a fact table that thirty dashboards reference. In a rebuild mindset, you’d design the new schema, build it, and then coordinate a cutover where every consumer updates their queries simultaneously. Good luck with that. In my experience, “simultaneously” means “over the course of several painful weeks, with a growing list of things that are broken in the meantime.”

Expand and Contract takes a different approach, in three phases:

In the Expand phase, you add the new column alongside the old one. Both exist. Both contain the same data. Nothing breaks, because nothing has been removed.

In the Migrate phase, you update consumers one at a time. Dashboard A switches to the new column. Pipeline B switches. Report C switches. Each migration is independent, low-risk, and reversible. There’s no coordination required between teams.

In the Contract phase — and only after every consumer has migrated — you drop the old column. By this point, nobody is using it, so removing it is a non-event.

Does this take more deployments? Yes. it might require up to five deployments before the change is fully in effect. But each deployment is safe. Each deployment is reversible. And at no point does anyone lose access to their data because you renamed a column in a way they weren’t expecting.

For data models specifically, Expand and Contract shines when you’re restructuring dimensions, splitting fact tables, or changing grain. Add the new structure alongside the old one. Migrate consumers gradually. Remove the old structure once it’s no longer referenced. It’s boring. It’s methodical. It works.

The ship that replaced itself

If you want a single case study that demonstrates what successful incremental replacement looks like at scale, look at what Slack did with their desktop client in 2019.

Slack’s engineering team faced a classic dilemma. Their desktop application — built on jQuery and a custom framework — had accumulated years of complexity. It was slow. It used too much memory. The architecture made certain improvements difficult or impossible. Every instinct said: rewrite it in React.

Their engineering blog explicitly acknowledged the risk: running code knows things. Hard-won knowledge gained through billions of hours of cumulative usage and tens of thousands of bug fixes.

So instead of a big-bang rewrite, they did something much harder and much smarter. Over two years, they replaced every component of the desktop client while shipping continuously. They called it modernising “bit by bit” — enforcing strict interfaces between existing and modern code, shipping every change to production as it was completed, and never asking users to endure a disruptive cutover.

The results: 50% memory reduction, 33% load time improvement. Users never experienced a broken release. Had they waited until the entire application was rewritten before releasing it, their users would have had a worse experience for years while the team worked in a vacuum.

Now, Slack is an application, not a data warehouse. But the principles translate directly. Strict interfaces between old and new code? That’s the Legacy Façade. Shipping continuously? That’s WAP combined with Expand and Contract. Never asking users to endure a disruption? That’s the entire point.

The ancient Greeks had a thought experiment about this: if you replace every plank of a ship over time, is it still the same ship? Philosophers argue about it. Engineers don’t care. The ship still floats. The passengers still get where they’re going.

The numbers, for the sceptics

I held back on the quantitative evidence in Part I because I wanted to make the argument from experience and conviction. But some people want receipts, so here they are.

McKinsey and Oxford University studied 5,400 IT projects with initial budgets exceeding $15 million. On average, large projects ran 45% over budget and 7% over time, while delivering 56% less value than predicted. Seventeen percent were “black swans” — cost overruns exceeding 200% that threatened the company’s viability. One retailer abandoned a $1.4 billion IT modernisation, failed on a $600 million follow-up, and filed for bankruptcy.

The Standish Group has been tracking IT project outcomes since 1994. In their original CHAOS report, only 16.2% of projects succeeded on time, on budget, with specified features. Average cost overrun: 189%. By 2020, the overall success rate had improved to 31% — but here’s the statistic that matters most: small projects succeeded approximately 90% of the time. Large projects succeeded less than 10% of the time.

Read that again. Small projects: 90%. Large projects: under 10%.

That’s not a technology problem. That’s a scope problem. And the single most reliable way to reduce scope is to stop rebuilding entire systems and start refactoring them one piece at a time.

For data systems specifically, Gartner reports that more than 50% of data warehouse projects fail to reach user acceptance. Other analyses put the number closer to 80%. CDInsights reports that 70% of data warehouse modernisation projects exceed budget or fail. Integrate.io found that approximately 80% of data migration projects exceed timelines or budgets, with large-scale projects showing 50% higher failure rates than incremental approaches.

I don’t know how much clearer the data can be. Big-bang rewrites of data platforms fail most of the time. Incremental approaches succeed most of the time. The methodology I’ve described in this article — strangler fig, WAP, dbt refactoring, expand and contract — isn’t just safer or more philosophically sound. It’s what the evidence says actually works.

Refactoring as a practice, not a project

Refactoring should not be a project. It should not have a start date and an end date and a steering committee and a Gantt chart. Refactoring should be a practice — something your team does continuously, as a natural part of how you work with your data models.

Every time you touch a model to add a feature or fix a bug, you leave it a little cleaner than you found it. You extract a hardcoded value into a reference table. You add a test that didn’t exist before. You rename a column from col_7 to something a human can understand. You break a 200-line CTE into two smaller, focused ones.

None of these changes are dramatic. None of them require a business case or a migration plan. They’re small, safe, incremental improvements that compound over time. A year from now, the model is cleaner, better-tested, and easier to understand — not because you stopped everything to rebuild it, but because you improved it a little bit every time you were in the neighbourhood.

This is what Fowler meant when he defined refactoring as changing internal structure without modifying observable behaviour. It’s not a phase. It’s a discipline. And it’s the discipline that separates data teams that thrive from data teams that periodically torch everything and start over every three years, wondering why they never seem to make lasting progress.

The closing argument

In Part I, I said your data model isn’t broken — it’s battle-scarred, and those scars are knowledge. In Part II, I’ve tried to show you what the alternative to demolition looks like: patient, methodical, verified improvement. Strangler figs instead of bulldozers. Write-Audit-Publish instead of deploy-and-pray. Expand and contract instead of coordinate-and-hope.

It’s not glamorous work. Nobody’s going to invite you to give a conference talk about the time you renamed some columns and extracted a few staging models. You won’t get a promotion for migrating a legacy fact table so smoothly that nobody noticed it happened.

But that’s exactly the point. The best infrastructure work is invisible. The best migration is the one where the business never had to care. The best data model is the one that quietly absorbed twenty years of business complexity and still answers questions accurately at 7am on a Monday when a Department Head needs numbers for a board meeting.

I wrote Part I of this series last week. The technology has changed — Netscape to Snowflake, CGI scripts to dbt — but the lesson hasn’t. Old code that works is more valuable than new code that doesn’t exist yet. Embedded knowledge is harder to create than clean architecture. And the patient, unglamorous work of incremental improvement will always outperform the seductive fantasy of starting fresh.

Your data model isn’t broken. Stop trying to replace it. Start making it better.

You Don't Need Permission to Fix Your Data

Chris Hillman — Sat, 21 Mar 2026 09:00:00 +1100

Let me tell you about a junior engineer called Sam.

Sam had been on the team about four months when I noticed something in a pull request. Tucked between two routine model changes was a new schema.yml entry — five accepted_values tests on a column called customer_status that had been silently accumulating fourteen different spellings of “active” for the better part of a year.

Nobody asked Sam to do this. It wasn’t in a sprint. There was no Jira ticket. Sam had just been working in that part of the warehouse, noticed the mess, and decided to clean it up on the way through.

I left a comment on the PR: “Nice catch.” Sam’s reply was almost apologetic — “Hope it’s okay that I added these, I wasn’t sure if I was supposed to.”

That response stuck with me. I wasn’t sure if I was supposed to.

Here was someone who’d spotted a real problem, built a real solution, tested it, and shipped it — and their instinct was to wonder whether they had permission. Not whether the fix was correct. Not whether the tests were well-written. Whether they were allowed.

I’ve thought about that moment a lot since, because it captures something I see in almost every data team I work with. The people closest to the problems — the ones running the queries, staring at the nulls, fielding the Microsoft Teams messages when dashboards look wrong — are often the last to believe they can do anything about it. Not because they lack skill. Because somewhere along the way, they absorbed the idea that fixing things is someone else’s job.

That belief is expensive. And it’s wrong.

I’m telling you this because data quality — the real, unglamorous, column-by-column work of making data trustworthy — is one of those things that sounds like a senior engineering problem until you see a four-month-old team member fix something a dozen experienced engineers walked past every day. It doesn’t take authority. It takes the willingness to act before you’re told to.

This article is about how to be more like Sam. Five tactics you can start this week — no permission required — backed by real case studies from companies like Airbnb, Google, and Warner Bros. Discovery that figured this out the hard way.

The $12.9 million reason your quality problem isn’t “too small” to fix

Before we get into tactics, let’s talk about why your “small” fix matters more than you think.

Gartner pegs the average annual cost of poor data quality at $12.9 million per organisation. IBM and Harvard Business Review put the aggregate U.S. cost at $3.1 trillion annually. These numbers feel abstract until you realise what they actually represent: knowledge workers spending roughly half their time hunting for data, correcting errors, and searching for confirmatory sources instead of doing their actual jobs.

Monte Carlo’s 2024 State of Data Quality survey found bad data now impacts 31% of company revenue on average — up from 26% just two years earlier. Their 2022 survey found data engineers spend 40% of their time evaluating or checking data quality. Soda’s research put that number even higher: 61% of data engineers spend half or more of their time handling data issues.

And the human cost? A DataKitchen survey of 600 data engineers found 87% are frequently blamed when things go wrong with data. 78% wish their job came with a therapist.

The implication for you: the problem is so large and so costly that any measurable improvement you create has real dollar value attached to it. That null check you added last Thursday? That’s not a minor fix — it’s a tiny piece of a multi-million-dollar problem that nobody else bothered to address.

Tactic 1: write tests for your own models without asking anyone

The single best case study for this tactic comes from Airbnb, and it didn’t start with a VP mandate or an OKR. It started with one engineer.

Airbnb’s engineering blog documents how they went from “an engineering culture in which most new code shipped without a single test” to one where proposing a change without tests got called out immediately. The transformation wasn’t top-down. As the blog explains: “One approach is to rule by edict, but given our culture of engineer autonomy this would have been very poorly received. The other approach is to lead by example and build a movement.”

That single champion’s playbook was remarkably tactical. They spoke in engineering meetings. They held office hours showing teammates how to write tests. They published a weekly newsletter highlighting well-written test specs — giving public credit to engineers who included tests. They revamped the testing bootcamp for new hires, effectively making each new hire a champion. They improved tooling, reducing CI build times from over an hour to roughly six minutes.

The result? Airbnb later applied the same grassroots philosophy to data quality, building a DQ Score on a 0–100 scale across their entire warehouse. Their anomaly detection system now prevents quality issues in new pipelines before they reach production. The Minerva metrics platform manages 12,000+ metrics and 4,000+ dimensions with 200+ data producers.

One engineer. No mandate. A newsletter and some office hours.

What this looks like for you

If you’re working with dbt, you already have the tools. dbt ships with four built-in generic tests — not_null, unique, accepted_values, relationships — that you define in YAML alongside your model definitions. No framework to build. No proposal to write. Just add a few lines to your schema.yml:

models:
 - name: dim_customers
 columns:
 - name: customer_id
 tests:
 - not_null
 - unique
 - name: customer_status
 tests:
 - accepted_values:
 values: ["active", "inactive", "churned", "suspended"]
 - name: email
 tests:
 - not_null

That’s it. Next time the pipeline runs, those tests execute automatically. If someone pushes a change that breaks uniqueness on customer_id, the pipeline catches it before it reaches production.

The dbt-expectations package goes further, porting Great Expectations’ advanced tests into dbt. Unit tests — introduced in dbt v1.8 — let you validate SQL logic with static mock inputs before transformations run. Tests integrate directly into CI/CD, blocking PRs with failures from reaching production.

Not on dbt? You can still write assertion queries. Here’s a completeness check you can run against any SQL database — no framework required:

-- Quick completeness check: what percentage of each critical
-- field is actually populated?
SELECT
 COUNT(*) AS total_records,

 -- How many emails do we actually have?
 ROUND(100.0 * COUNT(email) / COUNT(*), 1) AS email_pct,

 -- How many phone numbers?
 ROUND(100.0 * COUNT(phone) / COUNT(*), 1) AS phone_pct,

 -- How many addresses?
 ROUND(100.0 * COUNT(address_line_1) / COUNT(*), 1) AS address_pct,

 -- Flag: are we below the 95% threshold on anything critical?
 CASE
 WHEN COUNT(email) < COUNT(*) * 0.95 THEN 'BELOW THRESHOLD'
 ELSE 'OK'
 END AS email_status
FROM
 main.customers;

Schedule that to run daily. Store the results in a logging table. You’ve just built a monitoring system — and you didn’t ask anyone’s permission.

The numbers back this up

Monte Carlo’s 2022 survey found that organisations conducting at least three types of data tests weekly experienced 46 incidents per month compared to 61 incidents per month for less rigorous testers — a 25% reduction. The Forrester Total Economic Impact study of data observability platforms (April 2025) found 358% ROI, 6,500+ data personnel hours reclaimed annually, and $1.5M+ in avoided lost revenue from reduced data downtime.

Daniel Beach of Data Engineering Central offers a pragmatic path for getting started: begin with end-to-end tests first when inheriting a pipeline with no tests, then add data quality checks, then unit tests. You don’t need to boil the ocean. You need to start.

Tactic 2: document one thing every time you touch it

The numbers on missing documentation are genuinely staggering. According to DX, developers spend 3–10 hours per week searching for information that should be documented. For a 100-person engineering team, that equals 300–1,000 hours weekly — the equivalent of 8–25 full-time engineers doing nothing but hunting for answers. Stack Overflow’s developer survey found 62% of developers spend over 30 minutes daily on poorly documented issues. And 38% listed poor documentation among their top reasons for leaving a company.

One practitioner tracked “Time to First Production Commit” at 28 days for new engineers — versus an industry benchmark of 3–5 days. Engineers on that team spent 30% of their time rediscovering what previous engineers had already learned. 40% of sprint tickets required inter-team clarification, averaging 12 clarifications per ticket.

This is what documentation debt looks like. Not a gap in a wiki — a tax on every single person who touches the codebase.

“Documentation Fridays” don’t work. This does.

Multiple sources confirm that big-bang documentation efforts fail. As one case study puts it: “‘Okay team, let’s spend Friday afternoon documenting our processes!’ Nobody wants to do that. And it doesn’t work.”

The successful pattern is document-as-you-go. Every time you encounter an issue, you write it down. Shopify Engineering treats documentation as a product, explicitly encouraging new team members to fill documentation gaps from day one. Their philosophy: “The most effective way to encourage others to write is to actually write.”

For data teams specifically, the approach is even simpler. If you’re using dbt, column descriptions live in schema.yml alongside your model definitions. dbt’s column-level lineage means passthrough and renamed columns automatically inherit descriptions from upstream models. Write a description once, and it flows downstream through every model that references that column.

Here’s what this looks like in practice:

models:
 - name: dim_customers
 description: >
 One row per customer. Grain: customer_id.
 Source: CRM extract via Fivetran, refreshed daily at 03:00 UTC.
 Owner: data-eng-team@company.com
 columns:
 - name: customer_id
 description: "Primary key. Sourced from CRM.contacts.id. Unique, never null."
 - name: customer_status
 description: >
 Current lifecycle status. Valid values: active, inactive, 
 churned, suspended. Updated by CRM webhook on status change.
 Known issue: legacy records from pre-2023 may contain 
 'Active' (capitalised) — normalised in this model.
 - name: lifetime_value
 description: >
 Cumulative revenue attributed to this customer, in AUD.
 Calculated as SUM(order_total) from fct_orders.
 Excludes refunded orders. Updated daily.

That “Known issue” note on customer_status? That’s worth its weight in gold to the next person who touches this model. It took thirty seconds to write and it’ll save hours of confusion.

Microsoft’s engineering blogs describe the alternative bluntly: “When tribal knowledge holders leave, reverse engineering is the only way to understand what was done.” Enboarder’s 2025 research found 47% of organisations cite institutional knowledge loss as their top offboarding challenge. Atlan’s Modern Data Survey found data professionals waste one full day per week simply trying to figure out what data to use.

You don’t need to document the entire warehouse. You need to document the thing you touched today. Tomorrow, document the next thing. In three months, you’ll have built something genuinely valuable — and you’ll have done it one commit at a time.

Tactic 3: build a quality dashboard nobody asked for

At Warner Bros. Discovery, the DICE team built data quality management frameworks and created a “Data Quality Forum” that became a bridge between teams. During the Olympics livestream, they used custom SQL checks to detect missing content metadata — fixing issues before they broke reporting. The forum evolved into what Pam Zirpoli called a “cultural flywheel.” Patterns surfaced in the forum were incorporated into onboarding, translated into new monitors, and used to refine their priority matrix. Teams shifted from reactive — reacting after dashboards broke — to proactive.

At Tempus, a data engineer documented how adopting Elementary (an open-source dbt package) transformed their monitoring. Before: “test alerting relied on a dashboard in Data Studio built on top of a log sink, and the ‘alerts’ were a daily email that you actually had to open.” After: Microsoft Teams alerts that “@-ed a user and continued to do so until they’re turned off.” The key outcome: tests no longer fail silently.

The six metrics that actually matter

Drawing from IBM, dbt Labs, Databricks, and Alation, the consensus metrics for data quality monitoring are:

Freshness: time since last data update. If your orders table hasn’t been updated since yesterday, something’s broken.
Volume: expected versus actual row counts. Did today’s load bring in 50,000 rows when you normally get 48,000–52,000? Fine. Did it bring 12? Problem.
Schema changes: added, deleted, or altered columns. These should never surprise you.
Completeness: null rates per column. Simple COUNT(*) vs COUNT(column) tells you a lot.
Distribution shifts: statistical changes in data values. If the average order value suddenly doubles, that’s worth investigating.
Test pass/fail rates: the percentage of your quality assertions that are passing. Track this over time.

Here’s a freshness check you can run right now — no tools, no frameworks, just SQL:

-- How stale is each of your critical tables?
-- Run this daily and store results in a monitoring table
SELECT
 'fct_orders' AS table_name,
 MAX(updated_at) AS last_record_timestamp,
 CURRENT_TIMESTAMP AS checked_at,
 ROUND(
 EXTRACT(EPOCH FROM (CURRENT_TIMESTAMP - MAX(updated_at))) / 3600.0,
 1
 ) AS hours_since_update,
 CASE
 WHEN MAX(updated_at) IS NULL THEN 'CRITICAL - No data'
 WHEN CURRENT_TIMESTAMP - MAX(updated_at) > INTERVAL '24 hours'
 THEN 'STALE - Exceeds 24hr SLA'
 WHEN CURRENT_TIMESTAMP - MAX(updated_at) > INTERVAL '12 hours'
 THEN 'WARNING - Approaching SLA'
 ELSE 'FRESH'
 END AS freshness_status
FROM analytics.fct_orders;

Store that in a data_quality_log table, connect it to Looker Studio or Metabase (both free), and you’ve got a monitoring dashboard. That’s it. No Kubernetes cluster. No proposal document.

Visibility changes behaviour

Anna Geller documented how sending data quality alerts to a shared Microsoft Teams channel transforms team behaviour. Her core insight: “As social creatures, we are more motivated to fix issues if other people can see the effort we put into it.” Data owners reply with root cause analysis, add checkmarks, or link tickets — creating a social proof loop that issues are no longer ignored.

This isn’t just anecdotal. Research on the Hawthorne effect shows healthcare hand hygiene compliance increases by up to 70% when workers know they’re observed, and employees increase productivity by an average of 13% when activity is monitored. Making data quality visible doesn’t just inform people — it changes their behaviour.

Tactic 4: create a #data-bugs channel

MIT Sloan Management Review’s HelloFresh case study documents one of the best examples of how visibility catalyses organisational change. HelloFresh transitioned through three modes of data quality management. In Mode 1, issues were dealt with reactively by whoever happened to be affected. In Mode 2, a central engineering team handled cleanup but lacked understanding of how data was actually used — “data customers grew increasingly frustrated as they continued to spend large fractions of their workdays dealing with data that did not meet their needs.”

The transition to Mode 3 — proactive, collaborative quality management — happened precisely because “quality issues became increasingly visible.” Not because someone wrote a strategy doc. Not because a VP mandated it. Because the problems became impossible to ignore.

You can accelerate that transition. Create a Microsoft Teams channel. Call it #data-bugs or #data-quality or whatever feels natural. Start posting issues you find — with specifics. Not “the data looks wrong” but “fct_orders has 3,247 rows with null customer_id from the 2026-02-15 load, affecting the weekly revenue dashboard.”

That’s it. You’ve just created the visibility mechanism that HelloFresh spent years arriving at.

The broken windows effect is real — and it’s been measured

The Pragmatic Programmer introduced the “broken windows” software analogy decades ago: “Don’t leave ‘broken windows’ (bad designs, wrong decisions, or poor code) unrepaired. Fix each one as soon as it is discovered.”

This isn’t just metaphor anymore. A 2024 arXiv paper by Diomidis Spinellis et al. analysed 2 million code commits across 122 projects comprising 5.5 million lines of code. The key finding: “History matters — developers behave differently depending on some aspects of the code quality they encounter.” Developers tailor the quality of their commits based on the quality of the file they’re committing to. Low quality begets lower quality. High quality attracts higher quality.

Mat Ryer extends this to everyday engineering: “If you work on a project that has flaky tests, then you’re more likely to add more flaky tests. If there is a hacky design, you’re more likely to hack more in.”

Fixing one visible data quality issue — or even just making it visible — can interrupt the decay spiral.

Why juniors don’t speak up (and why that’s exactly the problem)

Here’s the uncomfortable part. A Culture Shift UK survey found junior colleagues were twice as likely (54%) as senior leaders (27%) to say speaking up about issues is “pointless.” 37% said speaking up “isn’t worth the personal risk.” A Blind survey found 58% of tech workers experience imposter syndrome.

Google’s Project Aristotle studied 180+ teams over two years and found psychological safety was “by far the most important” of five dynamics for team effectiveness. Amy Edmondson’s foundational research contains a striking finding: while studying hospital teams, she expected high-performing teams to report fewer errors. Instead, better teams reported MORE errors. Her insight: “Maybe the good teams don’t make more mistakes, maybe they report more.”

This is the paradox. The teams that look like they have the most data quality issues are often the healthiest — because they’ve created an environment where problems surface instead of festering.

Etsy pioneered blameless postmortems under CTO John Allspaw. Engineers send company-wide emails confessing mistakes. Etsy even gives out an annual “three-armed sweater” award to the employee who made the most surprising error. As a Hadley Wickham–curated Stanford reading list notes, while blameless postmortems are standard in DevOps, they “have not yet infiltrated standard data science practices.” That’s a gap you can help fill.

Creating a #data-bugs channel isn’t just about tracking issues. It’s about establishing the norm that finding problems is good work, not a sign that something’s wrong with you.

Tactic 5: ship an example, don’t write a proposal

Marcus Blankenship’s article on learned helplessness in software engineering — 96,000+ reads, 900+ Reddit comments — contains a story about a coworker called Milind. Milind wasn’t a manager. He didn’t have a fancy title. But he was an informal leader through pure action. He asked questions about why particular approaches were taken. He admitted ignorance in group settings. He insisted the team understand root causes before moving on.

Blankenship’s observation: “Milind changed our environment by his actions… He showed me that I had much more power than I thought, if I would only stop expecting to be spoon-fed everything. His actions showed me that true leadership isn’t granted, it’s grasped.”

A junior engineer on Irina Stanescu’s newsletter shared something similar: “Even as a junior engineer, I remember influencing my chief architect and brought Terraform into my organisation 7 years ago by simply pointing out a common pain point and how much longer things took without managing our infrastructure in code.” They added: “Being able to influence without authority is a very underrated skill, and I believe this is what helped me get promoted to senior engineer even more than my technical skills.”

The pattern is consistent. Don’t propose a data quality framework. Don’t write a 15-page RFC. Build the thing. Make it work. Show people.

Why working examples spread faster than proposals

Everett Rogers’ Diffusion of Innovations theory identifies five characteristics that determine how fast an idea spreads: relative advantage, compatibility, complexity (lower is better), trialability, and observability. A working example maximises both trialability and observability — the two factors most under your control as a junior engineer.

A proposal says “we should do this.” A working example says “look, I already did this, and here’s what happened.” One requires people to imagine the benefit. The other puts the benefit directly in front of them.

Addy Osmani, after 14 years at Google, captured the complementary lesson: “Early in my career, I believed great work would speak for itself. I was wrong. Code sits silently in a repository.” You need to build the example and make its impact visible. Post in Microsoft Teams. Show the before and after. Quantify what changed.

The junior advantage nobody talks about

Ant Weiss’s widely-read article on organisational learned helplessness identifies common symptoms: “This is how we were told to work,” “These are the limitations of the system,” “We don’t have the resources.” His core observation: “No engineer starts their career willing to be unproductive, pessimistic and stressed out. But the companies they work at teach them to.”

The cure, drawn from Seligman’s original research, is critical: “Threats, rewards, and observed demonstrations had no effect on the ‘helpless’ dogs. The dogs had to be physically moved through the escape action at least twice before they would do it themselves.” You can’t just tell engineers things can be better. You have to walk them through the experience of making something better.

Here’s the part that should give you confidence. JazzTeam reports a remarkable finding: “To prove that any problem can be solved, our team many times gave a task to Junior and even Intern Engineers who did not have psychological fears and tunnel thinking.” Juniors, unencumbered by years of “that’s impossible” conditioning, were able to tackle problems seniors had declared unsolvable.

Your fresh eyes aren’t a weakness. They’re an asset. You haven’t yet learned that “it can’t be done.” So go do it.

Putting it all together

None of these tactics require a committee meeting. None require architectural approval. None require you to be senior.

Here’s what they do require: the willingness to act before you’re told to.

Write a test this week. Add a column description tomorrow. Build a freshness check and schedule it. Create a Slack channel and post the first issue. Turn one of your quality checks into a reusable example your team can copy.

DORA’s research — surveying 39,000+ professionals — finds that with focused effort and the right tooling, teams typically see measurable improvements within 1–3 months. A GitHub engineer who went from junior to mid-level in 2.5 years described the compound effect: “If I got stuck on some undocumented functionality, I made sure to update the docs and let the team know. Tackling a tricky bug requiring cross-team collaboration? I’d summarise everything we discovered so that it would be easier for others later. Post about it in Microsoft Teams and highlight its impact.”

Kevin Workman, a former Google engineer, captured the mindset that makes all of this work: “Realising that I had agency and even a responsibility to advocate for my ideas is one of the most important lessons I learned on my journey to becoming a ‘senior’ software engineer… Some of the most interesting and most successful moments of my career have come from advocating for changes I thought nobody would agree to.”

I think about Sam sometimes — that four-month-old team member whose instinct was to apologise for fixing something. The tests Sam added that day caught three data quality issues in the following month. Other engineers started adding their own tests alongside model changes. Six months later, the team had 200+ automated quality checks running on every pipeline, and the weekly “data looks wrong” Microsoft Teams messages had dropped to nearly zero.

Sam never got a mandate. Never wrote a proposal. Never waited for Q3.

The data quality problem in your organisation is a $12.9 million issue wearing a “we’ll get to it in Q3” disguise. And you — yes, you, the person who’s been here five months and noticed fourteen spellings of “active” — are exactly the right person to start fixing it.

Not because someone gave you permission. Because the data doesn’t care about your title.

Your Friends Will Be There for You. Your Work Won't.

Chris Hillman — Wed, 18 Mar 2026 09:00:00 +1100

There’s a table in a rented house in Queenscliff — nothing fancy, just whatever furniture came with the place — with a hand-drawn map spread across it, dice scattered at the edges, and snacks slowly migrating toward the centre of the board.

Once a year, Steve, one of our friends, organises for us to make a trip down to Queenscliff to play tabletop RPGs for a weekend. More recently, we’ve started playing monthly or fortnightly at home too. Nothing glamorous about any of it. We’re not Glass Cannon Network or Critical Role — no cameras, no production values, no audience watching us roleplay with solemn gravity.

We’re just a bunch of friends. Laughing at the missed sword swing. Losing our minds when someone makes a completely unreasonable jump and somehow lands the killing blow on a copper golem with a natural twenty. Rolling dice and arguing about the rules and eating too much while the world outside keeps spinning without us.

I’ve been thinking about why it matters so much to me.

It’s not the game, really — or at least, the game isn’t the point. It’s what the game creates. A ritual. An excuse to sit in the same room for a weekend and actually be present with people I care about. A context where you ask “how are you going?” and actually mean it, and stay long enough to hear the real answer. The dragon we’re fighting is almost beside the point. What matters is being there for each other.

When you’re up at 2am resolving a critical data pipeline failure, your employer will not remember that next year. They might not remember it next month. The organisation absorbs it, thanks you if you’re lucky, and moves on to the next incident.

But fighting dragons with your friends — showing up for a weekend in Queenscliff year after year — those become stories. Stories that become memories. Memories that become the fabric of who you are to each other, and who they are to you.

Your friends will be there for you. Your work won’t.

And yet.

When was the last time someone you loved lost someone, and you climbed into bed next to them — not to fix it, not to say the right thing, just to be there in the dark with them? When something hard was happening in your own life, did you pick up the phone and call a friend? Or did the thought cross your mind — “they’re busy, they don’t have time to listen to me” — and you put the phone down and carried it alone?

Do you have a friend you’d call when you got the promotion — someone who’d hear the news and mean it when they say “that’s great”? Not quietly threatened by it. Not performing enthusiasm. Just genuinely proud of you, because your win is their win. That kind of friend is rarer than it should be.

Those moments. That’s what friendship actually is. Not the catch-up drinks you schedule six weeks out and cancel twice. The showing up in the hard moments. The willingness to burden someone with your struggle, and the willingness to carry theirs.

We don’t build trust by offering help. We build trust by asking for it. The act of calling someone when things are falling apart — not suffering quietly, not performing fine — is the thing that makes the friendship real on both sides. It’s not a burden. For the right person, it’s an honour.

This is something I’m still learning. I want to talk about what it actually means to be a good friend — because I think most of us, if we’re honest, have let that muscle go slack.

Are You Actually a Good Friend?

Most people, if you ask them directly, will say yes. Of course they’re a good friend. They’re there for the people they care about.

But ask yourself: do you call your friends on their birthday — actually call, sing happy birthday — or do you post on their Facebook because you saw the notification and everyone else was doing it? When a friend is going through something hard, do you show up? Not message. Show up. Do you say “I love you” to the people in your life that you love?

Most of us, if we’re being honest, have outsourced friendship to the low-effort channel. The emoji response. The “we should catch up soon!” comment thread that leads nowhere. We maintain the social graph without maintaining the relationships.

But when you’re in a dark place, who do you call? And more confronting — who would call you?

The answer to that second question is the real measure. Friendship isn’t what you feel for someone. It’s what you demonstrate over time. It’s the accumulated weight of showing up — for the birthday dinner that falls on a work night, for the hospital waiting room at 7am, for the phone call that starts “I just need to talk to someone” at an inconvenient hour.

Those moments don’t happen automatically. They have to be built. And they have to be built before you need them, not when you’re reaching for them in a crisis.

When our daughter Natasha was born, she wouldn’t latch on, and breastfeeding wasn’t working. Lena and I tried everything, exhausted and running on no sleep, seriously weighing up whether to just switch to formula. At some point we gave up trying to solve it ourselves and called our friends Tom and Rachel. We ended up talking for somewhere between four and six hours straight. We didn’t need advice, really — we needed to not be alone with it. And something about that phone call, about calming down, settled Natasha too. By the end of it she was feeding. I still think about that night. We’d been treating a human problem like a technical problem, trying to fix it in isolation, when all we’d actually needed was to pick up the phone.

What Women Already Know

Women are better at friendship than men. Not universally, not as a stereotype — but as a pattern that shows up consistently enough to be worth talking about honestly.

The research on women’s friendship patterns tells part of the story. Women’s friendships tend to orient toward direct emotional disclosure — what psychologists call face-to-face friendship. Personal sharing, vulnerability, actual conversation about what’s happening in your life. Men’s friendships tend to be shoulder-to-shoulder — bonding through shared activity, talking while doing something else together. Neither mode is better. But when the activity disappears — when the sport stops, when the shared project ends, when the workplace changes — shoulder-to-shoulder friendship has nothing to stand on. Face-to-face friendship survives the context change because the context was never the point.

What women understand, and what most men have to learn later and harder, is that the friendship is the thing. Not the activity. The person.

The good news is that the shoulder-to-shoulder model doesn’t have to disappear. The Queenscliff table is shoulder-to-shoulder. The hack is building ritual around it — making the activity a recurring container for the friendship, rather than the friendship being a side effect of the activity. When you have the ritual, the friendship survives when everything else changes.

I saw this firsthand when I was working in India. Some colleagues (Sarasa, Nidhi, Abhilash, Deepak) took me out to play badminton — I’d never played before, and I was genuinely terrible. But what struck me wasn’t the sport. It was that this group did it every week, without fail. They weren’t there to compete or keep score. They were there to be together, to unwind, to laugh at each other and at themselves. What I remember is leaving that court feeling like I’d glimpsed something. These people had built a ritual, and the ritual had built them.

What the Harvard Longevity Study Actually Found

Harvard physician Arlie Bock began following the lives of a few hundred Harvard sophomores. The study is now 85 years old, has followed over 1,300 people including the children of original participants, and is the longest-running study of adult development in history.

The fourth director of the study, psychiatrist Robert Waldinger, gave what became one of the ten most-watched TED talks ever. And the headline finding, distilled from nearly nine decades of data, is deceptively simple:

Good relationships lead to health and happiness. Not wealth. Not professional achievement. Not optimised nutrition or fitness metrics. Relationships.

Waldinger and his co-author Marc Schulz published the full findings in The Good Life in 2023. Two things from that research hit particularly hard.

First: relationship satisfaction at age 50 was a better predictor of physical health at 80 than cholesterol levels. That’s the kind of finding that should make every analytically-minded person stop and recalibrate. We spend enormous energy optimising for things we can measure while ignoring the metric that turns out to be most predictive of how we age and how long we live.

Second: when participants reached their 80s, their biggest regret was almost universally the same. Too much time at work. Not enough time with the people they loved. And their proudest achievements were almost entirely relational — being a good parent, a good friend, a good partner, a good mentor.

George Vaillant, who directed the study for over three decades, summarised it: “When the study began, nobody cared about empathy or attachment. But the key to healthy aging is relationships, relationships, relationships.”

What We Actually Traded

Reflecting on how high achievers use the word “sacrifice”. When we say we’ve sacrificed something for our career, we shouldn’t be afraid to put a name to who that sacrifice was. Because often it was the people in our lives that we call friends.

Put a name to it.

We’re not sacrificing some abstract concept. We’re sacrificing specific people. Former colleagues whose numbers we keep but never dial. School friends who live in the same city and see us once a year if they’re lucky. The people who would drop everything if we called — but who we don’t call, because we’re heads-down on the next deliverable.

So many of us have cancelled on friends because a meeting came up, telling ourselves they’ll understand. Yet the reverse almost never happens — we wouldn’t reschedule a meeting for a friend. We’ve quietly trained ourselves to treat friendship as the flexible commitment, the one that can move.

And the friendship debt is unlike technical debt — you can’t see it slowing you down, it doesn’t show up in velocity metrics or retrospectives. It just quietly accumulates until something breaks. The incident you pushed through alone. The difficult conversation at home that had nowhere to go because there were no friends to process it with. The creeping sense that despite being deeply capable and professionally respected, you’re not quite sure who you’d call.

Ubuntu: A Different Operating System

There’s a Zulu/Nguni philosophy called Ubuntu. The phrase “Umuntu ngumuntu ngabantu” translates roughly as “a person is a person through other people.” You’ll also hear it rendered as “I am because we are.”

Desmond Tutu described Ubuntu this way: “In our African worldview, we need other human beings for us to learn how to be human. For none of us comes fully formed into the world. The solitary human being is a contradiction in terms.”

He contrasted it explicitly with Descartes: “It is not ‘I think therefore I am.’ It says rather: ‘I am human because I belong. I participate. I share.’”

Most of the professional identity we build in data engineering runs on Descartes. I architect, therefore I am. I optimise the pipeline, therefore I am. I resolved the incident, therefore I am. We define ourselves by what we produce, alone, with headphones on, in a flow state that the rest of the world is just interrupting.

Ubuntu is a different operating system. One where your identity is constituted by your relationships — where the question “who are you?” is answered not by your job title or your GitHub commit history, but by who you show up for and who shows up for you.

The longest-running study of adult development in history arrived at the same conclusion Tutu was describing. Every major piece of research on social connection and mortality found it too. The architecture most of us are running on isn’t wrong in a subtle way. It’s wrong in the foundational way.

Mark Shuttleworth — South African entrepreneur — named the Ubuntu Linux operating system after this philosophy in 2004, explicitly to emphasise community, open-source contribution, and the idea that the work is better when people build it together. Engineers understood that intuitively, because it maps to how the best software actually gets built. It also maps, as it turns out, to how the best lives get built.

The Dragon Will Be There Next Year

There’s a table in a rented house in Queenscliff, with a hand-drawn map on it and dice scattered at the edges.

Every year, we make the trip. We gather around a table and roll dice and laugh at the missed sword swing and the impossible jump. The dragons change. The characters level up. The rules get argued over and amended and house-ruled into something nobody outside the group would recognise.

But the people are constant. And the ritual is the point.

It is more amazing to have an extraordinary experience with someone than by yourself. You can go somewhere alone and say “look what I did” — versus “do you remember that time we did that?”

The 2am pipeline failure you resolved will be forgotten. By you. By your employer. By the Jira ticket that gets closed and archived.

The dragon you fought with your friends — the copper golem that went down to a natural twenty on an impossible jump — that becomes a story. The story becomes a memory. The memory becomes part of who you are to each other and who they are to you.

That’s not sentimentality. That’s the mechanism by which good lives actually get built. Eighty-five years of longitudinal data says so.

Researcher Robin Dunbar found that building a close friendship takes roughly 200 hours of time together. Those hours don’t come from nowhere. They come from showing up, again and again, at the table. From protecting that time the same way you’d protect a critical production window. From understanding that the maintenance cost of friendship is the highest-ROI investment available to you as a human being.

Your work will absorb your best years and move on when it’s convenient. Your friends will still be at the table.

Put a name to what you’re protecting.

Then protect it.

Your Data Model Isn't Broken, Part I: Why Refactoring Beats Rebuilding

Chris Hillman — Sat, 14 Mar 2026 09:00:00 +1100

In the early 2000’s - Netscape’s decision to rewrite their browser from scratch was the single worst strategic mistake a software company could make.

At the time, Netscape was winning. They had the dominant browser. They had market share. They had momentum. And then they decided the codebase was too messy, too tangled, too hard to work with — so they threw it all away and started over. Navigator 4.0 became the foundation for a rewrite that would eventually ship as version 6.0. There was no 5.0. Three years of development. No shipping product. And while Netscape’s engineers were busy building their beautiful new browser in a vacuum, Internet Explorer ate their lunch, their dinner, and most of their market share.

It’s haunted me ever since: old code isn’t ugly because it’s bad. Old code is ugly because it works. Every strange condition, every seemingly redundant check, every patch that makes a new developer wince — those are battle scars. Each one represents a bug that took weeks to find in production, a customer workflow nobody anticipated, or an edge case that only surfaces on the third Tuesday of months ending in “R.”

I think about that every time I hear a data team say “let’s just rebuild it.”

The sentence that starts every failed data project

I’ve been on data teams long enough to recognise the pattern. It always starts the same way. Someone — usually someone new, often someone senior — opens a dimensional model they didn’t build, scrolls through a few hundred lines of transformation logic, and says the words that should make every data leader’s blood run cold:

“This is a mess. We need to start from scratch.”

And look, I get it. I’ve felt that impulse myself, and likely have said the words myself. You open a fact table with 200 columns. You find bridge tables that reference other bridge tables. You discover a slowly-changing dimension nested inside another slowly-changing dimension, and you think: who built this? What were they thinking?

But here’s what I’ve learned the hard way, across multiple teams and more warehouse migrations than I care to count: they were thinking about the business. That bizarre WHERE clause filtering out both “Unknown” and “unknown”? That’s not sloppy code. That’s a case-sensitivity bug someone found in production data from a source system that nobody controlled. The seemingly redundant join that adds three seconds to your query? It handles a quarterly reconciliation edge case that cost the finance team two days of manual work before someone encoded the fix.

That fact table with 200 columns isn’t a design failure. It’s an accurate representation of a business that has 200 things it needs to measure. The “clean” replacement model will eventually have 200 columns too — they’ll just have different names and it’ll take you eighteen months to figure out why you need them all.

What data teams keep forgetting

This isn’t about browsers or even about code quality. It was about knowledge. When you throw away a codebase and start fresh, you’re not just discarding syntax. You’re discarding years of accumulated understanding about how the real world actually behaves — understanding that was earned through production incidents, user complaints, and painful debugging sessions.

Every collected bug fix in that old code represents something learned. Each fix might be just one line, a couple of characters even, but a lot of work and time went into figuring out those two characters were needed. And that knowledge — the why behind the fix — almost never makes it into documentation. It lives in the code itself, or it lives nowhere.

This is doubly true for data systems. A software application has unit tests, integration tests, user acceptance testing. A data model has… what exactly? Row counts? Spot checks? A business user who eyeballs the dashboard and says “yeah, that looks about right”? The knowledge embedded in a mature data warehouse is far more fragile than application code, because the testing infrastructure around it is almost always weaker. Throw it away and you’re not just rebuilding a codebase — you’re rebuilding an institutional memory that was never written down in the first place.

Fred Brooks saw this coming fifty years ago. His “Second System Effect” from The Mythical Man-Month describes what happens when an engineer builds their second version of something: they over-design it. All the features they wisely deferred from the first version, all the architectural improvements they dreamed about, all the “if only we’d done it this way” ideas — they dump everything into the replacement. Brooks noted that a designer’s first system tends to be spare and clean because they know their limitations. The second system becomes a dumping ground for ambition.

Sound familiar? Every data warehouse rebuild I’ve witnessed follows this arc. The team doesn’t just want to replicate what exists — they want to add a semantic layer, implement a medallion architecture, introduce data contracts, add real-time streaming, switch to Data Vault, build a self-serve analytics platform, and migrate to a new cloud provider. All at once. In a single initiative.

And then they wonder why, two years later, they’ve delivered nothing and the business is still running reports off the “legacy” warehouse that was supposed to be decommissioned eighteen months ago.

The Teradata-to-Snowflake reality check

If you want to see the rebuild-vs-refactor debate play out in real time, watch a Teradata-to-Snowflake migration. The pattern is remarkably consistent.

The pitch is compelling: Snowflake is cheaper, scales elastically, separates compute from storage, and runs standard SQL. Moving from Teradata should be straightforward — convert the SQL, move the data, validate the results. Easy, right?

Roland Wenzlofsky, a Snowflake Solutions Architect, wrote about this in early 2026 and his observations line up perfectly with what I’ve seen. Most organisations approach these migrations as a translation exercise. The technical migration succeeds. Then the first quarterly cloud bill arrives at double the projected budget. Dashboards that loaded in seconds now queue for twenty minutes during batch windows. Pipelines that ran in 45 minutes on Teradata take five hours on Snowflake.

The problem isn’t Snowflake. The problem is that Teradata professionals carry assumptions into Snowflake that aren’t just incomplete — they’re actively counterproductive. Distribution keys, join strategies, indexing patterns — all the hard-won optimisation knowledge from Teradata becomes a liability in a platform built on fundamentally different architecture.

One of the largest Teradata-to-Snowflake migrations in North America involved 1.5 petabytes across 600 databases and roughly 45,000 objects. Before they could even begin, the team had to inventory every orphan object and document every behavioural difference between platforms. That’s not a “lift and shift.” That’s an archaeological dig.

But here’s what I find most telling: even when the migration succeeds technically, the data quality problems survive the move perfectly intact. William Flaiz documented a healthcare organisation that spent $1.8 million migrating from Siebel to Salesforce. Not a single record lost. Zero downtime. Perfect data mapping. Post-migration, the sales team still couldn’t run accurate pipeline reports. The “opportunity stage” field contained 89 different values for what should have been six standard stages — including 1,247 records with the typo “Closeing Soon.” That typo migrated perfectly to the new $2.3 million platform. As Flaiz put it: when performance is bad, we assume the technology is the limiting factor. Data quality problems are messier. They implicate people, processes, training gaps, and years of accumulated shortcuts.

New platform, same chaos. Because the platform was never the problem.

Chesterton’s Fence, or: why you shouldn’t delete that WHERE clause

There’s a principle in philosophy that every data engineer should have tattooed somewhere visible. G.K. Chesterton wrote it in 1929, and it goes roughly like this: if you come across a fence in the middle of a field and can’t see what purpose it serves, don’t tear it down. Go away and figure out why someone built it. Once you understand the reason, then you can decide whether it still needs to be there.

In data engineering, the fences are everywhere. That WHERE status NOT IN ('Unknown', 'unknown', 'UNKNOWN') clause. That filter excluding records from a specific date range in 2019. The join to a reference table that only has 12 rows and hasn’t been updated in three years. They all look pointless until you remove one and discover that the finance reconciliation breaks, or that a regulatory report starts including test transactions that were supposed to be filtered out, or that a dashboard starts showing a revenue spike from a data migration artifact that happened four years ago.

Hyrum Wright — formerly at Google, now at Adobe — formalised a related idea that’s become known as Hyrum’s Law: with enough users of an API, every observable behaviour will be depended on by somebody, regardless of what the documentation promises. For data systems, this is an absolute nightmare during rebuilds. Your downstream consumers don’t just depend on documented schemas. They depend on output ordering, null handling patterns, timestamp precision, and format quirks that were never specified anywhere. You rebuild the pipeline, and suddenly a report that’s worked for three years breaks — not because the data is wrong, but because the columns come back in a different order and someone’s Excel macro was hardcoded to column positions.

I’ve seen teams spend more time debugging these “invisible contracts” after a rebuild than they would have spent refactoring the original system over two years.

The psychology that makes us do it anyway

So if the evidence against big-bang rewrites is this overwhelming — and it is, from Spolsky to Brooks to McKinsey studies showing that large IT projects run 45% over budget and deliver 56% less value than predicted — why do smart, experienced data teams keep proposing them?

Because the impulse to rebuild isn’t rational. It’s psychological. And once you understand the cognitive biases at play, you start seeing them everywhere.

The planning fallacy is the first culprit. Kahneman and Tversky showed that humans systematically underestimate how long tasks will take, even when they have direct experience with similar tasks that took longer than expected. In controlled studies, only 13% of subjects finished their project by the time they’d assigned a 50% probability of completion. The Sydney Opera House was estimated at AU$7 million and four years; it was delivered at AU$102 million and fourteen years. Every data warehouse rebuild proposal I’ve seen exhibits this same delusional optimism. “Six months, maybe nine.” It’s never six months.

And it gets worse, because of what one writer calls the Achilles Paradox of rewrites: while you’re building the new version, features keep getting added to the old one. The business doesn’t stop generating requirements just because you’ve decided to rebuild. So the target keeps moving. The new system is perpetually six months from matching the functionality of the old one.

Not Invented Here syndrome is the second bias, and it’s the one that nobody wants to admit. Katz and Allen’s research on R&D project groups found that teams with stable composition develop a decreasing ability to absorb ideas from outside the group over time. They start believing they possess a monopoly on knowledge in their domain. The diagnostic sign? What researchers call “thought-terminating clichés” — phrases like “we already tried that” or “our situation is different.”

In data teams, NIH manifests as building custom orchestration frameworks instead of using Airflow. Writing bespoke quality checks instead of adopting Great Expectations or dbt tests. Designing proprietary transformation layers instead of using tools that thousands of other teams have battle-tested. And most relevantly: insisting that the existing data model is fundamentally broken and needs to be replaced with something designed in-house from the ground up.

The new leader rewrite is the third pattern, and it might be the most destructive. A new head of data arrives. They look at the legacy warehouse they’ve inherited. They don’t understand the history behind any of the design decisions. They don’t know about the regulatory edge case that explains the weird date filter, or the source system quirk that necessitates the redundant join. All they see is complexity that they didn’t create — and complexity you didn’t create always looks worse than complexity you did.

So they propose a rebuild. It’s partly strategic — they want to put their stamp on the architecture. And it’s partly genuine — they really do think they can do better. But they’re falling prey to the same illusion Spolsky identified: reading code is harder than writing it, and unfamiliar code always looks worse than it is.

There’s a Stack Overflow survey finding that sits underneath all of this: “feeling unproductive” was the number one cause of developer unhappiness at 45%. Working with legacy systems is slow. It’s frustrating. Progress is incremental and often invisible. A greenfield rebuild, by contrast, feels amazing — for the first few months. You’re making decisions, building things, moving fast. The architecture is clean and the tests pass and the world makes sense. And then reality seeps in, the edge cases accumulate, and eighteen months later you’ve got a codebase that looks suspiciously like the one you replaced.

The complexity was always essential

Fred Brooks drew a distinction that I think about constantly. He separated essential complexity — complexity that’s inherent to the problem domain and can’t be removed by better engineering — from accidental complexity, which comes from poor implementation choices and can theoretically be eliminated.

The truth about mature data warehouses: most of the complexity is essential. It’s not there because the original engineers were incompetent. It’s there because the business is genuinely that complicated.

The transformation handling fifteen date formats? That’s because you have fifteen source systems and none of them agree on how to represent a date. The slowly-changing dimension with Type 2 tracking on attributes that seem trivial? Someone in compliance needed audit trails on those fields after a regulatory inquiry. The bridge table that connects customers to accounts through an intermediate entity that nobody can explain? It models a many-to-many relationship that emerged when the company acquired a subsidiary with a different customer hierarchy.

A rebuild won’t make this complexity disappear. It’ll just redistribute it. Instead of one tangled fact table, you’ll have twelve microservices each handling a piece of the logic. Instead of one confusing WHERE clause, you’ll have business rules scattered across a semantic layer, a transformation layer, and a data quality framework. The total complexity will be identical — or worse, because now it’s distributed across more systems with more failure modes.

Ward Cunningham — the person who coined the term “technical debt” — has said he wishes he’d used the word “opportunity” instead. His original metaphor wasn’t about sloppy code at all. It was about the gap between your code’s current model of the problem domain and your team’s evolved understanding of that domain. Technical debt, properly understood, is learning that hasn’t been applied yet. The legacy data model doesn’t represent failure. It represents the team’s best understanding at the time it was built, plus every correction that production reality demanded afterward.

Refactoring respects this. It asks: which parts of this complexity are essential (keep them, clarify them, test them) and which parts are accidental (remove them incrementally, one safe step at a time)? It preserves institutional knowledge while improving structure. It delivers value continuously instead of asking the business to wait years for a payoff that — statistically, based on everything we know about large-scale IT projects — probably won’t arrive.

So what do you do instead?

I’m going to save the detailed playbook for Part II of this series, but the short version is this: you refactor. Methodically. Incrementally. With tests.

Martin Fowler and Pramod Sadalage proved that database refactoring is possible back in 2006, when conventional wisdom said it couldn’t be done. Their principle was simple: make each change as small as possible, because the pain of integration increases exponentially with the size of the integration.

dbt has turned this into a practical workflow for analytics engineering teams. Bring your legacy SQL in unchanged. Wrap it in a model. Verify the output matches. Then — and only then — start decomposing. Extract CTEs. Introduce staging layers. Build tests. Refactor one model at a time, auditing each change against the original output. It’s not glamorous. It doesn’t let you redesign everything from first principles. But it works, and it works without asking the business to lose access to their reports for six months.

The Strangler Fig pattern, the Write-Audit-Publish pattern, the Expand and Contract pattern — these are all variations on the same theme: replace incrementally, verify continuously, and never throw away working logic until you’ve proved the replacement handles every case the original did.

I’ll dig into each of these in Part II. For now, the point is simpler.

The scars are data

When I look at a legacy data model — really look at it, with patience and without ego — I don’t see a mess anymore. I see a record of everything a business has learned about itself. Every weird join is a relationship somebody fought to understand. Every cryptic transformation is a business rule that was discovered through painful experience. Every seemingly arbitrary filter is a production incident that someone fixed at some point, probably at an hour they’d rather not remember.

The Netscape rewrite remains one of the most studied failures in software history. And yet here we are, in 2026, watching data teams propose the same mistake — just with different technology names. Teradata becomes Snowflake. Informatica becomes dbt. The on-prem warehouse becomes the cloud lakehouse. The pitch changes, but the impulse doesn’t: tear it down, start over, do it right this time.

My view? Your data model isn’t broken. It’s battle-tested. It’s messy because reality is messy. And the right response to inherited complexity isn’t demolition — it’s archaeology. Understand what’s there. Understand why it’s there. Then improve it, one careful change at a time.

The scars in your data model aren’t flaws. They’re knowledge. Treat them accordingly.

12 Steps to Better Data Engineering

Chris Hillman — Sat, 07 Mar 2026 09:00:00 +1100

Let me tell you about the moment I stopped trusting architecture diagrams.

I was three days into a new role, getting up to speed with the data team. Smart people. Modern stack. On paper, everything looked right. They walked me through a beautiful data platform diagram: clean lines, labelled layers, colour-coded domains. It looked like something you’d see in a data conference.

Then I asked a question that changed everything: “Can you rebuild your finance table from scratch right now?”

The room went quiet. One person started explaining that it was “mostly possible” but there were “a few manual steps” and “some seeds that someone uploads”. Someone else mentioned an incremental model that “hasn’t been full-refreshed in about eight months because last time it broke.”

That gap — between the diagram on the wall and the reality in the warehouse — is where I’ve spent most of my career. And over the years, working across team after team, I started noticing something. The teams that were struggling weren’t missing knowledge. They had smart engineers who knew about testing, CI/CD, documentation, data contracts. They’d read the blog posts. They’d watched the conference talks. What they didn’t have was a way to honestly assess where they stood and what to fix first.

I kept wishing someone would build a simple diagnostic. Maybe they just needed a handful of honest questions that would tell you, in ten minutes, whether your team was engineering or firefighting.

The moment that turned wishing into doing came about six months later, with a team I was leading. We implemented just two practices — CI testing on pull requests and mandatory code review — and I watched the compound effect over twelve weeks. Our incidents dropped. The analysts stopped asking “which table do I use?” A new hire shipped a model on day four. One of the senior engineers pulled me aside and said, “I wish we’d known how bad things were before you got here. We thought we were fine because nothing was on fire.”

Nothing was on fire. But everything was smouldering.

That’s why I built this test. Not because the world needs another framework — it doesn’t. But because the gap between thinking you’re fine and knowing where you stand is where data teams lose months of momentum. And the only way to close that gap is to ask questions honest enough that the answers sting a little.

The twelve questions

Before we get into the detail, here they are. Score yourself. One point for each “yes.”

Can you rebuild any table from raw data in one command?
Do you have a data catalog that people actually use?
Can a new analyst find the data they need without asking you?
Do you test transformations before deploying them?
Do you fix data quality issues before building new pipelines?
Do you have SLAs for your critical tables?
Do you have a single source of truth for business definitions?
Can you explain the lineage of any metric in under 2 minutes?
Do data producers know when they break downstream consumers?
Do you do code review on SQL and dbt models?
Do new hires build a real pipeline in their first week?
Do you regularly talk to the people who actually use your data?

A score of 12 is unicorn territory. I’ve never seen it in the wild. Most teams I’ve worked with score 3 or 4. Anything below 6 and you’re not doing data engineering — you’re doing data triage.

Now let’s break each one down. For every question, I’ll show you what good looks like and what amazing looks like — with concrete implementations using dbt, Snowflake, GitHub Actions, and AWS. Because “yes” isn’t binary in practice. There’s a massive gap between “sort of” and “absolutely.”

1. Can you rebuild any table from raw data in one command?

This is the foundation. If you can’t reproduce your outputs from your inputs, you don’t have a pipeline — you have a prayer.

The core principle here is idempotency: running the same operation twice produces identical results. dbt is designed around this idea, but achieving it requires deliberate choices, especially with incremental models. Too many teams build incremental models that silently drift from what a full refresh would produce, and they don’t discover the discrepancy until something breaks spectacularly.

What good looks like

Individual tables rebuild cleanly with dbt run --full-refresh --select model_name. Your incremental models are tested against full refreshes during development. You’ve configured on_schema_change: sync_all_columns so schema evolution doesn’t silently break things. And you’ve got a CI pipeline in GitHub Actions that runs dbt build on pull requests.

That’s good. That’s better than most. But it’s still table-by-table.

What amazing looks like

Your entire warehouse rebuilds from raw data using a Write-Audit-Publish (WAP) pattern. If you’re not familiar with WAP, I wrote a deep dive on implementing it with Airflow — the core idea is that new data gets written to a staging environment, audited against quality checks, and only published to production once it passes. It’s the data engineering equivalent of a preflight checklist: nothing reaches consumers until it’s been verified. And if you’re on Snowflake with Iceberg tables, the game has changed — WAP with Iceberg branching gives you git-like isolation at the table level, which means you can stage, audit, and publish without creating separate schemas or databases at all.

The practical implementation: dbt always targets a staging schema or database (or an Iceberg branch), runs all models and tests, and only on success does the data get promoted to production. Combined with Snowflake’s zero-copy cloning, this becomes practical even at scale — cloning a multi-terabyte database takes seconds and costs nothing in storage. You’re not duplicating data; you’re creating metadata pointers.

And when things do go wrong — and they will — the question becomes how quickly you can recover. If your incremental models have drifted, or a source system silently changed its schema three weeks ago, you need a backfill strategy that doesn’t involve rebuilding everything from scratch. I wrote about the pitfalls of day-by-day backfills and how to heal tables properly — it’s the companion piece to idempotency, because reproducibility isn’t just about building forward. It’s about being able to go back.

The really mature teams take it further with Slim CI. Instead of rebuilding everything on every PR, they store the production manifest.json in S3 and compare it against the new build. Only modified models and their downstream dependencies get rebuilt:

# .github/workflows/dbt-ci.yml
on:
 pull_request:
 types: [opened, reopened, synchronize]

steps:
 - name: Download production manifest
 run: aws s3 cp s3://dbt-artifacts/manifest.json ./state/manifest.json

 - name: Run dbt build (modified models only)
 run: dbt build -s 'state:modified+' --defer --state ./state --target ci

One team I worked with reported a 30% reduction in compute costs just by splitting their monolithic CI job into targeted workflows.

Watch out for this: Protect massive tables with full_refresh = false in your dbt config. One accidental --full-refresh on a billion-row fact table can ruin your morning. Override it with a variable when you actually mean it: full_refresh = var("force_full_refresh", false).

2. Do you have a data catalog that people actually use?

The emphasis is on “actually use.” I’ve lost count of how many teams have shown me a data catalog, only for me to discover that nobody’s opened it in months. The data catalog is the gym membership of the data world — everyone has one, nobody goes.

The failure mode is almost always the same: a team selects a tool, loads metadata, declares victory, and moves on. Six months later it’s a ghost town of stale descriptions and orphaned tables.

What good looks like

dbt Docs generated and hosted somewhere accessible. Your major tables have descriptions in schema.yml. The lineage graph exists and gets pulled up occasionally during incident response or onboarding. It’s not perfect, but it’s there.

What amazing looks like

The catalog is the default entry point for data questions — not Microsoft Teams, not the data engineer sitting three desks over.

Spotify’s internal tool Lexikon pushed data scientist adoption from 75% to 95% by doing something clever: they added personalised dataset recommendations, people/team pages showing who uses each dataset, common joins, and popular fields. It became a top-five internal tool because it was genuinely faster than asking a colleague.

Airbnb’s Metis serves over 1,000 data users weekly. Their secret? A Google-like search interface that works for every skill level, from SQL-fluent engineers to product managers who’ve never written a query.

The pattern that actually works for smaller teams: author descriptions in whatever UI is easiest, auto-generate a PR back to dbt’s schema.yml daily, review in Git. This keeps your definitions version-controlled without making people learn Git just to document a table.

And here’s the leading indicator that your catalog is working: track the decline in Microsoft Teams questions about data. Productboard used exactly this metric. Fewer “which table do I use for revenue?” messages meant the catalog was earning its keep.

3. Can a new analyst find the data they need without asking you?

This is the catalog question’s evil twin. A catalog helps, but self-service discovery is really about the cumulative effect of naming conventions, documentation-as-code, semantic layers, and project structure.

dbt Labs published guidance that should be printed and taped to every data engineer’s monitor: “Assume your end-user will have no other context than the model name.” Model names persist across databases, BI tools, DAGs, and docs. Folder names and schemas don’t follow the data the same way.

What good looks like

You’re using the canonical dbt naming convention: stg_ for staging, int_ for intermediate, fct_ for facts, dim_ for dimensions. Your project follows the three-layer structure — staging (1:1 with sources, materialised as views), intermediate (composable business logic), and marts (final business-ready datasets organised by domain). A reasonably technical person can navigate the DAG and find what they need within ten minutes.

What amazing looks like

A new analyst finds and understands any dataset within minutes, without messaging anyone.

Every model has both table-level and column-level descriptions. Schema field search works across all tables — “find every table with customer_id” returns results instantly. Personalised dataset recommendations surface relevant tables based on role.

And the crown jewel: a semantic layer that lets business users query metrics in plain language without writing SQL. The dbt Semantic Layer (powered by MetricFlow) centralises metric definitions in YAML alongside your models — entities, dimensions, and measures declared once, then consumed via API in Tableau, Power BI, Looker, Python notebooks, and increasingly, AI agents.

Bilt Rewards reported an 80% decrease in data costs after implementing it for embedded analytics. And dbt Labs’ testing showed 83% of natural-language questions answered correctly when routed through the semantic layer. That last number matters more than you’d think — it’s the difference between AI tooling that actually works and AI tooling that confidently gives the wrong answer.

4. Do you test transformations before deploying them?

This is probably the single highest-impact practice on the list. If I could only get a team to adopt one of these twelve, it would be this one.

But here’s what most teams get wrong: they think “testing” means slapping not_null on a few columns and calling it done. Real testing means understanding the dimensions of data quality — completeness, uniqueness, timeliness, validity, accuracy, consistency — and building checks that cover each one deliberately. I’ve written about why data quality is a deeper problem than most teams realise, and the short version is this: testing in CI is where quality becomes a habit rather than a hope. But only if your tests are actually measuring the things that matter.

The consensus across every practitioner I’ve studied is clear: CI for dbt is the most impactful thing you can do for data quality. Three complementary testing layers form a comprehensive strategy: generic dbt tests (not_null, unique, accepted_values, relationships), unit tests (introduced in dbt v1.8 for testing complex business logic with static inputs), and data diffing for value-level comparison between production and development.

What good looks like

Basic dbt tests on key models. Tests run as part of scheduled production jobs using dbt build (which runs tests immediately after each model, not in a separate pass). Pull requests exist for dbt changes, and someone eyeballs them before merging.

What amazing looks like

Full Slim CI with per-PR isolated Snowflake schemas, data diffing integrated into PR comments, and automated linting that catches style issues before a human ever looks at the code.

The numbers from real teams are staggering. Thumbtack — 50-plus analysts, five data engineers, over 100 PRs per month — previously spent one to two hours manually validating each pull request with SQL queries and spreadsheets. After integrating data diffing into their GitHub CI pipeline, they saved over 200 hours per month. That’s not a rounding error. That’s a full-time engineer’s worth of capacity recovered.

Dutchie caught timezone corruption on created_at fields and case-when logic errors that were silently shifting 20% of data between columns. Nutrafol caught a transformation that would have shown net revenue plummeting — before it reached production and before the CFO’s Monday morning dashboard refreshed.

Zscaler went even further. They built PRISM, a multi-agent AI PR review system that reduced manual review time by 90% — auto-approving conformant PRs and posting targeted comments on complex logic changes.

Here’s what the mature pipeline looks like in practice:

# On every PR: lint, compile, build modified models, diff against prod
steps:
 - name: SQLFluff lint
 run: sqlfluff lint models/ --dialect snowflake

 - name: dbt compile
 run: dbt compile

 - name: Slim CI build
 run: dbt build -s 'state:modified+' --defer --state ./state --target ci

 - name: Data Diff
 run: datafold ci submit --ci-config-id ${{ secrets.DATAFOLD_CI_CONFIG }}

5. Do you fix data quality issues before building new pipelines?

Here’s an uncomfortable stat: Monte Carlo’s 2023 State of Data Quality report found that 74% of organisations reported that business stakeholders identify data quality issues first — up from 47% the prior year. Most data teams learn about broken data from the people who are supposed to trust it. That’s not a technology problem. That’s a culture problem.

What good looks like

Basic dbt tests catch obvious issues. Tests run in production jobs. The team has an informal sense of which data matters most, usually earned through painful experience — someone got burned by bad revenue numbers in an executive review, and now those tables have tests.

What amazing looks like

Data quality SLAs enforced with error budgets, borrowed from the SRE playbook. A 99.5% data availability target allows approximately 3.6 hours of acceptable downtime per month. When the error budget is consumed, reliability work takes priority over feature delivery. Full stop. No new pipelines until you’ve earned back your quality margin.

Warner Bros. Discovery created something they call a Data Quality Forum — not a reactive incident-review meeting, but an operational bridge between teams. They used observability tooling to surface anomalies early, developed a priority matrix (P0/P1/P2), and made the forum’s patterns part of new hire onboarding. During Olympics livestreaming, custom SQL checks detected missing content metadata before it could break reporting.

Snowflake now offers native Data Metric Functions — FRESHNESS, NULL_COUNT, DUPLICATE_COUNT, UNIQUE_COUNT, ROW_COUNT — that can be scheduled to run on DML changes or time intervals. Results land in SNOWFLAKE.LOCAL.DATA_QUALITY_MONITORING_RESULTS. It’s not a replacement for dbt tests, but it provides a warehouse-level safety net that catches issues your transformation layer might miss.

The cultural shift matters more than the tooling. HelloFresh’s VP of Data drove a transformation through three stages: ad-hoc individual fixes, organised cleanup by a central team, and finally proactive quality at the source. The final stage required embedding data product owners within business domains and running data literacy programs. It’s not glamorous work. But it’s the work that changes outcomes.

6. Do you have SLAs for your critical tables?

“The dashboard is stale” is the Slack message that launches a thousand fire drills. SLAs turn that reactive scramble into a measured, prioritised response.

The breakthrough insight from practitioners: don’t aim for 100%. A 99.5% target gives you a realistic buffer for maintenance, edge cases, and the occasional Snowflake service hiccup. Different data products warrant different targets — which means you need a tiered classification system.

What good looks like

Your team knows which tables are critical, usually because they’ve been burned before. Some dbt source freshness checks are running. Basic Slack alerts fire when jobs fail. It’s reactive, but at least you know when things break.

What amazing looks like

A formal tiered classification published in the data catalog with specific, measurable commitments:

Tier 1 (Gold): ML systems, revenue reporting → PagerDuty on-call, must have unique + not_null tests + assigned owner, freshness SLA under 4 hours
Tier 2 (Silver): Executive dashboards, KPI reports → Slack team channel alerts, owner assigned, freshness SLA under 12 hours
Tier 3 (Bronze): Ad-hoc analytics, exploration tables → weekly digest, freshness SLA under 24 hours

The implementation lives in dbt meta config:

models:
 - name: fct_revenue
 meta:
 owner: "finance-data-team"
 criticality: "tier_1"
 sla_freshness: "4 hours"
 tests:
 - unique:
 column_name: order_id
 - not_null:
 column_name: order_id

SLA compliance dashboards track attainment percentage, breach counts, and trends over time. When the error budget is exhausted, feature development freezes — same principle as practice #5. Dedicated Snowflake warehouses isolate critical workloads from exploratory queries so that someone’s SELECT * on a billion-row table doesn’t cause your revenue report to miss its 8 AM deadline.

7. Do you have a single source of truth for business definitions?

Everyone’s had this conversation: “Which revenue number is right?” Two dashboards, two numbers, two teams who each think the other is wrong. This is what happens when metric definitions live inside BI tools, tribal knowledge, and the head of that one analyst who’s been here since 2019. If you’ve ever worked with dimensional models, you’ll recognise this as the conformed dimension problem — I covered why dimensional modeling still matters and how conformed dimensions are the integration backbone that prevents exactly this kind of mess.

What good looks like

Key metrics are defined in dbt docs or a wiki. The team knows which mart table is the canonical source for revenue, orders, or whatever your core entities are. When someone asks “which table do I use?”, there’s a consistent answer — even if it’s only communicated verbally.

What amazing looks like

A centralised semantic layer enforced as the only path to metrics. All definitions are version-controlled in dbt YAML, reviewed via PR, and validated in CI.

Here’s what a metric definition looks like in MetricFlow:

metrics:
 - name: cancellation_rate
 description: "Percentage of orders cancelled within 24 hours of placement"
 type: ratio
 type_params:
 numerator: cancellations
 denominator: order_total
 filter: |
 {{ TimeDimension('metric_time', 'day') }}

That definition gets consumed identically whether someone queries it from Tableau, Power BI, a Python notebook, or an AI agent. One definition. One answer. Everywhere.

Whatnot — the livestream marketplace growing at breakneck speed — solved a different version of this problem. They used Protobuf schemas as a single source of truth for event definitions, consolidating hundreds of chaotic Snowflake tables down to two clean “exposure” tables: backend_events and frontend_events. Brutal simplification. But it worked because everyone could find the data.

dbt sl validate in CI catches breaking changes to semantic definitions before merge. That’s the key differentiator — your metric definitions aren’t just documented, they’re enforced.

8. Can you explain the lineage of any metric in under 2 minutes?

When your CFO asks “where does this number come from?”, you need an answer faster than “let me check and get back to you.” Two minutes is generous. In most incident situations, you’ve got about thirty seconds before people start making assumptions.

What good looks like

The dbt DAG is viewable, hosted on S3 or an internal site. Your team can trace major mart tables back to their sources. During incident response, someone pulls up the lineage graph and walks through the chain. It takes some squinting, but the information is there.

What amazing looks like

Column-level lineage automated across the full stack — from source systems through dbt transformations to the BI dashboards consumers see.

dbt Cloud provides column-level lineage showing whether each column is “transformed” versus “passthrough/rename.” On the Snowflake side, the ACCESS_HISTORY view (Enterprise Edition) tracks both read and write operations with column-level mappings. Snowflake Labs published an open-source adapter that converts ACCESS_HISTORY data into OpenLineage JSON format, making cross-platform lineage possible.

But here’s where it gets practical. The open-source tool Recce compares two dbt environments and produces a “Lineage Diff” in CI. It categorises changes as breaking, partial-breaking, or non-breaking. PR reviewers get an instant risk assessment: “this change affects 47 downstream models, including three Tier 1 dashboards” versus “this change is isolated to a staging model with no downstream consumers.”

PRs get annotated automatically with impact analysis. Cross-project lineage via dbt Mesh connects multiple dbt projects. Lineage isn’t a pretty graph that nobody looks at — it’s used daily for onboarding, incident response, impact analysis, and cost optimisation.

9. Do data producers know when they break downstream consumers?

This is where data engineering starts borrowing seriously from software engineering. In a microservices world, you wouldn’t deploy an API change without knowing who calls your endpoint. But in data, teams routinely change source schemas, modify column semantics, or sunset tables with zero awareness of who’s consuming them downstream.

Data contracts change that dynamic.

What good looks like

dbt sources are defined with dbt source freshness running and alerting. The team is aware of major upstream dependencies. When Fivetran’s sync breaks or a source system changes its schema, someone notices within a few hours — usually because a test fails.

What amazing looks like

dbt model contracts (v1.5+) enforced on all gold-layer and public models:

models:
 - name: dim_users
 config:
 contract:
 enforced: true
 columns:
 - name: user_id
 data_type: int
 constraints:
 - type: primary_key
 - name: email
 data_type: varchar
 constraints:
 - type: not_null

This is a preflight check: dbt verifies the model’s compiled SQL returns columns matching the contract before building. Breaking changes — removed columns, changed data types, modified constraints — are caught by state:modified in CI. Model access levels (public, protected, private) control cross-project visibility.

Whatnot went further with Protobuf schemas enforced via Buf linter in CI. Every event producer runs against a common testing harness. Post-deployment monitoring catches semantic drift — when a field still exists but its meaning has changed.

Gotcha worth knowing: dbt contracts validate the compiled SQL, not the actual Snowflake table. If tools like Fivetran add columns directly to your raw layer, dbt won’t know until the next run. Monitor schema changes externally using Snowflake’s INFORMATION_SCHEMA or ACCESS_HISTORY to catch drift between runs.

10. Do you do code review on SQL and dbt models?

I’m constantly surprised by how many data teams still deploy SQL changes without review. In software engineering, unreviewed code going to production would be considered reckless. In data engineering, it’s Tuesday.

What good looks like

Pull requests are required for all dbt changes — branch protection is enabled on main. At least one human reviewer looks at each PR. There’s a basic PR description explaining what changed and why.

What amazing looks like

Automated CI running SQLFluff lint + dbt compile + dbt build + data diff on every PR. Pre-commit hooks catch issues before code is even committed. PR templates with structured sections: description, linked tickets, impact zone, testing evidence.

The recommended pre-commit stack combines SQLFluff for linting with dbt-checkpoint for structural validation:

# .pre-commit-config.yaml
repos:
 - repo: https://github.com/sqlfluff/sqlfluff
 hooks:
 - id: sqlfluff-lint
 args: [--dialect, snowflake]

 - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
 hooks:
 - id: check-model-has-tests
 - id: check-model-columns-have-desc
 - id: check-source-has-freshness

That last hook — check-model-columns-have-desc — is quietly revolutionary. It means documentation isn’t optional. You literally cannot merge a model without column descriptions. The “we’ll document it later” excuse dies right there in the CI pipeline.

Surfline (700+ dbt models) reported that after integrating SQLFluff, SQL consistency improved dramatically and reviewer burden dropped. New engineers learned “good SQL” from day one because the linter enforced it automatically. Another team, Markerr, found that automated style enforcement freed up review time to focus on deeper logic questions rather than arguing about capitalisation and trailing commas.

The dbt_project_evaluator package is worth mentioning too — it audits your entire DAG structure against dbt Labs’ published best practices. Run it in CI and it’ll flag things like models that reference sources directly (bypassing staging), duplicate sources, or models with no tests.

11. Do new hires build a real pipeline in their first week?

The speed at which a new hire becomes productive tells you everything about the state of your documentation, tooling, and team culture. If it takes three weeks before someone can make a meaningful contribution, you don’t have an onboarding problem — you have a platform problem. I wrote a guide to navigating your first 90 days as a data engineer — and the teams that make those first 90 days count are almost always the ones who’ve invested in the infrastructure described here.

What good looks like

New hire has access to tools within a day or two. Some documentation exists. A buddy or mentor is assigned. They can run the dbt project locally by end of week one, even if they haven’t contributed anything yet.

What amazing looks like

Pre-arrival setup is complete before Day 1: machine provisioned, logins ready, Snowflake roles assigned, dbt Cloud account active. No engineer should spend their first morning installing things.

Each developer gets a personal Snowflake sandbox — dbt_<username> schema in the dev database, with write access only to dev and read-only access to raw/staging. Snowflake’s zero-copy cloning makes production data available instantly without duplicating storage costs. Productboard open-sourced dbt-snowflake-sandbox — a set of dbt macros that create isolated sandboxes by cloning only the specific model dependencies a developer needs.

The first-week project is structured and progressive:

Day 1-2: Run the existing dbt project locally. Explore the DAG. Read the key model documentation.
Day 3: Add a new source or staging model for a real (but low-risk) dataset.
Day 4: Build a staging model with tests and documentation.
Day 5: Open a PR, go through the review process, and get it merged.

By Friday, they’ve shipped something real. They understand the workflow. They’ve experienced CI, code review, and deployment. And critically, they feel like a contributor rather than a tourist.

7shifts estimated cutting onboarding time by over a week per new hire just by having documentation and data questions accessible in their data catalog.

12. Do you regularly talk to the people who actually use your data?

This is the question that separates data teams who build for their portfolio from data teams who build for their business. You can nail every technical practice on this list and still fail if you’re building the wrong things. I’ve written before about how to maximise your data team’s impact — and the through-line is always the same: the teams that create real value are the ones who stay close to the people consuming their work.

What good looks like

A Slack channel exists for data questions. Communication happens when issues arise. There are occasional stakeholder meetings, usually prompted by something breaking or a new request coming in.

What amazing looks like

Data office hours. Weekly or bi-weekly open sessions where anyone in the organisation can bring data questions. Not a presentation. Not a status update. An open door.

Holistics published a detailed playbook for running what they call “data clinics.” The core principles: teach, don’t serve. Show business users how to self-serve rather than just answering their question and sending them on their way. Montreal Analytics reported that after implementing data clinics, the number of self-serve business users on their BI tool grew tenfold. Tenfold.

Pair that with a monthly Data NPS survey — a single question: “How likely are you to recommend our data team’s products to a colleague?” It sounds corporate, but without this metric, you have no way to quantify whether your consumers are satisfied or just silently building workarounds in Excel.

dbt exposures formalise the connection between data models and their consumers:

exposures:
 - name: weekly_revenue_dashboard
 type: dashboard
 owner:
 name: Finance Team
 email: finance@company.com
 depends_on:
 - ref('fct_revenue')
 - ref('dim_date')

When a model changes, the PR shows which exposures are affected. Stakeholder impact becomes visible in code review. You can proactively notify the finance team that their revenue dashboard’s upstream model is changing before they discover it themselves.

Frame your communication in business terms. Not “model precision is up 12%” but “the sales team can now identify high-value leads 40% faster.” Nobody outside your team cares about your DAG. They care about whether they can trust the numbers. If you’re not sure how to bridge that gap, I wrote a guide on breaking down business context — because the hardest part of talking to stakeholders isn’t the talking, it’s knowing what they actually need to hear.

Scoring your team

Here’s the scoring framework, and I want you to be ruthless with yourself:

Score	What it means
10-12	You’re in elite territory. Your data platform is a competitive advantage.
7-9	You’ve got strong foundations with clear areas to improve.
4-6	You’re functional but fragile. One bad incident away from a crisis of trust.
1-3	You’re firefighting, not engineering. The business tolerates your team — it doesn’t trust it.

Most teams land at 3-4. That’s not a criticism — it’s where the industry is. The gap between knowing these practices exist and actually implementing them is where the real work lives.

Where to start

If you’re staring at a low score and feeling overwhelmed, here’s what I’d suggest: start with practices 4 and 10. CI testing and code review.

Here’s why. Once dbt changes go through pull requests with automated checks, you have the infrastructure to enforce everything else. Contracts? They’re a CI check. Quality gates? CI check. Documentation requirements? CI check. SLA validation? CI check. You’re not adopting twelve practices — you’re building one pipeline that gates on twelve things.

The canonical architecture appears across virtually every mature team I’ve studied: GitHub Actions triggering Slim CI with dbt build -s state:modified+ against per-PR Snowflake schemas, with production manifests stored in S3 for state comparison. Start there. Layer on the remaining practices as your team’s maturity grows.

Three patterns emerged from the research that I think are worth calling out explicitly:

“As code” wins everywhere. Documentation-as-code, style-guides-as-code, permissions-as-code, quality-checks-as-code, contracts-as-code. Every manual process that gets codified becomes version-controlled, reviewable, and enforceable. Every one that stays manual eventually drifts.

Snowflake’s zero-copy cloning is the force multiplier. Isolated PR environments, sandbox onboarding, rebuild testing — all of these become cheap and instant with cloning. If you’re on Snowflake and not using this feature aggressively, you’re leaving the most powerful tool in the shed.

Culture matters more than tooling. Catalog adoption fails without meeting users in their existing workflows. Data quality requires executive sponsorship and error budgets, not just more tests. Producer-consumer contracts are first and foremost a cultural change. You can buy every tool on this list and still score a 3 if the organisation doesn’t value the practices behind them.

The diagram and the warehouse

Remember that team — the one with the beautiful diagram and the revenue table nobody could rebuild?

I caught up with one of the engineers about four months after I’d moved on. They’d started with exactly what I’d pushed for: CI testing and code review. Then they added contracts on their gold-layer models. Then SLAs on their Tier 1 tables. Then a first-week onboarding project for new hires.

Their score when I first asked these twelve questions? Three. Four months later? Eight. Not perfect. But the difference wasn’t really the number. The difference was that when they pulled up that architecture diagram now, it matched what was actually running in Snowflake. The gap between the wall and the warehouse had closed.

That’s what this test is really measuring. Not whether you have the right tools — you probably do. Not whether you know the right practices — you’re reading this, so clearly you care. It’s measuring whether there’s a gap between what you think your data platform looks like and what it actually looks like.

These twelve questions just make you say it out loud.

So print it. Score yourself honestly. Share it with your team. And the next time someone asks you how your data platform is doing, give them a number between 1 and 12.

That number is worth more than any architecture diagram.