Posts on Ghost in the data

SQL Tells You What. Comments Tell You Why.

Chris Hillman — Sat, 06 Jun 2026 09:00:00 +1000

The best SQL doesn’t need comments. Write meaningful CTE names, descriptive aliases, clear column labels — and a skilled reader will follow your logic without a single annotation. That’s the right instinct.

It’s also only half right.

SQL is a declarative language. You’re not writing how the database retrieves your data; you’re writing what you want. That’s a useful distinction, because “what” and “why” are very different questions, and SQL can answer exactly one of them.

A query can be perfectly, elegantly readable and still be completely opaque about the reason it exists. The name of your CTE, no matter how well chosen, cannot explain the business decision that gave birth to it.

The self-documenting ceiling

Consider this query:

SELECT
 customer_id,
 SUM(order_total) AS lifetime_value,
 DATEDIFF(day, MIN(order_date), CURRENT_DATE) AS customer_age_days
FROM
 orders
WHERE
 order_date >= DATEADD(day, -90, CURRENT_DATE)
 AND status_code NOT IN (1, 2, 9)
GROUP BY
 customer_id

Clean SQL. No ambiguity about what it does. But try answering these questions from the code alone:

Why 90 days? Why not 60, or 180, or the full history? Is 90 the standard customer lifetime window, a finance reporting period, the SLA in a merchant agreement, or the number someone picked in a meeting four years ago and nobody has questioned since?

What are status codes 1, 2, and 9? You might guess from context — pending, draft, cancelled — but you’re guessing. More importantly: why are they excluded? Is this a business rule about what counts as “real” revenue, or a data quality workaround because those status codes appear when an upstream webhook fires twice?

These are not trivial questions. The 90-day cutoff defines what “active customer” means across your entire reporting layer. The excluded status codes determine what gets counted as revenue. Change either without understanding the original intent, and you’re not refactoring — you’re silently redefining business logic that stakeholders are relying on.

Better SQL helps, but only so far:

SELECT
 customer_id,
 SUM(order_total) AS lifetime_value,
 DATEDIFF(day, MIN(order_date), CURRENT_DATE) AS customer_age_days
FROM
 orders
WHERE
 order_date >= DATEADD(day, -90, CURRENT_DATE)
 AND status_code NOT IN (
 1, -- pending
 2, -- draft
 9 -- cancelled
 )
GROUP BY
 customer_id

This is better. You’ve replaced the mystery with labels. But inline labels aren’t explanations — they’re just naming the what more explicitly. You still don’t know why.

What SQL literally cannot tell you

SQL cannot explain why the program was written the way it was. It cannot discuss the reasons certain alternative approaches were taken. It cannot tell you that the business definition changed, that the upstream system is broken, or that this logic exists specifically to handle a problem that was supposed to be fixed in Q2 and wasn’t.

Here’s the kind of context that belongs in a comment:

-- Customer lifetime window: 90-day lookback aligns with the SLA in the Merchant Agreement (v3.2).
-- Reviewed and confirmed with Finance, Dec 2024. Don't change without checking with that team first.
-- status_codes 1, 2, 9 excluded per the order lifecycle defined in the legacy Salesforce migration.
-- These codes appear as artefacts when the Salesforce sync fires before the order is confirmed.
-- The upstream fix was deprioritised (JIRA: DATA-3841). Removing this filter will cause double-counting.
SELECT
 customer_id,
 SUM(order_total) AS lifetime_value,
 DATEDIFF(day, MIN(order_date), CURRENT_DATE) AS customer_age_days
FROM
 orders
WHERE
 order_date >= DATEADD(day, -90, CURRENT_DATE)
 AND status_code NOT IN (1, 2, 9)
GROUP BY
 customer_id

That comment block is five lines. It contains: the business rationale, the stakeholder who owns the decision, the date it was confirmed, the upstream system causing the problem, the known issue reference, and the consequence of removing the filter. None of that information is recoverable from the SQL. All of it is load-bearing.

The deduplication that nobody remembers adding

Have you seen this line before:

QUALIFY ROW_NUMBER() OVER (
 PARTITION BY customer_id, order_id
 ORDER BY ingested_at DESC
) = 1

The SQL is completely transparent about what it does: take the most recently ingested row for each customer/order combination. What it cannot tell you is why duplicates exist in the first place, whether the source of those duplicates has been fixed, whether this deduplication logic is still necessary, or whether “most recently ingested” is actually the right tiebreaker for your use case.

-- Deduplicating on customer_id + order_id to handle duplicate webhook events from Eftpos.
-- Eftpos retries failed webhooks, which can fire the same order_created event multiple times.
-- Taking the latest ingested_at gives us the most recent event state (status, amount, etc.)
-- Revisit if we adopt Eftpos's idempotency keys at the integration layer — this dedup may become redundant.
-- Background: this logic was added after the October 2024 incident (post-mortem in Confluence).
QUALIFY ROW_NUMBER() OVER (
 PARTITION BY customer_id, order_id
 ORDER BY ingested_at DESC
) = 1

The second version tells the reader everything they need to know to maintain this safely: the upstream cause, the logic behind the tiebreaker, when to reconsider it, and where to find the history. Without that comment, every future engineer who touches this code has to reverse-engineer context that no longer exists anywhere.

When two models define the same word differently

This one doesn’t generate an error. It generates confusion at the worst possible moment.

-- IMPORTANT: 'churned' here means >90 days since last order.
-- This differs from mkt_customer_segments, which uses >60 days for winback campaign targeting.
-- Both definitions are intentional. Customer Success uses 90 days because cohort analysis
-- showed that 30% of "60-day churned" customers reorder organically within the next 30 days
-- and shouldn't receive a discount. Marketing uses the stricter threshold to maximise winback opportunities.
-- The discrepancy is documented and known. Do not "fix" one to match the other.
CASE
 WHEN days_since_last_order <= 30 THEN 'active'
 WHEN days_since_last_order <= 90 THEN 'at_risk'
 ELSE 'churned'
END AS customer_segment

Without that comment, someone eventually “cleans up” one of the two definitions because having different churn thresholds in two models looks like a bug. They’re wrong, and they’re about to break a winback campaign that’s been performing well for six months.

The SQL cannot warn them. The comment can.

dbt descriptions aren’t exempt

dbt gives you a proper home for the “why”: model descriptions and column-level descriptions in your YAML.

Bad:

models:
 - name: fct_orders
 description: "The orders fact table. Contains order data."
 columns:
 - name: customer_segment
 description: "The customer segment based on days since last order."

That description restates what the model name already says. It is a comment-shaped void.

Good:

models:
 - name: fct_orders
 description: >
 Order-level fact table used as the single source of truth for revenue reporting.
 Revenue excludes test accounts (prefixed 'INTERNAL_') and gift card redemptions,
 per the Finance revenue recognition policy agreed January 2025.
 See Confluence: Revenue Recognition Standards v3.
 Do not use this model for marketing attribution — use mkt_attributed_revenue instead,
 which applies different channel logic.
 columns:
 - name: customer_segment
 description: >
 Lifecycle segment defined by the Customer Success team, Q1 2025.
 Thresholds (30/90 days) were derived from cohort analysis showing a median
 reorder window of 28 days. Intentionally differs from the Marketing segment
 definition in mkt_customer_segments, which uses a 60-day churn threshold
 for winback campaign purposes. Both are correct for their respective use cases.

That description does something a column name never can: it tells you who owns the definition, when it was set, what analysis underpins it, and — critically — where the definition intentionally diverges from a similar field elsewhere in the warehouse. That last part is the thing that saves someone three hours of confused Teams messages.

The archaeology problem

Data teams inherit codebases. It happens constantly, and it will keep happening. Engineers leave. Consultants finish their contracts. Reorgs shuffle ownership. What gets left behind is the SQL, and whatever context wasn’t written down is gone.

Good SQL survives the handover. It’s readable, consistent, and correct. But “readable” in the sense of mechanically parseable is not the same as “understandable” in the sense of knowing what business problem the code was solving, whose decision it was, and what you’d need to change if that decision changed.

Comments are how you write for the engineer who inherits this in two years. They’re how you write for yourself in six months. They’re how you ensure that a working pipeline stays working through the context collapse that happens every time anyone leaves a team.

SQL can only tell you what. Try not to shortchange the people reading it in either direction.

Don't Go Dark: Visibility Is a Data Engineering Skill

Chris Hillman — Sat, 23 May 2026 09:00:00 +1000

There’s a specific kind of silence in data engineering that I’ve learned to fear.

Not the silence of a system that’s working well. Not the comfortable quiet of a team in flow. I mean the silence of a project that’s been running for three weeks and you still can’t point to a single visible thing it has produced. The kind of silence where, if your manager stopped you in the hallway and asked “how’s that migration going?”, you’d say “fine” because saying anything more accurate would require explaining things you haven’t fully articulated yet — even to yourself.

I watched a talented engineer do this once. Let’s call her Mia. She’d been handed a significant refactor: six months of accumulated technical debt in a set of dbt models that sat at the centre of our revenue reporting. The kind of work that is genuinely unglamorous, invisible by nature, and deeply important. She approached it the way a lot of good engineers approach hard problems — she went quiet and started digging.

For three weeks, she showed up, she worked hard, and she said almost nothing. Her pull requests were sparse. Her standups were “still investigating.” Her Teams messages were infrequent. She wasn’t slacking off. She was grinding through some of the most complex lineage work I’d ever seen someone untangle.

Then a finance stakeholder messaged me — not her, me. Someone had noticed a spike in our Snowflake credit usage. They were nervous. They’d been burned by silent changes before.

I didn’t have an answer. Mia had one, but nobody knew to ask her.

I went to her and said, “walk me through what you’ve been doing.” What she showed me was extraordinary. She’d traced dependency chains through twelve models. She’d found three bugs that had been silently wrong for months. She’d drafted a migration plan that would cut query time in half. It was genuinely impressive work — and it had been completely invisible for twenty-one days.

That’s the moment I understood something that Jeff Atwood wrote in 2008, in a post called Don’t Go Dark, that I wish someone had made me read before I managed my first data team. Atwood’s essay was aimed at software engineers. It quoted a Microsoft engineering rule that said, in essence: three weeks is going dark. The rule wasn’t about laziness. It wasn’t about bad engineers. It was about a failure mode so common in technical work that it needed a name.

Mia wasn’t lazy. She was going dark.

What Atwood Actually Meant

Atwood’s post is short — shorter than most people expect from the weight it carries. He builds on two sources: a software engineering principle from Jim McCarthy’s Dynamics of Software Development, and an observation from open-source contributor Ben Collins-Sussman. Together they identify a pattern that every engineering lead will recognise the moment they read it.

The pattern is this: programmers don’t want to show work in progress. They want to disappear into the problem, solve it completely, and emerge holding something finished. Collins-Sussman pinned it to fear — not arrogance — arguing that most developers treat coding as an arena for personal heroics rather than an inherently social activity. They’ll share code once it’s polished. They’ll share failures once they’re resolved. But the messy in-between, the half-formed thinking, the models with todos and the branches with four commits and no description? That stays private.

McCarthy’s rule was direct: manage your tasks in short enough increments that you always have a visible deliverable at the end of each one. Three weeks without something to show is too long. The early warning of slippage — one day, discovered this week — is worth ten times more than six months of slippage discovered at deadline.

Atwood’s additional observation is the one that has aged best: agile development structures made going dark mechanically difficult. If you’re running sprints, you’re producing something reviewable every two weeks regardless. The iteration boundary forces surfacing. Joel Spolsky, Atwood’s co-founder at Stack Overflow and another writer in this intellectual lineage, made the same argument about daily builds a few years earlier: a team that builds its software every day has, at minimum, a daily signal that things are either working or not. The build is the forcing function. It converts invisible progress into visible evidence.

Open source projects, Atwood observed, face the highest risk of going dark — because there’s no manager, no sprint ceremony, no daily build process unless someone chooses to implement one. The discipline has to be entirely internal. The only currency that matters on an open source project is visible, reviewable work.

Data engineering, as a discipline, has no such forcing function. Our work doesn’t end at a sprint boundary with a demo. Our pipelines run on schedules that have nothing to do with how long the underlying thinking took. Our models transform quietly, in the background, producing outputs that are visible only when something breaks — or worse, when something breaks subtly and doesn’t look broken at all.

That’s the gap Atwood’s advice never closed. He fixed going dark for software teams. Nobody translated it for us.

Why Data Engineering Goes Dark More Than Anyone Else

There’s a structural reason data work is so prone to invisibility.

When a front-end engineer ships a feature, there’s an artifact: a button, a page, a flow you can click through. When a backend engineer merges a PR, there’s a diff: before and after, visible in the tool, reviewable by anyone. The work and the evidence of work are coupled. They arrive together.

Data work decouples them. Benn Stancil, writing about the data industry, put it well when he described the fuzzy boundary between production and everything else in data teams — there’s often no clean staging/prod line, no clear “shipped” moment, no obvious artifact to point to. A dbt model runs. A dashboard refreshes. A pipeline completes. Did anything change? Did anything improve? Is this better than it was last week? The answers require knowing the context that lives in the engineer’s head, which is precisely the context that isn’t written down.

This gets worse the longer the project. Migrations are the canonical example. A warehouse migration — say, moving from Redshift to Snowflake, or rebuilding a legacy reporting layer in dbt — might take three months of continuous work before a stakeholder sees a single dashboard change. During those three months, the data engineer is doing real, difficult, valuable work: tracing lineage, resolving naming conflicts, negotiating data contracts with upstream producers, writing tests that have never existed before. All of it is invisible. All of it looks like nothing to anyone watching from the outside.

There’s also the problem that Locally Optimistic describes as the distinction between linear and circular projects. Linear projects have a known path and a clear endpoint. Circular projects — most of the genuinely interesting data work — have outcomes that depend heavily on what you discover along the way. The best-laid migration plan doesn’t survive contact with a production schema that hasn’t been properly documented since 2019. Exploration takes time that doesn’t map cleanly to sprint cards or progress percentages.

When you combine those two things — structural invisibility and genuinely unpredictable progress — you get a discipline that almost naturally produces going-dark behaviour, even among engineers who are working diligently and in good faith. Mia wasn’t hiding. The nature of her work was hiding it for her.

The IC Chapter: Staying Visible Without Performing Busyness

Staying visible doesn’t mean performing busyness. It doesn’t mean posting standup updates about tasks you haven’t started. It doesn’t mean narrating every dead end like it’s a breakthrough. That’s noise, and experienced leads learn to tune it out.

What it does mean is producing artifacts — real, durable, reviewable traces of your thinking — on a cadence that keeps your work legible to the people around you. The goal is to make silence meaningful. When there’s nothing from you for three days, the people you work with should know that’s fine because you’ve pre-declared it and pre-committed to what you’ll surface when you come up for air. When you’re grinding through something genuinely hard, the work itself should leave evidence that can be reviewed asynchronously, without requiring a meeting to explain it.

Draft PRs as the lowest-friction visibility tool you already have.

Open a pull request on day one of a significant piece of work. Not when it’s ready for review — when you have a scaffold and an intent. GitHub’s draft PR feature exists exactly for this: your code can’t be merged, reviewers aren’t auto-requested, and everyone in the team can see that the work exists, what it’s targeting, and roughly how it’s going. Update it with a comment as you progress. “Blocked on upstream source delay — will resume Thursday.” “Discovered three extra joins needed — revised estimate to end of week.” “Models passing, running full test suite now.”

This pattern turns the PR into the canonical artifact of the work — a living document that replaces scattered messages, status check-ins, and “hey, how’s that thing going?” conversations. The PR description, written well, tells the story of what changed, why it changed, and how it was tested. Six months from now, when a model breaks at 2am and the engineer who built it is on leave, that description is the thing that saves the on-call.

Commit messages as team communication.

There’s a widely-cited piece by Chris Beams on how to write a good git commit message that’s been floating around engineering circles since 2014. The core argument is simple: the subject line tells you what changed; the body tells you why. A commit message that says fix bug costs the next engineer ten minutes of archaeology. A commit message that says Fix null handling in fct_orders when currency column is missing and then explains that a vendor renamed a field and the defensive .get() was silently returning zero for three months — that commit message is worth something.

The Conventional Commits specification formalises this further: feat:, fix:, refactor:, docs:, chore:, each with optional scope. It sounds bureaucratic until you try to understand a six-month-old change at 9pm during an incident, and someone had the discipline to write refactor(revenue): consolidate gross margin calculation into shared macro in the subject line. Then you understand immediately.

dbt documentation as a visibility discipline.

Every model in a dbt project can carry descriptions at the model level and at the column level, written in the same schema.yml file that defines your tests. Most teams don’t write them. Or they write them at launch and never update them. Or they live in a separate Confluence page that nobody visits.

Here’s the thing: schema.yml descriptions review in the same PR as the code they describe. They can’t silently drift from the implementation because they’re changed in the same commit. A model description that says “Revenue aggregated by market, excluding internal transfers — see ADR-004 for the exclusion logic” is a piece of team communication that persists across every engineer who will ever touch that model.

Go further. Use dbt Exposures to document which dashboards and downstream artifacts depend on your models. A handful of YAML lines, committing which report or ML model consumes each table, converts “will this refactor break anything?” from a Slack archaeology project into something you can run a query against. When a finance stakeholder asks if their dashboard will be affected by a model change, you can show them the dependency chain. That’s a conversation that previously required guesswork. Now it requires a terminal command.

Architecture Decision Records for the decisions people will question.

ADRs — short markdown files committed to docs/adr/ in your repository — document the decisions that will otherwise live only in one engineer’s memory. Why did you choose Snowflake over BigQuery? Why does the revenue model exclude returns that arrived after 30 days? Why is this model incremental rather than table? These questions come up six months later, in a meeting, when the person who made the decision has moved on or forgotten the reasoning. An ADR is three paragraphs and a status field. It takes fifteen minutes to write. It has saved me more than one painful meeting where the alternative was reconstructing a decision from Slack history.

Async standups that actually work.

The daily standup ritual made sense when software teams sat together and needed a quick synchronising pulse. For data teams — often distributed across timezones, frequently doing work that looks the same from the outside regardless of whether it’s going well or terribly — the fifteen-minute morning video call has become a source of collective fiction. Everyone says “still on X, should be done soon” and nobody learns anything.

Async standup tools like Geekbot (for Slack) or equivalent bots for Teams send each person the standup questions by DM on a schedule and post responses to a shared channel. The format that consistently performs best isn’t the classic three questions — it’s a single question: “What would you like the team to know today?” That question has enough latitude to surface a blocker, a discovery, a risk, or a win. It invites signal rather than mandating a ritual. Responses posted to a shared channel are searchable, linkable, and skimmable — a week of them gives a manager more information than five daily standups would.

The written standup format also decouples availability from communication. A data engineer in Melbourne doesn’t need to be online at the same time as her lead in London for him to understand what’s happening. The post is there when he arrives. She’s not interrupted in her maker’s block to attend a meeting she’ll spend eight minutes waiting through before she has sixty seconds to talk.

One discipline worth enforcing: when you name a blocker in a written standup, link to it. Don’t write “blocked on upstream data delay” — write “blocked on upstream raw_orders freshness, opened Issue #247 to track.” The standup post becomes navigable rather than descriptive. Anyone who wants to understand the blocker can follow the link. Anyone who just needs the pulse can read the post and move on.

Data diffs in CI — making the invisible visible before it ships.

The traditional CI check for a dbt project tells you whether models compile and whether your schema-level tests pass. Unique keys, not-null constraints, accepted values — these are important. They catch structural problems before they reach production.

What they don’t catch is when a model that was returning 65.00 starts returning 65.10, or when a currency denomination silently shifts from whole dollars to cents because an upstream source changed its representation without changing its schema. Those changes are invisible to schema tests. They’re invisible to the log output. They’re invisible right up until a finance stakeholder notices that last month’s revenue looks 100× higher than expected in a board deck.

Data diff tooling — whether through Datafold integrated into GitHub CI, or the open-source audit_helper package in dbt, or a homegrown comparison query — runs value-level comparisons between the current state of a model and a reference point. Merge a PR that refactors how gross margin is calculated, and the CI comment shows you a row-level diff of what changed. Every affected row. Every affected column. Before the code reaches production.

This changes the nature of what a PR review means. Instead of reviewing whether the SQL is logically correct — a genuinely hard thing to assess from reading code alone — reviewers can look at the data that would result. A refactor that should be semantically identical shows zero diff. One that inadvertently changes three percent of records in a specific market shows exactly which records and by how much. The artifact created by data diff in CI is one of the most powerful anti-going-dark tools available to a data team, because it makes the consequence of code changes legible to anyone who can read a table — not just the engineers who wrote the SQL.

Async documentation that travels with the work.

One pattern that consistently separates data teams that are legible from those that aren’t: Loom or short screen recordings attached to PR descriptions on significant changes. Not for every fix. Not as a replacement for the written description. But for a model redesign, a migration milestone, or a new data product, a two-minute screen recording walking through the dbt DAG and explaining the decisions is worth more than three paragraphs of prose that nobody will read.

The video doesn’t have to be polished. It has to exist. It converts work that is legible only to the person who did it into work that is legible to a finance analyst, a product manager, or a new data engineer joining in six months. It replaces the meeting where you’d have explained this anyway, except now it’s available asynchronously, can be rewatched, and is linked from the artifact it describes.

The Distributed Team Problem

Remote and distributed data teams face a compounded version of the going-dark problem. The structural invisibility of data work — no natural demo format, no clean shipped/not-shipped boundary — gets worse when your team spans timezones and your primary communication channel is a text-based messaging app with a scroll-back limit.

The teams that navigate this best tend to share a principle: optimize for knowledge retrieval, not knowledge transfer. The goal isn’t to ensure that information passes from person A to person B in a meeting. The goal is to ensure that the information exists in a form that person B can find when they need it, regardless of whether person A is online.

GitLab, operating as a fully remote company since its founding, formalised this as handbook-first communication: any decision, policy, or significant piece of information gets documented before it gets communicated or implemented. Not as bureaucracy, but because documentation creates the retrievable artifact. A verbal decision made in a meeting exists only in the attendees’ memories. A written decision linked from a PR exists for the lifetime of the repository.

For data teams, this translates practically into a few habits. Meeting decisions get a one-paragraph summary posted to a Teams thread, linked from the relevant GitHub issue. A verbal conversation about model design gets a brief written follow-up: “Following on from our chat — going with incremental strategy for fct_orders because full refresh at scale was timing out in staging. ADR drafted at docs/adr/0012-incremental-fct-orders.md.” The conversation happened. Now it has an artifact that travels.

Long-running projects — migrations, platform changes, major refactors — benefit from a written brief at the start that defines the scope, the success criteria, and the milestone cadence. Not a Jira epic with fifty tasks. A two-page document that answers: what are we doing, why are we doing it, how will we know when we’re done, and what are the milestones where we’ll regroup. The brief is the thing you hand to a new stakeholder who asks what’s happening with the platform migration. The brief is the thing you update when the scope changes. The brief is what prevents the three-week silence.

Locally Optimistic describes a useful scripting pattern for circular projects — the kind where the outcomes depend heavily on what you discover along the way. Rather than promising a deliverable, you promise a process: “We will investigate X, Y, and Z over the next two weeks. Those might not give us a conclusive answer, but we’ll regroup to discuss the findings and determine next steps.” That framing is honest. It sets appropriate expectations. And critically, it includes a defined moment of resurfacing — the regroup — which prevents the silence from becoming open-ended.

Julia Evans has written persuasively about the value of maintaining a running document of your own accomplishments — not as a vanity project but as a memory aid. The problem she identifies is real: at review time, you’ve forgotten half of what you did in the last six months, and your manager has forgotten more than that. A brag document is a log of things that mattered: bugs found before they became incidents, migrations completed, models refactored, stakeholders unblocked, junior engineers mentored.

This isn’t about inflating your achievements. It’s about making them visible in a context where the work itself produces no natural evidence trail.

Equally useful: weekly notes. A short message to your team lead, every Friday, that says “this week I did X, I’m picking up Y on Monday, and I want to flag Z as a potential blocker.” It doesn’t have to be long. It doesn’t have to be polished. It just has to exist. Will Larson calls this the drip — communication on cadence, regardless of whether there’s exciting news. The drip is what makes silence informative. When you’ve been sending a Friday note every week for three months and then you stop, the absence is a signal. When you’ve never sent one, silence tells nobody anything.

Tanya Reilly’s glue trap — for senior ICs especially.

If you’re a staff or principal data engineer, you’re probably doing a lot of work that doesn’t look like work from a promotion perspective: reviewing designs, unblocking others, noticing what’s slipping, maintaining relationships with stakeholders. Reilly calls this glue work — the coordination and communication and knowledge-transfer that holds projects together. It’s real work. It’s often the most valuable work on the team. And it’s completely invisible without a deliberate effort to create artifacts.

Design proposals, written-up meeting decisions, documented onboarding pathways, group emails that summarise a decision made in a verbal conversation — these are the artifacts that make glue work visible. Without them, you’re building organisational infrastructure that has no evidence it exists.

The Fear Dimension: What’s Really Happening When Engineers Go Dark

Here’s something I think we don’t talk about honestly enough — and I’m including myself in this.

I’ve been on both sides of it. I’ve gone dark on work I wasn’t confident about, telling myself I’d surface when I had something worth showing. I’ve also been the lead who didn’t notice, or didn’t ask, until a stakeholder beat me to it. Neither version felt like failure at the time. Both were.

Most going-dark behaviour isn’t strategic. It isn’t laziness. It’s fear — specifically, the fear that surfacing in-progress work will reveal that you don’t have it as together as you implied you did.

Collins-Sussman put this precisely in the piece Atwood quoted: developers don’t want their peers to see mistakes or failures. They want to present themselves as infallible. They’re comfortable sharing code once it’s polished. The messy middle, the exploratory branches, the models with big question marks in the comments — that stays private.

This fear is rational in the short term. Showing a half-finished thing invites feedback that can derail you. Showing a broken thing invites questions you can’t answer yet. Staying silent avoids both. The problem is that in the medium term, silence accumulates into something much worse: an engineer who surfaces three weeks into a problem with nothing to show, facing a stakeholder conversation where the first question is “why didn’t you just tell us?”

Paloma Medina’s BICEPS framework identifies six core needs that, when threatened, produce withdrawal: Belonging, Improvement, Choice, Equality, Predictability, Significance. (Yes, it spells BICEPS. Yes, it will appear on a workshop slide near you. The underlying insight is still useful.) Lara Hogan, who has done more than anyone I know of to translate this framework into engineering management practice, makes the observation that going dark is often a flight response to one of these needs being threatened. A project pivot that makes the last three months of work feel wasted threatens Significance — and the engineer who feels that their work no longer matters is likely to disengage before they articulate why. An abrupt change in priorities threatens Predictability and Choice simultaneously. A team reorganisation threatens Belonging.

The leader’s job, when they notice going-dark behaviour, isn’t to increase accountability pressure. It’s to identify which need has been threatened and address it directly. Fournier and others recommend this as the actual content of the 1:1 when something feels off: not “how’s the project going?” but “how are you doing?” The project status comes second. The human state comes first.

Hogan’s Red/Yellow/Green check-in is a lightweight mechanism for this. At the start of a 1:1, an engineer can say “red” without having to explain why — just to signal that something is wrong. The permission to say “red” without an explanation is itself the thing that makes the saying possible. When explanation is required before disclosure, disclosure gets deferred until the explanation is formed — and by then, the problem has grown. Hogan’s observation: only having to say red, and not having to explain the why, is huge. The absence of the explanation requirement is the mechanism. It lowers the cost of disclosure below the threshold where people start editing themselves.

Amy Edmondson’s research on psychological safety is directly relevant here. Her finding, counterintuitively, was that better hospital units didn’t make fewer medication errors — they reported more of them. Her reframe: the better units weren’t more error-prone, they were more willing to surface problems early. The error rate looked higher because the environment was safe enough that people actually said what was happening.

Google’s Project Aristotle, which studied 180 teams over several years, found the same pattern at scale: the single strongest predictor of team performance was whether team members felt safe to take interpersonal risks. Safe to say “I’m stuck.” Safe to say “I found a bug.” Safe to say “I don’t know how long this will take.”

The data engineering implication is uncomfortable but important: if your team never has visible incidents, never surfaces broken assumptions, never flags a model that’s been silently wrong — that’s probably not a sign of excellence. It’s probably a sign that the environment doesn’t feel safe enough to tell the truth. The team that surfaces the most broken pipelines, the most failed tests, the most “we got this wrong” disclosures might be your best team. They’re the ones telling you what’s actually happening.

The “I’ll tell them when it’s fixed” trap is the going-dark failure mode in its purest form. An engineer discovers a problem — a data quality issue, a budget overrun, a model that produces subtly wrong results. Rather than surface it, they decide to fix it first. Each day that passes raises the psychological cost of disclosure. The problem grows. The fix gets harder. Eventually, either the engineer surfaces in a state of exhaustion with a fix and hopes nobody asks hard questions, or a stakeholder finds the problem externally — at which point the disclosure isn’t just about the bug. It’s about the three weeks of silence.

Fixing the number is not the same as fixing what the number did to people who trusted it.

The Lead Chapter: Building Conditions Where Your Team Doesn’t Go Dark

If you lead a data team, the going-dark problem is partly yours to own — not because your engineers are your responsibility in a parental sense, but because the conditions that make going dark feel safe are conditions you set.

Camille Fournier’s framing, in The Manager’s Path, is that 1:1s are like oil changes: if you skip them, plan to get stranded. The 1:1 is the primary diagnostic for whether someone is going dark, and it works only if it creates genuine psychological safety — which means it has to be something other than a status update. If every 1:1 is “what are you working on this week?”, you’ll get accurate project status and zero signal about whether someone is actually stuck, scared, or burning out.

Lara Hogan recommends opening the relationship with simple questions: what makes you grumpy? How will I know when you’re grumpy? What do you need from me when you’re struggling? These aren’t soft questions — they’re diagnostic instruments. They establish that the 1:1 is a place where the real state of things can be named.

Her Red/Yellow/Green check-in is a lightweight version of the same tool. At the start of a 1:1, an engineer can say “red” without having to explain why — just to signal that something is wrong. The permission to say “red” without an explanation is itself the thing that makes the saying possible. When explanation is required before disclosure, disclosure gets deferred until the explanation is formed — and by then, the problem has grown.

Will Larson frames visibility for leads as a mechanical practice rather than an art form. The goal isn’t to inspire your team to communicate — it’s to build systems that make communication the path of least resistance. A weekly update template, committed to as a team norm. An async standup channel where the format is so lightweight that posting takes two minutes. A retrospective cadence where surfacing what went wrong is the expected behaviour, not the exception.

Warning signs to watch for.

Going dark rarely announces itself. The early signals are easy to miss.

Standups that shift from specific to vague — “still on X” across multiple days with no detail — are an indicator. PRs that open in draft and stay there, accumulating commits but never progressing to review. Estimates that consistently slip by a day or two, week after week, without the engineer naming the slippage explicitly. A previously engaged person who starts attending meetings on camera-off, contributing less, leaving threads unresponded to.

The most useful signal is often external. When a stakeholder or adjacent team member asks you what’s happening with a project before the engineer working on it has raised anything — that’s the canary. The information is flowing around the engineer rather than through them. That’s a conversation to have in the next 1:1, and it’s worth naming directly: “I heard from finance about the revenue models. Tell me where things actually are.”

Deep work is not going dark — and the distinction matters.

There’s a real risk of over-correcting. Data engineering requires extended, cognitively-demanding focus. Paul Graham’s framing of the maker’s schedule versus the manager’s schedule is directly applicable: a data engineer tracing lineage through a DAG, rebuilding a model from first principles, or debugging an intermittent pipeline failure needs hours of uninterrupted context, not a constant stream of check-in messages.

Deep work and going dark are not the same thing. The difference is legibility.

Deep work has a pre-declared output and timebox. Going dark does not. Deep work is legible at a coarse grain — “I’m heads-down on X until Thursday, update Friday” — even when it’s illegible at a fine grain. Going dark is illegible at every grain.

The mechanism that makes the distinction: the cadence contract. Before going heads-down, the engineer declares: here’s what I’m working on, here’s when you’ll hear from me, here’s what would make me surface early. “If I’m stuck for more than a day, I’ll tell you. If I’m on track, Friday update. Ping me in #urgent if there’s a fire.” With that contract in place, silence during the week means “on track.” Without it, silence means nothing — or worse, triggers anxiety that prompts the exact interruptions that fragment deep work.

The goal is to make silence meaningful. That requires prior investment in communication, not more communication during the work itself.

The Visible Data Team Playbook

There are a handful of practices that the best data teams I’ve seen use consistently. None of them are revolutionary. Most of them are embarrassingly simple. They work because they’re habitual, not because they’re clever.

Weekly insight posts.

A practice described from data teams at companies like Netlify involves a dedicated Teams channel where each analyst posts one insight per week — not a status update, not a task list, but one thing they found interesting in the data. The format is designed to answer five questions: what am I looking at, why should you care, what caught my eye, where can you learn more, and who do you ask if you have questions.

The explicit benefit isn’t just visibility — it’s demonstrating the range of what the data team does. Stakeholders who only see the data team when something breaks, or when they submit a request, have no model of what the team actually knows. A weekly insight post gives them one. Over time, it becomes the evidence base for why the team should be resourced, why data work is complex, why the business should care about data quality. It’s not reporting. It’s positioning.

Parity dashboards during migrations.

One of the most effective anti-going-dark tools for long migrations is a parity dashboard: a side-by-side view of critical metrics computed in the old and new systems simultaneously. Before you’ve finished the migration, you build the parallel calculation and make it visible. Divergence between the two numbers becomes the artifact of progress — something demoable, something reviewable, something that gives a stakeholder a concrete thing to look at even when the underlying models are still being rebuilt.

The parity dashboard also builds confidence in a way that purely technical communication can’t. When a finance stakeholder sees that their ARR number matches between the legacy and new system for three weeks running, they start to trust the migration in a way that no amount of “we’ve tested this thoroughly” can produce. Evidence beats assertion.

Incident communication as a team norm.

The instinct to fix a data quality issue quietly and move on is almost universal among data engineers. The impulse is understandable — nobody wants to be the person who caused the problem, and surfacing it draws attention to the failure. But the silent fix has a consistent failure mode: it gets discovered later, retrospectively, in a context where the silence becomes the story rather than the fix.

Monte Carlo’s research showed that the data teams with the highest stakeholder trust tend to over-communicate around incidents — sending post-mortems after significant issues, flagging discoveries before fixes are complete, maintaining a clear status trail from “investigating” to “resolved.” What one team described as transformative wasn’t their mean time to resolution. It was the shift from reactive firefighting to proactive communication that demonstrated awareness and control. The communication is the product, as much as the fix.

A simple practice: any time you discover a data quality issue, the first act is naming it in writing — a Teams message, a GitHub issue, an incident channel post — before you start fixing it. Not to perform distress, but to start the paper trail. “Found an issue with the fct_orders null handling — investigating now, will update by EOD.” That message, and the chain of messages that follow it, is what turns an incident into evidence of a well-run team rather than evidence of a careless one.

ADRs for the decisions people will question.

The architecture decisions that feel obvious when you make them feel arbitrary when someone encounters them six months later without context. Why incremental and not table? Why Snowflake and not BigQuery? Why is revenue calculated at invoice date rather than payment receipt date? These questions have answers. The answers live in someone’s head, or in a Confluence page that nobody linked to the model.

ADRs — short markdown files in docs/adr/, committed to the repository — are the durable artifact of those decisions. They don’t need to be long. They need a title, a status, a description of the context, the decision, and the consequences. Once written and committed, they’re part of the repository’s history. They review in PRs. They can be linked from model descriptions. They don’t disappear when someone leaves the team.

dbt Exposures for impact legibility.

If you’re using dbt and you haven’t adopted Exposures, this is worth doing this week. Exposures are YAML-defined references to downstream artifacts — dashboards, ML models, reverse-ETL syncs — that extend the dbt DAG past the warehouse. A few lines in schema.yml that say “this model feeds the Weekly Revenue dashboard” means that when someone runs dbt ls --select +exposure:weekly_revenue, they get the complete list of upstream models that would need to change to affect that dashboard.

For a migration, this means you can answer “will this change break the CEO’s dashboard?” without archaeology. For a refactor, it means PR reviewers have a concrete list of stakeholders to notify. For a data quality incident, it means you know who to tell before they find out themselves.

Going Back to Mia

After my conversation with Mia — the one where she walked me through three weeks of extraordinary, invisible work — we had a more direct conversation about why none of it had been visible.

Her answer was the one Collins-Sussman identified in 2008: she didn’t want to show something until it was right. She was worried about raising a false alarm. She was concerned that surfacing work in progress would invite questions she couldn’t answer yet. There was also something she said that stuck with me: “I thought if I kept my head down long enough, I’d come up with something worth showing.”

All of those concerns were reasonable. None of them were wrong, exactly. What she’d underestimated was the cost of the silence — not to herself, but to the people around her. The finance stakeholder who’d gone to me with questions would have gone to Mia first if they’d had any signal that the work was in progress and legible. My not knowing, in a week where I’d been asked directly by leadership about the revenue model timeline, was a problem I hadn’t been equipped to solve.

What I should have done differently, and what I’ve done differently since: established the cadence contract at the start of the project, not after the silence had already built. Agreed, before the first commit, what “on track” communication would look like — what the update cadence would be, what would trigger an early surface, and what silence during a declared heads-down period meant. With that contract in place, three weeks of quiet is three weeks of “on track, as agreed.” Without it, three weeks of quiet is three weeks of “nobody knows what’s happening.”

The artifact we built together after that conversation was simple. For the remaining six weeks of the migration, Mia opened a draft PR on Monday morning with a brief plan for the week. She posted a comment each Friday with what had been done, what was pending, and any surprises. She added a parity dashboard that the finance lead could check at any time. She wrote three ADRs for the decisions that had taken the most deliberation.

The work didn’t change. The visibility changed. And the experience of the migration — for Mia, for the stakeholders, for me — changed entirely.

At the end of the project, the finance lead sent a message that I still think about. She said: “I’ve never felt so informed during a data migration. Usually I find out things are done when the thing I was waiting for just starts working.”

Mia had gone dark for three weeks. For the remaining six, she hadn’t. The difference wasn’t effort. It was artifacts.

The Closing Thought

Jeff Atwood closed his post with something direct: don’t go dark. Don’t be the developer in the room who hides their code until it’s done. Sharing work in progress feels riskier than it is. The feedback and connection it produces are worth more than the protection the silence provides.

He was right in 2008. The principle has only compounded since, as data teams have become more distributed, more remote, and more responsible for the kind of long-horizon work that naturally produces invisibility.

The tradeoff is real, though. Better visibility takes time that could go to the work itself. An ADR written at the end of a hard week is time you’re not spending fixing the problem. A parity dashboard adds engineering overhead before the migration is even done. Writing commit messages properly requires discipline you don’t always have at 5pm on a Friday when you just want to push and go home.

The case I’m making isn’t that artifacts are free. It’s that the alternative — silence — costs more than you can see while you’re inside it. The cost lands on the stakeholder who went to your manager instead of you. On the on-call engineer at 2am who has nothing to go on. On the project that looked fine until it suddenly didn’t.

Amy Edmondson’s hospital study keeps coming back to me. She’d expected to find that better units made fewer errors. Instead she found that they reported more. Her reframe: maybe the better units don’t have fewer problems. Maybe they’re just more willing to say so.

The best data teams are the same. Not the ones with the fewest broken pipelines — the ones where a broken pipeline doesn’t stay quiet for long.

Three weeks is going dark. One good artifact is enough. The rest follows from that.

The Broken Window in Your Data Pipeline

Chris Hillman — Sat, 09 May 2026 09:00:00 +1000

There’s a particular kind of data problem that doesn’t announce itself. It accumulates.

We were receiving Salesforce data through delta extraction — sensible in theory, because full snapshots can run to hundreds of terabytes and less than 1% of records change on any given day. The problem is that deltas require someone to know what “changed” means. In Salesforce, that’s less obvious than it sounds. Watch a last_modified column and you’ll miss objects that get updated when a related object changes, without their own timestamp reflecting it.

Over time: drift. Orphan records. Data that looks current because the record exists, but isn’t. The fix was documented — run a full snapshot periodically to correct the accumulated drift, painful as that was — and everyone with any context on the system knew it.

What happened was roughly this: the drift accumulated silently until something downstream looked wrong. An investigation traced it back to the delta logic. The full snapshot was run. The problem was resolved. Notes were written. The workaround was filed away.

And then, about twelve months later, the same conversation happened again.

What made that moment stick wasn’t the technical failure. It was the recognition that the window had been broken for a while. Everybody who walked past it knew it was broken. We’d even put a note next to it.

I’ve been that person — the one who wrote the documentation, filed the Jira ticket, and moved on. Which is probably why I remember the twelve-month cycle so clearly. And why I started paying attention to the pattern of it, across teams and companies, long after that particular incident.

We just hadn’t fixed it.

Bear with me here, because what I’m about to describe starts with an abandoned car in the Bronx in 1969 — and I promise it ends somewhere relevant to your dbt models.

A broken window in the Bronx, a broken window in your warehouse

A Stanford psychologist named Philip Zimbardo ran a strange experiment. He abandoned two cars — one in the Bronx, one in Palo Alto — and watched what happened.

The Bronx car was stripped within twenty-four hours. Within three days it was completely gutted. The Palo Alto car sat untouched for a week — until Zimbardo himself walked up with a sledgehammer and broke a window. Within hours, it had been stripped too.

Same car. Different signal.

Thirteen years later, criminologists James Q. Wilson and George Kelling built a theory on that experiment. If a window in a building is broken and left unrepaired, they argued, all the rest will soon follow. Not because there’s a particular breed of window-breaker lurking around, but because an unrepaired window sends a message: nobody here cares. And once that signal is broadcast, breaking more windows costs nothing.

The insight was semiotic, not structural. It wasn’t about windows. It was about what an unrepaired window communicates about the norms of a place.

The theory eventually found its way into software. Andrew Hunt and David Thomas put it into The Pragmatic Programmer almost verbatim: don’t leave broken windows — bad designs, wrong decisions, poor code — unrepaired. They’d watched clean systems deteriorate quickly once windows started breaking. The mechanism was the same. A developer looking at a messy codebase thinks: if someone else got away with being careless, maybe I can too. The norm shifts. The entropy accelerates.

Researchers at Empirical Software Engineering ran a controlled experiment — twenty-nine developers, codebases seeded with either high or low technical-debt density — and found exactly what Hunt and Thomas had intuited. Pre-existing debt measurably caused developers to introduce more debt. Non-descriptive variable names. Duplicated logic instead of reuse. Additional code smells. The broken window was statistically contagious.

It’s a compelling idea, and it maps well to software. But it doesn’t map perfectly to data engineering. And the gap between “maps well” and “maps perfectly” is where teams get into serious trouble.

The thing that’s different about data

When a broken window appears in a codebase, it’s local. A messy function, an undocumented module, a class that’s doing five things at once — these are ugly, they invite imitation, and they slow future development. But they stay where they are. They don’t go anywhere.

A broken window in a data pipeline doesn’t stay where it is.

It travels.

That’s the thing nobody really talks about when they apply broken-windows thinking to data work. In a pipeline, everything is connected. A schema drift in a source table doesn’t just make that table annoying to work with — it silently corrupts every model downstream that touches that field. Which means it corrupts every dashboard that uses those models. Which means it corrupts the metrics those dashboards expose. Which means it corrupts the business decisions made from those metrics.

The broken window is in row one. By the time someone notices, the damage is in the boardroom.

And unlike software, where the damage is visible — a stacktrace, a failing build, a crash — data damage is often invisible. The pipeline still runs. The dashboard still renders. The report still reconciles. The numbers just happen to be wrong, quietly, for reasons nobody can easily trace.

Software bugs produce noise. Bad data produces silence.

In software, the broken window signals disorder and invites imitation. In data pipelines, it does all of that and it propagates at machine speed through every downstream system that trusts the upstream to be clean. By the time the propagation is discovered, it’s usually been underway for a while.

This is the propagation problem. It’s not just that bad data begets more bad data. It’s that one bad window can contaminate an entire watershed.

What a contaminated watershed actually looks like

In May 2022, Unity Software — the game engine company — disclosed something remarkable in their SEC earnings filing. Their Audience Pinpointer ML model, the system that powered their ad targeting business, had ingested bad data from a large customer. That data had corrupted their training set.

CEO John Riccitiello on the earnings call: “we lost the value of a portion of our training data due in part to us ingesting bad data from a large customer.”

The bad data didn’t just produce one wrong prediction. It poisoned the model weights. Every subsequent retrain was built on a contaminated foundation. The model had to be taken offline, the bad data removed, and training restarted from scratch. The estimated impact was $110 million in revenue for 2022. The stock dropped 37% in a single day. Market cap losses in the billions.

The broken window wasn’t in Unity’s systems — it was in data they were ingesting. Once it crossed the boundary into their training pipeline, it propagated in the only direction data knows: forward and downstream, embedding itself into every layer of the system that touched it.

Riccitiello’s pledge after the fact was almost poignant: “We are deploying monitoring, alerting and recovery systems and processes to promptly mitigate future events.” The observability came after the disaster. The window had already broken every other window in the building.

These are the normal failure mode of connected data systems. The propagation isn’t a bug in the design — it’s inherent to the architecture. Data flows in one direction. Trust flows with it. The moment you have a pipeline, you have propagation risk.

The question isn’t whether your broken windows will propagate. It’s how far they’ll travel before anyone notices.

The tools we built to tolerate this

Modern data tooling has given us genuinely sophisticated ways to formally accept broken windows.

dbt — the transformation tool most data engineering teams live inside — has a severity configuration on data tests. You can set a test to warn instead of error. The test runs, detects a problem, and… doesn’t fail the pipeline. It records the warning and moves on.

The documentation is cheerful about it: “Maybe 1 duplicate record can count as a warning, but 10 duplicate records should count as an error.”

In principle, this is sensible. In practice, the warn threshold becomes a Schelling point. Teams configure tests to warn to avoid breaking CI. The warnings accumulate. And then — almost invariably — the warnings become wallpaper. (Go check your own test results right now. Count how many have been in warn state for more than two weeks. I’ll wait.)

dbt also has a feature called store_failures that saves failed records to an audit schema table. The test is doing its job. The failures are being recorded. They overwrite the previous run’s failures on the next run. If nobody is actively querying that audit table — and almost nobody is — the failures exist only to make the test feel like it’s being taken seriously.

It’s a passive graveyard. The window is monitored. Nobody fixes the glass.

Airflow has an equivalent pattern. The soft_fail parameter on sensors means that if an exception is raised — the source system is down, the file hasn’t arrived — the task is marked as skipped rather than failed. Downstream tasks, depending on their trigger rules, may also skip. An entire branch of your DAG quietly collapses to a skipped state, which most pipelines treat as benign, and your stakeholders get a dashboard with no data in it instead of an error message.

Retries do something similar. A flaky source that fails 30% of the time gets retries=3 configured. The task eventually succeeds on the third attempt. The 30% failure rate never surfaces as an anomaly in any meaningful way. Until the source dies entirely, at which point the symptom everyone responds to is not “this has been flaky for six months” but “this suddenly started failing today.”

None of this is the fault of the tools. dbt and Airflow are doing what they’re designed to do. The issue is that the default ergonomics of both make tolerating failure significantly easier than stopping propagation. “Don’t break the build” is a more convenient goal than “don’t ship broken data downstream.” The tools give you excellent knobs for the former and adequate knobs for the latter.

Chad Sanderson — formerly at Convoy, now building in the data contracts space — has a name for what this produces over time. He calls it the POSIWID principle: the Purpose Of a System Is What It Does. If your data pipelines are consistently producing low-quality data, and your team consistently tolerates that, then whatever you think the purpose of your data platform is, its actual purpose is to enable teams to move fast, ship without accountability, and tolerate breakages.

The broken windows aren’t exceptions. They’re the product.

We’ve built monitoring. Why isn’t it working?

Data observability emerged as a formal discipline. The concept — largely popularised by Barr Moses at Monte Carlo — drew directly from the software world’s SRE and distributed systems monitoring practices: freshness, distribution, volume, schema, lineage. Five pillars. Automated anomaly detection. Alerts in Slack when something looks wrong.

The framing was right. The problem is what happened next.

Monte Carlo’s own telemetry — across millions of monitored tables — shows that alert engagement drops 15% when a Slack channel exceeds 50 alerts per week, and a further 20% past 100 per week. The data observability system has a data quality problem: the signal-to-noise ratio degrades, and so does the human response.

This isn’t specific to data. The clinical literature on alarm fatigue is sobering. ICU environments average more than 150 alarms per bed per day. Studies consistently find that 72% to 99% of those alarms are non-actionable. The clinical community’s response to this was institutional — the Joint Commission made alarm safety a National Patient Safety Goal in 2014 — because they recognised that a monitoring system that produces more noise than signal doesn’t just fail to help. It actively worsens outcomes by training humans to stop responding.

Cybersecurity teams face the same thing. Security operations centres receive thousands of alerts daily; most go unaddressed, not because analysts are lazy, but because the ratio of genuine signals to false positives has degraded to the point where sustained attention is cognitively impossible.

The Google SRE book is direct on this: “Every page should be actionable. If a page merely merits a robotic response, it shouldn’t be a page.”

In data engineering, the equivalent of a “robotic response” is the acknowledged-and-unresolved alert. The monitor that fires every Tuesday morning, gets a thumbs up in Teams, gets added to the “known issues” document, and never gets fixed. The window is now monitored. The fact that it’s monitored makes the team feel responsible. The window stays broken.

There’s a term for this: observability theatre. The infrastructure of visibility exists. The dashboards are green. Nobody’s actually looking at the glass.

This is where the broken windows metaphor earns its keep most fully. Wilson and Kelling weren’t saying that disorder is bad because it looks bad. They were saying that an unrepaired broken window sends a signal that no one cares — and that signal is the actual mechanism of decay. Monitoring a broken window without repairing it sends exactly the same signal. Possibly a worse one, because now everyone knows the problem is being tracked and nobody’s acting on it. The norm becomes: acknowledged problems are not necessarily fixed problems.

The stakeholder sees it differently

Here is the asymmetry that data teams consistently underestimate.

When an engineer looks at a broken window in a data pipeline — a column with a known null rate, a test that’s been in warn state for three weeks, a DAG that soft-fails every second run — they see technical debt. A problem to eventually be addressed. Something that exists on a spectrum of severity, probably not at the top of the list.

When a business stakeholder encounters the downstream consequence of that broken window — a dashboard that contradicts a number they just presented to the CFO, a metric that moved in an unexplained direction, a report that doesn’t reconcile with another report — they don’t think “technical debt.” They think: can I trust the data team?

Benn Stancil — one of the clearest thinkers writing about this — put it well: “Trust is built, and blown up, by the outputs — and specifically, the consistency of those outputs.” He uses a vivid image that I keep coming back to: “Data and the dashboards that display it create a shared sense of reality. Looking at two dashboards that don’t match is like looking out two adjacent windows and not seeing the same thing.”

That’s the experience of broken-window propagation from the stakeholder’s side. Two adjacent windows. Different views. A reality that doesn’t cohere.

Monte Carlo’s 2023 State of Data Quality research put a number to the trust inversion that most data engineers already sense. In 2022, 47% of respondents reported that business stakeholders identified data issues first “all or most of the time” — more often than the data team itself. By 2023, that figure had risen to 74%. (Three in four. Let that land.)

Think about what that means. In three out of four data incidents, the people who rely on the data found the problem before the people who built it. The pipeline runs, the test passes, the dashboard renders, and somewhere downstream a business analyst is staring at a number that doesn’t look right and is about to send a message that starts with: “Quick question about this figure…”

The damage isn’t technical. It’s relational. And it compounds in a way that’s harder to reverse than any schema migration.

Thomas Redman — who has spent decades studying data quality — made this point in Harvard Business Review: “When data are unreliable, managers quickly lose faith in them and fall back on their intuition to make decisions.” Once that happens, you haven’t just produced bad data. You’ve trained decision-makers to ignore good data too, because they can no longer distinguish between the two.

The broken window didn’t just propagate through the pipeline. It propagated into the culture.

Nobody owns the broken window

There’s a specific pattern worth naming, because it’s where most data teams I’ve worked with have the most unacknowledged broken windows: the orphaned asset.

The pipeline that was built by an engineer who left eighteen months ago. The table in the warehouse that exists in the data catalogue but whose owner field says “Unknown” or, worse, the name of someone who’s no longer at the company. The dbt model that runs in production, that feeds two dashboards, that nobody on the current team can fully explain.

These aren’t broken in any obvious sense. They run. They load. The tests pass, or they’re set to warn. But they’re broken windows in the original sense: they signal that nobody cares. And because nobody owns them, nobody’s in a position to repair them even when something goes wrong.

What typically happens is this: a new engineer joins the team. They need to understand how the data flows. They look at the catalogue. They look at the undocumented table. They look at the model that references it. They look at the Jira ticket from fourteen months ago that says “Known issue — downstream teams aware.” And they make a rational decision: don’t touch it, build around it, replicate it if necessary.

The broken window has now inspired a new window. Same mechanism as the Bronx car. Different materials.

The organisational research on this is unambiguous. Ron Westrum’s typology of organisational cultures — validated empirically by the DORA research programme across thousands of software teams — found that information flow predicts safety and performance more reliably than almost any structural variable. In pathological cultures, information is hoarded or withheld for political reasons. In generative cultures, information flows freely, failures are shared, and problems get fixed because surfacing problems is rewarded rather than penalised.

Amy Edmondson’s research on psychological safety adds the other half: 85% of employees have withheld important information from their manager due to fear of speaking up. In data teams, this looks like: the junior engineer who noticed the null rate had been climbing for two weeks and didn’t raise it because they weren’t sure it was their call to make. The analyst who suspected the metric definition had drifted but didn’t want to slow down the dashboard delivery. The data engineer who knew the pipeline was flaky but figured someone more senior would have noticed if it really mattered.

The broken window gets left unrepaired not because nobody sees it, but because the culture hasn’t made repair feel safe or worthwhile.

Edmondson on this: “If there’s no bad news, remind yourself: It’s not that it’s not there. It’s that you’re not hearing about it.”

The original theory’s mistake — and ours

Before we get to what to actually do about this, it’s worth pausing on where the original theory went wrong. Because data engineering is at risk of making exactly the same mistake.

When Wilson and Kelling published their 1982 Atlantic article, the theory was nuanced. It was about signals and norms. It was explicitly a theory about community — about how residents and police could work together to maintain shared standards in public spaces. It was, in Kelling’s own framing, a theory of collective efficacy.

What cities did with it was zero tolerance. Mass arrests for minor offences. Stop-and-frisk. 685,000 stops in New York City in a single year, more than 85% of them finding nothing at all.

Kelling’s reaction, when he learned how comprehensively his theory had been misapplied: “Oh, shit.”

The software world made a version of the same mistake when it imported broken windows thinking. “Don’t live with broken windows” became, for some teams, a linting rule for everything, a zero-tolerance policy for code smells, a culture of perfectionism that burned people out and produced beautiful codebases that never shipped.

Adam Tornhill’s research at CodeScene offers the corrective: not all broken windows matter equally. In a 400,000-line codebase with 89 developers, his analysis found that 4% of the code was responsible for 72% of the defects. The broken windows that needed fixing weren’t distributed evenly. They clustered in hotspots — files that were both frequently changed and highly complex. Fix the hotspots. Let the cold, ugly, stable corner of the codebase sit.

The principle for data engineering is the same. Zero-tolerance data quality enforcement — failing every pipeline on every test at severity error — produces fragile systems and tired teams. It’s the data equivalent of arresting everyone for jaywalking. The signal gets lost in the noise.

What the original theory was actually pointing at was something more like: maintain the norm. Make it visible that people care. Ensure that the broken windows that matter — the ones that propagate, the ones in the high-traffic, high-trust parts of the system — get fixed promptly and publicly. Not every window. The ones that signal whether anyone’s paying attention.

What collective efficacy looks like in a data team

Sampson, Raudenbush, and Earls — the Chicago sociologists who ran the most rigorous empirical test of broken windows theory — found that the variable that actually predicted neighbourhood safety wasn’t the presence or absence of disorder. It was collective efficacy: the combination of social cohesion and shared willingness to intervene. Neighbours who knew each other, trusted each other, and were willing to act on behalf of each other’s wellbeing.

The policy prescription that emerges from their work is very different from zero tolerance. It’s community investment. Shared ownership. Making the costs of non-intervention visible.

The equivalent in data engineering starts with one thing, and if you do nothing else in this list, do this one:

Make propagation visible before anything else.

Column-level data lineage — now available in most modern data observability platforms and increasingly in the open-source OpenLineage standard — lets you answer the question: if this column is wrong, what does it break? That question should be answerable in seconds, not hours. Teams that can visualise propagation chains respond to broken windows faster because they can see the radius of the damage before they decide whether to act.

This matters beyond incident response. When you can show an engineer that the null column they’re tolerating in a staging model feeds seven downstream gold-layer tables, three dashboards, and a Snowflake share that two other teams consume — the calculus on whether to fix it changes. The broken window stops being an abstract code quality concern and becomes a blast radius. That’s a much more compelling argument for repair than “we should improve our data quality culture.”

Treat ownership as load-bearing infrastructure, not housekeeping.

Every pipeline, every table, every model should have a named owner. Not a team. A person. The Jira ticket that says “Known issue — downstream teams aware” with no assigned owner is the data equivalent of a broken window with an orange cone next to it. The cone acknowledges the hazard. Nobody’s fixing the glass.

The counterargument is always resourcing — “we don’t have time to own everything properly.” That’s true, and worth taking seriously. But the right response isn’t to pretend you own things you don’t. It’s to make orphaned assets visible and have an honest conversation about whether the organisation can afford to run production pipelines with no accountable maintainer. Most of the time, when the question is asked that directly, the answer is no.

Calibrate your tolerance patterns deliberately — and actually revisit them.

The dbt severity: warn setting and Airflow’s soft_fail are legitimate tools when they’re conscious decisions: “this condition is a signal worth tracking but not a pipeline-stopper, and here’s the threshold at which that changes.” The problem is that almost nobody uses them that way. They’re the path of least resistance to avoid a broken CI build — and six months later you audit your test results and discover forty tests set to warn that haven’t been at zero failures since the day they were written.

Treat warn-severity tests the way you treat Jira tickets that never get triaged. Set a review cadence. If a test has been consistently warning for more than two weeks without an associated investigation, it’s either a bug that needs fixing or a threshold that needs changing. “Known issue” is not a status. It’s an admission.

Fix alert fatigue before it hollows out your monitoring culture.

The Teams channel where every data quality alert lands is the digital equivalent of a neighbourhood where every broken window gets photographed and logged and nobody ever fixes one. The log is evidence that someone noticed. It’s not evidence that anyone will act.

Set alert thresholds that produce actionable signals. The Google SRE principle applies directly: if an alert merits a robotic response, it shouldn’t be an alert. If your first instinct on seeing a particular monitor fire is to click acknowledge and move on, that monitor is producing noise, not signal. Change it or delete it. Grouping alerts by lineage — “these five monitors fired because of one upstream schema change” rather than five separate pings — reduces volume while making propagation visible at the moment of failure, which is exactly when you want it.

Make broken windows visible to leadership, because right now the cost is invisible.

Data quality work is famously hard to demonstrate. Pipelines that run cleanly leave no artefact. A well-maintained model with a 0% null rate looks identical to a freshly built one. There’s no natural demo format for “nothing bad happened this week.”

This invisibility is part of why broken windows accumulate. The cost of prevention is hidden in maintenance time that doesn’t get counted. The cost of failure gets absorbed by analysts who spend their Tuesdays reconciling numbers instead of answering strategic questions, by data engineers who spend their Fridays investigating stakeholder tickets, by business decisions made on incorrect information that can’t be traced back to a specific incident.

Quantify the propagation radius when incidents do occur. How many downstream models were affected? How many stakeholders were exposed to incorrect data, and for how long? What was the resolution effort in hours? Those numbers, tracked consistently, build a case for investment that abstract arguments about data quality never will.

The window was broken before I noticed

What we ended up doing was building the fix into the operating rhythm. A scheduled data drift catch-up: a full snapshot weekly, or monthly, depending on the volatility of the Salesforce object — not to respond to an incident, but to systematically correct the accumulated drift before it became visible to anyone downstream. We stopped waiting for the escalation. We made the repair part of the architecture.

The broken window was still there, technically. The delta logic still had its blind spots around hidden relationships. But we stopped waiting for it to propagate before we addressed it. Some broken windows aren’t things you permanently seal — they’re things you build a maintenance routine around. Knowing that is the fix.

What stayed with me wasn’t the technical solution, which was straightforward enough once we committed to it. What stayed with me was the calendar predictability of the escalation cycle before the fix. The fact that we had a known issue, a known workaround, documentation of both — and still managed to have the same downstream discovery, the same investigation, the same remediation conversation, almost exactly twelve months apart.

The window wasn’t hidden. It was visible in the delta logs if you knew where to look. We just hadn’t made it someone’s job to look, and we hadn’t made repair part of the routine. We’d fixed the glass, filed the paperwork, and assumed the problem was solved. Until the same signal appeared in the same downstream reports, a year later.

What I keep coming back to, though, is something that was true of that experience and is true of most of the data quality failures I’ve seen since: the window wasn’t broken secretly. It wasn’t hidden. It was visible, it was acknowledged, and it was left.

We had the monitoring. We had the test. We had the Jira ticket.

We had, in other words, all the infrastructure of concern. What we didn’t have was the collective agreement that repair mattered — that the propagation radius of that one broken window was large enough, and trust-eroding enough, that it justified stopping what we were doing and fixing the glass.

That’s the thing broken windows theory is actually about, underneath all the criminology and the code smells and the schema drift. It’s about the signal that unrepaired damage sends. Not to the criminals or the developers or the data consumers. To the people who are supposed to care about the system.

When a broken window sits long enough in a data pipeline, it stops being a problem and starts being a norm. The new engineer doesn’t flag it — they work around it. The analyst doesn’t escalate it — they add a caveat to their report. The data engineer doesn’t fix it — they document it.

And somewhere downstream, a business decision gets made on numbers that were broken before anyone thought to check.

The window isn’t just in the pipeline. The window is in the standard you’re willing to keep.

Five Worlds of Data Engineering

Chris Hillman — Sat, 02 May 2026 09:00:00 +1000

You watch a conference talk about implementing data contracts, and nobody mentions that the advice assumes you have multiple teams producing data — which you don’t. You read a post declaring “if you’re still using stored procedures in 2026, you’re doing it wrong,” and the comments erupt. Half the people are nodding along. Half are furious. Both sides are right. They’re just living in different worlds and don’t realise it.

That mismatch — smart people giving each other advice that doesn’t apply — is the thing that almost never gets named. And the reason it doesn’t get named is that most public data engineering discourse is produced by and for one particular world, while pretending to speak for all of them.

I’ve worked across a few of these worlds myself — SQL Server migrations to Teradata, then Teradata to S3, regulatory reporting under APRA and ASIC, a data mesh initiative at scale, and now a university environment that runs closer to a startup than anything I expected. The advice that kept me out of trouble in one would have gotten me fired in the other. That experience is what’s behind this taxonomy.

I think there are five worlds here. Sometimes they intersect. Often they don’t.

World 1: The Modern Analytics Shop

This is the world most people picture when they hear “data engineering.” A team of two to ten engineers at a startup, scale-up, or mid-size company. Cloud-native from day one, or recently migrated. The stack reads like a vendor sponsor list: Snowflake or BigQuery or Databricks, dbt for transformations, Fivetran or Airbyte for ingestion, Looker or Metabase for dashboards. Everything managed. Everything SaaS. GitHub Actions wiring it together.

What matters here is speed. Getting value to stakeholders fast. Shipping a working dashboard before the quarterly review. Iterating on models without a change advisory board. The team is small enough that everyone knows the codebase, and requirements come from a Teams message, not a 40-page specification document.

I find myself in this world more than I expected right now. The current role is an institution that by any measure is large and complex — but the data team is small, the autonomy is real, and the energy genuinely feels like a startup. New tools, fast decisions, the ability to actually make change. It’s a reminder that World 1 isn’t only about company size. It’s about how the team operates.

This is also a great world to work in, especially early in your career. The tooling is mature, the feedback loops are tight, and the problems are tractable.

Here’s the thing, though. This is also the world that roughly 80% of public data engineering content is written for and about. Not because it’s the most common world — it isn’t — but because it’s the most fundable. The vendor-funded conference talks, the sponsored podcasts, the developer advocate blog posts, the LinkedIn hot takes — they overwhelmingly reflect this world. Developer advocates write about the stack their employer sells (this is not a dig — it’s the job). Conference sponsors want talks that showcase their tools in the most flattering light. The technical depth suffers because the incentive is awareness, not education.

None of that is wrong. But it creates a gravitational pull that distorts the whole discourse. When someone writes “the right way to do data engineering,” they almost always mean this world. And if you’re in a different one and don’t recognise the mismatch, you end up feeling like you’re doing it wrong when you’re actually just solving a different problem.

Advice that works here but rarely travels: “Just use dbt.” “Schema-on-read is fine for now.” “You don’t need a data catalog yet.” “Start with a star schema and iterate.”

World 2: The Enterprise Legacy Estate

This is the world of large organisations — banks, insurers, manufacturers, utilities, healthcare systems, government agencies — with data infrastructure that predates the cloud. Often predates the people currently maintaining it. Teams of twenty to two hundred data professionals spread across business units that don’t always talk to each other.

The stack tells a different story: Informatica, SSIS, Ab Initio, Teradata, Oracle, maybe an on-prem Hadoop cluster that someone championed in 2014 and nobody’s had the political capital to decommission. Perhaps some Snowflake or Databricks grafted on top, creating a hybrid that’s more complex than either system alone.

I lived this migration arc twice — once from SQL Server to Teradata, and again from Teradata to S3 and Starburst. Each time, the temptation was to treat the old system as a problem to escape rather than a body of knowledge to decode. Each time, the engineers who did the most damage were the ones who arrived with the new stack and immediately started designing around the old one rather than understanding it first.

What matters here is stability. Not breaking things. Migration plans that span years, not sprints. Governance and lineage aren’t aspirational — they’re audit requirements. And politics, because the data warehouse your predecessor built in 2011 is somebody’s empire (you know the one), and you can’t “just replace it” without navigating a web of organisational power dynamics that no architecture diagram captures.

This is the world where refactoring beats rebuilding every time. That fact table with 200 columns? Those bridge tables nobody understands? The slowly-changing-dimension-within-a-slowly-changing-dimension? They’re not bugs — they’re reality encoded. Every weird modelling choice represents a business rule someone fought to understand. The “clean” data vault remodel will eventually end up with the same complexity, just distributed across more tables with more confusing names.

Advice from World 1 can be actively dangerous here. “Just rewrite it” destroys institutional memory. “Adopt a lakehouse architecture” sounds great until you realise you have 4,000 stored procedures that encode fifteen years of business rules, and nobody documented them. The engineer in this world isn’t slow because they’re behind the curve. They’re careful because the cost of breaking something is measured in regulatory findings and executive phone calls, not a failed CI check.

Advice that works here but rarely travels: “Document everything before you touch it.” “Strangler fig, never big bang.” “Spend more time understanding why it was built this way than planning what to replace it with.” “The weird WHERE clause isn’t a bug — it’s institutional memory.”

World 3: The Product Engine

This is the world where data engineering directly powers an outcome the business delivers. That might be real-time personalisation, recommendation engines, or pricing algorithms. But it also includes building data products that drive campaigns, marketing attribution, and business decisions — cases where the engineer’s output flows into something customers or commercial teams act on directly, rather than landing in a dashboard that an analyst reviews on Monday morning.

The real-time variant of this world has its own stack: Kafka or Kinesis for streaming, Spark or Flink for processing, feature stores for serving machine learning models. Often custom infrastructure because off-the-shelf tools can’t meet the latency requirements.

But the defining characteristic isn’t milliseconds — it’s consequence. If a pipeline breaks in this world, someone notices immediately. A campaign fires with the wrong audience. A pricing decision gets made on stale data. A recommendation engine serves the same product to everyone because the features stopped updating overnight. The engineer is accountable to an outcome, not just a pipeline.

I’ve worked in this space building data products that powered business campaigns and marketing decisions. Not real-time serving in the p99 latency sense, but consequential enough that a broken pipeline meant a broken business process. That accountability changes how you think about testing, monitoring, and what “done” actually means.

The skills that matter here — operational thinking, system design, understanding how your data is consumed downstream — overlap more with software engineering and product thinking than with SQL and dbt. When someone says “data engineering is just SQL and orchestration,” someone in this world quietly closes the tab.

Advice that works here but rarely travels: “Treat pipelines like production services.” “Your tests need to run in CI, not in a notebook.” “Schema evolution is a deployment problem, not a modelling problem.” “The consumer of your data is your customer — know what breaks their day.”

World 4: The Regulated Pipeline

This is the world defined by compliance. In Australia: APRA for prudential standards, ASIC for financial services conduct, AFCA for dispute resolution obligations, and the Privacy Act for anything touching personal information. Internationally: HIPAA, SOX, MiFID, GDPR. The defining characteristic isn’t the technology — it’s that regulatory requirements shape every architectural decision before the first line of code is written.

The stack is whatever passed the security review. Often years behind the cutting edge because new tools need months of compliance evaluation before they’re approved. Data lineage tools aren’t nice-to-haves — they’re audit requirements. Encryption isn’t a best practice — it’s a legal mandate. Access controls aren’t “we should get around to that” — they’re the first thing you build.

What matters is auditability. Can you answer “who accessed what, when, and why” for any record in the system? Can you prove data lineage from source to report? Can you demonstrate that a deletion request was honoured within the legally mandated timeframe? Your data deletion pipeline is as important as your ingestion pipeline, and it needs the same rigour.

I spent significant time in this world — financial services, where regulatory reporting wasn’t a background concern but a core deliverable. APRA and ASIC don’t ask nicely. The audit wasn’t a hypothetical and the regulator’s question list arrived without warning. The engineers I worked alongside weren’t slow because they lacked ambition. They were deliberate because the cost of getting it wrong wasn’t a postmortem — it was a regulatory finding, a remediation programme, and occasionally a front-page story.

The trap for engineers entering this world from World 1 is assuming that governance is bureaucracy. It isn’t. Governance is architecture. The compliance requirements aren’t obstacles to good engineering — they’re constraints that shape what good engineering looks like. The best engineers I’ve worked with in regulated environments don’t fight the constraints. They design systems where compliance is a property of the architecture itself, not a layer bolted on top.

Advice that works here but rarely travels: “Governance is architecture, not bureaucracy.” “If you can’t prove lineage, you can’t ship it.” “The security review IS the sprint.” “Your data deletion pipeline is as important as your data ingestion pipeline.”

World 5: The Internal Platform

This is the world of organisations large enough that the data team has split into “platform” and “domain” functions. The platform team builds the infrastructure, tooling, and self-service capabilities that other data teams consume. Data mesh adopters. Companies with fifty or more data practitioners who realised that a centralised team can’t scale to serve every domain’s needs.

The stack is about abstraction and enablement: Kubernetes, self-hosted Airflow or Dagster or Prefect, internal developer portals, data contracts, schema registries, internal PyPI packages. Often custom abstractions layered on top of cloud services, designed to give domain teams guardrails without bottlenecks.

I experienced this world through ANZx — ANZ’s digital platform initiative — where data mesh concepts were applied at scale. Not as a theoretical framework on a conference slide, but as a real attempt to distribute data ownership across domains while maintaining coherence at the platform level. The challenge wasn’t the technology. It was getting domain teams to think like data product owners rather than data consumers. That shift is harder than any infrastructure problem.

What matters here is adoption. Not how many pipelines you build, but how many pipelines your consumers build without needing your help. Your success metric isn’t “pipelines shipped” — it’s “time to first pipeline for a new domain team.” Developer experience for internal consumers is your product, and if the experience is poor, your consumers will route around you. They’ll spin up their own Snowflake account, write their own ingestion scripts, and create exactly the kind of ungoverned sprawl your platform was supposed to prevent.

If adopting your platform requires a Jira ticket and a two-week wait, you’ve already lost. The shadow pipelines will multiply, and nobody will tell you until the audit.

Advice that works here but rarely travels: “Treat internal teams as customers with choices.” “Self-service is the goal, not centralised delivery.” “Your documentation IS the product.” “Measure adoption, not output.”

So What?

Most things in data engineering are the same no matter which world you’re in. Data quality matters everywhere. Testing matters everywhere. Documentation matters everywhere. Understanding the business context behind the data — that’s universal, and I’d argue it’s the most undervalued skill in the profession.

But not everything transfers. And when somebody tells you about methodology — at a conference, on LinkedIn, in a blog post (including mine) — it’s worth thinking about which world they’re coming from.

Here’s something I’ve noticed when hiring vetran data engineers: the ones who stand out aren’t necessarily the deepest specialists in any one world. They’re the ones who’ve visited enough worlds to know the laws. They don’t have to have lived in each one for years — but they’ve spent enough time in the regulated environment to understand why governance isn’t bureaucracy. Enough time in the enterprise to know that “just rewrite it” destroys things. Enough time in the product engine to feel what accountability to an outcome actually means. That layered experience is what hardens an engineer. It’s what lets them walk into an unfamiliar environment and read it quickly — recognising the constraints, the trade-offs, the things that will matter before anyone tells them.

There’s also something worth saying about what these worlds are not: interchangeable, or combinable. I’ve never seen an organisation that genuinely encompasses all five. You occasionally see attempts — usually during a data mesh initiative — and they tend to produce something more complex than any individual world, with the clarity of none of them. The worlds don’t collapse into a super-universe. They coexist, imperfectly, inside large organisations. A regulated pipeline team and a modern analytics shop can operate within the same company and have almost nothing in common in terms of how they work, what they value, and what good looks like.

The tradeoff is real and worth naming plainly: the content that’s most abundant — the conference talks, the vendor posts, the LinkedIn hot takes — is calibrated for a world that isn’t yours if you’re in World 2, 3, 4, or 5. There’s more of it than ever, and it’s easier than ever to access. What you give up is the ability to consume it passively. Filtering for relevance is now part of the job, and nobody puts that in the job description.

When you read advice — any advice, including everything on this blog — ask yourself which world it’s coming from. If it doesn’t apply to yours, that’s not a failing on your part or theirs. It just means they’re in a different world.

Now you know to notice.

Your Data Platform Costs More Than It Should

Chris Hillman — Sat, 25 Apr 2026 09:00:00 +1100

Let me tell you about the moment I stopped treating cloud costs as someone else’s problem.

We were three months into a Snowflake migration. Everything was humming. Pipelines were green, dashboards were fast, the analytics team was happier than I’d seen them before. I felt good about the work we’d done.

Then finance forwarded me the invoice.

The number wasn’t catastrophic. But it was significantly higher than what we’d budgeted, and when I started digging, I couldn’t explain where most of it was going. I knew we had warehouses running. I knew we had pipelines executing. But I couldn’t tell you which warehouse was responsible for what cost, which pipelines were the expensive ones, or whether the money was well spent. I had built a platform I was proud of — and I had no idea what it actually cost to operate.

That’s the moment that changed how I think about data engineering. Not because of the dollar amount, but because of the realisation underneath it: I had built something I couldn’t explain to the people paying for it. And if I couldn’t explain it, I couldn’t defend it. And if I couldn’t defend it, someone else would make the decisions for me — someone who didn’t understand why the platform mattered.

I’m telling you this because cost management is one of those things that sounds like a finance problem until you experience the consequences firsthand. It’s not about being cheap. It’s about being intentional. It’s about knowing that every credit you spend is buying something valuable — and being able to prove it when someone asks.

The data engineers who understand their costs don’t just save money. They earn trust. They get budget for the projects that matter. They sleep better because they’ve eliminated the waste that eventually becomes someone else’s excuse to cut headcount or freeze hiring.

This article is about building that understanding. Not with a vendor’s optimisation tool or a consultant’s audit — but with the habits, queries, and mental models that let you own your platform’s economics from the inside. Everything here is grounded in Snowflake and AWS, with specific code you can run today.

You can’t optimise what you can’t see

Before you touch a single warehouse configuration, you need to answer one question: where is the money going?

Most teams skip this step. They read a blog post about auto-suspend settings, change a few defaults, and call it optimisation. That’s like going on a diet by switching to diet soda while eating three pizzas a day. The soda wasn’t the problem.

Here’s the query I run first on every Snowflake environment I touch. It tells you which warehouses are consuming the most credits over the last 30 days:

SELECT
 warehouse_name,
 SUM(credits_used) AS total_credits,
 SUM(credits_used) * 3.00 AS estimated_cost_usd, -- adjust your credit price
 COUNT(DISTINCT DATE_TRUNC('day', start_time)) AS active_days,
 ROUND(SUM(credits_used) / COUNT(DISTINCT DATE_TRUNC('day', start_time)), 2) AS credits_per_day
FROM snowflake.account_usage.warehouse_metering_history
WHERE start_time >= DATEADD('day', -30, CURRENT_TIMESTAMP())
GROUP BY warehouse_name
ORDER BY total_credits DESC;

Run that. Right now. I’ll wait.

If you’re like most teams, 70–80% of your credits come from two or three warehouses. That’s your starting point. Not everything — just the expensive stuff.

Now do the same thing for queries. This one finds your top 20 most expensive queries by bytes scanned:

SELECT
 query_id,
 query_text,
 warehouse_name,
 user_name,
 total_elapsed_time / 1000 AS elapsed_seconds,
 bytes_scanned / POWER(1024, 3) AS gb_scanned,
 partitions_scanned,
 partitions_total,
 ROUND(partitions_scanned / NULLIF(partitions_total, 0) * 100, 1) AS pct_partitions_scanned
FROM
 snowflake.account_usage.query_history
WHERE
 start_time >= DATEADD('day', -7, CURRENT_TIMESTAMP())
 AND bytes_scanned > 0
ORDER BY
 bytes_scanned DESC
LIMIT 20;

Pay attention to that pct_partitions_scanned column. If you see queries scanning 90–100% of a table’s partitions, those queries aren’t benefiting from clustering or partition pruning. That’s where the big wins hide.

This 15-minute exercise — top warehouses, top queries — tells you more about your cost profile than any dashboard. It’s the equivalent of checking your bank statement before creating a budget. Obvious in hindsight. Almost nobody does it.

The idle tax is your single biggest waste

Here’s what nobody tells you about Snowflake billing: you pay for compute in 60-second minimums. Every time a warehouse resumes from suspension, you’re billed for at least one minute — regardless of whether the query takes 2 seconds or 58 seconds.

This matters more than it sounds. Picture a BI tool like Metabase or Tableau hitting your warehouse with 20 small metadata queries over 15 minutes. If your warehouse auto-suspends after 5 minutes (Snowflake’s default for many setups), it might suspend and resume multiple times during that window. Each resume triggers another 60-second charge.

Twenty queries that take 3 seconds each? That’s 60 seconds of actual compute. But if the warehouse suspends and resumes 4 times, you’re billed for 240 seconds. A 4x overhead.

The fix is straightforward but requires thought:

-- For transformation warehouses (predictable, bursty workloads)
ALTER WAREHOUSE transform_wh
 SET AUTO_SUSPEND = 60
 AUTO_RESUME = TRUE
 WAREHOUSE_SIZE = 'MEDIUM';

-- For BI/dashboard warehouses (frequent small queries)
ALTER WAREHOUSE bi_wh
 SET AUTO_SUSPEND = 300 -- 5 min keeps it warm between dashboard interactions
 AUTO_RESUME = TRUE
 WAREHOUSE_SIZE = 'SMALL';

-- For ad-hoc/analyst warehouses (unpredictable, intermittent)
ALTER WAREHOUSE adhoc_wh
 SET AUTO_SUSPEND = 60
 AUTO_RESUME = TRUE
 WAREHOUSE_SIZE = 'XSMALL';

The principle: transformation warehouses should suspend aggressively because they run in defined windows with gaps between runs. BI warehouses should stay warm a bit longer because dashboard users generate clusters of queries with short pauses in between. Ad-hoc warehouses should be small and aggressive — analysts can tolerate a 1–2 second resume delay.

Here’s how to find warehouses that are running but idle — the silent cost killer:

SELECT
 warehouse_name,
 SUM(credits_used) AS total_credits,
 SUM(credits_used_compute) AS compute_credits,
 SUM(credits_used_cloud_services) AS cloud_credits,
 ROUND(
 (SUM(credits_used) - SUM(credits_used_compute))
 / NULLIF(SUM(credits_used), 0) * 100, 1
 ) AS pct_idle_cost
FROM
 snowflake.account_usage.warehouse_metering_history
WHERE
 start_time >= DATEADD('day', -30, CURRENT_TIMESTAMP())
GROUP BY
 warehouse_name
HAVING
 pct_idle_cost > 20
ORDER BY
 total_credits DESC;

If pct_idle_cost is above 20% on any warehouse, you’re burning money on idle time. Tighten the auto-suspend, or investigate what’s keeping the warehouse awake between queries.

The most expensive query is the one that scans everything

Snowflake charges you for the compute time your queries consume, and nothing drives compute time like full table scans. The two most common culprits are queries that wrap filter columns in functions, and queries that select more columns than they need.

Here’s what I mean. This query looks innocent:

-- Expensive: function on the filter column disables pruning
SELECT
 order_id, customer_id, total_amount
FROM
 analytics.fct_orders
WHERE
 DATE(order_timestamp) = '2026-03-01';

But it’s quietly terrible. Wrapping order_timestamp in DATE() forces Snowflake to evaluate every row before filtering. The query planner can’t use micro-partition metadata to skip irrelevant partitions.

This version does the same thing, but lets Snowflake prune:

-- Cheap: range filter on raw column enables pruning
SELECT
 order_id, customer_id, total_amount
FROM
 analytics.fct_orders
WHERE
 order_timestamp >= '2026-03-01'
 AND order_timestamp < '2026-03-02';

The difference can be enormous on large tables — I’ve seen this single change reduce bytes scanned by 95% on billion-row fact tables.

The other silent killer is SELECT * in intermediate transformations. In a dbt project, I often find staging models that look like this:

-- stg_orders.sql (before)
SELECT *
FROM
 {{ source('raw', 'orders') }}
WHERE
 _loaded_at > (SELECT MAX(_loaded_at) FROM {{ this }})

That SELECT * pulls every column from the source — including columns nobody downstream ever uses. In a columnar store like Snowflake, you only pay to scan the columns you reference. Trimming to the columns you actually need is free performance:

-- stg_orders.sql (after)
SELECT
 order_id,
 customer_id,
 order_timestamp,
 total_amount,
 status,
 _loaded_at
FROM
 {{ source('raw', 'orders') }}
WHERE
 _loaded_at > (SELECT MAX(_loaded_at) FROM {{ this }})

This isn’t premature optimisation. It’s hygiene. Every SELECT * in your transformation layer is a small tax you pay on every single run.

Full refreshes are the most expensive default in data engineering

If you’re using dbt — and most teams on Snowflake are — the default materialisation is either view or table. Both have the same problem at scale: they rebuild everything, every time.

A table materialisation on a 500-million-row fact table means Snowflake reads, transforms, and writes 500 million rows every run. Even if only 50,000 rows changed since yesterday. That’s a 10,000x overhead.

Switching to incremental materialisation is the single highest-impact cost change most teams can make:

-- fct_orders.sql
{{
 config(
 materialized='incremental',
 unique_key='order_id',
 incremental_strategy='merge',
 on_schema_change='append_new_columns'
 )
}}

SELECT
 order_id,
 customer_id,
 order_timestamp,
 total_amount,
 status,
 updated_at
FROM
 {{ ref('stg_orders') }}

{% if is_incremental() %}
WHERE updated_at > (
 SELECT DATEADD('hour', -3, MAX(updated_at))
 FROM {{ this }}
)
{% endif %}

A few things worth noting here. The DATEADD('hour', -3, ...) creates a 3-hour lookback window. This catches late-arriving data and handles clock skew between source systems. Without it, you’ll miss rows that arrive slightly out of order — and your numbers will silently drift.

The unique_key with incremental_strategy='merge' means Snowflake will update existing rows and insert new ones. This is essential for tables where source records get modified after initial load (order status changes, for example).

The rule of thumb: if a table has more than a million rows and less than 20% changes per run, make it incremental. The cost savings are usually 80–95% on that model’s compute.

But — and this is important — schedule a periodic full refresh to correct any drift. I typically set up a weekly dbt build --full-refresh --select fct_orders via a separate Airflow DAG or GitHub Actions workflow. Belt and suspenders.

The hidden costs your Snowflake dashboard won’t show you

Most teams monitor warehouse credits because that’s what the Snowflake UI makes visible. But there are cost vectors that don’t show up in the obvious places.

Cloud services credits accrue when queries use Snowflake’s coordination layer — query compilation, metadata operations, result set caching. Normally this is covered by a 10% “free” adjustment against your compute credits. But if you have BI tools making thousands of small metadata queries, cloud services can exceed that adjustment and start costing real money.

Find them:

SELECT
 DATE_TRUNC('day', usage_date) AS day,
 SUM(credits_used) AS total_credits,
 SUM(credits_adjustment_cloud_services) AS cloud_services_adjustment,
 CASE
 WHEN SUM(credits_adjustment_cloud_services) < 0
 THEN ABS(SUM(credits_adjustment_cloud_services))
 ELSE 0
 END AS excess_cloud_services_cost
FROM
 snowflake.account_usage.metering_daily_history
WHERE
 usage_date >= DATEADD('day', -30, CURRENT_DATE())
GROUP BY
 day
ORDER BY
 day DESC;

Serverless feature credits — Snowpipe, automatic clustering, materialized view maintenance, search optimisation — all consume credits outside your warehouse billing. They don’t show up in warehouse_metering_history. You need to check separately:

-- Snowpipe costs
SELECT
 pipe_name,
 SUM(credits_used) AS total_credits
FROM
 snowflake.account_usage.pipe_usage_history
WHERE
 start_time >= DATEADD('day', -30, CURRENT_TIMESTAMP())
GROUP BY
 pipe_name
ORDER BY
 total_credits DESC;

-- Automatic clustering costs
SELECT
 table_name,
 SUM(credits_used) AS total_credits
FROM
 snowflake.account_usage.automatic_clustering_history
WHERE
 start_time >= DATEADD('day', -30, CURRENT_TIMESTAMP())
GROUP BY
 table_name
ORDER BY
 total_credits DESC;

On the AWS side, the sneaky costs tend to be data transfer. Moving data between S3 regions, or from S3 out to the internet, adds up quietly. If your Snowflake account is in us-east-1 but your S3 landing zone is in ap-southeast-2, every byte of ingestion carries a cross-region transfer charge. Check your AWS Cost Explorer with the “Data Transfer” service filter — the numbers are often surprising.

You cannot reduce costs you can’t attribute. The simplest and most underused tool in Snowflake is the query tag — a piece of metadata you attach to every query that tells you what generated it.

If you’re using dbt, this takes about five minutes to set up. Create a macro:

-- macros/set_query_tag.sql
{% macro set_query_tag() %}
 {% set query_tag = {
 "dbt_model": model.name,
 "dbt_schema": model.schema,
 "dbt_materialized": model.config.materialized,
 "dbt_invocation_id": invocation_id,
 "environment": target.name
 } %}
 {% do run_query("ALTER SESSION SET QUERY_TAG = '{}'".format(query_tag | tojson)) %}
{% endmacro %}

Then add it as a pre-hook in your dbt_project.yml:

# dbt_project.yml
models:
 your_project:
 +pre-hook: "{{ set_query_tag() }}"

Now every query dbt runs is tagged with the model name, materialisation type, and environment. You can query cost by model:

SELECT
 PARSE_JSON(query_tag):dbt_model::STRING AS model_name,
 PARSE_JSON(query_tag):dbt_materialized::STRING AS materialization,
 COUNT(*) AS query_count,
 SUM(total_elapsed_time) / 1000 AS total_seconds,
 SUM(bytes_scanned) / POWER(1024, 4) AS tb_scanned
FROM
 snowflake.account_usage.query_history
WHERE
 query_tag IS NOT NULL
 AND TRY_PARSE_JSON(query_tag) IS NOT NULL
 AND start_time >= DATEADD('day', -7, CURRENT_TIMESTAMP())
GROUP BY
 model_name, materialization
ORDER BY
 tb_scanned DESC
LIMIT 20;

This is how you find the one dbt model that’s responsible for 40% of your bill. Without tags, it’s a guessing game.

For non-dbt workloads — Airflow tasks, Lambda functions, BI tools — set query tags at the session level in your connection configuration. In Python:

import snowflake.connector
import json

conn = snowflake.connector.connect(
 account='your_account',
 user='your_user',
 password='your_password',
 warehouse='transform_wh',
 session_parameters={
 'QUERY_TAG': json.dumps({
 'pipeline': 'customer_ingestion',
 'task': 'load_raw_customers',
 'environment': 'production'
 })
 }
)

The goal is simple: every query that runs on your platform should be attributable to a team, a pipeline, or a tool. Start with the big consumers and work outward.

Kill the zombies

Every data platform accumulates dead weight. Tables nobody queries. Pipelines that run faithfully every morning, transforming data that no dashboard, no analyst, and no model has touched in months.

These zombies cost you twice: once in compute (the pipeline that refreshes them) and once in storage (the data that sits there). Finding them is straightforward:

-- Tables with zero reads in the last 90 days
SELECT
 t.table_schema,
 t.table_name,
 t.row_count,
 t.bytes / POWER(1024, 3) AS size_gb,
 t.last_altered,
 MAX(ah.query_start_time) AS last_queried
FROM
 snowflake.account_usage.tables t

 LEFT JOIN snowflake.account_usage.access_history ah
 ON ah.base_objects_accessed LIKE '%' || t.table_name || '%'
 AND ah.query_start_time >= DATEADD('day', -90, CURRENT_TIMESTAMP())
WHERE
 t.table_schema NOT IN ('INFORMATION_SCHEMA')
 AND t.deleted IS NULL
 AND t.row_count > 0
GROUP BY
 t.table_schema, t.table_name, t.row_count, t.bytes, t.last_altered
HAVING
 last_queried IS NULL
ORDER BY
 size_gb DESC;

On the AWS side, check your S3 storage for orphaned data. Landing zones accumulate raw files that were loaded months ago and never cleaned up. A simple lifecycle policy handles this:

{
 "Rules": [
 {
 "ID": "Archive raw data after 90 days",
 "Status": "Enabled",
 "Filter": {
 "Prefix": "raw-landing/"
 },
 "Transitions": [
 {
 "Days": 90,
 "StorageClass": "GLACIER_INSTANT_RETRIEVAL"
 }
 ],
 "Expiration": {
 "Days": 365
 }
 }
 ]
}

The cultural shift matters as much as the tooling. Make it a quarterly habit: pull the zombie table list, review it with the team, and deprecate what nobody uses. If someone screams, you can always restore from Time Travel. But in my experience, nobody screams. The data was already dead — you’re just acknowledging it.

Cost anomalies will find you — or you can find them first

The scariest cost event isn’t the gradual creep. It’s the single bad query or misconfigured pipeline that doubles your weekly bill overnight. I’ve seen a single Snowflake query with a missing WHERE clause scan an entire 2TB table repeatedly inside a loop — burning through hundreds of credits in an hour.

On Snowflake, set up resource monitors as a basic guardrail:

-- Create a resource monitor with alerts and hard stop
CREATE OR REPLACE RESOURCE MONITOR monthly_budget
 WITH CREDIT_QUOTA = 5000 -- adjust to your monthly budget
 FREQUENCY = MONTHLY
 START_TIMESTAMP = IMMEDIATELY
 TRIGGERS
 ON 75 PERCENT DO NOTIFY -- email alert at 75%
 ON 90 PERCENT DO NOTIFY -- email alert at 90%
 ON 100 PERCENT DO SUSPEND; -- hard stop at 100%

-- Apply it to a warehouse
ALTER WAREHOUSE transform_wh
 SET RESOURCE_MONITOR = monthly_budget;

On AWS, enable Cost Anomaly Detection in the AWS Cost Management console. It uses ML to detect unusual spending patterns and sends alerts via SNS. For Snowflake-specific monitoring, a lightweight approach is a scheduled task that checks daily credit consumption against a rolling average:

-- Create a simple anomaly detection view
CREATE OR REPLACE VIEW monitoring.daily_cost_anomalies AS
WITH daily_usage AS (
 SELECT
 DATE_TRUNC('day', start_time) AS usage_date,
 warehouse_name,
 SUM(credits_used) AS daily_credits
 FROM
 snowflake.account_usage.warehouse_metering_history
 WHERE
 start_time >= DATEADD('day', -60, CURRENT_TIMESTAMP())
 GROUP BY
 usage_date, warehouse_name
),
averages AS
(
 SELECT
 warehouse_name,
 AVG(daily_credits) AS avg_daily_credits,
 STDDEV(daily_credits) AS stddev_daily_credits
 FROM
 daily_usage
 WHERE
 usage_date < CURRENT_DATE()
 GROUP BY
 warehouse_name
)
SELECT
 du.usage_date,
 du.warehouse_name,
 du.daily_credits,
 a.avg_daily_credits,
 ROUND((du.daily_credits - a.avg_daily_credits) / NULLIF(a.stddev_daily_credits, 0), 2) AS z_score
FROM
 daily_usage du

 JOIN averages a
 ON du.warehouse_name = a.warehouse_name
WHERE
 du.usage_date = CURRENT_DATE()
 AND (du.daily_credits - a.avg_daily_credits) / NULLIF(a.stddev_daily_credits, 0) > 2
ORDER BY
 z_score DESC;

A z-score above 2 means today’s spend is more than two standard deviations above the 60-day average. That’s worth investigating. Pipe this into a Teams or Slack alert via an AWS Lambda function and you’ve got same-day cost anomaly detection for the price of a few lines of SQL.

What your data model costs you

This one’s subtle but powerful. The way you model your data directly affects how much compute you burn on every query.

In columnar warehouses like Snowflake, wide denormalised tables query faster than star schemas with multiple JOINs — because each JOIN has overhead, and Snowflake is optimised for scanning columns from flat structures. But wide tables cost more to maintain because updating a single dimension attribute means rewriting many rows in the fact table.

The practical approach is a layered architecture:

Staging models (views or ephemeral): Minimal transformation, column selection, type casting. Cheap to run, no storage cost.
Intermediate models (tables or incremental): Business logic, deduplication, SCD handling. Materialised because they’re referenced by multiple downstream models.
Mart models (incremental or table): Wide, denormalised, optimised for consumer queries. These are what analysts and BI tools actually hit.

The cost trap is materialising too much too early. Every table materialisation means Snowflake stores and maintains that data. Every dbt run rebuilds it. If an intermediate model is only referenced by one downstream model, make it ephemeral or a view — let Snowflake inline it at query time.

Check your dbt DAG for this pattern: a staging model materialised as a table, referenced by a single intermediate model, which is also materialised as a table, referenced by a single mart. That’s three materialisations where one (the mart) would suffice. The staging and intermediate models can be views or ephemeral — the compute happens once when the mart builds, not three separate times.

Coming back to why this matters

I started this article with a story about not being able to explain my own platform’s costs. That wasn’t a technical failure. It was a leadership failure. I’d built something valuable and then neglected the part that kept it funded.

The data engineers I respect most aren’t the ones who build the most elegant pipelines. They’re the ones who can walk into a budget conversation and say: “Here’s what we spend. Here’s what we get for it. Here’s what I’d cut, and here’s what I’d invest more in.” That’s the kind of clarity that earns trust — and trust is what keeps data teams alive when the inevitable cost-cutting conversations happen.

You don’t need a FinOps certification or an expensive monitoring tool to get there. You need the queries in this article, a quarterly habit of reviewing them, and the willingness to kill the zombies nobody wants to admit are dead.

Start with the 15-minute audit. Tag your queries. Set up a resource monitor. Do those three things this week and you’ll know more about your platform’s economics than most data teams learn in a year.

The cheapest query is the one you never need to run. But the most valuable skill is knowing which queries are worth paying for.

Why Your Pipeline Finishes Later Every Month

Chris Hillman — Sat, 18 Apr 2026 09:00:00 +1100

Let me tell you about a graph that changed how I think about data engineering.

A junior engineer on my team — let’s call her Priya — had been tracking something nobody asked her to track. Every morning for two months, she’d noted the timestamp when our main analytics pipeline completed. She wasn’t trying to make a point. She was just curious, because the finance team kept mentioning their dashboards weren’t ready when they arrived at 8 AM anymore.

One afternoon she pulled me aside and showed me a scatter plot on her laptop. Pipeline completion time, plotted daily over several months. The trend was unmistakable: a slow, steady drift to the right. What used to finish at 5:47 AM was now finishing at 7:23 AM. And the slope wasn’t flattening.

“If this keeps going,” she said, “we’ll miss the 9 AM SLA in about six weeks.”

She was right. And nobody else on the team — including me — had noticed. We were watching for failures. Green DAGs, clean logs, no alerts. But the pipeline wasn’t failing. It was slowing. And slow is harder to see than broken, because slow doesn’t trigger an alert. Slow just quietly erodes trust until one day someone in finance builds their own spreadsheet and stops asking you for anything.

That’s the moment I understood that pipeline health isn’t about pass/fail. It’s about trajectory. A pipeline that runs successfully but takes 5% longer every month is a ticking clock. And the people who notice first aren’t the engineers watching the DAG — they’re the consumers waiting for their data.

I’m telling you this because pipeline optimisation sounds like a performance engineering problem, and it is. But underneath the technical work, it’s really about a commitment: the commitment to deliver data when you said you would, every single day. That’s what builds trust between data teams and the rest of the organisation. Not fancy architectures. Not real-time everything. Just showing up on time, reliably.

This article is about diagnosing why your pipelines get slower, identifying the bottlenecks that actually matter, and fixing the patterns that cause data to arrive later than it should. Everything is grounded in Snowflake, AWS, Airflow, and dbt — with specific patterns you can apply immediately.

Measure the pipeline in stages, not as a single number

When a pipeline is slow, the instinct is to look at the longest-running task. That’s often the wrong place to start.

A pipeline is a chain of stages: extract, transform, load, test. The total runtime is the sum of all stages on the critical path — the longest chain of dependent tasks. But the bottleneck might not be the longest task. It might be a 30-second task that blocks five parallel branches from starting.

Before you optimise anything, instrument your pipeline to measure each stage independently. In Airflow, the task instance metadata already captures this.

Two numbers matter here: duration_seconds (how long the task actually ran) and queue_wait_seconds (how long it waited before it could run). If queue wait is high, your problem isn’t the task — it’s resource contention. Too many tasks competing for too few Airflow worker slots, or too many queries competing for the same Snowflake warehouse.

For dbt runs specifically, the run_results.json that dbt generates after every invocation is a goldmine:

import json

with open('target/run_results.json') as f:
 results = json.load(f)

# Find your slowest models
models = sorted(
 [r for r in results['results'] if r['status'] == 'success'],
 key=lambda x: x['execution_time'],
 reverse=True
)

for m in models[:10]:
 print(f"{m['unique_id']:60s} {m['execution_time']:8.1f}s")

Run that after your next dbt build. You’ll immediately see which models dominate your pipeline runtime. In my experience, 3–5 models account for 60–80% of total execution time. Those are the only models worth optimising.

Your DAG probably has dependencies it doesn’t need

This is the most common structural problem in data pipelines, and it comes in two forms. One is obvious once you know to look for it. The other is sneaky, because it starts as a completely reasonable engineering decision.

The first form: phantom dependencies.

An engineer builds Model C and declares it depends on Model B — because Model B produces a table that Model C reads. Fair enough. But six months later, someone refactors Model C and it no longer reads from Model B. The ref() gets removed from the SQL. But the dependency in the Airflow DAG? That stays. Or in dbt, someone adds a ref() to a model they don’t actually need, just to “make sure it runs first.”

The result is a DAG with phantom dependencies — tasks that wait for other tasks to complete even though they don’t use those tasks’ outputs. Every phantom dependency adds serial wait time to your pipeline.

The second form: dependency monsters.

This one is trickier, because it starts with good intentions.

The customer team needs three new enrichment attributes on dim_customers — regional segment codes sourced from a CRM export, tenure tier derived from a subscription history table, and a propensity score from a data science model. All reasonable requests. Each one approved and added without much ceremony.

But dim_customers is upstream of 40+ models across your DAG. Adding those three attributes means pulling in the CRM extract, joining the subscription history table — which has its own upstream dependencies — and waiting on the propensity score model to complete before any of those 40 downstream models can start. Your pipeline used to finish at 9 AM. Eighteen months and a dozen enrichment requests later, it finishes at 3 PM.

Each addition was individually justified. Nobody modelled what they’d cost collectively. That’s how you build a dependency monster.

The fix isn’t to refuse enrichment requests — it’s to stop embedding enrichment into your core spine model. Keep dim_customers lean: identifiers, names, status, the attributes that nearly every consumer genuinely needs. Build a separate dim_customers_extended model that joins in the expensive enrichment for the consumers who actually need it. Most of your downstream models will never touch the propensity score. There’s no reason to make them wait for it.

-- dim_customers.sql: lean spine — runs fast, unblocks the DAG
SELECT
 customer_id,
 customer_name,
 email,
 status,
 created_at
FROM
 {{ ref('stg_customers') }}

-- dim_customers_extended.sql: enrichment for consumers who need it
-- runs independently, doesn't block the main pipeline
SELECT
 c.customer_id,
 c.customer_name,
 c.status,
 crm.regional_segment,
 sub.tenure_tier,
 ps.propensity_score
FROM
 {{ ref('dim_customers') }} c

 LEFT JOIN {{ ref('stg_crm_segments') }} crm
 on crm.customer_id = c.customer_id

 LEFT JOIN {{ ref('int_subscription_tenure') }} sub
 on sub.customer_id = c.customer_id

 LEFT JOIN {{ ref('ml_propensity_scores') }} ps
 on ps.customer_id = c.customer_id

Now dim_customers_extended and its expensive upstream dependencies sit on their own branch of the DAG. Consumers who need the enrichment reference the extended model. The rest of the pipeline is unblocked.

A useful governance rule of thumb before adding a field to a core model: if fewer than half your consumers will ever query that attribute, it probably doesn’t belong there. The team who needs it should own the enrichment themselves, as a downstream model they maintain.

Auditing for both problems

In dbt, the dbt_project_evaluator package surfaces structural issues systematically. Install it:

# packages.yml
packages:
 - package: dbt-labs/dbt_project_evaluator
 version: ">=0.8.0 <1.0.0"

Then run:

dbt build --select package:dbt_project_evaluator

It will flag models with fan-out issues (one model referenced by too many downstream models), unnecessary dependencies, and models that could be parallelised but aren’t. It won’t catch dependency monsters directly — those require a human asking “does every consumer of this model actually use every attribute in it?” — but it’s a good starting point for the structural audit.

For Airflow DAGs, the visual DAG view tells you a lot at a glance. A healthy DAG looks like a tree with wide parallel branches. An unhealthy DAG looks like a chain — or worse, a funnel where everything converges through a single overloaded task before it can branch out again. That funnel shape is the visual signature of a dependency monster: many expensive upstreams flowing into one hub model, with dozens of downstream models queued behind it.

If your DAG looks like a chain, ask this question for every dependency edge: does downstream task B actually read data produced by upstream task A? If the answer is no, remove the dependency. Let them run in parallel.

I once audited a DAG with 47 tasks running in a strict serial chain. After removing phantom dependencies and restructuring, 31 of those tasks could run in parallel. The pipeline went from 2 hours 15 minutes to 38 minutes. Same tasks, same compute, same data. Just fewer unnecessary wait states.

The critical path is the only thing worth optimising

Here’s a concept from project management that applies directly to pipeline engineering: the critical path.

The critical path is the longest chain of dependent tasks through your DAG. It determines your pipeline’s minimum possible runtime. Every other path through the DAG has “float” — slack time where tasks can be delayed without affecting the overall completion time.

This means: optimising a task that isn’t on the critical path has zero impact on your pipeline’s end-to-end runtime. You could make a non-critical task 10x faster and your pipeline would finish at exactly the same time.

Finding the critical path requires knowing each task’s duration and dependency structure. For a dbt project, you can approximate it:

-- Using dbt's run results stored in Snowflake (if you log them)
-- Or query your orchestrator's task instance table
WITH task_durations AS (
 SELECT
 task_id AS model_name,
 AVG(EXTRACT(EPOCH FROM (end_date - start_date))) AS avg_duration
 FROM task_instance
 WHERE dag_id = 'dbt_daily'
 AND state = 'success'
 AND execution_date >= CURRENT_DATE - INTERVAL '7 days'
 GROUP BY task_id
)
SELECT
 model_name,
 ROUND(avg_duration, 1) AS avg_seconds,
 ROUND(avg_duration / 60, 1) AS avg_minutes
FROM task_durations
ORDER BY avg_duration DESC
LIMIT 20;

The longest tasks are candidates for the critical path, but only if they’re on the dependency chain that determines total runtime. A 20-minute model that runs in parallel with a 45-minute model isn’t the bottleneck — the 45-minute model is.

Once you’ve identified the critical path, you have three options for shortening it: make the slow tasks faster (query optimisation, incremental models), reduce the number of tasks on the path (remove unnecessary dependencies), or parallelise sequential tasks (split a monolithic model into independent pieces).

Shifting right: diagnose it before it breaks your SLA

Remember Priya’s scatter plot? That pattern — pipeline completion drifting later and later — has a name: shifting right. And it has predictable causes.

Track it with a simple query against your orchestrator’s metadata:

-- Airflow: track pipeline completion time drift
SELECT
 execution_date::DATE AS run_date,
 MAX(end_date)::TIME AS completion_time,
 EXTRACT(EPOCH FROM (MAX(end_date) - MIN(start_date))) / 60 AS total_runtime_minutes
FROM
 task_instance
WHERE
 dag_id = 'daily_analytics'
 AND state = 'success'
 AND execution_date >= CURRENT_DATE - INTERVAL '60 days'
GROUP BY
 run_date
ORDER BY
 run_date;

Plot completion_time over run_date. If it trends upward, you’re shifting right. The four root causes, in order of how often I see them:

1. Data volume growth. Your table had 10 million rows when you built the model. Now it has 200 million. That JOIN that took 8 seconds now takes 3 minutes. This is the most common cause and the easiest to fix — switch to incremental materialisation and the growth stops mattering.

2. Resource contention. You’ve added more pipelines and they all run in the same window, competing for the same Snowflake warehouse. In Snowflake, you can see this directly:

-- Find warehouse queueing (queries waiting for compute)
SELECT
 warehouse_name,
 DATE_TRUNC('hour', start_time) AS hour,
 COUNT(*) AS total_queries,
 SUM(CASE WHEN queued_overload_time > 0 THEN 1 ELSE 0 END) AS queued_queries,
 AVG(queued_overload_time) / 1000 AS avg_queue_seconds
FROM
 snowflake.account_usage.query_history
WHERE
 start_time >= DATEADD('day', -7, CURRENT_TIMESTAMP())
 AND warehouse_name = 'TRANSFORM_WH'
GROUP BY
 warehouse_name, hour
HAVING
 queued_queries > 0
ORDER BY
 hour;

If queued_queries is non-zero during your pipeline window, queries are waiting for warehouse compute. Either scale up the warehouse during that window, split workloads across dedicated warehouses, or stagger pipeline start times to reduce concurrency.

3. Dependency chain lengthening. Someone added a new intermediate model between your staging and mart layers. That model takes 4 minutes. But because it’s on the critical path, the entire pipeline now finishes 4 minutes later. This is death by a thousand cuts — each addition is small, but they accumulate.

4. Upstream source delays. Your pipeline starts at 4 AM because the source system’s extract used to land in S3 by 3:45 AM. But the source system has grown too, and now the extract doesn’t land until 4:30 AM. Your pipeline sensors wait, and everything shifts right.

For upstream delays in an AWS environment, replace time-based scheduling with event-driven triggers. Instead of scheduling your Airflow DAG at 4 AM and hoping the data is there, trigger it when the data actually arrives:

# Airflow DAG: trigger on S3 file landing using AWS sensor
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
from airflow import DAG
from datetime import datetime

with DAG(
 'daily_analytics',
 start_date=datetime(2026, 1, 1),
 schedule_interval=None, # triggered externally, not on a schedule
 catchup=False,
) as dag:

 wait_for_source = S3KeySensor(
 task_id='wait_for_orders_extract',
 bucket_name='your-data-lake',
 bucket_key='raw/orders/dt={{ ds }}/orders.parquet',
 aws_conn_id='aws_default',
 mode='reschedule', # frees the worker slot while waiting
 poke_interval=300, # check every 5 minutes
 timeout=7200, # fail after 2 hours if file never arrives
 )

The mode='reschedule' is critical. Without it, the sensor occupies a worker slot for the entire time it’s waiting. With reschedule, it checks, releases the slot, and checks again later. This prevents the classic deadlock where all your worker slots are consumed by sensors and no actual work can run.

Even better: use S3 event notifications to trigger a Lambda function that kicks off the DAG via Airflow’s REST API. Zero polling, zero wasted slots:

# Lambda function triggered by S3 PutObject event
import boto3
import requests
import os

def handler(event, context):
 bucket = event['Records'][0]['s3']['bucket']['name']
 key = event['Records'][0]['s3']['object']['key']

 # Trigger Airflow DAG via REST API
 airflow_url = os.environ['AIRFLOW_API_URL']
 response = requests.post(
 f"{airflow_url}/api/v1/dags/daily_analytics/dagRuns",
 json={
 "conf": {"source_bucket": bucket, "source_key": key}
 },
 auth=(os.environ['AIRFLOW_USER'], os.environ['AIRFLOW_PASSWORD']),
 headers={"Content-Type": "application/json"}
 )

 return {"statusCode": response.status_code}

This eliminates the “safety margin” scheduling pattern entirely. Your pipeline runs as soon as the data is available — not 30 minutes after you hope it will be available.

Partition and cluster to eliminate full table scans

The single most effective performance optimisation in Snowflake is making sure your queries only read the data they need. Snowflake’s micro-partition pruning does this automatically — but only if your data is physically organised in a way that aligns with your query patterns.

Clustering keys tell Snowflake how to organise data within micro-partitions. If you consistently filter on order_date, clustering on that column means queries with a WHERE order_date = '2026-03-01' clause scan a tiny fraction of the table instead of the whole thing.

-- Add clustering to a large fact table
ALTER TABLE analytics.fct_orders
 CLUSTER BY (order_date, customer_segment);

Choose clustering keys based on how the table is actually queried, not how it’s loaded. The best candidates are columns that appear in WHERE clauses, JOIN conditions, and range filters. Limit yourself to 2–3 keys — more than that and Snowflake can’t maintain effective clustering.

Check whether your existing tables benefit from clustering:

-- Check clustering depth (lower is better, 0 is perfect)
SELECT
 table_name,
 clustering_key,
 total_constant_partition_count,
 total_partition_count,
 average_overlaps,
 average_depth
FROM
 snowflake.account_usage.table_storage_metrics
WHERE
 table_catalog = 'YOUR_DATABASE'
 AND clustering_key IS NOT NULL
 AND active_bytes > 0
ORDER BY
 average_depth DESC;

An average_depth above 5 means your clustering isn’t effective — queries are still scanning more partitions than they should. Either the clustering key doesn’t match query patterns, or the table has had so many small writes that the clustering has degraded. Running ALTER TABLE ... RECLUSTER manually or relying on automatic clustering (which costs credits) addresses the latter.

On the AWS side, if you’re using S3 as a data lake with Parquet or Iceberg, the equivalent is partition layout. Partition your S3 data by the columns you filter most:

When Snowflake external tables or AWS Athena query this structure with a date filter, they skip entire directories. A query for March 15th reads two files instead of scanning the entire events/ prefix. The savings compound with data volume — at a billion rows, proper partitioning can reduce query times from minutes to seconds.

Make every task idempotent or debugging becomes a nightmare

Here’s a pattern I see constantly: a pipeline fails midway through, the engineer reruns it, and now there are duplicate rows in the target table. Because the first run inserted half the data before failing, and the rerun inserted all the data — including the half that already existed.

Idempotency means a task produces the same result whether it runs once or ten times. This isn’t just a nice-to-have — it’s the foundation that makes everything else in pipeline engineering possible. Without it, you can’t safely retry. You can’t backfill. You can’t debug.

In dbt, incremental models with a unique_key are idempotent by default — the MERGE statement handles duplicates:

{{
 config(
 materialized='incremental',
 unique_key='event_id',
 incremental_strategy='merge'
 )
}}

SELECT
 event_id,
 user_id,
 event_type,
 event_timestamp,
 properties
FROM
 {{ ref('stg_events') }}

{% if is_incremental() %}
WHERE
 event_timestamp >= (
 SELECT DATEADD('day', -3, MAX(event_timestamp))
 FROM {{ this }}
)
{% endif %}

For non-dbt loads — say, a Lambda function loading data from an API into Snowflake — use MERGE instead of INSERT:

MERGE INTO raw.api_customers AS target
USING (
 SELECT $1:id::STRING AS customer_id,
 $1:name::STRING AS customer_name,
 $1:email::STRING AS email,
 $1:updated_at::TIMESTAMP_NTZ AS updated_at
 FROM @raw.s3_stage/customers/
 (FILE_FORMAT => 'json_format')
) AS source
ON target.customer_id = source.customer_id

 WHEN MATCHED AND source.updated_at > target.updated_at THEN
 UPDATE SET
 customer_name = source.customer_name,
 email = source.email,
 updated_at = source.updated_at

 WHEN NOT MATCHED THEN
 INSERT (customer_id, customer_name, email, updated_at)
 VALUES (source.customer_id, source.customer_name, source.email, source.updated_at);

The WHEN MATCHED AND source.updated_at > target.updated_at condition prevents overwriting newer data with older data during reruns. This matters when you’re replaying historical loads or running overlapping backfills.

Most “we need real-time” requests actually need “we need faster batch”

Before you reach for Kafka, Kinesis, or any streaming infrastructure, have this conversation with your stakeholders: “When you say real-time, what do you actually mean?”

In my experience, the answers fall into three categories:

“I want data from today, not yesterday.” That’s daily batch with a morning refresh. You already have this. Maybe you need to move the refresh earlier.

“I want data that’s at most an hour old.” That’s micro-batch — running your existing pipeline every 15–60 minutes instead of once a day. No new infrastructure required. Your existing SQL, dbt, and Airflow tools work unchanged at shorter intervals.

“I need to see events within seconds of them happening.” This is actual real-time. And it’s genuinely rare. Fraud detection, safety monitoring, real-time bidding — these need streaming. Your internal sales dashboard almost certainly does not.

For the micro-batch pattern in Airflow, it’s a scheduling change:

with DAG(
 'micro_batch_analytics',
 schedule_interval='*/15 * * * *', # every 15 minutes
 catchup=False,
 max_active_runs=1, # prevent overlap
) as dag:
 # ...tasks here

The max_active_runs=1 is important. Without it, if a run takes longer than 15 minutes, the next run starts before the previous one finishes. They compete for the same warehouse, both run slower, and you’ve created the resource contention problem from the shifting-right section.

In Snowflake, pair this with Snowpipe for continuous S3 ingestion:

-- Create a pipe for continuous loading from S3
CREATE OR REPLACE PIPE raw.orders_pipe
 AUTO_INGEST = TRUE
 AS
 COPY INTO raw.orders_stream
 FROM @raw.s3_orders_stage
 FILE_FORMAT = (TYPE = 'PARQUET')
 MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;

Snowpipe loads files within minutes of them landing in S3. Your micro-batch dbt pipeline then transforms whatever arrived since the last run. The combination delivers data freshness measured in minutes — not seconds, but close enough for the vast majority of business use cases — without any streaming infrastructure.

Measure time-to-consumer, not just pipeline runtime

Pipeline runtime tells you how long your DAG takes. Time-to-consumer tells you how long a business event takes to reach a human decision-maker. These are different numbers, and the second one is what your stakeholders actually care about.

Time-to-consumer decomposes into stages:

Source extraction delay: Time between the event occurring and the data landing in your lake
Pipeline processing time: Your DAG runtime (the part you control most directly)
Warehouse serving time: Query execution time when a dashboard or report reads the data
Cache/refresh delay: How often the BI tool refreshes its cache

Track it by stamping timestamps at each handoff:

-- Build a freshness tracking model in dbt
-- mart_data_freshness.sql
SELECT
 'fct_orders' AS model_name,
 MAX(order_timestamp) AS latest_source_event,
 MAX(_loaded_at) AS latest_load_time,
 CURRENT_TIMESTAMP() AS measured_at,
 TIMESTAMPDIFF(
 'minute',
 MAX(order_timestamp),
 CURRENT_TIMESTAMP()
 ) AS minutes_since_latest_event,
 TIMESTAMPDIFF(
 'minute',
 MAX(_loaded_at),
 CURRENT_TIMESTAMP()
 ) AS minutes_since_latest_load
FROM
 {{ ref('fct_orders') }}

Schedule this model to run after every pipeline completion. Over time, you’ll see whether minutes_since_latest_event is stable, improving, or drifting. If it’s drifting, you can pinpoint which stage is responsible by comparing the event timestamp, load timestamp, and measurement timestamp.

In dbt, you can also use source freshness tests to alert when upstream data is stale:

# models/staging/_sources.yml
sources:
 - name: raw
 database: raw_db
 schema: public
 loaded_at_field: _loaded_at
 freshness:
 warn_after: { count: 2, period: hour }
 error_after: { count: 6, period: hour }
 tables:
 - name: orders
 - name: customers
 - name: events

Running dbt source freshness checks whether the upstream data is as fresh as you expect. If raw.orders hasn’t received new rows in 6 hours, the test fails — and that’s a signal that the source system’s extract is delayed, not that your pipeline is broken.

The view-on-view problem: the compound interest of bad performance

This is the architectural anti-pattern I see most often in Snowflake environments, and it’s one of those problems that starts small and compounds quietly.

A view in Snowflake doesn’t store any data — it re-executes its SQL every time it’s queried. That’s fine for a simple view. But when View C references View B, which references View A, which scans a large table — every query against View C triggers the entire chain from scratch.

I’ve seen environments where analysts querying a “simple” dashboard view were unknowingly triggering five layers of nested views, each with its own JOINs and aggregations. The query took 4 minutes. After materialising the two most expensive intermediate layers as tables (refreshed daily), the same query took 3 seconds.

The fix is to audit your materialisation strategy:

-- Find views that reference other views (potential nesting)
SELECT
 referencing_object_name AS downstream_view,
 referenced_object_name AS upstream_object,
 referenced_object_type
FROM
 snowflake.account_usage.object_dependencies
WHERE
 referencing_object_type = 'VIEW'
 AND referenced_object_type = 'VIEW'
ORDER BY
 downstream_view;

If this query returns results, you have view-on-view nesting. Not all of it is bad — a simple renaming view on top of a complex view is fine. But if you find three or more layers, or if any of the intermediate views involve heavy JOINs or aggregations, materialise the expensive layers as tables.

In dbt terms, the decision framework is:

Ephemeral: Simple CTEs, column renaming, type casting. Zero cost.
View: Light transformations queried infrequently by a few consumers.
Table: Complex transformations queried frequently. Rebuilt every run.
Incremental: Large tables where only a fraction of rows change. The right choice for most fact tables.

If a model takes more than 30 seconds to build and is queried more than once between builds, it should be a table or incremental — not a view.

Coming back to Priya’s scatter plot

Six weeks after Priya showed me that graph, we’d fixed the three root causes: an intermediate model that had grown from 5 million to 180 million rows without being switched to incremental, a phantom dependency chain that added 12 minutes of serial wait time, and a warehouse that was being shared between transformation and BI queries during the morning refresh window.

Pipeline completion moved from 7:23 AM back to 5:15 AM. The finance team got their dashboards before their morning coffee. And Priya? She built a monitoring dashboard that tracked completion time drift automatically — the thing I should have built from the start.

Here’s what I learned from that experience, and what I hope you take from this article: pipeline performance isn’t a problem you solve once. It’s a trajectory you manage. Data grows. Teams add models. Source systems change. The pipeline that’s fast today will be slow in six months unless someone is watching the trend line.

The engineers who build reliable data platforms aren’t the ones who build the fastest pipeline on day one. They’re the ones who notice when completion time drifts by 3 minutes per week — and fix it before anyone else has to ask.

Track your completion times. Audit your dependencies quarterly. Profile your critical path after every significant change. And teach your team that a green DAG doesn’t mean a healthy pipeline. Sometimes green just means it hasn’t failed yet.

Stop Building Salesforce Integrations From Scratch

Chris Hillman — Sat, 04 Apr 2026 09:00:00 +1100

Let me tell you about Marcus.

Marcus was on a team I led a few years back. Sharp, motivated, the kind of engineer who actually read documentation before writing code. When the business asked us to get Salesforce data into our warehouse, Marcus volunteered. He’d done API work before. He figured a few weeks, tops.

He scoped it carefully. Built a Python service that authenticated via OAuth, pulled Account, Contact, and Opportunity objects through the Bulk API, flattened the nested JSON into relational tables, handled pagination, managed rate limits. Wrote solid tests. Documented everything. The kind of work you’d point to in a code review and say this is how it’s done.

It worked beautifully. For about three months.

Then our Salesforce admin added a custom field to the Opportunity object — Renewal_Likelihood__c — and nobody told the data team. The pipeline didn’t fail. That’s the insidious part. It kept running, kept landing data. It just quietly dropped the new field on the floor.

When Marcus tracked it down, he added the field. Then he realised there were eleven other custom fields that had been added since go-live that the pipeline was silently ignoring. And the compound address fields — BillingAddress, ShippingAddress — had never worked with the Bulk API in the first place. He’d been extracting the component parts (BillingStreet, BillingCity) as a workaround, but the workaround had a bug that truncated postal codes for international addresses.

Marcus spent three weeks patching all of it. Three weeks of a talented engineer doing work that a managed connector handles automatically.

When I watched a good engineer spend the best part of a month on a problem that shouldn’t exist. Marcus wasn’t learning anything. He wasn’t growing. He was hand-coding schema detection logic for the fourth time because Salesforce ships three major releases a year and our sales ops team adds custom fields like they’re decorating a Christmas tree.

I’m telling you this because Snowflake OpenFlow is one of those things that sounds like “just another integration tool” until you’ve lived through the alternative. What I want to show you today is how to set it up with Salesforce — step by step, with enough detail that you could do it tomorrow — and more importantly, why the schema evolution feature alone justifies the switch from custom code.

If you’ve ever maintained a custom Salesforce integration, you already know why this matters.

The Problem Nobody Warns You About

Here’s what the “just build it yourself” crowd doesn’t tell you: the Salesforce API isn’t one API. It’s an ecosystem of overlapping interfaces, each with its own quirks, limits, and failure modes.

The REST API handles small, synchronous requests well — but try pulling a million Opportunity records through it and you’ll burn through your rate limits before lunch. The Bulk API 2.0 is designed for volume — it can handle up to 150 million records in a 24-hour window — but it doesn’t support compound fields like BillingAddress or MailingAddress. Those silently return nothing. Not an error. Nothing.

Then there’s SOQL, Salesforce’s proprietary query language, which looks enough like SQL to trick you into thinking you understand it. Until you hit the 2,000-record offset limit, or try to resolve a polymorphic relationship where a WhoId field on a Task could reference either a Contact or a Lead depending on the record. That’s not a data modelling problem — that’s a “your pipeline logic needs to branch based on the referenced object type” problem.

And rate limits. Enterprise Edition gives you 100,000 API calls per 24 hours, plus 1,000 per user licence. Sounds generous until you remember that your BI tool, marketing automation platform, customer support system, and your data pipeline are all drinking from the same well.

But schema evolution is the silent killer.

Every Salesforce org is different because every business is different. Your Salesforce admin creates custom objects and fields to match how your company actually works. That’s the point — it’s a customisation platform. But every custom field added to Salesforce is a potential break point for any integration that doesn’t handle schema changes automatically.

And those changes happen constantly. Research suggests schemas in enterprise SaaS tools change roughly every three days on average. In Salesforce specifically, between admin-driven customisation and Salesforce’s own three-times-a-year release cycle (Spring, Summer, Winter), the schema you built against last quarter may not be the schema you’re ingesting from today.

This is the trap. Building the initial integration isn’t hard. Any competent engineer can get Salesforce data into a warehouse. Keeping it working — across schema changes, API version deprecations, authentication token rotations, and rate limit adjustments — is where it eats your life.

I’ve seen teams eliminate two full days of monthly engineering maintenance by migrating off custom connectors. Two days, every month, spent on work that a managed tool does automatically. That’s not engineering — it’s janitorial work wearing an engineering costume.

What Is Snowflake OpenFlow?

OpenFlow is Snowflake’s native, fully managed data integration service. It’s not a partnership or a marketplace app — it’s a first-party feature built into the platform. If you’re already running Snowflake, OpenFlow runs inside your existing environment.

The backstory matters. In November 2024, Snowflake acquired a company called Datavolo for roughly $110 million. Datavolo was founded by Joe Witt, who co-created Apache NiFi back when it was an NSA project. That NiFi heritage is the foundation of OpenFlow — curated, versioned connector definitions powered by NiFi’s flow engine, wrapped in a managed Snowflake experience.

Architecturally, it splits into two pieces. The control plane lives inside Snowflake and gives you pipeline management, scheduling, and monitoring through the Snowsight UI. The data plane handles the actual work and can run in two modes: Snowflake Deployments (running on Snowpark Container Services within Snowflake’s own infrastructure) or BYOC (Bring Your Own Cloud, running as a Kubernetes cluster in your VPC). For most teams starting out, the SPCS option is simpler — it went generally available in November 2025 on AWS and Azure.

Data flows into Snowflake primarily through Snowpipe Streaming for the initial load, with merge queries handling incremental updates. The result is structured tables in your Snowflake database — not staged JSON files, not semi-structured VARIANT columns. Actual, queryable tables with proper column types.

The connector catalogue currently covers about 20 curated sources: databases (PostgreSQL, MySQL, SQL Server, Oracle), SaaS platforms (Salesforce, Workday, Jira, LinkedIn Ads, Meta Ads, Google Ads), cloud storage (Google Drive, SharePoint, Box), and streaming sources (Kafka, Kinesis). It’s not Fivetran’s 500+ connector library, and nobody’s pretending it is. But the sources it does cover are the ones that account for the vast majority of enterprise data movement.

One important caveat up front: the Salesforce Bulk API connector is still in preview as of March 2026. It works. Preview in Snowflake terms means the feature is functional but not yet covered by production support SLAs. Keep that in your mental model as we walk through the setup.

Setting Up the Salesforce Connector: A Walkthrough

This section is the step-by-step. I’m going to walk through the full setup — Salesforce configuration, Snowflake preparation, and connector deployment — with enough detail that you could follow along on your own org.

Phase 1: Salesforce Configuration

The connector uses JWT Bearer Flow for authentication, which means you need an RSA key pair and a Connected App configured in Salesforce.

Generate your RSA key pair. Open a terminal and run:

# Generate 2048-bit RSA private key
openssl genrsa -out salesforce_private_key.pem 2048

# Generate the corresponding public certificate (valid 1 year)
openssl req -new -x509 -key salesforce_private_key.pem \
 -out salesforce_certificate.pem -days 365

Keep that private key safe. You’ll need it when configuring the connector in Snowflake.

Create the External Client App in Salesforce. Navigate to Setup → Apps → App Manager → New External Client App. Enable OAuth and add two scopes: api and refresh_token (sometimes labelled offline_access). Enable the JWT Bearer Flow toggle and upload your salesforce_certificate.pem public certificate. Save the app and record the Consumer Key and Consumer Secret that Salesforce generates.

Configure access. On the app’s Manage settings, set the Permitted Users policy to “Admin approved users are pre-authorized.” Then assign the appropriate user profiles or permission sets. The user account the connector will authenticate as needs read access to every object you intend to sync.

Phase 2: Snowflake Preparation

Before deploying the connector, you need a database, schema, role, and warehouse ready for it.

-- Create a dedicated database for Salesforce data
CREATE DATABASE IF NOT EXISTS SALESFORCE_RAW;
CREATE SCHEMA IF NOT EXISTS SALESFORCE_RAW.OPENFLOW;

-- Create a connector-specific role
CREATE ROLE IF NOT EXISTS OPENFLOW_SF_CONNECTOR;

-- Grant the role permissions to write to the destination
GRANT USAGE ON DATABASE SALESFORCE_RAW TO ROLE OPENFLOW_SF_CONNECTOR;
GRANT USAGE ON SCHEMA SALESFORCE_RAW.OPENFLOW TO ROLE OPENFLOW_SF_CONNECTOR;
GRANT CREATE TABLE ON SCHEMA SALESFORCE_RAW.OPENFLOW TO ROLE OPENFLOW_SF_CONNECTOR;

-- Create a warehouse for merge operations
CREATE WAREHOUSE IF NOT EXISTS OPENFLOW_WH
 WAREHOUSE_SIZE = 'SMALL'
 AUTO_SUSPEND = 60
 AUTO_RESUME = TRUE;

GRANT USAGE ON WAREHOUSE OPENFLOW_WH TO ROLE OPENFLOW_SF_CONNECTOR;
GRANT OPERATE ON WAREHOUSE OPENFLOW_WH TO ROLE OPENFLOW_SF_CONNECTOR;

For SPCS deployments, you also need a network rule allowing the connector to reach your Salesforce instance. OpenFlow runs inside Snowflake’s container environment, and by default it can’t reach external endpoints without explicit permission:

-- Allow egress to your Salesforce instance
CREATE NETWORK RULE IF NOT EXISTS OPENFLOW_SF_EGRESS
 TYPE = HOST_PORT
 MODE = EGRESS
 VALUE_LIST = ('yourinstance.salesforce.com:443');

-- Wrap it in an external access integration
CREATE EXTERNAL ACCESS INTEGRATION IF NOT EXISTS OPENFLOW_SF_ACCESS
 ALLOWED_NETWORK_RULES = (OPENFLOW_SF_EGRESS)
 ENABLED = TRUE;

Replace yourinstance.salesforce.com with your actual Salesforce instance hostname. If you’re not sure what that is, log in to Salesforce and look at the URL bar — it’s the domain before .salesforce.com.

Phase 3: Deploying the Connector

Now the fun part. Navigate to Snowsight → Data → Openflow in the left sidebar. Find the “Openflow connector for Salesforce Bulk API” tile and add it to your runtime.

This opens the NiFi canvas — a visual flow editor where the connector’s pre-built process groups appear. Right-click the Salesforce process group and select Configure Parameters. Here’s where you’ll enter:

Salesforce Instance URL — e.g., https://yourinstance.salesforce.com
Consumer Key — from the Connected App you created
Consumer Secret — same source
Private Key — the contents of salesforce_private_key.pem (the full PEM text, headers included)
Username — the Salesforce user the connector authenticates as
Filter — the objects to sync, by API name

The Filter parameter is where you define what to pull. List standard objects by name (Account, Contact, Opportunity, Lead, Task, Event) and custom objects using their API name with the __c suffix:

There’s also a Special Objects Filter for objects that require different handling (like User or RecordType).

Once the parameters are configured, enable all controller services in the process group, then start the connector. The initial sync will pull a full snapshot of every object in your filter list. Subsequent runs perform incremental syncs using Salesforce’s SystemModstamp field to identify changed records.

What You Get

Once the connector runs, you’ll find one table per Salesforce object in SALESFORCE_RAW.OPENFLOW. The tables are flattened and structured — every Salesforce field becomes a column with an appropriate Snowflake data type. No VARIANT columns to parse. No nested JSON to unpack.

An Account table will have columns like ID, NAME, INDUSTRY, ANNUALREVENUE, BILLINGSTREET, BILLINGCITY, BILLINGSTATE, BILLINGPOSTALCODE, CREATEDDATE, LASTMODIFIEDDATE, and so on. Custom fields like Renewal_Likelihood__c appear as RENEWAL_LIKELIHOOD__C.

Two metadata columns are added automatically: ISDELETED (tracking Salesforce soft deletes) and the system timestamp fields. This is important — hard deletes in Salesforce are not reflected. If a record is permanently deleted in Salesforce, it won’t disappear from your Snowflake table. You’ll need to handle that in your downstream dbt models if it matters for your use case.

A Note on Multiple Sync Frequencies

Not every Salesforce object changes at the same pace. Your Opportunity table might need hourly syncs while Account is fine with daily. OpenFlow handles this by letting you deploy multiple connector instances in the same runtime — each with its own filter list and schedule — at no additional compute cost beyond the shared runtime.

Schema Evolution: Where OpenFlow Earns Its Keep

This is the feature that matters most, and the one that would have saved Marcus three weeks.

When someone adds a new custom field to a Salesforce object — say the sales ops team creates Deal_Confidence_Score__c on the Opportunity object — the OpenFlow connector automatically detects the new field and adds a corresponding column to the Snowflake destination table on the next sync. No configuration change. No redeployment. No Teams message at 7am asking the data team to “add that new field we made yesterday.”

The column appears, data starts flowing into it, and your dbt models can reference it whenever you’re ready.

For field renames, OpenFlow takes a conservative approach: the old column stays in place (with stale data from the last sync before the rename), and a new column is created under the new field name. This means you’ll have both Old_Field_Name__c and New_Field_Name__c in your table for a period. That’s actually useful for audit purposes — you can see exactly when the rename happened by comparing timestamps — but it does mean your downstream queries need updating.

Here’s the honest gap: new custom objects are not auto-discovered. If someone creates an entirely new custom object in Salesforce, you have to manually add it to the connector’s Filter parameter. Schema evolution handles field-level changes automatically, but object-level discovery is still a manual step.

And field type changes — say someone changes a text field to a picklist, or a number to a formula — aren’t comprehensively documented in the current OpenFlow materials. In practice, this is a rare enough occurrence that it hasn’t been a problem in my experience, but it’s worth knowing the edge case exists.

Compare this to what Marcus had to maintain. His custom integration used a hardcoded list of fields per object. Every time a field was added, renamed, or retyped, someone had to update the Python code, test it, deploy it, and verify the data. Usually that “someone” was Marcus, usually at an inconvenient time, and usually because nobody told him the change was coming.

The value of automatic schema evolution isn’t technical elegance. It’s that your engineers stop spending time on schema babysitting and start spending it on work that actually matters — building models, improving data quality, answering the questions the business is actually asking.

A Word About Fivetran

I’d be doing you a disservice if I wrote about managed Salesforce connectors without mentioning Fivetran. For most of the last decade, Fivetran has been the gold standard here, and their Salesforce connector is genuinely excellent.

Fivetran’s schema evolution handling is more mature and more configurable than OpenFlow’s current offering. They offer three schema change policies: ALLOW_ALL (sync everything automatically, including new tables), ALLOW_COLUMNS (auto-add new columns but not new tables), and BLOCK_ALL (manual opt-in for everything). They also generate a LOG table that records every schema change event — create_table, alter_table, drop_table — giving you an audit trail that’s useful for debugging and compliance. Field type widening (like INT to LONG) happens automatically; narrowing changes are blocked to prevent data loss.

Their pricing model is fundamentally different from OpenFlow. Fivetran charges per Monthly Active Row (MAR) — the count of distinct rows synced per connection per month. A unique primary key counts once regardless of how many times it’s updated. Approximate pricing clusters around $500 per million MAR on the Standard plan, with a $12,000 annual minimum commitment. Since early 2025, MAR is calculated per connection rather than account-wide, which eliminated the bulk discount that previously benefited multi-connector setups.

OpenFlow, by contrast, charges for compute — SPCS credits for the container runtime, Snowpipe Streaming costs for ingestion, and warehouse costs for merge operations. There’s no per-row fee and no separate licence. It’s included with Snowflake. The trade-off is that your management compute pool runs continuously while a deployment exists, which means you’re paying a base cost even when no data is flowing. One estimate I’ve seen puts idle costs at roughly $10 per day, though this varies with configuration.

For a single large Salesforce org, OpenFlow’s compute-based model may end up cheaper than Fivetran’s per-row pricing — especially if your Salesforce data doesn’t churn heavily. For environments where you need 30+ connectors across different source types, Fivetran’s breadth and maturity is hard to argue with. They have 500+ connectors. OpenFlow has about 20.

The honest framing: OpenFlow is the better choice if you’re already deeply invested in Snowflake, value platform consolidation, and your primary sources are in the current connector catalogue. Fivetran is the safer choice if you need broad connector coverage, battle-tested reliability, and you’d rather pay a premium for someone else to handle the ops.

They’re not mutually exclusive, either. I’ve seen teams run Fivetran for the long tail of small connectors while using OpenFlow for their highest-volume sources where compute-based pricing wins.

The Zero-Copy Alternative

There’s a third path I haven’t mentioned yet, and it’s worth understanding even if it’s not right for every team: zero-copy data federation.

Salesforce Data Cloud offers a zero-copy integration with Snowflake that flips the entire model on its head. Instead of extracting data from Salesforce and loading it into your warehouse, zero-copy gives you direct read access to Salesforce data without moving it at all. Salesforce’s query pushdown engine handles the work — when you query the federated data, Salesforce pushes the query to the source, filters and aggregates there, and returns only the results you need. No pipelines to build. No schemas to manage. No sync schedules to configure.

It works bidirectionally, too. You can share data from Snowflake back into Salesforce Data Cloud, letting sales reps see warehouse-enriched signals directly on Account and Opportunity pages without anyone exporting a CSV or building a reverse ETL pipeline. The integration runs on Apache Iceberg under the hood, which at least means the underlying format is open.

For handling new custom objects — the gap I flagged with OpenFlow — zero-copy sidesteps the problem entirely. There’s nothing to discover because there’s nothing to sync. The data stays where it is, and you access it in place. As your Salesforce org evolves, the federated view evolves with it.

That’s genuinely appealing. For proof-of-concept work, rapid prototyping, or use cases where you need real-time access to Salesforce data without building infrastructure, zero-copy is fast and effective.

But here’s where I get cautious.

It comes at a premium. Salesforce Data Cloud runs on a consumption credit model — credits are purchased in bundles (roughly $500 per 100,000 credits at list price), and every action burns them. Zero-copy federation queries consume about 70 credits per million records, which sounds reasonable until you’re running it at scale across multiple business units. The pricing model has simplified since late 2025 — Salesforce consolidated multiple credit types into a single fungible credit and made ingestion from core Salesforce products free — but it’s still a consumption model with real costs that compound quickly if you’re not watching.

More importantly, zero-copy creates a deep platform dependency. Your data access is mediated entirely through Salesforce and Snowflake’s partnership. If Salesforce changes their terms, adjusts their pricing (which they’ve done repeatedly), or if you decide to move off Snowflake to Databricks or BigQuery, that zero-copy integration doesn’t come with you. You’d need to build actual data movement infrastructure — the thing you avoided — under time pressure and with no existing pipeline to fall back on.

I value keeping infrastructure portable. Not because I’m paranoid about vendor lock-in — I’m pragmatic about it. I’ve been in this industry long enough to see what happens when a new CEO arrives and wants to renegotiate every vendor contract, or when a cloud provider changes their pricing structure overnight, or when the company you depend on gets acquired and the roadmap shifts. Having your data physically in your own warehouse, in formats you control, with pipelines you can redirect to a different destination — that’s insurance. Not glamorous insurance, but the kind you’re grateful for when you need it.

The balance is real, though. You always have to weigh portability against the time you spend building and maintaining pipelines. If your team is drowning in pipeline maintenance, zero-copy might buy you breathing room while you stand up proper infrastructure. Just go in with your eyes open about what you’re trading for that convenience.

There’s a more fundamental concern, though, and it’s the one that won’t appear in any vendor comparison chart.

Zero-copy makes it dangerously easy to skip the data warehouse entirely. When you give analysts and report builders direct read access to raw Salesforce data, the temptation to build dashboards straight from it is enormous. And for a quick proof of concept or a one-off analysis, fine. But the moment those PoC dashboards become “the dashboard the VP checks every Monday,” you’ve got a problem.

Raw source data doesn’t tell you which CustomerNo field to use — the one from the legacy migration, the one from the 2023 CRM consolidation, or the one from the current integration. It doesn’t encode the business rule that says “Closed Won” opportunities in APAC exclude training deals under $5,000 because those are tracked separately. It doesn’t flag that the Last_Activity_Date field is unreliable for accounts owned by the partner team because they log activities in a different system.

That knowledge lives in your data warehouse layer. It lives in the dbt models that your team built over months of conversations with stakeholders, through change management cycles, through debugging sessions where someone finally said “oh, we stopped using that field in Q3 because…” That’s not transformation logic — it’s institutional memory encoded as code. It connects raw data across domains and gives the business a full landscape picture instead of a narrow, single-source view. It catches the behavioural anomalies unique to your organisation — the sales rep who bulk-updates 500 records every Friday afternoon, the integration that occasionally double-fires during DST transitions, the custom object that gets repurposed every time the business restructures.

So by all means, evaluate zero-copy for the use cases where it shines. Real-time signals on Account pages for sales reps? Great fit. Federated access for a specific analytics team that knows the source data intimately? Reasonable. But don’t let it replace the warehouse layer. That layer is where the hard-won understanding of your data lives, and it’s harder to recreate than any pipeline.

The Honest Trade-Offs

I’ve been positive about OpenFlow because I think it solves a real problem well. But I’d be violating my own writing principles if I didn’t lay out the limitations clearly.

The Salesforce connector is still in preview. It works, but it’s not covered by production support SLAs. If you’re running a mission-critical pipeline that your CFO stares at every Monday morning, that matters.

Custom Salesforce domains aren’t supported. If your org uses a vanity URL (like mycompany.my.salesforce.com with a custom domain), check the documentation carefully before committing.

Hard deletes aren’t tracked. The ISDELETED column catches soft deletes, but records that are permanently purged from Salesforce will persist in your Snowflake tables indefinitely. You’ll need a reconciliation process if that matters for your use case.

Certain field types are silently dropped. location, address (compound), and base64 fields are not synced. The connector doesn’t error on these — it just ignores them. This is the kind of thing that bites you three months in when someone asks why geographic data isn’t in the warehouse.

Formula fields require full refresh. Because formula field values are computed server-side and don’t update SystemModstamp, they can’t be synced incrementally. You need a separate connector instance running full refreshes to capture formula field changes. That’s not a bug — it’s how Salesforce works — but it’s a gotcha if you’re not expecting it.

No relationship traversal. You can’t configure the connector to follow lookups and pull related objects automatically. Each object is pulled independently. Joining them is your job in the transformation layer, which is where it belongs anyway if you’re running dbt.

None of these are dealbreakers. But every one of them is the kind of thing that bites an engineer at 4pm on a Friday if they weren’t expecting it. Now you’re expecting it.

Getting Started

If you want to try this on your own Salesforce org, here’s my recommended order of operations:

Start with a non-production Salesforce sandbox and a Snowflake trial account if you don’t have a dev environment handy. Pick two or three standard objects — Account, Contact, Opportunity — and one custom object if your org has them. Run through the three-phase setup I described above. Get data flowing, verify the table structures, and then add a custom field to one of the objects in Salesforce. Wait for the next sync. Watch the column appear automatically in Snowflake.

That’s the moment it clicks. That’s the moment you stop thinking about managed connectors as a luxury and start thinking about custom API code as technical debt you’re choosing to carry.

Once you’ve validated the core flow, layer on your dbt models. OpenFlow lands raw data — it doesn’t transform it. Your staging models handle naming conventions, type casting, soft delete filtering, and the join logic that stitches related objects together. This is the same pattern you’d use with Fivetran or any other EL tool: land it raw, transform it in the warehouse.

For teams already running Fivetran for Salesforce, I’m not suggesting you rip it out tomorrow. If it’s working and the cost is acceptable, that’s a solved problem. But the next time you’re adding a new high-volume source that’s in OpenFlow’s catalogue — especially if you’re already paying for Snowflake — it’s worth running the numbers. You might find that native integration with compute-based pricing is the better deal.

The Real Why

I started this article with Marcus’s story because it illustrates something I feel strongly about: your engineers’ time is the most expensive resource in your data organisation, and spending it on solved problems is a leadership failure.

Building a custom Salesforce integration isn’t impressive. It was impressive in 2016, when the tooling didn’t exist. Today, it’s a choice to take on maintenance burden that a managed service handles automatically. It’s choosing to babysit schema changes instead of building the models and analyses that actually move the business forward.

Marcus is a senior engineer now. He builds dimensional models and designs data products that directly influence how the sales team allocates resources. He doesn’t write API integration code anymore. Not because he can’t — because his time is worth more than that.

But here’s what I keep coming back to. The tools keep getting better — OpenFlow, Fivetran, zero-copy, and yes, AI-assisted pipeline generation. Every year there’s a new thing that promises to automate another piece of the data engineering workflow. And many of them genuinely do.

What they don’t automate is the part that actually matters.

No tool — not OpenFlow, not Fivetran, not an AI agent — is going to sit in a room with your head of sales and ask “what keeps you up at night?” No connector is going to notice that the way your company defines “active customer” has quietly drifted from what the CRM tracks. No pipeline is going to push back and say “we could build that dashboard, but the metric you’re asking for doesn’t answer the question you actually have.”

That’s the connective tissue between raw data and business value. It’s a person who understands the domain, who’s been present through the change management cycles, who’s built relationships with the stakeholders and knows which CustomerNo field to use because they were in the room when the decision was made three years ago. It’s someone who asks clarifying questions instead of presuming they already know the answer.

AI will keep getting better at the mechanical parts of data engineering. Schema detection, pipeline code generation, anomaly detection — those are well-defined problems that automation is suited for. But the assumption that any tool can skip the human step — the clarification, the context, the judgment about what’s actually worth building — that’s where I’ve seen projects go sideways. Not because the technology failed, but because nobody asked the right questions before building.

OpenFlow isn’t perfect. The connector catalogue is young, the Salesforce connector is in preview, and Fivetran still wins on breadth and battle-hardened maturity. But OpenFlow represents something important: the data platform taking responsibility for data movement, not just data storage and compute. It’s Snowflake saying “we’ll handle getting the data in — you focus on making it useful.”

That last part — making it useful — is still your job. And it’s the part that no tool can do for you. The best thing a managed connector gives you isn’t fewer lines of code. It’s time back. Time to spend on the work that actually requires a human: understanding the business, building the right models, and making sure the data tells an accurate story.

The next time someone on your team says “I’ll just build a quick Salesforce integration” — send them this article. And the next time someone says “AI can just handle the data pipeline” — ask them who’s going to sit with the stakeholders and figure out what the pipeline should actually deliver.

That’s still you. Make sure you have the time for it.

Your Data Model Isn't Broken, Part II: The Refactoring Playbook

Chris Hillman — Sat, 28 Mar 2026 09:00:00 +1100

In [Part I], I made the case that your legacy data model isn’t the disaster it looks like. That the strange WHERE clauses, the bridge tables nobody can explain, and the slowly-changing-dimension-within-a-slowly-changing-dimension aren’t bugs — they’re business rules earned through years of production reality. I argued that big-bang rebuilds fail at alarming rates, that the complexity you’re fighting is mostly essential rather than accidental, and that the impulse to “start from scratch” is driven more by cognitive bias than by engineering judgment.

A few people messaged me afterward and said, roughly: “Okay, I’m convinced. But what am I supposed to do about it?”

Fair question. Let’s talk about the playbook.

The vine that eats the tree

In 2004, Martin Fowler was visiting a rainforest in Queensland when he noticed something remarkable about the strangler fig trees. These vines start as seeds deposited in the upper branches of a host tree by birds. They grow downward, wrapping around the trunk, eventually establishing their own root system in the soil. Over years — sometimes decades — the fig gradually replaces the host tree. The original tree decays inside the fig’s lattice until there’s nothing left of it. But at no point during this process does the canopy collapse. The forest continues to function the entire time.

Fowler looked at those trees and thought: that’s how you should replace software systems.

The Strangler Fig pattern is, in my experience, the single most useful idea in all of software architecture — and it maps onto data warehouse migrations with almost uncomfortable precision. The concept is deceptively simple: instead of replacing the old system all at once, you build the new system around it. You intercept one piece of functionality at a time, route it through the new implementation, verify it works, and move on. The old system slowly shrinks. The new system slowly grows. And at no point does the business lose access to anything.

Fowler updated his essay on this pattern as recently as August 2024, and his framing is worth internalising. He described the alternative — the big-bang rewrite — and noted that he’d watched it fail most of the time. The strangler fig approach, by contrast, gives value steadily and allows you to monitor progress more carefully through frequent releases.

Now, here’s where it gets interesting for data teams specifically. Application systems have clean interfaces — APIs, endpoints, well-defined contracts. You can intercept a request, route it to the new system, and nobody downstream knows the difference. Data systems are messier. Elliott Cordo, who documented migrating three legacy data warehouses using this pattern, identified the core challenge: analytics systems have porous boundaries and poorly defined interfaces with consumers. You don’t always know who’s querying what, or how.

Cordo’s solution — and this is the variant I’ve seen work best — is what he calls the Legacy Façade. You build a new interface that mimics the legacy system’s contract. Same table names, same column names, same output format. Behind the façade, you’ve plumbed in the new logic. Consumers don’t change anything. They don’t even need to know a migration is happening. And this, critically, eliminates the organisational bottleneck that kills most migrations: getting twenty different teams to simultaneously update their queries, dashboards, and downstream pipelines to point at the new system.

Cordo’s experience is particularly telling because one of the three warehouses he migrated existed precisely because a previous attempt to replace one of the others had failed — mainly due to change management. The strangler fig approach bypasses that failure mode entirely by asking consumers to do little or nothing to adopt the new system.

I think this matters so much: the majority of data warehouse migrations don’t fail for technical reasons. They fail because you can’t convince the entire organisation to cut over on the same Tuesday. The Legacy Façade removes that dependency. You migrate at your pace, not at the pace of every team that consumes your data.

Stop publishing before you’ve checked

I’m continually surprised by how few people have heard of this pattern. When Robin Moffatt polled data engineers on Reddit about Write-Audit-Publish, most hadn’t even encountered the term — let alone used it. That’s a problem, because WAP solves the single scariest moment in any refactoring effort: the moment you push changed logic to production and hope you didn’t break anything.

Netflix introduced WAP at the DataWorks Summit in 2017, and the concept is elegant in its simplicity. Instead of the typical pipeline flow — transform data, write it to the production table, then maybe run some checks afterward — you split the process into three distinct stages:

Write to an isolated staging area. Not production. Not anywhere consumers can see it.

Audit the output against your quality checks. Row counts. Schema validation. Business rule assertions. Whatever gives you confidence.

Publish only if the audit passes. If it doesn’t, production never sees the bad data. Nobody’s dashboard breaks. Nobody’s quarterly report includes garbage. The corrupted data dies quietly in staging where it belongs.

The reason this matters for refactoring specifically is that refactoring requires you to change transformation logic while producing identical output. You’re restructuring the internals — breaking a 400-line SQL query into modular CTEs, moving logic from stored procedures into dbt models, replacing hardcoded values with reference table lookups. Each of these changes is supposed to be behaviour-preserving. WAP gives you the safety net to verify that it actually was.

Julien Hurault described the common anti-pattern — what he calls Write-(Publish)-Audit — where if an error is detected during testing, it’s already too late because the corrupted data has already been released to downstream systems. I’ve seen this play out more times than I want to admit. Someone refactors a model, the tests pass in the development environment, the change gets deployed, and then a stakeholder notices that revenue is off by 12% in a dashboard three hours later. By that point, the bad data has propagated through six downstream models and someone’s already screenshot the dashboard for a presentation.

WAP prevents this entirely. And with Apache Iceberg now supporting it natively through branch-based isolation, the tooling has finally caught up with the idea. You write to an audit branch, run your validations, and only fast-forward merge to main when everything checks out. It’s the data engineering equivalent of a pull request with automated tests — and honestly, it’s bizarre that we’ve accepted as normal the practice of deploying data transformations straight to production without a verification step in between.

If you’re on Snowflake, the implementation is straightforward using CLONE and SWAP operations. Clone the production table, run your refactored pipeline against the clone, validate the output, and swap the clone into the production role. If validation fails, the clone gets dropped and production is untouched. I’ve written about this pattern in more detail here — but the key point for this article is simpler: WAP turns refactoring from a high-wire act into a methodical, reversible process.

The archaeology of SQL

How does refactoring a data model actually works in practice, because it’s less dramatic than people expect. That’s sort of the point.

dbt Labs published a refactoring course that formalises the approach, and having used variants of this workflow on multiple teams, I think they’ve got it right. The core philosophy is one sentence: while refactoring, you’ll be moving around a lot of logic, but ideally you won’t be changing the logic.

Read that again. You are not improving business rules during a refactor. You are not fixing data quality issues during a refactor. You are not adding new features during a refactor. You are rearranging the existing logic into a better structure while proving, at every step, that the output hasn’t changed. The improvements come later, once the structure is clean enough to support them safely.

The workflow goes something like this. First, you take the legacy SQL — the 800-line stored procedure, the five nested subqueries, the transformation that nobody wants to touch because the last person who edited it left the company — and you port it into a dbt model unchanged. You don’t clean it up. You don’t refactor it. You bring it over exactly as it is, warts and all, and you verify that it produces identical output.

This is the step most people want to skip, and it’s the most important one. That ugly SQL is your baseline. It’s your source of truth. Until you’ve proved your refactored version matches it row-for-row, column-for-column, you don’t have a refactor — you have a rewrite wearing a refactor’s clothing.

Once you’ve got your baseline, you start decomposing. Extract the source references into dbt sources. Break the monolithic query into CTEs, each one doing a single logical operation. Move shared logic into staging models. Separate business rules from data cleaning from aggregation. At each step — and this is non-negotiable — you use something like dbt’s audit_helper package to compare the output of your refactored model against the original. If the outputs match, you move on. If they don’t, you figure out what you changed and fix it before going further.

Tristan Handy talks about his formative experience at Casper in 2016, where his team refactored all of their existing pipelines and brought them over to dbt in a single week, delivering work at least ten times faster than they would have been able to do otherwise. That speed came not from heroics but from methodology: migrate unchanged, then decompose with verification at every step.

I’ve watched a team at refactor over 200 dbt models with complex logic in three days using audit_helper. Three days. Not three months. Not the eighteen-month rebuild timeline that some stakeholder approved in a steering committee. Three days, because the tooling verified every change automatically and the team could move fast with confidence.

That’s the real payoff of refactoring over rebuilding. Not that it’s safer — though it is. Not that it preserves business logic — though it does. The payoff is that it’s faster. Faster to start delivering value. Faster to reach a clean architecture. Faster to build the confidence you need to actually make meaningful improvements.

Expand, then contract

There’s a pattern for handling breaking schema changes that I don’t see data teams use nearly enough, and it deserves more attention. Fowler calls it Parallel Change. Others call it Expand and Contract. The name doesn’t matter.

Imagine you need to rename a column in a fact table that thirty dashboards reference. In a rebuild mindset, you’d design the new schema, build it, and then coordinate a cutover where every consumer updates their queries simultaneously. Good luck with that. In my experience, “simultaneously” means “over the course of several painful weeks, with a growing list of things that are broken in the meantime.”

Expand and Contract takes a different approach, in three phases:

In the Expand phase, you add the new column alongside the old one. Both exist. Both contain the same data. Nothing breaks, because nothing has been removed.

In the Migrate phase, you update consumers one at a time. Dashboard A switches to the new column. Pipeline B switches. Report C switches. Each migration is independent, low-risk, and reversible. There’s no coordination required between teams.

In the Contract phase — and only after every consumer has migrated — you drop the old column. By this point, nobody is using it, so removing it is a non-event.

Does this take more deployments? Yes. it might require up to five deployments before the change is fully in effect. But each deployment is safe. Each deployment is reversible. And at no point does anyone lose access to their data because you renamed a column in a way they weren’t expecting.

For data models specifically, Expand and Contract shines when you’re restructuring dimensions, splitting fact tables, or changing grain. Add the new structure alongside the old one. Migrate consumers gradually. Remove the old structure once it’s no longer referenced. It’s boring. It’s methodical. It works.

The ship that replaced itself

If you want a single case study that demonstrates what successful incremental replacement looks like at scale, look at what Slack did with their desktop client in 2019.

Slack’s engineering team faced a classic dilemma. Their desktop application — built on jQuery and a custom framework — had accumulated years of complexity. It was slow. It used too much memory. The architecture made certain improvements difficult or impossible. Every instinct said: rewrite it in React.

Their engineering blog explicitly acknowledged the risk: running code knows things. Hard-won knowledge gained through billions of hours of cumulative usage and tens of thousands of bug fixes.

So instead of a big-bang rewrite, they did something much harder and much smarter. Over two years, they replaced every component of the desktop client while shipping continuously. They called it modernising “bit by bit” — enforcing strict interfaces between existing and modern code, shipping every change to production as it was completed, and never asking users to endure a disruptive cutover.

The results: 50% memory reduction, 33% load time improvement. Users never experienced a broken release. Had they waited until the entire application was rewritten before releasing it, their users would have had a worse experience for years while the team worked in a vacuum.

Now, Slack is an application, not a data warehouse. But the principles translate directly. Strict interfaces between old and new code? That’s the Legacy Façade. Shipping continuously? That’s WAP combined with Expand and Contract. Never asking users to endure a disruption? That’s the entire point.

The ancient Greeks had a thought experiment about this: if you replace every plank of a ship over time, is it still the same ship? Philosophers argue about it. Engineers don’t care. The ship still floats. The passengers still get where they’re going.

The numbers, for the sceptics

I held back on the quantitative evidence in Part I because I wanted to make the argument from experience and conviction. But some people want receipts, so here they are.

McKinsey and Oxford University studied 5,400 IT projects with initial budgets exceeding $15 million. On average, large projects ran 45% over budget and 7% over time, while delivering 56% less value than predicted. Seventeen percent were “black swans” — cost overruns exceeding 200% that threatened the company’s viability. One retailer abandoned a $1.4 billion IT modernisation, failed on a $600 million follow-up, and filed for bankruptcy.

The Standish Group has been tracking IT project outcomes since 1994. In their original CHAOS report, only 16.2% of projects succeeded on time, on budget, with specified features. Average cost overrun: 189%. By 2020, the overall success rate had improved to 31% — but here’s the statistic that matters most: small projects succeeded approximately 90% of the time. Large projects succeeded less than 10% of the time.

Read that again. Small projects: 90%. Large projects: under 10%.

That’s not a technology problem. That’s a scope problem. And the single most reliable way to reduce scope is to stop rebuilding entire systems and start refactoring them one piece at a time.

For data systems specifically, Gartner reports that more than 50% of data warehouse projects fail to reach user acceptance. Other analyses put the number closer to 80%. CDInsights reports that 70% of data warehouse modernisation projects exceed budget or fail. Integrate.io found that approximately 80% of data migration projects exceed timelines or budgets, with large-scale projects showing 50% higher failure rates than incremental approaches.

I don’t know how much clearer the data can be. Big-bang rewrites of data platforms fail most of the time. Incremental approaches succeed most of the time. The methodology I’ve described in this article — strangler fig, WAP, dbt refactoring, expand and contract — isn’t just safer or more philosophically sound. It’s what the evidence says actually works.

Refactoring as a practice, not a project

Refactoring should not be a project. It should not have a start date and an end date and a steering committee and a Gantt chart. Refactoring should be a practice — something your team does continuously, as a natural part of how you work with your data models.

Every time you touch a model to add a feature or fix a bug, you leave it a little cleaner than you found it. You extract a hardcoded value into a reference table. You add a test that didn’t exist before. You rename a column from col_7 to something a human can understand. You break a 200-line CTE into two smaller, focused ones.

None of these changes are dramatic. None of them require a business case or a migration plan. They’re small, safe, incremental improvements that compound over time. A year from now, the model is cleaner, better-tested, and easier to understand — not because you stopped everything to rebuild it, but because you improved it a little bit every time you were in the neighbourhood.

This is what Fowler meant when he defined refactoring as changing internal structure without modifying observable behaviour. It’s not a phase. It’s a discipline. And it’s the discipline that separates data teams that thrive from data teams that periodically torch everything and start over every three years, wondering why they never seem to make lasting progress.

The closing argument

In Part I, I said your data model isn’t broken — it’s battle-scarred, and those scars are knowledge. In Part II, I’ve tried to show you what the alternative to demolition looks like: patient, methodical, verified improvement. Strangler figs instead of bulldozers. Write-Audit-Publish instead of deploy-and-pray. Expand and contract instead of coordinate-and-hope.

It’s not glamorous work. Nobody’s going to invite you to give a conference talk about the time you renamed some columns and extracted a few staging models. You won’t get a promotion for migrating a legacy fact table so smoothly that nobody noticed it happened.

But that’s exactly the point. The best infrastructure work is invisible. The best migration is the one where the business never had to care. The best data model is the one that quietly absorbed twenty years of business complexity and still answers questions accurately at 7am on a Monday when a Department Head needs numbers for a board meeting.

I wrote Part I of this series last week. The technology has changed — Netscape to Snowflake, CGI scripts to dbt — but the lesson hasn’t. Old code that works is more valuable than new code that doesn’t exist yet. Embedded knowledge is harder to create than clean architecture. And the patient, unglamorous work of incremental improvement will always outperform the seductive fantasy of starting fresh.

Your data model isn’t broken. Stop trying to replace it. Start making it better.

You Don't Need Permission to Fix Your Data

Chris Hillman — Sat, 21 Mar 2026 09:00:00 +1100

Let me tell you about a junior engineer called Sam.

Sam had been on the team about four months when I noticed something in a pull request. Tucked between two routine model changes was a new schema.yml entry — five accepted_values tests on a column called customer_status that had been silently accumulating fourteen different spellings of “active” for the better part of a year.

Nobody asked Sam to do this. It wasn’t in a sprint. There was no Jira ticket. Sam had just been working in that part of the warehouse, noticed the mess, and decided to clean it up on the way through.

I left a comment on the PR: “Nice catch.” Sam’s reply was almost apologetic — “Hope it’s okay that I added these, I wasn’t sure if I was supposed to.”

That response stuck with me. I wasn’t sure if I was supposed to.

Here was someone who’d spotted a real problem, built a real solution, tested it, and shipped it — and their instinct was to wonder whether they had permission. Not whether the fix was correct. Not whether the tests were well-written. Whether they were allowed.

I’ve thought about that moment a lot since, because it captures something I see in almost every data team I work with. The people closest to the problems — the ones running the queries, staring at the nulls, fielding the Microsoft Teams messages when dashboards look wrong — are often the last to believe they can do anything about it. Not because they lack skill. Because somewhere along the way, they absorbed the idea that fixing things is someone else’s job.

That belief is expensive. And it’s wrong.

I’m telling you this because data quality — the real, unglamorous, column-by-column work of making data trustworthy — is one of those things that sounds like a senior engineering problem until you see a four-month-old team member fix something a dozen experienced engineers walked past every day. It doesn’t take authority. It takes the willingness to act before you’re told to.

This article is about how to be more like Sam. Five tactics you can start this week — no permission required — backed by real case studies from companies like Airbnb, Google, and Warner Bros. Discovery that figured this out the hard way.

The $12.9 million reason your quality problem isn’t “too small” to fix

Before we get into tactics, let’s talk about why your “small” fix matters more than you think.

Gartner pegs the average annual cost of poor data quality at $12.9 million per organisation. IBM and Harvard Business Review put the aggregate U.S. cost at $3.1 trillion annually. These numbers feel abstract until you realise what they actually represent: knowledge workers spending roughly half their time hunting for data, correcting errors, and searching for confirmatory sources instead of doing their actual jobs.

Monte Carlo’s 2024 State of Data Quality survey found bad data now impacts 31% of company revenue on average — up from 26% just two years earlier. Their 2022 survey found data engineers spend 40% of their time evaluating or checking data quality. Soda’s research put that number even higher: 61% of data engineers spend half or more of their time handling data issues.

And the human cost? A DataKitchen survey of 600 data engineers found 87% are frequently blamed when things go wrong with data. 78% wish their job came with a therapist.

The implication for you: the problem is so large and so costly that any measurable improvement you create has real dollar value attached to it. That null check you added last Thursday? That’s not a minor fix — it’s a tiny piece of a multi-million-dollar problem that nobody else bothered to address.

Tactic 1: write tests for your own models without asking anyone

The single best case study for this tactic comes from Airbnb, and it didn’t start with a VP mandate or an OKR. It started with one engineer.

Airbnb’s engineering blog documents how they went from “an engineering culture in which most new code shipped without a single test” to one where proposing a change without tests got called out immediately. The transformation wasn’t top-down. As the blog explains: “One approach is to rule by edict, but given our culture of engineer autonomy this would have been very poorly received. The other approach is to lead by example and build a movement.”

That single champion’s playbook was remarkably tactical. They spoke in engineering meetings. They held office hours showing teammates how to write tests. They published a weekly newsletter highlighting well-written test specs — giving public credit to engineers who included tests. They revamped the testing bootcamp for new hires, effectively making each new hire a champion. They improved tooling, reducing CI build times from over an hour to roughly six minutes.

The result? Airbnb later applied the same grassroots philosophy to data quality, building a DQ Score on a 0–100 scale across their entire warehouse. Their anomaly detection system now prevents quality issues in new pipelines before they reach production. The Minerva metrics platform manages 12,000+ metrics and 4,000+ dimensions with 200+ data producers.

One engineer. No mandate. A newsletter and some office hours.

What this looks like for you

If you’re working with dbt, you already have the tools. dbt ships with four built-in generic tests — not_null, unique, accepted_values, relationships — that you define in YAML alongside your model definitions. No framework to build. No proposal to write. Just add a few lines to your schema.yml:

models:
 - name: dim_customers
 columns:
 - name: customer_id
 tests:
 - not_null
 - unique
 - name: customer_status
 tests:
 - accepted_values:
 values: ["active", "inactive", "churned", "suspended"]
 - name: email
 tests:
 - not_null

That’s it. Next time the pipeline runs, those tests execute automatically. If someone pushes a change that breaks uniqueness on customer_id, the pipeline catches it before it reaches production.

The dbt-expectations package goes further, porting Great Expectations’ advanced tests into dbt. Unit tests — introduced in dbt v1.8 — let you validate SQL logic with static mock inputs before transformations run. Tests integrate directly into CI/CD, blocking PRs with failures from reaching production.

Not on dbt? You can still write assertion queries. Here’s a completeness check you can run against any SQL database — no framework required:

-- Quick completeness check: what percentage of each critical
-- field is actually populated?
SELECT
 COUNT(*) AS total_records,

 -- How many emails do we actually have?
 ROUND(100.0 * COUNT(email) / COUNT(*), 1) AS email_pct,

 -- How many phone numbers?
 ROUND(100.0 * COUNT(phone) / COUNT(*), 1) AS phone_pct,

 -- How many addresses?
 ROUND(100.0 * COUNT(address_line_1) / COUNT(*), 1) AS address_pct,

 -- Flag: are we below the 95% threshold on anything critical?
 CASE
 WHEN COUNT(email) < COUNT(*) * 0.95 THEN 'BELOW THRESHOLD'
 ELSE 'OK'
 END AS email_status
FROM
 main.customers;

Schedule that to run daily. Store the results in a logging table. You’ve just built a monitoring system — and you didn’t ask anyone’s permission.

The numbers back this up

Monte Carlo’s 2022 survey found that organisations conducting at least three types of data tests weekly experienced 46 incidents per month compared to 61 incidents per month for less rigorous testers — a 25% reduction. The Forrester Total Economic Impact study of data observability platforms (April 2025) found 358% ROI, 6,500+ data personnel hours reclaimed annually, and $1.5M+ in avoided lost revenue from reduced data downtime.

Daniel Beach of Data Engineering Central offers a pragmatic path for getting started: begin with end-to-end tests first when inheriting a pipeline with no tests, then add data quality checks, then unit tests. You don’t need to boil the ocean. You need to start.

Tactic 2: document one thing every time you touch it

The numbers on missing documentation are genuinely staggering. According to DX, developers spend 3–10 hours per week searching for information that should be documented. For a 100-person engineering team, that equals 300–1,000 hours weekly — the equivalent of 8–25 full-time engineers doing nothing but hunting for answers. Stack Overflow’s developer survey found 62% of developers spend over 30 minutes daily on poorly documented issues. And 38% listed poor documentation among their top reasons for leaving a company.

One practitioner tracked “Time to First Production Commit” at 28 days for new engineers — versus an industry benchmark of 3–5 days. Engineers on that team spent 30% of their time rediscovering what previous engineers had already learned. 40% of sprint tickets required inter-team clarification, averaging 12 clarifications per ticket.

This is what documentation debt looks like. Not a gap in a wiki — a tax on every single person who touches the codebase.

“Documentation Fridays” don’t work. This does.

Multiple sources confirm that big-bang documentation efforts fail. As one case study puts it: “‘Okay team, let’s spend Friday afternoon documenting our processes!’ Nobody wants to do that. And it doesn’t work.”

The successful pattern is document-as-you-go. Every time you encounter an issue, you write it down. Shopify Engineering treats documentation as a product, explicitly encouraging new team members to fill documentation gaps from day one. Their philosophy: “The most effective way to encourage others to write is to actually write.”

For data teams specifically, the approach is even simpler. If you’re using dbt, column descriptions live in schema.yml alongside your model definitions. dbt’s column-level lineage means passthrough and renamed columns automatically inherit descriptions from upstream models. Write a description once, and it flows downstream through every model that references that column.

Here’s what this looks like in practice:

models:
 - name: dim_customers
 description: >
 One row per customer. Grain: customer_id.
 Source: CRM extract via Fivetran, refreshed daily at 03:00 UTC.
 Owner: data-eng-team@company.com
 columns:
 - name: customer_id
 description: "Primary key. Sourced from CRM.contacts.id. Unique, never null."
 - name: customer_status
 description: >
 Current lifecycle status. Valid values: active, inactive, 
 churned, suspended. Updated by CRM webhook on status change.
 Known issue: legacy records from pre-2023 may contain 
 'Active' (capitalised) — normalised in this model.
 - name: lifetime_value
 description: >
 Cumulative revenue attributed to this customer, in AUD.
 Calculated as SUM(order_total) from fct_orders.
 Excludes refunded orders. Updated daily.

That “Known issue” note on customer_status? That’s worth its weight in gold to the next person who touches this model. It took thirty seconds to write and it’ll save hours of confusion.

Microsoft’s engineering blogs describe the alternative bluntly: “When tribal knowledge holders leave, reverse engineering is the only way to understand what was done.” Enboarder’s 2025 research found 47% of organisations cite institutional knowledge loss as their top offboarding challenge. Atlan’s Modern Data Survey found data professionals waste one full day per week simply trying to figure out what data to use.

You don’t need to document the entire warehouse. You need to document the thing you touched today. Tomorrow, document the next thing. In three months, you’ll have built something genuinely valuable — and you’ll have done it one commit at a time.

Tactic 3: build a quality dashboard nobody asked for

At Warner Bros. Discovery, the DICE team built data quality management frameworks and created a “Data Quality Forum” that became a bridge between teams. During the Olympics livestream, they used custom SQL checks to detect missing content metadata — fixing issues before they broke reporting. The forum evolved into what Pam Zirpoli called a “cultural flywheel.” Patterns surfaced in the forum were incorporated into onboarding, translated into new monitors, and used to refine their priority matrix. Teams shifted from reactive — reacting after dashboards broke — to proactive.

At Tempus, a data engineer documented how adopting Elementary (an open-source dbt package) transformed their monitoring. Before: “test alerting relied on a dashboard in Data Studio built on top of a log sink, and the ‘alerts’ were a daily email that you actually had to open.” After: Microsoft Teams alerts that “@-ed a user and continued to do so until they’re turned off.” The key outcome: tests no longer fail silently.

The six metrics that actually matter

Drawing from IBM, dbt Labs, Databricks, and Alation, the consensus metrics for data quality monitoring are:

Freshness: time since last data update. If your orders table hasn’t been updated since yesterday, something’s broken.
Volume: expected versus actual row counts. Did today’s load bring in 50,000 rows when you normally get 48,000–52,000? Fine. Did it bring 12? Problem.
Schema changes: added, deleted, or altered columns. These should never surprise you.
Completeness: null rates per column. Simple COUNT(*) vs COUNT(column) tells you a lot.
Distribution shifts: statistical changes in data values. If the average order value suddenly doubles, that’s worth investigating.
Test pass/fail rates: the percentage of your quality assertions that are passing. Track this over time.

Here’s a freshness check you can run right now — no tools, no frameworks, just SQL:

-- How stale is each of your critical tables?
-- Run this daily and store results in a monitoring table
SELECT
 'fct_orders' AS table_name,
 MAX(updated_at) AS last_record_timestamp,
 CURRENT_TIMESTAMP AS checked_at,
 ROUND(
 EXTRACT(EPOCH FROM (CURRENT_TIMESTAMP - MAX(updated_at))) / 3600.0,
 1
 ) AS hours_since_update,
 CASE
 WHEN MAX(updated_at) IS NULL THEN 'CRITICAL - No data'
 WHEN CURRENT_TIMESTAMP - MAX(updated_at) > INTERVAL '24 hours'
 THEN 'STALE - Exceeds 24hr SLA'
 WHEN CURRENT_TIMESTAMP - MAX(updated_at) > INTERVAL '12 hours'
 THEN 'WARNING - Approaching SLA'
 ELSE 'FRESH'
 END AS freshness_status
FROM analytics.fct_orders;

Store that in a data_quality_log table, connect it to Looker Studio or Metabase (both free), and you’ve got a monitoring dashboard. That’s it. No Kubernetes cluster. No proposal document.

Visibility changes behaviour

Anna Geller documented how sending data quality alerts to a shared Microsoft Teams channel transforms team behaviour. Her core insight: “As social creatures, we are more motivated to fix issues if other people can see the effort we put into it.” Data owners reply with root cause analysis, add checkmarks, or link tickets — creating a social proof loop that issues are no longer ignored.

This isn’t just anecdotal. Research on the Hawthorne effect shows healthcare hand hygiene compliance increases by up to 70% when workers know they’re observed, and employees increase productivity by an average of 13% when activity is monitored. Making data quality visible doesn’t just inform people — it changes their behaviour.

Tactic 4: create a #data-bugs channel

MIT Sloan Management Review’s HelloFresh case study documents one of the best examples of how visibility catalyses organisational change. HelloFresh transitioned through three modes of data quality management. In Mode 1, issues were dealt with reactively by whoever happened to be affected. In Mode 2, a central engineering team handled cleanup but lacked understanding of how data was actually used — “data customers grew increasingly frustrated as they continued to spend large fractions of their workdays dealing with data that did not meet their needs.”

The transition to Mode 3 — proactive, collaborative quality management — happened precisely because “quality issues became increasingly visible.” Not because someone wrote a strategy doc. Not because a VP mandated it. Because the problems became impossible to ignore.

You can accelerate that transition. Create a Microsoft Teams channel. Call it #data-bugs or #data-quality or whatever feels natural. Start posting issues you find — with specifics. Not “the data looks wrong” but “fct_orders has 3,247 rows with null customer_id from the 2026-02-15 load, affecting the weekly revenue dashboard.”

That’s it. You’ve just created the visibility mechanism that HelloFresh spent years arriving at.

The broken windows effect is real — and it’s been measured

The Pragmatic Programmer introduced the “broken windows” software analogy decades ago: “Don’t leave ‘broken windows’ (bad designs, wrong decisions, or poor code) unrepaired. Fix each one as soon as it is discovered.”

This isn’t just metaphor anymore. A 2024 arXiv paper by Diomidis Spinellis et al. analysed 2 million code commits across 122 projects comprising 5.5 million lines of code. The key finding: “History matters — developers behave differently depending on some aspects of the code quality they encounter.” Developers tailor the quality of their commits based on the quality of the file they’re committing to. Low quality begets lower quality. High quality attracts higher quality.

Mat Ryer extends this to everyday engineering: “If you work on a project that has flaky tests, then you’re more likely to add more flaky tests. If there is a hacky design, you’re more likely to hack more in.”

Fixing one visible data quality issue — or even just making it visible — can interrupt the decay spiral.

Why juniors don’t speak up (and why that’s exactly the problem)

Here’s the uncomfortable part. A Culture Shift UK survey found junior colleagues were twice as likely (54%) as senior leaders (27%) to say speaking up about issues is “pointless.” 37% said speaking up “isn’t worth the personal risk.” A Blind survey found 58% of tech workers experience imposter syndrome.

Google’s Project Aristotle studied 180+ teams over two years and found psychological safety was “by far the most important” of five dynamics for team effectiveness. Amy Edmondson’s foundational research contains a striking finding: while studying hospital teams, she expected high-performing teams to report fewer errors. Instead, better teams reported MORE errors. Her insight: “Maybe the good teams don’t make more mistakes, maybe they report more.”

This is the paradox. The teams that look like they have the most data quality issues are often the healthiest — because they’ve created an environment where problems surface instead of festering.

Etsy pioneered blameless postmortems under CTO John Allspaw. Engineers send company-wide emails confessing mistakes. Etsy even gives out an annual “three-armed sweater” award to the employee who made the most surprising error. As a Hadley Wickham–curated Stanford reading list notes, while blameless postmortems are standard in DevOps, they “have not yet infiltrated standard data science practices.” That’s a gap you can help fill.

Creating a #data-bugs channel isn’t just about tracking issues. It’s about establishing the norm that finding problems is good work, not a sign that something’s wrong with you.

Tactic 5: ship an example, don’t write a proposal

Marcus Blankenship’s article on learned helplessness in software engineering — 96,000+ reads, 900+ Reddit comments — contains a story about a coworker called Milind. Milind wasn’t a manager. He didn’t have a fancy title. But he was an informal leader through pure action. He asked questions about why particular approaches were taken. He admitted ignorance in group settings. He insisted the team understand root causes before moving on.

Blankenship’s observation: “Milind changed our environment by his actions… He showed me that I had much more power than I thought, if I would only stop expecting to be spoon-fed everything. His actions showed me that true leadership isn’t granted, it’s grasped.”

A junior engineer on Irina Stanescu’s newsletter shared something similar: “Even as a junior engineer, I remember influencing my chief architect and brought Terraform into my organisation 7 years ago by simply pointing out a common pain point and how much longer things took without managing our infrastructure in code.” They added: “Being able to influence without authority is a very underrated skill, and I believe this is what helped me get promoted to senior engineer even more than my technical skills.”

The pattern is consistent. Don’t propose a data quality framework. Don’t write a 15-page RFC. Build the thing. Make it work. Show people.

Why working examples spread faster than proposals

Everett Rogers’ Diffusion of Innovations theory identifies five characteristics that determine how fast an idea spreads: relative advantage, compatibility, complexity (lower is better), trialability, and observability. A working example maximises both trialability and observability — the two factors most under your control as a junior engineer.

A proposal says “we should do this.” A working example says “look, I already did this, and here’s what happened.” One requires people to imagine the benefit. The other puts the benefit directly in front of them.

Addy Osmani, after 14 years at Google, captured the complementary lesson: “Early in my career, I believed great work would speak for itself. I was wrong. Code sits silently in a repository.” You need to build the example and make its impact visible. Post in Microsoft Teams. Show the before and after. Quantify what changed.

The junior advantage nobody talks about

Ant Weiss’s widely-read article on organisational learned helplessness identifies common symptoms: “This is how we were told to work,” “These are the limitations of the system,” “We don’t have the resources.” His core observation: “No engineer starts their career willing to be unproductive, pessimistic and stressed out. But the companies they work at teach them to.”

The cure, drawn from Seligman’s original research, is critical: “Threats, rewards, and observed demonstrations had no effect on the ‘helpless’ dogs. The dogs had to be physically moved through the escape action at least twice before they would do it themselves.” You can’t just tell engineers things can be better. You have to walk them through the experience of making something better.

Here’s the part that should give you confidence. JazzTeam reports a remarkable finding: “To prove that any problem can be solved, our team many times gave a task to Junior and even Intern Engineers who did not have psychological fears and tunnel thinking.” Juniors, unencumbered by years of “that’s impossible” conditioning, were able to tackle problems seniors had declared unsolvable.

Your fresh eyes aren’t a weakness. They’re an asset. You haven’t yet learned that “it can’t be done.” So go do it.

Putting it all together

None of these tactics require a committee meeting. None require architectural approval. None require you to be senior.

Here’s what they do require: the willingness to act before you’re told to.

Write a test this week. Add a column description tomorrow. Build a freshness check and schedule it. Create a Slack channel and post the first issue. Turn one of your quality checks into a reusable example your team can copy.

DORA’s research — surveying 39,000+ professionals — finds that with focused effort and the right tooling, teams typically see measurable improvements within 1–3 months. A GitHub engineer who went from junior to mid-level in 2.5 years described the compound effect: “If I got stuck on some undocumented functionality, I made sure to update the docs and let the team know. Tackling a tricky bug requiring cross-team collaboration? I’d summarise everything we discovered so that it would be easier for others later. Post about it in Microsoft Teams and highlight its impact.”

Kevin Workman, a former Google engineer, captured the mindset that makes all of this work: “Realising that I had agency and even a responsibility to advocate for my ideas is one of the most important lessons I learned on my journey to becoming a ‘senior’ software engineer… Some of the most interesting and most successful moments of my career have come from advocating for changes I thought nobody would agree to.”

I think about Sam sometimes — that four-month-old team member whose instinct was to apologise for fixing something. The tests Sam added that day caught three data quality issues in the following month. Other engineers started adding their own tests alongside model changes. Six months later, the team had 200+ automated quality checks running on every pipeline, and the weekly “data looks wrong” Microsoft Teams messages had dropped to nearly zero.

Sam never got a mandate. Never wrote a proposal. Never waited for Q3.

The data quality problem in your organisation is a $12.9 million issue wearing a “we’ll get to it in Q3” disguise. And you — yes, you, the person who’s been here five months and noticed fourteen spellings of “active” — are exactly the right person to start fixing it.

Not because someone gave you permission. Because the data doesn’t care about your title.

Your Friends Will Be There for You. Your Work Won't.

Chris Hillman — Wed, 18 Mar 2026 09:00:00 +1100

There’s a table in a rented house in Queenscliff — nothing fancy, just whatever furniture came with the place — with a hand-drawn map spread across it, dice scattered at the edges, and snacks slowly migrating toward the centre of the board.

Once a year, Steve, one of our friends, organises for us to make a trip down to Queenscliff to play tabletop RPGs for a weekend. More recently, we’ve started playing monthly or fortnightly at home too. Nothing glamorous about any of it. We’re not Glass Cannon Network or Critical Role — no cameras, no production values, no audience watching us roleplay with solemn gravity.

We’re just a bunch of friends. Laughing at the missed sword swing. Losing our minds when someone makes a completely unreasonable jump and somehow lands the killing blow on a copper golem with a natural twenty. Rolling dice and arguing about the rules and eating too much while the world outside keeps spinning without us.

I’ve been thinking about why it matters so much to me.

It’s not the game, really — or at least, the game isn’t the point. It’s what the game creates. A ritual. An excuse to sit in the same room for a weekend and actually be present with people I care about. A context where you ask “how are you going?” and actually mean it, and stay long enough to hear the real answer. The dragon we’re fighting is almost beside the point. What matters is being there for each other.

When you’re up at 2am resolving a critical data pipeline failure, your employer will not remember that next year. They might not remember it next month. The organisation absorbs it, thanks you if you’re lucky, and moves on to the next incident.

But fighting dragons with your friends — showing up for a weekend in Queenscliff year after year — those become stories. Stories that become memories. Memories that become the fabric of who you are to each other, and who they are to you.

Your friends will be there for you. Your work won’t.

And yet.

When was the last time someone you loved lost someone, and you climbed into bed next to them — not to fix it, not to say the right thing, just to be there in the dark with them? When something hard was happening in your own life, did you pick up the phone and call a friend? Or did the thought cross your mind — “they’re busy, they don’t have time to listen to me” — and you put the phone down and carried it alone?

Do you have a friend you’d call when you got the promotion — someone who’d hear the news and mean it when they say “that’s great”? Not quietly threatened by it. Not performing enthusiasm. Just genuinely proud of you, because your win is their win. That kind of friend is rarer than it should be.

Those moments. That’s what friendship actually is. Not the catch-up drinks you schedule six weeks out and cancel twice. The showing up in the hard moments. The willingness to burden someone with your struggle, and the willingness to carry theirs.

We don’t build trust by offering help. We build trust by asking for it. The act of calling someone when things are falling apart — not suffering quietly, not performing fine — is the thing that makes the friendship real on both sides. It’s not a burden. For the right person, it’s an honour.

This is something I’m still learning. I want to talk about what it actually means to be a good friend — because I think most of us, if we’re honest, have let that muscle go slack.

Are You Actually a Good Friend?

Most people, if you ask them directly, will say yes. Of course they’re a good friend. They’re there for the people they care about.

But ask yourself: do you call your friends on their birthday — actually call, sing happy birthday — or do you post on their Facebook because you saw the notification and everyone else was doing it? When a friend is going through something hard, do you show up? Not message. Show up. Do you say “I love you” to the people in your life that you love?

Most of us, if we’re being honest, have outsourced friendship to the low-effort channel. The emoji response. The “we should catch up soon!” comment thread that leads nowhere. We maintain the social graph without maintaining the relationships.

But when you’re in a dark place, who do you call? And more confronting — who would call you?

The answer to that second question is the real measure. Friendship isn’t what you feel for someone. It’s what you demonstrate over time. It’s the accumulated weight of showing up — for the birthday dinner that falls on a work night, for the hospital waiting room at 7am, for the phone call that starts “I just need to talk to someone” at an inconvenient hour.

Those moments don’t happen automatically. They have to be built. And they have to be built before you need them, not when you’re reaching for them in a crisis.

When our daughter Natasha was born, she wouldn’t latch on, and breastfeeding wasn’t working. Lena and I tried everything, exhausted and running on no sleep, seriously weighing up whether to just switch to formula. At some point we gave up trying to solve it ourselves and called our friends Tom and Rachel. We ended up talking for somewhere between four and six hours straight. We didn’t need advice, really — we needed to not be alone with it. And something about that phone call, about calming down, settled Natasha too. By the end of it she was feeding. I still think about that night. We’d been treating a human problem like a technical problem, trying to fix it in isolation, when all we’d actually needed was to pick up the phone.

What Women Already Know

Women are better at friendship than men. Not universally, not as a stereotype — but as a pattern that shows up consistently enough to be worth talking about honestly.

The research on women’s friendship patterns tells part of the story. Women’s friendships tend to orient toward direct emotional disclosure — what psychologists call face-to-face friendship. Personal sharing, vulnerability, actual conversation about what’s happening in your life. Men’s friendships tend to be shoulder-to-shoulder — bonding through shared activity, talking while doing something else together. Neither mode is better. But when the activity disappears — when the sport stops, when the shared project ends, when the workplace changes — shoulder-to-shoulder friendship has nothing to stand on. Face-to-face friendship survives the context change because the context was never the point.

What women understand, and what most men have to learn later and harder, is that the friendship is the thing. Not the activity. The person.

The good news is that the shoulder-to-shoulder model doesn’t have to disappear. The Queenscliff table is shoulder-to-shoulder. The hack is building ritual around it — making the activity a recurring container for the friendship, rather than the friendship being a side effect of the activity. When you have the ritual, the friendship survives when everything else changes.

I saw this firsthand when I was working in India. Some colleagues (Sarasa, Nidhi, Abhilash, Deepak) took me out to play badminton — I’d never played before, and I was genuinely terrible. But what struck me wasn’t the sport. It was that this group did it every week, without fail. They weren’t there to compete or keep score. They were there to be together, to unwind, to laugh at each other and at themselves. What I remember is leaving that court feeling like I’d glimpsed something. These people had built a ritual, and the ritual had built them.

What the Harvard Longevity Study Actually Found

Harvard physician Arlie Bock began following the lives of a few hundred Harvard sophomores. The study is now 85 years old, has followed over 1,300 people including the children of original participants, and is the longest-running study of adult development in history.

The fourth director of the study, psychiatrist Robert Waldinger, gave what became one of the ten most-watched TED talks ever. And the headline finding, distilled from nearly nine decades of data, is deceptively simple:

Good relationships lead to health and happiness. Not wealth. Not professional achievement. Not optimised nutrition or fitness metrics. Relationships.

Waldinger and his co-author Marc Schulz published the full findings in The Good Life in 2023. Two things from that research hit particularly hard.

First: relationship satisfaction at age 50 was a better predictor of physical health at 80 than cholesterol levels. That’s the kind of finding that should make every analytically-minded person stop and recalibrate. We spend enormous energy optimising for things we can measure while ignoring the metric that turns out to be most predictive of how we age and how long we live.

Second: when participants reached their 80s, their biggest regret was almost universally the same. Too much time at work. Not enough time with the people they loved. And their proudest achievements were almost entirely relational — being a good parent, a good friend, a good partner, a good mentor.

George Vaillant, who directed the study for over three decades, summarised it: “When the study began, nobody cared about empathy or attachment. But the key to healthy aging is relationships, relationships, relationships.”

What We Actually Traded

Reflecting on how high achievers use the word “sacrifice”. When we say we’ve sacrificed something for our career, we shouldn’t be afraid to put a name to who that sacrifice was. Because often it was the people in our lives that we call friends.

Put a name to it.

We’re not sacrificing some abstract concept. We’re sacrificing specific people. Former colleagues whose numbers we keep but never dial. School friends who live in the same city and see us once a year if they’re lucky. The people who would drop everything if we called — but who we don’t call, because we’re heads-down on the next deliverable.

So many of us have cancelled on friends because a meeting came up, telling ourselves they’ll understand. Yet the reverse almost never happens — we wouldn’t reschedule a meeting for a friend. We’ve quietly trained ourselves to treat friendship as the flexible commitment, the one that can move.

And the friendship debt is unlike technical debt — you can’t see it slowing you down, it doesn’t show up in velocity metrics or retrospectives. It just quietly accumulates until something breaks. The incident you pushed through alone. The difficult conversation at home that had nowhere to go because there were no friends to process it with. The creeping sense that despite being deeply capable and professionally respected, you’re not quite sure who you’d call.

Ubuntu: A Different Operating System

There’s a Zulu/Nguni philosophy called Ubuntu. The phrase “Umuntu ngumuntu ngabantu” translates roughly as “a person is a person through other people.” You’ll also hear it rendered as “I am because we are.”

Desmond Tutu described Ubuntu this way: “In our African worldview, we need other human beings for us to learn how to be human. For none of us comes fully formed into the world. The solitary human being is a contradiction in terms.”

He contrasted it explicitly with Descartes: “It is not ‘I think therefore I am.’ It says rather: ‘I am human because I belong. I participate. I share.’”

Most of the professional identity we build in data engineering runs on Descartes. I architect, therefore I am. I optimise the pipeline, therefore I am. I resolved the incident, therefore I am. We define ourselves by what we produce, alone, with headphones on, in a flow state that the rest of the world is just interrupting.

Ubuntu is a different operating system. One where your identity is constituted by your relationships — where the question “who are you?” is answered not by your job title or your GitHub commit history, but by who you show up for and who shows up for you.

The longest-running study of adult development in history arrived at the same conclusion Tutu was describing. Every major piece of research on social connection and mortality found it too. The architecture most of us are running on isn’t wrong in a subtle way. It’s wrong in the foundational way.

Mark Shuttleworth — South African entrepreneur — named the Ubuntu Linux operating system after this philosophy in 2004, explicitly to emphasise community, open-source contribution, and the idea that the work is better when people build it together. Engineers understood that intuitively, because it maps to how the best software actually gets built. It also maps, as it turns out, to how the best lives get built.

The Dragon Will Be There Next Year

There’s a table in a rented house in Queenscliff, with a hand-drawn map on it and dice scattered at the edges.

Every year, we make the trip. We gather around a table and roll dice and laugh at the missed sword swing and the impossible jump. The dragons change. The characters level up. The rules get argued over and amended and house-ruled into something nobody outside the group would recognise.

But the people are constant. And the ritual is the point.

It is more amazing to have an extraordinary experience with someone than by yourself. You can go somewhere alone and say “look what I did” — versus “do you remember that time we did that?”

The 2am pipeline failure you resolved will be forgotten. By you. By your employer. By the Jira ticket that gets closed and archived.

The dragon you fought with your friends — the copper golem that went down to a natural twenty on an impossible jump — that becomes a story. The story becomes a memory. The memory becomes part of who you are to each other and who they are to you.

That’s not sentimentality. That’s the mechanism by which good lives actually get built. Eighty-five years of longitudinal data says so.

Researcher Robin Dunbar found that building a close friendship takes roughly 200 hours of time together. Those hours don’t come from nowhere. They come from showing up, again and again, at the table. From protecting that time the same way you’d protect a critical production window. From understanding that the maintenance cost of friendship is the highest-ROI investment available to you as a human being.

Your work will absorb your best years and move on when it’s convenient. Your friends will still be at the table.

Put a name to what you’re protecting.

Then protect it.

Your Data Model Isn't Broken, Part I: Why Refactoring Beats Rebuilding

Chris Hillman — Sat, 14 Mar 2026 09:00:00 +1100

In the early 2000’s - Netscape’s decision to rewrite their browser from scratch was the single worst strategic mistake a software company could make.

At the time, Netscape was winning. They had the dominant browser. They had market share. They had momentum. And then they decided the codebase was too messy, too tangled, too hard to work with — so they threw it all away and started over. Navigator 4.0 became the foundation for a rewrite that would eventually ship as version 6.0. There was no 5.0. Three years of development. No shipping product. And while Netscape’s engineers were busy building their beautiful new browser in a vacuum, Internet Explorer ate their lunch, their dinner, and most of their market share.

It’s haunted me ever since: old code isn’t ugly because it’s bad. Old code is ugly because it works. Every strange condition, every seemingly redundant check, every patch that makes a new developer wince — those are battle scars. Each one represents a bug that took weeks to find in production, a customer workflow nobody anticipated, or an edge case that only surfaces on the third Tuesday of months ending in “R.”

I think about that every time I hear a data team say “let’s just rebuild it.”

The sentence that starts every failed data project

I’ve been on data teams long enough to recognise the pattern. It always starts the same way. Someone — usually someone new, often someone senior — opens a dimensional model they didn’t build, scrolls through a few hundred lines of transformation logic, and says the words that should make every data leader’s blood run cold:

“This is a mess. We need to start from scratch.”

And look, I get it. I’ve felt that impulse myself, and likely have said the words myself. You open a fact table with 200 columns. You find bridge tables that reference other bridge tables. You discover a slowly-changing dimension nested inside another slowly-changing dimension, and you think: who built this? What were they thinking?

But here’s what I’ve learned the hard way, across multiple teams and more warehouse migrations than I care to count: they were thinking about the business. That bizarre WHERE clause filtering out both “Unknown” and “unknown”? That’s not sloppy code. That’s a case-sensitivity bug someone found in production data from a source system that nobody controlled. The seemingly redundant join that adds three seconds to your query? It handles a quarterly reconciliation edge case that cost the finance team two days of manual work before someone encoded the fix.

That fact table with 200 columns isn’t a design failure. It’s an accurate representation of a business that has 200 things it needs to measure. The “clean” replacement model will eventually have 200 columns too — they’ll just have different names and it’ll take you eighteen months to figure out why you need them all.

What data teams keep forgetting

This isn’t about browsers or even about code quality. It was about knowledge. When you throw away a codebase and start fresh, you’re not just discarding syntax. You’re discarding years of accumulated understanding about how the real world actually behaves — understanding that was earned through production incidents, user complaints, and painful debugging sessions.

Every collected bug fix in that old code represents something learned. Each fix might be just one line, a couple of characters even, but a lot of work and time went into figuring out those two characters were needed. And that knowledge — the why behind the fix — almost never makes it into documentation. It lives in the code itself, or it lives nowhere.

This is doubly true for data systems. A software application has unit tests, integration tests, user acceptance testing. A data model has… what exactly? Row counts? Spot checks? A business user who eyeballs the dashboard and says “yeah, that looks about right”? The knowledge embedded in a mature data warehouse is far more fragile than application code, because the testing infrastructure around it is almost always weaker. Throw it away and you’re not just rebuilding a codebase — you’re rebuilding an institutional memory that was never written down in the first place.

Fred Brooks saw this coming fifty years ago. His “Second System Effect” from The Mythical Man-Month describes what happens when an engineer builds their second version of something: they over-design it. All the features they wisely deferred from the first version, all the architectural improvements they dreamed about, all the “if only we’d done it this way” ideas — they dump everything into the replacement. Brooks noted that a designer’s first system tends to be spare and clean because they know their limitations. The second system becomes a dumping ground for ambition.

Sound familiar? Every data warehouse rebuild I’ve witnessed follows this arc. The team doesn’t just want to replicate what exists — they want to add a semantic layer, implement a medallion architecture, introduce data contracts, add real-time streaming, switch to Data Vault, build a self-serve analytics platform, and migrate to a new cloud provider. All at once. In a single initiative.

And then they wonder why, two years later, they’ve delivered nothing and the business is still running reports off the “legacy” warehouse that was supposed to be decommissioned eighteen months ago.

The Teradata-to-Snowflake reality check

If you want to see the rebuild-vs-refactor debate play out in real time, watch a Teradata-to-Snowflake migration. The pattern is remarkably consistent.

The pitch is compelling: Snowflake is cheaper, scales elastically, separates compute from storage, and runs standard SQL. Moving from Teradata should be straightforward — convert the SQL, move the data, validate the results. Easy, right?

Roland Wenzlofsky, a Snowflake Solutions Architect, wrote about this in early 2026 and his observations line up perfectly with what I’ve seen. Most organisations approach these migrations as a translation exercise. The technical migration succeeds. Then the first quarterly cloud bill arrives at double the projected budget. Dashboards that loaded in seconds now queue for twenty minutes during batch windows. Pipelines that ran in 45 minutes on Teradata take five hours on Snowflake.

The problem isn’t Snowflake. The problem is that Teradata professionals carry assumptions into Snowflake that aren’t just incomplete — they’re actively counterproductive. Distribution keys, join strategies, indexing patterns — all the hard-won optimisation knowledge from Teradata becomes a liability in a platform built on fundamentally different architecture.

One of the largest Teradata-to-Snowflake migrations in North America involved 1.5 petabytes across 600 databases and roughly 45,000 objects. Before they could even begin, the team had to inventory every orphan object and document every behavioural difference between platforms. That’s not a “lift and shift.” That’s an archaeological dig.

But here’s what I find most telling: even when the migration succeeds technically, the data quality problems survive the move perfectly intact. William Flaiz documented a healthcare organisation that spent $1.8 million migrating from Siebel to Salesforce. Not a single record lost. Zero downtime. Perfect data mapping. Post-migration, the sales team still couldn’t run accurate pipeline reports. The “opportunity stage” field contained 89 different values for what should have been six standard stages — including 1,247 records with the typo “Closeing Soon.” That typo migrated perfectly to the new $2.3 million platform. As Flaiz put it: when performance is bad, we assume the technology is the limiting factor. Data quality problems are messier. They implicate people, processes, training gaps, and years of accumulated shortcuts.

New platform, same chaos. Because the platform was never the problem.

Chesterton’s Fence, or: why you shouldn’t delete that WHERE clause

There’s a principle in philosophy that every data engineer should have tattooed somewhere visible. G.K. Chesterton wrote it in 1929, and it goes roughly like this: if you come across a fence in the middle of a field and can’t see what purpose it serves, don’t tear it down. Go away and figure out why someone built it. Once you understand the reason, then you can decide whether it still needs to be there.

In data engineering, the fences are everywhere. That WHERE status NOT IN ('Unknown', 'unknown', 'UNKNOWN') clause. That filter excluding records from a specific date range in 2019. The join to a reference table that only has 12 rows and hasn’t been updated in three years. They all look pointless until you remove one and discover that the finance reconciliation breaks, or that a regulatory report starts including test transactions that were supposed to be filtered out, or that a dashboard starts showing a revenue spike from a data migration artifact that happened four years ago.

Hyrum Wright — formerly at Google, now at Adobe — formalised a related idea that’s become known as Hyrum’s Law: with enough users of an API, every observable behaviour will be depended on by somebody, regardless of what the documentation promises. For data systems, this is an absolute nightmare during rebuilds. Your downstream consumers don’t just depend on documented schemas. They depend on output ordering, null handling patterns, timestamp precision, and format quirks that were never specified anywhere. You rebuild the pipeline, and suddenly a report that’s worked for three years breaks — not because the data is wrong, but because the columns come back in a different order and someone’s Excel macro was hardcoded to column positions.

I’ve seen teams spend more time debugging these “invisible contracts” after a rebuild than they would have spent refactoring the original system over two years.

The psychology that makes us do it anyway

So if the evidence against big-bang rewrites is this overwhelming — and it is, from Spolsky to Brooks to McKinsey studies showing that large IT projects run 45% over budget and deliver 56% less value than predicted — why do smart, experienced data teams keep proposing them?

Because the impulse to rebuild isn’t rational. It’s psychological. And once you understand the cognitive biases at play, you start seeing them everywhere.

The planning fallacy is the first culprit. Kahneman and Tversky showed that humans systematically underestimate how long tasks will take, even when they have direct experience with similar tasks that took longer than expected. In controlled studies, only 13% of subjects finished their project by the time they’d assigned a 50% probability of completion. The Sydney Opera House was estimated at AU$7 million and four years; it was delivered at AU$102 million and fourteen years. Every data warehouse rebuild proposal I’ve seen exhibits this same delusional optimism. “Six months, maybe nine.” It’s never six months.

And it gets worse, because of what one writer calls the Achilles Paradox of rewrites: while you’re building the new version, features keep getting added to the old one. The business doesn’t stop generating requirements just because you’ve decided to rebuild. So the target keeps moving. The new system is perpetually six months from matching the functionality of the old one.

Not Invented Here syndrome is the second bias, and it’s the one that nobody wants to admit. Katz and Allen’s research on R&D project groups found that teams with stable composition develop a decreasing ability to absorb ideas from outside the group over time. They start believing they possess a monopoly on knowledge in their domain. The diagnostic sign? What researchers call “thought-terminating clichés” — phrases like “we already tried that” or “our situation is different.”

In data teams, NIH manifests as building custom orchestration frameworks instead of using Airflow. Writing bespoke quality checks instead of adopting Great Expectations or dbt tests. Designing proprietary transformation layers instead of using tools that thousands of other teams have battle-tested. And most relevantly: insisting that the existing data model is fundamentally broken and needs to be replaced with something designed in-house from the ground up.

The new leader rewrite is the third pattern, and it might be the most destructive. A new head of data arrives. They look at the legacy warehouse they’ve inherited. They don’t understand the history behind any of the design decisions. They don’t know about the regulatory edge case that explains the weird date filter, or the source system quirk that necessitates the redundant join. All they see is complexity that they didn’t create — and complexity you didn’t create always looks worse than complexity you did.

So they propose a rebuild. It’s partly strategic — they want to put their stamp on the architecture. And it’s partly genuine — they really do think they can do better. But they’re falling prey to the same illusion Spolsky identified: reading code is harder than writing it, and unfamiliar code always looks worse than it is.

There’s a Stack Overflow survey finding that sits underneath all of this: “feeling unproductive” was the number one cause of developer unhappiness at 45%. Working with legacy systems is slow. It’s frustrating. Progress is incremental and often invisible. A greenfield rebuild, by contrast, feels amazing — for the first few months. You’re making decisions, building things, moving fast. The architecture is clean and the tests pass and the world makes sense. And then reality seeps in, the edge cases accumulate, and eighteen months later you’ve got a codebase that looks suspiciously like the one you replaced.

The complexity was always essential

Fred Brooks drew a distinction that I think about constantly. He separated essential complexity — complexity that’s inherent to the problem domain and can’t be removed by better engineering — from accidental complexity, which comes from poor implementation choices and can theoretically be eliminated.

The truth about mature data warehouses: most of the complexity is essential. It’s not there because the original engineers were incompetent. It’s there because the business is genuinely that complicated.

The transformation handling fifteen date formats? That’s because you have fifteen source systems and none of them agree on how to represent a date. The slowly-changing dimension with Type 2 tracking on attributes that seem trivial? Someone in compliance needed audit trails on those fields after a regulatory inquiry. The bridge table that connects customers to accounts through an intermediate entity that nobody can explain? It models a many-to-many relationship that emerged when the company acquired a subsidiary with a different customer hierarchy.

A rebuild won’t make this complexity disappear. It’ll just redistribute it. Instead of one tangled fact table, you’ll have twelve microservices each handling a piece of the logic. Instead of one confusing WHERE clause, you’ll have business rules scattered across a semantic layer, a transformation layer, and a data quality framework. The total complexity will be identical — or worse, because now it’s distributed across more systems with more failure modes.

Ward Cunningham — the person who coined the term “technical debt” — has said he wishes he’d used the word “opportunity” instead. His original metaphor wasn’t about sloppy code at all. It was about the gap between your code’s current model of the problem domain and your team’s evolved understanding of that domain. Technical debt, properly understood, is learning that hasn’t been applied yet. The legacy data model doesn’t represent failure. It represents the team’s best understanding at the time it was built, plus every correction that production reality demanded afterward.

Refactoring respects this. It asks: which parts of this complexity are essential (keep them, clarify them, test them) and which parts are accidental (remove them incrementally, one safe step at a time)? It preserves institutional knowledge while improving structure. It delivers value continuously instead of asking the business to wait years for a payoff that — statistically, based on everything we know about large-scale IT projects — probably won’t arrive.

So what do you do instead?

I’m going to save the detailed playbook for Part II of this series, but the short version is this: you refactor. Methodically. Incrementally. With tests.

Martin Fowler and Pramod Sadalage proved that database refactoring is possible back in 2006, when conventional wisdom said it couldn’t be done. Their principle was simple: make each change as small as possible, because the pain of integration increases exponentially with the size of the integration.

dbt has turned this into a practical workflow for analytics engineering teams. Bring your legacy SQL in unchanged. Wrap it in a model. Verify the output matches. Then — and only then — start decomposing. Extract CTEs. Introduce staging layers. Build tests. Refactor one model at a time, auditing each change against the original output. It’s not glamorous. It doesn’t let you redesign everything from first principles. But it works, and it works without asking the business to lose access to their reports for six months.

The Strangler Fig pattern, the Write-Audit-Publish pattern, the Expand and Contract pattern — these are all variations on the same theme: replace incrementally, verify continuously, and never throw away working logic until you’ve proved the replacement handles every case the original did.

I’ll dig into each of these in Part II. For now, the point is simpler.

The scars are data

When I look at a legacy data model — really look at it, with patience and without ego — I don’t see a mess anymore. I see a record of everything a business has learned about itself. Every weird join is a relationship somebody fought to understand. Every cryptic transformation is a business rule that was discovered through painful experience. Every seemingly arbitrary filter is a production incident that someone fixed at some point, probably at an hour they’d rather not remember.

The Netscape rewrite remains one of the most studied failures in software history. And yet here we are, in 2026, watching data teams propose the same mistake — just with different technology names. Teradata becomes Snowflake. Informatica becomes dbt. The on-prem warehouse becomes the cloud lakehouse. The pitch changes, but the impulse doesn’t: tear it down, start over, do it right this time.

My view? Your data model isn’t broken. It’s battle-tested. It’s messy because reality is messy. And the right response to inherited complexity isn’t demolition — it’s archaeology. Understand what’s there. Understand why it’s there. Then improve it, one careful change at a time.

The scars in your data model aren’t flaws. They’re knowledge. Treat them accordingly.

12 Steps to Better Data Engineering

Chris Hillman — Sat, 07 Mar 2026 09:00:00 +1100

Let me tell you about the moment I stopped trusting architecture diagrams.

I was three days into a new role, getting up to speed with the data team. Smart people. Modern stack. On paper, everything looked right. They walked me through a beautiful data platform diagram: clean lines, labelled layers, colour-coded domains. It looked like something you’d see in a data conference.

Then I asked a question that changed everything: “Can you rebuild your finance table from scratch right now?”

The room went quiet. One person started explaining that it was “mostly possible” but there were “a few manual steps” and “some seeds that someone uploads”. Someone else mentioned an incremental model that “hasn’t been full-refreshed in about eight months because last time it broke.”

That gap — between the diagram on the wall and the reality in the warehouse — is where I’ve spent most of my career. And over the years, working across team after team, I started noticing something. The teams that were struggling weren’t missing knowledge. They had smart engineers who knew about testing, CI/CD, documentation, data contracts. They’d read the blog posts. They’d watched the conference talks. What they didn’t have was a way to honestly assess where they stood and what to fix first.

I kept wishing someone would build a simple diagnostic. Maybe they just needed a handful of honest questions that would tell you, in ten minutes, whether your team was engineering or firefighting.

The moment that turned wishing into doing came about six months later, with a team I was leading. We implemented just two practices — CI testing on pull requests and mandatory code review — and I watched the compound effect over twelve weeks. Our incidents dropped. The analysts stopped asking “which table do I use?” A new hire shipped a model on day four. One of the senior engineers pulled me aside and said, “I wish we’d known how bad things were before you got here. We thought we were fine because nothing was on fire.”

Nothing was on fire. But everything was smouldering.

That’s why I built this test. Not because the world needs another framework — it doesn’t. But because the gap between thinking you’re fine and knowing where you stand is where data teams lose months of momentum. And the only way to close that gap is to ask questions honest enough that the answers sting a little.

The twelve questions

Before we get into the detail, here they are. Score yourself. One point for each “yes.”

Can you rebuild any table from raw data in one command?
Do you have a data catalog that people actually use?
Can a new analyst find the data they need without asking you?
Do you test transformations before deploying them?
Do you fix data quality issues before building new pipelines?
Do you have SLAs for your critical tables?
Do you have a single source of truth for business definitions?
Can you explain the lineage of any metric in under 2 minutes?
Do data producers know when they break downstream consumers?
Do you do code review on SQL and dbt models?
Do new hires build a real pipeline in their first week?
Do you regularly talk to the people who actually use your data?

A score of 12 is unicorn territory. I’ve never seen it in the wild. Most teams I’ve worked with score 3 or 4. Anything below 6 and you’re not doing data engineering — you’re doing data triage.

Now let’s break each one down. For every question, I’ll show you what good looks like and what amazing looks like — with concrete implementations using dbt, Snowflake, GitHub Actions, and AWS. Because “yes” isn’t binary in practice. There’s a massive gap between “sort of” and “absolutely.”

1. Can you rebuild any table from raw data in one command?

This is the foundation. If you can’t reproduce your outputs from your inputs, you don’t have a pipeline — you have a prayer.

The core principle here is idempotency: running the same operation twice produces identical results. dbt is designed around this idea, but achieving it requires deliberate choices, especially with incremental models. Too many teams build incremental models that silently drift from what a full refresh would produce, and they don’t discover the discrepancy until something breaks spectacularly.

What good looks like

Individual tables rebuild cleanly with dbt run --full-refresh --select model_name. Your incremental models are tested against full refreshes during development. You’ve configured on_schema_change: sync_all_columns so schema evolution doesn’t silently break things. And you’ve got a CI pipeline in GitHub Actions that runs dbt build on pull requests.

That’s good. That’s better than most. But it’s still table-by-table.

What amazing looks like

Your entire warehouse rebuilds from raw data using a Write-Audit-Publish (WAP) pattern. If you’re not familiar with WAP, I wrote a deep dive on implementing it with Airflow — the core idea is that new data gets written to a staging environment, audited against quality checks, and only published to production once it passes. It’s the data engineering equivalent of a preflight checklist: nothing reaches consumers until it’s been verified. And if you’re on Snowflake with Iceberg tables, the game has changed — WAP with Iceberg branching gives you git-like isolation at the table level, which means you can stage, audit, and publish without creating separate schemas or databases at all.

The practical implementation: dbt always targets a staging schema or database (or an Iceberg branch), runs all models and tests, and only on success does the data get promoted to production. Combined with Snowflake’s zero-copy cloning, this becomes practical even at scale — cloning a multi-terabyte database takes seconds and costs nothing in storage. You’re not duplicating data; you’re creating metadata pointers.

And when things do go wrong — and they will — the question becomes how quickly you can recover. If your incremental models have drifted, or a source system silently changed its schema three weeks ago, you need a backfill strategy that doesn’t involve rebuilding everything from scratch. I wrote about the pitfalls of day-by-day backfills and how to heal tables properly — it’s the companion piece to idempotency, because reproducibility isn’t just about building forward. It’s about being able to go back.

The really mature teams take it further with Slim CI. Instead of rebuilding everything on every PR, they store the production manifest.json in S3 and compare it against the new build. Only modified models and their downstream dependencies get rebuilt:

# .github/workflows/dbt-ci.yml
on:
 pull_request:
 types: [opened, reopened, synchronize]

steps:
 - name: Download production manifest
 run: aws s3 cp s3://dbt-artifacts/manifest.json ./state/manifest.json

 - name: Run dbt build (modified models only)
 run: dbt build -s 'state:modified+' --defer --state ./state --target ci

One team I worked with reported a 30% reduction in compute costs just by splitting their monolithic CI job into targeted workflows.

Watch out for this: Protect massive tables with full_refresh = false in your dbt config. One accidental --full-refresh on a billion-row fact table can ruin your morning. Override it with a variable when you actually mean it: full_refresh = var("force_full_refresh", false).

2. Do you have a data catalog that people actually use?

The emphasis is on “actually use.” I’ve lost count of how many teams have shown me a data catalog, only for me to discover that nobody’s opened it in months. The data catalog is the gym membership of the data world — everyone has one, nobody goes.

The failure mode is almost always the same: a team selects a tool, loads metadata, declares victory, and moves on. Six months later it’s a ghost town of stale descriptions and orphaned tables.

What good looks like

dbt Docs generated and hosted somewhere accessible. Your major tables have descriptions in schema.yml. The lineage graph exists and gets pulled up occasionally during incident response or onboarding. It’s not perfect, but it’s there.

What amazing looks like

The catalog is the default entry point for data questions — not Microsoft Teams, not the data engineer sitting three desks over.

Spotify’s internal tool Lexikon pushed data scientist adoption from 75% to 95% by doing something clever: they added personalised dataset recommendations, people/team pages showing who uses each dataset, common joins, and popular fields. It became a top-five internal tool because it was genuinely faster than asking a colleague.

Airbnb’s Metis serves over 1,000 data users weekly. Their secret? A Google-like search interface that works for every skill level, from SQL-fluent engineers to product managers who’ve never written a query.

The pattern that actually works for smaller teams: author descriptions in whatever UI is easiest, auto-generate a PR back to dbt’s schema.yml daily, review in Git. This keeps your definitions version-controlled without making people learn Git just to document a table.

And here’s the leading indicator that your catalog is working: track the decline in Microsoft Teams questions about data. Productboard used exactly this metric. Fewer “which table do I use for revenue?” messages meant the catalog was earning its keep.

3. Can a new analyst find the data they need without asking you?

This is the catalog question’s evil twin. A catalog helps, but self-service discovery is really about the cumulative effect of naming conventions, documentation-as-code, semantic layers, and project structure.

dbt Labs published guidance that should be printed and taped to every data engineer’s monitor: “Assume your end-user will have no other context than the model name.” Model names persist across databases, BI tools, DAGs, and docs. Folder names and schemas don’t follow the data the same way.

What good looks like

You’re using the canonical dbt naming convention: stg_ for staging, int_ for intermediate, fct_ for facts, dim_ for dimensions. Your project follows the three-layer structure — staging (1:1 with sources, materialised as views), intermediate (composable business logic), and marts (final business-ready datasets organised by domain). A reasonably technical person can navigate the DAG and find what they need within ten minutes.

What amazing looks like

A new analyst finds and understands any dataset within minutes, without messaging anyone.

Every model has both table-level and column-level descriptions. Schema field search works across all tables — “find every table with customer_id” returns results instantly. Personalised dataset recommendations surface relevant tables based on role.

And the crown jewel: a semantic layer that lets business users query metrics in plain language without writing SQL. The dbt Semantic Layer (powered by MetricFlow) centralises metric definitions in YAML alongside your models — entities, dimensions, and measures declared once, then consumed via API in Tableau, Power BI, Looker, Python notebooks, and increasingly, AI agents.

Bilt Rewards reported an 80% decrease in data costs after implementing it for embedded analytics. And dbt Labs’ testing showed 83% of natural-language questions answered correctly when routed through the semantic layer. That last number matters more than you’d think — it’s the difference between AI tooling that actually works and AI tooling that confidently gives the wrong answer.

4. Do you test transformations before deploying them?

This is probably the single highest-impact practice on the list. If I could only get a team to adopt one of these twelve, it would be this one.

But here’s what most teams get wrong: they think “testing” means slapping not_null on a few columns and calling it done. Real testing means understanding the dimensions of data quality — completeness, uniqueness, timeliness, validity, accuracy, consistency — and building checks that cover each one deliberately. I’ve written about why data quality is a deeper problem than most teams realise, and the short version is this: testing in CI is where quality becomes a habit rather than a hope. But only if your tests are actually measuring the things that matter.

The consensus across every practitioner I’ve studied is clear: CI for dbt is the most impactful thing you can do for data quality. Three complementary testing layers form a comprehensive strategy: generic dbt tests (not_null, unique, accepted_values, relationships), unit tests (introduced in dbt v1.8 for testing complex business logic with static inputs), and data diffing for value-level comparison between production and development.

What good looks like

Basic dbt tests on key models. Tests run as part of scheduled production jobs using dbt build (which runs tests immediately after each model, not in a separate pass). Pull requests exist for dbt changes, and someone eyeballs them before merging.

What amazing looks like

Full Slim CI with per-PR isolated Snowflake schemas, data diffing integrated into PR comments, and automated linting that catches style issues before a human ever looks at the code.

The numbers from real teams are staggering. Thumbtack — 50-plus analysts, five data engineers, over 100 PRs per month — previously spent one to two hours manually validating each pull request with SQL queries and spreadsheets. After integrating data diffing into their GitHub CI pipeline, they saved over 200 hours per month. That’s not a rounding error. That’s a full-time engineer’s worth of capacity recovered.

Dutchie caught timezone corruption on created_at fields and case-when logic errors that were silently shifting 20% of data between columns. Nutrafol caught a transformation that would have shown net revenue plummeting — before it reached production and before the CFO’s Monday morning dashboard refreshed.

Zscaler went even further. They built PRISM, a multi-agent AI PR review system that reduced manual review time by 90% — auto-approving conformant PRs and posting targeted comments on complex logic changes.

Here’s what the mature pipeline looks like in practice:

# On every PR: lint, compile, build modified models, diff against prod
steps:
 - name: SQLFluff lint
 run: sqlfluff lint models/ --dialect snowflake

 - name: dbt compile
 run: dbt compile

 - name: Slim CI build
 run: dbt build -s 'state:modified+' --defer --state ./state --target ci

 - name: Data Diff
 run: datafold ci submit --ci-config-id ${{ secrets.DATAFOLD_CI_CONFIG }}

5. Do you fix data quality issues before building new pipelines?

Here’s an uncomfortable stat: Monte Carlo’s 2023 State of Data Quality report found that 74% of organisations reported that business stakeholders identify data quality issues first — up from 47% the prior year. Most data teams learn about broken data from the people who are supposed to trust it. That’s not a technology problem. That’s a culture problem.

What good looks like

Basic dbt tests catch obvious issues. Tests run in production jobs. The team has an informal sense of which data matters most, usually earned through painful experience — someone got burned by bad revenue numbers in an executive review, and now those tables have tests.

What amazing looks like

Data quality SLAs enforced with error budgets, borrowed from the SRE playbook. A 99.5% data availability target allows approximately 3.6 hours of acceptable downtime per month. When the error budget is consumed, reliability work takes priority over feature delivery. Full stop. No new pipelines until you’ve earned back your quality margin.

Warner Bros. Discovery created something they call a Data Quality Forum — not a reactive incident-review meeting, but an operational bridge between teams. They used observability tooling to surface anomalies early, developed a priority matrix (P0/P1/P2), and made the forum’s patterns part of new hire onboarding. During Olympics livestreaming, custom SQL checks detected missing content metadata before it could break reporting.

Snowflake now offers native Data Metric Functions — FRESHNESS, NULL_COUNT, DUPLICATE_COUNT, UNIQUE_COUNT, ROW_COUNT — that can be scheduled to run on DML changes or time intervals. Results land in SNOWFLAKE.LOCAL.DATA_QUALITY_MONITORING_RESULTS. It’s not a replacement for dbt tests, but it provides a warehouse-level safety net that catches issues your transformation layer might miss.

The cultural shift matters more than the tooling. HelloFresh’s VP of Data drove a transformation through three stages: ad-hoc individual fixes, organised cleanup by a central team, and finally proactive quality at the source. The final stage required embedding data product owners within business domains and running data literacy programs. It’s not glamorous work. But it’s the work that changes outcomes.

6. Do you have SLAs for your critical tables?

“The dashboard is stale” is the Slack message that launches a thousand fire drills. SLAs turn that reactive scramble into a measured, prioritised response.

The breakthrough insight from practitioners: don’t aim for 100%. A 99.5% target gives you a realistic buffer for maintenance, edge cases, and the occasional Snowflake service hiccup. Different data products warrant different targets — which means you need a tiered classification system.

What good looks like

Your team knows which tables are critical, usually because they’ve been burned before. Some dbt source freshness checks are running. Basic Slack alerts fire when jobs fail. It’s reactive, but at least you know when things break.

What amazing looks like

A formal tiered classification published in the data catalog with specific, measurable commitments:

Tier 1 (Gold): ML systems, revenue reporting → PagerDuty on-call, must have unique + not_null tests + assigned owner, freshness SLA under 4 hours
Tier 2 (Silver): Executive dashboards, KPI reports → Slack team channel alerts, owner assigned, freshness SLA under 12 hours
Tier 3 (Bronze): Ad-hoc analytics, exploration tables → weekly digest, freshness SLA under 24 hours

The implementation lives in dbt meta config:

models:
 - name: fct_revenue
 meta:
 owner: "finance-data-team"
 criticality: "tier_1"
 sla_freshness: "4 hours"
 tests:
 - unique:
 column_name: order_id
 - not_null:
 column_name: order_id

SLA compliance dashboards track attainment percentage, breach counts, and trends over time. When the error budget is exhausted, feature development freezes — same principle as practice #5. Dedicated Snowflake warehouses isolate critical workloads from exploratory queries so that someone’s SELECT * on a billion-row table doesn’t cause your revenue report to miss its 8 AM deadline.

7. Do you have a single source of truth for business definitions?

Everyone’s had this conversation: “Which revenue number is right?” Two dashboards, two numbers, two teams who each think the other is wrong. This is what happens when metric definitions live inside BI tools, tribal knowledge, and the head of that one analyst who’s been here since 2019. If you’ve ever worked with dimensional models, you’ll recognise this as the conformed dimension problem — I covered why dimensional modeling still matters and how conformed dimensions are the integration backbone that prevents exactly this kind of mess.

What good looks like

Key metrics are defined in dbt docs or a wiki. The team knows which mart table is the canonical source for revenue, orders, or whatever your core entities are. When someone asks “which table do I use?”, there’s a consistent answer — even if it’s only communicated verbally.

What amazing looks like

A centralised semantic layer enforced as the only path to metrics. All definitions are version-controlled in dbt YAML, reviewed via PR, and validated in CI.

Here’s what a metric definition looks like in MetricFlow:

metrics:
 - name: cancellation_rate
 description: "Percentage of orders cancelled within 24 hours of placement"
 type: ratio
 type_params:
 numerator: cancellations
 denominator: order_total
 filter: |
 {{ TimeDimension('metric_time', 'day') }}

That definition gets consumed identically whether someone queries it from Tableau, Power BI, a Python notebook, or an AI agent. One definition. One answer. Everywhere.

Whatnot — the livestream marketplace growing at breakneck speed — solved a different version of this problem. They used Protobuf schemas as a single source of truth for event definitions, consolidating hundreds of chaotic Snowflake tables down to two clean “exposure” tables: backend_events and frontend_events. Brutal simplification. But it worked because everyone could find the data.

dbt sl validate in CI catches breaking changes to semantic definitions before merge. That’s the key differentiator — your metric definitions aren’t just documented, they’re enforced.

8. Can you explain the lineage of any metric in under 2 minutes?

When your CFO asks “where does this number come from?”, you need an answer faster than “let me check and get back to you.” Two minutes is generous. In most incident situations, you’ve got about thirty seconds before people start making assumptions.

What good looks like

The dbt DAG is viewable, hosted on S3 or an internal site. Your team can trace major mart tables back to their sources. During incident response, someone pulls up the lineage graph and walks through the chain. It takes some squinting, but the information is there.

What amazing looks like

Column-level lineage automated across the full stack — from source systems through dbt transformations to the BI dashboards consumers see.

dbt Cloud provides column-level lineage showing whether each column is “transformed” versus “passthrough/rename.” On the Snowflake side, the ACCESS_HISTORY view (Enterprise Edition) tracks both read and write operations with column-level mappings. Snowflake Labs published an open-source adapter that converts ACCESS_HISTORY data into OpenLineage JSON format, making cross-platform lineage possible.

But here’s where it gets practical. The open-source tool Recce compares two dbt environments and produces a “Lineage Diff” in CI. It categorises changes as breaking, partial-breaking, or non-breaking. PR reviewers get an instant risk assessment: “this change affects 47 downstream models, including three Tier 1 dashboards” versus “this change is isolated to a staging model with no downstream consumers.”

PRs get annotated automatically with impact analysis. Cross-project lineage via dbt Mesh connects multiple dbt projects. Lineage isn’t a pretty graph that nobody looks at — it’s used daily for onboarding, incident response, impact analysis, and cost optimisation.

9. Do data producers know when they break downstream consumers?

This is where data engineering starts borrowing seriously from software engineering. In a microservices world, you wouldn’t deploy an API change without knowing who calls your endpoint. But in data, teams routinely change source schemas, modify column semantics, or sunset tables with zero awareness of who’s consuming them downstream.

Data contracts change that dynamic.

What good looks like

dbt sources are defined with dbt source freshness running and alerting. The team is aware of major upstream dependencies. When Fivetran’s sync breaks or a source system changes its schema, someone notices within a few hours — usually because a test fails.

What amazing looks like

dbt model contracts (v1.5+) enforced on all gold-layer and public models:

models:
 - name: dim_users
 config:
 contract:
 enforced: true
 columns:
 - name: user_id
 data_type: int
 constraints:
 - type: primary_key
 - name: email
 data_type: varchar
 constraints:
 - type: not_null

This is a preflight check: dbt verifies the model’s compiled SQL returns columns matching the contract before building. Breaking changes — removed columns, changed data types, modified constraints — are caught by state:modified in CI. Model access levels (public, protected, private) control cross-project visibility.

Whatnot went further with Protobuf schemas enforced via Buf linter in CI. Every event producer runs against a common testing harness. Post-deployment monitoring catches semantic drift — when a field still exists but its meaning has changed.

Gotcha worth knowing: dbt contracts validate the compiled SQL, not the actual Snowflake table. If tools like Fivetran add columns directly to your raw layer, dbt won’t know until the next run. Monitor schema changes externally using Snowflake’s INFORMATION_SCHEMA or ACCESS_HISTORY to catch drift between runs.

10. Do you do code review on SQL and dbt models?

I’m constantly surprised by how many data teams still deploy SQL changes without review. In software engineering, unreviewed code going to production would be considered reckless. In data engineering, it’s Tuesday.

What good looks like

Pull requests are required for all dbt changes — branch protection is enabled on main. At least one human reviewer looks at each PR. There’s a basic PR description explaining what changed and why.

What amazing looks like

Automated CI running SQLFluff lint + dbt compile + dbt build + data diff on every PR. Pre-commit hooks catch issues before code is even committed. PR templates with structured sections: description, linked tickets, impact zone, testing evidence.

The recommended pre-commit stack combines SQLFluff for linting with dbt-checkpoint for structural validation:

# .pre-commit-config.yaml
repos:
 - repo: https://github.com/sqlfluff/sqlfluff
 hooks:
 - id: sqlfluff-lint
 args: [--dialect, snowflake]

 - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
 hooks:
 - id: check-model-has-tests
 - id: check-model-columns-have-desc
 - id: check-source-has-freshness

That last hook — check-model-columns-have-desc — is quietly revolutionary. It means documentation isn’t optional. You literally cannot merge a model without column descriptions. The “we’ll document it later” excuse dies right there in the CI pipeline.

Surfline (700+ dbt models) reported that after integrating SQLFluff, SQL consistency improved dramatically and reviewer burden dropped. New engineers learned “good SQL” from day one because the linter enforced it automatically. Another team, Markerr, found that automated style enforcement freed up review time to focus on deeper logic questions rather than arguing about capitalisation and trailing commas.

The dbt_project_evaluator package is worth mentioning too — it audits your entire DAG structure against dbt Labs’ published best practices. Run it in CI and it’ll flag things like models that reference sources directly (bypassing staging), duplicate sources, or models with no tests.

11. Do new hires build a real pipeline in their first week?

The speed at which a new hire becomes productive tells you everything about the state of your documentation, tooling, and team culture. If it takes three weeks before someone can make a meaningful contribution, you don’t have an onboarding problem — you have a platform problem. I wrote a guide to navigating your first 90 days as a data engineer — and the teams that make those first 90 days count are almost always the ones who’ve invested in the infrastructure described here.

What good looks like

New hire has access to tools within a day or two. Some documentation exists. A buddy or mentor is assigned. They can run the dbt project locally by end of week one, even if they haven’t contributed anything yet.

What amazing looks like

Pre-arrival setup is complete before Day 1: machine provisioned, logins ready, Snowflake roles assigned, dbt Cloud account active. No engineer should spend their first morning installing things.

Each developer gets a personal Snowflake sandbox — dbt_<username> schema in the dev database, with write access only to dev and read-only access to raw/staging. Snowflake’s zero-copy cloning makes production data available instantly without duplicating storage costs. Productboard open-sourced dbt-snowflake-sandbox — a set of dbt macros that create isolated sandboxes by cloning only the specific model dependencies a developer needs.

The first-week project is structured and progressive:

Day 1-2: Run the existing dbt project locally. Explore the DAG. Read the key model documentation.
Day 3: Add a new source or staging model for a real (but low-risk) dataset.
Day 4: Build a staging model with tests and documentation.
Day 5: Open a PR, go through the review process, and get it merged.

By Friday, they’ve shipped something real. They understand the workflow. They’ve experienced CI, code review, and deployment. And critically, they feel like a contributor rather than a tourist.

7shifts estimated cutting onboarding time by over a week per new hire just by having documentation and data questions accessible in their data catalog.

12. Do you regularly talk to the people who actually use your data?

This is the question that separates data teams who build for their portfolio from data teams who build for their business. You can nail every technical practice on this list and still fail if you’re building the wrong things. I’ve written before about how to maximise your data team’s impact — and the through-line is always the same: the teams that create real value are the ones who stay close to the people consuming their work.

What good looks like

A Slack channel exists for data questions. Communication happens when issues arise. There are occasional stakeholder meetings, usually prompted by something breaking or a new request coming in.

What amazing looks like

Data office hours. Weekly or bi-weekly open sessions where anyone in the organisation can bring data questions. Not a presentation. Not a status update. An open door.

Holistics published a detailed playbook for running what they call “data clinics.” The core principles: teach, don’t serve. Show business users how to self-serve rather than just answering their question and sending them on their way. Montreal Analytics reported that after implementing data clinics, the number of self-serve business users on their BI tool grew tenfold. Tenfold.

Pair that with a monthly Data NPS survey — a single question: “How likely are you to recommend our data team’s products to a colleague?” It sounds corporate, but without this metric, you have no way to quantify whether your consumers are satisfied or just silently building workarounds in Excel.

dbt exposures formalise the connection between data models and their consumers:

exposures:
 - name: weekly_revenue_dashboard
 type: dashboard
 owner:
 name: Finance Team
 email: finance@company.com
 depends_on:
 - ref('fct_revenue')
 - ref('dim_date')

When a model changes, the PR shows which exposures are affected. Stakeholder impact becomes visible in code review. You can proactively notify the finance team that their revenue dashboard’s upstream model is changing before they discover it themselves.

Frame your communication in business terms. Not “model precision is up 12%” but “the sales team can now identify high-value leads 40% faster.” Nobody outside your team cares about your DAG. They care about whether they can trust the numbers. If you’re not sure how to bridge that gap, I wrote a guide on breaking down business context — because the hardest part of talking to stakeholders isn’t the talking, it’s knowing what they actually need to hear.

Scoring your team

Here’s the scoring framework, and I want you to be ruthless with yourself:

Score	What it means
10-12	You’re in elite territory. Your data platform is a competitive advantage.
7-9	You’ve got strong foundations with clear areas to improve.
4-6	You’re functional but fragile. One bad incident away from a crisis of trust.
1-3	You’re firefighting, not engineering. The business tolerates your team — it doesn’t trust it.

Most teams land at 3-4. That’s not a criticism — it’s where the industry is. The gap between knowing these practices exist and actually implementing them is where the real work lives.

Where to start

If you’re staring at a low score and feeling overwhelmed, here’s what I’d suggest: start with practices 4 and 10. CI testing and code review.

Here’s why. Once dbt changes go through pull requests with automated checks, you have the infrastructure to enforce everything else. Contracts? They’re a CI check. Quality gates? CI check. Documentation requirements? CI check. SLA validation? CI check. You’re not adopting twelve practices — you’re building one pipeline that gates on twelve things.

The canonical architecture appears across virtually every mature team I’ve studied: GitHub Actions triggering Slim CI with dbt build -s state:modified+ against per-PR Snowflake schemas, with production manifests stored in S3 for state comparison. Start there. Layer on the remaining practices as your team’s maturity grows.

Three patterns emerged from the research that I think are worth calling out explicitly:

“As code” wins everywhere. Documentation-as-code, style-guides-as-code, permissions-as-code, quality-checks-as-code, contracts-as-code. Every manual process that gets codified becomes version-controlled, reviewable, and enforceable. Every one that stays manual eventually drifts.

Snowflake’s zero-copy cloning is the force multiplier. Isolated PR environments, sandbox onboarding, rebuild testing — all of these become cheap and instant with cloning. If you’re on Snowflake and not using this feature aggressively, you’re leaving the most powerful tool in the shed.

Culture matters more than tooling. Catalog adoption fails without meeting users in their existing workflows. Data quality requires executive sponsorship and error budgets, not just more tests. Producer-consumer contracts are first and foremost a cultural change. You can buy every tool on this list and still score a 3 if the organisation doesn’t value the practices behind them.

The diagram and the warehouse

Remember that team — the one with the beautiful diagram and the revenue table nobody could rebuild?

I caught up with one of the engineers about four months after I’d moved on. They’d started with exactly what I’d pushed for: CI testing and code review. Then they added contracts on their gold-layer models. Then SLAs on their Tier 1 tables. Then a first-week onboarding project for new hires.

Their score when I first asked these twelve questions? Three. Four months later? Eight. Not perfect. But the difference wasn’t really the number. The difference was that when they pulled up that architecture diagram now, it matched what was actually running in Snowflake. The gap between the wall and the warehouse had closed.

That’s what this test is really measuring. Not whether you have the right tools — you probably do. Not whether you know the right practices — you’re reading this, so clearly you care. It’s measuring whether there’s a gap between what you think your data platform looks like and what it actually looks like.

These twelve questions just make you say it out loud.

So print it. Score yourself honestly. Share it with your team. And the next time someone asks you how your data platform is doing, give them a number between 1 and 12.

That number is worth more than any architecture diagram.

The CSV Test Suite Nobody Writes

Chris Hillman — Wed, 04 Mar 2026 09:00:00 +1100

In October 2020, roughly 16,000 positive COVID-19 test results vanished from the UK’s public health reporting for nearly a week. Not because the tests weren’t run. Not because the labs didn’t report them. The results were collected, transmitted, and received — inside CSV files.

The problem? Public Health England was importing those CSV files into Microsoft Excel’s legacy .xls format. The format has a hard row limit of 65,536. When the files grew past that limit, Excel didn’t throw an error. It didn’t warn anyone. It just silently dropped the extra rows. Sixteen thousand people who tested positive for a deadly virus during a second wave went untraced. An estimated 50,000 of their contacts were never notified. And the system this happened in? Part of a £12 billion Test and Trace programme.

I remember reading that story and feeling something beyond frustration. I felt recognition. Because I’d seen this pattern before — not at that scale, not with those stakes — but the same fundamental failure. Someone produced a CSV. Someone else consumed it. And nobody, at any point in the chain, had written a test that asked: “What happens when this file gets bigger than we expect?”

That’s the moment I became a bit obsessive about CSV testing. Not because CSV is complicated — it’s deceptively simple. But because that simplicity breeds a dangerous confidence. Everyone thinks they know how to handle a CSV. Everyone thinks the format is too basic to need a test suite. And that assumption has cost organisations billions of dollars, corrupted scientific research, and — in the case of COVID — put actual lives at risk.

I’m telling you this because CSV testing is one of those things that sounds unnecessary until you’ve spent a weekend rebuilding a pipeline because a vendor changed their quoting convention and nobody noticed for three weeks. The engineers I’ve worked with who do this well — who write proper CSV test suites — don’t do it because they’re cautious by nature. They do it because they’ve been burned. And they decided it wouldn’t happen again.

This article is the test suite I wish someone had handed me years ago. We’re going to take RFC 4180, the closest thing CSV has to a standard, and turn it into concrete test cases. We’ll build manifest files that make file delivery verifiable. And we’ll walk through how to systematically break down everything you know about CSV edge cases — embedded commas, encoding traps, null representation chaos, the lot — into tests you can run during development, not discoveries you make in production at 2am.

Whether you’re producing CSVs for a downstream consumer or receiving them from a vendor who swears the file is “fine,” this is your playbook.

Why “I Know How to Test” Isn’t Enough

Here’s what I’ve noticed when I ask data engineers how they test their CSV files: most of them describe validation, not testing. They’ll say things like “we check the row count” or “we verify the columns match the schema.” And those are good things to do. But they’re runtime checks — they catch problems after they’ve already arrived at your door.

Testing is different. Testing happens during development. Testing means deliberately creating malformed files and confirming your system handles them correctly. Testing means asking “what’s the worst thing a vendor could send me?” and then actually building that worst-case file and feeding it through your pipeline.

The gap between validation and testing is where the expensive surprises live. Validation tells you this specific file has a problem. Testing tells you your system has a problem — before any real data gets near it.

So how do you build a proper CSV test suite? You need a specification to test against. And despite its limitations, that specification is RFC 4180.

RFC 4180: Your Test Specification (and Its Gaps)

RFC 4180 was published in October 2005 by Yakov Shafranovich. It’s universally treated as “the CSV standard,” but its opening line tells a different story: it’s an “Informational” memo that “does not specify an Internet standard of any kind.” It was written to document common practice and register the text/csv MIME type. That’s it.

But here’s the thing — even a flawed specification is better than no specification. RFC 4180 gives us seven concrete rules, and each one translates directly into test cases. Let’s walk through them.

Rule 1: Records Are Separated by CRLF

The spec mandates \r\n (carriage return + line feed) as the line ending. Not just \n (Unix/macOS default). Not just \r (old Mac).

Test cases for producers:

def test_line_endings_are_crlf(csv_content: bytes):
 """Every record must end with CRLF, not LF-only or CR-only."""
 lines = csv_content.split(b'\r\n')
 # Check no bare LF exists outside of quoted fields
 for i, line in enumerate(lines):
 assert b'\n' not in line, (
 f"Row {i+1}: Found bare LF (Unix line ending). "
 f"RFC 4180 requires CRLF."
 )
 # Check no bare CR exists outside of quoted fields
 for i, line in enumerate(lines):
 assert b'\r' not in line, (
 f"Row {i+1}: Found bare CR (old Mac line ending). "
 f"RFC 4180 requires CRLF."
 )

Test cases for consumers:

def test_consumer_handles_mixed_line_endings(parser):
 """Consumer should handle LF, CR, and CRLF gracefully."""
 test_cases = {
 "unix_lf": b"name,age\nAlice,30\nBob,25\n",
 "windows_crlf": b"name,age\r\nAlice,30\r\nBob,25\r\n",
 "old_mac_cr": b"name,age\rAlice,30\rBob,25\r",
 "mixed": b"name,age\r\nAlice,30\nBob,25\r",
 }
 for label, content in test_cases.items():
 result = parser(content)
 assert len(result) == 2, (
 f"Line ending style '{label}': Expected 2 data rows, "
 f"got {len(result)}"
 )

This distinction matters. If you’re a producer, your job is to emit compliant files. If you’re a consumer, your job is to survive whatever arrives. The test mindset is different for each role — producers test for correctness, consumers test for resilience.

Rule 2: The Last Record May or May Not Have a Trailing CRLF

This sounds trivial, but it catches people. Some systems emit a final newline, others don’t. Your parser needs to handle both without creating a phantom empty row at the end.

def test_trailing_newline_does_not_create_empty_row(parser):
 """A trailing CRLF should not produce an extra empty record."""
 with_trailing = b"name,age\r\nAlice,30\r\n"
 without_trailing = b"name,age\r\nAlice,30"
 assert len(parser(with_trailing)) == len(parser(without_trailing))

Rule 3: There Should Be a Header Row

The RFC says “should” — not “must.” This ambiguity is a source of real problems. Some vendors send headers, some don’t, and some send headers that change between deliveries.

Not having headers isn’t always a mistake. From a security perspective, omitting headers is actually defensible. Think about what a header row really is: it’s a plaintext schema sitting right there in the first line of your file. If someone intercepts a CSV with columns named ssn, credit_score, annual_income, and account_balance, they don’t need to decode anything. The header tells them exactly what they’ve stolen and which column to exploit first.

Stripping headers from files in transit — especially files containing PII, financial data, or health records — reduces the value of an intercepted file to an attacker. A file full of numbers and strings without column names is significantly harder to weaponise than one with a helpful schema attached.

But here’s the catch: headerless files only work if you have a reliable mechanism to reconstruct the schema at the consumer end. The column definitions need to live somewhere — and that somewhere should be your manifest file or your data contract. The manifest we build later in this article includes a columns array with names, types, and ordering for exactly this reason. The producer strips the headers, the manifest carries the schema, and the consumer rebuilds the structure on ingestion.

This means your tests need to handle both scenarios:

def test_header_row_present_and_matches_schema(csv_file, expected_columns):
 """When headers are expected, verify they match the schema exactly."""
 header = csv_file.readline().strip().split(',')
 assert header == expected_columns, (
 f"Header mismatch.\n"
 f"Expected: {expected_columns}\n"
 f"Got: {header}"
 )

def test_no_duplicate_column_names(csv_file):
 """Duplicate column names cause silent column overwrites in pandas."""
 header = csv_file.readline().strip().split(',')
 duplicates = [h for h in header if header.count(h) > 1]
 assert not duplicates, (
 f"Duplicate column names found: {set(duplicates)}. "
 f"Pandas will silently rename these to name.1, name.2, etc."
 )

def test_headerless_file_matches_manifest_schema(csv_file, manifest):
 """When headers are intentionally omitted for security,
 the first data row must match the manifest's column count
 and the manifest must define column names and types."""
 first_row = csv_file.readline().strip().split(',')
 expected_cols = manifest['files'][0]['columns']
 assert len(first_row) == len(expected_cols), (
 f"Headerless file has {len(first_row)} fields but manifest "
 f"defines {len(expected_cols)} columns. Without headers, "
 f"column alignment is entirely dependent on the manifest."
 )
 # Verify manifest actually defines names (not just count)
 for col_def in expected_cols:
 assert 'name' in col_def, (
 "Manifest column definitions must include 'name' — "
 "headerless files are only safe if the schema is "
 "fully specified in the manifest."
 )

Whether you include headers or not, the decision must be explicit and documented in your data contract. The worst scenario isn’t a headerless file or a headered file — it’s a file where nobody agreed which one it should be, and the answer changes depending on who ran the export.

Rule 4: Each Row Should Contain the Same Number of Fields

This is where “ragged rows” come from — and they’re surprisingly common. A free-text field containing an unescaped comma silently creates an extra column in that row, shifting everything to the right.

def test_consistent_column_count(csv_reader):
 """Every row must have the same number of fields as the header."""
 header = next(csv_reader)
 expected = len(header)
 ragged_rows = []
 for i, row in enumerate(csv_reader, start=2):
 if len(row) != expected:
 ragged_rows.append({
 'row': i,
 'expected': expected,
 'got': len(row),
 'content': row[:3] # First 3 fields for debugging
 })
 assert not ragged_rows, (
 f"Found {len(ragged_rows)} ragged rows. "
 f"First: row {ragged_rows[0]['row']} has "
 f"{ragged_rows[0]['got']} fields (expected {expected}). "
 f"This usually means an unescaped comma or newline in the data."
 )

This test is the single most valuable check you can run against a CSV file. If it passes, most of the structural problems aren’t present. If it fails, you know exactly where to look.

Rules 5-7: Quoting and Escaping

These three rules govern how fields handle special characters:

Fields containing commas, double-quotes, or line breaks must be enclosed in double-quotes
Double-quotes inside a quoted field are escaped by doubling them: ""
Spaces are part of field data and must not be trimmed

These rules are where the most insidious bugs hide:

# -- Producer tests: Do we generate valid quoted fields? --

def test_fields_with_commas_are_quoted():
 """A field containing a comma must be wrapped in double-quotes."""
 record = generate_csv_row(address="456 Oak Ave, Apt 2B")
 assert record == 'id,"456 Oak Ave, Apt 2B",other_field'

def test_fields_with_quotes_are_escaped():
 """Double-quotes in data must be doubled, and field must be quoted."""
 record = generate_csv_row(company='Acme "Holdings" Ltd')
 assert '"Acme ""Holdings"" Ltd"' in record

def test_fields_with_newlines_are_quoted():
 """A field containing a line break must be wrapped in double-quotes."""
 record = generate_csv_row(notes="Line one\r\nLine two")
 # The entire field, including the newline, should be inside quotes
 assert '"Line one\r\nLine two"' in record

# -- Consumer tests: Can we parse valid-but-tricky fields? --

def test_parse_embedded_comma():
 content = b'name,address\r\n"Alice","456 Oak Ave, Apt 2B"\r\n'
 result = parse(content)
 assert result[0]['address'] == '456 Oak Ave, Apt 2B'

def test_parse_embedded_newline():
 content = b'name,notes\r\n"Alice","Line one\r\nLine two"\r\n'
 result = parse(content)
 assert result[0]['notes'] == 'Line one\r\nLine two'

def test_parse_escaped_quotes():
 content = b'name,company\r\n"Alice","Acme ""Holdings"" Ltd"\r\n'
 result = parse(content)
 assert result[0]['company'] == 'Acme "Holdings" Ltd'

That covers the seven RFC rules. But here’s the catch — the RFC is only half the story. The really nasty problems come from everything the RFC doesn’t cover.

Testing What the RFC Forgot

RFC 4180 says nothing about character encoding, data types, null values, or how Excel will mangle your data. These are the gaps where production failures breed. Each one becomes a test category.

Encoding Tests

The most common encoding problem is invisible: the UTF-8 BOM (Byte Order Mark). Excel on Windows prepends three bytes (EF BB BF) to CSV files saved as UTF-8. Those bytes become an invisible character attached to your first column name. You try df['Name'] and get a KeyError because the actual column is '\ufeffName'.

def test_no_bom_prefix(csv_path):
 """UTF-8 BOM causes invisible characters in the first column name."""
 with open(csv_path, 'rb') as f:
 first_bytes = f.read(3)
 assert first_bytes != b'\xef\xbb\xbf', (
 "File contains UTF-8 BOM. This will prepend an invisible "
 "character to the first column name and break column lookups."
 )

def test_file_is_valid_utf8(csv_path):
 """Catch encoding mismatches before they become mojibake."""
 with open(csv_path, 'rb') as f:
 content = f.read()
 try:
 content.decode('utf-8')
 except UnicodeDecodeError as e:
 # Try to identify the actual encoding
 import chardet
 detected = chardet.detect(content)
 raise AssertionError(
 f"File is not valid UTF-8. Decode error at byte {e.start}. "
 f"Detected encoding: {detected['encoding']} "
 f"(confidence: {detected['confidence']:.0%}). "
 f"This is likely a Latin-1 or Windows-1252 file."
 )

def test_no_null_bytes(csv_path):
 """Null bytes (0x00) indicate binary data or a UTF-16 file
 being read as UTF-8."""
 with open(csv_path, 'rb') as f:
 content = f.read()
 null_positions = [i for i, b in enumerate(content) if b == 0]
 assert not null_positions, (
 f"Found {len(null_positions)} null bytes. "
 f"First at position {null_positions[0]}. "
 f"This might be a UTF-16 encoded file."
 )

If you’re a consumer, you should also test that your parser survives non-UTF-8 input gracefully — does it raise a clear error, or does it silently produce mojibake? Build test fixtures with Latin-1 encoded files containing accented characters (café, naïve, München) and verify your system either decodes correctly or fails explicitly.

Null Value Tests

RFC 4180 has no concept of null. An empty field (,,) is just an empty string. But every system represents null differently, and when you’re receiving files from multiple vendors, the chaos compounds.

# These are all real null representations I've seen in production
NULL_VARIANTS = [
 '', # Empty string (most common)
 'NULL', # MySQL Workbench default
 'null', # JavaScript/JSON convention
 'None', # Python convention
 r'\N', # PostgreSQL COPY convention
 'N/A', # Business user convention
 'n/a', # Lowercase variant
 'NA', # R convention
 '#N/A', # Excel formula error
 '-', # Dashboard export convention
 'NaN', # Pandas/NumPy convention
 '(null)', # Some BI tools
]

def test_null_representation_is_consistent(csv_path, null_convention=''):
 """All null values should use the agreed convention.
 Mixed conventions cause type inference failures."""
 import csv
 with open(csv_path, newline='') as f:
 reader = csv.reader(f)
 header = next(reader)
 violations = []
 for row_num, row in enumerate(reader, start=2):
 for col_idx, value in enumerate(row):
 if value in NULL_VARIANTS and value != null_convention:
 violations.append({
 'row': row_num,
 'column': header[col_idx],
 'value': repr(value),
 'expected': repr(null_convention)
 })
 assert not violations, (
 f"Found {len(violations)} non-standard null representations. "
 f"First: row {violations[0]['row']}, "
 f"column '{violations[0]['column']}' contains "
 f"{violations[0]['value']} (expected {violations[0]['expected']})"
 )

This is a test you should agree on with your vendor as part of a data contract — which we’ll get to shortly. Without explicit agreement, you’ll spend more time debugging null handling than you will on actual analysis.

The Leading Zeros Problem

Excel silently strips leading zeros from any field it interprets as numeric. ZIP code 07102 becomes 7102. Product code 00456 becomes 456. Numbers over 15 digits lose precision permanently. This is not a CSV problem — it’s an Excel problem — but it affects every CSV that passes through Excel, which is most of them.

def test_leading_zeros_preserved(csv_path, zero_padded_columns):
 """Fields that should retain leading zeros still have them.
 Excel strips these silently on open-and-save."""
 import csv
 with open(csv_path, newline='') as f:
 reader = csv.DictReader(f)
 for row_num, row in enumerate(reader, start=2):
 for col in zero_padded_columns:
 value = row[col]
 if value and not value[0].isdigit():
 continue # Skip non-numeric values
 if value and value[0] == '0' and len(value) > 1:
 pass # Good — leading zero preserved
 elif value and col in zero_padded_columns:
 # Check against expected length
 # e.g., ZIP should be 5 digits
 pass

def test_no_scientific_notation(csv_path, numeric_columns):
 """Numbers should not be in scientific notation (1.23E+11).
 This happens when Excel converts long numbers."""
 import csv, re
 sci_notation = re.compile(r'^-?\d+\.?\d*[eE][+-]?\d+$')
 with open(csv_path, newline='') as f:
 reader = csv.DictReader(f)
 violations = []
 for row_num, row in enumerate(reader, start=2):
 for col in numeric_columns:
 if sci_notation.match(row[col] or ''):
 violations.append((row_num, col, row[col]))
 assert not violations, (
 f"Scientific notation found — data may have been "
 f"corrupted by Excel. First: row {violations[0][0]}, "
 f"column '{violations[0][1]}', value '{violations[0][2]}'"
 )

Date Format Tests

Is 03/04/2024 March 4th or April 3rd? Unless you know the source system’s locale, you literally cannot tell for dates where both day and month values are 12 or below. The only safe format is ISO 8601: YYYY-MM-DD.

def test_dates_are_iso8601(csv_path, date_columns):
 """Date fields should use ISO 8601 (YYYY-MM-DD) format.
 Ambiguous formats like MM/DD/YYYY vs DD/MM/YYYY
 are programmatically unresolvable."""
 import csv, re
 iso_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}')
 with open(csv_path, newline='') as f:
 reader = csv.DictReader(f)
 violations = []
 for row_num, row in enumerate(reader, start=2):
 for col in date_columns:
 val = row.get(col, '')
 if val and not iso_pattern.match(val):
 violations.append((row_num, col, val))
 assert not violations, (
 f"Non-ISO date format found. First: row {violations[0][0]}, "
 f"column '{violations[0][1]}', value '{violations[0][2]}'. "
 f"Ambiguous date formats cannot be safely parsed."
 )

Invisible Character Tests

These are the ones that make you question your sanity. Non-breaking spaces (U+00A0), zero-width spaces (U+200B), zero-width joiners — they look identical to normal characters (or invisible entirely) but break numeric conversion, string matching, and joins.

INVISIBLE_CHARS = {
 '\u00a0': 'Non-breaking space',
 '\u200b': 'Zero-width space',
 '\u200c': 'Zero-width non-joiner',
 '\u200d': 'Zero-width joiner',
 '\ufeff': 'BOM / Zero-width no-break space',
 '\u2028': 'Line separator',
 '\u2029': 'Paragraph separator',
 '\u00ad': 'Soft hyphen',
}

def test_no_invisible_characters(csv_path):
 """Invisible Unicode characters break joins, type casting,
 and string comparisons while being impossible to spot visually."""
 with open(csv_path, encoding='utf-8') as f:
 content = f.read()
 found = []
 for char, name in INVISIBLE_CHARS.items():
 positions = [i for i, c in enumerate(content) if c == char]
 if positions:
 found.append(f"{name} (U+{ord(char):04X}): "
 f"{len(positions)} occurrences")
 assert not found, (
 "Invisible characters detected:\n" +
 "\n".join(f" - {f}" for f in found)
 )

Manifest Files: Testing Secure Delivery

All of those tests assume you’ve received the file intact. But how do you know? A file could be truncated during transfer, silently corrupted on disk, or swapped out entirely. This is where manifest files come in — they’re the contract between producer and consumer.

A manifest is metadata about the file that travels alongside it. At minimum, it should contain the filename, a cryptographic checksum, the row count, and the file size. NIST has formally deprecated MD5 and SHA-1, so use SHA-256.

Here’s what a solid manifest looks like:

{
 "manifest_version": "1.0",
 "generated_at": "2026-03-04T08:00:00Z",
 "source_system": "vendor-crm",
 "batch_id": "batch-20260304-001",
 "files": [
 {
 "filename": "customers_20260304.csv",
 "sha256": "a3f2b8c9d1e4f5a6b7c8d9e0f1a2b3c4d5e6f7...",
 "row_count": 50432,
 "row_count_includes_header": true,
 "file_size_bytes": 8234567,
 "column_count": 12,
 "encoding": "UTF-8",
 "delimiter": ",",
 "quote_char": "\"",
 "has_header": true,
 "columns": [
 { "name": "id", "type": "integer" },
 { "name": "name", "type": "string" },
 { "name": "email", "type": "string" },
 { "name": "created_date", "type": "date", "format": "YYYY-MM-DD" }
 ]
 }
 ]
}

Now here’s the test suite that validates a delivery against its manifest:

import hashlib
import json
import os

def validate_delivery(manifest_path, data_dir):
 """Validate a CSV delivery against its manifest.
 Returns list of errors. Empty list = valid delivery."""
 errors = []

 # 1. Parse manifest
 with open(manifest_path) as f:
 manifest = json.load(f)

 for file_spec in manifest['files']:
 filepath = os.path.join(data_dir, file_spec['filename'])

 # 2. File existence
 if not os.path.exists(filepath):
 errors.append(f"MISSING: {file_spec['filename']} not found")
 continue

 # 3. File size
 actual_size = os.path.getsize(filepath)
 if actual_size == 0:
 errors.append(f"EMPTY: {file_spec['filename']} is zero bytes")
 continue
 if actual_size != file_spec['file_size_bytes']:
 errors.append(
 f"SIZE MISMATCH: {file_spec['filename']} "
 f"expected {file_spec['file_size_bytes']} bytes, "
 f"got {actual_size}"
 )

 # 4. Checksum
 sha256 = hashlib.sha256()
 with open(filepath, 'rb') as f:
 for chunk in iter(lambda: f.read(8192), b''):
 sha256.update(chunk)
 actual_hash = sha256.hexdigest()
 if actual_hash != file_spec['sha256']:
 errors.append(
 f"CHECKSUM MISMATCH: {file_spec['filename']} "
 f"has been modified or corrupted in transit"
 )

 # 5. Row count
 with open(filepath, encoding=file_spec.get('encoding', 'utf-8')) as f:
 actual_rows = sum(1 for _ in f)
 expected = file_spec['row_count']
 if actual_rows != expected:
 errors.append(
 f"ROW COUNT MISMATCH: {file_spec['filename']} "
 f"expected {expected} rows, got {actual_rows}"
 )

 # 6. Column count and names
 with open(filepath, encoding=file_spec.get('encoding', 'utf-8')) as f:
 import csv
 reader = csv.reader(f)
 header = next(reader)
 if len(header) != file_spec['column_count']:
 errors.append(
 f"COLUMN COUNT MISMATCH: {file_spec['filename']} "
 f"expected {file_spec['column_count']} columns, "
 f"got {len(header)}"
 )
 if 'columns' in file_spec:
 expected_names = [c['name'] for c in file_spec['columns']]
 if header != expected_names:
 errors.append(
 f"COLUMN NAME MISMATCH: {file_spec['filename']}\n"
 f" Expected: {expected_names}\n"
 f" Got: {header}"
 )

 return errors

The key insight about manifest validation is that it should be a gate, not a check. If the manifest fails, you don’t process the file. Period. No “well, the row count is close enough.” No “the checksum is probably fine.” Either the delivery matches the contract or it gets quarantined.

For orchestration tools like Airflow, the pattern looks like this:

# Airflow DAG pattern: manifest-gated processing
wait_for_manifest = FileSensor(
 task_id="wait_for_manifest",
 filepath="/landing/{{ ds_nodash }}/manifest.json",
 poke_interval=120,
 timeout=7200,
)

validate = BranchPythonOperator(
 task_id="validate_delivery",
 python_callable=validate_and_route,
 # Routes to either 'load_data' or 'quarantine_and_alert'
)

wait_for_manifest >> validate >> [load_data, quarantine_and_alert]

As a producer, you should generate the manifest after writing the CSV file — computing the checksum on the exact bytes that were written. Ship the manifest alongside the data file. If you’re using SFTP or S3, upload the data file first, then the manifest. The consumer watches for the manifest as a “ready” signal — its presence means the data file is complete.

As a consumer, you should test your manifest validation against deliberately corrupted scenarios:

def test_detects_truncated_file():
 """Simulate a file that was cut short during transfer."""
 # Write a file, then truncate it
 ...

def test_detects_missing_file():
 """Manifest references a file that doesn't exist."""
 ...

def test_detects_extra_rows():
 """File has more rows than manifest claims —
 could indicate duplicate data or appended garbage."""
 ...

def test_detects_wrong_checksum():
 """File was modified after manifest was generated."""
 ...

def test_rejects_zero_byte_file():
 """Empty files should never pass validation."""
 ...

The CSV Test Matrix: Thinking Like a Tester

Most people know about edge cases — embedded commas, encoding problems, null ambiguity — but they don’t know how to systematically translate that knowledge into a test suite. They end up with ad hoc checks scattered across their codebase, catching some things and missing others.

The trick is to think in categories, not individual cases. Every CSV problem falls into one of five categories, and each category has a predictable set of test cases.

Category 1: Structural Integrity

Does the file conform to the expected shape?

Test	What It Catches	Producer/Consumer
Consistent column count across all rows	Unescaped delimiters in data	Both
Header row present and matches schema (when expected)	Schema drift, renamed columns	Consumer
Headerless file matches manifest column count	Misaligned data when headers stripped for security	Consumer
No duplicate column names	Silent column overwrites	Consumer
No trailing delimiter creating phantom column	Off-by-one column counts	Both
File is not empty / not just a header	Failed exports, empty result sets	Consumer
Row count within expected range	Truncation, duplication, data loss	Consumer

Category 2: Encoding and Character Safety

Are the bytes what we think they are?

Test	What It Catches	Producer/Consumer
File is valid UTF-8	Latin-1/Windows-1252 masquerading as UTF-8	Both
No BOM prefix	Invisible first-column corruption	Producer
No null bytes	Binary data or wrong encoding family	Consumer
No invisible Unicode characters	Broken joins, failed type casts	Both
No smart quotes or em-dashes	Copy-paste from Word/email	Both

Category 3: Quoting and Escaping

Are special characters handled correctly?

Test	What It Catches	Producer/Consumer
Fields with commas are quoted	Row splitting, column misalignment	Producer
Fields with double-quotes use `""` escaping	Unclosed quotes consuming multiple rows	Producer
Fields with newlines are quoted	Row count mismatches, multi-line chaos	Producer
Consumer survives unclosed quotes gracefully	Vendor bugs eating your pipeline	Consumer
No backslash escaping (non-standard)	Interop failures with strict parsers	Producer

Category 4: Data Type Safety

Will downstream systems interpret values correctly?

Test	What It Catches	Producer/Consumer
Dates in ISO 8601 format	Ambiguous MM/DD vs DD/MM	Both
Leading zeros preserved	Excel corruption of ZIPs, product codes	Both
No scientific notation in numeric fields	Excel corruption of long numbers	Both
Consistent null representation	Mixed NULL/None/N/A/empty conventions	Both
Boolean fields use consistent values	TRUE/True/true/1/yes/Y chaos	Both
Numeric fields contain only valid numbers	Currency symbols, percentage signs, commas in numbers	Consumer

Category 5: Delivery Integrity

Did the file arrive intact and complete?

Test	What It Catches	Producer/Consumer
SHA-256 checksum matches manifest	Corruption in transit	Consumer
File size matches manifest	Truncation, wrong file	Consumer
Row count matches manifest	Silent data loss (the COVID problem)	Consumer
Column count matches manifest	Schema changes without notice	Consumer
File is not a duplicate of previous delivery	Accidental re-sends	Consumer
Manifest is present and valid JSON	Incomplete delivery	Consumer

This matrix is your test plan. Print it out. Stick it on the wall next to your desk. When you’re building a new CSV integration, walk through each category and ask: “Do I have a test for this?”

Building Test Fixtures: The Art of Deliberate Breakage

A test suite is only as good as its fixtures. And for CSV testing, your fixtures need to be deliberately, creatively broken. Here’s how I build mine.

Start with a golden file — a perfectly compliant CSV that passes every test. This is your baseline. Then create variations that each break exactly one thing:

Each fixture is a test case. Your producer tests verify you never generate files that look like the broken fixtures. Your consumer tests verify you detect and handle each type of breakage correctly.

The most valuable fixture? unclosed_quote.csv. A single unclosed double-quote can cause a parser to consume every subsequent row as part of one giant field — potentially the entire rest of the file. I’ve seen this eat 2 million rows. Test for it explicitly.

What to Put in Your Data Contract

Tests tell you whether a specific file is good or bad. A data contract tells you what “good” means. Without one, you’re testing against assumptions — and your assumptions will eventually differ from your vendor’s.

Here’s what a data contract for CSV exchange should specify. Think of it as the RFC that RFC 4180 forgot to write:

File format:

Delimiter (comma, pipe, tab — be explicit)
Quote character (double-quote)
Escape method (doubled quotes, per RFC 4180)
Line ending (CRLF)
Encoding (UTF-8, no BOM)
Header row policy (present, omitted for security, or conditional — and if omitted, where the schema lives)
Column names and order (in manifest if headers are stripped)

Data type conventions:

Date format (ISO 8601: YYYY-MM-DD)
Timestamp format (ISO 8601: YYYY-MM-DDTHH:MM:SSZ)
Null representation (empty field — no “NULL”, no “N/A”)
Boolean representation (true/false)
Numeric precision (no scientific notation, no thousands separators)

Delivery protocol:

Transfer method (SFTP, S3, etc.)
Manifest file required (JSON, SHA-256 checksum)
Signal file or completion marker
Delivery schedule and SLA
Who to contact when something breaks

Change management:

Minimum notice period for schema changes
Versioning strategy (new columns appended, never reordered)
Breaking vs non-breaking change definitions

You don’t need a 50-page legal document. A one-page markdown file committed to a shared repository works. The point isn’t formality — it’s explicitness. When the vendor’s export changes and your pipeline breaks at 3am, you want something to point to that says “this is what we agreed.”

Putting It All Together: A Reusable Validation Function

Here’s a practical, copy-paste-ready validation function that covers the most critical checks. This isn’t a framework — it’s a function you can drop into any Python project and start using immediately.

import csv
import hashlib
import os
import re
from dataclasses import dataclass, field
from typing import Optional


@dataclass
class CSVValidationResult:
 """Results from validating a CSV file."""
 filepath: str
 is_valid: bool = True
 errors: list = field(default_factory=list)
 warnings: list = field(default_factory=list)
 stats: dict = field(default_factory=dict)

 def add_error(self, message: str):
 self.errors.append(message)
 self.is_valid = False

 def add_warning(self, message: str):
 self.warnings.append(message)


def validate_csv(
 filepath: str,
 expected_columns: Optional[list] = None,
 expected_encoding: str = 'utf-8',
 date_columns: Optional[list] = None,
 zero_padded_columns: Optional[list] = None,
 null_convention: str = '',
 max_row_count: Optional[int] = None,
 min_row_count: Optional[int] = None,
) -> CSVValidationResult:
 """Validate a CSV file against common quality checks.

 Works for both producers (checking your own output)
 and consumers (checking vendor deliveries).
 """
 result = CSVValidationResult(filepath=filepath)

 # --- Check 1: File exists and isn't empty ---
 if not os.path.exists(filepath):
 result.add_error(f"File not found: {filepath}")
 return result

 file_size = os.path.getsize(filepath)
 if file_size == 0:
 result.add_error("File is empty (zero bytes)")
 return result
 result.stats['file_size_bytes'] = file_size

 # --- Check 2: Encoding ---
 with open(filepath, 'rb') as f:
 raw_head = f.read(10000)

 # BOM check
 if raw_head[:3] == b'\xef\xbb\xbf':
 result.add_warning(
 "File contains UTF-8 BOM — this will corrupt "
 "the first column name in most parsers"
 )

 # Null byte check
 if b'\x00' in raw_head:
 result.add_error(
 "File contains null bytes — likely wrong encoding "
 "(e.g., UTF-16 read as UTF-8) or binary data"
 )
 return result

 # Encoding validity
 try:
 raw_full = open(filepath, 'rb').read()
 raw_full.decode(expected_encoding)
 except UnicodeDecodeError as e:
 result.add_error(
 f"File is not valid {expected_encoding}. "
 f"Decode error at byte position {e.start}."
 )
 return result

 # --- Check 3: Structural integrity ---
 encoding = 'utf-8-sig' if raw_head[:3] == b'\xef\xbb\xbf' else expected_encoding
 with open(filepath, newline='', encoding=encoding) as f:
 reader = csv.reader(f)

 # Header check
 try:
 header = next(reader)
 except StopIteration:
 result.add_error("File has no rows (not even a header)")
 return result

 result.stats['column_count'] = len(header)
 result.stats['columns'] = header

 # Duplicate column check
 seen = {}
 for col in header:
 seen[col] = seen.get(col, 0) + 1
 dupes = {k: v for k, v in seen.items() if v > 1}
 if dupes:
 result.add_error(f"Duplicate column names: {dupes}")

 # Expected columns check
 if expected_columns and header != expected_columns:
 result.add_error(
 f"Column mismatch.\n"
 f" Expected: {expected_columns}\n"
 f" Got: {header}"
 )

 # Row-level checks
 row_count = 0
 ragged_rows = []
 iso_date = re.compile(r'^\d{4}-\d{2}-\d{2}')

 for row_num, row in enumerate(reader, start=2):
 row_count += 1

 # Column count consistency
 if len(row) != len(header):
 ragged_rows.append((row_num, len(row)))

 # Date format check
 if date_columns:
 for col in date_columns:
 idx = header.index(col) if col in header else None
 if idx is not None and row[idx]:
 if not iso_date.match(row[idx]):
 result.add_warning(
 f"Row {row_num}, column '{col}': "
 f"Non-ISO date '{row[idx]}'"
 )

 result.stats['row_count'] = row_count

 if ragged_rows:
 result.add_error(
 f"{len(ragged_rows)} rows have wrong column count. "
 f"Expected {len(header)} columns. "
 f"First bad row: line {ragged_rows[0][0]} "
 f"has {ragged_rows[0][1]} fields."
 )

 # Row count bounds
 if min_row_count and row_count < min_row_count:
 result.add_error(
 f"Row count {row_count} is below minimum "
 f"threshold of {min_row_count}"
 )
 if max_row_count and row_count > max_row_count:
 result.add_error(
 f"Row count {row_count} exceeds maximum "
 f"threshold of {max_row_count}"
 )

 # --- Check 4: Invisible characters ---
 invisible = {
 '\u00a0': 'Non-breaking space',
 '\u200b': 'Zero-width space',
 '\ufeff': 'BOM (mid-file)',
 }
 text = raw_full.decode(encoding, errors='replace')
 for char, name in invisible.items():
 count = text.count(char)
 if count > 0:
 result.add_warning(
 f"Found {count} '{name}' characters (U+{ord(char):04X})"
 )

 # --- Compute checksum for manifest comparison ---
 sha256 = hashlib.sha256(raw_full).hexdigest()
 result.stats['sha256'] = sha256

 return result

Use it like this:

result = validate_csv(
 'vendor_delivery/customers_20260304.csv',
 expected_columns=['id', 'name', 'email', 'city', 'state', 'zip'],
 date_columns=['created_date', 'updated_date'],
 min_row_count=1000,
 max_row_count=100000,
)

if not result.is_valid:
 for error in result.errors:
 print(f"ERROR: {error}")
 # Route to quarantine
else:
 print(f"Valid. {result.stats['row_count']} rows, "
 f"SHA-256: {result.stats['sha256'][:16]}...")
 # Proceed to loading

What 16,000 Missing Records Taught Us

Let’s come back to where we started. Sixteen thousand COVID test results, dropped silently because nobody tested what would happen when the file exceeded a row limit. The labs did their job. The CSV was generated correctly. The file was transmitted successfully. Every individual component worked. The system failed because nobody tested the boundaries — the places where one component’s assumptions meet another component’s limitations.

That’s what CSV testing is really about. It’s not about whether you can parse a comma-separated file — of course you can. It’s about whether you’ve thought through every place where the producer’s assumptions and the consumer’s assumptions might diverge. The vendor who exports NULL as the literal string “NULL.” The CRM that allows newlines in address fields. The business user who opens your carefully formatted CSV in Excel, saves it, and sends it back with stripped zeros and mangled dates.

Every one of those is a test case. Every one of those is a boundary you can verify during development instead of discovering in production.

The engineers who sleep well at night aren’t the ones who assume their CSV files are fine. They’re the ones who wrote the tests to prove it.

Write-Audit-Publish with Iceberg Tables in Snowflake

Chris Hillman — Fri, 27 Feb 2026 09:00:00 +1100

It was a Tuesday afternoon when the analyst pinged me on Microsoft Teams: “Hey, the Total Portfolio numbers just jumped 40% overnight. Did we land a whale?”

We hadn’t.

What actually happened was more mundane and significantly more painful. A schema change in the source system introduced a currency conversion bug. Our pipeline dutifully loaded the corrupted data into production at 3 AM, the dashboards updated by 6 AM, and the Department Head opened her morning report to numbers that looked like champagne-worthy growth.

Here’s the thing — we had data quality tests. They ran after the data hit production. A classic case of closing the gate after the horse has bolted, kicked over the fence, and trampled the neighbour’s garden.

That incident changed how I think about data pipelines entirely. It led me to the Write-Audit-Publish pattern — a concept championed heavily by Zach Wilson. Zach frames it simply: write to a staging area with an identical schema to production, run your data quality checks, and only publish to production if they pass. It’s deceptively simple. And it works.

I’ve written about WAP before using Airflow as the orchestration layer, with traditional staging tables and swap mechanics. That approach is solid, battle-tested, and it’ll serve most teams well. But now that Apache Iceberg tables in Snowflake have matured — there’s a more elegant way to implement WAP that eliminates one of the pattern’s biggest practical headaches: the I/O overhead.

In this tutorial, we’re going to build a complete WAP test harness from scratch on Snowflake Iceberg tables. We’ll create synthetic data, simulate a change that introduces bad data, catch it with quality gates, and publish only the clean data to production. Along the way, I’ll show you why the zero-copy mechanics of Iceberg’s architecture make the publish step insanely fast — and why that speed changes the calculus on when WAP is worth the effort.

Why the Publish Step Is Where Traditional WAP Hurts

If you’ve worked in data engineering for more than a few months, you’ve probably built some version of WAP without calling it that. You load data into a staging table, run some checks, then copy the good stuff into production. That works. Sort of.

The problem is what happens at the publish step. In a traditional staging-table WAP implementation, publishing means physically moving data. You’re running an INSERT INTO production SELECT * FROM staging, or you’re doing a table swap that involves Snowflake creating new micro-partitions. Either way, you’re generating I/O. You’re writing data that already exists in a perfectly good format, to another location, in another format, just so it lives under the right table name.

For a table with a few million rows, nobody notices. For a table with a few billion rows partitioned across hundreds of files? That publish step becomes a pipeline bottleneck. I’ve seen WAP implementations where the audit takes 30 seconds and the publish takes 45 minutes. The quality gate is fast. The bureaucratic data shuffle is slow.

This is where Iceberg changes the equation.

Iceberg tables don’t store data inside Snowflake’s proprietary micro-partition format. The data lives as Parquet files in your external storage — S3, GCS, or Azure Blob. Snowflake manages the metadata that tells it which Parquet files belong to which snapshot of the table. When you do a CLONE of an Iceberg table in Snowflake, it creates a new metadata pointer to the same underlying Parquet files. No data moves. No files are copied. No I/O is generated.

And when you SWAP a validated clone back into production? Again — it’s a metadata operation. Snowflake updates which metadata pointer the production table name resolves to. The Parquet files don’t move. The snapshots don’t change. The swap is essentially instant regardless of table size.

Let me put that in concrete terms. Say you have a 500 GB Iceberg table with three years of order history. In a traditional WAP setup, publishing validated staging data means writing hundreds of gigabytes to new micro-partitions — that’s real compute cost, real wall-clock time, and a real window where downstream consumers might see partial data. With Iceberg’s zero-copy clone and swap, the same publish step takes seconds. Not minutes. Seconds. Because you’re flipping a pointer, not copying bytes.

This isn’t a marginal improvement. It fundamentally changes when WAP is worth implementing. When publish is expensive, you think twice about adding WAP to every pipeline. When publish is free, the question flips — why wouldn’t you gate every production write with quality checks?

The Conceptual Model: How WAP Works on Iceberg

Here’s the flow we’re going to build. Every step maps to actual SQL you can run in Snowflake today:

The beauty of this model is that production is never exposed to unvalidated data. The clone is a sandbox. Downstream consumers querying fct_orders see the same clean data they saw yesterday, right up until the moment we swap — and the swap is atomic. There’s no window of inconsistency. No partial writes. No “oops, the pipeline failed halfway through the publish.”

Setting Up the Foundation

Let’s build this thing. You’ll need a Snowflake account with access to create Iceberg tables, which means an external volume pointing to your cloud storage and the appropriate privileges.

Prerequisites

-- Create our tutorial workspace
CREATE DATABASE IF NOT EXISTS wap_tutorial;
CREATE SCHEMA IF NOT EXISTS wap_tutorial.iceberg_wap;
USE DATABASE wap_tutorial;
USE SCHEMA iceberg_wap;

-- Create external volume (adjust for your cloud provider)
-- You'll need to configure the IAM trust relationship first
-- See Snowflake docs for the cloud-specific IAM setup
CREATE OR REPLACE EXTERNAL VOLUME wap_ext_vol
 STORAGE_LOCATIONS = (
 (
 NAME = 'wap-iceberg-storage'
 STORAGE_BASE_URL = 's3://your-bucket/iceberg-wap/'
 STORAGE_PROVIDER = 'S3'
 STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/snowflake-iceberg-role'
 )
 );

-- Verify the volume is healthy
DESC EXTERNAL VOLUME wap_ext_vol;

If the external volume setup feels unfamiliar, don’t stress — the Snowflake documentation for this is thorough. The IAM configuration is the fiddly part, but it’s a one-time cost. Once the volume exists, everything downstream is straightforward SQL.

Step 1: WRITE — Creating the Iceberg Table and Synthetic Data

We’re going to simulate an e-commerce order pipeline. This is a scenario practically every data engineer has encountered — daily order data flowing from a transactional system into the warehouse.

First, create the Iceberg table using Snowflake as the catalog:

-- Create the Iceberg table
-- Using Snowflake as the catalog means Snowflake handles
-- all lifecycle maintenance (compaction, metadata updates)
CREATE OR REPLACE ICEBERG TABLE fct_orders (
 order_id INT,
 customer_id INT,
 order_date DATE,
 product_id INT,
 quantity INT,
 unit_price DECIMAL(10,2),
 currency_code VARCHAR(3),
 total_amount DECIMAL(12,2),
 order_status VARCHAR(20),
 region VARCHAR(50),
 loaded_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
)
 CATALOG = 'SNOWFLAKE'
 EXTERNAL_VOLUME = 'wap_ext_vol'
 BASE_LOCATION = 'fct_orders/';

Now let’s generate our initial “production” data. This represents the clean, trusted baseline that’s been accumulating for the past 30 days. Five thousand orders, all in USD, no nulls, no negative amounts — a pristine starting point:

-- Generate 30 days of clean, synthetic order data
-- This is our trusted production baseline
INSERT INTO fct_orders (
 order_id, customer_id, order_date, product_id,
 quantity, unit_price, currency_code, total_amount,
 order_status, region
)
SELECT
 ROW_NUMBER() OVER (ORDER BY SEQ4()) + 10000 AS order_id,
 UNIFORM(1, 500, RANDOM()) AS customer_id,
 DATEADD('day', -UNIFORM(0, 29, RANDOM()), CURRENT_DATE()) AS order_date,
 UNIFORM(1, 50, RANDOM()) AS product_id,
 UNIFORM(1, 10, RANDOM()) AS quantity,
 ROUND(UNIFORM(5.00, 200.00, RANDOM())::DECIMAL(10,2), 2) AS unit_price,
 'USD' AS currency_code,
 -- total_amount = quantity * unit_price (clean, correct calculation)
 ROUND(
 UNIFORM(1, 10, RANDOM()) * UNIFORM(5.00, 200.00, RANDOM())::DECIMAL(12,2),
 2
 ) AS total_amount,
 CASE UNIFORM(1, 5, RANDOM())
 WHEN 1 THEN 'PENDING'
 WHEN 2 THEN 'CONFIRMED'
 WHEN 3 THEN 'SHIPPED'
 WHEN 4 THEN 'DELIVERED'
 ELSE 'CANCELLED'
 END AS order_status,
 CASE UNIFORM(1, 5, RANDOM())
 WHEN 1 THEN 'APAC'
 WHEN 2 THEN 'EMEA'
 WHEN 3 THEN 'AMERICAS'
 WHEN 4 THEN 'LATAM'
 ELSE 'ANZ'
 END AS region
FROM TABLE(GENERATOR(ROWCOUNT => 5000));

Let’s verify our baseline is healthy before we mess it up:

-- Quick sanity check on our production data
SELECT
 COUNT(*) AS total_orders,
 COUNT(DISTINCT customer_id) AS unique_customers,
 MIN(order_date) AS earliest_order,
 MAX(order_date) AS latest_order,
 ROUND(AVG(total_amount), 2) AS avg_order_value,
 COUNT(CASE WHEN total_amount <= 0 THEN 1 END) AS negative_amounts,
 COUNT(CASE WHEN currency_code != 'USD' THEN 1 END) AS non_usd_orders
FROM fct_orders;

You should see 5,000 orders, zero negative amounts, and zero non-USD orders. This is clean production data. Remember these numbers — because we’re about to introduce some chaos.

Simulating the Change: When Bad Data Arrives

Now the realistic part. A new batch of 500 orders arrives from the source system. But this batch has problems — the kind of subtle, infuriating problems that slip past casual inspection but wreck downstream analytics.

First, we create the zero-copy clone. This is our staging sandbox:

-- Create a zero-copy clone of production
-- This is a METADATA operation — no Parquet files are copied
-- The clone shares the same underlying data files as production
CREATE OR REPLACE ICEBERG TABLE fct_orders_staging
 CLONE fct_orders;

That CLONE just happened in under a second regardless of whether your table is 5,000 rows or 500 million. No data moved. The staging clone points to the same Parquet files in your external storage. Snowflake just created a new metadata entry that says “this table currently looks like that table.”

Now let’s load the problematic batch into our staging clone:

-- Simulate incoming data with realistic bugs:
-- 1. Some NULL currency codes (source system migration issue)
-- 2. Some invalid currency codes (mapping error)
-- 3. Some negative total_amounts (conversion bug)
-- 4. Some EUR amounts that weren't actually converted
INSERT INTO fct_orders_staging (
 order_id, customer_id, order_date, product_id,
 quantity, unit_price, currency_code, total_amount,
 order_status, region
)
SELECT
 ROW_NUMBER() OVER (ORDER BY SEQ4()) + 20000 AS order_id,
 UNIFORM(1, 500, RANDOM()) AS customer_id,
 CURRENT_DATE() AS order_date,
 UNIFORM(1, 50, RANDOM()) AS product_id,
 UNIFORM(1, 10, RANDOM()) AS quantity,
 ROUND(UNIFORM(5.00, 200.00, RANDOM())::DECIMAL(10,2), 2) AS unit_price,
 -- HERE'S THE BUG: some orders arrive with wrong currency codes
 CASE
 WHEN UNIFORM(1, 100, RANDOM()) <= 85 THEN 'USD'
 WHEN UNIFORM(1, 100, RANDOM()) <= 5 THEN NULL -- Null currency!
 WHEN UNIFORM(1, 100, RANDOM()) <= 5 THEN 'INVALID' -- Invalid code!
 ELSE 'EUR' -- EUR amount but never converted to USD
 END AS currency_code,
 -- And some totals went negative due to a sign error
 CASE
 WHEN UNIFORM(1, 100, RANDOM()) <= 90
 THEN ROUND(
 UNIFORM(1, 10, RANDOM())
 * UNIFORM(5.00, 200.00, RANDOM())::DECIMAL(12,2), 2)
 ELSE ROUND(
 -1 * UNIFORM(1, 10, RANDOM())
 * UNIFORM(5.00, 200.00, RANDOM())::DECIMAL(12,2), 2)
 END AS total_amount,
 CASE UNIFORM(1, 5, RANDOM())
 WHEN 1 THEN 'PENDING'
 WHEN 2 THEN 'CONFIRMED'
 WHEN 3 THEN 'SHIPPED'
 WHEN 4 THEN 'DELIVERED'
 ELSE 'CANCELLED'
 END AS order_status,
 CASE UNIFORM(1, 5, RANDOM())
 WHEN 1 THEN 'APAC'
 WHEN 2 THEN 'EMEA'
 WHEN 3 THEN 'AMERICAS'
 WHEN 4 THEN 'LATAM'
 ELSE 'ANZ'
 END AS region
FROM
 TABLE(GENERATOR(ROWCOUNT => 500));

The write step is done. We have 500 new orders in our staging clone — some clean, some dirty. And critically, production is completely untouched. Anyone querying fct_orders right now sees exactly what they saw before. No Department Head is going to message you on Teams this morning about phantom Total Portfolio growth.

Step 2: AUDIT — The Quality Gate Battery

This is where WAP earns its keep. We run every check we care about against the staging clone. If anything fails, we either fix it or drop the clone and production never sees the bad data. As Zach puts it — downstream pipelines wait on production data to be ready. Not just present.

Let’s build a comprehensive audit battery. Each check targets a different failure mode:

Check 1: Null Detection on Critical Fields

Nulls in currency or amount fields are deal-breakers. A null currency means we can’t convert the amount. A null amount means we can’t aggregate Total Portfolio numbers. Either one poisons everything downstream.

SELECT
 'NULL_CHECK' AS check_name,
 COUNT(CASE WHEN currency_code IS NULL THEN 1 END) AS null_currency,
 COUNT(CASE WHEN total_amount IS NULL THEN 1 END) AS null_amount,
 COUNT(CASE WHEN order_id IS NULL THEN 1 END) AS null_order_id,
 CASE
 WHEN COUNT(CASE WHEN currency_code IS NULL THEN 1 END) > 0
 OR COUNT(CASE WHEN total_amount IS NULL THEN 1 END) > 0
 OR COUNT(CASE WHEN order_id IS NULL THEN 1 END) > 0
 THEN 'FAIL'
 ELSE 'PASS'
 END AS check_result
FROM
 fct_orders_staging
WHERE
 order_date = CURRENT_DATE();

Check 2: Valid Currency Codes

Only accepted ISO currency codes should appear. Anything else means a mapping went sideways upstream.

SELECT
 'CURRENCY_VALIDATION' AS check_name,
 COUNT(*) AS invalid_currency_count,
 LISTAGG(DISTINCT currency_code, ', ') AS invalid_values,
 CASE
 WHEN COUNT(*) > 0 THEN 'FAIL'
 ELSE 'PASS'
 END AS check_result
FROM
 fct_orders_staging
WHERE
 order_date = CURRENT_DATE()
 AND currency_code NOT IN ('USD', 'EUR', 'GBP', 'AUD', 'CAD', 'JPY')
 AND currency_code IS NOT NULL;

Check 3: No Negative Amounts

A negative total_amount on an order that isn’t a refund is a sign of a calculation bug. This is exactly the kind of error that inflated our Total Portfolio numbers that Tuesday morning.

SELECT
 'NEGATIVE_AMOUNT_CHECK' AS check_name,
 COUNT(*) AS negative_count,
 MIN(total_amount) AS worst_amount,
 CASE
 WHEN COUNT(*) > 0 THEN 'FAIL'
 ELSE 'PASS'
 END AS check_result
FROM
 fct_orders_staging
WHERE
 order_date = CURRENT_DATE()
 AND total_amount < 0;

Check 4: Duplicate Detection

Duplicate order IDs mean the source system sent the same batch twice, or our pipeline ran a retry without proper idempotency.

SELECT
 'DUPLICATE_CHECK' AS check_name,
 COUNT(*) - COUNT(DISTINCT order_id) AS duplicate_count,
 CASE
 WHEN COUNT(*) - COUNT(DISTINCT order_id) > 0 THEN 'FAIL'
 ELSE 'PASS'
 END AS check_result
FROM
 fct_orders_staging
WHERE
 order_date = CURRENT_DATE();

Check 5: Volume Anomaly Detection

If today’s batch is wildly larger or smaller than the historical daily average, something structural has changed. Maybe a filter broke, maybe a new source came online, maybe someone’s test data leaked into production.

WITH daily_volumes AS (
 SELECT
 order_date,
 COUNT(*) AS daily_count
 FROM
 fct_orders_staging
 WHERE
 order_date < CURRENT_DATE()
 GROUP BY
 order_date
),
today_volume AS
(
 SELECT COUNT(*) AS today_count
 FROM
 fct_orders_staging
 WHERE
 order_date = CURRENT_DATE()
)
SELECT
 'VOLUME_ANOMALY' AS check_name,
 t.today_count,
 ROUND(AVG(d.daily_count), 0) AS avg_daily_count,
 CASE
 WHEN t.today_count > AVG(d.daily_count) * 3 THEN 'FAIL'
 WHEN t.today_count < AVG(d.daily_count) * 0.1 THEN 'FAIL'
 ELSE 'PASS'
 END AS check_result
FROM
 daily_volumes d

 CROSS JOIN today_volume t
GROUP BY
 t.today_count;

Aggregating the Verdict

In a real pipeline, you’d wrap these checks in a stored procedure or Snowflake Task. Here’s the aggregation logic that gives you a single pass/fail decision:

-- Aggregate all quality checks into a single verdict
CREATE OR REPLACE TEMPORARY TABLE audit_results AS
WITH checks AS (
 -- Null check
 SELECT 'NULL_CHECK' AS check_name,
 CASE WHEN COUNT(CASE WHEN currency_code IS NULL
 OR total_amount IS NULL THEN 1 END) > 0
 THEN 'FAIL' ELSE 'PASS' END AS result
 FROM fct_orders_staging
 WHERE order_date = CURRENT_DATE()

 UNION ALL

 -- Negative amounts
 SELECT 'NEGATIVE_AMOUNTS',
 CASE WHEN COUNT(*) > 0 THEN 'FAIL' ELSE 'PASS' END
 FROM fct_orders_staging
 WHERE order_date = CURRENT_DATE() AND total_amount < 0

 UNION ALL

 -- Invalid currency
 SELECT 'INVALID_CURRENCY',
 CASE WHEN COUNT(*) > 0 THEN 'FAIL' ELSE 'PASS' END
 FROM fct_orders_staging
 WHERE order_date = CURRENT_DATE()
 AND (currency_code NOT IN ('USD','EUR','GBP','AUD','CAD','JPY')
 OR currency_code IS NULL)

 UNION ALL

 -- Duplicates
 SELECT 'DUPLICATES',
 CASE WHEN COUNT(*) - COUNT(DISTINCT order_id) > 0
 THEN 'FAIL' ELSE 'PASS' END
 FROM fct_orders_staging
 WHERE order_date = CURRENT_DATE()
)
SELECT * FROM checks;

-- See the full verdict
SELECT * FROM audit_results;

-- Check if we're clear to publish
SELECT
 COUNT(*) AS total_checks,
 SUM(CASE WHEN result = 'PASS' THEN 1 ELSE 0 END) AS passed,
 SUM(CASE WHEN result = 'FAIL' THEN 1 ELSE 0 END) AS failed,
 CASE
 WHEN SUM(CASE WHEN result = 'FAIL' THEN 1 ELSE 0 END) = 0
 THEN 'CLEAR TO PUBLISH'
 ELSE 'BLOCKED — FIX REQUIRED'
 END AS verdict
FROM audit_results;

With our deliberately corrupted data, you’ll see multiple failures here. Null currencies, invalid codes, negative amounts — exactly the problems we planted. The audit caught everything. Production is still clean.

Step 3: PUBLISH — The Happy Path (When Audits Pass)

Let’s first walk through what happens when everything is clean. This is the path your pipeline will take 95% of the time — and it’s where the Iceberg zero-copy advantage really shines.

-- PUBLISH: Swap the validated clone into production
-- This is a METADATA OPERATION — no Parquet files are copied or moved
-- The swap is ATOMIC — downstream sees old data or new data, never partial
ALTER ICEBERG TABLE fct_orders
 SWAP WITH fct_orders_staging;

-- Clean up — the old production reference is now the staging table name
DROP TABLE IF EXISTS fct_orders_staging;

That’s it. Two statements. The SWAP updated which metadata pointer the table name fct_orders resolves to. The Parquet files sitting in your S3 bucket didn’t budge. No new files were written. No compute was burned copying data. No warehouse credits were consumed on I/O.

Think about what this means for pipeline design. In a traditional WAP implementation with standard Snowflake tables, the publish step is the expensive one — it’s the INSERT INTO ... SELECT * or the CREATE TABLE AS SELECT that physically writes data. That cost scales linearly with data volume. Double the table size, double the publish time and cost.

With Iceberg’s zero-copy swap, publish cost is constant regardless of table size. A 10 MB table and a 10 TB table swap in the same amount of time. That’s not a rounding error. That’s a fundamental shift in the economics of data quality enforcement.

For context, I timed this on a table with ~200 million rows across several hundred Parquet files. The clone took about a second. The audit checks took about 40 seconds. The swap? Sub-second. The most expensive part of the entire WAP pipeline was the actual data quality logic — which is exactly where you want the time to be spent.

Step 3 (Alternate): QUARANTINE — When Audits Fail

Now the realistic scenario. Our audit found problems. We have options.

Option A: Drop everything, investigate, retry. This is the nuclear option. Production stays on yesterday’s data until the source is fixed and a clean batch arrives.

-- Nuclear option — reject the entire batch
DROP TABLE IF EXISTS fct_orders_staging;
-- Production is exactly as it was. Nothing changed.

Option B: Quarantine the bad records, publish the clean ones. This is more practical for most teams. You log the failures for investigation, surgically remove the bad data, re-audit, and publish what’s clean.

-- First, capture the failures for investigation
-- This gives your team evidence to fix the source system
CREATE OR REPLACE TABLE quarantine_log AS
SELECT
 *,
 CURRENT_TIMESTAMP() AS quarantined_at,
 CASE
 WHEN currency_code IS NULL THEN 'NULL_CURRENCY'
 WHEN currency_code NOT IN ('USD','EUR','GBP','AUD','CAD','JPY')
 THEN 'INVALID_CURRENCY'
 WHEN total_amount < 0 THEN 'NEGATIVE_AMOUNT'
 END AS failure_reason
FROM
 fct_orders_staging
WHERE
 order_date = CURRENT_DATE()
 AND (
 currency_code IS NULL
 OR currency_code NOT IN ('USD','EUR','GBP','AUD','CAD','JPY')
 OR total_amount < 0
 );

-- See what we caught
SELECT failure_reason, COUNT(*) AS record_count
FROM quarantine_log
GROUP BY failure_reason
ORDER BY record_count DESC;

Now remove the bad records from the staging clone and re-validate:

-- Surgically remove the bad records
DELETE FROM fct_orders_staging
WHERE order_date = CURRENT_DATE()
 AND (
 currency_code IS NULL
 OR currency_code NOT IN ('USD','EUR','GBP','AUD','CAD','JPY')
 OR total_amount < 0
 );

-- Re-run the audit against the cleaned staging data
-- (Repeat the audit_results query from Step 2)
-- If it passes now, publish the clean subset
ALTER ICEBERG TABLE fct_orders
 SWAP WITH fct_orders_staging;

DROP TABLE IF EXISTS fct_orders_staging;

The quarantine log is important. It’s not enough to just filter out bad data — you need to understand why it was bad so you can fix the source. Every quarantined record is a symptom. The disease is upstream.

Putting It All Together: The Orchestrated Pipeline

In practice, you’d wrap this entire flow in a Snowflake Task or an external orchestrator. Here’s a stored procedure that encapsulates the full WAP pattern. This is something you can adapt and parameterise for any Iceberg table in your warehouse:

CREATE OR REPLACE PROCEDURE run_wap_pipeline(
 target_table VARCHAR,
 staging_table VARCHAR
)
RETURNS VARCHAR
LANGUAGE SQL
AS
$$
DECLARE
 fail_count INT;
 result_msg VARCHAR;
BEGIN
 -- WRITE: Clone production
 EXECUTE IMMEDIATE 'CREATE OR REPLACE ICEBERG TABLE '
 || staging_table || ' CLONE ' || target_table;

 -- (Your data loading logic goes here)
 -- INSERT INTO staging_table SELECT ... FROM source

 -- AUDIT: Run quality checks
 SELECT COUNT(*) INTO fail_count
 FROM (
 -- Null check
 SELECT 1 WHERE EXISTS (
 SELECT 1 FROM IDENTIFIER(:staging_table)
 WHERE order_date = CURRENT_DATE()
 AND (currency_code IS NULL OR total_amount IS NULL)
 )
 UNION ALL
 -- Negative amount check
 SELECT 1 WHERE EXISTS (
 SELECT 1 FROM IDENTIFIER(:staging_table)
 WHERE order_date = CURRENT_DATE()
 AND total_amount < 0
 )
 UNION ALL
 -- Invalid currency check
 SELECT 1 WHERE EXISTS (
 SELECT 1 FROM IDENTIFIER(:staging_table)
 WHERE order_date = CURRENT_DATE()
 AND currency_code NOT IN ('USD','EUR','GBP','AUD','CAD','JPY')
 )
 );

 IF (fail_count = 0) THEN
 -- PUBLISH: Swap (zero-copy, instant)
 EXECUTE IMMEDIATE 'ALTER ICEBERG TABLE '
 || target_table || ' SWAP WITH ' || staging_table;
 EXECUTE IMMEDIATE 'DROP TABLE IF EXISTS ' || staging_table;
 result_msg := 'PUBLISHED — All checks passed. Swap completed.';
 ELSE
 -- ROLLBACK: Drop the clone
 EXECUTE IMMEDIATE 'DROP TABLE IF EXISTS ' || staging_table;
 result_msg := 'BLOCKED — ' || fail_count
 || ' check(s) failed. Production unchanged.';
 END IF;

 RETURN result_msg;
END;
$$;

-- Run it
CALL run_wap_pipeline('fct_orders', 'fct_orders_staging');

A Note on Iceberg Branching — The Future Path

If you’ve been following the Iceberg ecosystem, you might be wondering about branching. Apache Iceberg’s spec includes a branching feature that’s purpose-built for WAP — you create a named branch within the same table, write to it in isolation, and fast-forward the main branch when audits pass. It’s like a Git pull request for your data.

The feature exists and it works beautifully in Spark and Flink. You can create branches with ALTER TABLE ... CREATE BRANCH, write to them via the spark.wap.branch session config, and publish with CALL system.fast_forward(). The semantics are cleaner than the clone-and-swap approach because the branch is the table — just a different lineage of snapshots within the same metadata.

As of writing this it appears Snowflake does not natively support Iceberg branching through its SQL engine today. The branching and tagging metadata operations are Iceberg spec features implemented in the Java library, with engine integrations for Spark and Flink. Snowflake’s Iceberg support gives you full DML, Time Travel, and schema evolution — but not yet the CREATE BRANCH or fast_forward operations.

For teams already running Spark workloads alongside Snowflake, the branching approach is worth exploring now. You run the WAP pipeline in Spark with native Iceberg branching, and Snowflake reads the published data as an externally managed Iceberg table. It’s a composable architecture where you push quality enforcement to cheaper compute and keep the warehouse for analytics.

For teams that are Snowflake-native, the clone-and-swap approach we’ve built in this tutorial gives you the vast majority of the benefits today. When Snowflake adds native branching support — and given the competitive landscape with Databricks, I’d expect it — we can revisit this with a cleaner implementation. The quality checks and audit battery you build now will carry forward regardless of the underlying publish mechanism.

Best Practices That’ll Save You Pain

After running WAP patterns in various forms across multiple teams, these are the lessons that kept surfacing.

Start with hard gates, evolve to soft gates. Initially, any check failure should block publication. Full stop. As your quality checks mature and you understand your data better, you’ll learn which checks are hard blockers — null primary keys, negative amounts, schema violations — and which are warnings worth logging but not blocking on — volume slightly above the historical average, a handful of records with an unusual but valid status. Build your audit results as structured data with severity levels so the publish decision can be nuanced.

Log quarantined records, always. Even when you drop a staging clone, capture the failing records somewhere for investigation. A quarantine table or a dead-letter S3 prefix gives your team the evidence they need to fix the source system — which is the real solution, not just filtering out bad data forever. Every quarantined record should answer three questions: what failed, which check caught it, and when it arrived.

Make your quality checks idempotent. The audit battery should be safe to run multiple times without side effects. No state mutations, no “mark as audited” flags that prevent re-runs. If the pipeline fails halfway through auditing, you should be able to restart from the top of the audit phase without creating a new clone.

Monitor your audit pass rates over time. A pipeline that fails WAP checks once a month is healthy — the pattern is catching occasional source issues. A pipeline that fails WAP checks every other day is telling you something about your upstream data source that needs fixing. Track the pass/fail ratio, the specific checks that fail most often, and the volume of quarantined records. This data is ammunition for the conversation with the source system owners.

Don’t forget the cleanup. Snowflake’s CLONE creates metadata that references the same underlying Parquet files. If you create clones and forget to drop them, you’re not duplicating storage (the Parquet files are shared), but you are accumulating metadata that Snowflake tracks. Expired clones should be dropped promptly. Set up a cleanup task that sweeps for orphaned staging tables older than your retention threshold.

Tag your successful publishes. Not Iceberg tags (Snowflake doesn’t support those natively yet), but a metadata log. Keep a table that records every successful WAP publish — the timestamp, the table name, the row count of the new batch, and the audit results. When something goes wrong downstream three weeks from now and someone asks “what changed?”, that log is your first stop.

The Bigger Picture: Data Quality as a Deploy Gate

Zach Wilson has a line that sticks with me: “Data engineering shouldn’t have any less engineering rigor than software engineering.” He’s right. We’ve spent years building CI/CD pipelines for application code — linting, unit tests, integration tests, staging environments, approval gates — and then we load data straight into production with a prayer and a Teams message that says “run looks good.”

WAP is the data equivalent of a deployment gate. You wouldn’t merge a pull request that fails tests. You shouldn’t publish data that fails quality checks. The mechanics are different — we’re swapping table metadata instead of deploying container images — but the principle is identical: validate before you publish, and make the publish step as cheap and reversible as possible.

Iceberg tables make the “as cheap as possible” part genuinely achievable. When publish is a zero-copy metadata operation that completes in under a second regardless of table size, the overhead of WAP effectively disappears. The only cost is the compute for your quality checks — and that’s compute you want to be spending.

That Tuesday afternoon with the phantom Total Portfolio growth? It cost us three days of investigation, two days of reprocessing historical data, and an unknowable amount of credibility with the Department Head and her team. Implementing WAP on that pipeline would have cost us nothing but a few lines of SQL and a staging clone that gets created and destroyed automatically every morning.

The pattern works. The tooling is ready. Start where you are, build the quality gates, and let the zero-copy publish mechanics handle the rest.

Your production data — and your Department Head — will thank you for it.

Healing Tables: When Day-by-Day Backfills Become a Slow-Motion Disaster

Chris Hillman — Sat, 07 Feb 2026 09:00:00 +1100

It was 2 AM on a Saturday when I realized we’d been loading data wrong for six months.

The situation: a customer dimension with three years of history needed to be backfilled after a source system migration. The previous team’s approach was straightforward—run the daily incremental process 1,095 times, once for each day of history. They estimated three weeks to complete.

What they hadn’t accounted for was how errors compound. By the time I looked at the data, we had 47,000 records with overlapping date ranges, 12,000 timeline gaps where customers seemed to vanish and reappear, and an unknowable number of missed changes from when source systems updated the same record multiple times in a single day.

The dimension wasn’t just wrong. It was unfixably wrong using traditional methods. Every incremental run had layered new errors on top of old ones, creating a Jenga tower of data quality issues that couldn’t be untangled without starting over.

That night, I started building what I now call a Healing Table—a dimension that can be completely rebuilt from source data at any point, “healing” whatever accumulated inconsistencies have crept in over months or years of incremental loads.

This isn’t just a technique for disaster recovery. It’s a fundamentally different approach to SCD Type 2 that separates change detection from period construction, processes entire historical ranges in a single pass, and produces dimensions that are deterministically reproducible from source data.

Here’s how it works.

Why Day-by-Day Backfills Fail

The traditional approach to backfilling historical SCD2 dimensions follows a pattern most data engineers know well: take your daily incremental process, wrap it in a loop, and execute it once for each historical date. Simple, reuses existing code, and catastrophically fragile.

The problems start immediately.

Errors compound across iterations. If your change detection logic misses an edge case on January 3rd, that incorrect record becomes the baseline for January 4th’s comparison. The error doesn’t stay contained—it propagates forward through every subsequent run. By the time you discover the issue months later, you can’t simply fix January 3rd because hundreds of downstream records depend on that incorrect state.

Performance degrades non-linearly. Each daily run needs to compare incoming records against the existing dimension. As the dimension grows through backfill, comparison costs increase. A process that takes 5 minutes for a single day’s delta might take 45 minutes per day when the target table contains three years of history. That three-week estimate becomes three months.

Source system quirks multiply. Real source systems don’t change data once per day at a predictable time. They update records multiple times, delete and recreate rows, and occasionally backdate changes. Day-by-day processing either misses these patterns entirely or handles them inconsistently across runs.

Recovery requires complete rebuild anyway. When—not if—something goes wrong, fixing it requires blowing away the dimension and starting over. But if you’re going to rebuild from scratch regardless, why not design for that from the beginning?

The Healing Tables framework embraces this reality. Instead of trying to prevent rebuilds, it makes them fast, reliable, and deterministic.

The Six-Step Framework

Healing Tables work by separating two concerns that traditional SCD2 implementations conflate: change detection (identifying when attributes changed) and period construction (building valid time slices with proper start and end dates).

Traditional approaches detect changes and construct periods simultaneously, comparing incoming records to existing dimension state. This creates tight coupling between current dimension contents and processing logic—if the dimension is wrong, future processing will also be wrong.

Healing Tables decouple these concerns through a six-step pipeline that operates entirely on source data, constructing the dimension from scratch without reference to any existing target state:

Effectivity Table Creation — Extract all change points from sources
Time Slice Generation — Build date ranges with proper valid_from/valid_to
Source Table Joining — Conform attributes from multiple sources
Hash Computation — Enable efficient change detection
Row Compression — Eliminate consecutive identical states
Validation Testing — Verify temporal integrity before loading

Let me walk through each step.

Step 1: Building the Effectivity Table

The Effectivity Table captures every moment when any tracked attribute changed for any business key. Think of it as a timeline of “something happened here” markers that we’ll later fill in with actual attribute values.

For sources with explicit timestamps, extraction looks like this:

-- Extract all change points from source
WITH source_ordered AS (
 SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,credit_limit
 ,updated_at AS change_timestamp
 ,ROW_NUMBER() OVER (
 PARTITION BY customer_id
 ORDER BY updated_at
 ) AS version_sequence
 FROM
 raw_customers
 WHERE
 updated_at IS NOT NULL
),

-- Detect actual changes using LAG comparison
change_detection AS (
 SELECT
 *
 ,LAG(customer_name) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ) AS prev_name
 ,LAG(customer_status) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ) AS prev_status
 ,LAG(credit_limit) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ) AS prev_credit_limit
 FROM
 source_ordered
)

-- Keep only first record OR records where attributes actually changed
SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,credit_limit
 ,change_timestamp
FROM
 change_detection
WHERE
 version_sequence = 1
 OR customer_name IS DISTINCT FROM prev_name
 OR customer_status IS DISTINCT FROM prev_status
 OR credit_limit IS DISTINCT FROM prev_credit_limit

The critical insight here is filtering to actual changes only. Raw source tables often contain records where nothing meaningful changed—perhaps a batch process touched every row, or a status field was updated and then immediately reverted. Including these non-changes creates unnecessary dimension versions that complicate queries and waste storage.

When working with daily snapshot sources (where you don’t have change timestamps, just daily extracts), the pattern shifts to detecting differences between consecutive snapshots:

WITH daily_ordered AS (
 SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,snapshot_date
 ,LAG(customer_name) OVER (
 PARTITION BY customer_id
 ORDER BY snapshot_date
 ) AS prev_name
 ,LAG(customer_status) OVER (
 PARTITION BY customer_id
 ORDER BY snapshot_date
 ) AS prev_status
 FROM
 daily_customer_snapshots
)

SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,snapshot_date AS effective_from
FROM
 daily_ordered
WHERE
 prev_name IS NULL -- First occurrence
 OR customer_name IS DISTINCT FROM prev_name
 OR customer_status IS DISTINCT FROM prev_status

The IS DISTINCT FROM operator handles NULL comparisons correctly—a common source of bugs in change detection logic. Standard != treats NULL as unknown, meaning NULL != 'value' returns NULL rather than TRUE. IS DISTINCT FROM treats NULL as a concrete value, giving us the comparison semantics we actually want.

Step 2: Generating Time Slices

Time Slices transform point-in-time change events into date ranges with valid_from and valid_to columns. The LEAD window function does the heavy lifting, calculating when each version expires based on when the next version begins:

WITH effectivity_data AS (
 SELECT
 customer_id, customer_name, customer_status, change_timestamp
 FROM
 effectivity_table
)

SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,change_timestamp AS valid_from
 -- Calculate valid_to from next version's start date
 ,COALESCE(
 LEAD(change_timestamp) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ),
 DATE '9999-12-31' -- High date for current records
 ) AS valid_to
 -- Flag current record for easy filtering
 ,CASE
 WHEN LEAD(change_timestamp) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ) IS NULL THEN TRUE
 ELSE FALSE
 END AS is_current
FROM
 effectivity_data

A few implementation decisions deserve attention here.

High dates versus NULLs for current records. Using 9999-12-31 for current record end dates simplifies BETWEEN queries but can confuse some BI tools that try to calculate date differences. Using NULL for valid_to has clearer semantic meaning but requires IS NULL predicates everywhere. My recommendation: use the high date for the column value, but also include an explicit is_current boolean flag. This gives query authors flexibility without forcing them to remember magic date values.

Interval conventions matter. Use left-closed, right-open intervals where valid_from <= date < valid_to. This ensures every point in time maps to exactly one dimension version without ambiguity at boundaries. The alternative—inclusive on both ends—creates situations where midnight timestamps belong to two periods simultaneously.

Point-in-time lookup pattern:

SELECT d.*
FROM
 dim_customer d
WHERE
 @lookup_date >= d.valid_from
 AND @lookup_date < d.valid_to

Step 3: Joining Multiple Sources

When dimension attributes come from multiple systems, you need to conform those sources before SCD2 processing begins. The temptation is to load directly from multiple sources into a single target—don’t do this. You’ll create conflicting versions where the same timestamp has different attribute combinations depending on which source processed first.

The solution is a Unified Timeline that collects all change points from all sources, then joins each source back using as-of-date logic:

-- Step 1: Collect ALL change points from ALL sources
WITH unified_change_points AS (
 SELECT customer_id, change_timestamp FROM crm_customers
 UNION
 SELECT customer_id, change_timestamp FROM erp_customers
 UNION
 SELECT customer_id, change_timestamp FROM web_profiles
),

-- Step 2: Build time slices from unified timeline
unified_time_slices AS (
 SELECT
 customer_id
 ,change_timestamp AS valid_from
 ,COALESCE(
 LEAD(change_timestamp) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ),
 TIMESTAMP '9999-12-31 23:59:59'
 ) AS valid_to
 FROM
 unified_change_points
),

-- Step 3: Join each source back using as-of logic
final_dimension AS (
 SELECT
 ts.customer_id
 ,ts.valid_from
 ,ts.valid_to
 -- CRM attributes (authoritative for name)
 ,crm.customer_name
 -- ERP attributes (authoritative for financial data)
 ,erp.credit_limit
 ,erp.payment_terms
 -- Web attributes (authoritative for contact preferences)
 ,web.email_opt_in
 ,web.preferred_contact_method
 FROM
 unified_time_slices ts

 LEFT JOIN crm_customers crm
 ON ts.customer_id = crm.customer_id
 AND crm.change_timestamp = (
 SELECT MAX(change_timestamp)
 FROM crm_customers
 WHERE customer_id = ts.customer_id
 AND change_timestamp <= ts.valid_from
 )

 LEFT JOIN erp_customers erp
 ON ts.customer_id = erp.customer_id
 AND erp.change_timestamp = (
 SELECT MAX(change_timestamp)
 FROM erp_customers
 WHERE customer_id = ts.customer_id
 AND change_timestamp <= ts.valid_from
 )

 LEFT JOIN web_profiles web
 ON ts.customer_id = web.customer_id
 AND web.change_timestamp = (
 SELECT MAX(change_timestamp)
 FROM web_profiles
 WHERE customer_id = ts.customer_id
 AND change_timestamp <= ts.valid_from
 )
)

SELECT * FROM final_dimension

Notice the correlated subqueries for each join—we’re finding the most recent version from each source that was effective at each unified timeline point. This handles cases where one source updates frequently while another updates rarely.

Document attribute ownership explicitly. When CRM and ERP both have a customer name field, which wins? When web profile email conflicts with CRM email, which is authoritative? These decisions belong in documentation and code comments, not in tribal knowledge. Build a table:

Attribute	Primary Source	Fallback	Resolution Rule
Customer Name	CRM	ERP	CRM always wins
Credit Limit	ERP	None	ERP authoritative
Email	Web	CRM	Most recent wins

Then implement those rules explicitly in your SQL using COALESCE or CASE expressions.

Step 4: Hash Computation

Hash-based change detection replaces expensive multi-column comparisons with single-column hash comparisons. The Healing Tables framework uses a two-hash strategy: one hash for business key identification, another for attribute change detection.

key_hash identifies the business entity. Hash the natural key columns together:

-- Trino/Starburst syntax
lower(to_hex(sha256(to_utf8(
 COALESCE(CAST(source_system AS VARCHAR), '') || '|' ||
 COALESCE(CAST(customer_id AS VARCHAR), '')
)))) AS key_hash

row_hash detects when any tracked attribute changes. Hash all business attributes excluding keys and metadata:

lower(to_hex(sha256(to_utf8(
 COALESCE(customer_name, '^^NULL^^') || '|' ||
 COALESCE(customer_status, '^^NULL^^') || '|' ||
 COALESCE(CAST(credit_limit AS VARCHAR), '^^NULL^^') || '|' ||
 COALESCE(CAST(birth_date AS VARCHAR), '^^NULL^^')
)))) AS row_hash

The ^^NULL^^ placeholder distinguishes NULL values from empty strings—a critical detail that trips up many implementations. Without this, ('John', NULL, 'Active') and ('John', '', 'Active') produce identical hashes despite having semantically different data.

Why not use the built-in CONCAT_WS? Because CONCAT_WS skips NULL values entirely. The inputs ('A', NULL, 'B') and ('A', 'B') produce identical output, creating false positive matches where none exist. Always wrap columns in COALESCE before concatenation.

Column ordering best practice: concatenate in alphabetical order. This creates deterministic, reproducible hashes that survive schema evolution—adding a new column doesn’t change the hash of existing columns.

Platform-specific hash functions vary:

Platform	Function	Syntax
Snowflake	SHA2	`SHA2(expression, 256)`
BigQuery	SHA256 + TO_HEX	`TO_HEX(SHA256(expression))`
Trino/Starburst	sha256 + to_hex	`to_hex(sha256(to_utf8(expression)))`
SQL Server	HASHBYTES	`HASHBYTES('SHA2_256', expression)`

Step 5: Row Compression

Row compression eliminates consecutive duplicate states—adjacent time periods where all attributes are identical. This happens more often than you’d expect: source systems frequently touch records without changing meaningful data, or changes get reverted within the same processing window.

The technique is a classic Islands and Gaps pattern. Assign a group identifier to runs of consecutive identical rows, then collapse each group into a single time slice:

-- Step 1: Detect group boundaries
WITH boundary_detection AS (
 SELECT
 *
 ,CASE
 -- New group if: periods don't connect OR attributes changed
 WHEN LAG(valid_to) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) = valid_from
 AND row_hash = LAG(row_hash) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 )
 THEN 0 -- Continuation of previous group
 ELSE 1 -- Start of new group
 END AS is_group_start
 FROM
 dimension_with_hashes
),

-- Step 2: Assign group IDs via running sum
group_assignment AS (
 SELECT
 *
 ,SUM(is_group_start) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ROWS UNBOUNDED PRECEDING
 ) AS group_id
 FROM
 boundary_detection
)

-- Step 3: Collapse each group to single row
SELECT
 customer_id
 ,MIN(valid_from) AS valid_from
 ,MAX(valid_to) AS valid_to
 ,MAX(CASE WHEN is_current THEN 1 ELSE 0 END) = 1 AS is_current
 ,MAX(customer_name) AS customer_name -- All identical within group
 ,MAX(customer_status) AS customer_status
 ,MAX(row_hash) AS row_hash
 ,MAX(key_hash) AS key_hash
 ,COUNT(*) AS compressed_row_count -- Audit trail
FROM
 group_assignment
GROUP BY
 customer_id, group_id
ORDER BY
 customer_id, valid_from

Two conditions must both be true for rows to compress:

Temporal contiguity: Previous row’s valid_to equals current row’s valid_from
Attribute identity: row_hash values match

The contiguity check prevents incorrectly merging records that happen to have identical attributes but represent separate valid periods. If a customer had credit limit 10000 from January to March, then credit limit 5000 from March to June, then credit limit 10000 again from June to December—those are three distinct business states, not one, even though the first and third have identical attribute values.

Before compression (6 rows):

customer_id	valid_from	valid_to	credit_limit
1002	2020-01-01	2020-01-05	40000
1002	2020-01-05	2020-01-09	40000
1002	2020-01-09	2020-01-11	30000
1002	2022-01-01	2022-03-01	30000
1002	2022-03-01	2022-05-01	30000
1002	2022-05-01	2022-06-01	30000

After compression (3 rows):

customer_id	valid_from	valid_to	credit_limit
1002	2020-01-01	2020-01-09	40000
1002	2020-01-09	2020-01-11	30000
1002	2022-01-01	2022-06-01	30000

The gap between 2020-01-11 and 2022-01-01 correctly prevents merging the two “30000” groups—they represent distinct validity periods separated by time when the customer didn’t exist in source data.

Step 6: Validation Testing

Production SCD2 dimensions require comprehensive validation to ensure temporal integrity. These tests should run after every load and block pipeline completion on failure. Silent failures in dimension loading corrupt downstream fact tables in ways that are extremely difficult to diagnose later.

Test 1: Single current record per key

Each business key must have exactly one active (current) record:

-- CRITICAL: Fails if any key has multiple current records
SELECT
 customer_id
 ,COUNT(*) AS current_record_count
FROM
 dim_customer
WHERE
 is_current = TRUE
GROUP BY
 customer_id
HAVING
 COUNT(*) <> 1

-- Expected: Zero rows returned

Test 2: No overlapping date ranges

Time periods for the same key must not overlap. This is where I learned a painful lesson: the obvious LEAD-based check is insufficient.

-- INSUFFICIENT - Only checks adjacent records
WITH overlap_check AS (
 SELECT
 customer_id
 ,valid_from
 ,valid_to
 ,LEAD(valid_from) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) AS next_valid_from
 FROM
 dim_customer
)
SELECT *
FROM
 overlap_check
WHERE
 next_valid_from <= valid_to

This misses overlaps between non-adjacent records. Records at positions 1 and 3 could overlap even if record 2 doesn’t overlap with either. The comprehensive check requires a self-join:

-- COMPREHENSIVE - Checks all pairs
SELECT a.*, b.*
FROM
 dim_customer a

 INNER JOIN dim_customer b
 ON a.customer_id = b.customer_id
 AND a.dbt_scd_id != b.dbt_scd_id -- Different records
WHERE
 a.valid_from < b.valid_to
 AND a.valid_to > b.valid_from

-- Expected: Zero rows returned

Test 3: No timeline gaps

For dimensions requiring continuous history, adjacent periods must connect:

WITH gap_check AS (
 SELECT
 customer_id
 ,valid_from
 ,LAG(valid_to) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) AS prev_valid_to
 FROM
 dim_customer
)
SELECT
 customer_id
 ,prev_valid_to AS gap_start
 ,valid_from AS gap_end
FROM
 gap_check
WHERE
 prev_valid_to IS NOT NULL
 AND
 prev_valid_to <> valid_from

-- Expected: Zero rows (or documented acceptable gaps)

Test 4: Valid date ordering

Every record must have valid_from < valid_to:

SELECT *
FROM
 dim_customer
WHERE
 valid_from >= valid_to

-- Expected: Zero rows

Test 5: No consecutive duplicate versions

After compression, no adjacent rows should have identical attributes:

WITH version_check AS (
 SELECT
 customer_id
 ,valid_from
 ,row_hash
 ,LAG(row_hash) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) AS prev_row_hash
 ,LAG(valid_to) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) AS prev_valid_to
 FROM
 dim_customer
)
SELECT *
FROM
 version_check
WHERE
 row_hash = prev_row_hash
 AND valid_from = prev_valid_to -- Contiguous AND identical

-- Expected: Zero rows

In dbt, implement these as reusable tests:

# schema.yml
models:
 - name: dim_customer
 tests:
 - dbt_utils.unique_combination_of_columns:
 combination_of_columns:
 - customer_id
 - valid_from
 columns:
 - name: customer_id
 tests:
 - unique:
 config:
 where: "is_current = TRUE"
 - name: valid_from
 tests:
 - not_null
 - name: valid_to
 tests:
 - not_null

The Self-Healing Property

The name “Healing Tables” comes from a property that emerges from this design: dimensions built this way can recover from accumulated data quality issues simply by reprocessing source data.

Traditional incremental SCD2 implementations are path-dependent—the current state depends on the exact sequence of historical loads. If Tuesday’s load had a bug, and Wednesday through Friday ran correctly, you can’t just re-run Tuesday. Those later loads built on Tuesday’s incorrect state.

Healing Tables are path-independent. The dimension produced depends only on source data contents, not on the history of load operations. Re-run any date range with corrected logic, and the result is identical to what you’d get from a full rebuild.

This property transforms how teams think about data quality issues:

Bug in change detection logic? Fix it, reprocess affected date range, done.
Source system correction applied retroactively? Reprocess includes those corrections automatically.
New attribute added to dimension? Reprocess rebuilds all historical versions with the new column.
Suspected data corruption? Rebuild and compare—if results differ, you’ve found your problem.

The dimension becomes reproducible in the same way good software builds are reproducible. Given the same inputs and logic, you always get the same outputs. This isn’t just convenient for debugging—it’s a fundamental quality characteristic that enables reliable data pipelines.

When to Use This Approach

Healing Tables aren’t appropriate for every situation. They work best when:

You have access to complete source history. The pattern assumes you can extract all historical changes, not just deltas. If your source system only provides “current state” with no history, you’ll need a different approach.
Source data is contiguous. The timeline construction assumes no gaps in source timestamps. If customers can appear, disappear, and reappear in source data, you’ll need additional logic to handle these discontinuities.
Rebuild time is acceptable. Processing years of history takes longer than processing a single day’s changes. For dimensions with billions of historical rows, full rebuilds might be impractical even with optimization.
Your sources don’t have hard deletes. The pattern assumes deletions are logical (flagged) rather than physical. If source systems physically remove rows, you’ll lose that history.

Traditional incremental approaches remain appropriate for:

Very high-volume dimensions where full rebuilds are impractical
Real-time or near-real-time requirements
Situations where source history is unavailable
Simple dimensions with stable, well-understood source systems

The choice isn’t binary. Many teams use Healing Tables for periodic “refresh” operations while running standard incremental processes daily. If the daily process accumulates issues, the weekly or monthly heal repairs them.

Wrapping Up

That 2 AM debugging session taught me something important: the traditional approach to SCD2—incremental changes applied to existing dimension state—creates tight coupling between historical load operations and current data quality. Every bug, every edge case, every source system quirk gets baked into the dimension permanently.

Healing Tables break that coupling. By extracting change points from source data, constructing time slices independently, compressing duplicate states, and validating results before loading, you create dimensions that are deterministically reproducible from source data.

The six-step framework isn’t complicated, but it requires discipline:

Build the Effectivity Table with actual changes only
Generate Time Slices with proper valid_from/valid_to boundaries
Join multiple sources using unified timeline logic
Compute hashes for efficient change detection
Compress consecutive identical states
Validate temporal integrity before loading

Implement those steps correctly, and you’ll have dimensions that can heal themselves from whatever accumulated issues have crept in over months or years of operation.

The next time someone asks you to backfill three years of history by running the daily process 1,095 times, you’ll have a better answer.

For Sooty

Chris Hillman — Tue, 03 Feb 2026 09:00:00 +1100

This one isn’t about data pipelines. There’s no framework, no architecture diagram, no code snippet at the end.

Yesterday I said goodbye to my best friend.

Sooty was a miniature schnauzer — nine kilograms of stubbornness, loyalty, and heart. Born October 2011. Gone February 2026. Fourteen years that changed the shape of everything.

I don’t have the right words for this. I’m not sure anyone does. But I tried to write something that comes close, and I wanted to share it here — because this is my corner of the internet, and she deserves a place in it.

Sooty

There is something about having a dog that nobody warns you about. They don’t tell you that the first time she falls asleep with her head on your lap, something inside your chest will rearrange itself, and from then on there will always be a space in you that is exactly the shape of her.

She is going to know where you are. By the sound of your keys in the door, by the way you sigh when you sit down, by the particular shuffle of your feet on the kitchen floor at six in the morning. She will never once say your name and you will never once doubt that she knows it.

And every time you come home — every single time, whether you were gone for eight hours or eight minutes — she will greet you at the door with the excitement and joy and emotion of a relative she hasn’t seen in years. As though your return is the most remarkable thing that has ever happened. As though the world had gone quiet without you and now, finally, it is loud again.

She will follow you throughout the house, every room, no matter how small the journey — just to check you’re still there. She will sit beside you during every bad movie, every 2am bout of insomnia when the ceiling has nothing useful to offer. She will follow you into rooms you didn’t even mean to walk into and look up at you as if to say, Well? We’re here now. What’s the plan?

She is your partner in crime before you even commit one. She will be at the door before you even reach for the leash. She will bark at the mailman every time she sees him, because someone has to warn him — my house — and she has appointed herself to the role.

And there will be days when the world does what the world does best, which is to come at you sideways and keep swinging. Days when the meetings don’t stop, when the inbox is on fire, when your body feels like it was assembled wrong.

And on those days, she will not ask you what happened. She will not offer advice. She will press her whole warm ridiculous body against yours and stay. Just stay. And occasionally lick your hand — not to fix you, but to say I’m checking. I’m still here. Are you still here?

And yes, she will also be the reason you cannot sleep in. Every morning a fresh day — a stretch with energy, a shake, and a tail wag, sometimes all at once, as though the night never happened and everything is new again.

She will remind you that the sun has come up and that this, apparently, is cause for celebration. She will remind you that her bowl is empty and that this, apparently, is an emergency.

She will stand at the door and look back at you with those unbearable eyes and you will put on your shoes even when you don’t want to, and you will step outside even when the sky is grey, and you will walk — because she asked, because she needs you, because somehow the needing is the gift. And every walk a new adventure. She knew where to go. Lead on.

Sometimes she sniffed where she shouldn’t have. Under the fence — and got bitten. The neighbour’s cat — and got a claw slap across the nose. But she bounced back up every time, because that is the thing about dogs: they do not hold grudges against the world for being sharp. They just shake it off and go looking for the next thing to love.

When she was young she would whimsically play throughout the house, tearing from room to room like joy had legs, and then — no matter where she was, middle of the floor en route to the bowl, or outside in the windy grass — she would zonk out. Batteries depleted. Gone. As though living that hard required sleeping that deep.

And as she got older she would remind you — time for bed, come on — at 7pm. A bit too early for you, but the right time for her. And you would learn that she was not wrong about most things, and maybe not wrong about this either.

She will see you through the worst of it. She will be there when the phone call comes. She will be there when you can’t get off the couch. She will be there when you forget what it feels like to be okay, lying beside you like a kind of proof — that warmth exists, that loyalty is not a theory, that someone in this world looked at you and thought yes, this one, always this one, no matter what.

And she will be there for the best of it too. She will be in the background of your happiest memories, tail going, ears up, a blur of joy in the corner of every photograph. She will teach you that happiness is not complicated. That it lives in tennis balls and puddles and the space between your hand and the top of her head.

And here is the part they really should warn you about.

She will teach you that love does not need language. That it does not need to be earned or explained or defended. That it can just exist, simple and enormous, in the weight of a head on your chest, in the sound of breathing that is not yours but that has become as necessary as your own.

She will teach you all of this, and she will teach you so well that by the time you have to let her go, you will understand exactly what you are losing.

And I think that is why it is so hard to say goodbye. Not because we didn’t know it was coming. But because they spent every single day of their lives making sure we knew exactly how much there was to love.

And they were right. And we did. And it was everything.

Rest easy, Sooty. You were the best girl.

What an NBA Coach Can Teach Data Leaders About Building Teams That Actually Work

Chris Hillman — Mon, 02 Feb 2026 09:00:00 +1100

I was three hours into a retrospective that had devolved into blame-shifting when the most senior engineer on the team finally spoke up. “Look,” he said, “we can keep pointing fingers at the data model, or we can admit we don’t actually trust each other enough to have an honest conversation about what went wrong.”

The room went quiet. He was right.

That moment stuck with me because it exposed something I’ve seen destroy more data teams than bad architecture ever could: the absence of genuine connection between people who spend forty-plus hours a week depending on each other.

We obsess over tool selection. We debate medallion architectures until our Teams threads reach scroll-fatigue territory. We write detailed runbooks and build elaborate monitoring systems. But we rarely invest in the thing that actually determines whether a data team succeeds or fails—whether the people on it feel like they belong to something worth sacrificing for.

A while back, I stumbled across a deep dive into Gregg Popovich’s leadership approach. If you don’t follow basketball, Pop coached the San Antonio Spurs to five NBA championships across twenty-two consecutive playoff appearances—the most sustained dynasty in modern professional sports. His winning percentage over three decades is unmatched in North American sports.

And his secret? It wasn’t X’s and O’s. It was wine, tears, and showing up at a player’s hotel room at 2 AM to cry with him after his father died.

What I found wasn’t a basketball story at all. It was a masterclass in building teams that actually work. And everything in it applies to leading data engineers—if you’re willing to do the uncomfortable work.

The problem with how we lead data teams

When managing data team’s: we’ve inherited a leadership model built for factory floors and sales quotas, then awkwardly grafted it onto creative, highly collaborative technical work.

The standard playbook goes something like this:

Set clear goals. Define KPIs. Track story points. Measure velocity. Hold quarterly reviews where you show charts that go up and to the right (or explain why they didn’t).

Provide feedback. Annual performance reviews. Maybe some lightweight 360 input. A few uncomfortable conversations about “areas for growth.”

Remove blockers. Clear the technical debt backlog. Negotiate for headcount. Shield the team from stakeholder noise.

None of this is wrong, exactly. But it’s not sufficient.

Data teams live in a peculiar tension. The work is deeply technical—you need people who genuinely understand distributed systems, query optimization, and schema design. But the outcomes are entirely social—dashboards nobody uses, pipelines that don’t answer the questions stakeholders actually have, data quality that erodes trust with every “Why do these numbers look different?” conversation.

Success requires people who can hold both realities simultaneously. And that kind of cognitive flexibility only emerges in environments where people feel safe enough to be vulnerable, connected enough to sacrifice personal metrics for collective success, and invested enough to push back when the team is heading in the wrong direction.

Popovich understood this intuitively. His teams consistently featured players who took less money, accepted smaller roles, and subordinated personal statistics for championships. How? Not through inspirational speeches or performance bonuses. Through dinners, emotional presence, and a relentless refusal to separate the person from the player.

Belonging cues: the invisible architecture of high-performing teams

Daniel Coyle, who studied Popovich extensively for his book The Culture Code, identified three types of signals that create what he calls “belonging cues”:

Personal, up-close connection — behavior that communicates “I care about you as a person”
Performance feedback — relentless criticism that says “we have high standards here”
Big-picture perspective — conversations that say “life is bigger than this immediate task”

Most data leaders nail one of these and completely miss the others.

The technically-focused leader provides excellent feedback on code quality and architecture decisions but treats personal connection as optional—something that happens organically in Teams channels or maybe at the occasional team happy hour.

The people-focused leader builds genuine relationships but struggles to deliver hard feedback, allowing mediocrity to persist because confrontation feels mean.

The visionary leader connects everything to grand purpose—“we’re democratizing data access across the organization!"—but the lofty framing feels hollow when day-to-day work involves debugging pipeline failures at 10 PM on a Friday.

Popovich weaves all three together constantly. The same evening might include crying with a player over a family tragedy, reviewing film and yelling about a missed rotation, and discussing the Civil Rights Movement. As one assistant coach put it: “He’ll tell you the truth, with no bullshit, and then he’ll love you to death.”

This isn’t contradictory. It’s complete.

Sound familiar? It should. Google’s Project Aristotle research found essentially the same thing—that psychological safety (the belief that you won’t be punished for taking risks or admitting mistakes) was the single strongest predictor of team performance. But psychological safety isn’t warm fuzziness. It’s the confidence that your team has high standards and will support you through the inevitable failures that come from pushing against those standards.

“Get over yourself” — the four-word philosophy that builds dynasties

Popovich’s core belief reduces to four words: “Get over yourself.”

It sounds harsh. But it’s actually an invitation to freedom.

“It’s not about any one person,” he explains. “You’ve got to get over yourself and realize that it takes a group to get this thing done.”

When evaluating potential players, he watched for this quality specifically: “When there’s a guy who talks about himself all day long, you start to get the sense that he doesn’t listen real well… Has this person gotten over himself? If he has then he’s going to accept parameters. He’s going to accept the role.”

This is the hardest cultural shift for data teams to make.

We work in a field that rewards individual technical brilliance. The engineer who builds the elegant solution gets recognition. The one who writes the clever query gets quoted in architecture reviews. We celebrate the heroic debugging session that saved the quarterly report, rarely asking why the pipeline was fragile enough to need heroics in the first place.

Getting over yourself means accepting that the best data architecture might not showcase your technical sophistication. It means implementing the boring, proven solution instead of the interesting one. It means writing documentation instead of starting a new project. It means saying “I don’t know” instead of confidently bullshitting through a stakeholder meeting.

Manu Ginobili—an All-Star caliber player—started only 349 of 1,057 games because the team needed him as a sixth man. When the coaching staff asked him to permanently accept the bench role, they told him honestly: if he wasn’t good with it, he was going to start. Whatever he said, they would do it. He deserved that choice.

Ginobili agreed to come off the bench.

Tim Duncan was “blown away”: “Are you kidding? He’s Manu! He’s a star! He can’t not start.” But Ginobili embraced the sacrifice because he understood something crucial: his ego was getting between him and the team’s success.

How many data engineers would make that trade? How many would accept that their personal visibility, their resume-building features, their chance to work on the exciting project needed to take a back seat to what the team actually needed?

The uncomfortable truth: they’ll only make that sacrifice if they trust you enough to believe it’s genuine. If they feel like the sacrifice is worth it because they belong to something worth sacrificing for.

The dinner table as sacred ground

Popovich spends an estimated seven figures annually on food and wine for team gatherings. After road games, the team frequently stays overnight specifically to share dinner, with Pop personally scouting restaurants weeks in advance.

These aren’t casual meals. He positions himself at tables of exactly six people—“believing this number fosters diversity of conversation without people breaking off into side chats”—and works the room methodically. The dinners serve a strategic purpose: “Dinners help us have a better understanding of each individual person, which brings us closer to each other—and, on the court, understand each other better.”

Now here’s the challenge for modern data leaders: how do you create this in a world where your team might be spread across six time zones?

You can’t replicate the intimacy of shared meals and expensive wine. But you can replicate the intention behind them.

What Popovich’s dinners actually accomplish:

Dedicated time for connection that isn’t about work deliverables
Small enough groups to have real conversations
Physical proximity and eye contact (or as close as you can get)
Consistent repetition that builds cumulative trust
Leader modeling vulnerability by being genuinely present

Translation for distributed data teams:

Intentional one-on-ones that go beyond status updates. Not “what are you working on?” but “what’s actually going on with you?” Schedule them consistently. Protect them from cancellation. Come prepared with questions about their life outside work.

Small-group video calls for real conversation. Skip the all-hands. Create rotating groups of 4-6 people for unstructured time. No agenda. No recording. Just talking.

Occasional in-person gatherings that prioritize relationship-building. If you bring the team together once a year, don’t fill the schedule with strategy sessions and team-building exercises. Create space for meals, walks, and conversations that would feel weird to have over Zoom.

Consistent rituals that build cumulative connection. A Slack channel where people share weekend plans. A monthly “show and tell” of something non-work related. Anything that repeats often enough to become meaningful.

The point isn’t to replicate Popovich’s exact methods. The point is to be as intentional about relationship-building as you are about sprint planning.

When tragedy strikes, leaders appear

The most powerful evidence of Popovich’s approach emerges during personal crises.

When DeMar DeRozan’s father passed away during the 2021 season, DeRozan called the team’s GM privately—“I didn’t want nobody to know.” Ninety seconds later, Pop was knocking on his hotel room door. “Pop sat in the room with me and cried with me for about two hours. He was like, ‘I’m not leaving until you leave.’”

When Dejounte Murray’s mother was shot during his rookie year, Popovich called her directly—without Murray’s knowledge—offering to move her to San Antonio on his own dime.

When Robert Horry’s daughter was hospitalized at the start of a season, Popovich told him: “Don’t come back until she’s out of the hospital.” Three weeks later, upon returning, Pop asked if she was out. Horry said yes. Pop responded: “She ain’t out of the woods yet. Go back home. We don’t really need you right now.”

“Family is the most important thing.”

These aren’t management techniques. They’re expressions of genuine care. And they’re the reason players sacrifice for Popovich in ways they wouldn’t for other coaches.

Data leaders: when was the last time you showed up for someone beyond work boundaries?

I’m not suggesting you need to cry with your direct reports or offer to pay for their family members’ relocation. But I am suggesting that most of us dramatically underinvest in being present during the moments that actually matter.

The life events where showing up builds lasting trust:

Family emergencies and deaths
Health crises (theirs or family members')
Major life transitions (new babies, divorces, moves)
Career anxieties and disappointments
Personal struggles they’re brave enough to share

What “showing up” looks like in practice:

Proactively reaching out, not waiting for them to come to you
Asking what they need, not assuming you know
Offering flexibility without making them negotiate for it
Following up later to see how they’re doing
Remembering the details and checking in over time

The leaders I’ve worked with who inspire genuine loyalty share one trait: they remember the human stuff. They ask about your kid’s soccer tournament. They notice when you seem off. They don’t treat personal life as interference with work productivity.

High standards and deep care aren’t opposites

Here’s the false dichotomy that destroys most leadership: you’re either a demanding, results-oriented leader who pushes for excellence, or you’re a supportive, people-first leader who prioritizes relationships. Pick your lane.

Popovich refuses the choice.

Boris Diaw once visited Tony Parker at Pop’s house for Christmas dinner. After a warm meal, both Parker and Popovich disappeared. Diaw went searching and found Pop reviewing film with Tony about the game the night before—and yelling at him about missed shots and turnovers.

Diaw’s takeaway captures the essence: “On the same night, you could have the family setting, all the love and the care, and at the same time caring about making Tony a better player.”

The warmth makes the feedback land differently. Not softer—still direct, still pointed, still demanding. But the player receiving it knows it comes from someone who genuinely wants them to succeed, who has invested in them as a complete person, who will be there for the hard stuff outside work.

Research backs this up. Amy Edmondson’s psychological safety research shows that high-performing teams combine high standards with high support—“learning zones” where people push themselves precisely because they feel safe enough to fail.

For data leaders, this means:

Being specific and direct about performance issues without softening the message into meaninglessness. “The pipeline you built has reliability problems that are affecting stakeholder trust” is clearer and more respectful than “there might be some opportunities to improve things.”

Providing the context behind criticism so it lands as investment rather than attack. “I’m telling you this because you’re capable of leading our data platform work, and these patterns would prevent that.”

Following hard conversations with continued connection rather than distance. The relationship doesn’t end after difficult feedback—it continues.

Modeling receiving feedback yourself so the team sees that criticism flows in all directions. Pop regularly admitted his own failures publicly. Your team needs to see you do the same.

Connecting work to something larger

Before the 2014 NBA Finals, Popovich skipped X’s and O’s entirely. Instead, he told his team about Eddie Mabo Day—a significant date for Indigenous Australians—to honor point guard Patty Mills’ heritage.

This reflects Pop’s broader practice of engaging players on issues far beyond basketball. He’s taken teams to see Hamilton on Broadway. He brought civil rights activist John Carlos to speak. He gave players Ta-Nehisi Coates’ Between the World and Me to read.

Why does this matter for building teams?

Because it sends a message: we are complete human beings who care about the world beyond our immediate work.

Coyle noted: “Popovich would create similar conversations on the war in Syria, or a change of government in Argentina, gay marriage, institutional racism, terrorism—it doesn’t really matter, as long as it delivers the message he wants to deliver: There are bigger things than basketball to which we are all connected.”

This is the belonging cue that most data leaders miss entirely.

We act as if the only things worth discussing are data models, pipeline reliability, and stakeholder requirements. We treat outside interests as distractions rather than windows into who our team members actually are.

For data teams, connecting to something larger might include:

Discussions about the ethical implications of data work (privacy, algorithmic bias, surveillance)
Conversations about industry trends and where the field is heading
Space for people to share what they care about outside work
Recognition of cultural backgrounds and experiences
Honest conversation about work-life balance, burnout, and sustainability

None of this requires seven-figure wine budgets. It requires treating your team members as complete humans who exist in a world larger than your Jira board.

The evidence speaks across decades

Popovich’s methods produce measurable results. Five championships. Twenty-two consecutive playoff appearances. Eighteen straight 50-win seasons—unprecedented in NBA history. The best winning percentage of any major professional sports franchise in North America over a three-decade span.

The 2014 championship team became the first since the ABA-NBA merger to win a title without any player averaging 30+ minutes per game. Four different players led the team in scoring during the playoffs. No dominant statistics—just beautiful, unselfish basketball.

Beyond the Spurs, his coaching tree now shapes the entire league. When The Athletic surveyed NBA players on which coach they’d most want to play for, Popovich received 41%—the highest percentage by far.

Kevin Durant captured the appeal: “When you are with somebody like that, a person who cares about all those things on top of wanting you to be the best player you can be, then you want to go to work with that person.”

The same principle applies to data teams. The leaders who build lasting excellence aren’t the ones with the cleverest technical strategies. They’re the ones who create environments where talented people choose to stay, sacrifice, and push each other toward genuine accomplishment.

The uncomfortable question for data leaders

At his Hall of Fame induction, Popovich offered unusual clarity: “Basketball doesn’t love us back, does it? We use it like a bar of soap, right? It pays our bills. It gives us a wonderful life. But I don’t remember it saying, ‘I love you, Pop.’ It’s different. It’s the family.”

Here’s the uncomfortable question for data leaders: what’s the family equivalent for your team?

Data pipelines don’t love you back. Dashboards don’t remember your birthday. The perfectly optimized query won’t show up at your door when your father dies.

The work matters. Technical excellence matters. Delivering value to the organization matters.

But none of it persists if you haven’t built the human foundation that makes sustained performance possible.

The false choice is between being a great technical leader and being a great people leader. Popovich proves you need both—that technical excellence requires the human foundation, not despite it.

So the next question isn’t whether you have the right technology stack or the optimal team structure. It’s whether you’re investing in the belonging cues, the dinners (literal or figurative), the showing up during crisis, the relentless standards delivered with genuine love.

It’s whether you’re willing to do the uncomfortable work of building something worth sacrificing for.

Because here’s the thing: your team knows the difference. They know when connection is performative and when it’s real. They know when high standards come from caring and when they come from ego. They know whether they’re resources to be optimized or humans to be invested in.

And they’ll calibrate their own investment accordingly.

Tim Duncan, accepting his Hall of Fame honor, said what countless Spurs have felt about Popovich: “You showed up after I got drafted. You came to my island. You sat with my friends, my family. You talked with my dad. I thought that was normal. It’s not.”

I’m nowhere near Pop’s level. Most of us aren’t. But stories like this matter because they remind us what good can look like—and give us something to measure ourselves against.

Not to feel inadequate. To feel inspired.

The gap between where we are and where Pop operates isn’t a source of shame. It’s a direction. And the first step is simply asking the question: Would my team say I showed up?

Context Engineering: The New Must-Have Skill for Data Engineers

Chris Hillman — Sat, 31 Jan 2026 09:00:00 +1100

Last year I watched a colleague ask AI to help write a dbt model. The AI spit out perfectly functional SQL—clean syntax, proper CTEs, the works. Looked great.

Then I noticed the table would eventually hold 800 million rows. No partitioning. No clustering. Just a raw, unoptimised heap waiting to turn into a query performance nightmare (that would likely become my nightmare to fix).

The engineer wasn’t at fault. The AI wasn’t at fault either, really. The AI simply didn’t know that our environment clusters large tables by date. It didn’t know our team’s conventions around incremental models. It couldn’t know, because nobody had told it.

Here’s the thing: most data engineers treat AI assistants like they’re infinitely capable strangers. We give them tasks, accept their output, maybe tweak a few lines. But we never invest in teaching them our context—the hard-won lessons, the team conventions, the mistakes we’ve already made (and would rather not make again).

What if you could? What if every partitioning decision, every naming convention, every “never do this” rule you’ve learned over the years lived in a file that your AI assistant read before every single interaction?

That’s exactly what context engineering enables. Once you set it up, your AI stops being a generic autocomplete and starts acting like a peer who actually knows your codebase.

The Problem With Generic AI Assistance

I’ve been using AI coding assistants for a while. First Copilot (that was a terrible experience), then ChatGPT, and now more recently Claude. I can write up blocks faster with the right prompts.

But something kept nagging at me.

The AI would suggest something technically correct but contextually wrong. It would generate a model without tests. Propose a column name that violated our conventions. Use VARCHAR when we standardise on STRING. Small things, individually. Which would lead to more and more refining on the prompts.

The root cause was simple: the AI had no memory. Every conversation started fresh. It didn’t know that we’d spent three painful weeks last quarter cleaning up inconsistent date formats. It didn’t know that our dim_customer table had a specific SCD Type 2 pattern we’d refined over months.

Every session, I found myself re-explaining the same context. “Remember to add tests.” “We use snake_case for everything.” “Always partition by event_date.”

This is always a problem in work generally. You could have an equally talented data engineer sitting next to you, but you don’t share the same problems faced, the same challenges solved. You’d find out months or weeks later—we ran into this problem in that table, and we resolved it by doing this. And you’d say, oh we had the same problem elsewhere, but here’s how we tackled it.

The solution can be better prompting, as I discussed in this article - https://ghostinthedata.info/posts/2026/2026-01-04-talk/. It can also be solved with persistent context.

But there’s another approach that’s more fundamental: teaching your AI assistant your context once, so it remembers across every session. That’s what context engineering is about.

What Context Engineering Actually Looks Like

Context engineering is a file (or set of files) that contains your team’s accumulated wisdom, loaded automatically into every AI session. Think of it as institutional memory for your AI assistant.

The core concept is straightforward: you maintain markdown files that describe your conventions, standards, and hard-won lessons. Your AI assistant reads these files at the start of every session.

Different tools implement this differently:

Claude Code (Anthropic’s CLI tool) looks for a CLAUDE.md file in your project root. It loads this automatically when you start a session.

Cline (the VSCode extension I use) supports custom system prompts and can reference local files for context. You can configure it to always include specific markdown files in its context window.

Cursor has similar capabilities with its rules files.

The implementation details vary, but the principle is identical: give the AI your context before it starts generating code.

Here’s what a basic context file might look like for a data engineering project:

# Analytics Pipeline Context

## Stack

- Snowflake (Enterprise tier)
- dbt Core 1.7+
- Fivetran for ingestion
- Tableau for BI

## Critical Standards

### Partitioning & Clustering

- **ALWAYS** cluster tables >10M rows by their primary date column
- Use `cluster_by` in dbt config, not raw DDL
- Partition Staging tables by data_date

### Naming Conventions

- Models: `stg_source__entity`, `fct_entity`, `dim_entity`
- Columns: snake_case, never camelCase
- Booleans: prefix with `is_` or `has_`
- Dates: suffix with `_at` for timestamps, `_date` for dates
- IDs: suffix with `_id` (never `_key` for surrogate keys)

### Testing Requirements

- Every model needs at least: unique, not_null on primary key
- Fact tables need referential integrity tests to all dimensions
- SCD2 dimensions need: valid_from < valid_to, no gaps or overlaps in history

This isn’t complicated. It’s just written-down knowledge. The magic happens when this knowledge persists across sessions.

Structuring Your Context Files

I use Cline for most of my dbt work because it integrates cleanly with VSCode and connects directly to the Claude API. Rather than one massive context file, I structure my context into logical chunks—this helps the AI (and me) find relevant information quickly.

Here’s the directory structure I use:

When I start a session, this context is already loaded. The AI knows our conventions before I type a single character.

The specific configuration will depend on your tool of choice—Cline, Cursor, and Claude Code each have their own settings. The important part is the content itself: clear, specific rules that capture how your team actually works.

The “Every Mistake Becomes a Rule” Philosophy

Here’s where context engineering gets genuinely powerful.

Boris Cherny, who created Claude Code, maintains his context file at exactly 2,500 tokens. His practice: “Every mistake becomes a rule.” When something goes wrong that’s generalisable, it gets added to the context file immediately.

This resonated with me because it mirrors how data engineers actually accumulate wisdom. We don’t learn by reading documentation. We learn by screwing things up, fixing them, and vowing to never make that mistake again.

The difference now? Those lessons don’t just live in our heads. They live in a file that our AI assistant reads before every interaction.

Last quarter, a seemingly innocent change to an SCD2 dimension caused our dashboards to double-count certain metrics. The root cause was subtle: we’d accidentally created overlapping validity windows during a backfill. It took three days to diagnose.

The fix took an hour. But more importantly, I added this to my testing patterns file:

SCD Type 2 Validation Rules

## SCD Type 2 Validation

Every SCD2 dimension MUST include these tests:

### No Overlapping Windows

```yaml
tests:
 - dbt_utils.unique_combination_of_columns:
 combination_of_columns:
 - customer_id
 - valid_from
 - dbt_expectations.expect_column_pair_values_A_to_be_less_than_B:
 column_A: valid_from
 column_B: valid_to
 or_equal: false
```

The Duct Tape Data Engineer

Chris Hillman — Sat, 24 Jan 2026 09:00:00 +1100

The Engineer Who Ships

I want to tell you about a data engineer I worked with. Let’s call her Sarah.

Sarah had a reputation. When business stakeholders had an urgent question—the kind that arrives at 4 PM on a Friday with the CEO’s name in the subject line—they went to Sarah. Not to the senior architect with the impeccable data model. Not to the platform team with their carefully orchestrated Airflow DAGs. They went to Sarah.

Her code wasn’t pretty. She’d pull data from production databases with SQL queries that made the architecture team wince. She’d transform things in Python scripts that violated every design pattern in the book. She’d dump results into Excel because that’s where the CFO could actually use them.

And here’s the thing: Sarah shipped. Every time. On time. With answers that moved the business forward.

At the time, I thought Sarah was the best data engineer I’d ever worked with. I’m less certain now.

Sarah is what I would call a “duct tape programmer.” Someone who ships working solutions while others are still debating architectures. In the moment, there’s no one you’d rather have on your team.

But moments pass. And the code remains.

Three Types of Data Engineer

I’ve come to believe there are three distinct approaches to data engineering, and understanding the differences matters more than most of us realise.

The Architecture Astronaut builds for a future that never arrives. They spend months perfecting data contracts and tenant-level data mesh architecture. They have governance frameworks, schema registries, and architectural purity that would make Zhamak Dehghani proud. What they don’t have is a single stakeholder whose problem they’ve solved.

The Duct Tape Engineer ships at all costs. They’re Sarah at 4 PM on Friday—pulling data from production, hacking together Python scripts, dumping results into Excel. They solve problems. They move fast. They leave a trail of undocumented queries and mystery scripts in their wake.

The Foundational Pragmatist does something different. They invest in the foundations that make future speed possible—proper dimensional models, conformed dimensions, clean grain definitions—and then they duct tape on top of that prepared surface. They ship fast because they built well, not instead of building well.

The industry talks endlessly about the first two. The conferences are full of astronauts evangelising the latest architecture. The war stories celebrate the duct tape heroes who saved the day. But the engineers who actually deliver sustained value? They’re usually the third type, and they’re rarely on stage.

The Architecture Astronaut Visits Your Desk

You know this person. They show up at your desk, coffee in hand, ready to explain why your simple ETL pipeline is fundamentally wrong.

“What you really need,” they say, eyes lighting up, “is a proper lakehouse architecture. Bronze, silver, gold layers. Delta Lake for ACID compliance. Apache Kafka for real-time ingestion. And have you considered data mesh? Each domain team should own their own data products with standardized contracts and federated governance.”

You nod politely, but inside you’re doing the math. Your company has three data sources. Your biggest table has 50 million rows. Your most demanding stakeholder needs a weekly report. And this person wants you to implement an architecture designed for companies processing petabytes per day.

They’re the over-engineering colleague who “comes up to your desk, coffee mug in hand, and starts rattling on about how if you implement event-driven architecture with a proper schema registry, your pipeline will be 34% more scalable.” Scalable for what? You’re processing 100MB of data. On a good day.

The architecture astronaut isn’t stupid. They’re often brilliant. They’ve read every blog post, attended every conference, mastered every tool. They can diagram a system that would make Netflix jealous.

But they have a disease: they’re allergic to shipping.

They’d rather spend six months perfecting a medallion architecture than six hours writing the SQL query that answers the CEO’s question. They optimise for theoretical elegance at the expense of actual impact. And they leave a trail of half-finished platforms in their wake.

A Tale of Two Teams

Let me tell you about two data teams I’ve observed over the years.

The first was building the future. Data contracts. Federated governance. Tenant-level data mesh with domain ownership and self-serve capabilities. They had architecture review boards, schema evolution policies, and compatibility matrices. Every decision went through rigorous evaluation. Every interface was documented exhaustively.

Three years in, they were still perfecting their approach. The platform team had become a bureaucracy that produced governance documents, not insights. Architecture astronauts, every one of them.

The second team took a different path. They weren’t duct tape engineers—they didn’t hack things together and hope for the best. Instead, they invested in a proper dimensional model. Conformed dimensions. Clean fact tables. Well-defined grain. Nothing fancy. No distributed systems. No streaming. Just solid data modeling fundamentals executed well.

When the general manager asked a question, they answered it in minutes. Not because they had magical technology, but because the foundation was sound. The conformed customer dimension meant they didn’t have to reconcile six different definitions of “active customer.” The properly-grained fact tables meant they could slice and dice without fear of double-counting. The slowly changing dimensions meant they could answer “what did this look like last quarter?” without archaeological expeditions through backup tapes.

One team had years of architectural innovation with nothing to show for it. The other had basic dimensional modeling and could answer almost anything.

The difference wasn’t intelligence or effort. It was understanding which investments actually compound. The second team wasn’t avoiding architecture—they were choosing the right architecture. The kind that creates a foundation for speed rather than a monument to complexity.

The 85% Failure Rate

But wait, you might say. Surely all these complex architectures exist for a reason. Surely the industry’s best minds have converged on data mesh and lakehouses because they work.

In 2015, Gartner estimated that most big data projects would fail. By 2017, analyst Nick Heudecker admitted they’d been “too conservative.” The actual failure rate was higher—in the 80% range. Most data migrations fail or exceed their budget and scope.

Eighty percent. In what other engineering discipline would we accept failure rates like that? In what other field would we keep advocating for the same approaches that fail four out of five times?

The UK’s National Health Service attempted to centralise patient records in one of the largest data projects ever undertaken. They flushed $15 billion down the drain. Fifteen billion dollars. And they got nothing.

Lyft spent years migrating to Hive. By the time they finished in 2020, the entire industry had moved on. They ended up stuck on an outdated system, having invested countless engineering hours chasing an architecture that was already obsolete.

These aren’t exceptions. They’re the pattern. Complex architectures fail. Simple foundations, built well, endure.

The Philosophy of Worse

This tension isn’t new. Richard Gabriel identified it in 1989 with his famous essay “Worse is Better.”

The core idea is counterintuitive: “Implementation simplicity has highest priority.” Not correctness. Not completeness. Not elegance. Simplicity.

Gabriel argued that “it is better to get half of the right thing available so that it spreads like a virus. Once people are hooked on it, take the time to improve it to 90% of the right thing.”

The four principles he outlined still apply:

Simplicity: The system should be simple above all else.
Correctness: It’s slightly better to be simple than correct.
Consistency: Can be sacrificed for simplicity.
Completeness: Can be sacrificed in favour of any other quality.

This feels wrong to engineers. We’re trained to build things correctly. We’re judged on technical sophistication. Our peers admire elegant architectures and comprehensive solutions.

But Gabriel was describing how Unix beat more sophisticated operating systems. How C beat more powerful languages. How TCP/IP beat more theoretically correct protocols. The simpler, “worse” solutions won because they shipped, spread, and evolved.

John Gall formalised the pattern in 1975: “A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”

Don’t architect a lakehouse from scratch. Start with a simple pipeline that works. Evolve it as you understand your actual requirements. That’s not compromise—that’s how successful systems are built.

The Foundation That Enables Speed

Here’s where the duct tape philosophy goes wrong: it assumes all architecture is overhead.

I believe strongly in dimensional modeling. Not because it’s elegant—though it is. Not because it’s theoretically pure—though it can be. But because when a proper dimensional model exists, I can answer the general manager’s Friday afternoon question in fifteen minutes instead of four hours.

The conformed dimension that took a week to build properly? It’s saved me hundreds of hours of query writing since. The fact table with clean grain definitions? It means I’m not debugging data quality issues at 11 PM.

This is what separates the foundational pragmatist from the pure duct tape engineer. Sarah could answer Friday’s question. But she’d have to re-derive the customer definition from scratch every time. She’d spend hours reconciling data that a conformed dimension would have reconciled once. She could produce 3 reports but they might differ in definitions of what a customer is.

The architecture astronaut spends six months building infrastructure nobody uses. The duct tape engineer hacks together solutions that become unmaintainable. The foundational pragmatist invests two weeks in dimensional modeling and creates the conditions for rapid, reliable delivery forever after.

The question isn’t whether to invest in architecture. It’s which architecture and when. A solid dimensional model? Almost always worth it. A lakehouse for your 50GB dataset? Probably not. Data contracts and federated governance for a team of three? You’re solving problems you don’t have.

That team I mentioned—the one answering questions in minutes—they weren’t magicians. They’d done the foundational work. They had clean dimensions. They had documented grain. They had conformed definitions.

Their duct tape stuck better because they’d prepared the surface.

The Maturity Gradient

There’s a progression that the conference talks never mention:

Monthly reports → Weekly reports → Daily pipelines → Micro-batches → Streaming

Each step leftward costs exponentially more in complexity. Each step rightward gives you exponentially more time to get the quality right.

When I build something new, I start as far right as the business can tolerate. Monthly reporting sounds unsexy, but it gives you thirty days to catch quality issues before anyone sees bad data. You have time to validate, to cross-check, to build confidence.

Once monthly is bulletproof, moving to weekly is straightforward. You’ve already solved the hard problems—the business logic, the edge cases, the data quality rules. Now you’re just increasing frequency. Daily follows. Micro-batch follows.

Can you skip straight to real-time? Sure. I’ve seen it done. I’ve also seen the pain involved—debugging race conditions, explaining to stakeholders why the “real-time” dashboard showed impossible numbers, rebuilding pipelines from scratch because the streaming-first design couldn’t handle a schema change.

The streaming-first approach means you’re solving data quality, business logic, edge cases, AND distributed systems problems simultaneously. That’s a recipe for the 85% failure rate we talked about.

Ship the monthly report. Prove the logic. Harden the quality. Then earn your way to daily.

Duct Tape With a Sunset Date

Here’s what separates strategic pragmatism from technical negligence: the hardening plan.

Every quick solution I ship comes with an implicit contract. Yes, I’ll wrangle this data together to plug the compliance gap by Friday. Yes, I’ll build that dashboard for the legal team using queries that would make a data modeler weep. But written somewhere—in Jira, in a Confluence page, in my own notes—is the plan to do it properly.

The CFO’s urgent analysis gets a spreadsheet today. It gets a proper pipeline next quarter.

This isn’t compromise—it’s sequencing. The business gets value immediately. The team gets breathing room. And the quick solution becomes a specification for the permanent one. You’ve proven the logic works. You’ve validated the business need. Now you can invest in building it right, with far less risk than starting from scratch.

The failure mode isn’t duct tape itself. It’s duct tape without a sunset date. It’s the “temporary” query that’s still running five years later because nobody remembered to harden it. It’s technical debt accumulated without a repayment plan.

I’ve used this approach to protect my teams from massive pieces of work that would have consumed them for months. The compliance hole that needed plugging immediately? Duct tape with a ticket. The legal team’s urgent risk analysis? Duct tape with a documented plan. The high-value project that couldn’t wait for proper architecture? Duct tape with a sunset date.

The duct tape isn’t the problem. Forgetting to remove it is.

Sarah’s Legacy

Every organisation has Bob’s query.

You know the one. Bob wrote it three years ago. Bob left eighteen months ago. The query is still running every Tuesday morning, feeding a report that twelve people apparently depend on, documented nowhere, understood by no one.

Here’s the uncomfortable truth: Bob’s query is what Sarah’s Friday afternoon heroics look like three years later.

Sarah shipped. She solved the problem. The CEO got their answer. But she didn’t write the ticket. She didn’t document the logic. She didn’t create a sunset date. She moved on to the next fire, and the quick fix quietly became load-bearing infrastructure.

The person who inherited Sarah’s work doesn’t think she’s a hero. They think she’s the reason they’re debugging mystery scripts at 11 PM instead of building something new.

This is what duct tape looks like when the sunset date gets forgotten. It’s technical debt, pure and simple. A liability waiting to break at the worst possible moment.

But here’s the reframe: Sarah’s legacy is also an opportunity.

That query is a perfect project for a junior engineer. The risk is low—it’s already working, so they have a baseline to compare against. The scope is contained—one query, one output, one business purpose. They can pull it apart, understand the logic, document what they find, and rebuild it as a proper pipeline.

They learn data modeling. They learn testing. They learn documentation. They learn how to interview stakeholders about requirements that were never written down. They harden their skills in a safe environment while simultaneously paying down technical debt. This is what builds up Junior to Senior levels.

Every piece of Sarah’s legacy is both a problem to solve and a training ground to offer. The mature data engineer sees both. They create the ticket, assign the junior, and turn a liability into a development opportunity.

The debt gets paid. The junior grows. The organisation gets a documented, tested, maintainable pipeline instead of a mystery script.

That’s pragmatism with a long-term view.

The Pragmatist’s Manifesto

Here’s what the foundational pragmatist understands:

Shipping is a feature. The most important feature. Your data pipeline doesn’t exist until it’s serving production queries. Your data model doesn’t matter until it’s powering real decisions. “Your product must have it.”

The pipeline that ships insights beats the pipeline that ships nothing. A “50%-good solution that people actually have solves more problems and survives longer than a 99% solution that nobody has because it’s in your lab where you’re endlessly polishing the damn thing.”

But foundations compound. A solid dimensional model isn’t architecture astronaut behaviour—it’s the investment that makes all future delivery faster and more reliable. Build the foundation once, benefit forever.

Directionally correct beats precisely wrong. The dashboard that’s 80% accurate and delivered today creates more value than the perfectly governed data product delivered never.

Start with CSV, graduate to Parquet, earn your way to streaming. You don’t need real-time until you do. You don’t need a lakehouse until you do. You don’t need Kafka until you do. And you probably don’t.

Every shortcut needs a sunset. Ship the hack. Document the plan. Pay down the debt. Quick fixes without a hardening plan aren’t pragmatism—they’re negligence with extra steps.

Jamie Zawinski said it best: “At the end of the day, ship the fucking thing! It’s great to rewrite your code and make it cleaner and by the third time it’ll actually be pretty. But that’s not the point—you’re not here to write code; you’re here to ship products.”

For data engineers, the adaptation is obvious: You’re not here to build architectures. You’re here to ship insights. But the insights you ship today shouldn’t sabotage the insights you need to ship tomorrow.

The Caveat

Here’s where I need to be careful.

Pragmatic shortcuts aren’t an excuse for incompetence. “Duct tape programmers must be talented enough to pull it off.”

Sarah wasn’t shipping ugly code because she didn’t know better. She was making conscious trade-offs. She understood the implications of pulling from production databases. She knew how to write cleaner code—she just knew when not to bother.

Her failure wasn’t the duct tape. It was stopping there.

There’s a difference between strategic simplicity and reckless shortcuts. The foundational pragmatist understands the trade-offs. They know what they’re sacrificing and why it’s worth it. They can explain their decisions and defend them. And they write the ticket for the hardening work.

Netflix actually needs Kafka. Meta actually needs petabyte-scale systems. Google actually needs distributed query engines.

The question is whether your company does.

Most don’t. The data supports this overwhelmingly. But knowing when you’re the exception requires understanding the fundamentals. You can’t break the rules effectively until you know what the rules are.

So yes, learn the proper patterns. Understand data modeling. Know why slowly changing dimensions exist. Study the medallion architecture. Read about data mesh.

Then make conscious decisions about when to use them. Not because a blog post said so. Not because a vendor promised ROI. Not because that’s what the architecture astronaut wants.

Because they solve a problem you actually have.

The Engineer Who Holds All Three

Here’s what I’ve learned after years of watching these patterns play out: the best data engineers understand all three modes and know when each applies.

I read every blog post about data mesh. I attend the conferences. I evaluate new tools. I think constantly about how we might evolve our architecture. That’s not being an astronaut—that’s staying informed.

When my team faces a Friday afternoon crisis, I can be Sarah. I can hack together the solution, ship the answer, buy us breathing room. That’s not recklessness—that’s responsiveness.

But unlike Sarah, I write the ticket. I document the sunset date. And when we have the space, I advocate for the dimensional modeling that will make the next crisis easier to handle.

The architecture astronaut’s failure is pursuing elegance without urgency. The duct tape engineer’s failure is urgency without investment in foundations. The pure pragmatist ships fast on solid ground and pays down their debts.

I’ve watched teams spend years on data mesh with nothing to show for it. I’ve inherited Sarah’s legacy code and cursed her name at midnight. I’ve also seen teams with solid data models answer any question in minutes, year after year.

The difference wasn’t the sophistication of their approach—it was whether their investments actually compounded into capability, and whether their shortcuts came with expiration dates.

So yes, ship the dashboard. Answer the executive’s question. Plug the compliance hole today.

But write down the hardening plan. Build the dimensional model when you can. Start with monthly, earn your way to daily. Invest in the foundations that make future speed possible.

And when the junior engineer asks why Sarah’s query is still running after five years, hand them the ticket to fix it.

That’s the job.

That Tuesday Morning When I Finally Fixed Our Ten-Minute Queries

Chris Hillman — Fri, 16 Jan 2026 09:00:00 +1100

The Ten-Minute Query

I’m sitting at my laptop on a Tuesday morning, waiting. The progress bar on my screen says ‘Query running… 4 minutes, 37 seconds.’ I lean back in my chair and let out this long sigh that probably says more than I intended.

My manager walks past my desk. She glances at my screen, and I can see that look—the one that says she already knows what I’m about to tell her. I didn’t need to explain.

“Still pulling that customer report?”

“Yeah.”

She shakes her head. “We need to fix this.”

It was an ad-hoc piece of work that we’d been running for months. But I just nod.

Here’s what was happening: every time someone needed to know ‘how many orders did customer X place?’ or ‘which products sold best last month?’, we’d query this massive table. Three million rows. Every single customer’s name, address, email—repeated on every order row. The system was scanning through all that redundant data every single time.

And then, two weeks ago, it got worse. Someone ran a data import script that had a bug. It replaced half our customer names with NULL. By the time we caught it, twelve reports had already gone out with blank customer names. I had to send apology emails. My team and I had to stay until midnight fixing it.

I thought, there has to be a better way to do this. People have been building data systems for decades—someone must have figured this out.

And they have. His name is Ralph Kimball, and his dimensional modeling methodology has been the gold standard for data warehouses since the 1990s.

The Problem I Couldn’t Unsee

Let me show you what our data looked like.

order_id | date | customer_name | address | email | product | amount
---------|------------|---------------|--------------|--------------|---------|-------
1 | 2024-11-15 | John Smith | 123 Main St | john@... | Pizza | 25
2 | 2024-11-15 | John Smith | 123 Main St | john@... | Soda | 3
3 | 2024-11-15 | Jane Doe | 456 Oak Ave | jane@... | Pizza | 25
4 | 2024-11-16 | John Smith | 123 Main St | john@... | Burger | 12

See what’s happening? John Smith’s address and email are stored three times. Multiply that by thousands of customers and millions of orders, and you’ve got:

Every query scanning way more data than necessary
Storage costs that make your CFO unhappy
One bug corrupting your entire history
Query times that make you question your career choices

Every single query was like asking someone to find your keys by searching every room in your house, including the ones you never use. Sure, they’ll find the keys eventually. But there’s a better way.

The Kitchen That Changed Everything

Let me paint you a hypothetical scenario. Imagine it’s a Saturday, and you’re standing in your kitchen after just moving house. You can’t find anything. Everything’s still in boxes.

You’re rummaging through the third box, looking for a spatula. You’re pulling out random stuff—a whisk, some Tupperware lids, a pizza cutter, yesterday’s takeout containers, a bag of flour. Everything mixed together. This is chaos.

Now imagine a friend comes over. They take one look at your kitchen and say, “What is this?”

“It’s… I haven’t unpacked yet.”

“You’ve been here a week.” They walk over to the mess. “Here’s what you should do. The stuff that rarely changes—your plates, your pots, your cutting boards—those go in the cabinets. Organised, labelled, easy to find. The stuff that comes and goes constantly—the groceries, the leftovers, what you’re cooking today—that goes in the fridge and on the counter. Don’t mix the permanent stuff with the temporary stuff.”

Two hours later, the kitchen makes sense. You can find anything in seconds. And here’s the wild part—it actually takes up less space because you’re not duplicating storage for the same items.

That’s dimensional modeling.

When your data is like those boxes, everything’s mixed together. Customer information sitting right next to order information, repeated over and over. Every time you need something, you’re digging through all the boxes.

Dimensional modeling says: let’s organise this. Put stable descriptive data—things that rarely change—in one place. We call these dimension tables. Put the events that happen constantly—the transactions, the updates, the measurements—in another place. We call these fact tables. Link them with keys.

Here’s what that transformation looks like:

BEFORE (Messy Boxes - everything mixed together):

order_id | customer_name | address | product | amount
1 | John Smith | 123 Main St | Pizza | 25
2 | John Smith | 123 Main St | Soda | 3

↑ John’s details repeated every single order!

AFTER (Organised Kitchen): DIM_CUSTOMER (Cabinets - stable stuff, stored ONCE):

customer_key | customer_id | name | address | email
1001 | C-123 | John Smith | 123 Main St | john@...
1002 | C-456 | Jane Doe | 456 Oak Ave | jane@...

DIM_DATE (Calendar on the wall - one row per day):

date_key | full_date | day_name | month | quarter | year | is_weekend
20241115 | 2024-11-15 | Friday | Nov | Q4 | 2024 | N
20241116 | 2024-11-16 | Saturday | Nov | Q4 | 2024 | Y

FACT_ORDERS (Fridge/counter - events that happen constantly):

order_key | date_key | customer_key | product_key | amount
1 | 20241115 | 1001 | 5 | 25
2 | 20241115 | 1001 | 8 | 3

Customer details stored one time. Not a thousand times. When you need to answer ‘how much did we sell?’, you scan the compact fact table. When you need ‘which customers bought?’, you JOIN to get the descriptive details.

Notice something important in that structure? The dimension tables have two types of keys:

customer_key (1001, 1002) — This is a surrogate key. A meaningless integer we generate ourselves.
customer_id (C-123, C-456) — This is the natural key. The ID from the source system.

Why both? Because source systems are chaotic. They recycle IDs. They change formats. They get merged in acquisitions. The surrogate key is our key—stable, predictable, and under our control. Every dimension table gets a surrogate key as its primary key.

And that date dimension? It’s “the most important dimension.” Every fact table should have one. It lets you slice data by day of week, month, quarter, fiscal period, holidays—without calculating these on every query.

The core principle: things that rarely change go in dedicated storage, organised and easy to find. Things that happen constantly go in their own space, linked back by keys. Same rule applies to data:

Dimensions = Stable descriptive stuff (customers, products, locations, dates)
Facts = Events that happen constantly (orders, clicks, updates, measurements)

The Conference Room I’ll Never Forget

I’m sitting in a conference room. Fluorescent lights buzzing. That weird hum they make when it’s too quiet—I’m wondering why we haven’t changed to LEDs. My manager is standing at the whiteboard, and she’s pissed.

“Someone,” she says, not looking at anyone in particular, “pushed bad data to production last night.”

I feel my stomach drop. The data pipeline my team had been working on.

“The revenue report that went to the board this morning showed every transaction as zero dollars. Zero. The board emailed me asking if the company made any money last quarter.”

Oh god. Oh no. I sank down in my chair a little.

Turns out it wasn’t actually our release—it was someone else’s. But in that moment, sitting there, I realised something crucial: we had no safety net. Data went straight from transformation to production. No checks. No validation. If you made a mistake, everyone saw it immediately.

That conference room moment stuck with me. I started researching how other teams prevented this kind of disaster. That’s when I learned about Write-Audit-Publish. And honestly? It’s probably saved me from that same sick feeling countless times since.

Here’s the idea:

WRITE — You finish your transformation. Instead of putting the data straight into production, you write it to a staging branch. Think of it like a draft folder for emails. It’s there, but it’s not official yet.

AUDIT — This is the crucial part. You run every check you can think of:

Did I lose any rows?
Are there any NULL values where there shouldn’t be?
Do all the foreign keys match up?
Does the math make sense?
Do surrogate keys have duplicates?

It’s like re-reading an important email before you hit send. You check for typos, you verify the recipients, you make sure the attachment is right.

Important: In the future when you find incidents, bugs, or problems—add them to this AUDIT layer. Ensure you don’t make the same mistakes twice.

PUBLISH — Only if everything passes, you merge the branch to production. If anything fails, you stop. You fix it. You try again. The bad data never reaches production. Never reaches that board meeting.

The traditional way to implement this involves copying data between folders or tables. But there’s a better approach using Apache Iceberg’s branching feature. Instead of physically copying data, you create a branch—like in Git. Your audit runs against the branch. If it passes, you merge the branch to main. If it fails, you drop the branch. No data was ever at risk.

Critical design decision: WAP should happen at each table level, not just at the end of the pipeline. Each dimension job does its own Write-Audit-Publish cycle. Each fact job does its own. This way:

Failures are isolated to the specific table that failed
You catch issues early, not at the very end
Each job is self-contained and independently testable
Downstream jobs read from published (main) tables, not staging branches

Let me show you what this looks like when an audit catches something:

AUDIT PHASE (dim_system):

Row count check.......................... PASS
 Source: 1,198 rows
 Target: 1,198 rows

Primary key uniqueness................... FAIL
 Found 13 duplicate system_keys in dim_system!

AUDIT FAILED — BRANCH NOT MERGED

Data remains in staging branch.
Production table unchanged.
Fix the duplicates and re-run the job.

Look at that. The audit caught duplicate surrogate keys before they reached production. Nobody sees this except you. You fix the problem. You run it again. Then it passes, and the branch merges to main.

This is Write-Audit-Publish. It’s the difference between sending an email with a typo versus catching it first. Between serving undercooked food versus checking with a thermometer. Between publishing bad data versus validating before it matters.

That conference room moment? Doesn’t happen with this pattern.

Building This Thing for Real

Alright, enough theory. Let me show you how to actually build this in AWS.

We’re going to use real (game) space exploration data—systems, planets, updates. It’s way more interesting than order data, and the patterns work exactly the same.

In this section:

Architecture Overview
Step 1: Setting Up S3 Buckets
Step 2: Creating the Iceberg Tables
The Glue Jobs (with Per-Table WAP)
Step Functions Workflow
Dependency Triggers with S3 Events

Architecture Overview

Here’s our architecture:

 +------------------------------------------+
 | AWS Step Functions Workflow |
 +------------------------------------------+
 |
 +----------------------------+----------------------------+
 | | |
 v v v
 +------------+ +------------+ +------------+
 | Glue Job: | | Glue Job: | | Glue Job: |
 | dim_date | | dim_system | | dim_planet |
 | (WAP) | | (WAP) | | (WAP) |
 +------------+ +------------+ +------------+
 | | |
 +----------------------------+----------------------------+
 |
 (Wait for all dimensions)
 |
 v
 +------------------------+
 | Glue Job: |
 | fact_system_updates |
 | (WAP + Incremental) |
 +------------------------+

Data Flow:

Landing Layer	Staging Layer (PSA)	Semantic Layer
Raw 1:1 from source	Iceberg + data_date partitioning	Production dimensional model
`systems/`	`stg_systems/`	`dim_date`
`planets/`	`stg_planets/`	`dim_system`
`updates/`	`stg_updates/`	`dim_planet`
		`fact_system_updates`

Let me explain the three layers:

Landing Layer — Raw data, 1:1 from source systems. No transformations except what’s needed to land it. Contains only the latest fetch. If sourcing fails, this layer fails. Get the data close to you as quickly as possible (into your ecosystem). Typically all the files will be string or varchar to mitigate data type issues from a source system.

Staging Layer (Persistent Staging Area) — This is where we convert data types and prepare for dimensional modeling. Critically, we partition by data_date—the date the source data was fetched. This gives us:

Historical reprocessing capability (reload any past date)
Debugging (what did the data look like on January 15th?)
Dependency triggers (when a partition is written, trigger downstream jobs)

Semantic Layer — Production-ready dimensional model. This is where business users query. Tables are partitioned appropriately for query performance.

Also notice the job structure: each job does its own WAP cycle. Dimensions build in parallel, each publishing to main when their audits pass. Fact tables wait for all dimensions to publish, then do their own WAP cycle. This ensures:

Fact tables read from published dimensions (main branch), not staging branches
Foreign key lookups use production-ready surrogate keys
Each job is independently testable

Step 1: Setting Up S3 Buckets

I’m in the S3 console. Click “Create bucket”. I’m calling mine space-data-pipeline-demo.

Important settings:

Region: ap-southeast-2 (Sydney—pick one close to you)
Block all public access: Yes
Versioning: Enabled
Default encryption: Enabled

Inside this bucket:

space-data-pipeline-demo/
├── landing/ ← Raw 1:1 from source (latest only)
│ ├── systems/
│ ├── planets/
│ └── updates/
├── staging/ ← PSA with data_date partitioning
│ ├── stg_systems/
│ ├── stg_planets/
│ └── stg_updates/
└── semantic/ ← Production dimensional model
 ├── dim_date/
 ├── dim_system/
 ├── dim_planet/
 └── fact_system_updates/

Step 2: Creating the Iceberg Tables

Here’s the DDL for our dimensional model. Notice the partitioning strategies and SCD Type 2 columns.

Click to expand: dim_date DDL

This is a reference dimension—typically loaded once with all dates, then rarely updated. Partitioned by year for query performance.

CREATE TABLE space_exploration.dim_date (
 date_key INT,
 full_date DATE,
 day_of_week STRING,
 day_of_month INT,
 day_of_year INT,
 week_of_year INT,
 month_number INT,
 month_name STRING,
 quarter INT,
 year INT,
 is_weekend STRING,
 is_holiday STRING,
 fiscal_quarter INT,
 fiscal_year INT,
 -- Audit columns
 load_timestamp TIMESTAMP,
 source_system STRING
)
PARTITIONED BY (year)
LOCATION 's3://space-data-pipeline-demo/semantic/dim_date/'
TBLPROPERTIES (
 'table_type' = 'ICEBERG',
 'format' = 'parquet'
);

Click to expand: dim_system DDL (SCD Type 2)

Tracks history with effective_date/expiration_date. Partitioned by effective_year for efficient history queries.

CREATE TABLE space_exploration.dim_system (
 system_key INT, -- Surrogate key (generated)
 system_id STRING, -- Natural key (from source)
 system_name STRING,
 sector STRING,
 main_star_type STRING,
 coordinates_x DOUBLE,
 coordinates_y DOUBLE,
 coordinates_z DOUBLE,
 discovery_date DATE,
 exploration_status STRING,
 -- SCD Type 2 columns
 effective_date DATE,
 expiration_date DATE,
 current_flag STRING, -- 'Y' or 'N'
 -- Audit columns
 load_timestamp TIMESTAMP,
 source_system STRING,
 -- Partition column
 effective_year INT
)
PARTITIONED BY (effective_year)
LOCATION 's3://space-data-pipeline-demo/semantic/dim_system/'
TBLPROPERTIES (
 'table_type' = 'ICEBERG',
 'format' = 'parquet'
);

Click to expand: dim_planet DDL (SCD Type 2)

Foreign key to dim_system uses the surrogate key. Partitioned by effective_year.

CREATE TABLE space_exploration.dim_planet (
 planet_key INT, -- Surrogate key
 planet_id STRING, -- Natural key
 system_key INT, -- FK to dim_system (surrogate!)
 planet_name STRING,
 planet_type STRING,
 orbital_period DOUBLE,
 has_atmosphere STRING,
 habitability_score DOUBLE,
 -- SCD Type 2 columns
 effective_date DATE,
 expiration_date DATE,
 current_flag STRING,
 -- Audit columns
 load_timestamp TIMESTAMP,
 source_system STRING,
 -- Partition column
 effective_year INT
)
PARTITIONED BY (effective_year)
LOCATION 's3://space-data-pipeline-demo/semantic/dim_planet/'
TBLPROPERTIES (
 'table_type' = 'ICEBERG',
 'format' = 'parquet'
);

Click to expand: fact_system_updates DDL

Incremental fact table partitioned by date_year. Loaded with delete-then-insert pattern.

CREATE TABLE space_exploration.fact_system_updates (
 update_key BIGINT, -- Surrogate key
 date_key INT, -- FK to dim_date
 system_key INT, -- FK to dim_system (current surrogate!)
 update_timestamp TIMESTAMP, -- Original precision preserved
 status STRING,
 ships_present INT, -- Additive fact
 resource_level DOUBLE, -- Additive fact
 -- Audit columns
 load_timestamp TIMESTAMP,
 source_system STRING,
 data_date DATE, -- Source data date (for reprocessing)
 -- Partition column
 date_year INT
)
PARTITIONED BY (date_year)
LOCATION 's3://space-data-pipeline-demo/semantic/fact_system_updates/'
TBLPROPERTIES (
 'table_type' = 'ICEBERG',
 'format' = 'parquet'
);

Click to expand: Staging Tables DDL (Persistent Staging Area)

Partitioned by data_date for historical reprocessing.

CREATE TABLE space_exploration.stg_systems (
 system_id STRING,
 system_name STRING,
 sector STRING,
 main_star_type STRING,
 coordinates_x DOUBLE,
 coordinates_y DOUBLE,
 coordinates_z DOUBLE,
 discovery_date DATE,
 exploration_status STRING,
 -- PSA columns
 data_date DATE,
 load_timestamp TIMESTAMP
)
PARTITIONED BY (data_date)
LOCATION 's3://space-data-pipeline-demo/staging/stg_systems/'
TBLPROPERTIES ('table_type' = 'ICEBERG', 'format' = 'parquet');

CREATE TABLE space_exploration.stg_planets (
 planet_id STRING,
 system_id STRING,
 planet_name STRING,
 planet_type STRING,
 orbital_period DOUBLE,
 has_atmosphere STRING,
 habitability_score DOUBLE,
 -- PSA columns
 data_date DATE,
 load_timestamp TIMESTAMP
)
PARTITIONED BY (data_date)
LOCATION 's3://space-data-pipeline-demo/staging/stg_planets/'
TBLPROPERTIES ('table_type' = 'ICEBERG', 'format' = 'parquet');

CREATE TABLE space_exploration.stg_updates (
 system_id STRING,
 update_timestamp TIMESTAMP,
 status STRING,
 ships_present INT,
 resource_level DOUBLE,
 -- PSA columns
 data_date DATE,
 load_timestamp TIMESTAMP
)
PARTITIONED BY (data_date)
LOCATION 's3://space-data-pipeline-demo/staging/stg_updates/'
TBLPROPERTIES ('table_type' = 'ICEBERG', 'format' = 'parquet');

Key design decisions:

dim_date: Partitioned by year. This is a reference dimension—typically loaded once with all dates from 1900-01-01 to 9999-12-31, then rarely updated.
dim_system / dim_planet: SCD Type 2 with effective_date, expiration_date, current_flag. Partitioned by effective_year for efficient history queries. Updated daily.
fact_system_updates: Partitioned by date_year. Loaded incrementally using delete-then-insert pattern.
Staging tables (PSA): Partitioned by data_date. This lets us reprocess any historical date and provides dependency triggers.

The Heart of the System: Orchestrated Glue Jobs with Per-Table WAP

Each job does its own Write-Audit-Publish cycle. Let me walk you through each one.

Job 1: Build Date Dimension (Reference Dimension)

The date dimension is special—it’s a reference dimension. We generate it ourselves with all calendar attributes, typically as a one-time load. You’d only rerun this if you need to extend the date range or add new attributes (like new holidays).

Click to expand: build_dim_date Glue Job

This job generates a complete date dimension from 2000-01-01 to 2050-12-31 (for demo purposes—production would use a wider range). It creates date attributes like day_of_week, quarter, fiscal_year, and is_weekend, then runs WAP to validate before publishing.

# Glue Job: build_dim_date
# This is typically a ONE-TIME RUN to populate all dates
# Rerun only when extending date range or adding attributes

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *
from pyspark.sql.types import *
from datetime import datetime, timedelta

# Initialize
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

print("="*60)
print("Building Date Dimension (Reference Dimension)")
print("This is typically a ONE-TIME load")
print("="*60)

# Generate comprehensive date range
# Standard practice: 1900-01-01 to 9999-12-31
start_date = datetime(1900, 1, 1)
end_date = datetime(9999, 12, 31)

# For demo purposes, let's use a more reasonable range
# In production, you'd use the full range
start_date = datetime(2000, 1, 1)
end_date = datetime(2050, 12, 31)

print(f"Generating dates from {start_date.date()} to {end_date.date()}")

date_list = []
current = start_date
while current <= end_date:
 date_list.append((current,))
 current += timedelta(days=1)

date_df = spark.createDataFrame(date_list, ["full_date"])

# Build dimension with rich attributes
dim_date = date_df.select(
 # Surrogate key: YYYYMMDD format
 (year("full_date") * 10000 +
 month("full_date") * 100 +
 dayofmonth("full_date")).cast("int").alias("date_key"),

 col("full_date"),
 date_format("full_date", "EEEE").alias("day_of_week"),
 dayofmonth("full_date").alias("day_of_month"),
 dayofyear("full_date").alias("day_of_year"),
 weekofyear("full_date").alias("week_of_year"),
 month("full_date").alias("month_number"),
 date_format("full_date", "MMMM").alias("month_name"),
 quarter("full_date").alias("quarter"),
 year("full_date").alias("year"),

 # Business-friendly flags
 when(dayofweek("full_date").isin([1, 7]), "Y")
 .otherwise("N").alias("is_weekend"),
 lit("N").alias("is_holiday"), # Populate from holiday calendar

 # Fiscal calendar (assuming fiscal year = calendar year)
 quarter("full_date").alias("fiscal_quarter"),
 year("full_date").alias("fiscal_year"),

 # Audit columns
 current_timestamp().alias("load_timestamp"),
 lit("GENERATED").alias("source_system")
)

row_count = dim_date.count()
print(f"Generated {row_count:,} date rows")

# ============================================
# WRITE PHASE: Write to staging branch
# ============================================
print("\n--- WRITE PHASE ---")
print("Writing to staging branch...")

spark.sql("ALTER TABLE space_exploration.dim_date CREATE BRANCH IF NOT EXISTS staging")

dim_date.writeTo("space_exploration.dim_date") \
 .option("branch", "staging") \
 .overwritePartitions()

print("Written to staging branch")

# ============================================
# AUDIT PHASE: Validate before publishing
# ============================================
print("\n--- AUDIT PHASE ---")

staging_df = spark.read \
 .option("branch", "staging") \
 .table("space_exploration.dim_date")

audit_failures = []

# Audit 1: Row count
staged_count = staging_df.count()
print(f"Audit 1 - Row count: {staged_count:,}")
if staged_count == 0:
 audit_failures.append("FAIL: dim_date is empty")

# Audit 2: Primary key uniqueness
unique_keys = staging_df.select("date_key").distinct().count()
if staged_count != unique_keys:
 duplicates = staged_count - unique_keys
 audit_failures.append(f"FAIL: {duplicates} duplicate date_keys")
 print(f"Audit 2 - PK uniqueness: FAIL ({duplicates} duplicates)")
else:
 print(f"Audit 2 - PK uniqueness: PASS (all {unique_keys:,} unique)")

# Audit 3: No NULLs in critical fields
null_count = staging_df.filter(
 col("date_key").isNull() | col("full_date").isNull()
).count()
if null_count > 0:
 audit_failures.append(f"FAIL: {null_count} NULL values in critical fields")
 print(f"Audit 3 - NULL check: FAIL ({null_count} NULLs)")
else:
 print("Audit 3 - NULL check: PASS")

# Audit 4: Date range coverage
date_range = staging_df.agg(
 min("full_date").alias("min_date"),
 max("full_date").alias("max_date")
).collect()[0]
print(f"Audit 4 - Date range: {date_range['min_date']} to {date_range['max_date']}")

# ============================================
# PUBLISH PHASE: Merge to main if audits pass
# ============================================
print("\n--- PUBLISH PHASE ---")

if len(audit_failures) > 0:
 print("\nAUDIT FAILED:")
 for failure in audit_failures:
 print(f" {failure}")
 print("\nBranch NOT merged. Production unchanged.")
 spark.sql("ALTER TABLE space_exploration.dim_date DROP BRANCH staging")
 raise Exception(f"Audit failed with {len(audit_failures)} errors")
else:
 print("\nALL AUDITS PASSED")
 print("Merging staging branch to main...")

 # Fast-forward main to staging
 spark.sql("""
 CALL system.fast_forward(
 table => 'space_exploration.dim_date',
 branch => 'main',
 to => 'staging'
 )
 """)

 # Clean up staging branch
 spark.sql("ALTER TABLE space_exploration.dim_date DROP BRANCH staging")

 print(f"PUBLISHED: dim_date ({staged_count:,} rows)")

print("\n" + "="*60)
print("dim_date job complete")
print("="*60)

job.commit()

Job 2: Build System Dimension (SCD Type 2)

This dimension tracks history. When a system’s attributes change (like exploration_status), we expire the old row and create a new one.

Click to expand: build_dim_system Glue Job

This job implements SCD Type 2 change detection. It compares incoming data against the current dimension, identifies NEW/CHANGED/UNCHANGED records, expires old versions, and creates new versions with updated surrogate keys.

# Glue Job: build_dim_system
# SCD Type 2 implementation with effective/expiration dates

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *
from pyspark.sql.window import Window

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'DATA_DATE'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

data_date = args['DATA_DATE'] # e.g., '2026-01-17'
print("="*60)
print(f"Building System Dimension (SCD Type 2)")
print(f"Data Date: {data_date}")
print("="*60)

# ============================================
# LOAD: Read from Persistent Staging Area
# ============================================
print("\n--- LOAD PHASE ---")

# Read today's staged data (already landed in PSA)
source_df = spark.read.table("space_exploration.stg_systems") \
 .filter(col("data_date") == data_date)

source_count = source_df.count()
print(f"Source rows from PSA (data_date={data_date}): {source_count:,}")

if source_count == 0:
 print("No data for this date. Exiting.")
 job.commit()
 sys.exit(0)

# Read current production dimension (for SCD2 comparison)
try:
 current_dim = spark.read.table("space_exploration.dim_system") \
 .filter(col("current_flag") == "Y")
 current_count = current_dim.count()
 print(f"Current dimension rows (current_flag='Y'): {current_count:,}")
 has_existing_data = current_count > 0
except:
 print("No existing dimension data (first load)")
 has_existing_data = False
 current_dim = None

# ============================================
# TRANSFORM: SCD Type 2 Logic
# ============================================
print("\n--- TRANSFORM PHASE ---")

if not has_existing_data:
 # First load: all rows are new
 print("First load - all rows are INSERT")

 window_spec = Window.orderBy("system_id")

 new_dim = source_df.select(
 row_number().over(window_spec).alias("system_key"),
 col("system_id"),
 col("system_name"),
 col("sector"),
 col("main_star_type"),
 col("coordinates_x"),
 col("coordinates_y"),
 col("coordinates_z"),
 col("discovery_date"),
 col("exploration_status"),
 lit(data_date).cast("date").alias("effective_date"),
 lit("9999-12-31").cast("date").alias("expiration_date"),
 lit("Y").alias("current_flag"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 year(lit(data_date)).alias("effective_year")
 ).distinct()

else:
 # Incremental load: detect changes for SCD2
 print("Incremental load - detecting changes for SCD2")

 # Define which columns to track for changes
 tracked_columns = [
 "system_name", "sector", "main_star_type",
 "coordinates_x", "coordinates_y", "coordinates_z",
 "exploration_status"
 ]

 # Join source with current to detect changes
 comparison = source_df.alias("src").join(
 current_dim.alias("curr"),
 col("src.system_id") == col("curr.system_id"),
 "full_outer"
 )

 # Identify: NEW (no match in current), CHANGED (values differ), UNCHANGED
 change_detection = comparison.withColumn(
 "change_type",
 when(col("curr.system_id").isNull(), "NEW")
 .when(col("src.system_id").isNull(), "DELETED")
 .when(
 (col("src.system_name") != col("curr.system_name")) |
 (col("src.sector") != col("curr.sector")) |
 (col("src.main_star_type") != col("curr.main_star_type")) |
 (col("src.exploration_status") != col("curr.exploration_status")) |
 (abs(col("src.coordinates_x") - col("curr.coordinates_x")) > 0.001) |
 (abs(col("src.coordinates_y") - col("curr.coordinates_y")) > 0.001) |
 (abs(col("src.coordinates_z") - col("curr.coordinates_z")) > 0.001),
 "CHANGED"
 )
 .otherwise("UNCHANGED")
 )

 # Count changes
 change_counts = change_detection.groupBy("change_type").count().collect()
 for row in change_counts:
 print(f" {row['change_type']}: {row['count']:,}")

 # Get max surrogate key for new key generation
 max_key = current_dim.agg(max("system_key")).collect()[0][0] or 0

 # Build new dimension rows for NEW and CHANGED records
 new_records = change_detection.filter(col("change_type").isin(["NEW", "CHANGED"]))

 window_spec = Window.orderBy("src.system_id")

 new_rows = new_records.select(
 (row_number().over(window_spec) + max_key).alias("system_key"),
 col("src.system_id").alias("system_id"),
 col("src.system_name").alias("system_name"),
 col("src.sector").alias("sector"),
 col("src.main_star_type").alias("main_star_type"),
 col("src.coordinates_x").alias("coordinates_x"),
 col("src.coordinates_y").alias("coordinates_y"),
 col("src.coordinates_z").alias("coordinates_z"),
 col("src.discovery_date").alias("discovery_date"),
 col("src.exploration_status").alias("exploration_status"),
 lit(data_date).cast("date").alias("effective_date"),
 lit("9999-12-31").cast("date").alias("expiration_date"),
 lit("Y").alias("current_flag"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 year(lit(data_date)).alias("effective_year")
 )

 # Build expired rows (for CHANGED records, expire the old version)
 changed_ids = change_detection.filter(col("change_type") == "CHANGED") \
 .select(col("curr.system_id").alias("system_id"))

 expired_rows = current_dim.join(changed_ids, "system_id") \
 .withColumn("expiration_date", lit(data_date).cast("date") - 1) \
 .withColumn("current_flag", lit("N"))

 # Unchanged rows stay as-is
 unchanged_rows = current_dim.join(
 change_detection.filter(col("change_type") == "UNCHANGED")
 .select(col("curr.system_id").alias("system_id")),
 "system_id"
 )

 # Historical rows (already expired) - read from full table
 historical_rows = spark.read.table("space_exploration.dim_system") \
 .filter(col("current_flag") == "N")

 # Combine all
 new_dim = new_rows \
 .unionByName(expired_rows, allowMissingColumns=True) \
 .unionByName(unchanged_rows, allowMissingColumns=True) \
 .unionByName(historical_rows, allowMissingColumns=True)

transform_count = new_dim.count()
print(f"Total dimension rows after SCD2: {transform_count:,}")

# ============================================
# WRITE PHASE: Write to staging branch
# ============================================
print("\n--- WRITE PHASE ---")

spark.sql("ALTER TABLE space_exploration.dim_system CREATE BRANCH IF NOT EXISTS staging")

new_dim.writeTo("space_exploration.dim_system") \
 .option("branch", "staging") \
 .overwritePartitions()

print("Written to staging branch")

# ============================================
# AUDIT PHASE
# ============================================
print("\n--- AUDIT PHASE ---")

staging_df = spark.read \
 .option("branch", "staging") \
 .table("space_exploration.dim_system")

audit_failures = []

# Audit 1: Row count
staged_count = staging_df.count()
print(f"Audit 1 - Row count: {staged_count:,}")
if staged_count == 0:
 audit_failures.append("FAIL: dim_system is empty")

# Audit 2: Surrogate key uniqueness
unique_keys = staging_df.select("system_key").distinct().count()
if staged_count != unique_keys:
 duplicates = staged_count - unique_keys
 audit_failures.append(f"FAIL: {duplicates} duplicate system_keys")
 print(f"Audit 2 - PK uniqueness: FAIL ({duplicates} duplicates)")
else:
 print(f"Audit 2 - PK uniqueness: PASS")

# Audit 3: Only one current row per natural key
current_rows = staging_df.filter(col("current_flag") == "Y")
current_count = current_rows.count()
unique_natural_keys = current_rows.select("system_id").distinct().count()
if current_count != unique_natural_keys:
 audit_failures.append(f"FAIL: Multiple current rows for same system_id")
 print(f"Audit 3 - Single current: FAIL")
else:
 print(f"Audit 3 - Single current: PASS ({current_count:,} current rows)")

# Audit 4: No NULLs in critical fields
null_count = staging_df.filter(
 col("system_key").isNull() |
 col("system_id").isNull() |
 col("effective_date").isNull() |
 col("current_flag").isNull()
).count()
if null_count > 0:
 audit_failures.append(f"FAIL: {null_count} NULL values in critical fields")
 print(f"Audit 4 - NULL check: FAIL")
else:
 print("Audit 4 - NULL check: PASS")

# Audit 5: Effective/expiration date logic
invalid_dates = staging_df.filter(
 col("effective_date") > col("expiration_date")
).count()
if invalid_dates > 0:
 audit_failures.append(f"FAIL: {invalid_dates} rows with effective > expiration")
 print(f"Audit 5 - Date logic: FAIL")
else:
 print("Audit 5 - Date logic: PASS")

# ============================================
# PUBLISH PHASE
# ============================================
print("\n--- PUBLISH PHASE ---")

if len(audit_failures) > 0:
 print("\nAUDIT FAILED:")
 for failure in audit_failures:
 print(f" {failure}")
 print("\nBranch NOT merged. Production unchanged.")
 spark.sql("ALTER TABLE space_exploration.dim_system DROP BRANCH staging")
 raise Exception(f"Audit failed with {len(audit_failures)} errors")
else:
 print("\nALL AUDITS PASSED")
 print("Merging staging branch to main...")

 spark.sql("""
 CALL system.fast_forward(
 table => 'space_exploration.dim_system',
 branch => 'main',
 to => 'staging'
 )
 """)

 spark.sql("ALTER TABLE space_exploration.dim_system DROP BRANCH staging")

 print(f"PUBLISHED: dim_system ({staged_count:,} rows)")

job.commit()

Job 3: Build Planet Dimension (SCD Type 2)

Similar pattern to dimsystem, but with a foreign key lookup to the _published dim_system.

Click to expand: build_dim_planet Glue Job

This job demonstrates FK lookup to a published dimension. It reads from the main branch of dim_system (not staging), ensuring it uses production-ready surrogate keys. The FK integrity audit validates this relationship.

# Glue Job: build_dim_planet
# SCD Type 2 with FK lookup to published dim_system

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *
from pyspark.sql.window import Window

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'DATA_DATE'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

data_date = args['DATA_DATE']
print("="*60)
print(f"Building Planet Dimension (SCD Type 2)")
print(f"Data Date: {data_date}")
print("="*60)

# ============================================
# LOAD PHASE
# ============================================
print("\n--- LOAD PHASE ---")

source_df = spark.read.table("space_exploration.stg_planets") \
 .filter(col("data_date") == data_date)

source_count = source_df.count()
print(f"Source rows: {source_count:,}")

if source_count == 0:
 print("No data. Exiting.")
 job.commit()
 sys.exit(0)

# CRITICAL: Read from PUBLISHED dim_system (main branch, not staging)
# This is why dim_planet must run AFTER dim_system publishes
dim_system = spark.read.table("space_exploration.dim_system") \
 .filter(col("current_flag") == "Y") \
 .select("system_key", "system_id")

print(f"dim_system lookup rows: {dim_system.count():,}")

# ============================================
# TRANSFORM PHASE
# ============================================
print("\n--- TRANSFORM PHASE ---")

# Check for existing data
try:
 current_dim = spark.read.table("space_exploration.dim_planet") \
 .filter(col("current_flag") == "Y")
 has_existing = current_dim.count() > 0
 max_key = current_dim.agg(max("planet_key")).collect()[0][0] or 0
except:
 has_existing = False
 max_key = 0

# Join source with dim_system to get surrogate key
source_with_sk = source_df.join(
 dim_system,
 source_df.system_id == dim_system.system_id,
 "left"
).select(
 source_df["*"],
 dim_system.system_key
)

# Track orphaned planets (no matching system)
orphaned = source_with_sk.filter(col("system_key").isNull()).count()
if orphaned > 0:
 print(f"WARNING: {orphaned} planets have no matching system!")

# SCD2 logic (simplified for first load demonstration)
window_spec = Window.orderBy("planet_id")

new_dim = source_with_sk.select(
 (row_number().over(window_spec) + max_key).alias("planet_key"),
 col("planet_id"),
 col("system_key"),
 col("planet_name"),
 col("planet_type"),
 col("orbital_period"),
 col("has_atmosphere"),
 col("habitability_score"),
 lit(data_date).cast("date").alias("effective_date"),
 lit("9999-12-31").cast("date").alias("expiration_date"),
 lit("Y").alias("current_flag"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 year(lit(data_date)).alias("effective_year")
).distinct()

# (Full SCD2 change detection would follow same pattern as dim_system)

transform_count = new_dim.count()
print(f"Dimension rows: {transform_count:,}")

# ============================================
# WRITE PHASE
# ============================================
print("\n--- WRITE PHASE ---")

spark.sql("ALTER TABLE space_exploration.dim_planet CREATE BRANCH IF NOT EXISTS staging")

new_dim.writeTo("space_exploration.dim_planet") \
 .option("branch", "staging") \
 .overwritePartitions()

print("Written to staging branch")

# ============================================
# AUDIT PHASE
# ============================================
print("\n--- AUDIT PHASE ---")

staging_df = spark.read \
 .option("branch", "staging") \
 .table("space_exploration.dim_planet")

audit_failures = []

# Audit 1: PK uniqueness
staged_count = staging_df.count()
unique_keys = staging_df.select("planet_key").distinct().count()
if staged_count != unique_keys:
 audit_failures.append(f"FAIL: Duplicate planet_keys")
else:
 print(f"Audit 1 - PK uniqueness: PASS")

# Audit 2: FK integrity to dim_system (published!)
published_system_keys = spark.read.table("space_exploration.dim_system") \
 .select("system_key").distinct()

orphaned_fk = staging_df \
 .filter(col("system_key").isNotNull()) \
 .select("system_key").distinct() \
 .join(published_system_keys, "system_key", "left_anti") \
 .count()

if orphaned_fk > 0:
 audit_failures.append(f"FAIL: {orphaned_fk} orphaned system_keys")
 print(f"Audit 2 - FK integrity: FAIL")
else:
 print("Audit 2 - FK integrity: PASS")

# Audit 3: NULL check
null_count = staging_df.filter(
 col("planet_key").isNull() | col("planet_id").isNull()
).count()
if null_count > 0:
 audit_failures.append(f"FAIL: {null_count} NULLs in critical fields")
else:
 print("Audit 3 - NULL check: PASS")

# ============================================
# PUBLISH PHASE
# ============================================
print("\n--- PUBLISH PHASE ---")

if len(audit_failures) > 0:
 print("\nAUDIT FAILED:")
 for f in audit_failures:
 print(f" {f}")
 spark.sql("ALTER TABLE space_exploration.dim_planet DROP BRANCH staging")
 raise Exception(f"Audit failed")
else:
 print("\nALL AUDITS PASSED")
 spark.sql("""
 CALL system.fast_forward(
 table => 'space_exploration.dim_planet',
 branch => 'main',
 to => 'staging'
 )
 """)
 spark.sql("ALTER TABLE space_exploration.dim_planet DROP BRANCH staging")
 print(f"PUBLISHED: dim_planet ({staged_count:,} rows)")

job.commit()

Job 4: Build Fact Table (Incremental with Delete-Insert)

This is the critical job. It reads from published dimensions (main branch), uses delete-then-insert pattern for the date partition, and includes left anti-join as a defensive check for missing records.

Click to expand: build_fact_system_updates Glue Job

This job loads facts incrementally by date. For a given data_date, it deletes existing records and inserts new ones. It validates FK integrity against all published dimensions before merging to main.

# Glue Job: build_fact_system_updates
# Incremental load with delete-then-insert + left anti-join defensive coding

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *
from pyspark.sql.window import Window

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'DATA_DATE'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

data_date = args['DATA_DATE']
print("="*60)
print(f"Building Fact Table: System Updates (Incremental)")
print(f"Data Date: {data_date}")
print("="*60)

# ============================================
# LOAD PHASE
# ============================================
print("\n--- LOAD PHASE ---")

# Read from PSA for this data_date
source_df = spark.read.table("space_exploration.stg_updates") \
 .filter(col("data_date") == data_date)

source_count = source_df.count()
print(f"Source rows: {source_count:,}")

if source_count == 0:
 print("No data. Exiting.")
 job.commit()
 sys.exit(0)

# Read PUBLISHED dimensions (main branch)
# Fact tables MUST read from published dimensions, not staging branches
dim_system = spark.read.table("space_exploration.dim_system") \
 .filter(col("current_flag") == "Y") \
 .select("system_key", "system_id")

dim_date = spark.read.table("space_exploration.dim_date") \
 .select("date_key", "full_date", "year")

print(f"dim_system rows: {dim_system.count():,}")
print(f"dim_date rows: {dim_date.count():,}")

# Read existing fact table for incremental logic
try:
 existing_fact = spark.read.table("space_exploration.fact_system_updates")
 existing_count = existing_fact.count()
 print(f"Existing fact rows: {existing_count:,}")
 max_key = existing_fact.agg(max("update_key")).collect()[0][0] or 0
except:
 print("No existing fact data (first load)")
 existing_fact = None
 max_key = 0

# ============================================
# TRANSFORM PHASE
# ============================================
print("\n--- TRANSFORM PHASE ---")

# Add date column for joining
source_with_date = source_df.withColumn(
 "update_date",
 to_date("update_timestamp")
)

# Lookup surrogate keys from published dimensions
fact_transformed = source_with_date \
 .join(dim_system, source_with_date.system_id == dim_system.system_id, "left") \
 .join(dim_date, source_with_date.update_date == dim_date.full_date, "left") \
 .select(
 col("date_key"),
 dim_system.system_key,
 source_with_date.update_timestamp,
 col("status"),
 col("ships_present"),
 col("resource_level"),
 lit(data_date).cast("date").alias("data_date"),
 dim_date.year.alias("date_year")
 )

# Track orphaned facts
orphaned_systems = fact_transformed.filter(col("system_key").isNull()).count()
orphaned_dates = fact_transformed.filter(col("date_key").isNull()).count()

if orphaned_systems > 0:
 print(f"WARNING: {orphaned_systems} facts have no matching system")
if orphaned_dates > 0:
 print(f"WARNING: {orphaned_dates} facts have no matching date")

# ============================================
# INCREMENTAL LOGIC: Delete-then-Insert + Left Anti-Join
# ============================================
print("\n--- INCREMENTAL LOGIC ---")

# Extract the date partition we're loading
date_partition_year = int(data_date[:4]) # e.g., 2026

if existing_fact is not None:
 # Step 1: Get existing records for this data_date
 existing_for_date = existing_fact.filter(col("data_date") == data_date)
 existing_for_date_count = existing_for_date.count()
 print(f"Existing records for data_date={data_date}: {existing_for_date_count:,}")

 # Step 2: Records NOT in this date (keep as-is)
 records_to_keep = existing_fact.filter(col("data_date") != data_date)
 keep_count = records_to_keep.count()
 print(f"Records to keep (other dates): {keep_count:,}")

 # Step 3: New records from source (delete-then-insert for this date)
 # Generate new surrogate keys starting after max existing key
 window_spec = Window.orderBy("update_timestamp", "system_key")

 new_records = fact_transformed.select(
 (row_number().over(window_spec) + max_key).alias("update_key"),
 col("date_key"),
 col("system_key"),
 col("update_timestamp"),
 col("status"),
 col("ships_present"),
 col("resource_level"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 col("data_date"),
 col("date_year")
 )

 new_count = new_records.count()
 print(f"New records for data_date={data_date}: {new_count:,}")

 # Step 4: Left anti-join defensive check
 # Find any records in destination that should exist but don't
 # (This catches edge cases where source has records not in our transform)
 # For this example, we're doing a full replace for the date, so this is informational

 # Step 5: Combine: kept records + new records (effectively delete-insert)
 final_fact = records_to_keep.unionByName(new_records)

else:
 # First load: all records are new
 window_spec = Window.orderBy("update_timestamp", "system_key")

 final_fact = fact_transformed.select(
 row_number().over(window_spec).alias("update_key"),
 col("date_key"),
 col("system_key"),
 col("update_timestamp"),
 col("status"),
 col("ships_present"),
 col("resource_level"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 col("data_date"),
 col("date_year")
 )

transform_count = final_fact.count()
print(f"Final fact rows: {transform_count:,}")

# ============================================
# WRITE PHASE
# ============================================
print("\n--- WRITE PHASE ---")

spark.sql("ALTER TABLE space_exploration.fact_system_updates CREATE BRANCH IF NOT EXISTS staging")

final_fact.writeTo("space_exploration.fact_system_updates") \
 .option("branch", "staging") \
 .overwritePartitions()

print("Written to staging branch")

# ============================================
# AUDIT PHASE
# ============================================
print("\n--- AUDIT PHASE ---")

staging_df = spark.read \
 .option("branch", "staging") \
 .table("space_exploration.fact_system_updates")

audit_failures = []

# Audit 1: Row count (source to target for this date)
staged_for_date = staging_df.filter(col("data_date") == data_date).count()
print(f"Audit 1 - Row count for {data_date}: Source={source_count:,}, Target={staged_for_date:,}")
if source_count != staged_for_date:
 # This might be OK if we filtered orphans, but flag for review
 print(f" Note: Difference of {source_count - staged_for_date} (likely orphaned records)")

# Audit 2: PK uniqueness
staged_count = staging_df.count()
unique_keys = staging_df.select("update_key").distinct().count()
if staged_count != unique_keys:
 audit_failures.append(f"FAIL: Duplicate update_keys")
 print(f"Audit 2 - PK uniqueness: FAIL")
else:
 print(f"Audit 2 - PK uniqueness: PASS ({staged_count:,} unique)")

# Audit 3: FK integrity to dim_system
published_system_keys = spark.read.table("space_exploration.dim_system") \
 .select("system_key").distinct()

orphaned_fk_system = staging_df \
 .filter(col("system_key").isNotNull()) \
 .select("system_key").distinct() \
 .join(published_system_keys, "system_key", "left_anti") \
 .count()

if orphaned_fk_system > 0:
 audit_failures.append(f"FAIL: {orphaned_fk_system} orphaned system_keys")
 print(f"Audit 3 - FK to dim_system: FAIL")
else:
 print("Audit 3 - FK to dim_system: PASS")

# Audit 4: FK integrity to dim_date
published_date_keys = spark.read.table("space_exploration.dim_date") \
 .select("date_key").distinct()

orphaned_fk_date = staging_df \
 .filter(col("date_key").isNotNull()) \
 .select("date_key").distinct() \
 .join(published_date_keys, "date_key", "left_anti") \
 .count()

if orphaned_fk_date > 0:
 audit_failures.append(f"FAIL: {orphaned_fk_date} orphaned date_keys")
 print(f"Audit 4 - FK to dim_date: FAIL")
else:
 print("Audit 4 - FK to dim_date: PASS")

# Audit 5: No NULLs in mandatory FKs
# (We allow NULLs for orphaned records, but track them)
null_system = staging_df.filter(col("system_key").isNull()).count()
null_date = staging_df.filter(col("date_key").isNull()).count()
print(f"Audit 5 - NULL FKs: {null_system} null system_key, {null_date} null date_key")

# Audit 6: Value range checks
invalid_resources = staging_df.filter(
 (col("resource_level") < 0) | (col("resource_level") > 1)
).count()
if invalid_resources > 0:
 audit_failures.append(f"FAIL: {invalid_resources} invalid resource_level values")
 print(f"Audit 6 - Value ranges: FAIL")
else:
 print("Audit 6 - Value ranges: PASS")

# Audit 7: No future timestamps
future_ts = staging_df.filter(col("update_timestamp") > current_timestamp()).count()
if future_ts > 0:
 audit_failures.append(f"FAIL: {future_ts} future timestamps")
 print(f"Audit 7 - Timestamps: FAIL")
else:
 print("Audit 7 - Timestamps: PASS")

# ============================================
# PUBLISH PHASE
# ============================================
print("\n--- PUBLISH PHASE ---")

if len(audit_failures) > 0:
 print("\nAUDIT FAILED:")
 for f in audit_failures:
 print(f" {f}")
 spark.sql("ALTER TABLE space_exploration.fact_system_updates DROP BRANCH staging")
 raise Exception(f"Audit failed with {len(audit_failures)} errors")
else:
 print("\nALL AUDITS PASSED")
 spark.sql("""
 CALL system.fast_forward(
 table => 'space_exploration.fact_system_updates',
 branch => 'main',
 to => 'staging'
 )
 """)
 spark.sql("ALTER TABLE space_exploration.fact_system_updates DROP BRANCH staging")
 print(f"PUBLISHED: fact_system_updates ({staged_count:,} rows)")

job.commit()

Step Functions Workflow

The Step Functions workflow orchestrates everything. Here’s the execution flow:

Stage 1 (Parallel): dim_date and dim_system run simultaneously—neither depends on the other
Stage 2: dim_planet runs after Stage 1 completes (needs dim_system’s surrogate keys)
Stage 3: fact_system_updates runs last (needs all dimension surrogate keys)

If any job fails, the workflow stops and sends an SNS notification.

Click to expand: Step Functions JSON Definition

{
 "Comment": "Dimensional Modeling Pipeline - Per-Table WAP Pattern",
 "StartAt": "Parallel Dimensions Stage 1",
 "States": {
 "Parallel Dimensions Stage 1": {
 "Type": "Parallel",
 "Comment": "dim_date and dim_system have no dependencies",
 "Branches": [
 {
 "StartAt": "Build dim_date (WAP)",
 "States": {
 "Build dim_date (WAP)": {
 "Type": "Task",
 "Resource": "arn:aws:states:::glue:startJobRun.sync",
 "Parameters": {
 "JobName": "build_dim_date"
 },
 "End": true
 }
 }
 },
 {
 "StartAt": "Build dim_system (WAP)",
 "States": {
 "Build dim_system (WAP)": {
 "Type": "Task",
 "Resource": "arn:aws:states:::glue:startJobRun.sync",
 "Parameters": {
 "JobName": "build_dim_system",
 "Arguments": {
 "--DATA_DATE.$": "$.data_date"
 }
 },
 "End": true
 }
 }
 }
 ],
 "Next": "Build dim_planet (WAP)",
 "Catch": [
 {
 "ErrorEquals": ["States.ALL"],
 "Next": "Notify Failure",
 "ResultPath": "$.error"
 }
 ]
 },
 "Build dim_planet (WAP)": {
 "Type": "Task",
 "Comment": "Depends on dim_system being published",
 "Resource": "arn:aws:states:::glue:startJobRun.sync",
 "Parameters": {
 "JobName": "build_dim_planet",
 "Arguments": {
 "--DATA_DATE.$": "$.data_date"
 }
 },
 "Next": "Build fact_system_updates (WAP)",
 "Catch": [
 {
 "ErrorEquals": ["States.ALL"],
 "Next": "Notify Failure",
 "ResultPath": "$.error"
 }
 ]
 },
 "Build fact_system_updates (WAP)": {
 "Type": "Task",
 "Comment": "Depends on ALL dimensions being published",
 "Resource": "arn:aws:states:::glue:startJobRun.sync",
 "Parameters": {
 "JobName": "build_fact_system_updates",
 "Arguments": {
 "--DATA_DATE.$": "$.data_date"
 }
 },
 "Next": "Pipeline Succeeded",
 "Catch": [
 {
 "ErrorEquals": ["States.ALL"],
 "Next": "Notify Failure",
 "ResultPath": "$.error"
 }
 ]
 },
 "Notify Failure": {
 "Type": "Task",
 "Resource": "arn:aws:states:::sns:publish",
 "Parameters": {
 "TopicArn": "arn:aws:sns:ap-southeast-2:ACCOUNT:pipeline-alerts",
 "Message.$": "States.Format('Pipeline failed: {}', $.error.Cause)",
 "Subject": "Data Pipeline Failed"
 },
 "Next": "Pipeline Failed"
 },
 "Pipeline Succeeded": {
 "Type": "Succeed"
 },
 "Pipeline Failed": {
 "Type": "Fail",
 "Error": "PipelineError",
 "Cause": "One or more jobs failed - check individual job logs"
 }
 }
}

Start the workflow with input:

{
 "data_date": "2026-01-17"
}

This data_date propagates to each job, enabling:

PSA partition reads (WHERE data_date = '2026-01-17')
Historical reprocessing (re-run with a past date)
Dependency triggers (S3 event on partition write triggers workflow)

Dependency Triggers with S3 Events

You can improve this further by using partition writes as triggers. Here’s how to set that up:

Option 1: S3 Event → EventBridge → Step Functions

When a file lands in landing/updates/, trigger the pipeline:

# EventBridge Rule
{
 "source": ["aws.s3"],
 "detail-type": ["Object Created"],
 "detail":
 {
 "bucket": { "name": ["space-data-pipeline-demo"] },
 "object": { "key": [{ "prefix": "landing/updates/" }] },
 },
}

Option 2: Glue Trigger on Partition

Configure Glue to watch for new partitions:

# Check if partition exists before running
partition_path = f"s3://space-data-pipeline-demo/staging/stg_updates/data_date={data_date}/"

import boto3
s3 = boto3.client('s3')

try:
 s3.head_object(
 Bucket='space-data-pipeline-demo',
 Key=f'staging/stg_updates/data_date={data_date}/_SUCCESS'
 )
 print(f"Partition exists - proceeding")
except:
 print(f"Partition not ready - exiting")
 sys.exit(0)

This lets you build dependency chains based on partition availability.

The Moment of Truth: Querying

Now let’s query the properly partitioned dimensional model:

-- Query leverages date partition pruning
SELECT
 dt.year,
 dt.quarter,
 d.main_star_type,
 COUNT(*) as updates,
 SUM(f.ships_present) as total_ships,
 ROUND(AVG(f.resource_level), 3) as avg_resources
FROM
 space_exploration.fact_system_updates f

 LEFT JOIN space_exploration.dim_system d
 ON f.system_key = d.system_key
 AND d.current_flag = 'Y' -- Only current dimension records!

 LEFT JOIN space_exploration.dim_date dt
 ON f.date_key = dt.date_key
WHERE
 dt.year = 2026 -- Partition pruning!
 AND dt.quarter = 1
GROUP BY
 dt.year, dt.quarter, d.main_star_type
ORDER BY
 total_ships DESC;

Notice d.current_flag = 'Y' — this is how you query SCD Type 2 dimensions for current state. For point-in-time analysis:

-- Point-in-time analysis: What was system status on Jan 15, 2026?
SELECT
 d.system_name,
 d.exploration_status,
 d.effective_date,
 d.expiration_date
FROM
 space_exploration.dim_system d
WHERE
 d.system_id = 'SOL'
 AND DATE '2026-01-15' BETWEEN d.effective_date AND d.expiration_date;

A Few Things Worth Remembering

WAP at each table, not just at the end. Each job should do its own Write-Audit-Publish cycle. Failures are isolated. Downstream jobs read from published (main) tables.

Landing is raw, Staging is persistent. Landing = 1:1 from source, latest only. Staging (PSA) = partitioned by data_date for historical reprocessing.

dim_date is a reference dimension. Load it once with a wide date range. Partition by year. Rarely needs updates.

SCD Type 2 for dimensions that change. Add effective_date, expiration_date, current_flag. Partition by effective year. Query with current_flag = 'Y' for current state.

Incremental facts with delete-then-insert. For a given date partition: delete existing, insert new. Use left anti-join as defensive check for missing records.

Partition strategically:

dim_date: by year
dim_system/planet: by effective_year (SCD2)
fact: by date_year

Use data_date for dependencies. When stg_updates/data_date=2026-01-17/ is written, trigger the downstream pipeline. This makes dependencies explicit and reprocessing easy.

That ten-minute query? Down to 1.2 seconds with partition pruning.

That conference room moment? Each job’s WAP cycle catches issues before they propagate.

These patterns work. They’ve saved me from embarrassment, late nights, and angry emails. They’ll do the same for you.

Posts on Ghost in the data

SQL Tells You What. Comments Tell You Why.

The self-documenting ceiling

What SQL literally cannot tell you

The deduplication that nobody remembers adding

When two models define the same word differently

dbt descriptions aren’t exempt

The archaeology problem

Don't Go Dark: Visibility Is a Data Engineering Skill

What Atwood Actually Meant

Why Data Engineering Goes Dark More Than Anyone Else

The IC Chapter: Staying Visible Without Performing Busyness

The Distributed Team Problem

The Fear Dimension: What’s Really Happening When Engineers Go Dark

The Lead Chapter: Building Conditions Where Your Team Doesn’t Go Dark

The Visible Data Team Playbook

Going Back to Mia

The Closing Thought

The Broken Window in Your Data Pipeline

A broken window in the Bronx, a broken window in your warehouse

The thing that’s different about data

What a contaminated watershed actually looks like

The tools we built to tolerate this

We’ve built monitoring. Why isn’t it working?

The stakeholder sees it differently

Nobody owns the broken window

The original theory’s mistake — and ours

What collective efficacy looks like in a data team

The window was broken before I noticed

Five Worlds of Data Engineering

World 1: The Modern Analytics Shop

World 2: The Enterprise Legacy Estate

World 3: The Product Engine

World 4: The Regulated Pipeline

World 5: The Internal Platform

So What?

Your Data Platform Costs More Than It Should

You can’t optimise what you can’t see

The idle tax is your single biggest waste

The most expensive query is the one that scans everything

Full refreshes are the most expensive default in data engineering

The hidden costs your Snowflake dashboard won’t show you

Tag everything or you’ll optimise blind

Kill the zombies

Cost anomalies will find you — or you can find them first

What your data model costs you

Coming back to why this matters

Why Your Pipeline Finishes Later Every Month

Measure the pipeline in stages, not as a single number

Your DAG probably has dependencies it doesn’t need

The critical path is the only thing worth optimising

Shifting right: diagnose it before it breaks your SLA

Partition and cluster to eliminate full table scans

Make every task idempotent or debugging becomes a nightmare

Most “we need real-time” requests actually need “we need faster batch”

Measure time-to-consumer, not just pipeline runtime

The view-on-view problem: the compound interest of bad performance

Coming back to Priya’s scatter plot

Stop Building Salesforce Integrations From Scratch

The Problem Nobody Warns You About

What Is Snowflake OpenFlow?

Setting Up the Salesforce Connector: A Walkthrough

Phase 1: Salesforce Configuration

Phase 2: Snowflake Preparation

Phase 3: Deploying the Connector

What You Get

A Note on Multiple Sync Frequencies

Schema Evolution: Where OpenFlow Earns Its Keep

A Word About Fivetran

The Zero-Copy Alternative

The Honest Trade-Offs

Getting Started

The Real Why

Your Data Model Isn't Broken, Part II: The Refactoring Playbook

The vine that eats the tree

Stop publishing before you’ve checked

The archaeology of SQL

Expand, then contract

The ship that replaced itself

The numbers, for the sceptics