Posts on Ghost in the data

Stop Building Salesforce Integrations From Scratch

Chris Hillman — Sat, 04 Apr 2026 09:00:00 +1100

Let me tell you about Marcus.

Marcus was on a team I led a few years back. Sharp, motivated, the kind of engineer who actually read documentation before writing code. When the business asked us to get Salesforce data into our warehouse, Marcus volunteered. He’d done API work before. He figured a few weeks, tops.

He scoped it carefully. Built a Python service that authenticated via OAuth, pulled Account, Contact, and Opportunity objects through the Bulk API, flattened the nested JSON into relational tables, handled pagination, managed rate limits. Wrote solid tests. Documented everything. The kind of work you’d point to in a code review and say this is how it’s done.

It worked beautifully. For about three months.

Then our Salesforce admin added a custom field to the Opportunity object — Renewal_Likelihood__c — and nobody told the data team. The pipeline didn’t fail. That’s the insidious part. It kept running, kept landing data. It just quietly dropped the new field on the floor.

When Marcus tracked it down, he added the field. Then he realised there were eleven other custom fields that had been added since go-live that the pipeline was silently ignoring. And the compound address fields — BillingAddress, ShippingAddress — had never worked with the Bulk API in the first place. He’d been extracting the component parts (BillingStreet, BillingCity) as a workaround, but the workaround had a bug that truncated postal codes for international addresses.

Marcus spent three weeks patching all of it. Three weeks of a talented engineer doing work that a managed connector handles automatically.

When I watched a good engineer spend the best part of a month on a problem that shouldn’t exist. Marcus wasn’t learning anything. He wasn’t growing. He was hand-coding schema detection logic for the fourth time because Salesforce ships three major releases a year and our sales ops team adds custom fields like they’re decorating a Christmas tree.

I’m telling you this because Snowflake OpenFlow is one of those things that sounds like “just another integration tool” until you’ve lived through the alternative. What I want to show you today is how to set it up with Salesforce — step by step, with enough detail that you could do it tomorrow — and more importantly, why the schema evolution feature alone justifies the switch from custom code.

If you’ve ever maintained a custom Salesforce integration, you already know why this matters.

The Problem Nobody Warns You About

Here’s what the “just build it yourself” crowd doesn’t tell you: the Salesforce API isn’t one API. It’s an ecosystem of overlapping interfaces, each with its own quirks, limits, and failure modes.

The REST API handles small, synchronous requests well — but try pulling a million Opportunity records through it and you’ll burn through your rate limits before lunch. The Bulk API 2.0 is designed for volume — it can handle up to 150 million records in a 24-hour window — but it doesn’t support compound fields like BillingAddress or MailingAddress. Those silently return nothing. Not an error. Nothing.

Then there’s SOQL, Salesforce’s proprietary query language, which looks enough like SQL to trick you into thinking you understand it. Until you hit the 2,000-record offset limit, or try to resolve a polymorphic relationship where a WhoId field on a Task could reference either a Contact or a Lead depending on the record. That’s not a data modelling problem — that’s a “your pipeline logic needs to branch based on the referenced object type” problem.

And rate limits. Enterprise Edition gives you 100,000 API calls per 24 hours, plus 1,000 per user licence. Sounds generous until you remember that your BI tool, marketing automation platform, customer support system, and your data pipeline are all drinking from the same well.

But schema evolution is the silent killer.

Every Salesforce org is different because every business is different. Your Salesforce admin creates custom objects and fields to match how your company actually works. That’s the point — it’s a customisation platform. But every custom field added to Salesforce is a potential break point for any integration that doesn’t handle schema changes automatically.

And those changes happen constantly. Research suggests schemas in enterprise SaaS tools change roughly every three days on average. In Salesforce specifically, between admin-driven customisation and Salesforce’s own three-times-a-year release cycle (Spring, Summer, Winter), the schema you built against last quarter may not be the schema you’re ingesting from today.

This is the trap. Building the initial integration isn’t hard. Any competent engineer can get Salesforce data into a warehouse. Keeping it working — across schema changes, API version deprecations, authentication token rotations, and rate limit adjustments — is where it eats your life.

I’ve seen teams eliminate two full days of monthly engineering maintenance by migrating off custom connectors. Two days, every month, spent on work that a managed tool does automatically. That’s not engineering — it’s janitorial work wearing an engineering costume.

What Is Snowflake OpenFlow?

OpenFlow is Snowflake’s native, fully managed data integration service. It’s not a partnership or a marketplace app — it’s a first-party feature built into the platform. If you’re already running Snowflake, OpenFlow runs inside your existing environment.

The backstory matters. In November 2024, Snowflake acquired a company called Datavolo for roughly $110 million. Datavolo was founded by Joe Witt, who co-created Apache NiFi back when it was an NSA project. That NiFi heritage is the foundation of OpenFlow — curated, versioned connector definitions powered by NiFi’s flow engine, wrapped in a managed Snowflake experience.

Architecturally, it splits into two pieces. The control plane lives inside Snowflake and gives you pipeline management, scheduling, and monitoring through the Snowsight UI. The data plane handles the actual work and can run in two modes: Snowflake Deployments (running on Snowpark Container Services within Snowflake’s own infrastructure) or BYOC (Bring Your Own Cloud, running as a Kubernetes cluster in your VPC). For most teams starting out, the SPCS option is simpler — it went generally available in November 2025 on AWS and Azure.

Data flows into Snowflake primarily through Snowpipe Streaming for the initial load, with merge queries handling incremental updates. The result is structured tables in your Snowflake database — not staged JSON files, not semi-structured VARIANT columns. Actual, queryable tables with proper column types.

The connector catalogue currently covers about 20 curated sources: databases (PostgreSQL, MySQL, SQL Server, Oracle), SaaS platforms (Salesforce, Workday, Jira, LinkedIn Ads, Meta Ads, Google Ads), cloud storage (Google Drive, SharePoint, Box), and streaming sources (Kafka, Kinesis). It’s not Fivetran’s 500+ connector library, and nobody’s pretending it is. But the sources it does cover are the ones that account for the vast majority of enterprise data movement.

One important caveat up front: the Salesforce Bulk API connector is still in preview as of March 2026. It works. Preview in Snowflake terms means the feature is functional but not yet covered by production support SLAs. Keep that in your mental model as we walk through the setup.

Setting Up the Salesforce Connector: A Walkthrough

This section is the step-by-step. I’m going to walk through the full setup — Salesforce configuration, Snowflake preparation, and connector deployment — with enough detail that you could follow along on your own org.

Phase 1: Salesforce Configuration

The connector uses JWT Bearer Flow for authentication, which means you need an RSA key pair and a Connected App configured in Salesforce.

Generate your RSA key pair. Open a terminal and run:

# Generate 2048-bit RSA private key
openssl genrsa -out salesforce_private_key.pem 2048

# Generate the corresponding public certificate (valid 1 year)
openssl req -new -x509 -key salesforce_private_key.pem \
 -out salesforce_certificate.pem -days 365

Keep that private key safe. You’ll need it when configuring the connector in Snowflake.

Create the External Client App in Salesforce. Navigate to Setup → Apps → App Manager → New External Client App. Enable OAuth and add two scopes: api and refresh_token (sometimes labelled offline_access). Enable the JWT Bearer Flow toggle and upload your salesforce_certificate.pem public certificate. Save the app and record the Consumer Key and Consumer Secret that Salesforce generates.

Configure access. On the app’s Manage settings, set the Permitted Users policy to “Admin approved users are pre-authorized.” Then assign the appropriate user profiles or permission sets. The user account the connector will authenticate as needs read access to every object you intend to sync.

Phase 2: Snowflake Preparation

Before deploying the connector, you need a database, schema, role, and warehouse ready for it.

-- Create a dedicated database for Salesforce data
CREATE DATABASE IF NOT EXISTS SALESFORCE_RAW;
CREATE SCHEMA IF NOT EXISTS SALESFORCE_RAW.OPENFLOW;

-- Create a connector-specific role
CREATE ROLE IF NOT EXISTS OPENFLOW_SF_CONNECTOR;

-- Grant the role permissions to write to the destination
GRANT USAGE ON DATABASE SALESFORCE_RAW TO ROLE OPENFLOW_SF_CONNECTOR;
GRANT USAGE ON SCHEMA SALESFORCE_RAW.OPENFLOW TO ROLE OPENFLOW_SF_CONNECTOR;
GRANT CREATE TABLE ON SCHEMA SALESFORCE_RAW.OPENFLOW TO ROLE OPENFLOW_SF_CONNECTOR;

-- Create a warehouse for merge operations
CREATE WAREHOUSE IF NOT EXISTS OPENFLOW_WH
 WAREHOUSE_SIZE = 'SMALL'
 AUTO_SUSPEND = 60
 AUTO_RESUME = TRUE;

GRANT USAGE ON WAREHOUSE OPENFLOW_WH TO ROLE OPENFLOW_SF_CONNECTOR;
GRANT OPERATE ON WAREHOUSE OPENFLOW_WH TO ROLE OPENFLOW_SF_CONNECTOR;

For SPCS deployments, you also need a network rule allowing the connector to reach your Salesforce instance. OpenFlow runs inside Snowflake’s container environment, and by default it can’t reach external endpoints without explicit permission:

-- Allow egress to your Salesforce instance
CREATE NETWORK RULE IF NOT EXISTS OPENFLOW_SF_EGRESS
 TYPE = HOST_PORT
 MODE = EGRESS
 VALUE_LIST = ('yourinstance.salesforce.com:443');

-- Wrap it in an external access integration
CREATE EXTERNAL ACCESS INTEGRATION IF NOT EXISTS OPENFLOW_SF_ACCESS
 ALLOWED_NETWORK_RULES = (OPENFLOW_SF_EGRESS)
 ENABLED = TRUE;

Replace yourinstance.salesforce.com with your actual Salesforce instance hostname. If you’re not sure what that is, log in to Salesforce and look at the URL bar — it’s the domain before .salesforce.com.

Phase 3: Deploying the Connector

Now the fun part. Navigate to Snowsight → Data → Openflow in the left sidebar. Find the “Openflow connector for Salesforce Bulk API” tile and add it to your runtime.

This opens the NiFi canvas — a visual flow editor where the connector’s pre-built process groups appear. Right-click the Salesforce process group and select Configure Parameters. Here’s where you’ll enter:

Salesforce Instance URL — e.g., https://yourinstance.salesforce.com
Consumer Key — from the Connected App you created
Consumer Secret — same source
Private Key — the contents of salesforce_private_key.pem (the full PEM text, headers included)
Username — the Salesforce user the connector authenticates as
Filter — the objects to sync, by API name

The Filter parameter is where you define what to pull. List standard objects by name (Account, Contact, Opportunity, Lead, Task, Event) and custom objects using their API name with the __c suffix:

There’s also a Special Objects Filter for objects that require different handling (like User or RecordType).

Once the parameters are configured, enable all controller services in the process group, then start the connector. The initial sync will pull a full snapshot of every object in your filter list. Subsequent runs perform incremental syncs using Salesforce’s SystemModstamp field to identify changed records.

What You Get

Once the connector runs, you’ll find one table per Salesforce object in SALESFORCE_RAW.OPENFLOW. The tables are flattened and structured — every Salesforce field becomes a column with an appropriate Snowflake data type. No VARIANT columns to parse. No nested JSON to unpack.

An Account table will have columns like ID, NAME, INDUSTRY, ANNUALREVENUE, BILLINGSTREET, BILLINGCITY, BILLINGSTATE, BILLINGPOSTALCODE, CREATEDDATE, LASTMODIFIEDDATE, and so on. Custom fields like Renewal_Likelihood__c appear as RENEWAL_LIKELIHOOD__C.

Two metadata columns are added automatically: ISDELETED (tracking Salesforce soft deletes) and the system timestamp fields. This is important — hard deletes in Salesforce are not reflected. If a record is permanently deleted in Salesforce, it won’t disappear from your Snowflake table. You’ll need to handle that in your downstream dbt models if it matters for your use case.

A Note on Multiple Sync Frequencies

Not every Salesforce object changes at the same pace. Your Opportunity table might need hourly syncs while Account is fine with daily. OpenFlow handles this by letting you deploy multiple connector instances in the same runtime — each with its own filter list and schedule — at no additional compute cost beyond the shared runtime.

Schema Evolution: Where OpenFlow Earns Its Keep

This is the feature that matters most, and the one that would have saved Marcus three weeks.

When someone adds a new custom field to a Salesforce object — say the sales ops team creates Deal_Confidence_Score__c on the Opportunity object — the OpenFlow connector automatically detects the new field and adds a corresponding column to the Snowflake destination table on the next sync. No configuration change. No redeployment. No Teams message at 7am asking the data team to “add that new field we made yesterday.”

The column appears, data starts flowing into it, and your dbt models can reference it whenever you’re ready.

For field renames, OpenFlow takes a conservative approach: the old column stays in place (with stale data from the last sync before the rename), and a new column is created under the new field name. This means you’ll have both Old_Field_Name__c and New_Field_Name__c in your table for a period. That’s actually useful for audit purposes — you can see exactly when the rename happened by comparing timestamps — but it does mean your downstream queries need updating.

Here’s the honest gap: new custom objects are not auto-discovered. If someone creates an entirely new custom object in Salesforce, you have to manually add it to the connector’s Filter parameter. Schema evolution handles field-level changes automatically, but object-level discovery is still a manual step.

And field type changes — say someone changes a text field to a picklist, or a number to a formula — aren’t comprehensively documented in the current OpenFlow materials. In practice, this is a rare enough occurrence that it hasn’t been a problem in my experience, but it’s worth knowing the edge case exists.

Compare this to what Marcus had to maintain. His custom integration used a hardcoded list of fields per object. Every time a field was added, renamed, or retyped, someone had to update the Python code, test it, deploy it, and verify the data. Usually that “someone” was Marcus, usually at an inconvenient time, and usually because nobody told him the change was coming.

The value of automatic schema evolution isn’t technical elegance. It’s that your engineers stop spending time on schema babysitting and start spending it on work that actually matters — building models, improving data quality, answering the questions the business is actually asking.

A Word About Fivetran

I’d be doing you a disservice if I wrote about managed Salesforce connectors without mentioning Fivetran. For most of the last decade, Fivetran has been the gold standard here, and their Salesforce connector is genuinely excellent.

Fivetran’s schema evolution handling is more mature and more configurable than OpenFlow’s current offering. They offer three schema change policies: ALLOW_ALL (sync everything automatically, including new tables), ALLOW_COLUMNS (auto-add new columns but not new tables), and BLOCK_ALL (manual opt-in for everything). They also generate a LOG table that records every schema change event — create_table, alter_table, drop_table — giving you an audit trail that’s useful for debugging and compliance. Field type widening (like INT to LONG) happens automatically; narrowing changes are blocked to prevent data loss.

Their pricing model is fundamentally different from OpenFlow. Fivetran charges per Monthly Active Row (MAR) — the count of distinct rows synced per connection per month. A unique primary key counts once regardless of how many times it’s updated. Approximate pricing clusters around $500 per million MAR on the Standard plan, with a $12,000 annual minimum commitment. Since early 2025, MAR is calculated per connection rather than account-wide, which eliminated the bulk discount that previously benefited multi-connector setups.

OpenFlow, by contrast, charges for compute — SPCS credits for the container runtime, Snowpipe Streaming costs for ingestion, and warehouse costs for merge operations. There’s no per-row fee and no separate licence. It’s included with Snowflake. The trade-off is that your management compute pool runs continuously while a deployment exists, which means you’re paying a base cost even when no data is flowing. One estimate I’ve seen puts idle costs at roughly $10 per day, though this varies with configuration.

For a single large Salesforce org, OpenFlow’s compute-based model may end up cheaper than Fivetran’s per-row pricing — especially if your Salesforce data doesn’t churn heavily. For environments where you need 30+ connectors across different source types, Fivetran’s breadth and maturity is hard to argue with. They have 500+ connectors. OpenFlow has about 20.

The honest framing: OpenFlow is the better choice if you’re already deeply invested in Snowflake, value platform consolidation, and your primary sources are in the current connector catalogue. Fivetran is the safer choice if you need broad connector coverage, battle-tested reliability, and you’d rather pay a premium for someone else to handle the ops.

They’re not mutually exclusive, either. I’ve seen teams run Fivetran for the long tail of small connectors while using OpenFlow for their highest-volume sources where compute-based pricing wins.

The Zero-Copy Alternative

There’s a third path I haven’t mentioned yet, and it’s worth understanding even if it’s not right for every team: zero-copy data federation.

Salesforce Data Cloud offers a zero-copy integration with Snowflake that flips the entire model on its head. Instead of extracting data from Salesforce and loading it into your warehouse, zero-copy gives you direct read access to Salesforce data without moving it at all. Salesforce’s query pushdown engine handles the work — when you query the federated data, Salesforce pushes the query to the source, filters and aggregates there, and returns only the results you need. No pipelines to build. No schemas to manage. No sync schedules to configure.

It works bidirectionally, too. You can share data from Snowflake back into Salesforce Data Cloud, letting sales reps see warehouse-enriched signals directly on Account and Opportunity pages without anyone exporting a CSV or building a reverse ETL pipeline. The integration runs on Apache Iceberg under the hood, which at least means the underlying format is open.

For handling new custom objects — the gap I flagged with OpenFlow — zero-copy sidesteps the problem entirely. There’s nothing to discover because there’s nothing to sync. The data stays where it is, and you access it in place. As your Salesforce org evolves, the federated view evolves with it.

That’s genuinely appealing. For proof-of-concept work, rapid prototyping, or use cases where you need real-time access to Salesforce data without building infrastructure, zero-copy is fast and effective.

But here’s where I get cautious.

It comes at a premium. Salesforce Data Cloud runs on a consumption credit model — credits are purchased in bundles (roughly $500 per 100,000 credits at list price), and every action burns them. Zero-copy federation queries consume about 70 credits per million records, which sounds reasonable until you’re running it at scale across multiple business units. The pricing model has simplified since late 2025 — Salesforce consolidated multiple credit types into a single fungible credit and made ingestion from core Salesforce products free — but it’s still a consumption model with real costs that compound quickly if you’re not watching.

More importantly, zero-copy creates a deep platform dependency. Your data access is mediated entirely through Salesforce and Snowflake’s partnership. If Salesforce changes their terms, adjusts their pricing (which they’ve done repeatedly), or if you decide to move off Snowflake to Databricks or BigQuery, that zero-copy integration doesn’t come with you. You’d need to build actual data movement infrastructure — the thing you avoided — under time pressure and with no existing pipeline to fall back on.

I value keeping infrastructure portable. Not because I’m paranoid about vendor lock-in — I’m pragmatic about it. I’ve been in this industry long enough to see what happens when a new CEO arrives and wants to renegotiate every vendor contract, or when a cloud provider changes their pricing structure overnight, or when the company you depend on gets acquired and the roadmap shifts. Having your data physically in your own warehouse, in formats you control, with pipelines you can redirect to a different destination — that’s insurance. Not glamorous insurance, but the kind you’re grateful for when you need it.

The balance is real, though. You always have to weigh portability against the time you spend building and maintaining pipelines. If your team is drowning in pipeline maintenance, zero-copy might buy you breathing room while you stand up proper infrastructure. Just go in with your eyes open about what you’re trading for that convenience.

There’s a more fundamental concern, though, and it’s the one that won’t appear in any vendor comparison chart.

Zero-copy makes it dangerously easy to skip the data warehouse entirely. When you give analysts and report builders direct read access to raw Salesforce data, the temptation to build dashboards straight from it is enormous. And for a quick proof of concept or a one-off analysis, fine. But the moment those PoC dashboards become “the dashboard the VP checks every Monday,” you’ve got a problem.

Raw source data doesn’t tell you which CustomerNo field to use — the one from the legacy migration, the one from the 2023 CRM consolidation, or the one from the current integration. It doesn’t encode the business rule that says “Closed Won” opportunities in APAC exclude training deals under $5,000 because those are tracked separately. It doesn’t flag that the Last_Activity_Date field is unreliable for accounts owned by the partner team because they log activities in a different system.

That knowledge lives in your data warehouse layer. It lives in the dbt models that your team built over months of conversations with stakeholders, through change management cycles, through debugging sessions where someone finally said “oh, we stopped using that field in Q3 because…” That’s not transformation logic — it’s institutional memory encoded as code. It connects raw data across domains and gives the business a full landscape picture instead of a narrow, single-source view. It catches the behavioural anomalies unique to your organisation — the sales rep who bulk-updates 500 records every Friday afternoon, the integration that occasionally double-fires during DST transitions, the custom object that gets repurposed every time the business restructures.

So by all means, evaluate zero-copy for the use cases where it shines. Real-time signals on Account pages for sales reps? Great fit. Federated access for a specific analytics team that knows the source data intimately? Reasonable. But don’t let it replace the warehouse layer. That layer is where the hard-won understanding of your data lives, and it’s harder to recreate than any pipeline.

The Honest Trade-Offs

I’ve been positive about OpenFlow because I think it solves a real problem well. But I’d be violating my own writing principles if I didn’t lay out the limitations clearly.

The Salesforce connector is still in preview. It works, but it’s not covered by production support SLAs. If you’re running a mission-critical pipeline that your CFO stares at every Monday morning, that matters.

Custom Salesforce domains aren’t supported. If your org uses a vanity URL (like mycompany.my.salesforce.com with a custom domain), check the documentation carefully before committing.

Hard deletes aren’t tracked. The ISDELETED column catches soft deletes, but records that are permanently purged from Salesforce will persist in your Snowflake tables indefinitely. You’ll need a reconciliation process if that matters for your use case.

Certain field types are silently dropped. location, address (compound), and base64 fields are not synced. The connector doesn’t error on these — it just ignores them. This is the kind of thing that bites you three months in when someone asks why geographic data isn’t in the warehouse.

Formula fields require full refresh. Because formula field values are computed server-side and don’t update SystemModstamp, they can’t be synced incrementally. You need a separate connector instance running full refreshes to capture formula field changes. That’s not a bug — it’s how Salesforce works — but it’s a gotcha if you’re not expecting it.

No relationship traversal. You can’t configure the connector to follow lookups and pull related objects automatically. Each object is pulled independently. Joining them is your job in the transformation layer, which is where it belongs anyway if you’re running dbt.

None of these are dealbreakers. But every one of them is the kind of thing that bites an engineer at 4pm on a Friday if they weren’t expecting it. Now you’re expecting it.

Getting Started

If you want to try this on your own Salesforce org, here’s my recommended order of operations:

Start with a non-production Salesforce sandbox and a Snowflake trial account if you don’t have a dev environment handy. Pick two or three standard objects — Account, Contact, Opportunity — and one custom object if your org has them. Run through the three-phase setup I described above. Get data flowing, verify the table structures, and then add a custom field to one of the objects in Salesforce. Wait for the next sync. Watch the column appear automatically in Snowflake.

That’s the moment it clicks. That’s the moment you stop thinking about managed connectors as a luxury and start thinking about custom API code as technical debt you’re choosing to carry.

Once you’ve validated the core flow, layer on your dbt models. OpenFlow lands raw data — it doesn’t transform it. Your staging models handle naming conventions, type casting, soft delete filtering, and the join logic that stitches related objects together. This is the same pattern you’d use with Fivetran or any other EL tool: land it raw, transform it in the warehouse.

For teams already running Fivetran for Salesforce, I’m not suggesting you rip it out tomorrow. If it’s working and the cost is acceptable, that’s a solved problem. But the next time you’re adding a new high-volume source that’s in OpenFlow’s catalogue — especially if you’re already paying for Snowflake — it’s worth running the numbers. You might find that native integration with compute-based pricing is the better deal.

The Real Why

I started this article with Marcus’s story because it illustrates something I feel strongly about: your engineers’ time is the most expensive resource in your data organisation, and spending it on solved problems is a leadership failure.

Building a custom Salesforce integration isn’t impressive. It was impressive in 2016, when the tooling didn’t exist. Today, it’s a choice to take on maintenance burden that a managed service handles automatically. It’s choosing to babysit schema changes instead of building the models and analyses that actually move the business forward.

Marcus is a senior engineer now. He builds dimensional models and designs data products that directly influence how the sales team allocates resources. He doesn’t write API integration code anymore. Not because he can’t — because his time is worth more than that.

But here’s what I keep coming back to. The tools keep getting better — OpenFlow, Fivetran, zero-copy, and yes, AI-assisted pipeline generation. Every year there’s a new thing that promises to automate another piece of the data engineering workflow. And many of them genuinely do.

What they don’t automate is the part that actually matters.

No tool — not OpenFlow, not Fivetran, not an AI agent — is going to sit in a room with your head of sales and ask “what keeps you up at night?” No connector is going to notice that the way your company defines “active customer” has quietly drifted from what the CRM tracks. No pipeline is going to push back and say “we could build that dashboard, but the metric you’re asking for doesn’t answer the question you actually have.”

That’s the connective tissue between raw data and business value. It’s a person who understands the domain, who’s been present through the change management cycles, who’s built relationships with the stakeholders and knows which CustomerNo field to use because they were in the room when the decision was made three years ago. It’s someone who asks clarifying questions instead of presuming they already know the answer.

AI will keep getting better at the mechanical parts of data engineering. Schema detection, pipeline code generation, anomaly detection — those are well-defined problems that automation is suited for. But the assumption that any tool can skip the human step — the clarification, the context, the judgment about what’s actually worth building — that’s where I’ve seen projects go sideways. Not because the technology failed, but because nobody asked the right questions before building.

OpenFlow isn’t perfect. The connector catalogue is young, the Salesforce connector is in preview, and Fivetran still wins on breadth and battle-hardened maturity. But OpenFlow represents something important: the data platform taking responsibility for data movement, not just data storage and compute. It’s Snowflake saying “we’ll handle getting the data in — you focus on making it useful.”

That last part — making it useful — is still your job. And it’s the part that no tool can do for you. The best thing a managed connector gives you isn’t fewer lines of code. It’s time back. Time to spend on the work that actually requires a human: understanding the business, building the right models, and making sure the data tells an accurate story.

The next time someone on your team says “I’ll just build a quick Salesforce integration” — send them this article. And the next time someone says “AI can just handle the data pipeline” — ask them who’s going to sit with the stakeholders and figure out what the pipeline should actually deliver.

That’s still you. Make sure you have the time for it.

Your Data Model Isn't Broken, Part II: The Refactoring Playbook

Chris Hillman — Sat, 28 Mar 2026 09:00:00 +1100

In [Part I], I made the case that your legacy data model isn’t the disaster it looks like. That the strange WHERE clauses, the bridge tables nobody can explain, and the slowly-changing-dimension-within-a-slowly-changing-dimension aren’t bugs — they’re business rules earned through years of production reality. I argued that big-bang rebuilds fail at alarming rates, that the complexity you’re fighting is mostly essential rather than accidental, and that the impulse to “start from scratch” is driven more by cognitive bias than by engineering judgment.

A few people messaged me afterward and said, roughly: “Okay, I’m convinced. But what am I supposed to do about it?”

Fair question. Let’s talk about the playbook.

The vine that eats the tree

In 2004, Martin Fowler was visiting a rainforest in Queensland when he noticed something remarkable about the strangler fig trees. These vines start as seeds deposited in the upper branches of a host tree by birds. They grow downward, wrapping around the trunk, eventually establishing their own root system in the soil. Over years — sometimes decades — the fig gradually replaces the host tree. The original tree decays inside the fig’s lattice until there’s nothing left of it. But at no point during this process does the canopy collapse. The forest continues to function the entire time.

Fowler looked at those trees and thought: that’s how you should replace software systems.

The Strangler Fig pattern is, in my experience, the single most useful idea in all of software architecture — and it maps onto data warehouse migrations with almost uncomfortable precision. The concept is deceptively simple: instead of replacing the old system all at once, you build the new system around it. You intercept one piece of functionality at a time, route it through the new implementation, verify it works, and move on. The old system slowly shrinks. The new system slowly grows. And at no point does the business lose access to anything.

Fowler updated his essay on this pattern as recently as August 2024, and his framing is worth internalising. He described the alternative — the big-bang rewrite — and noted that he’d watched it fail most of the time. The strangler fig approach, by contrast, gives value steadily and allows you to monitor progress more carefully through frequent releases.

Now, here’s where it gets interesting for data teams specifically. Application systems have clean interfaces — APIs, endpoints, well-defined contracts. You can intercept a request, route it to the new system, and nobody downstream knows the difference. Data systems are messier. Elliott Cordo, who documented migrating three legacy data warehouses using this pattern, identified the core challenge: analytics systems have porous boundaries and poorly defined interfaces with consumers. You don’t always know who’s querying what, or how.

Cordo’s solution — and this is the variant I’ve seen work best — is what he calls the Legacy Façade. You build a new interface that mimics the legacy system’s contract. Same table names, same column names, same output format. Behind the façade, you’ve plumbed in the new logic. Consumers don’t change anything. They don’t even need to know a migration is happening. And this, critically, eliminates the organisational bottleneck that kills most migrations: getting twenty different teams to simultaneously update their queries, dashboards, and downstream pipelines to point at the new system.

Cordo’s experience is particularly telling because one of the three warehouses he migrated existed precisely because a previous attempt to replace one of the others had failed — mainly due to change management. The strangler fig approach bypasses that failure mode entirely by asking consumers to do little or nothing to adopt the new system.

I think this matters so much: the majority of data warehouse migrations don’t fail for technical reasons. They fail because you can’t convince the entire organisation to cut over on the same Tuesday. The Legacy Façade removes that dependency. You migrate at your pace, not at the pace of every team that consumes your data.

Stop publishing before you’ve checked

I’m continually surprised by how few people have heard of this pattern. When Robin Moffatt polled data engineers on Reddit about Write-Audit-Publish, most hadn’t even encountered the term — let alone used it. That’s a problem, because WAP solves the single scariest moment in any refactoring effort: the moment you push changed logic to production and hope you didn’t break anything.

Netflix introduced WAP at the DataWorks Summit in 2017, and the concept is elegant in its simplicity. Instead of the typical pipeline flow — transform data, write it to the production table, then maybe run some checks afterward — you split the process into three distinct stages:

Write to an isolated staging area. Not production. Not anywhere consumers can see it.

Audit the output against your quality checks. Row counts. Schema validation. Business rule assertions. Whatever gives you confidence.

Publish only if the audit passes. If it doesn’t, production never sees the bad data. Nobody’s dashboard breaks. Nobody’s quarterly report includes garbage. The corrupted data dies quietly in staging where it belongs.

The reason this matters for refactoring specifically is that refactoring requires you to change transformation logic while producing identical output. You’re restructuring the internals — breaking a 400-line SQL query into modular CTEs, moving logic from stored procedures into dbt models, replacing hardcoded values with reference table lookups. Each of these changes is supposed to be behaviour-preserving. WAP gives you the safety net to verify that it actually was.

Julien Hurault described the common anti-pattern — what he calls Write-(Publish)-Audit — where if an error is detected during testing, it’s already too late because the corrupted data has already been released to downstream systems. I’ve seen this play out more times than I want to admit. Someone refactors a model, the tests pass in the development environment, the change gets deployed, and then a stakeholder notices that revenue is off by 12% in a dashboard three hours later. By that point, the bad data has propagated through six downstream models and someone’s already screenshot the dashboard for a presentation.

WAP prevents this entirely. And with Apache Iceberg now supporting it natively through branch-based isolation, the tooling has finally caught up with the idea. You write to an audit branch, run your validations, and only fast-forward merge to main when everything checks out. It’s the data engineering equivalent of a pull request with automated tests — and honestly, it’s bizarre that we’ve accepted as normal the practice of deploying data transformations straight to production without a verification step in between.

If you’re on Snowflake, the implementation is straightforward using CLONE and SWAP operations. Clone the production table, run your refactored pipeline against the clone, validate the output, and swap the clone into the production role. If validation fails, the clone gets dropped and production is untouched. I’ve written about this pattern in more detail here — but the key point for this article is simpler: WAP turns refactoring from a high-wire act into a methodical, reversible process.

The archaeology of SQL

How does refactoring a data model actually works in practice, because it’s less dramatic than people expect. That’s sort of the point.

dbt Labs published a refactoring course that formalises the approach, and having used variants of this workflow on multiple teams, I think they’ve got it right. The core philosophy is one sentence: while refactoring, you’ll be moving around a lot of logic, but ideally you won’t be changing the logic.

Read that again. You are not improving business rules during a refactor. You are not fixing data quality issues during a refactor. You are not adding new features during a refactor. You are rearranging the existing logic into a better structure while proving, at every step, that the output hasn’t changed. The improvements come later, once the structure is clean enough to support them safely.

The workflow goes something like this. First, you take the legacy SQL — the 800-line stored procedure, the five nested subqueries, the transformation that nobody wants to touch because the last person who edited it left the company — and you port it into a dbt model unchanged. You don’t clean it up. You don’t refactor it. You bring it over exactly as it is, warts and all, and you verify that it produces identical output.

This is the step most people want to skip, and it’s the most important one. That ugly SQL is your baseline. It’s your source of truth. Until you’ve proved your refactored version matches it row-for-row, column-for-column, you don’t have a refactor — you have a rewrite wearing a refactor’s clothing.

Once you’ve got your baseline, you start decomposing. Extract the source references into dbt sources. Break the monolithic query into CTEs, each one doing a single logical operation. Move shared logic into staging models. Separate business rules from data cleaning from aggregation. At each step — and this is non-negotiable — you use something like dbt’s audit_helper package to compare the output of your refactored model against the original. If the outputs match, you move on. If they don’t, you figure out what you changed and fix it before going further.

Tristan Handy talks about his formative experience at Casper in 2016, where his team refactored all of their existing pipelines and brought them over to dbt in a single week, delivering work at least ten times faster than they would have been able to do otherwise. That speed came not from heroics but from methodology: migrate unchanged, then decompose with verification at every step.

I’ve watched a team at refactor over 200 dbt models with complex logic in three days using audit_helper. Three days. Not three months. Not the eighteen-month rebuild timeline that some stakeholder approved in a steering committee. Three days, because the tooling verified every change automatically and the team could move fast with confidence.

That’s the real payoff of refactoring over rebuilding. Not that it’s safer — though it is. Not that it preserves business logic — though it does. The payoff is that it’s faster. Faster to start delivering value. Faster to reach a clean architecture. Faster to build the confidence you need to actually make meaningful improvements.

Expand, then contract

There’s a pattern for handling breaking schema changes that I don’t see data teams use nearly enough, and it deserves more attention. Fowler calls it Parallel Change. Others call it Expand and Contract. The name doesn’t matter.

Imagine you need to rename a column in a fact table that thirty dashboards reference. In a rebuild mindset, you’d design the new schema, build it, and then coordinate a cutover where every consumer updates their queries simultaneously. Good luck with that. In my experience, “simultaneously” means “over the course of several painful weeks, with a growing list of things that are broken in the meantime.”

Expand and Contract takes a different approach, in three phases:

In the Expand phase, you add the new column alongside the old one. Both exist. Both contain the same data. Nothing breaks, because nothing has been removed.

In the Migrate phase, you update consumers one at a time. Dashboard A switches to the new column. Pipeline B switches. Report C switches. Each migration is independent, low-risk, and reversible. There’s no coordination required between teams.

In the Contract phase — and only after every consumer has migrated — you drop the old column. By this point, nobody is using it, so removing it is a non-event.

Does this take more deployments? Yes. it might require up to five deployments before the change is fully in effect. But each deployment is safe. Each deployment is reversible. And at no point does anyone lose access to their data because you renamed a column in a way they weren’t expecting.

For data models specifically, Expand and Contract shines when you’re restructuring dimensions, splitting fact tables, or changing grain. Add the new structure alongside the old one. Migrate consumers gradually. Remove the old structure once it’s no longer referenced. It’s boring. It’s methodical. It works.

The ship that replaced itself

If you want a single case study that demonstrates what successful incremental replacement looks like at scale, look at what Slack did with their desktop client in 2019.

Slack’s engineering team faced a classic dilemma. Their desktop application — built on jQuery and a custom framework — had accumulated years of complexity. It was slow. It used too much memory. The architecture made certain improvements difficult or impossible. Every instinct said: rewrite it in React.

Their engineering blog explicitly acknowledged the risk: running code knows things. Hard-won knowledge gained through billions of hours of cumulative usage and tens of thousands of bug fixes.

So instead of a big-bang rewrite, they did something much harder and much smarter. Over two years, they replaced every component of the desktop client while shipping continuously. They called it modernising “bit by bit” — enforcing strict interfaces between existing and modern code, shipping every change to production as it was completed, and never asking users to endure a disruptive cutover.

The results: 50% memory reduction, 33% load time improvement. Users never experienced a broken release. Had they waited until the entire application was rewritten before releasing it, their users would have had a worse experience for years while the team worked in a vacuum.

Now, Slack is an application, not a data warehouse. But the principles translate directly. Strict interfaces between old and new code? That’s the Legacy Façade. Shipping continuously? That’s WAP combined with Expand and Contract. Never asking users to endure a disruption? That’s the entire point.

The ancient Greeks had a thought experiment about this: if you replace every plank of a ship over time, is it still the same ship? Philosophers argue about it. Engineers don’t care. The ship still floats. The passengers still get where they’re going.

The numbers, for the sceptics

I held back on the quantitative evidence in Part I because I wanted to make the argument from experience and conviction. But some people want receipts, so here they are.

McKinsey and Oxford University studied 5,400 IT projects with initial budgets exceeding $15 million. On average, large projects ran 45% over budget and 7% over time, while delivering 56% less value than predicted. Seventeen percent were “black swans” — cost overruns exceeding 200% that threatened the company’s viability. One retailer abandoned a $1.4 billion IT modernisation, failed on a $600 million follow-up, and filed for bankruptcy.

The Standish Group has been tracking IT project outcomes since 1994. In their original CHAOS report, only 16.2% of projects succeeded on time, on budget, with specified features. Average cost overrun: 189%. By 2020, the overall success rate had improved to 31% — but here’s the statistic that matters most: small projects succeeded approximately 90% of the time. Large projects succeeded less than 10% of the time.

Read that again. Small projects: 90%. Large projects: under 10%.

That’s not a technology problem. That’s a scope problem. And the single most reliable way to reduce scope is to stop rebuilding entire systems and start refactoring them one piece at a time.

For data systems specifically, Gartner reports that more than 50% of data warehouse projects fail to reach user acceptance. Other analyses put the number closer to 80%. CDInsights reports that 70% of data warehouse modernisation projects exceed budget or fail. Integrate.io found that approximately 80% of data migration projects exceed timelines or budgets, with large-scale projects showing 50% higher failure rates than incremental approaches.

I don’t know how much clearer the data can be. Big-bang rewrites of data platforms fail most of the time. Incremental approaches succeed most of the time. The methodology I’ve described in this article — strangler fig, WAP, dbt refactoring, expand and contract — isn’t just safer or more philosophically sound. It’s what the evidence says actually works.

Refactoring as a practice, not a project

Refactoring should not be a project. It should not have a start date and an end date and a steering committee and a Gantt chart. Refactoring should be a practice — something your team does continuously, as a natural part of how you work with your data models.

Every time you touch a model to add a feature or fix a bug, you leave it a little cleaner than you found it. You extract a hardcoded value into a reference table. You add a test that didn’t exist before. You rename a column from col_7 to something a human can understand. You break a 200-line CTE into two smaller, focused ones.

None of these changes are dramatic. None of them require a business case or a migration plan. They’re small, safe, incremental improvements that compound over time. A year from now, the model is cleaner, better-tested, and easier to understand — not because you stopped everything to rebuild it, but because you improved it a little bit every time you were in the neighbourhood.

This is what Fowler meant when he defined refactoring as changing internal structure without modifying observable behaviour. It’s not a phase. It’s a discipline. And it’s the discipline that separates data teams that thrive from data teams that periodically torch everything and start over every three years, wondering why they never seem to make lasting progress.

The closing argument

In Part I, I said your data model isn’t broken — it’s battle-scarred, and those scars are knowledge. In Part II, I’ve tried to show you what the alternative to demolition looks like: patient, methodical, verified improvement. Strangler figs instead of bulldozers. Write-Audit-Publish instead of deploy-and-pray. Expand and contract instead of coordinate-and-hope.

It’s not glamorous work. Nobody’s going to invite you to give a conference talk about the time you renamed some columns and extracted a few staging models. You won’t get a promotion for migrating a legacy fact table so smoothly that nobody noticed it happened.

But that’s exactly the point. The best infrastructure work is invisible. The best migration is the one where the business never had to care. The best data model is the one that quietly absorbed twenty years of business complexity and still answers questions accurately at 7am on a Monday when a Department Head needs numbers for a board meeting.

I wrote Part I of this series last week. The technology has changed — Netscape to Snowflake, CGI scripts to dbt — but the lesson hasn’t. Old code that works is more valuable than new code that doesn’t exist yet. Embedded knowledge is harder to create than clean architecture. And the patient, unglamorous work of incremental improvement will always outperform the seductive fantasy of starting fresh.

Your data model isn’t broken. Stop trying to replace it. Start making it better.

You Don't Need Permission to Fix Your Data

Chris Hillman — Sat, 21 Mar 2026 09:00:00 +1100

Let me tell you about a junior engineer called Sam.

Sam had been on the team about four months when I noticed something in a pull request. Tucked between two routine model changes was a new schema.yml entry — five accepted_values tests on a column called customer_status that had been silently accumulating fourteen different spellings of “active” for the better part of a year.

Nobody asked Sam to do this. It wasn’t in a sprint. There was no Jira ticket. Sam had just been working in that part of the warehouse, noticed the mess, and decided to clean it up on the way through.

I left a comment on the PR: “Nice catch.” Sam’s reply was almost apologetic — “Hope it’s okay that I added these, I wasn’t sure if I was supposed to.”

That response stuck with me. I wasn’t sure if I was supposed to.

Here was someone who’d spotted a real problem, built a real solution, tested it, and shipped it — and their instinct was to wonder whether they had permission. Not whether the fix was correct. Not whether the tests were well-written. Whether they were allowed.

I’ve thought about that moment a lot since, because it captures something I see in almost every data team I work with. The people closest to the problems — the ones running the queries, staring at the nulls, fielding the Microsoft Teams messages when dashboards look wrong — are often the last to believe they can do anything about it. Not because they lack skill. Because somewhere along the way, they absorbed the idea that fixing things is someone else’s job.

That belief is expensive. And it’s wrong.

I’m telling you this because data quality — the real, unglamorous, column-by-column work of making data trustworthy — is one of those things that sounds like a senior engineering problem until you see a four-month-old team member fix something a dozen experienced engineers walked past every day. It doesn’t take authority. It takes the willingness to act before you’re told to.

This article is about how to be more like Sam. Five tactics you can start this week — no permission required — backed by real case studies from companies like Airbnb, Google, and Warner Bros. Discovery that figured this out the hard way.

The $12.9 million reason your quality problem isn’t “too small” to fix

Before we get into tactics, let’s talk about why your “small” fix matters more than you think.

Gartner pegs the average annual cost of poor data quality at $12.9 million per organisation. IBM and Harvard Business Review put the aggregate U.S. cost at $3.1 trillion annually. These numbers feel abstract until you realise what they actually represent: knowledge workers spending roughly half their time hunting for data, correcting errors, and searching for confirmatory sources instead of doing their actual jobs.

Monte Carlo’s 2024 State of Data Quality survey found bad data now impacts 31% of company revenue on average — up from 26% just two years earlier. Their 2022 survey found data engineers spend 40% of their time evaluating or checking data quality. Soda’s research put that number even higher: 61% of data engineers spend half or more of their time handling data issues.

And the human cost? A DataKitchen survey of 600 data engineers found 87% are frequently blamed when things go wrong with data. 78% wish their job came with a therapist.

The implication for you: the problem is so large and so costly that any measurable improvement you create has real dollar value attached to it. That null check you added last Thursday? That’s not a minor fix — it’s a tiny piece of a multi-million-dollar problem that nobody else bothered to address.

Tactic 1: write tests for your own models without asking anyone

The single best case study for this tactic comes from Airbnb, and it didn’t start with a VP mandate or an OKR. It started with one engineer.

Airbnb’s engineering blog documents how they went from “an engineering culture in which most new code shipped without a single test” to one where proposing a change without tests got called out immediately. The transformation wasn’t top-down. As the blog explains: “One approach is to rule by edict, but given our culture of engineer autonomy this would have been very poorly received. The other approach is to lead by example and build a movement.”

That single champion’s playbook was remarkably tactical. They spoke in engineering meetings. They held office hours showing teammates how to write tests. They published a weekly newsletter highlighting well-written test specs — giving public credit to engineers who included tests. They revamped the testing bootcamp for new hires, effectively making each new hire a champion. They improved tooling, reducing CI build times from over an hour to roughly six minutes.

The result? Airbnb later applied the same grassroots philosophy to data quality, building a DQ Score on a 0–100 scale across their entire warehouse. Their anomaly detection system now prevents quality issues in new pipelines before they reach production. The Minerva metrics platform manages 12,000+ metrics and 4,000+ dimensions with 200+ data producers.

One engineer. No mandate. A newsletter and some office hours.

What this looks like for you

If you’re working with dbt, you already have the tools. dbt ships with four built-in generic tests — not_null, unique, accepted_values, relationships — that you define in YAML alongside your model definitions. No framework to build. No proposal to write. Just add a few lines to your schema.yml:

models:
 - name: dim_customers
 columns:
 - name: customer_id
 tests:
 - not_null
 - unique
 - name: customer_status
 tests:
 - accepted_values:
 values: ["active", "inactive", "churned", "suspended"]
 - name: email
 tests:
 - not_null

That’s it. Next time the pipeline runs, those tests execute automatically. If someone pushes a change that breaks uniqueness on customer_id, the pipeline catches it before it reaches production.

The dbt-expectations package goes further, porting Great Expectations’ advanced tests into dbt. Unit tests — introduced in dbt v1.8 — let you validate SQL logic with static mock inputs before transformations run. Tests integrate directly into CI/CD, blocking PRs with failures from reaching production.

Not on dbt? You can still write assertion queries. Here’s a completeness check you can run against any SQL database — no framework required:

-- Quick completeness check: what percentage of each critical
-- field is actually populated?
SELECT
 COUNT(*) AS total_records,

 -- How many emails do we actually have?
 ROUND(100.0 * COUNT(email) / COUNT(*), 1) AS email_pct,

 -- How many phone numbers?
 ROUND(100.0 * COUNT(phone) / COUNT(*), 1) AS phone_pct,

 -- How many addresses?
 ROUND(100.0 * COUNT(address_line_1) / COUNT(*), 1) AS address_pct,

 -- Flag: are we below the 95% threshold on anything critical?
 CASE
 WHEN COUNT(email) < COUNT(*) * 0.95 THEN 'BELOW THRESHOLD'
 ELSE 'OK'
 END AS email_status
FROM
 main.customers;

Schedule that to run daily. Store the results in a logging table. You’ve just built a monitoring system — and you didn’t ask anyone’s permission.

The numbers back this up

Monte Carlo’s 2022 survey found that organisations conducting at least three types of data tests weekly experienced 46 incidents per month compared to 61 incidents per month for less rigorous testers — a 25% reduction. The Forrester Total Economic Impact study of data observability platforms (April 2025) found 358% ROI, 6,500+ data personnel hours reclaimed annually, and $1.5M+ in avoided lost revenue from reduced data downtime.

Daniel Beach of Data Engineering Central offers a pragmatic path for getting started: begin with end-to-end tests first when inheriting a pipeline with no tests, then add data quality checks, then unit tests. You don’t need to boil the ocean. You need to start.

Tactic 2: document one thing every time you touch it

The numbers on missing documentation are genuinely staggering. According to DX, developers spend 3–10 hours per week searching for information that should be documented. For a 100-person engineering team, that equals 300–1,000 hours weekly — the equivalent of 8–25 full-time engineers doing nothing but hunting for answers. Stack Overflow’s developer survey found 62% of developers spend over 30 minutes daily on poorly documented issues. And 38% listed poor documentation among their top reasons for leaving a company.

One practitioner tracked “Time to First Production Commit” at 28 days for new engineers — versus an industry benchmark of 3–5 days. Engineers on that team spent 30% of their time rediscovering what previous engineers had already learned. 40% of sprint tickets required inter-team clarification, averaging 12 clarifications per ticket.

This is what documentation debt looks like. Not a gap in a wiki — a tax on every single person who touches the codebase.

“Documentation Fridays” don’t work. This does.

Multiple sources confirm that big-bang documentation efforts fail. As one case study puts it: “‘Okay team, let’s spend Friday afternoon documenting our processes!’ Nobody wants to do that. And it doesn’t work.”

The successful pattern is document-as-you-go. Every time you encounter an issue, you write it down. Shopify Engineering treats documentation as a product, explicitly encouraging new team members to fill documentation gaps from day one. Their philosophy: “The most effective way to encourage others to write is to actually write.”

For data teams specifically, the approach is even simpler. If you’re using dbt, column descriptions live in schema.yml alongside your model definitions. dbt’s column-level lineage means passthrough and renamed columns automatically inherit descriptions from upstream models. Write a description once, and it flows downstream through every model that references that column.

Here’s what this looks like in practice:

models:
 - name: dim_customers
 description: >
 One row per customer. Grain: customer_id.
 Source: CRM extract via Fivetran, refreshed daily at 03:00 UTC.
 Owner: data-eng-team@company.com
 columns:
 - name: customer_id
 description: "Primary key. Sourced from CRM.contacts.id. Unique, never null."
 - name: customer_status
 description: >
 Current lifecycle status. Valid values: active, inactive, 
 churned, suspended. Updated by CRM webhook on status change.
 Known issue: legacy records from pre-2023 may contain 
 'Active' (capitalised) — normalised in this model.
 - name: lifetime_value
 description: >
 Cumulative revenue attributed to this customer, in AUD.
 Calculated as SUM(order_total) from fct_orders.
 Excludes refunded orders. Updated daily.

That “Known issue” note on customer_status? That’s worth its weight in gold to the next person who touches this model. It took thirty seconds to write and it’ll save hours of confusion.

Microsoft’s engineering blogs describe the alternative bluntly: “When tribal knowledge holders leave, reverse engineering is the only way to understand what was done.” Enboarder’s 2025 research found 47% of organisations cite institutional knowledge loss as their top offboarding challenge. Atlan’s Modern Data Survey found data professionals waste one full day per week simply trying to figure out what data to use.

You don’t need to document the entire warehouse. You need to document the thing you touched today. Tomorrow, document the next thing. In three months, you’ll have built something genuinely valuable — and you’ll have done it one commit at a time.

Tactic 3: build a quality dashboard nobody asked for

At Warner Bros. Discovery, the DICE team built data quality management frameworks and created a “Data Quality Forum” that became a bridge between teams. During the Olympics livestream, they used custom SQL checks to detect missing content metadata — fixing issues before they broke reporting. The forum evolved into what Pam Zirpoli called a “cultural flywheel.” Patterns surfaced in the forum were incorporated into onboarding, translated into new monitors, and used to refine their priority matrix. Teams shifted from reactive — reacting after dashboards broke — to proactive.

At Tempus, a data engineer documented how adopting Elementary (an open-source dbt package) transformed their monitoring. Before: “test alerting relied on a dashboard in Data Studio built on top of a log sink, and the ‘alerts’ were a daily email that you actually had to open.” After: Microsoft Teams alerts that “@-ed a user and continued to do so until they’re turned off.” The key outcome: tests no longer fail silently.

The six metrics that actually matter

Drawing from IBM, dbt Labs, Databricks, and Alation, the consensus metrics for data quality monitoring are:

Freshness: time since last data update. If your orders table hasn’t been updated since yesterday, something’s broken.
Volume: expected versus actual row counts. Did today’s load bring in 50,000 rows when you normally get 48,000–52,000? Fine. Did it bring 12? Problem.
Schema changes: added, deleted, or altered columns. These should never surprise you.
Completeness: null rates per column. Simple COUNT(*) vs COUNT(column) tells you a lot.
Distribution shifts: statistical changes in data values. If the average order value suddenly doubles, that’s worth investigating.
Test pass/fail rates: the percentage of your quality assertions that are passing. Track this over time.

Here’s a freshness check you can run right now — no tools, no frameworks, just SQL:

-- How stale is each of your critical tables?
-- Run this daily and store results in a monitoring table
SELECT
 'fct_orders' AS table_name,
 MAX(updated_at) AS last_record_timestamp,
 CURRENT_TIMESTAMP AS checked_at,
 ROUND(
 EXTRACT(EPOCH FROM (CURRENT_TIMESTAMP - MAX(updated_at))) / 3600.0,
 1
 ) AS hours_since_update,
 CASE
 WHEN MAX(updated_at) IS NULL THEN 'CRITICAL - No data'
 WHEN CURRENT_TIMESTAMP - MAX(updated_at) > INTERVAL '24 hours'
 THEN 'STALE - Exceeds 24hr SLA'
 WHEN CURRENT_TIMESTAMP - MAX(updated_at) > INTERVAL '12 hours'
 THEN 'WARNING - Approaching SLA'
 ELSE 'FRESH'
 END AS freshness_status
FROM analytics.fct_orders;

Store that in a data_quality_log table, connect it to Looker Studio or Metabase (both free), and you’ve got a monitoring dashboard. That’s it. No Kubernetes cluster. No proposal document.

Visibility changes behaviour

Anna Geller documented how sending data quality alerts to a shared Microsoft Teams channel transforms team behaviour. Her core insight: “As social creatures, we are more motivated to fix issues if other people can see the effort we put into it.” Data owners reply with root cause analysis, add checkmarks, or link tickets — creating a social proof loop that issues are no longer ignored.

This isn’t just anecdotal. Research on the Hawthorne effect shows healthcare hand hygiene compliance increases by up to 70% when workers know they’re observed, and employees increase productivity by an average of 13% when activity is monitored. Making data quality visible doesn’t just inform people — it changes their behaviour.

Tactic 4: create a #data-bugs channel

MIT Sloan Management Review’s HelloFresh case study documents one of the best examples of how visibility catalyses organisational change. HelloFresh transitioned through three modes of data quality management. In Mode 1, issues were dealt with reactively by whoever happened to be affected. In Mode 2, a central engineering team handled cleanup but lacked understanding of how data was actually used — “data customers grew increasingly frustrated as they continued to spend large fractions of their workdays dealing with data that did not meet their needs.”

The transition to Mode 3 — proactive, collaborative quality management — happened precisely because “quality issues became increasingly visible.” Not because someone wrote a strategy doc. Not because a VP mandated it. Because the problems became impossible to ignore.

You can accelerate that transition. Create a Microsoft Teams channel. Call it #data-bugs or #data-quality or whatever feels natural. Start posting issues you find — with specifics. Not “the data looks wrong” but “fct_orders has 3,247 rows with null customer_id from the 2026-02-15 load, affecting the weekly revenue dashboard.”

That’s it. You’ve just created the visibility mechanism that HelloFresh spent years arriving at.

The broken windows effect is real — and it’s been measured

The Pragmatic Programmer introduced the “broken windows” software analogy decades ago: “Don’t leave ‘broken windows’ (bad designs, wrong decisions, or poor code) unrepaired. Fix each one as soon as it is discovered.”

This isn’t just metaphor anymore. A 2024 arXiv paper by Diomidis Spinellis et al. analysed 2 million code commits across 122 projects comprising 5.5 million lines of code. The key finding: “History matters — developers behave differently depending on some aspects of the code quality they encounter.” Developers tailor the quality of their commits based on the quality of the file they’re committing to. Low quality begets lower quality. High quality attracts higher quality.

Mat Ryer extends this to everyday engineering: “If you work on a project that has flaky tests, then you’re more likely to add more flaky tests. If there is a hacky design, you’re more likely to hack more in.”

Fixing one visible data quality issue — or even just making it visible — can interrupt the decay spiral.

Why juniors don’t speak up (and why that’s exactly the problem)

Here’s the uncomfortable part. A Culture Shift UK survey found junior colleagues were twice as likely (54%) as senior leaders (27%) to say speaking up about issues is “pointless.” 37% said speaking up “isn’t worth the personal risk.” A Blind survey found 58% of tech workers experience imposter syndrome.

Google’s Project Aristotle studied 180+ teams over two years and found psychological safety was “by far the most important” of five dynamics for team effectiveness. Amy Edmondson’s foundational research contains a striking finding: while studying hospital teams, she expected high-performing teams to report fewer errors. Instead, better teams reported MORE errors. Her insight: “Maybe the good teams don’t make more mistakes, maybe they report more.”

This is the paradox. The teams that look like they have the most data quality issues are often the healthiest — because they’ve created an environment where problems surface instead of festering.

Etsy pioneered blameless postmortems under CTO John Allspaw. Engineers send company-wide emails confessing mistakes. Etsy even gives out an annual “three-armed sweater” award to the employee who made the most surprising error. As a Hadley Wickham–curated Stanford reading list notes, while blameless postmortems are standard in DevOps, they “have not yet infiltrated standard data science practices.” That’s a gap you can help fill.

Creating a #data-bugs channel isn’t just about tracking issues. It’s about establishing the norm that finding problems is good work, not a sign that something’s wrong with you.

Tactic 5: ship an example, don’t write a proposal

Marcus Blankenship’s article on learned helplessness in software engineering — 96,000+ reads, 900+ Reddit comments — contains a story about a coworker called Milind. Milind wasn’t a manager. He didn’t have a fancy title. But he was an informal leader through pure action. He asked questions about why particular approaches were taken. He admitted ignorance in group settings. He insisted the team understand root causes before moving on.

Blankenship’s observation: “Milind changed our environment by his actions… He showed me that I had much more power than I thought, if I would only stop expecting to be spoon-fed everything. His actions showed me that true leadership isn’t granted, it’s grasped.”

A junior engineer on Irina Stanescu’s newsletter shared something similar: “Even as a junior engineer, I remember influencing my chief architect and brought Terraform into my organisation 7 years ago by simply pointing out a common pain point and how much longer things took without managing our infrastructure in code.” They added: “Being able to influence without authority is a very underrated skill, and I believe this is what helped me get promoted to senior engineer even more than my technical skills.”

The pattern is consistent. Don’t propose a data quality framework. Don’t write a 15-page RFC. Build the thing. Make it work. Show people.

Why working examples spread faster than proposals

Everett Rogers’ Diffusion of Innovations theory identifies five characteristics that determine how fast an idea spreads: relative advantage, compatibility, complexity (lower is better), trialability, and observability. A working example maximises both trialability and observability — the two factors most under your control as a junior engineer.

A proposal says “we should do this.” A working example says “look, I already did this, and here’s what happened.” One requires people to imagine the benefit. The other puts the benefit directly in front of them.

Addy Osmani, after 14 years at Google, captured the complementary lesson: “Early in my career, I believed great work would speak for itself. I was wrong. Code sits silently in a repository.” You need to build the example and make its impact visible. Post in Microsoft Teams. Show the before and after. Quantify what changed.

The junior advantage nobody talks about

Ant Weiss’s widely-read article on organisational learned helplessness identifies common symptoms: “This is how we were told to work,” “These are the limitations of the system,” “We don’t have the resources.” His core observation: “No engineer starts their career willing to be unproductive, pessimistic and stressed out. But the companies they work at teach them to.”

The cure, drawn from Seligman’s original research, is critical: “Threats, rewards, and observed demonstrations had no effect on the ‘helpless’ dogs. The dogs had to be physically moved through the escape action at least twice before they would do it themselves.” You can’t just tell engineers things can be better. You have to walk them through the experience of making something better.

Here’s the part that should give you confidence. JazzTeam reports a remarkable finding: “To prove that any problem can be solved, our team many times gave a task to Junior and even Intern Engineers who did not have psychological fears and tunnel thinking.” Juniors, unencumbered by years of “that’s impossible” conditioning, were able to tackle problems seniors had declared unsolvable.

Your fresh eyes aren’t a weakness. They’re an asset. You haven’t yet learned that “it can’t be done.” So go do it.

Putting it all together

None of these tactics require a committee meeting. None require architectural approval. None require you to be senior.

Here’s what they do require: the willingness to act before you’re told to.

Write a test this week. Add a column description tomorrow. Build a freshness check and schedule it. Create a Slack channel and post the first issue. Turn one of your quality checks into a reusable example your team can copy.

DORA’s research — surveying 39,000+ professionals — finds that with focused effort and the right tooling, teams typically see measurable improvements within 1–3 months. A GitHub engineer who went from junior to mid-level in 2.5 years described the compound effect: “If I got stuck on some undocumented functionality, I made sure to update the docs and let the team know. Tackling a tricky bug requiring cross-team collaboration? I’d summarise everything we discovered so that it would be easier for others later. Post about it in Microsoft Teams and highlight its impact.”

Kevin Workman, a former Google engineer, captured the mindset that makes all of this work: “Realising that I had agency and even a responsibility to advocate for my ideas is one of the most important lessons I learned on my journey to becoming a ‘senior’ software engineer… Some of the most interesting and most successful moments of my career have come from advocating for changes I thought nobody would agree to.”

I think about Sam sometimes — that four-month-old team member whose instinct was to apologise for fixing something. The tests Sam added that day caught three data quality issues in the following month. Other engineers started adding their own tests alongside model changes. Six months later, the team had 200+ automated quality checks running on every pipeline, and the weekly “data looks wrong” Microsoft Teams messages had dropped to nearly zero.

Sam never got a mandate. Never wrote a proposal. Never waited for Q3.

The data quality problem in your organisation is a $12.9 million issue wearing a “we’ll get to it in Q3” disguise. And you — yes, you, the person who’s been here five months and noticed fourteen spellings of “active” — are exactly the right person to start fixing it.

Not because someone gave you permission. Because the data doesn’t care about your title.

Your Friends Will Be There for You. Your Work Won't.

Chris Hillman — Wed, 18 Mar 2026 09:00:00 +1100

There’s a table in a rented house in Queenscliff — nothing fancy, just whatever furniture came with the place — with a hand-drawn map spread across it, dice scattered at the edges, and snacks slowly migrating toward the centre of the board.

Once a year, Steve, one of our friends, organises for us to make a trip down to Queenscliff to play tabletop RPGs for a weekend. More recently, we’ve started playing monthly or fortnightly at home too. Nothing glamorous about any of it. We’re not Glass Cannon Network or Critical Role — no cameras, no production values, no audience watching us roleplay with solemn gravity.

We’re just a bunch of friends. Laughing at the missed sword swing. Losing our minds when someone makes a completely unreasonable jump and somehow lands the killing blow on a copper golem with a natural twenty. Rolling dice and arguing about the rules and eating too much while the world outside keeps spinning without us.

I’ve been thinking about why it matters so much to me.

It’s not the game, really — or at least, the game isn’t the point. It’s what the game creates. A ritual. An excuse to sit in the same room for a weekend and actually be present with people I care about. A context where you ask “how are you going?” and actually mean it, and stay long enough to hear the real answer. The dragon we’re fighting is almost beside the point. What matters is being there for each other.

When you’re up at 2am resolving a critical data pipeline failure, your employer will not remember that next year. They might not remember it next month. The organisation absorbs it, thanks you if you’re lucky, and moves on to the next incident.

But fighting dragons with your friends — showing up for a weekend in Queenscliff year after year — those become stories. Stories that become memories. Memories that become the fabric of who you are to each other, and who they are to you.

Your friends will be there for you. Your work won’t.

And yet.

When was the last time someone you loved lost someone, and you climbed into bed next to them — not to fix it, not to say the right thing, just to be there in the dark with them? When something hard was happening in your own life, did you pick up the phone and call a friend? Or did the thought cross your mind — “they’re busy, they don’t have time to listen to me” — and you put the phone down and carried it alone?

Do you have a friend you’d call when you got the promotion — someone who’d hear the news and mean it when they say “that’s great”? Not quietly threatened by it. Not performing enthusiasm. Just genuinely proud of you, because your win is their win. That kind of friend is rarer than it should be.

Those moments. That’s what friendship actually is. Not the catch-up drinks you schedule six weeks out and cancel twice. The showing up in the hard moments. The willingness to burden someone with your struggle, and the willingness to carry theirs.

We don’t build trust by offering help. We build trust by asking for it. The act of calling someone when things are falling apart — not suffering quietly, not performing fine — is the thing that makes the friendship real on both sides. It’s not a burden. For the right person, it’s an honour.

This is something I’m still learning. I want to talk about what it actually means to be a good friend — because I think most of us, if we’re honest, have let that muscle go slack.

Are You Actually a Good Friend?

Most people, if you ask them directly, will say yes. Of course they’re a good friend. They’re there for the people they care about.

But ask yourself: do you call your friends on their birthday — actually call, sing happy birthday — or do you post on their Facebook because you saw the notification and everyone else was doing it? When a friend is going through something hard, do you show up? Not message. Show up. Do you say “I love you” to the people in your life that you love?

Most of us, if we’re being honest, have outsourced friendship to the low-effort channel. The emoji response. The “we should catch up soon!” comment thread that leads nowhere. We maintain the social graph without maintaining the relationships.

But when you’re in a dark place, who do you call? And more confronting — who would call you?

The answer to that second question is the real measure. Friendship isn’t what you feel for someone. It’s what you demonstrate over time. It’s the accumulated weight of showing up — for the birthday dinner that falls on a work night, for the hospital waiting room at 7am, for the phone call that starts “I just need to talk to someone” at an inconvenient hour.

Those moments don’t happen automatically. They have to be built. And they have to be built before you need them, not when you’re reaching for them in a crisis.

When our daughter Natasha was born, she wouldn’t latch on, and breastfeeding wasn’t working. Lena and I tried everything, exhausted and running on no sleep, seriously weighing up whether to just switch to formula. At some point we gave up trying to solve it ourselves and called our friends Tom and Rachel. We ended up talking for somewhere between four and six hours straight. We didn’t need advice, really — we needed to not be alone with it. And something about that phone call, about calming down, settled Natasha too. By the end of it she was feeding. I still think about that night. We’d been treating a human problem like a technical problem, trying to fix it in isolation, when all we’d actually needed was to pick up the phone.

What Women Already Know

Women are better at friendship than men. Not universally, not as a stereotype — but as a pattern that shows up consistently enough to be worth talking about honestly.

The research on women’s friendship patterns tells part of the story. Women’s friendships tend to orient toward direct emotional disclosure — what psychologists call face-to-face friendship. Personal sharing, vulnerability, actual conversation about what’s happening in your life. Men’s friendships tend to be shoulder-to-shoulder — bonding through shared activity, talking while doing something else together. Neither mode is better. But when the activity disappears — when the sport stops, when the shared project ends, when the workplace changes — shoulder-to-shoulder friendship has nothing to stand on. Face-to-face friendship survives the context change because the context was never the point.

What women understand, and what most men have to learn later and harder, is that the friendship is the thing. Not the activity. The person.

The good news is that the shoulder-to-shoulder model doesn’t have to disappear. The Queenscliff table is shoulder-to-shoulder. The hack is building ritual around it — making the activity a recurring container for the friendship, rather than the friendship being a side effect of the activity. When you have the ritual, the friendship survives when everything else changes.

I saw this firsthand when I was working in India. Some colleagues (Sarasa, Nidhi, Abhilash, Deepak) took me out to play badminton — I’d never played before, and I was genuinely terrible. But what struck me wasn’t the sport. It was that this group did it every week, without fail. They weren’t there to compete or keep score. They were there to be together, to unwind, to laugh at each other and at themselves. What I remember is leaving that court feeling like I’d glimpsed something. These people had built a ritual, and the ritual had built them.

What the Harvard Longevity Study Actually Found

Harvard physician Arlie Bock began following the lives of a few hundred Harvard sophomores. The study is now 85 years old, has followed over 1,300 people including the children of original participants, and is the longest-running study of adult development in history.

The fourth director of the study, psychiatrist Robert Waldinger, gave what became one of the ten most-watched TED talks ever. And the headline finding, distilled from nearly nine decades of data, is deceptively simple:

Good relationships lead to health and happiness. Not wealth. Not professional achievement. Not optimised nutrition or fitness metrics. Relationships.

Waldinger and his co-author Marc Schulz published the full findings in The Good Life in 2023. Two things from that research hit particularly hard.

First: relationship satisfaction at age 50 was a better predictor of physical health at 80 than cholesterol levels. That’s the kind of finding that should make every analytically-minded person stop and recalibrate. We spend enormous energy optimising for things we can measure while ignoring the metric that turns out to be most predictive of how we age and how long we live.

Second: when participants reached their 80s, their biggest regret was almost universally the same. Too much time at work. Not enough time with the people they loved. And their proudest achievements were almost entirely relational — being a good parent, a good friend, a good partner, a good mentor.

George Vaillant, who directed the study for over three decades, summarised it: “When the study began, nobody cared about empathy or attachment. But the key to healthy aging is relationships, relationships, relationships.”

What We Actually Traded

Reflecting on how high achievers use the word “sacrifice”. When we say we’ve sacrificed something for our career, we shouldn’t be afraid to put a name to who that sacrifice was. Because often it was the people in our lives that we call friends.

Put a name to it.

We’re not sacrificing some abstract concept. We’re sacrificing specific people. Former colleagues whose numbers we keep but never dial. School friends who live in the same city and see us once a year if they’re lucky. The people who would drop everything if we called — but who we don’t call, because we’re heads-down on the next deliverable.

So many of us have cancelled on friends because a meeting came up, telling ourselves they’ll understand. Yet the reverse almost never happens — we wouldn’t reschedule a meeting for a friend. We’ve quietly trained ourselves to treat friendship as the flexible commitment, the one that can move.

And the friendship debt is unlike technical debt — you can’t see it slowing you down, it doesn’t show up in velocity metrics or retrospectives. It just quietly accumulates until something breaks. The incident you pushed through alone. The difficult conversation at home that had nowhere to go because there were no friends to process it with. The creeping sense that despite being deeply capable and professionally respected, you’re not quite sure who you’d call.

Ubuntu: A Different Operating System

There’s a Zulu/Nguni philosophy called Ubuntu. The phrase “Umuntu ngumuntu ngabantu” translates roughly as “a person is a person through other people.” You’ll also hear it rendered as “I am because we are.”

Desmond Tutu described Ubuntu this way: “In our African worldview, we need other human beings for us to learn how to be human. For none of us comes fully formed into the world. The solitary human being is a contradiction in terms.”

He contrasted it explicitly with Descartes: “It is not ‘I think therefore I am.’ It says rather: ‘I am human because I belong. I participate. I share.’”

Most of the professional identity we build in data engineering runs on Descartes. I architect, therefore I am. I optimise the pipeline, therefore I am. I resolved the incident, therefore I am. We define ourselves by what we produce, alone, with headphones on, in a flow state that the rest of the world is just interrupting.

Ubuntu is a different operating system. One where your identity is constituted by your relationships — where the question “who are you?” is answered not by your job title or your GitHub commit history, but by who you show up for and who shows up for you.

The longest-running study of adult development in history arrived at the same conclusion Tutu was describing. Every major piece of research on social connection and mortality found it too. The architecture most of us are running on isn’t wrong in a subtle way. It’s wrong in the foundational way.

Mark Shuttleworth — South African entrepreneur — named the Ubuntu Linux operating system after this philosophy in 2004, explicitly to emphasise community, open-source contribution, and the idea that the work is better when people build it together. Engineers understood that intuitively, because it maps to how the best software actually gets built. It also maps, as it turns out, to how the best lives get built.

The Dragon Will Be There Next Year

There’s a table in a rented house in Queenscliff, with a hand-drawn map on it and dice scattered at the edges.

Every year, we make the trip. We gather around a table and roll dice and laugh at the missed sword swing and the impossible jump. The dragons change. The characters level up. The rules get argued over and amended and house-ruled into something nobody outside the group would recognise.

But the people are constant. And the ritual is the point.

It is more amazing to have an extraordinary experience with someone than by yourself. You can go somewhere alone and say “look what I did” — versus “do you remember that time we did that?”

The 2am pipeline failure you resolved will be forgotten. By you. By your employer. By the Jira ticket that gets closed and archived.

The dragon you fought with your friends — the copper golem that went down to a natural twenty on an impossible jump — that becomes a story. The story becomes a memory. The memory becomes part of who you are to each other and who they are to you.

That’s not sentimentality. That’s the mechanism by which good lives actually get built. Eighty-five years of longitudinal data says so.

Researcher Robin Dunbar found that building a close friendship takes roughly 200 hours of time together. Those hours don’t come from nowhere. They come from showing up, again and again, at the table. From protecting that time the same way you’d protect a critical production window. From understanding that the maintenance cost of friendship is the highest-ROI investment available to you as a human being.

Your work will absorb your best years and move on when it’s convenient. Your friends will still be at the table.

Put a name to what you’re protecting.

Then protect it.

Your Data Model Isn't Broken, Part I: Why Refactoring Beats Rebuilding

Chris Hillman — Sat, 14 Mar 2026 09:00:00 +1100

In the early 2000’s - Netscape’s decision to rewrite their browser from scratch was the single worst strategic mistake a software company could make.

At the time, Netscape was winning. They had the dominant browser. They had market share. They had momentum. And then they decided the codebase was too messy, too tangled, too hard to work with — so they threw it all away and started over. Navigator 4.0 became the foundation for a rewrite that would eventually ship as version 6.0. There was no 5.0. Three years of development. No shipping product. And while Netscape’s engineers were busy building their beautiful new browser in a vacuum, Internet Explorer ate their lunch, their dinner, and most of their market share.

It’s haunted me ever since: old code isn’t ugly because it’s bad. Old code is ugly because it works. Every strange condition, every seemingly redundant check, every patch that makes a new developer wince — those are battle scars. Each one represents a bug that took weeks to find in production, a customer workflow nobody anticipated, or an edge case that only surfaces on the third Tuesday of months ending in “R.”

I think about that every time I hear a data team say “let’s just rebuild it.”

The sentence that starts every failed data project

I’ve been on data teams long enough to recognise the pattern. It always starts the same way. Someone — usually someone new, often someone senior — opens a dimensional model they didn’t build, scrolls through a few hundred lines of transformation logic, and says the words that should make every data leader’s blood run cold:

“This is a mess. We need to start from scratch.”

And look, I get it. I’ve felt that impulse myself, and likely have said the words myself. You open a fact table with 200 columns. You find bridge tables that reference other bridge tables. You discover a slowly-changing dimension nested inside another slowly-changing dimension, and you think: who built this? What were they thinking?

But here’s what I’ve learned the hard way, across multiple teams and more warehouse migrations than I care to count: they were thinking about the business. That bizarre WHERE clause filtering out both “Unknown” and “unknown”? That’s not sloppy code. That’s a case-sensitivity bug someone found in production data from a source system that nobody controlled. The seemingly redundant join that adds three seconds to your query? It handles a quarterly reconciliation edge case that cost the finance team two days of manual work before someone encoded the fix.

That fact table with 200 columns isn’t a design failure. It’s an accurate representation of a business that has 200 things it needs to measure. The “clean” replacement model will eventually have 200 columns too — they’ll just have different names and it’ll take you eighteen months to figure out why you need them all.

What data teams keep forgetting

This isn’t about browsers or even about code quality. It was about knowledge. When you throw away a codebase and start fresh, you’re not just discarding syntax. You’re discarding years of accumulated understanding about how the real world actually behaves — understanding that was earned through production incidents, user complaints, and painful debugging sessions.

Every collected bug fix in that old code represents something learned. Each fix might be just one line, a couple of characters even, but a lot of work and time went into figuring out those two characters were needed. And that knowledge — the why behind the fix — almost never makes it into documentation. It lives in the code itself, or it lives nowhere.

This is doubly true for data systems. A software application has unit tests, integration tests, user acceptance testing. A data model has… what exactly? Row counts? Spot checks? A business user who eyeballs the dashboard and says “yeah, that looks about right”? The knowledge embedded in a mature data warehouse is far more fragile than application code, because the testing infrastructure around it is almost always weaker. Throw it away and you’re not just rebuilding a codebase — you’re rebuilding an institutional memory that was never written down in the first place.

Fred Brooks saw this coming fifty years ago. His “Second System Effect” from The Mythical Man-Month describes what happens when an engineer builds their second version of something: they over-design it. All the features they wisely deferred from the first version, all the architectural improvements they dreamed about, all the “if only we’d done it this way” ideas — they dump everything into the replacement. Brooks noted that a designer’s first system tends to be spare and clean because they know their limitations. The second system becomes a dumping ground for ambition.

Sound familiar? Every data warehouse rebuild I’ve witnessed follows this arc. The team doesn’t just want to replicate what exists — they want to add a semantic layer, implement a medallion architecture, introduce data contracts, add real-time streaming, switch to Data Vault, build a self-serve analytics platform, and migrate to a new cloud provider. All at once. In a single initiative.

And then they wonder why, two years later, they’ve delivered nothing and the business is still running reports off the “legacy” warehouse that was supposed to be decommissioned eighteen months ago.

The Teradata-to-Snowflake reality check

If you want to see the rebuild-vs-refactor debate play out in real time, watch a Teradata-to-Snowflake migration. The pattern is remarkably consistent.

The pitch is compelling: Snowflake is cheaper, scales elastically, separates compute from storage, and runs standard SQL. Moving from Teradata should be straightforward — convert the SQL, move the data, validate the results. Easy, right?

Roland Wenzlofsky, a Snowflake Solutions Architect, wrote about this in early 2026 and his observations line up perfectly with what I’ve seen. Most organisations approach these migrations as a translation exercise. The technical migration succeeds. Then the first quarterly cloud bill arrives at double the projected budget. Dashboards that loaded in seconds now queue for twenty minutes during batch windows. Pipelines that ran in 45 minutes on Teradata take five hours on Snowflake.

The problem isn’t Snowflake. The problem is that Teradata professionals carry assumptions into Snowflake that aren’t just incomplete — they’re actively counterproductive. Distribution keys, join strategies, indexing patterns — all the hard-won optimisation knowledge from Teradata becomes a liability in a platform built on fundamentally different architecture.

One of the largest Teradata-to-Snowflake migrations in North America involved 1.5 petabytes across 600 databases and roughly 45,000 objects. Before they could even begin, the team had to inventory every orphan object and document every behavioural difference between platforms. That’s not a “lift and shift.” That’s an archaeological dig.

But here’s what I find most telling: even when the migration succeeds technically, the data quality problems survive the move perfectly intact. William Flaiz documented a healthcare organisation that spent $1.8 million migrating from Siebel to Salesforce. Not a single record lost. Zero downtime. Perfect data mapping. Post-migration, the sales team still couldn’t run accurate pipeline reports. The “opportunity stage” field contained 89 different values for what should have been six standard stages — including 1,247 records with the typo “Closeing Soon.” That typo migrated perfectly to the new $2.3 million platform. As Flaiz put it: when performance is bad, we assume the technology is the limiting factor. Data quality problems are messier. They implicate people, processes, training gaps, and years of accumulated shortcuts.

New platform, same chaos. Because the platform was never the problem.

Chesterton’s Fence, or: why you shouldn’t delete that WHERE clause

There’s a principle in philosophy that every data engineer should have tattooed somewhere visible. G.K. Chesterton wrote it in 1929, and it goes roughly like this: if you come across a fence in the middle of a field and can’t see what purpose it serves, don’t tear it down. Go away and figure out why someone built it. Once you understand the reason, then you can decide whether it still needs to be there.

In data engineering, the fences are everywhere. That WHERE status NOT IN ('Unknown', 'unknown', 'UNKNOWN') clause. That filter excluding records from a specific date range in 2019. The join to a reference table that only has 12 rows and hasn’t been updated in three years. They all look pointless until you remove one and discover that the finance reconciliation breaks, or that a regulatory report starts including test transactions that were supposed to be filtered out, or that a dashboard starts showing a revenue spike from a data migration artifact that happened four years ago.

Hyrum Wright — formerly at Google, now at Adobe — formalised a related idea that’s become known as Hyrum’s Law: with enough users of an API, every observable behaviour will be depended on by somebody, regardless of what the documentation promises. For data systems, this is an absolute nightmare during rebuilds. Your downstream consumers don’t just depend on documented schemas. They depend on output ordering, null handling patterns, timestamp precision, and format quirks that were never specified anywhere. You rebuild the pipeline, and suddenly a report that’s worked for three years breaks — not because the data is wrong, but because the columns come back in a different order and someone’s Excel macro was hardcoded to column positions.

I’ve seen teams spend more time debugging these “invisible contracts” after a rebuild than they would have spent refactoring the original system over two years.

The psychology that makes us do it anyway

So if the evidence against big-bang rewrites is this overwhelming — and it is, from Spolsky to Brooks to McKinsey studies showing that large IT projects run 45% over budget and deliver 56% less value than predicted — why do smart, experienced data teams keep proposing them?

Because the impulse to rebuild isn’t rational. It’s psychological. And once you understand the cognitive biases at play, you start seeing them everywhere.

The planning fallacy is the first culprit. Kahneman and Tversky showed that humans systematically underestimate how long tasks will take, even when they have direct experience with similar tasks that took longer than expected. In controlled studies, only 13% of subjects finished their project by the time they’d assigned a 50% probability of completion. The Sydney Opera House was estimated at AU$7 million and four years; it was delivered at AU$102 million and fourteen years. Every data warehouse rebuild proposal I’ve seen exhibits this same delusional optimism. “Six months, maybe nine.” It’s never six months.

And it gets worse, because of what one writer calls the Achilles Paradox of rewrites: while you’re building the new version, features keep getting added to the old one. The business doesn’t stop generating requirements just because you’ve decided to rebuild. So the target keeps moving. The new system is perpetually six months from matching the functionality of the old one.

Not Invented Here syndrome is the second bias, and it’s the one that nobody wants to admit. Katz and Allen’s research on R&D project groups found that teams with stable composition develop a decreasing ability to absorb ideas from outside the group over time. They start believing they possess a monopoly on knowledge in their domain. The diagnostic sign? What researchers call “thought-terminating clichés” — phrases like “we already tried that” or “our situation is different.”

In data teams, NIH manifests as building custom orchestration frameworks instead of using Airflow. Writing bespoke quality checks instead of adopting Great Expectations or dbt tests. Designing proprietary transformation layers instead of using tools that thousands of other teams have battle-tested. And most relevantly: insisting that the existing data model is fundamentally broken and needs to be replaced with something designed in-house from the ground up.

The new leader rewrite is the third pattern, and it might be the most destructive. A new head of data arrives. They look at the legacy warehouse they’ve inherited. They don’t understand the history behind any of the design decisions. They don’t know about the regulatory edge case that explains the weird date filter, or the source system quirk that necessitates the redundant join. All they see is complexity that they didn’t create — and complexity you didn’t create always looks worse than complexity you did.

So they propose a rebuild. It’s partly strategic — they want to put their stamp on the architecture. And it’s partly genuine — they really do think they can do better. But they’re falling prey to the same illusion Spolsky identified: reading code is harder than writing it, and unfamiliar code always looks worse than it is.

There’s a Stack Overflow survey finding that sits underneath all of this: “feeling unproductive” was the number one cause of developer unhappiness at 45%. Working with legacy systems is slow. It’s frustrating. Progress is incremental and often invisible. A greenfield rebuild, by contrast, feels amazing — for the first few months. You’re making decisions, building things, moving fast. The architecture is clean and the tests pass and the world makes sense. And then reality seeps in, the edge cases accumulate, and eighteen months later you’ve got a codebase that looks suspiciously like the one you replaced.

The complexity was always essential

Fred Brooks drew a distinction that I think about constantly. He separated essential complexity — complexity that’s inherent to the problem domain and can’t be removed by better engineering — from accidental complexity, which comes from poor implementation choices and can theoretically be eliminated.

The truth about mature data warehouses: most of the complexity is essential. It’s not there because the original engineers were incompetent. It’s there because the business is genuinely that complicated.

The transformation handling fifteen date formats? That’s because you have fifteen source systems and none of them agree on how to represent a date. The slowly-changing dimension with Type 2 tracking on attributes that seem trivial? Someone in compliance needed audit trails on those fields after a regulatory inquiry. The bridge table that connects customers to accounts through an intermediate entity that nobody can explain? It models a many-to-many relationship that emerged when the company acquired a subsidiary with a different customer hierarchy.

A rebuild won’t make this complexity disappear. It’ll just redistribute it. Instead of one tangled fact table, you’ll have twelve microservices each handling a piece of the logic. Instead of one confusing WHERE clause, you’ll have business rules scattered across a semantic layer, a transformation layer, and a data quality framework. The total complexity will be identical — or worse, because now it’s distributed across more systems with more failure modes.

Ward Cunningham — the person who coined the term “technical debt” — has said he wishes he’d used the word “opportunity” instead. His original metaphor wasn’t about sloppy code at all. It was about the gap between your code’s current model of the problem domain and your team’s evolved understanding of that domain. Technical debt, properly understood, is learning that hasn’t been applied yet. The legacy data model doesn’t represent failure. It represents the team’s best understanding at the time it was built, plus every correction that production reality demanded afterward.

Refactoring respects this. It asks: which parts of this complexity are essential (keep them, clarify them, test them) and which parts are accidental (remove them incrementally, one safe step at a time)? It preserves institutional knowledge while improving structure. It delivers value continuously instead of asking the business to wait years for a payoff that — statistically, based on everything we know about large-scale IT projects — probably won’t arrive.

So what do you do instead?

I’m going to save the detailed playbook for Part II of this series, but the short version is this: you refactor. Methodically. Incrementally. With tests.

Martin Fowler and Pramod Sadalage proved that database refactoring is possible back in 2006, when conventional wisdom said it couldn’t be done. Their principle was simple: make each change as small as possible, because the pain of integration increases exponentially with the size of the integration.

dbt has turned this into a practical workflow for analytics engineering teams. Bring your legacy SQL in unchanged. Wrap it in a model. Verify the output matches. Then — and only then — start decomposing. Extract CTEs. Introduce staging layers. Build tests. Refactor one model at a time, auditing each change against the original output. It’s not glamorous. It doesn’t let you redesign everything from first principles. But it works, and it works without asking the business to lose access to their reports for six months.

The Strangler Fig pattern, the Write-Audit-Publish pattern, the Expand and Contract pattern — these are all variations on the same theme: replace incrementally, verify continuously, and never throw away working logic until you’ve proved the replacement handles every case the original did.

I’ll dig into each of these in Part II. For now, the point is simpler.

The scars are data

When I look at a legacy data model — really look at it, with patience and without ego — I don’t see a mess anymore. I see a record of everything a business has learned about itself. Every weird join is a relationship somebody fought to understand. Every cryptic transformation is a business rule that was discovered through painful experience. Every seemingly arbitrary filter is a production incident that someone fixed at some point, probably at an hour they’d rather not remember.

The Netscape rewrite remains one of the most studied failures in software history. And yet here we are, in 2026, watching data teams propose the same mistake — just with different technology names. Teradata becomes Snowflake. Informatica becomes dbt. The on-prem warehouse becomes the cloud lakehouse. The pitch changes, but the impulse doesn’t: tear it down, start over, do it right this time.

My view? Your data model isn’t broken. It’s battle-tested. It’s messy because reality is messy. And the right response to inherited complexity isn’t demolition — it’s archaeology. Understand what’s there. Understand why it’s there. Then improve it, one careful change at a time.

The scars in your data model aren’t flaws. They’re knowledge. Treat them accordingly.

12 Steps to Better Data Engineering

Chris Hillman — Sat, 07 Mar 2026 09:00:00 +1100

Let me tell you about the moment I stopped trusting architecture diagrams.

I was three days into a new role, getting up to speed with the data team. Smart people. Modern stack. On paper, everything looked right. They walked me through a beautiful data platform diagram: clean lines, labelled layers, colour-coded domains. It looked like something you’d see in a data conference.

Then I asked a question that changed everything: “Can you rebuild your finance table from scratch right now?”

The room went quiet. One person started explaining that it was “mostly possible” but there were “a few manual steps” and “some seeds that someone uploads”. Someone else mentioned an incremental model that “hasn’t been full-refreshed in about eight months because last time it broke.”

That gap — between the diagram on the wall and the reality in the warehouse — is where I’ve spent most of my career. And over the years, working across team after team, I started noticing something. The teams that were struggling weren’t missing knowledge. They had smart engineers who knew about testing, CI/CD, documentation, data contracts. They’d read the blog posts. They’d watched the conference talks. What they didn’t have was a way to honestly assess where they stood and what to fix first.

I kept wishing someone would build a simple diagnostic. Maybe they just needed a handful of honest questions that would tell you, in ten minutes, whether your team was engineering or firefighting.

The moment that turned wishing into doing came about six months later, with a team I was leading. We implemented just two practices — CI testing on pull requests and mandatory code review — and I watched the compound effect over twelve weeks. Our incidents dropped. The analysts stopped asking “which table do I use?” A new hire shipped a model on day four. One of the senior engineers pulled me aside and said, “I wish we’d known how bad things were before you got here. We thought we were fine because nothing was on fire.”

Nothing was on fire. But everything was smouldering.

That’s why I built this test. Not because the world needs another framework — it doesn’t. But because the gap between thinking you’re fine and knowing where you stand is where data teams lose months of momentum. And the only way to close that gap is to ask questions honest enough that the answers sting a little.

The twelve questions

Before we get into the detail, here they are. Score yourself. One point for each “yes.”

Can you rebuild any table from raw data in one command?
Do you have a data catalog that people actually use?
Can a new analyst find the data they need without asking you?
Do you test transformations before deploying them?
Do you fix data quality issues before building new pipelines?
Do you have SLAs for your critical tables?
Do you have a single source of truth for business definitions?
Can you explain the lineage of any metric in under 2 minutes?
Do data producers know when they break downstream consumers?
Do you do code review on SQL and dbt models?
Do new hires build a real pipeline in their first week?
Do you regularly talk to the people who actually use your data?

A score of 12 is unicorn territory. I’ve never seen it in the wild. Most teams I’ve worked with score 3 or 4. Anything below 6 and you’re not doing data engineering — you’re doing data triage.

Now let’s break each one down. For every question, I’ll show you what good looks like and what amazing looks like — with concrete implementations using dbt, Snowflake, GitHub Actions, and AWS. Because “yes” isn’t binary in practice. There’s a massive gap between “sort of” and “absolutely.”

1. Can you rebuild any table from raw data in one command?

This is the foundation. If you can’t reproduce your outputs from your inputs, you don’t have a pipeline — you have a prayer.

The core principle here is idempotency: running the same operation twice produces identical results. dbt is designed around this idea, but achieving it requires deliberate choices, especially with incremental models. Too many teams build incremental models that silently drift from what a full refresh would produce, and they don’t discover the discrepancy until something breaks spectacularly.

What good looks like

Individual tables rebuild cleanly with dbt run --full-refresh --select model_name. Your incremental models are tested against full refreshes during development. You’ve configured on_schema_change: sync_all_columns so schema evolution doesn’t silently break things. And you’ve got a CI pipeline in GitHub Actions that runs dbt build on pull requests.

That’s good. That’s better than most. But it’s still table-by-table.

What amazing looks like

Your entire warehouse rebuilds from raw data using a Write-Audit-Publish (WAP) pattern. If you’re not familiar with WAP, I wrote a deep dive on implementing it with Airflow — the core idea is that new data gets written to a staging environment, audited against quality checks, and only published to production once it passes. It’s the data engineering equivalent of a preflight checklist: nothing reaches consumers until it’s been verified. And if you’re on Snowflake with Iceberg tables, the game has changed — WAP with Iceberg branching gives you git-like isolation at the table level, which means you can stage, audit, and publish without creating separate schemas or databases at all.

The practical implementation: dbt always targets a staging schema or database (or an Iceberg branch), runs all models and tests, and only on success does the data get promoted to production. Combined with Snowflake’s zero-copy cloning, this becomes practical even at scale — cloning a multi-terabyte database takes seconds and costs nothing in storage. You’re not duplicating data; you’re creating metadata pointers.

And when things do go wrong — and they will — the question becomes how quickly you can recover. If your incremental models have drifted, or a source system silently changed its schema three weeks ago, you need a backfill strategy that doesn’t involve rebuilding everything from scratch. I wrote about the pitfalls of day-by-day backfills and how to heal tables properly — it’s the companion piece to idempotency, because reproducibility isn’t just about building forward. It’s about being able to go back.

The really mature teams take it further with Slim CI. Instead of rebuilding everything on every PR, they store the production manifest.json in S3 and compare it against the new build. Only modified models and their downstream dependencies get rebuilt:

# .github/workflows/dbt-ci.yml
on:
 pull_request:
 types: [opened, reopened, synchronize]

steps:
 - name: Download production manifest
 run: aws s3 cp s3://dbt-artifacts/manifest.json ./state/manifest.json

 - name: Run dbt build (modified models only)
 run: dbt build -s 'state:modified+' --defer --state ./state --target ci

One team I worked with reported a 30% reduction in compute costs just by splitting their monolithic CI job into targeted workflows.

Watch out for this: Protect massive tables with full_refresh = false in your dbt config. One accidental --full-refresh on a billion-row fact table can ruin your morning. Override it with a variable when you actually mean it: full_refresh = var("force_full_refresh", false).

2. Do you have a data catalog that people actually use?

The emphasis is on “actually use.” I’ve lost count of how many teams have shown me a data catalog, only for me to discover that nobody’s opened it in months. The data catalog is the gym membership of the data world — everyone has one, nobody goes.

The failure mode is almost always the same: a team selects a tool, loads metadata, declares victory, and moves on. Six months later it’s a ghost town of stale descriptions and orphaned tables.

What good looks like

dbt Docs generated and hosted somewhere accessible. Your major tables have descriptions in schema.yml. The lineage graph exists and gets pulled up occasionally during incident response or onboarding. It’s not perfect, but it’s there.

What amazing looks like

The catalog is the default entry point for data questions — not Microsoft Teams, not the data engineer sitting three desks over.

Spotify’s internal tool Lexikon pushed data scientist adoption from 75% to 95% by doing something clever: they added personalised dataset recommendations, people/team pages showing who uses each dataset, common joins, and popular fields. It became a top-five internal tool because it was genuinely faster than asking a colleague.

Airbnb’s Metis serves over 1,000 data users weekly. Their secret? A Google-like search interface that works for every skill level, from SQL-fluent engineers to product managers who’ve never written a query.

The pattern that actually works for smaller teams: author descriptions in whatever UI is easiest, auto-generate a PR back to dbt’s schema.yml daily, review in Git. This keeps your definitions version-controlled without making people learn Git just to document a table.

And here’s the leading indicator that your catalog is working: track the decline in Microsoft Teams questions about data. Productboard used exactly this metric. Fewer “which table do I use for revenue?” messages meant the catalog was earning its keep.

3. Can a new analyst find the data they need without asking you?

This is the catalog question’s evil twin. A catalog helps, but self-service discovery is really about the cumulative effect of naming conventions, documentation-as-code, semantic layers, and project structure.

dbt Labs published guidance that should be printed and taped to every data engineer’s monitor: “Assume your end-user will have no other context than the model name.” Model names persist across databases, BI tools, DAGs, and docs. Folder names and schemas don’t follow the data the same way.

What good looks like

You’re using the canonical dbt naming convention: stg_ for staging, int_ for intermediate, fct_ for facts, dim_ for dimensions. Your project follows the three-layer structure — staging (1:1 with sources, materialised as views), intermediate (composable business logic), and marts (final business-ready datasets organised by domain). A reasonably technical person can navigate the DAG and find what they need within ten minutes.

What amazing looks like

A new analyst finds and understands any dataset within minutes, without messaging anyone.

Every model has both table-level and column-level descriptions. Schema field search works across all tables — “find every table with customer_id” returns results instantly. Personalised dataset recommendations surface relevant tables based on role.

And the crown jewel: a semantic layer that lets business users query metrics in plain language without writing SQL. The dbt Semantic Layer (powered by MetricFlow) centralises metric definitions in YAML alongside your models — entities, dimensions, and measures declared once, then consumed via API in Tableau, Power BI, Looker, Python notebooks, and increasingly, AI agents.

Bilt Rewards reported an 80% decrease in data costs after implementing it for embedded analytics. And dbt Labs’ testing showed 83% of natural-language questions answered correctly when routed through the semantic layer. That last number matters more than you’d think — it’s the difference between AI tooling that actually works and AI tooling that confidently gives the wrong answer.

4. Do you test transformations before deploying them?

This is probably the single highest-impact practice on the list. If I could only get a team to adopt one of these twelve, it would be this one.

But here’s what most teams get wrong: they think “testing” means slapping not_null on a few columns and calling it done. Real testing means understanding the dimensions of data quality — completeness, uniqueness, timeliness, validity, accuracy, consistency — and building checks that cover each one deliberately. I’ve written about why data quality is a deeper problem than most teams realise, and the short version is this: testing in CI is where quality becomes a habit rather than a hope. But only if your tests are actually measuring the things that matter.

The consensus across every practitioner I’ve studied is clear: CI for dbt is the most impactful thing you can do for data quality. Three complementary testing layers form a comprehensive strategy: generic dbt tests (not_null, unique, accepted_values, relationships), unit tests (introduced in dbt v1.8 for testing complex business logic with static inputs), and data diffing for value-level comparison between production and development.

What good looks like

Basic dbt tests on key models. Tests run as part of scheduled production jobs using dbt build (which runs tests immediately after each model, not in a separate pass). Pull requests exist for dbt changes, and someone eyeballs them before merging.

What amazing looks like

Full Slim CI with per-PR isolated Snowflake schemas, data diffing integrated into PR comments, and automated linting that catches style issues before a human ever looks at the code.

The numbers from real teams are staggering. Thumbtack — 50-plus analysts, five data engineers, over 100 PRs per month — previously spent one to two hours manually validating each pull request with SQL queries and spreadsheets. After integrating data diffing into their GitHub CI pipeline, they saved over 200 hours per month. That’s not a rounding error. That’s a full-time engineer’s worth of capacity recovered.

Dutchie caught timezone corruption on created_at fields and case-when logic errors that were silently shifting 20% of data between columns. Nutrafol caught a transformation that would have shown net revenue plummeting — before it reached production and before the CFO’s Monday morning dashboard refreshed.

Zscaler went even further. They built PRISM, a multi-agent AI PR review system that reduced manual review time by 90% — auto-approving conformant PRs and posting targeted comments on complex logic changes.

Here’s what the mature pipeline looks like in practice:

# On every PR: lint, compile, build modified models, diff against prod
steps:
 - name: SQLFluff lint
 run: sqlfluff lint models/ --dialect snowflake

 - name: dbt compile
 run: dbt compile

 - name: Slim CI build
 run: dbt build -s 'state:modified+' --defer --state ./state --target ci

 - name: Data Diff
 run: datafold ci submit --ci-config-id ${{ secrets.DATAFOLD_CI_CONFIG }}

5. Do you fix data quality issues before building new pipelines?

Here’s an uncomfortable stat: Monte Carlo’s 2023 State of Data Quality report found that 74% of organisations reported that business stakeholders identify data quality issues first — up from 47% the prior year. Most data teams learn about broken data from the people who are supposed to trust it. That’s not a technology problem. That’s a culture problem.

What good looks like

Basic dbt tests catch obvious issues. Tests run in production jobs. The team has an informal sense of which data matters most, usually earned through painful experience — someone got burned by bad revenue numbers in an executive review, and now those tables have tests.

What amazing looks like

Data quality SLAs enforced with error budgets, borrowed from the SRE playbook. A 99.5% data availability target allows approximately 3.6 hours of acceptable downtime per month. When the error budget is consumed, reliability work takes priority over feature delivery. Full stop. No new pipelines until you’ve earned back your quality margin.

Warner Bros. Discovery created something they call a Data Quality Forum — not a reactive incident-review meeting, but an operational bridge between teams. They used observability tooling to surface anomalies early, developed a priority matrix (P0/P1/P2), and made the forum’s patterns part of new hire onboarding. During Olympics livestreaming, custom SQL checks detected missing content metadata before it could break reporting.

Snowflake now offers native Data Metric Functions — FRESHNESS, NULL_COUNT, DUPLICATE_COUNT, UNIQUE_COUNT, ROW_COUNT — that can be scheduled to run on DML changes or time intervals. Results land in SNOWFLAKE.LOCAL.DATA_QUALITY_MONITORING_RESULTS. It’s not a replacement for dbt tests, but it provides a warehouse-level safety net that catches issues your transformation layer might miss.

The cultural shift matters more than the tooling. HelloFresh’s VP of Data drove a transformation through three stages: ad-hoc individual fixes, organised cleanup by a central team, and finally proactive quality at the source. The final stage required embedding data product owners within business domains and running data literacy programs. It’s not glamorous work. But it’s the work that changes outcomes.

6. Do you have SLAs for your critical tables?

“The dashboard is stale” is the Slack message that launches a thousand fire drills. SLAs turn that reactive scramble into a measured, prioritised response.

The breakthrough insight from practitioners: don’t aim for 100%. A 99.5% target gives you a realistic buffer for maintenance, edge cases, and the occasional Snowflake service hiccup. Different data products warrant different targets — which means you need a tiered classification system.

What good looks like

Your team knows which tables are critical, usually because they’ve been burned before. Some dbt source freshness checks are running. Basic Slack alerts fire when jobs fail. It’s reactive, but at least you know when things break.

What amazing looks like

A formal tiered classification published in the data catalog with specific, measurable commitments:

Tier 1 (Gold): ML systems, revenue reporting → PagerDuty on-call, must have unique + not_null tests + assigned owner, freshness SLA under 4 hours
Tier 2 (Silver): Executive dashboards, KPI reports → Slack team channel alerts, owner assigned, freshness SLA under 12 hours
Tier 3 (Bronze): Ad-hoc analytics, exploration tables → weekly digest, freshness SLA under 24 hours

The implementation lives in dbt meta config:

models:
 - name: fct_revenue
 meta:
 owner: "finance-data-team"
 criticality: "tier_1"
 sla_freshness: "4 hours"
 tests:
 - unique:
 column_name: order_id
 - not_null:
 column_name: order_id

SLA compliance dashboards track attainment percentage, breach counts, and trends over time. When the error budget is exhausted, feature development freezes — same principle as practice #5. Dedicated Snowflake warehouses isolate critical workloads from exploratory queries so that someone’s SELECT * on a billion-row table doesn’t cause your revenue report to miss its 8 AM deadline.

7. Do you have a single source of truth for business definitions?

Everyone’s had this conversation: “Which revenue number is right?” Two dashboards, two numbers, two teams who each think the other is wrong. This is what happens when metric definitions live inside BI tools, tribal knowledge, and the head of that one analyst who’s been here since 2019. If you’ve ever worked with dimensional models, you’ll recognise this as the conformed dimension problem — I covered why dimensional modeling still matters and how conformed dimensions are the integration backbone that prevents exactly this kind of mess.

What good looks like

Key metrics are defined in dbt docs or a wiki. The team knows which mart table is the canonical source for revenue, orders, or whatever your core entities are. When someone asks “which table do I use?”, there’s a consistent answer — even if it’s only communicated verbally.

What amazing looks like

A centralised semantic layer enforced as the only path to metrics. All definitions are version-controlled in dbt YAML, reviewed via PR, and validated in CI.

Here’s what a metric definition looks like in MetricFlow:

metrics:
 - name: cancellation_rate
 description: "Percentage of orders cancelled within 24 hours of placement"
 type: ratio
 type_params:
 numerator: cancellations
 denominator: order_total
 filter: |
 {{ TimeDimension('metric_time', 'day') }}

That definition gets consumed identically whether someone queries it from Tableau, Power BI, a Python notebook, or an AI agent. One definition. One answer. Everywhere.

Whatnot — the livestream marketplace growing at breakneck speed — solved a different version of this problem. They used Protobuf schemas as a single source of truth for event definitions, consolidating hundreds of chaotic Snowflake tables down to two clean “exposure” tables: backend_events and frontend_events. Brutal simplification. But it worked because everyone could find the data.

dbt sl validate in CI catches breaking changes to semantic definitions before merge. That’s the key differentiator — your metric definitions aren’t just documented, they’re enforced.

8. Can you explain the lineage of any metric in under 2 minutes?

When your CFO asks “where does this number come from?”, you need an answer faster than “let me check and get back to you.” Two minutes is generous. In most incident situations, you’ve got about thirty seconds before people start making assumptions.

What good looks like

The dbt DAG is viewable, hosted on S3 or an internal site. Your team can trace major mart tables back to their sources. During incident response, someone pulls up the lineage graph and walks through the chain. It takes some squinting, but the information is there.

What amazing looks like

Column-level lineage automated across the full stack — from source systems through dbt transformations to the BI dashboards consumers see.

dbt Cloud provides column-level lineage showing whether each column is “transformed” versus “passthrough/rename.” On the Snowflake side, the ACCESS_HISTORY view (Enterprise Edition) tracks both read and write operations with column-level mappings. Snowflake Labs published an open-source adapter that converts ACCESS_HISTORY data into OpenLineage JSON format, making cross-platform lineage possible.

But here’s where it gets practical. The open-source tool Recce compares two dbt environments and produces a “Lineage Diff” in CI. It categorises changes as breaking, partial-breaking, or non-breaking. PR reviewers get an instant risk assessment: “this change affects 47 downstream models, including three Tier 1 dashboards” versus “this change is isolated to a staging model with no downstream consumers.”

PRs get annotated automatically with impact analysis. Cross-project lineage via dbt Mesh connects multiple dbt projects. Lineage isn’t a pretty graph that nobody looks at — it’s used daily for onboarding, incident response, impact analysis, and cost optimisation.

9. Do data producers know when they break downstream consumers?

This is where data engineering starts borrowing seriously from software engineering. In a microservices world, you wouldn’t deploy an API change without knowing who calls your endpoint. But in data, teams routinely change source schemas, modify column semantics, or sunset tables with zero awareness of who’s consuming them downstream.

Data contracts change that dynamic.

What good looks like

dbt sources are defined with dbt source freshness running and alerting. The team is aware of major upstream dependencies. When Fivetran’s sync breaks or a source system changes its schema, someone notices within a few hours — usually because a test fails.

What amazing looks like

dbt model contracts (v1.5+) enforced on all gold-layer and public models:

models:
 - name: dim_users
 config:
 contract:
 enforced: true
 columns:
 - name: user_id
 data_type: int
 constraints:
 - type: primary_key
 - name: email
 data_type: varchar
 constraints:
 - type: not_null

This is a preflight check: dbt verifies the model’s compiled SQL returns columns matching the contract before building. Breaking changes — removed columns, changed data types, modified constraints — are caught by state:modified in CI. Model access levels (public, protected, private) control cross-project visibility.

Whatnot went further with Protobuf schemas enforced via Buf linter in CI. Every event producer runs against a common testing harness. Post-deployment monitoring catches semantic drift — when a field still exists but its meaning has changed.

Gotcha worth knowing: dbt contracts validate the compiled SQL, not the actual Snowflake table. If tools like Fivetran add columns directly to your raw layer, dbt won’t know until the next run. Monitor schema changes externally using Snowflake’s INFORMATION_SCHEMA or ACCESS_HISTORY to catch drift between runs.

10. Do you do code review on SQL and dbt models?

I’m constantly surprised by how many data teams still deploy SQL changes without review. In software engineering, unreviewed code going to production would be considered reckless. In data engineering, it’s Tuesday.

What good looks like

Pull requests are required for all dbt changes — branch protection is enabled on main. At least one human reviewer looks at each PR. There’s a basic PR description explaining what changed and why.

What amazing looks like

Automated CI running SQLFluff lint + dbt compile + dbt build + data diff on every PR. Pre-commit hooks catch issues before code is even committed. PR templates with structured sections: description, linked tickets, impact zone, testing evidence.

The recommended pre-commit stack combines SQLFluff for linting with dbt-checkpoint for structural validation:

# .pre-commit-config.yaml
repos:
 - repo: https://github.com/sqlfluff/sqlfluff
 hooks:
 - id: sqlfluff-lint
 args: [--dialect, snowflake]

 - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
 hooks:
 - id: check-model-has-tests
 - id: check-model-columns-have-desc
 - id: check-source-has-freshness

That last hook — check-model-columns-have-desc — is quietly revolutionary. It means documentation isn’t optional. You literally cannot merge a model without column descriptions. The “we’ll document it later” excuse dies right there in the CI pipeline.

Surfline (700+ dbt models) reported that after integrating SQLFluff, SQL consistency improved dramatically and reviewer burden dropped. New engineers learned “good SQL” from day one because the linter enforced it automatically. Another team, Markerr, found that automated style enforcement freed up review time to focus on deeper logic questions rather than arguing about capitalisation and trailing commas.

The dbt_project_evaluator package is worth mentioning too — it audits your entire DAG structure against dbt Labs’ published best practices. Run it in CI and it’ll flag things like models that reference sources directly (bypassing staging), duplicate sources, or models with no tests.

11. Do new hires build a real pipeline in their first week?

The speed at which a new hire becomes productive tells you everything about the state of your documentation, tooling, and team culture. If it takes three weeks before someone can make a meaningful contribution, you don’t have an onboarding problem — you have a platform problem. I wrote a guide to navigating your first 90 days as a data engineer — and the teams that make those first 90 days count are almost always the ones who’ve invested in the infrastructure described here.

What good looks like

New hire has access to tools within a day or two. Some documentation exists. A buddy or mentor is assigned. They can run the dbt project locally by end of week one, even if they haven’t contributed anything yet.

What amazing looks like

Pre-arrival setup is complete before Day 1: machine provisioned, logins ready, Snowflake roles assigned, dbt Cloud account active. No engineer should spend their first morning installing things.

Each developer gets a personal Snowflake sandbox — dbt_<username> schema in the dev database, with write access only to dev and read-only access to raw/staging. Snowflake’s zero-copy cloning makes production data available instantly without duplicating storage costs. Productboard open-sourced dbt-snowflake-sandbox — a set of dbt macros that create isolated sandboxes by cloning only the specific model dependencies a developer needs.

The first-week project is structured and progressive:

Day 1-2: Run the existing dbt project locally. Explore the DAG. Read the key model documentation.
Day 3: Add a new source or staging model for a real (but low-risk) dataset.
Day 4: Build a staging model with tests and documentation.
Day 5: Open a PR, go through the review process, and get it merged.

By Friday, they’ve shipped something real. They understand the workflow. They’ve experienced CI, code review, and deployment. And critically, they feel like a contributor rather than a tourist.

7shifts estimated cutting onboarding time by over a week per new hire just by having documentation and data questions accessible in their data catalog.

12. Do you regularly talk to the people who actually use your data?

This is the question that separates data teams who build for their portfolio from data teams who build for their business. You can nail every technical practice on this list and still fail if you’re building the wrong things. I’ve written before about how to maximise your data team’s impact — and the through-line is always the same: the teams that create real value are the ones who stay close to the people consuming their work.

What good looks like

A Slack channel exists for data questions. Communication happens when issues arise. There are occasional stakeholder meetings, usually prompted by something breaking or a new request coming in.

What amazing looks like

Data office hours. Weekly or bi-weekly open sessions where anyone in the organisation can bring data questions. Not a presentation. Not a status update. An open door.

Holistics published a detailed playbook for running what they call “data clinics.” The core principles: teach, don’t serve. Show business users how to self-serve rather than just answering their question and sending them on their way. Montreal Analytics reported that after implementing data clinics, the number of self-serve business users on their BI tool grew tenfold. Tenfold.

Pair that with a monthly Data NPS survey — a single question: “How likely are you to recommend our data team’s products to a colleague?” It sounds corporate, but without this metric, you have no way to quantify whether your consumers are satisfied or just silently building workarounds in Excel.

dbt exposures formalise the connection between data models and their consumers:

exposures:
 - name: weekly_revenue_dashboard
 type: dashboard
 owner:
 name: Finance Team
 email: finance@company.com
 depends_on:
 - ref('fct_revenue')
 - ref('dim_date')

When a model changes, the PR shows which exposures are affected. Stakeholder impact becomes visible in code review. You can proactively notify the finance team that their revenue dashboard’s upstream model is changing before they discover it themselves.

Frame your communication in business terms. Not “model precision is up 12%” but “the sales team can now identify high-value leads 40% faster.” Nobody outside your team cares about your DAG. They care about whether they can trust the numbers. If you’re not sure how to bridge that gap, I wrote a guide on breaking down business context — because the hardest part of talking to stakeholders isn’t the talking, it’s knowing what they actually need to hear.

Scoring your team

Here’s the scoring framework, and I want you to be ruthless with yourself:

Score	What it means
10-12	You’re in elite territory. Your data platform is a competitive advantage.
7-9	You’ve got strong foundations with clear areas to improve.
4-6	You’re functional but fragile. One bad incident away from a crisis of trust.
1-3	You’re firefighting, not engineering. The business tolerates your team — it doesn’t trust it.

Most teams land at 3-4. That’s not a criticism — it’s where the industry is. The gap between knowing these practices exist and actually implementing them is where the real work lives.

Where to start

If you’re staring at a low score and feeling overwhelmed, here’s what I’d suggest: start with practices 4 and 10. CI testing and code review.

Here’s why. Once dbt changes go through pull requests with automated checks, you have the infrastructure to enforce everything else. Contracts? They’re a CI check. Quality gates? CI check. Documentation requirements? CI check. SLA validation? CI check. You’re not adopting twelve practices — you’re building one pipeline that gates on twelve things.

The canonical architecture appears across virtually every mature team I’ve studied: GitHub Actions triggering Slim CI with dbt build -s state:modified+ against per-PR Snowflake schemas, with production manifests stored in S3 for state comparison. Start there. Layer on the remaining practices as your team’s maturity grows.

Three patterns emerged from the research that I think are worth calling out explicitly:

“As code” wins everywhere. Documentation-as-code, style-guides-as-code, permissions-as-code, quality-checks-as-code, contracts-as-code. Every manual process that gets codified becomes version-controlled, reviewable, and enforceable. Every one that stays manual eventually drifts.

Snowflake’s zero-copy cloning is the force multiplier. Isolated PR environments, sandbox onboarding, rebuild testing — all of these become cheap and instant with cloning. If you’re on Snowflake and not using this feature aggressively, you’re leaving the most powerful tool in the shed.

Culture matters more than tooling. Catalog adoption fails without meeting users in their existing workflows. Data quality requires executive sponsorship and error budgets, not just more tests. Producer-consumer contracts are first and foremost a cultural change. You can buy every tool on this list and still score a 3 if the organisation doesn’t value the practices behind them.

The diagram and the warehouse

Remember that team — the one with the beautiful diagram and the revenue table nobody could rebuild?

I caught up with one of the engineers about four months after I’d moved on. They’d started with exactly what I’d pushed for: CI testing and code review. Then they added contracts on their gold-layer models. Then SLAs on their Tier 1 tables. Then a first-week onboarding project for new hires.

Their score when I first asked these twelve questions? Three. Four months later? Eight. Not perfect. But the difference wasn’t really the number. The difference was that when they pulled up that architecture diagram now, it matched what was actually running in Snowflake. The gap between the wall and the warehouse had closed.

That’s what this test is really measuring. Not whether you have the right tools — you probably do. Not whether you know the right practices — you’re reading this, so clearly you care. It’s measuring whether there’s a gap between what you think your data platform looks like and what it actually looks like.

These twelve questions just make you say it out loud.

So print it. Score yourself honestly. Share it with your team. And the next time someone asks you how your data platform is doing, give them a number between 1 and 12.

That number is worth more than any architecture diagram.

The CSV Test Suite Nobody Writes

Chris Hillman — Wed, 04 Mar 2026 09:00:00 +1100

In October 2020, roughly 16,000 positive COVID-19 test results vanished from the UK’s public health reporting for nearly a week. Not because the tests weren’t run. Not because the labs didn’t report them. The results were collected, transmitted, and received — inside CSV files.

The problem? Public Health England was importing those CSV files into Microsoft Excel’s legacy .xls format. The format has a hard row limit of 65,536. When the files grew past that limit, Excel didn’t throw an error. It didn’t warn anyone. It just silently dropped the extra rows. Sixteen thousand people who tested positive for a deadly virus during a second wave went untraced. An estimated 50,000 of their contacts were never notified. And the system this happened in? Part of a £12 billion Test and Trace programme.

I remember reading that story and feeling something beyond frustration. I felt recognition. Because I’d seen this pattern before — not at that scale, not with those stakes — but the same fundamental failure. Someone produced a CSV. Someone else consumed it. And nobody, at any point in the chain, had written a test that asked: “What happens when this file gets bigger than we expect?”

That’s the moment I became a bit obsessive about CSV testing. Not because CSV is complicated — it’s deceptively simple. But because that simplicity breeds a dangerous confidence. Everyone thinks they know how to handle a CSV. Everyone thinks the format is too basic to need a test suite. And that assumption has cost organisations billions of dollars, corrupted scientific research, and — in the case of COVID — put actual lives at risk.

I’m telling you this because CSV testing is one of those things that sounds unnecessary until you’ve spent a weekend rebuilding a pipeline because a vendor changed their quoting convention and nobody noticed for three weeks. The engineers I’ve worked with who do this well — who write proper CSV test suites — don’t do it because they’re cautious by nature. They do it because they’ve been burned. And they decided it wouldn’t happen again.

This article is the test suite I wish someone had handed me years ago. We’re going to take RFC 4180, the closest thing CSV has to a standard, and turn it into concrete test cases. We’ll build manifest files that make file delivery verifiable. And we’ll walk through how to systematically break down everything you know about CSV edge cases — embedded commas, encoding traps, null representation chaos, the lot — into tests you can run during development, not discoveries you make in production at 2am.

Whether you’re producing CSVs for a downstream consumer or receiving them from a vendor who swears the file is “fine,” this is your playbook.

Why “I Know How to Test” Isn’t Enough

Here’s what I’ve noticed when I ask data engineers how they test their CSV files: most of them describe validation, not testing. They’ll say things like “we check the row count” or “we verify the columns match the schema.” And those are good things to do. But they’re runtime checks — they catch problems after they’ve already arrived at your door.

Testing is different. Testing happens during development. Testing means deliberately creating malformed files and confirming your system handles them correctly. Testing means asking “what’s the worst thing a vendor could send me?” and then actually building that worst-case file and feeding it through your pipeline.

The gap between validation and testing is where the expensive surprises live. Validation tells you this specific file has a problem. Testing tells you your system has a problem — before any real data gets near it.

So how do you build a proper CSV test suite? You need a specification to test against. And despite its limitations, that specification is RFC 4180.

RFC 4180: Your Test Specification (and Its Gaps)

RFC 4180 was published in October 2005 by Yakov Shafranovich. It’s universally treated as “the CSV standard,” but its opening line tells a different story: it’s an “Informational” memo that “does not specify an Internet standard of any kind.” It was written to document common practice and register the text/csv MIME type. That’s it.

But here’s the thing — even a flawed specification is better than no specification. RFC 4180 gives us seven concrete rules, and each one translates directly into test cases. Let’s walk through them.

Rule 1: Records Are Separated by CRLF

The spec mandates \r\n (carriage return + line feed) as the line ending. Not just \n (Unix/macOS default). Not just \r (old Mac).

Test cases for producers:

def test_line_endings_are_crlf(csv_content: bytes):
 """Every record must end with CRLF, not LF-only or CR-only."""
 lines = csv_content.split(b'\r\n')
 # Check no bare LF exists outside of quoted fields
 for i, line in enumerate(lines):
 assert b'\n' not in line, (
 f"Row {i+1}: Found bare LF (Unix line ending). "
 f"RFC 4180 requires CRLF."
 )
 # Check no bare CR exists outside of quoted fields
 for i, line in enumerate(lines):
 assert b'\r' not in line, (
 f"Row {i+1}: Found bare CR (old Mac line ending). "
 f"RFC 4180 requires CRLF."
 )

Test cases for consumers:

def test_consumer_handles_mixed_line_endings(parser):
 """Consumer should handle LF, CR, and CRLF gracefully."""
 test_cases = {
 "unix_lf": b"name,age\nAlice,30\nBob,25\n",
 "windows_crlf": b"name,age\r\nAlice,30\r\nBob,25\r\n",
 "old_mac_cr": b"name,age\rAlice,30\rBob,25\r",
 "mixed": b"name,age\r\nAlice,30\nBob,25\r",
 }
 for label, content in test_cases.items():
 result = parser(content)
 assert len(result) == 2, (
 f"Line ending style '{label}': Expected 2 data rows, "
 f"got {len(result)}"
 )

This distinction matters. If you’re a producer, your job is to emit compliant files. If you’re a consumer, your job is to survive whatever arrives. The test mindset is different for each role — producers test for correctness, consumers test for resilience.

Rule 2: The Last Record May or May Not Have a Trailing CRLF

This sounds trivial, but it catches people. Some systems emit a final newline, others don’t. Your parser needs to handle both without creating a phantom empty row at the end.

def test_trailing_newline_does_not_create_empty_row(parser):
 """A trailing CRLF should not produce an extra empty record."""
 with_trailing = b"name,age\r\nAlice,30\r\n"
 without_trailing = b"name,age\r\nAlice,30"
 assert len(parser(with_trailing)) == len(parser(without_trailing))

Rule 3: There Should Be a Header Row

The RFC says “should” — not “must.” This ambiguity is a source of real problems. Some vendors send headers, some don’t, and some send headers that change between deliveries.

Not having headers isn’t always a mistake. From a security perspective, omitting headers is actually defensible. Think about what a header row really is: it’s a plaintext schema sitting right there in the first line of your file. If someone intercepts a CSV with columns named ssn, credit_score, annual_income, and account_balance, they don’t need to decode anything. The header tells them exactly what they’ve stolen and which column to exploit first.

Stripping headers from files in transit — especially files containing PII, financial data, or health records — reduces the value of an intercepted file to an attacker. A file full of numbers and strings without column names is significantly harder to weaponise than one with a helpful schema attached.

But here’s the catch: headerless files only work if you have a reliable mechanism to reconstruct the schema at the consumer end. The column definitions need to live somewhere — and that somewhere should be your manifest file or your data contract. The manifest we build later in this article includes a columns array with names, types, and ordering for exactly this reason. The producer strips the headers, the manifest carries the schema, and the consumer rebuilds the structure on ingestion.

This means your tests need to handle both scenarios:

def test_header_row_present_and_matches_schema(csv_file, expected_columns):
 """When headers are expected, verify they match the schema exactly."""
 header = csv_file.readline().strip().split(',')
 assert header == expected_columns, (
 f"Header mismatch.\n"
 f"Expected: {expected_columns}\n"
 f"Got: {header}"
 )

def test_no_duplicate_column_names(csv_file):
 """Duplicate column names cause silent column overwrites in pandas."""
 header = csv_file.readline().strip().split(',')
 duplicates = [h for h in header if header.count(h) > 1]
 assert not duplicates, (
 f"Duplicate column names found: {set(duplicates)}. "
 f"Pandas will silently rename these to name.1, name.2, etc."
 )

def test_headerless_file_matches_manifest_schema(csv_file, manifest):
 """When headers are intentionally omitted for security,
 the first data row must match the manifest's column count
 and the manifest must define column names and types."""
 first_row = csv_file.readline().strip().split(',')
 expected_cols = manifest['files'][0]['columns']
 assert len(first_row) == len(expected_cols), (
 f"Headerless file has {len(first_row)} fields but manifest "
 f"defines {len(expected_cols)} columns. Without headers, "
 f"column alignment is entirely dependent on the manifest."
 )
 # Verify manifest actually defines names (not just count)
 for col_def in expected_cols:
 assert 'name' in col_def, (
 "Manifest column definitions must include 'name' — "
 "headerless files are only safe if the schema is "
 "fully specified in the manifest."
 )

Whether you include headers or not, the decision must be explicit and documented in your data contract. The worst scenario isn’t a headerless file or a headered file — it’s a file where nobody agreed which one it should be, and the answer changes depending on who ran the export.

Rule 4: Each Row Should Contain the Same Number of Fields

This is where “ragged rows” come from — and they’re surprisingly common. A free-text field containing an unescaped comma silently creates an extra column in that row, shifting everything to the right.

def test_consistent_column_count(csv_reader):
 """Every row must have the same number of fields as the header."""
 header = next(csv_reader)
 expected = len(header)
 ragged_rows = []
 for i, row in enumerate(csv_reader, start=2):
 if len(row) != expected:
 ragged_rows.append({
 'row': i,
 'expected': expected,
 'got': len(row),
 'content': row[:3] # First 3 fields for debugging
 })
 assert not ragged_rows, (
 f"Found {len(ragged_rows)} ragged rows. "
 f"First: row {ragged_rows[0]['row']} has "
 f"{ragged_rows[0]['got']} fields (expected {expected}). "
 f"This usually means an unescaped comma or newline in the data."
 )

This test is the single most valuable check you can run against a CSV file. If it passes, most of the structural problems aren’t present. If it fails, you know exactly where to look.

Rules 5-7: Quoting and Escaping

These three rules govern how fields handle special characters:

Fields containing commas, double-quotes, or line breaks must be enclosed in double-quotes
Double-quotes inside a quoted field are escaped by doubling them: ""
Spaces are part of field data and must not be trimmed

These rules are where the most insidious bugs hide:

# -- Producer tests: Do we generate valid quoted fields? --

def test_fields_with_commas_are_quoted():
 """A field containing a comma must be wrapped in double-quotes."""
 record = generate_csv_row(address="456 Oak Ave, Apt 2B")
 assert record == 'id,"456 Oak Ave, Apt 2B",other_field'

def test_fields_with_quotes_are_escaped():
 """Double-quotes in data must be doubled, and field must be quoted."""
 record = generate_csv_row(company='Acme "Holdings" Ltd')
 assert '"Acme ""Holdings"" Ltd"' in record

def test_fields_with_newlines_are_quoted():
 """A field containing a line break must be wrapped in double-quotes."""
 record = generate_csv_row(notes="Line one\r\nLine two")
 # The entire field, including the newline, should be inside quotes
 assert '"Line one\r\nLine two"' in record

# -- Consumer tests: Can we parse valid-but-tricky fields? --

def test_parse_embedded_comma():
 content = b'name,address\r\n"Alice","456 Oak Ave, Apt 2B"\r\n'
 result = parse(content)
 assert result[0]['address'] == '456 Oak Ave, Apt 2B'

def test_parse_embedded_newline():
 content = b'name,notes\r\n"Alice","Line one\r\nLine two"\r\n'
 result = parse(content)
 assert result[0]['notes'] == 'Line one\r\nLine two'

def test_parse_escaped_quotes():
 content = b'name,company\r\n"Alice","Acme ""Holdings"" Ltd"\r\n'
 result = parse(content)
 assert result[0]['company'] == 'Acme "Holdings" Ltd'

That covers the seven RFC rules. But here’s the catch — the RFC is only half the story. The really nasty problems come from everything the RFC doesn’t cover.

Testing What the RFC Forgot

RFC 4180 says nothing about character encoding, data types, null values, or how Excel will mangle your data. These are the gaps where production failures breed. Each one becomes a test category.

Encoding Tests

The most common encoding problem is invisible: the UTF-8 BOM (Byte Order Mark). Excel on Windows prepends three bytes (EF BB BF) to CSV files saved as UTF-8. Those bytes become an invisible character attached to your first column name. You try df['Name'] and get a KeyError because the actual column is '\ufeffName'.

def test_no_bom_prefix(csv_path):
 """UTF-8 BOM causes invisible characters in the first column name."""
 with open(csv_path, 'rb') as f:
 first_bytes = f.read(3)
 assert first_bytes != b'\xef\xbb\xbf', (
 "File contains UTF-8 BOM. This will prepend an invisible "
 "character to the first column name and break column lookups."
 )

def test_file_is_valid_utf8(csv_path):
 """Catch encoding mismatches before they become mojibake."""
 with open(csv_path, 'rb') as f:
 content = f.read()
 try:
 content.decode('utf-8')
 except UnicodeDecodeError as e:
 # Try to identify the actual encoding
 import chardet
 detected = chardet.detect(content)
 raise AssertionError(
 f"File is not valid UTF-8. Decode error at byte {e.start}. "
 f"Detected encoding: {detected['encoding']} "
 f"(confidence: {detected['confidence']:.0%}). "
 f"This is likely a Latin-1 or Windows-1252 file."
 )

def test_no_null_bytes(csv_path):
 """Null bytes (0x00) indicate binary data or a UTF-16 file
 being read as UTF-8."""
 with open(csv_path, 'rb') as f:
 content = f.read()
 null_positions = [i for i, b in enumerate(content) if b == 0]
 assert not null_positions, (
 f"Found {len(null_positions)} null bytes. "
 f"First at position {null_positions[0]}. "
 f"This might be a UTF-16 encoded file."
 )

If you’re a consumer, you should also test that your parser survives non-UTF-8 input gracefully — does it raise a clear error, or does it silently produce mojibake? Build test fixtures with Latin-1 encoded files containing accented characters (café, naïve, München) and verify your system either decodes correctly or fails explicitly.

Null Value Tests

RFC 4180 has no concept of null. An empty field (,,) is just an empty string. But every system represents null differently, and when you’re receiving files from multiple vendors, the chaos compounds.

# These are all real null representations I've seen in production
NULL_VARIANTS = [
 '', # Empty string (most common)
 'NULL', # MySQL Workbench default
 'null', # JavaScript/JSON convention
 'None', # Python convention
 r'\N', # PostgreSQL COPY convention
 'N/A', # Business user convention
 'n/a', # Lowercase variant
 'NA', # R convention
 '#N/A', # Excel formula error
 '-', # Dashboard export convention
 'NaN', # Pandas/NumPy convention
 '(null)', # Some BI tools
]

def test_null_representation_is_consistent(csv_path, null_convention=''):
 """All null values should use the agreed convention.
 Mixed conventions cause type inference failures."""
 import csv
 with open(csv_path, newline='') as f:
 reader = csv.reader(f)
 header = next(reader)
 violations = []
 for row_num, row in enumerate(reader, start=2):
 for col_idx, value in enumerate(row):
 if value in NULL_VARIANTS and value != null_convention:
 violations.append({
 'row': row_num,
 'column': header[col_idx],
 'value': repr(value),
 'expected': repr(null_convention)
 })
 assert not violations, (
 f"Found {len(violations)} non-standard null representations. "
 f"First: row {violations[0]['row']}, "
 f"column '{violations[0]['column']}' contains "
 f"{violations[0]['value']} (expected {violations[0]['expected']})"
 )

This is a test you should agree on with your vendor as part of a data contract — which we’ll get to shortly. Without explicit agreement, you’ll spend more time debugging null handling than you will on actual analysis.

The Leading Zeros Problem

Excel silently strips leading zeros from any field it interprets as numeric. ZIP code 07102 becomes 7102. Product code 00456 becomes 456. Numbers over 15 digits lose precision permanently. This is not a CSV problem — it’s an Excel problem — but it affects every CSV that passes through Excel, which is most of them.

def test_leading_zeros_preserved(csv_path, zero_padded_columns):
 """Fields that should retain leading zeros still have them.
 Excel strips these silently on open-and-save."""
 import csv
 with open(csv_path, newline='') as f:
 reader = csv.DictReader(f)
 for row_num, row in enumerate(reader, start=2):
 for col in zero_padded_columns:
 value = row[col]
 if value and not value[0].isdigit():
 continue # Skip non-numeric values
 if value and value[0] == '0' and len(value) > 1:
 pass # Good — leading zero preserved
 elif value and col in zero_padded_columns:
 # Check against expected length
 # e.g., ZIP should be 5 digits
 pass

def test_no_scientific_notation(csv_path, numeric_columns):
 """Numbers should not be in scientific notation (1.23E+11).
 This happens when Excel converts long numbers."""
 import csv, re
 sci_notation = re.compile(r'^-?\d+\.?\d*[eE][+-]?\d+$')
 with open(csv_path, newline='') as f:
 reader = csv.DictReader(f)
 violations = []
 for row_num, row in enumerate(reader, start=2):
 for col in numeric_columns:
 if sci_notation.match(row[col] or ''):
 violations.append((row_num, col, row[col]))
 assert not violations, (
 f"Scientific notation found — data may have been "
 f"corrupted by Excel. First: row {violations[0][0]}, "
 f"column '{violations[0][1]}', value '{violations[0][2]}'"
 )

Date Format Tests

Is 03/04/2024 March 4th or April 3rd? Unless you know the source system’s locale, you literally cannot tell for dates where both day and month values are 12 or below. The only safe format is ISO 8601: YYYY-MM-DD.

def test_dates_are_iso8601(csv_path, date_columns):
 """Date fields should use ISO 8601 (YYYY-MM-DD) format.
 Ambiguous formats like MM/DD/YYYY vs DD/MM/YYYY
 are programmatically unresolvable."""
 import csv, re
 iso_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}')
 with open(csv_path, newline='') as f:
 reader = csv.DictReader(f)
 violations = []
 for row_num, row in enumerate(reader, start=2):
 for col in date_columns:
 val = row.get(col, '')
 if val and not iso_pattern.match(val):
 violations.append((row_num, col, val))
 assert not violations, (
 f"Non-ISO date format found. First: row {violations[0][0]}, "
 f"column '{violations[0][1]}', value '{violations[0][2]}'. "
 f"Ambiguous date formats cannot be safely parsed."
 )

Invisible Character Tests

These are the ones that make you question your sanity. Non-breaking spaces (U+00A0), zero-width spaces (U+200B), zero-width joiners — they look identical to normal characters (or invisible entirely) but break numeric conversion, string matching, and joins.

INVISIBLE_CHARS = {
 '\u00a0': 'Non-breaking space',
 '\u200b': 'Zero-width space',
 '\u200c': 'Zero-width non-joiner',
 '\u200d': 'Zero-width joiner',
 '\ufeff': 'BOM / Zero-width no-break space',
 '\u2028': 'Line separator',
 '\u2029': 'Paragraph separator',
 '\u00ad': 'Soft hyphen',
}

def test_no_invisible_characters(csv_path):
 """Invisible Unicode characters break joins, type casting,
 and string comparisons while being impossible to spot visually."""
 with open(csv_path, encoding='utf-8') as f:
 content = f.read()
 found = []
 for char, name in INVISIBLE_CHARS.items():
 positions = [i for i, c in enumerate(content) if c == char]
 if positions:
 found.append(f"{name} (U+{ord(char):04X}): "
 f"{len(positions)} occurrences")
 assert not found, (
 "Invisible characters detected:\n" +
 "\n".join(f" - {f}" for f in found)
 )

Manifest Files: Testing Secure Delivery

All of those tests assume you’ve received the file intact. But how do you know? A file could be truncated during transfer, silently corrupted on disk, or swapped out entirely. This is where manifest files come in — they’re the contract between producer and consumer.

A manifest is metadata about the file that travels alongside it. At minimum, it should contain the filename, a cryptographic checksum, the row count, and the file size. NIST has formally deprecated MD5 and SHA-1, so use SHA-256.

Here’s what a solid manifest looks like:

{
 "manifest_version": "1.0",
 "generated_at": "2026-03-04T08:00:00Z",
 "source_system": "vendor-crm",
 "batch_id": "batch-20260304-001",
 "files": [
 {
 "filename": "customers_20260304.csv",
 "sha256": "a3f2b8c9d1e4f5a6b7c8d9e0f1a2b3c4d5e6f7...",
 "row_count": 50432,
 "row_count_includes_header": true,
 "file_size_bytes": 8234567,
 "column_count": 12,
 "encoding": "UTF-8",
 "delimiter": ",",
 "quote_char": "\"",
 "has_header": true,
 "columns": [
 { "name": "id", "type": "integer" },
 { "name": "name", "type": "string" },
 { "name": "email", "type": "string" },
 { "name": "created_date", "type": "date", "format": "YYYY-MM-DD" }
 ]
 }
 ]
}

Now here’s the test suite that validates a delivery against its manifest:

import hashlib
import json
import os

def validate_delivery(manifest_path, data_dir):
 """Validate a CSV delivery against its manifest.
 Returns list of errors. Empty list = valid delivery."""
 errors = []

 # 1. Parse manifest
 with open(manifest_path) as f:
 manifest = json.load(f)

 for file_spec in manifest['files']:
 filepath = os.path.join(data_dir, file_spec['filename'])

 # 2. File existence
 if not os.path.exists(filepath):
 errors.append(f"MISSING: {file_spec['filename']} not found")
 continue

 # 3. File size
 actual_size = os.path.getsize(filepath)
 if actual_size == 0:
 errors.append(f"EMPTY: {file_spec['filename']} is zero bytes")
 continue
 if actual_size != file_spec['file_size_bytes']:
 errors.append(
 f"SIZE MISMATCH: {file_spec['filename']} "
 f"expected {file_spec['file_size_bytes']} bytes, "
 f"got {actual_size}"
 )

 # 4. Checksum
 sha256 = hashlib.sha256()
 with open(filepath, 'rb') as f:
 for chunk in iter(lambda: f.read(8192), b''):
 sha256.update(chunk)
 actual_hash = sha256.hexdigest()
 if actual_hash != file_spec['sha256']:
 errors.append(
 f"CHECKSUM MISMATCH: {file_spec['filename']} "
 f"has been modified or corrupted in transit"
 )

 # 5. Row count
 with open(filepath, encoding=file_spec.get('encoding', 'utf-8')) as f:
 actual_rows = sum(1 for _ in f)
 expected = file_spec['row_count']
 if actual_rows != expected:
 errors.append(
 f"ROW COUNT MISMATCH: {file_spec['filename']} "
 f"expected {expected} rows, got {actual_rows}"
 )

 # 6. Column count and names
 with open(filepath, encoding=file_spec.get('encoding', 'utf-8')) as f:
 import csv
 reader = csv.reader(f)
 header = next(reader)
 if len(header) != file_spec['column_count']:
 errors.append(
 f"COLUMN COUNT MISMATCH: {file_spec['filename']} "
 f"expected {file_spec['column_count']} columns, "
 f"got {len(header)}"
 )
 if 'columns' in file_spec:
 expected_names = [c['name'] for c in file_spec['columns']]
 if header != expected_names:
 errors.append(
 f"COLUMN NAME MISMATCH: {file_spec['filename']}\n"
 f" Expected: {expected_names}\n"
 f" Got: {header}"
 )

 return errors

The key insight about manifest validation is that it should be a gate, not a check. If the manifest fails, you don’t process the file. Period. No “well, the row count is close enough.” No “the checksum is probably fine.” Either the delivery matches the contract or it gets quarantined.

For orchestration tools like Airflow, the pattern looks like this:

# Airflow DAG pattern: manifest-gated processing
wait_for_manifest = FileSensor(
 task_id="wait_for_manifest",
 filepath="/landing/{{ ds_nodash }}/manifest.json",
 poke_interval=120,
 timeout=7200,
)

validate = BranchPythonOperator(
 task_id="validate_delivery",
 python_callable=validate_and_route,
 # Routes to either 'load_data' or 'quarantine_and_alert'
)

wait_for_manifest >> validate >> [load_data, quarantine_and_alert]

As a producer, you should generate the manifest after writing the CSV file — computing the checksum on the exact bytes that were written. Ship the manifest alongside the data file. If you’re using SFTP or S3, upload the data file first, then the manifest. The consumer watches for the manifest as a “ready” signal — its presence means the data file is complete.

As a consumer, you should test your manifest validation against deliberately corrupted scenarios:

def test_detects_truncated_file():
 """Simulate a file that was cut short during transfer."""
 # Write a file, then truncate it
 ...

def test_detects_missing_file():
 """Manifest references a file that doesn't exist."""
 ...

def test_detects_extra_rows():
 """File has more rows than manifest claims —
 could indicate duplicate data or appended garbage."""
 ...

def test_detects_wrong_checksum():
 """File was modified after manifest was generated."""
 ...

def test_rejects_zero_byte_file():
 """Empty files should never pass validation."""
 ...

The CSV Test Matrix: Thinking Like a Tester

Most people know about edge cases — embedded commas, encoding problems, null ambiguity — but they don’t know how to systematically translate that knowledge into a test suite. They end up with ad hoc checks scattered across their codebase, catching some things and missing others.

The trick is to think in categories, not individual cases. Every CSV problem falls into one of five categories, and each category has a predictable set of test cases.

Category 1: Structural Integrity

Does the file conform to the expected shape?

Test	What It Catches	Producer/Consumer
Consistent column count across all rows	Unescaped delimiters in data	Both
Header row present and matches schema (when expected)	Schema drift, renamed columns	Consumer
Headerless file matches manifest column count	Misaligned data when headers stripped for security	Consumer
No duplicate column names	Silent column overwrites	Consumer
No trailing delimiter creating phantom column	Off-by-one column counts	Both
File is not empty / not just a header	Failed exports, empty result sets	Consumer
Row count within expected range	Truncation, duplication, data loss	Consumer

Category 2: Encoding and Character Safety

Are the bytes what we think they are?

Test	What It Catches	Producer/Consumer
File is valid UTF-8	Latin-1/Windows-1252 masquerading as UTF-8	Both
No BOM prefix	Invisible first-column corruption	Producer
No null bytes	Binary data or wrong encoding family	Consumer
No invisible Unicode characters	Broken joins, failed type casts	Both
No smart quotes or em-dashes	Copy-paste from Word/email	Both

Category 3: Quoting and Escaping

Are special characters handled correctly?

Test	What It Catches	Producer/Consumer
Fields with commas are quoted	Row splitting, column misalignment	Producer
Fields with double-quotes use `""` escaping	Unclosed quotes consuming multiple rows	Producer
Fields with newlines are quoted	Row count mismatches, multi-line chaos	Producer
Consumer survives unclosed quotes gracefully	Vendor bugs eating your pipeline	Consumer
No backslash escaping (non-standard)	Interop failures with strict parsers	Producer

Category 4: Data Type Safety

Will downstream systems interpret values correctly?

Test	What It Catches	Producer/Consumer
Dates in ISO 8601 format	Ambiguous MM/DD vs DD/MM	Both
Leading zeros preserved	Excel corruption of ZIPs, product codes	Both
No scientific notation in numeric fields	Excel corruption of long numbers	Both
Consistent null representation	Mixed NULL/None/N/A/empty conventions	Both
Boolean fields use consistent values	TRUE/True/true/1/yes/Y chaos	Both
Numeric fields contain only valid numbers	Currency symbols, percentage signs, commas in numbers	Consumer

Category 5: Delivery Integrity

Did the file arrive intact and complete?

Test	What It Catches	Producer/Consumer
SHA-256 checksum matches manifest	Corruption in transit	Consumer
File size matches manifest	Truncation, wrong file	Consumer
Row count matches manifest	Silent data loss (the COVID problem)	Consumer
Column count matches manifest	Schema changes without notice	Consumer
File is not a duplicate of previous delivery	Accidental re-sends	Consumer
Manifest is present and valid JSON	Incomplete delivery	Consumer

This matrix is your test plan. Print it out. Stick it on the wall next to your desk. When you’re building a new CSV integration, walk through each category and ask: “Do I have a test for this?”

Building Test Fixtures: The Art of Deliberate Breakage

A test suite is only as good as its fixtures. And for CSV testing, your fixtures need to be deliberately, creatively broken. Here’s how I build mine.

Start with a golden file — a perfectly compliant CSV that passes every test. This is your baseline. Then create variations that each break exactly one thing:

Each fixture is a test case. Your producer tests verify you never generate files that look like the broken fixtures. Your consumer tests verify you detect and handle each type of breakage correctly.

The most valuable fixture? unclosed_quote.csv. A single unclosed double-quote can cause a parser to consume every subsequent row as part of one giant field — potentially the entire rest of the file. I’ve seen this eat 2 million rows. Test for it explicitly.

What to Put in Your Data Contract

Tests tell you whether a specific file is good or bad. A data contract tells you what “good” means. Without one, you’re testing against assumptions — and your assumptions will eventually differ from your vendor’s.

Here’s what a data contract for CSV exchange should specify. Think of it as the RFC that RFC 4180 forgot to write:

File format:

Delimiter (comma, pipe, tab — be explicit)
Quote character (double-quote)
Escape method (doubled quotes, per RFC 4180)
Line ending (CRLF)
Encoding (UTF-8, no BOM)
Header row policy (present, omitted for security, or conditional — and if omitted, where the schema lives)
Column names and order (in manifest if headers are stripped)

Data type conventions:

Date format (ISO 8601: YYYY-MM-DD)
Timestamp format (ISO 8601: YYYY-MM-DDTHH:MM:SSZ)
Null representation (empty field — no “NULL”, no “N/A”)
Boolean representation (true/false)
Numeric precision (no scientific notation, no thousands separators)

Delivery protocol:

Transfer method (SFTP, S3, etc.)
Manifest file required (JSON, SHA-256 checksum)
Signal file or completion marker
Delivery schedule and SLA
Who to contact when something breaks

Change management:

Minimum notice period for schema changes
Versioning strategy (new columns appended, never reordered)
Breaking vs non-breaking change definitions

You don’t need a 50-page legal document. A one-page markdown file committed to a shared repository works. The point isn’t formality — it’s explicitness. When the vendor’s export changes and your pipeline breaks at 3am, you want something to point to that says “this is what we agreed.”

Putting It All Together: A Reusable Validation Function

Here’s a practical, copy-paste-ready validation function that covers the most critical checks. This isn’t a framework — it’s a function you can drop into any Python project and start using immediately.

import csv
import hashlib
import os
import re
from dataclasses import dataclass, field
from typing import Optional


@dataclass
class CSVValidationResult:
 """Results from validating a CSV file."""
 filepath: str
 is_valid: bool = True
 errors: list = field(default_factory=list)
 warnings: list = field(default_factory=list)
 stats: dict = field(default_factory=dict)

 def add_error(self, message: str):
 self.errors.append(message)
 self.is_valid = False

 def add_warning(self, message: str):
 self.warnings.append(message)


def validate_csv(
 filepath: str,
 expected_columns: Optional[list] = None,
 expected_encoding: str = 'utf-8',
 date_columns: Optional[list] = None,
 zero_padded_columns: Optional[list] = None,
 null_convention: str = '',
 max_row_count: Optional[int] = None,
 min_row_count: Optional[int] = None,
) -> CSVValidationResult:
 """Validate a CSV file against common quality checks.

 Works for both producers (checking your own output)
 and consumers (checking vendor deliveries).
 """
 result = CSVValidationResult(filepath=filepath)

 # --- Check 1: File exists and isn't empty ---
 if not os.path.exists(filepath):
 result.add_error(f"File not found: {filepath}")
 return result

 file_size = os.path.getsize(filepath)
 if file_size == 0:
 result.add_error("File is empty (zero bytes)")
 return result
 result.stats['file_size_bytes'] = file_size

 # --- Check 2: Encoding ---
 with open(filepath, 'rb') as f:
 raw_head = f.read(10000)

 # BOM check
 if raw_head[:3] == b'\xef\xbb\xbf':
 result.add_warning(
 "File contains UTF-8 BOM — this will corrupt "
 "the first column name in most parsers"
 )

 # Null byte check
 if b'\x00' in raw_head:
 result.add_error(
 "File contains null bytes — likely wrong encoding "
 "(e.g., UTF-16 read as UTF-8) or binary data"
 )
 return result

 # Encoding validity
 try:
 raw_full = open(filepath, 'rb').read()
 raw_full.decode(expected_encoding)
 except UnicodeDecodeError as e:
 result.add_error(
 f"File is not valid {expected_encoding}. "
 f"Decode error at byte position {e.start}."
 )
 return result

 # --- Check 3: Structural integrity ---
 encoding = 'utf-8-sig' if raw_head[:3] == b'\xef\xbb\xbf' else expected_encoding
 with open(filepath, newline='', encoding=encoding) as f:
 reader = csv.reader(f)

 # Header check
 try:
 header = next(reader)
 except StopIteration:
 result.add_error("File has no rows (not even a header)")
 return result

 result.stats['column_count'] = len(header)
 result.stats['columns'] = header

 # Duplicate column check
 seen = {}
 for col in header:
 seen[col] = seen.get(col, 0) + 1
 dupes = {k: v for k, v in seen.items() if v > 1}
 if dupes:
 result.add_error(f"Duplicate column names: {dupes}")

 # Expected columns check
 if expected_columns and header != expected_columns:
 result.add_error(
 f"Column mismatch.\n"
 f" Expected: {expected_columns}\n"
 f" Got: {header}"
 )

 # Row-level checks
 row_count = 0
 ragged_rows = []
 iso_date = re.compile(r'^\d{4}-\d{2}-\d{2}')

 for row_num, row in enumerate(reader, start=2):
 row_count += 1

 # Column count consistency
 if len(row) != len(header):
 ragged_rows.append((row_num, len(row)))

 # Date format check
 if date_columns:
 for col in date_columns:
 idx = header.index(col) if col in header else None
 if idx is not None and row[idx]:
 if not iso_date.match(row[idx]):
 result.add_warning(
 f"Row {row_num}, column '{col}': "
 f"Non-ISO date '{row[idx]}'"
 )

 result.stats['row_count'] = row_count

 if ragged_rows:
 result.add_error(
 f"{len(ragged_rows)} rows have wrong column count. "
 f"Expected {len(header)} columns. "
 f"First bad row: line {ragged_rows[0][0]} "
 f"has {ragged_rows[0][1]} fields."
 )

 # Row count bounds
 if min_row_count and row_count < min_row_count:
 result.add_error(
 f"Row count {row_count} is below minimum "
 f"threshold of {min_row_count}"
 )
 if max_row_count and row_count > max_row_count:
 result.add_error(
 f"Row count {row_count} exceeds maximum "
 f"threshold of {max_row_count}"
 )

 # --- Check 4: Invisible characters ---
 invisible = {
 '\u00a0': 'Non-breaking space',
 '\u200b': 'Zero-width space',
 '\ufeff': 'BOM (mid-file)',
 }
 text = raw_full.decode(encoding, errors='replace')
 for char, name in invisible.items():
 count = text.count(char)
 if count > 0:
 result.add_warning(
 f"Found {count} '{name}' characters (U+{ord(char):04X})"
 )

 # --- Compute checksum for manifest comparison ---
 sha256 = hashlib.sha256(raw_full).hexdigest()
 result.stats['sha256'] = sha256

 return result

Use it like this:

result = validate_csv(
 'vendor_delivery/customers_20260304.csv',
 expected_columns=['id', 'name', 'email', 'city', 'state', 'zip'],
 date_columns=['created_date', 'updated_date'],
 min_row_count=1000,
 max_row_count=100000,
)

if not result.is_valid:
 for error in result.errors:
 print(f"ERROR: {error}")
 # Route to quarantine
else:
 print(f"Valid. {result.stats['row_count']} rows, "
 f"SHA-256: {result.stats['sha256'][:16]}...")
 # Proceed to loading

What 16,000 Missing Records Taught Us

Let’s come back to where we started. Sixteen thousand COVID test results, dropped silently because nobody tested what would happen when the file exceeded a row limit. The labs did their job. The CSV was generated correctly. The file was transmitted successfully. Every individual component worked. The system failed because nobody tested the boundaries — the places where one component’s assumptions meet another component’s limitations.

That’s what CSV testing is really about. It’s not about whether you can parse a comma-separated file — of course you can. It’s about whether you’ve thought through every place where the producer’s assumptions and the consumer’s assumptions might diverge. The vendor who exports NULL as the literal string “NULL.” The CRM that allows newlines in address fields. The business user who opens your carefully formatted CSV in Excel, saves it, and sends it back with stripped zeros and mangled dates.

Every one of those is a test case. Every one of those is a boundary you can verify during development instead of discovering in production.

The engineers who sleep well at night aren’t the ones who assume their CSV files are fine. They’re the ones who wrote the tests to prove it.

Write-Audit-Publish with Iceberg Tables in Snowflake

Chris Hillman — Fri, 27 Feb 2026 09:00:00 +1100

It was a Tuesday afternoon when the analyst pinged me on Microsoft Teams: “Hey, the Total Portfolio numbers just jumped 40% overnight. Did we land a whale?”

We hadn’t.

What actually happened was more mundane and significantly more painful. A schema change in the source system introduced a currency conversion bug. Our pipeline dutifully loaded the corrupted data into production at 3 AM, the dashboards updated by 6 AM, and the Department Head opened her morning report to numbers that looked like champagne-worthy growth.

Here’s the thing — we had data quality tests. They ran after the data hit production. A classic case of closing the gate after the horse has bolted, kicked over the fence, and trampled the neighbour’s garden.

That incident changed how I think about data pipelines entirely. It led me to the Write-Audit-Publish pattern — a concept championed heavily by Zach Wilson. Zach frames it simply: write to a staging area with an identical schema to production, run your data quality checks, and only publish to production if they pass. It’s deceptively simple. And it works.

I’ve written about WAP before using Airflow as the orchestration layer, with traditional staging tables and swap mechanics. That approach is solid, battle-tested, and it’ll serve most teams well. But now that Apache Iceberg tables in Snowflake have matured — there’s a more elegant way to implement WAP that eliminates one of the pattern’s biggest practical headaches: the I/O overhead.

In this tutorial, we’re going to build a complete WAP test harness from scratch on Snowflake Iceberg tables. We’ll create synthetic data, simulate a change that introduces bad data, catch it with quality gates, and publish only the clean data to production. Along the way, I’ll show you why the zero-copy mechanics of Iceberg’s architecture make the publish step insanely fast — and why that speed changes the calculus on when WAP is worth the effort.

Why the Publish Step Is Where Traditional WAP Hurts

If you’ve worked in data engineering for more than a few months, you’ve probably built some version of WAP without calling it that. You load data into a staging table, run some checks, then copy the good stuff into production. That works. Sort of.

The problem is what happens at the publish step. In a traditional staging-table WAP implementation, publishing means physically moving data. You’re running an INSERT INTO production SELECT * FROM staging, or you’re doing a table swap that involves Snowflake creating new micro-partitions. Either way, you’re generating I/O. You’re writing data that already exists in a perfectly good format, to another location, in another format, just so it lives under the right table name.

For a table with a few million rows, nobody notices. For a table with a few billion rows partitioned across hundreds of files? That publish step becomes a pipeline bottleneck. I’ve seen WAP implementations where the audit takes 30 seconds and the publish takes 45 minutes. The quality gate is fast. The bureaucratic data shuffle is slow.

This is where Iceberg changes the equation.

Iceberg tables don’t store data inside Snowflake’s proprietary micro-partition format. The data lives as Parquet files in your external storage — S3, GCS, or Azure Blob. Snowflake manages the metadata that tells it which Parquet files belong to which snapshot of the table. When you do a CLONE of an Iceberg table in Snowflake, it creates a new metadata pointer to the same underlying Parquet files. No data moves. No files are copied. No I/O is generated.

And when you SWAP a validated clone back into production? Again — it’s a metadata operation. Snowflake updates which metadata pointer the production table name resolves to. The Parquet files don’t move. The snapshots don’t change. The swap is essentially instant regardless of table size.

Let me put that in concrete terms. Say you have a 500 GB Iceberg table with three years of order history. In a traditional WAP setup, publishing validated staging data means writing hundreds of gigabytes to new micro-partitions — that’s real compute cost, real wall-clock time, and a real window where downstream consumers might see partial data. With Iceberg’s zero-copy clone and swap, the same publish step takes seconds. Not minutes. Seconds. Because you’re flipping a pointer, not copying bytes.

This isn’t a marginal improvement. It fundamentally changes when WAP is worth implementing. When publish is expensive, you think twice about adding WAP to every pipeline. When publish is free, the question flips — why wouldn’t you gate every production write with quality checks?

The Conceptual Model: How WAP Works on Iceberg

Here’s the flow we’re going to build. Every step maps to actual SQL you can run in Snowflake today:

The beauty of this model is that production is never exposed to unvalidated data. The clone is a sandbox. Downstream consumers querying fct_orders see the same clean data they saw yesterday, right up until the moment we swap — and the swap is atomic. There’s no window of inconsistency. No partial writes. No “oops, the pipeline failed halfway through the publish.”

Setting Up the Foundation

Let’s build this thing. You’ll need a Snowflake account with access to create Iceberg tables, which means an external volume pointing to your cloud storage and the appropriate privileges.

Prerequisites

-- Create our tutorial workspace
CREATE DATABASE IF NOT EXISTS wap_tutorial;
CREATE SCHEMA IF NOT EXISTS wap_tutorial.iceberg_wap;
USE DATABASE wap_tutorial;
USE SCHEMA iceberg_wap;

-- Create external volume (adjust for your cloud provider)
-- You'll need to configure the IAM trust relationship first
-- See Snowflake docs for the cloud-specific IAM setup
CREATE OR REPLACE EXTERNAL VOLUME wap_ext_vol
 STORAGE_LOCATIONS = (
 (
 NAME = 'wap-iceberg-storage'
 STORAGE_BASE_URL = 's3://your-bucket/iceberg-wap/'
 STORAGE_PROVIDER = 'S3'
 STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/snowflake-iceberg-role'
 )
 );

-- Verify the volume is healthy
DESC EXTERNAL VOLUME wap_ext_vol;

If the external volume setup feels unfamiliar, don’t stress — the Snowflake documentation for this is thorough. The IAM configuration is the fiddly part, but it’s a one-time cost. Once the volume exists, everything downstream is straightforward SQL.

Step 1: WRITE — Creating the Iceberg Table and Synthetic Data

We’re going to simulate an e-commerce order pipeline. This is a scenario practically every data engineer has encountered — daily order data flowing from a transactional system into the warehouse.

First, create the Iceberg table using Snowflake as the catalog:

-- Create the Iceberg table
-- Using Snowflake as the catalog means Snowflake handles
-- all lifecycle maintenance (compaction, metadata updates)
CREATE OR REPLACE ICEBERG TABLE fct_orders (
 order_id INT,
 customer_id INT,
 order_date DATE,
 product_id INT,
 quantity INT,
 unit_price DECIMAL(10,2),
 currency_code VARCHAR(3),
 total_amount DECIMAL(12,2),
 order_status VARCHAR(20),
 region VARCHAR(50),
 loaded_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
)
 CATALOG = 'SNOWFLAKE'
 EXTERNAL_VOLUME = 'wap_ext_vol'
 BASE_LOCATION = 'fct_orders/';

Now let’s generate our initial “production” data. This represents the clean, trusted baseline that’s been accumulating for the past 30 days. Five thousand orders, all in USD, no nulls, no negative amounts — a pristine starting point:

-- Generate 30 days of clean, synthetic order data
-- This is our trusted production baseline
INSERT INTO fct_orders (
 order_id, customer_id, order_date, product_id,
 quantity, unit_price, currency_code, total_amount,
 order_status, region
)
SELECT
 ROW_NUMBER() OVER (ORDER BY SEQ4()) + 10000 AS order_id,
 UNIFORM(1, 500, RANDOM()) AS customer_id,
 DATEADD('day', -UNIFORM(0, 29, RANDOM()), CURRENT_DATE()) AS order_date,
 UNIFORM(1, 50, RANDOM()) AS product_id,
 UNIFORM(1, 10, RANDOM()) AS quantity,
 ROUND(UNIFORM(5.00, 200.00, RANDOM())::DECIMAL(10,2), 2) AS unit_price,
 'USD' AS currency_code,
 -- total_amount = quantity * unit_price (clean, correct calculation)
 ROUND(
 UNIFORM(1, 10, RANDOM()) * UNIFORM(5.00, 200.00, RANDOM())::DECIMAL(12,2),
 2
 ) AS total_amount,
 CASE UNIFORM(1, 5, RANDOM())
 WHEN 1 THEN 'PENDING'
 WHEN 2 THEN 'CONFIRMED'
 WHEN 3 THEN 'SHIPPED'
 WHEN 4 THEN 'DELIVERED'
 ELSE 'CANCELLED'
 END AS order_status,
 CASE UNIFORM(1, 5, RANDOM())
 WHEN 1 THEN 'APAC'
 WHEN 2 THEN 'EMEA'
 WHEN 3 THEN 'AMERICAS'
 WHEN 4 THEN 'LATAM'
 ELSE 'ANZ'
 END AS region
FROM TABLE(GENERATOR(ROWCOUNT => 5000));

Let’s verify our baseline is healthy before we mess it up:

-- Quick sanity check on our production data
SELECT
 COUNT(*) AS total_orders,
 COUNT(DISTINCT customer_id) AS unique_customers,
 MIN(order_date) AS earliest_order,
 MAX(order_date) AS latest_order,
 ROUND(AVG(total_amount), 2) AS avg_order_value,
 COUNT(CASE WHEN total_amount <= 0 THEN 1 END) AS negative_amounts,
 COUNT(CASE WHEN currency_code != 'USD' THEN 1 END) AS non_usd_orders
FROM fct_orders;

You should see 5,000 orders, zero negative amounts, and zero non-USD orders. This is clean production data. Remember these numbers — because we’re about to introduce some chaos.

Simulating the Change: When Bad Data Arrives

Now the realistic part. A new batch of 500 orders arrives from the source system. But this batch has problems — the kind of subtle, infuriating problems that slip past casual inspection but wreck downstream analytics.

First, we create the zero-copy clone. This is our staging sandbox:

-- Create a zero-copy clone of production
-- This is a METADATA operation — no Parquet files are copied
-- The clone shares the same underlying data files as production
CREATE OR REPLACE ICEBERG TABLE fct_orders_staging
 CLONE fct_orders;

That CLONE just happened in under a second regardless of whether your table is 5,000 rows or 500 million. No data moved. The staging clone points to the same Parquet files in your external storage. Snowflake just created a new metadata entry that says “this table currently looks like that table.”

Now let’s load the problematic batch into our staging clone:

-- Simulate incoming data with realistic bugs:
-- 1. Some NULL currency codes (source system migration issue)
-- 2. Some invalid currency codes (mapping error)
-- 3. Some negative total_amounts (conversion bug)
-- 4. Some EUR amounts that weren't actually converted
INSERT INTO fct_orders_staging (
 order_id, customer_id, order_date, product_id,
 quantity, unit_price, currency_code, total_amount,
 order_status, region
)
SELECT
 ROW_NUMBER() OVER (ORDER BY SEQ4()) + 20000 AS order_id,
 UNIFORM(1, 500, RANDOM()) AS customer_id,
 CURRENT_DATE() AS order_date,
 UNIFORM(1, 50, RANDOM()) AS product_id,
 UNIFORM(1, 10, RANDOM()) AS quantity,
 ROUND(UNIFORM(5.00, 200.00, RANDOM())::DECIMAL(10,2), 2) AS unit_price,
 -- HERE'S THE BUG: some orders arrive with wrong currency codes
 CASE
 WHEN UNIFORM(1, 100, RANDOM()) <= 85 THEN 'USD'
 WHEN UNIFORM(1, 100, RANDOM()) <= 5 THEN NULL -- Null currency!
 WHEN UNIFORM(1, 100, RANDOM()) <= 5 THEN 'INVALID' -- Invalid code!
 ELSE 'EUR' -- EUR amount but never converted to USD
 END AS currency_code,
 -- And some totals went negative due to a sign error
 CASE
 WHEN UNIFORM(1, 100, RANDOM()) <= 90
 THEN ROUND(
 UNIFORM(1, 10, RANDOM())
 * UNIFORM(5.00, 200.00, RANDOM())::DECIMAL(12,2), 2)
 ELSE ROUND(
 -1 * UNIFORM(1, 10, RANDOM())
 * UNIFORM(5.00, 200.00, RANDOM())::DECIMAL(12,2), 2)
 END AS total_amount,
 CASE UNIFORM(1, 5, RANDOM())
 WHEN 1 THEN 'PENDING'
 WHEN 2 THEN 'CONFIRMED'
 WHEN 3 THEN 'SHIPPED'
 WHEN 4 THEN 'DELIVERED'
 ELSE 'CANCELLED'
 END AS order_status,
 CASE UNIFORM(1, 5, RANDOM())
 WHEN 1 THEN 'APAC'
 WHEN 2 THEN 'EMEA'
 WHEN 3 THEN 'AMERICAS'
 WHEN 4 THEN 'LATAM'
 ELSE 'ANZ'
 END AS region
FROM
 TABLE(GENERATOR(ROWCOUNT => 500));

The write step is done. We have 500 new orders in our staging clone — some clean, some dirty. And critically, production is completely untouched. Anyone querying fct_orders right now sees exactly what they saw before. No Department Head is going to message you on Teams this morning about phantom Total Portfolio growth.

Step 2: AUDIT — The Quality Gate Battery

This is where WAP earns its keep. We run every check we care about against the staging clone. If anything fails, we either fix it or drop the clone and production never sees the bad data. As Zach puts it — downstream pipelines wait on production data to be ready. Not just present.

Let’s build a comprehensive audit battery. Each check targets a different failure mode:

Check 1: Null Detection on Critical Fields

Nulls in currency or amount fields are deal-breakers. A null currency means we can’t convert the amount. A null amount means we can’t aggregate Total Portfolio numbers. Either one poisons everything downstream.

SELECT
 'NULL_CHECK' AS check_name,
 COUNT(CASE WHEN currency_code IS NULL THEN 1 END) AS null_currency,
 COUNT(CASE WHEN total_amount IS NULL THEN 1 END) AS null_amount,
 COUNT(CASE WHEN order_id IS NULL THEN 1 END) AS null_order_id,
 CASE
 WHEN COUNT(CASE WHEN currency_code IS NULL THEN 1 END) > 0
 OR COUNT(CASE WHEN total_amount IS NULL THEN 1 END) > 0
 OR COUNT(CASE WHEN order_id IS NULL THEN 1 END) > 0
 THEN 'FAIL'
 ELSE 'PASS'
 END AS check_result
FROM
 fct_orders_staging
WHERE
 order_date = CURRENT_DATE();

Check 2: Valid Currency Codes

Only accepted ISO currency codes should appear. Anything else means a mapping went sideways upstream.

SELECT
 'CURRENCY_VALIDATION' AS check_name,
 COUNT(*) AS invalid_currency_count,
 LISTAGG(DISTINCT currency_code, ', ') AS invalid_values,
 CASE
 WHEN COUNT(*) > 0 THEN 'FAIL'
 ELSE 'PASS'
 END AS check_result
FROM
 fct_orders_staging
WHERE
 order_date = CURRENT_DATE()
 AND currency_code NOT IN ('USD', 'EUR', 'GBP', 'AUD', 'CAD', 'JPY')
 AND currency_code IS NOT NULL;

Check 3: No Negative Amounts

A negative total_amount on an order that isn’t a refund is a sign of a calculation bug. This is exactly the kind of error that inflated our Total Portfolio numbers that Tuesday morning.

SELECT
 'NEGATIVE_AMOUNT_CHECK' AS check_name,
 COUNT(*) AS negative_count,
 MIN(total_amount) AS worst_amount,
 CASE
 WHEN COUNT(*) > 0 THEN 'FAIL'
 ELSE 'PASS'
 END AS check_result
FROM
 fct_orders_staging
WHERE
 order_date = CURRENT_DATE()
 AND total_amount < 0;

Check 4: Duplicate Detection

Duplicate order IDs mean the source system sent the same batch twice, or our pipeline ran a retry without proper idempotency.

SELECT
 'DUPLICATE_CHECK' AS check_name,
 COUNT(*) - COUNT(DISTINCT order_id) AS duplicate_count,
 CASE
 WHEN COUNT(*) - COUNT(DISTINCT order_id) > 0 THEN 'FAIL'
 ELSE 'PASS'
 END AS check_result
FROM
 fct_orders_staging
WHERE
 order_date = CURRENT_DATE();

Check 5: Volume Anomaly Detection

If today’s batch is wildly larger or smaller than the historical daily average, something structural has changed. Maybe a filter broke, maybe a new source came online, maybe someone’s test data leaked into production.

WITH daily_volumes AS (
 SELECT
 order_date,
 COUNT(*) AS daily_count
 FROM
 fct_orders_staging
 WHERE
 order_date < CURRENT_DATE()
 GROUP BY
 order_date
),
today_volume AS
(
 SELECT COUNT(*) AS today_count
 FROM
 fct_orders_staging
 WHERE
 order_date = CURRENT_DATE()
)
SELECT
 'VOLUME_ANOMALY' AS check_name,
 t.today_count,
 ROUND(AVG(d.daily_count), 0) AS avg_daily_count,
 CASE
 WHEN t.today_count > AVG(d.daily_count) * 3 THEN 'FAIL'
 WHEN t.today_count < AVG(d.daily_count) * 0.1 THEN 'FAIL'
 ELSE 'PASS'
 END AS check_result
FROM
 daily_volumes d

 CROSS JOIN today_volume t
GROUP BY
 t.today_count;

Aggregating the Verdict

In a real pipeline, you’d wrap these checks in a stored procedure or Snowflake Task. Here’s the aggregation logic that gives you a single pass/fail decision:

-- Aggregate all quality checks into a single verdict
CREATE OR REPLACE TEMPORARY TABLE audit_results AS
WITH checks AS (
 -- Null check
 SELECT 'NULL_CHECK' AS check_name,
 CASE WHEN COUNT(CASE WHEN currency_code IS NULL
 OR total_amount IS NULL THEN 1 END) > 0
 THEN 'FAIL' ELSE 'PASS' END AS result
 FROM fct_orders_staging
 WHERE order_date = CURRENT_DATE()

 UNION ALL

 -- Negative amounts
 SELECT 'NEGATIVE_AMOUNTS',
 CASE WHEN COUNT(*) > 0 THEN 'FAIL' ELSE 'PASS' END
 FROM fct_orders_staging
 WHERE order_date = CURRENT_DATE() AND total_amount < 0

 UNION ALL

 -- Invalid currency
 SELECT 'INVALID_CURRENCY',
 CASE WHEN COUNT(*) > 0 THEN 'FAIL' ELSE 'PASS' END
 FROM fct_orders_staging
 WHERE order_date = CURRENT_DATE()
 AND (currency_code NOT IN ('USD','EUR','GBP','AUD','CAD','JPY')
 OR currency_code IS NULL)

 UNION ALL

 -- Duplicates
 SELECT 'DUPLICATES',
 CASE WHEN COUNT(*) - COUNT(DISTINCT order_id) > 0
 THEN 'FAIL' ELSE 'PASS' END
 FROM fct_orders_staging
 WHERE order_date = CURRENT_DATE()
)
SELECT * FROM checks;

-- See the full verdict
SELECT * FROM audit_results;

-- Check if we're clear to publish
SELECT
 COUNT(*) AS total_checks,
 SUM(CASE WHEN result = 'PASS' THEN 1 ELSE 0 END) AS passed,
 SUM(CASE WHEN result = 'FAIL' THEN 1 ELSE 0 END) AS failed,
 CASE
 WHEN SUM(CASE WHEN result = 'FAIL' THEN 1 ELSE 0 END) = 0
 THEN 'CLEAR TO PUBLISH'
 ELSE 'BLOCKED — FIX REQUIRED'
 END AS verdict
FROM audit_results;

With our deliberately corrupted data, you’ll see multiple failures here. Null currencies, invalid codes, negative amounts — exactly the problems we planted. The audit caught everything. Production is still clean.

Step 3: PUBLISH — The Happy Path (When Audits Pass)

Let’s first walk through what happens when everything is clean. This is the path your pipeline will take 95% of the time — and it’s where the Iceberg zero-copy advantage really shines.

-- PUBLISH: Swap the validated clone into production
-- This is a METADATA OPERATION — no Parquet files are copied or moved
-- The swap is ATOMIC — downstream sees old data or new data, never partial
ALTER ICEBERG TABLE fct_orders
 SWAP WITH fct_orders_staging;

-- Clean up — the old production reference is now the staging table name
DROP TABLE IF EXISTS fct_orders_staging;

That’s it. Two statements. The SWAP updated which metadata pointer the table name fct_orders resolves to. The Parquet files sitting in your S3 bucket didn’t budge. No new files were written. No compute was burned copying data. No warehouse credits were consumed on I/O.

Think about what this means for pipeline design. In a traditional WAP implementation with standard Snowflake tables, the publish step is the expensive one — it’s the INSERT INTO ... SELECT * or the CREATE TABLE AS SELECT that physically writes data. That cost scales linearly with data volume. Double the table size, double the publish time and cost.

With Iceberg’s zero-copy swap, publish cost is constant regardless of table size. A 10 MB table and a 10 TB table swap in the same amount of time. That’s not a rounding error. That’s a fundamental shift in the economics of data quality enforcement.

For context, I timed this on a table with ~200 million rows across several hundred Parquet files. The clone took about a second. The audit checks took about 40 seconds. The swap? Sub-second. The most expensive part of the entire WAP pipeline was the actual data quality logic — which is exactly where you want the time to be spent.

Step 3 (Alternate): QUARANTINE — When Audits Fail

Now the realistic scenario. Our audit found problems. We have options.

Option A: Drop everything, investigate, retry. This is the nuclear option. Production stays on yesterday’s data until the source is fixed and a clean batch arrives.

-- Nuclear option — reject the entire batch
DROP TABLE IF EXISTS fct_orders_staging;
-- Production is exactly as it was. Nothing changed.

Option B: Quarantine the bad records, publish the clean ones. This is more practical for most teams. You log the failures for investigation, surgically remove the bad data, re-audit, and publish what’s clean.

-- First, capture the failures for investigation
-- This gives your team evidence to fix the source system
CREATE OR REPLACE TABLE quarantine_log AS
SELECT
 *,
 CURRENT_TIMESTAMP() AS quarantined_at,
 CASE
 WHEN currency_code IS NULL THEN 'NULL_CURRENCY'
 WHEN currency_code NOT IN ('USD','EUR','GBP','AUD','CAD','JPY')
 THEN 'INVALID_CURRENCY'
 WHEN total_amount < 0 THEN 'NEGATIVE_AMOUNT'
 END AS failure_reason
FROM
 fct_orders_staging
WHERE
 order_date = CURRENT_DATE()
 AND (
 currency_code IS NULL
 OR currency_code NOT IN ('USD','EUR','GBP','AUD','CAD','JPY')
 OR total_amount < 0
 );

-- See what we caught
SELECT failure_reason, COUNT(*) AS record_count
FROM quarantine_log
GROUP BY failure_reason
ORDER BY record_count DESC;

Now remove the bad records from the staging clone and re-validate:

-- Surgically remove the bad records
DELETE FROM fct_orders_staging
WHERE order_date = CURRENT_DATE()
 AND (
 currency_code IS NULL
 OR currency_code NOT IN ('USD','EUR','GBP','AUD','CAD','JPY')
 OR total_amount < 0
 );

-- Re-run the audit against the cleaned staging data
-- (Repeat the audit_results query from Step 2)
-- If it passes now, publish the clean subset
ALTER ICEBERG TABLE fct_orders
 SWAP WITH fct_orders_staging;

DROP TABLE IF EXISTS fct_orders_staging;

The quarantine log is important. It’s not enough to just filter out bad data — you need to understand why it was bad so you can fix the source. Every quarantined record is a symptom. The disease is upstream.

Putting It All Together: The Orchestrated Pipeline

In practice, you’d wrap this entire flow in a Snowflake Task or an external orchestrator. Here’s a stored procedure that encapsulates the full WAP pattern. This is something you can adapt and parameterise for any Iceberg table in your warehouse:

CREATE OR REPLACE PROCEDURE run_wap_pipeline(
 target_table VARCHAR,
 staging_table VARCHAR
)
RETURNS VARCHAR
LANGUAGE SQL
AS
$$
DECLARE
 fail_count INT;
 result_msg VARCHAR;
BEGIN
 -- WRITE: Clone production
 EXECUTE IMMEDIATE 'CREATE OR REPLACE ICEBERG TABLE '
 || staging_table || ' CLONE ' || target_table;

 -- (Your data loading logic goes here)
 -- INSERT INTO staging_table SELECT ... FROM source

 -- AUDIT: Run quality checks
 SELECT COUNT(*) INTO fail_count
 FROM (
 -- Null check
 SELECT 1 WHERE EXISTS (
 SELECT 1 FROM IDENTIFIER(:staging_table)
 WHERE order_date = CURRENT_DATE()
 AND (currency_code IS NULL OR total_amount IS NULL)
 )
 UNION ALL
 -- Negative amount check
 SELECT 1 WHERE EXISTS (
 SELECT 1 FROM IDENTIFIER(:staging_table)
 WHERE order_date = CURRENT_DATE()
 AND total_amount < 0
 )
 UNION ALL
 -- Invalid currency check
 SELECT 1 WHERE EXISTS (
 SELECT 1 FROM IDENTIFIER(:staging_table)
 WHERE order_date = CURRENT_DATE()
 AND currency_code NOT IN ('USD','EUR','GBP','AUD','CAD','JPY')
 )
 );

 IF (fail_count = 0) THEN
 -- PUBLISH: Swap (zero-copy, instant)
 EXECUTE IMMEDIATE 'ALTER ICEBERG TABLE '
 || target_table || ' SWAP WITH ' || staging_table;
 EXECUTE IMMEDIATE 'DROP TABLE IF EXISTS ' || staging_table;
 result_msg := 'PUBLISHED — All checks passed. Swap completed.';
 ELSE
 -- ROLLBACK: Drop the clone
 EXECUTE IMMEDIATE 'DROP TABLE IF EXISTS ' || staging_table;
 result_msg := 'BLOCKED — ' || fail_count
 || ' check(s) failed. Production unchanged.';
 END IF;

 RETURN result_msg;
END;
$$;

-- Run it
CALL run_wap_pipeline('fct_orders', 'fct_orders_staging');

A Note on Iceberg Branching — The Future Path

If you’ve been following the Iceberg ecosystem, you might be wondering about branching. Apache Iceberg’s spec includes a branching feature that’s purpose-built for WAP — you create a named branch within the same table, write to it in isolation, and fast-forward the main branch when audits pass. It’s like a Git pull request for your data.

The feature exists and it works beautifully in Spark and Flink. You can create branches with ALTER TABLE ... CREATE BRANCH, write to them via the spark.wap.branch session config, and publish with CALL system.fast_forward(). The semantics are cleaner than the clone-and-swap approach because the branch is the table — just a different lineage of snapshots within the same metadata.

As of writing this it appears Snowflake does not natively support Iceberg branching through its SQL engine today. The branching and tagging metadata operations are Iceberg spec features implemented in the Java library, with engine integrations for Spark and Flink. Snowflake’s Iceberg support gives you full DML, Time Travel, and schema evolution — but not yet the CREATE BRANCH or fast_forward operations.

For teams already running Spark workloads alongside Snowflake, the branching approach is worth exploring now. You run the WAP pipeline in Spark with native Iceberg branching, and Snowflake reads the published data as an externally managed Iceberg table. It’s a composable architecture where you push quality enforcement to cheaper compute and keep the warehouse for analytics.

For teams that are Snowflake-native, the clone-and-swap approach we’ve built in this tutorial gives you the vast majority of the benefits today. When Snowflake adds native branching support — and given the competitive landscape with Databricks, I’d expect it — we can revisit this with a cleaner implementation. The quality checks and audit battery you build now will carry forward regardless of the underlying publish mechanism.

Best Practices That’ll Save You Pain

After running WAP patterns in various forms across multiple teams, these are the lessons that kept surfacing.

Start with hard gates, evolve to soft gates. Initially, any check failure should block publication. Full stop. As your quality checks mature and you understand your data better, you’ll learn which checks are hard blockers — null primary keys, negative amounts, schema violations — and which are warnings worth logging but not blocking on — volume slightly above the historical average, a handful of records with an unusual but valid status. Build your audit results as structured data with severity levels so the publish decision can be nuanced.

Log quarantined records, always. Even when you drop a staging clone, capture the failing records somewhere for investigation. A quarantine table or a dead-letter S3 prefix gives your team the evidence they need to fix the source system — which is the real solution, not just filtering out bad data forever. Every quarantined record should answer three questions: what failed, which check caught it, and when it arrived.

Make your quality checks idempotent. The audit battery should be safe to run multiple times without side effects. No state mutations, no “mark as audited” flags that prevent re-runs. If the pipeline fails halfway through auditing, you should be able to restart from the top of the audit phase without creating a new clone.

Monitor your audit pass rates over time. A pipeline that fails WAP checks once a month is healthy — the pattern is catching occasional source issues. A pipeline that fails WAP checks every other day is telling you something about your upstream data source that needs fixing. Track the pass/fail ratio, the specific checks that fail most often, and the volume of quarantined records. This data is ammunition for the conversation with the source system owners.

Don’t forget the cleanup. Snowflake’s CLONE creates metadata that references the same underlying Parquet files. If you create clones and forget to drop them, you’re not duplicating storage (the Parquet files are shared), but you are accumulating metadata that Snowflake tracks. Expired clones should be dropped promptly. Set up a cleanup task that sweeps for orphaned staging tables older than your retention threshold.

Tag your successful publishes. Not Iceberg tags (Snowflake doesn’t support those natively yet), but a metadata log. Keep a table that records every successful WAP publish — the timestamp, the table name, the row count of the new batch, and the audit results. When something goes wrong downstream three weeks from now and someone asks “what changed?”, that log is your first stop.

The Bigger Picture: Data Quality as a Deploy Gate

Zach Wilson has a line that sticks with me: “Data engineering shouldn’t have any less engineering rigor than software engineering.” He’s right. We’ve spent years building CI/CD pipelines for application code — linting, unit tests, integration tests, staging environments, approval gates — and then we load data straight into production with a prayer and a Teams message that says “run looks good.”

WAP is the data equivalent of a deployment gate. You wouldn’t merge a pull request that fails tests. You shouldn’t publish data that fails quality checks. The mechanics are different — we’re swapping table metadata instead of deploying container images — but the principle is identical: validate before you publish, and make the publish step as cheap and reversible as possible.

Iceberg tables make the “as cheap as possible” part genuinely achievable. When publish is a zero-copy metadata operation that completes in under a second regardless of table size, the overhead of WAP effectively disappears. The only cost is the compute for your quality checks — and that’s compute you want to be spending.

That Tuesday afternoon with the phantom Total Portfolio growth? It cost us three days of investigation, two days of reprocessing historical data, and an unknowable amount of credibility with the Department Head and her team. Implementing WAP on that pipeline would have cost us nothing but a few lines of SQL and a staging clone that gets created and destroyed automatically every morning.

The pattern works. The tooling is ready. Start where you are, build the quality gates, and let the zero-copy publish mechanics handle the rest.

Your production data — and your Department Head — will thank you for it.

Healing Tables: When Day-by-Day Backfills Become a Slow-Motion Disaster

Chris Hillman — Sat, 07 Feb 2026 09:00:00 +1100

It was 2 AM on a Saturday when I realized we’d been loading data wrong for six months.

The situation: a customer dimension with three years of history needed to be backfilled after a source system migration. The previous team’s approach was straightforward—run the daily incremental process 1,095 times, once for each day of history. They estimated three weeks to complete.

What they hadn’t accounted for was how errors compound. By the time I looked at the data, we had 47,000 records with overlapping date ranges, 12,000 timeline gaps where customers seemed to vanish and reappear, and an unknowable number of missed changes from when source systems updated the same record multiple times in a single day.

The dimension wasn’t just wrong. It was unfixably wrong using traditional methods. Every incremental run had layered new errors on top of old ones, creating a Jenga tower of data quality issues that couldn’t be untangled without starting over.

That night, I started building what I now call a Healing Table—a dimension that can be completely rebuilt from source data at any point, “healing” whatever accumulated inconsistencies have crept in over months or years of incremental loads.

This isn’t just a technique for disaster recovery. It’s a fundamentally different approach to SCD Type 2 that separates change detection from period construction, processes entire historical ranges in a single pass, and produces dimensions that are deterministically reproducible from source data.

Here’s how it works.

Why Day-by-Day Backfills Fail

The traditional approach to backfilling historical SCD2 dimensions follows a pattern most data engineers know well: take your daily incremental process, wrap it in a loop, and execute it once for each historical date. Simple, reuses existing code, and catastrophically fragile.

The problems start immediately.

Errors compound across iterations. If your change detection logic misses an edge case on January 3rd, that incorrect record becomes the baseline for January 4th’s comparison. The error doesn’t stay contained—it propagates forward through every subsequent run. By the time you discover the issue months later, you can’t simply fix January 3rd because hundreds of downstream records depend on that incorrect state.

Performance degrades non-linearly. Each daily run needs to compare incoming records against the existing dimension. As the dimension grows through backfill, comparison costs increase. A process that takes 5 minutes for a single day’s delta might take 45 minutes per day when the target table contains three years of history. That three-week estimate becomes three months.

Source system quirks multiply. Real source systems don’t change data once per day at a predictable time. They update records multiple times, delete and recreate rows, and occasionally backdate changes. Day-by-day processing either misses these patterns entirely or handles them inconsistently across runs.

Recovery requires complete rebuild anyway. When—not if—something goes wrong, fixing it requires blowing away the dimension and starting over. But if you’re going to rebuild from scratch regardless, why not design for that from the beginning?

The Healing Tables framework embraces this reality. Instead of trying to prevent rebuilds, it makes them fast, reliable, and deterministic.

The Six-Step Framework

Healing Tables work by separating two concerns that traditional SCD2 implementations conflate: change detection (identifying when attributes changed) and period construction (building valid time slices with proper start and end dates).

Traditional approaches detect changes and construct periods simultaneously, comparing incoming records to existing dimension state. This creates tight coupling between current dimension contents and processing logic—if the dimension is wrong, future processing will also be wrong.

Healing Tables decouple these concerns through a six-step pipeline that operates entirely on source data, constructing the dimension from scratch without reference to any existing target state:

Effectivity Table Creation — Extract all change points from sources
Time Slice Generation — Build date ranges with proper valid_from/valid_to
Source Table Joining — Conform attributes from multiple sources
Hash Computation — Enable efficient change detection
Row Compression — Eliminate consecutive identical states
Validation Testing — Verify temporal integrity before loading

Let me walk through each step.

Step 1: Building the Effectivity Table

The Effectivity Table captures every moment when any tracked attribute changed for any business key. Think of it as a timeline of “something happened here” markers that we’ll later fill in with actual attribute values.

For sources with explicit timestamps, extraction looks like this:

-- Extract all change points from source
WITH source_ordered AS (
 SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,credit_limit
 ,updated_at AS change_timestamp
 ,ROW_NUMBER() OVER (
 PARTITION BY customer_id
 ORDER BY updated_at
 ) AS version_sequence
 FROM
 raw_customers
 WHERE
 updated_at IS NOT NULL
),

-- Detect actual changes using LAG comparison
change_detection AS (
 SELECT
 *
 ,LAG(customer_name) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ) AS prev_name
 ,LAG(customer_status) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ) AS prev_status
 ,LAG(credit_limit) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ) AS prev_credit_limit
 FROM
 source_ordered
)

-- Keep only first record OR records where attributes actually changed
SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,credit_limit
 ,change_timestamp
FROM
 change_detection
WHERE
 version_sequence = 1
 OR customer_name IS DISTINCT FROM prev_name
 OR customer_status IS DISTINCT FROM prev_status
 OR credit_limit IS DISTINCT FROM prev_credit_limit

The critical insight here is filtering to actual changes only. Raw source tables often contain records where nothing meaningful changed—perhaps a batch process touched every row, or a status field was updated and then immediately reverted. Including these non-changes creates unnecessary dimension versions that complicate queries and waste storage.

When working with daily snapshot sources (where you don’t have change timestamps, just daily extracts), the pattern shifts to detecting differences between consecutive snapshots:

WITH daily_ordered AS (
 SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,snapshot_date
 ,LAG(customer_name) OVER (
 PARTITION BY customer_id
 ORDER BY snapshot_date
 ) AS prev_name
 ,LAG(customer_status) OVER (
 PARTITION BY customer_id
 ORDER BY snapshot_date
 ) AS prev_status
 FROM
 daily_customer_snapshots
)

SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,snapshot_date AS effective_from
FROM
 daily_ordered
WHERE
 prev_name IS NULL -- First occurrence
 OR customer_name IS DISTINCT FROM prev_name
 OR customer_status IS DISTINCT FROM prev_status

The IS DISTINCT FROM operator handles NULL comparisons correctly—a common source of bugs in change detection logic. Standard != treats NULL as unknown, meaning NULL != 'value' returns NULL rather than TRUE. IS DISTINCT FROM treats NULL as a concrete value, giving us the comparison semantics we actually want.

Step 2: Generating Time Slices

Time Slices transform point-in-time change events into date ranges with valid_from and valid_to columns. The LEAD window function does the heavy lifting, calculating when each version expires based on when the next version begins:

WITH effectivity_data AS (
 SELECT
 customer_id, customer_name, customer_status, change_timestamp
 FROM
 effectivity_table
)

SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,change_timestamp AS valid_from
 -- Calculate valid_to from next version's start date
 ,COALESCE(
 LEAD(change_timestamp) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ),
 DATE '9999-12-31' -- High date for current records
 ) AS valid_to
 -- Flag current record for easy filtering
 ,CASE
 WHEN LEAD(change_timestamp) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ) IS NULL THEN TRUE
 ELSE FALSE
 END AS is_current
FROM
 effectivity_data

A few implementation decisions deserve attention here.

High dates versus NULLs for current records. Using 9999-12-31 for current record end dates simplifies BETWEEN queries but can confuse some BI tools that try to calculate date differences. Using NULL for valid_to has clearer semantic meaning but requires IS NULL predicates everywhere. My recommendation: use the high date for the column value, but also include an explicit is_current boolean flag. This gives query authors flexibility without forcing them to remember magic date values.

Interval conventions matter. Use left-closed, right-open intervals where valid_from <= date < valid_to. This ensures every point in time maps to exactly one dimension version without ambiguity at boundaries. The alternative—inclusive on both ends—creates situations where midnight timestamps belong to two periods simultaneously.

Point-in-time lookup pattern:

SELECT d.*
FROM
 dim_customer d
WHERE
 @lookup_date >= d.valid_from
 AND @lookup_date < d.valid_to

Step 3: Joining Multiple Sources

When dimension attributes come from multiple systems, you need to conform those sources before SCD2 processing begins. The temptation is to load directly from multiple sources into a single target—don’t do this. You’ll create conflicting versions where the same timestamp has different attribute combinations depending on which source processed first.

The solution is a Unified Timeline that collects all change points from all sources, then joins each source back using as-of-date logic:

-- Step 1: Collect ALL change points from ALL sources
WITH unified_change_points AS (
 SELECT customer_id, change_timestamp FROM crm_customers
 UNION
 SELECT customer_id, change_timestamp FROM erp_customers
 UNION
 SELECT customer_id, change_timestamp FROM web_profiles
),

-- Step 2: Build time slices from unified timeline
unified_time_slices AS (
 SELECT
 customer_id
 ,change_timestamp AS valid_from
 ,COALESCE(
 LEAD(change_timestamp) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ),
 TIMESTAMP '9999-12-31 23:59:59'
 ) AS valid_to
 FROM
 unified_change_points
),

-- Step 3: Join each source back using as-of logic
final_dimension AS (
 SELECT
 ts.customer_id
 ,ts.valid_from
 ,ts.valid_to
 -- CRM attributes (authoritative for name)
 ,crm.customer_name
 -- ERP attributes (authoritative for financial data)
 ,erp.credit_limit
 ,erp.payment_terms
 -- Web attributes (authoritative for contact preferences)
 ,web.email_opt_in
 ,web.preferred_contact_method
 FROM
 unified_time_slices ts

 LEFT JOIN crm_customers crm
 ON ts.customer_id = crm.customer_id
 AND crm.change_timestamp = (
 SELECT MAX(change_timestamp)
 FROM crm_customers
 WHERE customer_id = ts.customer_id
 AND change_timestamp <= ts.valid_from
 )

 LEFT JOIN erp_customers erp
 ON ts.customer_id = erp.customer_id
 AND erp.change_timestamp = (
 SELECT MAX(change_timestamp)
 FROM erp_customers
 WHERE customer_id = ts.customer_id
 AND change_timestamp <= ts.valid_from
 )

 LEFT JOIN web_profiles web
 ON ts.customer_id = web.customer_id
 AND web.change_timestamp = (
 SELECT MAX(change_timestamp)
 FROM web_profiles
 WHERE customer_id = ts.customer_id
 AND change_timestamp <= ts.valid_from
 )
)

SELECT * FROM final_dimension

Notice the correlated subqueries for each join—we’re finding the most recent version from each source that was effective at each unified timeline point. This handles cases where one source updates frequently while another updates rarely.

Document attribute ownership explicitly. When CRM and ERP both have a customer name field, which wins? When web profile email conflicts with CRM email, which is authoritative? These decisions belong in documentation and code comments, not in tribal knowledge. Build a table:

Attribute	Primary Source	Fallback	Resolution Rule
Customer Name	CRM	ERP	CRM always wins
Credit Limit	ERP	None	ERP authoritative
Email	Web	CRM	Most recent wins

Then implement those rules explicitly in your SQL using COALESCE or CASE expressions.

Step 4: Hash Computation

Hash-based change detection replaces expensive multi-column comparisons with single-column hash comparisons. The Healing Tables framework uses a two-hash strategy: one hash for business key identification, another for attribute change detection.

key_hash identifies the business entity. Hash the natural key columns together:

-- Trino/Starburst syntax
lower(to_hex(sha256(to_utf8(
 COALESCE(CAST(source_system AS VARCHAR), '') || '|' ||
 COALESCE(CAST(customer_id AS VARCHAR), '')
)))) AS key_hash

row_hash detects when any tracked attribute changes. Hash all business attributes excluding keys and metadata:

lower(to_hex(sha256(to_utf8(
 COALESCE(customer_name, '^^NULL^^') || '|' ||
 COALESCE(customer_status, '^^NULL^^') || '|' ||
 COALESCE(CAST(credit_limit AS VARCHAR), '^^NULL^^') || '|' ||
 COALESCE(CAST(birth_date AS VARCHAR), '^^NULL^^')
)))) AS row_hash

The ^^NULL^^ placeholder distinguishes NULL values from empty strings—a critical detail that trips up many implementations. Without this, ('John', NULL, 'Active') and ('John', '', 'Active') produce identical hashes despite having semantically different data.

Why not use the built-in CONCAT_WS? Because CONCAT_WS skips NULL values entirely. The inputs ('A', NULL, 'B') and ('A', 'B') produce identical output, creating false positive matches where none exist. Always wrap columns in COALESCE before concatenation.

Column ordering best practice: concatenate in alphabetical order. This creates deterministic, reproducible hashes that survive schema evolution—adding a new column doesn’t change the hash of existing columns.

Platform-specific hash functions vary:

Platform	Function	Syntax
Snowflake	SHA2	`SHA2(expression, 256)`
BigQuery	SHA256 + TO_HEX	`TO_HEX(SHA256(expression))`
Trino/Starburst	sha256 + to_hex	`to_hex(sha256(to_utf8(expression)))`
SQL Server	HASHBYTES	`HASHBYTES('SHA2_256', expression)`

Step 5: Row Compression

Row compression eliminates consecutive duplicate states—adjacent time periods where all attributes are identical. This happens more often than you’d expect: source systems frequently touch records without changing meaningful data, or changes get reverted within the same processing window.

The technique is a classic Islands and Gaps pattern. Assign a group identifier to runs of consecutive identical rows, then collapse each group into a single time slice:

-- Step 1: Detect group boundaries
WITH boundary_detection AS (
 SELECT
 *
 ,CASE
 -- New group if: periods don't connect OR attributes changed
 WHEN LAG(valid_to) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) = valid_from
 AND row_hash = LAG(row_hash) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 )
 THEN 0 -- Continuation of previous group
 ELSE 1 -- Start of new group
 END AS is_group_start
 FROM
 dimension_with_hashes
),

-- Step 2: Assign group IDs via running sum
group_assignment AS (
 SELECT
 *
 ,SUM(is_group_start) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ROWS UNBOUNDED PRECEDING
 ) AS group_id
 FROM
 boundary_detection
)

-- Step 3: Collapse each group to single row
SELECT
 customer_id
 ,MIN(valid_from) AS valid_from
 ,MAX(valid_to) AS valid_to
 ,MAX(CASE WHEN is_current THEN 1 ELSE 0 END) = 1 AS is_current
 ,MAX(customer_name) AS customer_name -- All identical within group
 ,MAX(customer_status) AS customer_status
 ,MAX(row_hash) AS row_hash
 ,MAX(key_hash) AS key_hash
 ,COUNT(*) AS compressed_row_count -- Audit trail
FROM
 group_assignment
GROUP BY
 customer_id, group_id
ORDER BY
 customer_id, valid_from

Two conditions must both be true for rows to compress:

Temporal contiguity: Previous row’s valid_to equals current row’s valid_from
Attribute identity: row_hash values match

The contiguity check prevents incorrectly merging records that happen to have identical attributes but represent separate valid periods. If a customer had credit limit 10000 from January to March, then credit limit 5000 from March to June, then credit limit 10000 again from June to December—those are three distinct business states, not one, even though the first and third have identical attribute values.

Before compression (6 rows):

customer_id	valid_from	valid_to	credit_limit
1002	2020-01-01	2020-01-05	40000
1002	2020-01-05	2020-01-09	40000
1002	2020-01-09	2020-01-11	30000
1002	2022-01-01	2022-03-01	30000
1002	2022-03-01	2022-05-01	30000
1002	2022-05-01	2022-06-01	30000

After compression (3 rows):

customer_id	valid_from	valid_to	credit_limit
1002	2020-01-01	2020-01-09	40000
1002	2020-01-09	2020-01-11	30000
1002	2022-01-01	2022-06-01	30000

The gap between 2020-01-11 and 2022-01-01 correctly prevents merging the two “30000” groups—they represent distinct validity periods separated by time when the customer didn’t exist in source data.

Step 6: Validation Testing

Production SCD2 dimensions require comprehensive validation to ensure temporal integrity. These tests should run after every load and block pipeline completion on failure. Silent failures in dimension loading corrupt downstream fact tables in ways that are extremely difficult to diagnose later.

Test 1: Single current record per key

Each business key must have exactly one active (current) record:

-- CRITICAL: Fails if any key has multiple current records
SELECT
 customer_id
 ,COUNT(*) AS current_record_count
FROM
 dim_customer
WHERE
 is_current = TRUE
GROUP BY
 customer_id
HAVING
 COUNT(*) <> 1

-- Expected: Zero rows returned

Test 2: No overlapping date ranges

Time periods for the same key must not overlap. This is where I learned a painful lesson: the obvious LEAD-based check is insufficient.

-- INSUFFICIENT - Only checks adjacent records
WITH overlap_check AS (
 SELECT
 customer_id
 ,valid_from
 ,valid_to
 ,LEAD(valid_from) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) AS next_valid_from
 FROM
 dim_customer
)
SELECT *
FROM
 overlap_check
WHERE
 next_valid_from <= valid_to

This misses overlaps between non-adjacent records. Records at positions 1 and 3 could overlap even if record 2 doesn’t overlap with either. The comprehensive check requires a self-join:

-- COMPREHENSIVE - Checks all pairs
SELECT a.*, b.*
FROM
 dim_customer a

 INNER JOIN dim_customer b
 ON a.customer_id = b.customer_id
 AND a.dbt_scd_id != b.dbt_scd_id -- Different records
WHERE
 a.valid_from < b.valid_to
 AND a.valid_to > b.valid_from

-- Expected: Zero rows returned

Test 3: No timeline gaps

For dimensions requiring continuous history, adjacent periods must connect:

WITH gap_check AS (
 SELECT
 customer_id
 ,valid_from
 ,LAG(valid_to) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) AS prev_valid_to
 FROM
 dim_customer
)
SELECT
 customer_id
 ,prev_valid_to AS gap_start
 ,valid_from AS gap_end
FROM
 gap_check
WHERE
 prev_valid_to IS NOT NULL
 AND
 prev_valid_to <> valid_from

-- Expected: Zero rows (or documented acceptable gaps)

Test 4: Valid date ordering

Every record must have valid_from < valid_to:

SELECT *
FROM
 dim_customer
WHERE
 valid_from >= valid_to

-- Expected: Zero rows

Test 5: No consecutive duplicate versions

After compression, no adjacent rows should have identical attributes:

WITH version_check AS (
 SELECT
 customer_id
 ,valid_from
 ,row_hash
 ,LAG(row_hash) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) AS prev_row_hash
 ,LAG(valid_to) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) AS prev_valid_to
 FROM
 dim_customer
)
SELECT *
FROM
 version_check
WHERE
 row_hash = prev_row_hash
 AND valid_from = prev_valid_to -- Contiguous AND identical

-- Expected: Zero rows

In dbt, implement these as reusable tests:

# schema.yml
models:
 - name: dim_customer
 tests:
 - dbt_utils.unique_combination_of_columns:
 combination_of_columns:
 - customer_id
 - valid_from
 columns:
 - name: customer_id
 tests:
 - unique:
 config:
 where: "is_current = TRUE"
 - name: valid_from
 tests:
 - not_null
 - name: valid_to
 tests:
 - not_null

The Self-Healing Property

The name “Healing Tables” comes from a property that emerges from this design: dimensions built this way can recover from accumulated data quality issues simply by reprocessing source data.

Traditional incremental SCD2 implementations are path-dependent—the current state depends on the exact sequence of historical loads. If Tuesday’s load had a bug, and Wednesday through Friday ran correctly, you can’t just re-run Tuesday. Those later loads built on Tuesday’s incorrect state.

Healing Tables are path-independent. The dimension produced depends only on source data contents, not on the history of load operations. Re-run any date range with corrected logic, and the result is identical to what you’d get from a full rebuild.

This property transforms how teams think about data quality issues:

Bug in change detection logic? Fix it, reprocess affected date range, done.
Source system correction applied retroactively? Reprocess includes those corrections automatically.
New attribute added to dimension? Reprocess rebuilds all historical versions with the new column.
Suspected data corruption? Rebuild and compare—if results differ, you’ve found your problem.

The dimension becomes reproducible in the same way good software builds are reproducible. Given the same inputs and logic, you always get the same outputs. This isn’t just convenient for debugging—it’s a fundamental quality characteristic that enables reliable data pipelines.

When to Use This Approach

Healing Tables aren’t appropriate for every situation. They work best when:

You have access to complete source history. The pattern assumes you can extract all historical changes, not just deltas. If your source system only provides “current state” with no history, you’ll need a different approach.
Source data is contiguous. The timeline construction assumes no gaps in source timestamps. If customers can appear, disappear, and reappear in source data, you’ll need additional logic to handle these discontinuities.
Rebuild time is acceptable. Processing years of history takes longer than processing a single day’s changes. For dimensions with billions of historical rows, full rebuilds might be impractical even with optimization.
Your sources don’t have hard deletes. The pattern assumes deletions are logical (flagged) rather than physical. If source systems physically remove rows, you’ll lose that history.

Traditional incremental approaches remain appropriate for:

Very high-volume dimensions where full rebuilds are impractical
Real-time or near-real-time requirements
Situations where source history is unavailable
Simple dimensions with stable, well-understood source systems

The choice isn’t binary. Many teams use Healing Tables for periodic “refresh” operations while running standard incremental processes daily. If the daily process accumulates issues, the weekly or monthly heal repairs them.

Wrapping Up

That 2 AM debugging session taught me something important: the traditional approach to SCD2—incremental changes applied to existing dimension state—creates tight coupling between historical load operations and current data quality. Every bug, every edge case, every source system quirk gets baked into the dimension permanently.

Healing Tables break that coupling. By extracting change points from source data, constructing time slices independently, compressing duplicate states, and validating results before loading, you create dimensions that are deterministically reproducible from source data.

The six-step framework isn’t complicated, but it requires discipline:

Build the Effectivity Table with actual changes only
Generate Time Slices with proper valid_from/valid_to boundaries
Join multiple sources using unified timeline logic
Compute hashes for efficient change detection
Compress consecutive identical states
Validate temporal integrity before loading

Implement those steps correctly, and you’ll have dimensions that can heal themselves from whatever accumulated issues have crept in over months or years of operation.

The next time someone asks you to backfill three years of history by running the daily process 1,095 times, you’ll have a better answer.

For Sooty

Chris Hillman — Tue, 03 Feb 2026 09:00:00 +1100

This one isn’t about data pipelines. There’s no framework, no architecture diagram, no code snippet at the end.

Yesterday I said goodbye to my best friend.

Sooty was a miniature schnauzer — nine kilograms of stubbornness, loyalty, and heart. Born October 2011. Gone February 2026. Fourteen years that changed the shape of everything.

I don’t have the right words for this. I’m not sure anyone does. But I tried to write something that comes close, and I wanted to share it here — because this is my corner of the internet, and she deserves a place in it.

Sooty

There is something about having a dog that nobody warns you about. They don’t tell you that the first time she falls asleep with her head on your lap, something inside your chest will rearrange itself, and from then on there will always be a space in you that is exactly the shape of her.

She is going to know where you are. By the sound of your keys in the door, by the way you sigh when you sit down, by the particular shuffle of your feet on the kitchen floor at six in the morning. She will never once say your name and you will never once doubt that she knows it.

And every time you come home — every single time, whether you were gone for eight hours or eight minutes — she will greet you at the door with the excitement and joy and emotion of a relative she hasn’t seen in years. As though your return is the most remarkable thing that has ever happened. As though the world had gone quiet without you and now, finally, it is loud again.

She will follow you throughout the house, every room, no matter how small the journey — just to check you’re still there. She will sit beside you during every bad movie, every 2am bout of insomnia when the ceiling has nothing useful to offer. She will follow you into rooms you didn’t even mean to walk into and look up at you as if to say, Well? We’re here now. What’s the plan?

She is your partner in crime before you even commit one. She will be at the door before you even reach for the leash. She will bark at the mailman every time she sees him, because someone has to warn him — my house — and she has appointed herself to the role.

And there will be days when the world does what the world does best, which is to come at you sideways and keep swinging. Days when the meetings don’t stop, when the inbox is on fire, when your body feels like it was assembled wrong.

And on those days, she will not ask you what happened. She will not offer advice. She will press her whole warm ridiculous body against yours and stay. Just stay. And occasionally lick your hand — not to fix you, but to say I’m checking. I’m still here. Are you still here?

And yes, she will also be the reason you cannot sleep in. Every morning a fresh day — a stretch with energy, a shake, and a tail wag, sometimes all at once, as though the night never happened and everything is new again.

She will remind you that the sun has come up and that this, apparently, is cause for celebration. She will remind you that her bowl is empty and that this, apparently, is an emergency.

She will stand at the door and look back at you with those unbearable eyes and you will put on your shoes even when you don’t want to, and you will step outside even when the sky is grey, and you will walk — because she asked, because she needs you, because somehow the needing is the gift. And every walk a new adventure. She knew where to go. Lead on.

Sometimes she sniffed where she shouldn’t have. Under the fence — and got bitten. The neighbour’s cat — and got a claw slap across the nose. But she bounced back up every time, because that is the thing about dogs: they do not hold grudges against the world for being sharp. They just shake it off and go looking for the next thing to love.

When she was young she would whimsically play throughout the house, tearing from room to room like joy had legs, and then — no matter where she was, middle of the floor en route to the bowl, or outside in the windy grass — she would zonk out. Batteries depleted. Gone. As though living that hard required sleeping that deep.

And as she got older she would remind you — time for bed, come on — at 7pm. A bit too early for you, but the right time for her. And you would learn that she was not wrong about most things, and maybe not wrong about this either.

She will see you through the worst of it. She will be there when the phone call comes. She will be there when you can’t get off the couch. She will be there when you forget what it feels like to be okay, lying beside you like a kind of proof — that warmth exists, that loyalty is not a theory, that someone in this world looked at you and thought yes, this one, always this one, no matter what.

And she will be there for the best of it too. She will be in the background of your happiest memories, tail going, ears up, a blur of joy in the corner of every photograph. She will teach you that happiness is not complicated. That it lives in tennis balls and puddles and the space between your hand and the top of her head.

And here is the part they really should warn you about.

She will teach you that love does not need language. That it does not need to be earned or explained or defended. That it can just exist, simple and enormous, in the weight of a head on your chest, in the sound of breathing that is not yours but that has become as necessary as your own.

She will teach you all of this, and she will teach you so well that by the time you have to let her go, you will understand exactly what you are losing.

And I think that is why it is so hard to say goodbye. Not because we didn’t know it was coming. But because they spent every single day of their lives making sure we knew exactly how much there was to love.

And they were right. And we did. And it was everything.

Rest easy, Sooty. You were the best girl.

What an NBA Coach Can Teach Data Leaders About Building Teams That Actually Work

Chris Hillman — Mon, 02 Feb 2026 09:00:00 +1100

I was three hours into a retrospective that had devolved into blame-shifting when the most senior engineer on the team finally spoke up. “Look,” he said, “we can keep pointing fingers at the data model, or we can admit we don’t actually trust each other enough to have an honest conversation about what went wrong.”

The room went quiet. He was right.

That moment stuck with me because it exposed something I’ve seen destroy more data teams than bad architecture ever could: the absence of genuine connection between people who spend forty-plus hours a week depending on each other.

We obsess over tool selection. We debate medallion architectures until our Teams threads reach scroll-fatigue territory. We write detailed runbooks and build elaborate monitoring systems. But we rarely invest in the thing that actually determines whether a data team succeeds or fails—whether the people on it feel like they belong to something worth sacrificing for.

A while back, I stumbled across a deep dive into Gregg Popovich’s leadership approach. If you don’t follow basketball, Pop coached the San Antonio Spurs to five NBA championships across twenty-two consecutive playoff appearances—the most sustained dynasty in modern professional sports. His winning percentage over three decades is unmatched in North American sports.

And his secret? It wasn’t X’s and O’s. It was wine, tears, and showing up at a player’s hotel room at 2 AM to cry with him after his father died.

What I found wasn’t a basketball story at all. It was a masterclass in building teams that actually work. And everything in it applies to leading data engineers—if you’re willing to do the uncomfortable work.

The problem with how we lead data teams

When managing data team’s: we’ve inherited a leadership model built for factory floors and sales quotas, then awkwardly grafted it onto creative, highly collaborative technical work.

The standard playbook goes something like this:

Set clear goals. Define KPIs. Track story points. Measure velocity. Hold quarterly reviews where you show charts that go up and to the right (or explain why they didn’t).

Provide feedback. Annual performance reviews. Maybe some lightweight 360 input. A few uncomfortable conversations about “areas for growth.”

Remove blockers. Clear the technical debt backlog. Negotiate for headcount. Shield the team from stakeholder noise.

None of this is wrong, exactly. But it’s not sufficient.

Data teams live in a peculiar tension. The work is deeply technical—you need people who genuinely understand distributed systems, query optimization, and schema design. But the outcomes are entirely social—dashboards nobody uses, pipelines that don’t answer the questions stakeholders actually have, data quality that erodes trust with every “Why do these numbers look different?” conversation.

Success requires people who can hold both realities simultaneously. And that kind of cognitive flexibility only emerges in environments where people feel safe enough to be vulnerable, connected enough to sacrifice personal metrics for collective success, and invested enough to push back when the team is heading in the wrong direction.

Popovich understood this intuitively. His teams consistently featured players who took less money, accepted smaller roles, and subordinated personal statistics for championships. How? Not through inspirational speeches or performance bonuses. Through dinners, emotional presence, and a relentless refusal to separate the person from the player.

Belonging cues: the invisible architecture of high-performing teams

Daniel Coyle, who studied Popovich extensively for his book The Culture Code, identified three types of signals that create what he calls “belonging cues”:

Personal, up-close connection — behavior that communicates “I care about you as a person”
Performance feedback — relentless criticism that says “we have high standards here”
Big-picture perspective — conversations that say “life is bigger than this immediate task”

Most data leaders nail one of these and completely miss the others.

The technically-focused leader provides excellent feedback on code quality and architecture decisions but treats personal connection as optional—something that happens organically in Teams channels or maybe at the occasional team happy hour.

The people-focused leader builds genuine relationships but struggles to deliver hard feedback, allowing mediocrity to persist because confrontation feels mean.

The visionary leader connects everything to grand purpose—“we’re democratizing data access across the organization!"—but the lofty framing feels hollow when day-to-day work involves debugging pipeline failures at 10 PM on a Friday.

Popovich weaves all three together constantly. The same evening might include crying with a player over a family tragedy, reviewing film and yelling about a missed rotation, and discussing the Civil Rights Movement. As one assistant coach put it: “He’ll tell you the truth, with no bullshit, and then he’ll love you to death.”

This isn’t contradictory. It’s complete.

Sound familiar? It should. Google’s Project Aristotle research found essentially the same thing—that psychological safety (the belief that you won’t be punished for taking risks or admitting mistakes) was the single strongest predictor of team performance. But psychological safety isn’t warm fuzziness. It’s the confidence that your team has high standards and will support you through the inevitable failures that come from pushing against those standards.

“Get over yourself” — the four-word philosophy that builds dynasties

Popovich’s core belief reduces to four words: “Get over yourself.”

It sounds harsh. But it’s actually an invitation to freedom.

“It’s not about any one person,” he explains. “You’ve got to get over yourself and realize that it takes a group to get this thing done.”

When evaluating potential players, he watched for this quality specifically: “When there’s a guy who talks about himself all day long, you start to get the sense that he doesn’t listen real well… Has this person gotten over himself? If he has then he’s going to accept parameters. He’s going to accept the role.”

This is the hardest cultural shift for data teams to make.

We work in a field that rewards individual technical brilliance. The engineer who builds the elegant solution gets recognition. The one who writes the clever query gets quoted in architecture reviews. We celebrate the heroic debugging session that saved the quarterly report, rarely asking why the pipeline was fragile enough to need heroics in the first place.

Getting over yourself means accepting that the best data architecture might not showcase your technical sophistication. It means implementing the boring, proven solution instead of the interesting one. It means writing documentation instead of starting a new project. It means saying “I don’t know” instead of confidently bullshitting through a stakeholder meeting.

Manu Ginobili—an All-Star caliber player—started only 349 of 1,057 games because the team needed him as a sixth man. When the coaching staff asked him to permanently accept the bench role, they told him honestly: if he wasn’t good with it, he was going to start. Whatever he said, they would do it. He deserved that choice.

Ginobili agreed to come off the bench.

Tim Duncan was “blown away”: “Are you kidding? He’s Manu! He’s a star! He can’t not start.” But Ginobili embraced the sacrifice because he understood something crucial: his ego was getting between him and the team’s success.

How many data engineers would make that trade? How many would accept that their personal visibility, their resume-building features, their chance to work on the exciting project needed to take a back seat to what the team actually needed?

The uncomfortable truth: they’ll only make that sacrifice if they trust you enough to believe it’s genuine. If they feel like the sacrifice is worth it because they belong to something worth sacrificing for.

The dinner table as sacred ground

Popovich spends an estimated seven figures annually on food and wine for team gatherings. After road games, the team frequently stays overnight specifically to share dinner, with Pop personally scouting restaurants weeks in advance.

These aren’t casual meals. He positions himself at tables of exactly six people—“believing this number fosters diversity of conversation without people breaking off into side chats”—and works the room methodically. The dinners serve a strategic purpose: “Dinners help us have a better understanding of each individual person, which brings us closer to each other—and, on the court, understand each other better.”

Now here’s the challenge for modern data leaders: how do you create this in a world where your team might be spread across six time zones?

You can’t replicate the intimacy of shared meals and expensive wine. But you can replicate the intention behind them.

What Popovich’s dinners actually accomplish:

Dedicated time for connection that isn’t about work deliverables
Small enough groups to have real conversations
Physical proximity and eye contact (or as close as you can get)
Consistent repetition that builds cumulative trust
Leader modeling vulnerability by being genuinely present

Translation for distributed data teams:

Intentional one-on-ones that go beyond status updates. Not “what are you working on?” but “what’s actually going on with you?” Schedule them consistently. Protect them from cancellation. Come prepared with questions about their life outside work.

Small-group video calls for real conversation. Skip the all-hands. Create rotating groups of 4-6 people for unstructured time. No agenda. No recording. Just talking.

Occasional in-person gatherings that prioritize relationship-building. If you bring the team together once a year, don’t fill the schedule with strategy sessions and team-building exercises. Create space for meals, walks, and conversations that would feel weird to have over Zoom.

Consistent rituals that build cumulative connection. A Slack channel where people share weekend plans. A monthly “show and tell” of something non-work related. Anything that repeats often enough to become meaningful.

The point isn’t to replicate Popovich’s exact methods. The point is to be as intentional about relationship-building as you are about sprint planning.

When tragedy strikes, leaders appear

The most powerful evidence of Popovich’s approach emerges during personal crises.

When DeMar DeRozan’s father passed away during the 2021 season, DeRozan called the team’s GM privately—“I didn’t want nobody to know.” Ninety seconds later, Pop was knocking on his hotel room door. “Pop sat in the room with me and cried with me for about two hours. He was like, ‘I’m not leaving until you leave.’”

When Dejounte Murray’s mother was shot during his rookie year, Popovich called her directly—without Murray’s knowledge—offering to move her to San Antonio on his own dime.

When Robert Horry’s daughter was hospitalized at the start of a season, Popovich told him: “Don’t come back until she’s out of the hospital.” Three weeks later, upon returning, Pop asked if she was out. Horry said yes. Pop responded: “She ain’t out of the woods yet. Go back home. We don’t really need you right now.”

“Family is the most important thing.”

These aren’t management techniques. They’re expressions of genuine care. And they’re the reason players sacrifice for Popovich in ways they wouldn’t for other coaches.

Data leaders: when was the last time you showed up for someone beyond work boundaries?

I’m not suggesting you need to cry with your direct reports or offer to pay for their family members’ relocation. But I am suggesting that most of us dramatically underinvest in being present during the moments that actually matter.

The life events where showing up builds lasting trust:

Family emergencies and deaths
Health crises (theirs or family members')
Major life transitions (new babies, divorces, moves)
Career anxieties and disappointments
Personal struggles they’re brave enough to share

What “showing up” looks like in practice:

Proactively reaching out, not waiting for them to come to you
Asking what they need, not assuming you know
Offering flexibility without making them negotiate for it
Following up later to see how they’re doing
Remembering the details and checking in over time

The leaders I’ve worked with who inspire genuine loyalty share one trait: they remember the human stuff. They ask about your kid’s soccer tournament. They notice when you seem off. They don’t treat personal life as interference with work productivity.

High standards and deep care aren’t opposites

Here’s the false dichotomy that destroys most leadership: you’re either a demanding, results-oriented leader who pushes for excellence, or you’re a supportive, people-first leader who prioritizes relationships. Pick your lane.

Popovich refuses the choice.

Boris Diaw once visited Tony Parker at Pop’s house for Christmas dinner. After a warm meal, both Parker and Popovich disappeared. Diaw went searching and found Pop reviewing film with Tony about the game the night before—and yelling at him about missed shots and turnovers.

Diaw’s takeaway captures the essence: “On the same night, you could have the family setting, all the love and the care, and at the same time caring about making Tony a better player.”

The warmth makes the feedback land differently. Not softer—still direct, still pointed, still demanding. But the player receiving it knows it comes from someone who genuinely wants them to succeed, who has invested in them as a complete person, who will be there for the hard stuff outside work.

Research backs this up. Amy Edmondson’s psychological safety research shows that high-performing teams combine high standards with high support—“learning zones” where people push themselves precisely because they feel safe enough to fail.

For data leaders, this means:

Being specific and direct about performance issues without softening the message into meaninglessness. “The pipeline you built has reliability problems that are affecting stakeholder trust” is clearer and more respectful than “there might be some opportunities to improve things.”

Providing the context behind criticism so it lands as investment rather than attack. “I’m telling you this because you’re capable of leading our data platform work, and these patterns would prevent that.”

Following hard conversations with continued connection rather than distance. The relationship doesn’t end after difficult feedback—it continues.

Modeling receiving feedback yourself so the team sees that criticism flows in all directions. Pop regularly admitted his own failures publicly. Your team needs to see you do the same.

Connecting work to something larger

Before the 2014 NBA Finals, Popovich skipped X’s and O’s entirely. Instead, he told his team about Eddie Mabo Day—a significant date for Indigenous Australians—to honor point guard Patty Mills’ heritage.

This reflects Pop’s broader practice of engaging players on issues far beyond basketball. He’s taken teams to see Hamilton on Broadway. He brought civil rights activist John Carlos to speak. He gave players Ta-Nehisi Coates’ Between the World and Me to read.

Why does this matter for building teams?

Because it sends a message: we are complete human beings who care about the world beyond our immediate work.

Coyle noted: “Popovich would create similar conversations on the war in Syria, or a change of government in Argentina, gay marriage, institutional racism, terrorism—it doesn’t really matter, as long as it delivers the message he wants to deliver: There are bigger things than basketball to which we are all connected.”

This is the belonging cue that most data leaders miss entirely.

We act as if the only things worth discussing are data models, pipeline reliability, and stakeholder requirements. We treat outside interests as distractions rather than windows into who our team members actually are.

For data teams, connecting to something larger might include:

Discussions about the ethical implications of data work (privacy, algorithmic bias, surveillance)
Conversations about industry trends and where the field is heading
Space for people to share what they care about outside work
Recognition of cultural backgrounds and experiences
Honest conversation about work-life balance, burnout, and sustainability

None of this requires seven-figure wine budgets. It requires treating your team members as complete humans who exist in a world larger than your Jira board.

The evidence speaks across decades

Popovich’s methods produce measurable results. Five championships. Twenty-two consecutive playoff appearances. Eighteen straight 50-win seasons—unprecedented in NBA history. The best winning percentage of any major professional sports franchise in North America over a three-decade span.

The 2014 championship team became the first since the ABA-NBA merger to win a title without any player averaging 30+ minutes per game. Four different players led the team in scoring during the playoffs. No dominant statistics—just beautiful, unselfish basketball.

Beyond the Spurs, his coaching tree now shapes the entire league. When The Athletic surveyed NBA players on which coach they’d most want to play for, Popovich received 41%—the highest percentage by far.

Kevin Durant captured the appeal: “When you are with somebody like that, a person who cares about all those things on top of wanting you to be the best player you can be, then you want to go to work with that person.”

The same principle applies to data teams. The leaders who build lasting excellence aren’t the ones with the cleverest technical strategies. They’re the ones who create environments where talented people choose to stay, sacrifice, and push each other toward genuine accomplishment.

The uncomfortable question for data leaders

At his Hall of Fame induction, Popovich offered unusual clarity: “Basketball doesn’t love us back, does it? We use it like a bar of soap, right? It pays our bills. It gives us a wonderful life. But I don’t remember it saying, ‘I love you, Pop.’ It’s different. It’s the family.”

Here’s the uncomfortable question for data leaders: what’s the family equivalent for your team?

Data pipelines don’t love you back. Dashboards don’t remember your birthday. The perfectly optimized query won’t show up at your door when your father dies.

The work matters. Technical excellence matters. Delivering value to the organization matters.

But none of it persists if you haven’t built the human foundation that makes sustained performance possible.

The false choice is between being a great technical leader and being a great people leader. Popovich proves you need both—that technical excellence requires the human foundation, not despite it.

So the next question isn’t whether you have the right technology stack or the optimal team structure. It’s whether you’re investing in the belonging cues, the dinners (literal or figurative), the showing up during crisis, the relentless standards delivered with genuine love.

It’s whether you’re willing to do the uncomfortable work of building something worth sacrificing for.

Because here’s the thing: your team knows the difference. They know when connection is performative and when it’s real. They know when high standards come from caring and when they come from ego. They know whether they’re resources to be optimized or humans to be invested in.

And they’ll calibrate their own investment accordingly.

Tim Duncan, accepting his Hall of Fame honor, said what countless Spurs have felt about Popovich: “You showed up after I got drafted. You came to my island. You sat with my friends, my family. You talked with my dad. I thought that was normal. It’s not.”

I’m nowhere near Pop’s level. Most of us aren’t. But stories like this matter because they remind us what good can look like—and give us something to measure ourselves against.

Not to feel inadequate. To feel inspired.

The gap between where we are and where Pop operates isn’t a source of shame. It’s a direction. And the first step is simply asking the question: Would my team say I showed up?

Context Engineering: The New Must-Have Skill for Data Engineers

Chris Hillman — Sat, 31 Jan 2026 09:00:00 +1100

Last year I watched a colleague ask AI to help write a dbt model. The AI spit out perfectly functional SQL—clean syntax, proper CTEs, the works. Looked great.

Then I noticed the table would eventually hold 800 million rows. No partitioning. No clustering. Just a raw, unoptimised heap waiting to turn into a query performance nightmare (that would likely become my nightmare to fix).

The engineer wasn’t at fault. The AI wasn’t at fault either, really. The AI simply didn’t know that our environment clusters large tables by date. It didn’t know our team’s conventions around incremental models. It couldn’t know, because nobody had told it.

Here’s the thing: most data engineers treat AI assistants like they’re infinitely capable strangers. We give them tasks, accept their output, maybe tweak a few lines. But we never invest in teaching them our context—the hard-won lessons, the team conventions, the mistakes we’ve already made (and would rather not make again).

What if you could? What if every partitioning decision, every naming convention, every “never do this” rule you’ve learned over the years lived in a file that your AI assistant read before every single interaction?

That’s exactly what context engineering enables. Once you set it up, your AI stops being a generic autocomplete and starts acting like a peer who actually knows your codebase.

The Problem With Generic AI Assistance

I’ve been using AI coding assistants for a while. First Copilot (that was a terrible experience), then ChatGPT, and now more recently Claude. I can write up blocks faster with the right prompts.

But something kept nagging at me.

The AI would suggest something technically correct but contextually wrong. It would generate a model without tests. Propose a column name that violated our conventions. Use VARCHAR when we standardise on STRING. Small things, individually. Which would lead to more and more refining on the prompts.

The root cause was simple: the AI had no memory. Every conversation started fresh. It didn’t know that we’d spent three painful weeks last quarter cleaning up inconsistent date formats. It didn’t know that our dim_customer table had a specific SCD Type 2 pattern we’d refined over months.

Every session, I found myself re-explaining the same context. “Remember to add tests.” “We use snake_case for everything.” “Always partition by event_date.”

This is always a problem in work generally. You could have an equally talented data engineer sitting next to you, but you don’t share the same problems faced, the same challenges solved. You’d find out months or weeks later—we ran into this problem in that table, and we resolved it by doing this. And you’d say, oh we had the same problem elsewhere, but here’s how we tackled it.

The solution can be better prompting, as I discussed in this article - https://ghostinthedata.info/posts/2026/2026-01-04-talk/. It can also be solved with persistent context.

But there’s another approach that’s more fundamental: teaching your AI assistant your context once, so it remembers across every session. That’s what context engineering is about.

What Context Engineering Actually Looks Like

Context engineering is a file (or set of files) that contains your team’s accumulated wisdom, loaded automatically into every AI session. Think of it as institutional memory for your AI assistant.

The core concept is straightforward: you maintain markdown files that describe your conventions, standards, and hard-won lessons. Your AI assistant reads these files at the start of every session.

Different tools implement this differently:

Claude Code (Anthropic’s CLI tool) looks for a CLAUDE.md file in your project root. It loads this automatically when you start a session.

Cline (the VSCode extension I use) supports custom system prompts and can reference local files for context. You can configure it to always include specific markdown files in its context window.

Cursor has similar capabilities with its rules files.

The implementation details vary, but the principle is identical: give the AI your context before it starts generating code.

Here’s what a basic context file might look like for a data engineering project:

# Analytics Pipeline Context

## Stack

- Snowflake (Enterprise tier)
- dbt Core 1.7+
- Fivetran for ingestion
- Tableau for BI

## Critical Standards

### Partitioning & Clustering

- **ALWAYS** cluster tables >10M rows by their primary date column
- Use `cluster_by` in dbt config, not raw DDL
- Partition Staging tables by data_date

### Naming Conventions

- Models: `stg_source__entity`, `fct_entity`, `dim_entity`
- Columns: snake_case, never camelCase
- Booleans: prefix with `is_` or `has_`
- Dates: suffix with `_at` for timestamps, `_date` for dates
- IDs: suffix with `_id` (never `_key` for surrogate keys)

### Testing Requirements

- Every model needs at least: unique, not_null on primary key
- Fact tables need referential integrity tests to all dimensions
- SCD2 dimensions need: valid_from < valid_to, no gaps or overlaps in history

This isn’t complicated. It’s just written-down knowledge. The magic happens when this knowledge persists across sessions.

Structuring Your Context Files

I use Cline for most of my dbt work because it integrates cleanly with VSCode and connects directly to the Claude API. Rather than one massive context file, I structure my context into logical chunks—this helps the AI (and me) find relevant information quickly.

Here’s the directory structure I use:

When I start a session, this context is already loaded. The AI knows our conventions before I type a single character.

The specific configuration will depend on your tool of choice—Cline, Cursor, and Claude Code each have their own settings. The important part is the content itself: clear, specific rules that capture how your team actually works.

The “Every Mistake Becomes a Rule” Philosophy

Here’s where context engineering gets genuinely powerful.

Boris Cherny, who created Claude Code, maintains his context file at exactly 2,500 tokens. His practice: “Every mistake becomes a rule.” When something goes wrong that’s generalisable, it gets added to the context file immediately.

This resonated with me because it mirrors how data engineers actually accumulate wisdom. We don’t learn by reading documentation. We learn by screwing things up, fixing them, and vowing to never make that mistake again.

The difference now? Those lessons don’t just live in our heads. They live in a file that our AI assistant reads before every interaction.

Last quarter, a seemingly innocent change to an SCD2 dimension caused our dashboards to double-count certain metrics. The root cause was subtle: we’d accidentally created overlapping validity windows during a backfill. It took three days to diagnose.

The fix took an hour. But more importantly, I added this to my testing patterns file:

SCD Type 2 Validation Rules

## SCD Type 2 Validation

Every SCD2 dimension MUST include these tests:

### No Overlapping Windows

```yaml
tests:
 - dbt_utils.unique_combination_of_columns:
 combination_of_columns:
 - customer_id
 - valid_from
 - dbt_expectations.expect_column_pair_values_A_to_be_less_than_B:
 column_A: valid_from
 column_B: valid_to
 or_equal: false
```

The Duct Tape Data Engineer

Chris Hillman — Sat, 24 Jan 2026 09:00:00 +1100

The Engineer Who Ships

I want to tell you about a data engineer I worked with. Let’s call her Sarah.

Sarah had a reputation. When business stakeholders had an urgent question—the kind that arrives at 4 PM on a Friday with the CEO’s name in the subject line—they went to Sarah. Not to the senior architect with the impeccable data model. Not to the platform team with their carefully orchestrated Airflow DAGs. They went to Sarah.

Her code wasn’t pretty. She’d pull data from production databases with SQL queries that made the architecture team wince. She’d transform things in Python scripts that violated every design pattern in the book. She’d dump results into Excel because that’s where the CFO could actually use them.

And here’s the thing: Sarah shipped. Every time. On time. With answers that moved the business forward.

At the time, I thought Sarah was the best data engineer I’d ever worked with. I’m less certain now.

Sarah is what I would call a “duct tape programmer.” Someone who ships working solutions while others are still debating architectures. In the moment, there’s no one you’d rather have on your team.

But moments pass. And the code remains.

Three Types of Data Engineer

I’ve come to believe there are three distinct approaches to data engineering, and understanding the differences matters more than most of us realise.

The Architecture Astronaut builds for a future that never arrives. They spend months perfecting data contracts and tenant-level data mesh architecture. They have governance frameworks, schema registries, and architectural purity that would make Zhamak Dehghani proud. What they don’t have is a single stakeholder whose problem they’ve solved.

The Duct Tape Engineer ships at all costs. They’re Sarah at 4 PM on Friday—pulling data from production, hacking together Python scripts, dumping results into Excel. They solve problems. They move fast. They leave a trail of undocumented queries and mystery scripts in their wake.

The Foundational Pragmatist does something different. They invest in the foundations that make future speed possible—proper dimensional models, conformed dimensions, clean grain definitions—and then they duct tape on top of that prepared surface. They ship fast because they built well, not instead of building well.

The industry talks endlessly about the first two. The conferences are full of astronauts evangelising the latest architecture. The war stories celebrate the duct tape heroes who saved the day. But the engineers who actually deliver sustained value? They’re usually the third type, and they’re rarely on stage.

The Architecture Astronaut Visits Your Desk

You know this person. They show up at your desk, coffee in hand, ready to explain why your simple ETL pipeline is fundamentally wrong.

“What you really need,” they say, eyes lighting up, “is a proper lakehouse architecture. Bronze, silver, gold layers. Delta Lake for ACID compliance. Apache Kafka for real-time ingestion. And have you considered data mesh? Each domain team should own their own data products with standardized contracts and federated governance.”

You nod politely, but inside you’re doing the math. Your company has three data sources. Your biggest table has 50 million rows. Your most demanding stakeholder needs a weekly report. And this person wants you to implement an architecture designed for companies processing petabytes per day.

They’re the over-engineering colleague who “comes up to your desk, coffee mug in hand, and starts rattling on about how if you implement event-driven architecture with a proper schema registry, your pipeline will be 34% more scalable.” Scalable for what? You’re processing 100MB of data. On a good day.

The architecture astronaut isn’t stupid. They’re often brilliant. They’ve read every blog post, attended every conference, mastered every tool. They can diagram a system that would make Netflix jealous.

But they have a disease: they’re allergic to shipping.

They’d rather spend six months perfecting a medallion architecture than six hours writing the SQL query that answers the CEO’s question. They optimise for theoretical elegance at the expense of actual impact. And they leave a trail of half-finished platforms in their wake.

A Tale of Two Teams

Let me tell you about two data teams I’ve observed over the years.

The first was building the future. Data contracts. Federated governance. Tenant-level data mesh with domain ownership and self-serve capabilities. They had architecture review boards, schema evolution policies, and compatibility matrices. Every decision went through rigorous evaluation. Every interface was documented exhaustively.

Three years in, they were still perfecting their approach. The platform team had become a bureaucracy that produced governance documents, not insights. Architecture astronauts, every one of them.

The second team took a different path. They weren’t duct tape engineers—they didn’t hack things together and hope for the best. Instead, they invested in a proper dimensional model. Conformed dimensions. Clean fact tables. Well-defined grain. Nothing fancy. No distributed systems. No streaming. Just solid data modeling fundamentals executed well.

When the general manager asked a question, they answered it in minutes. Not because they had magical technology, but because the foundation was sound. The conformed customer dimension meant they didn’t have to reconcile six different definitions of “active customer.” The properly-grained fact tables meant they could slice and dice without fear of double-counting. The slowly changing dimensions meant they could answer “what did this look like last quarter?” without archaeological expeditions through backup tapes.

One team had years of architectural innovation with nothing to show for it. The other had basic dimensional modeling and could answer almost anything.

The difference wasn’t intelligence or effort. It was understanding which investments actually compound. The second team wasn’t avoiding architecture—they were choosing the right architecture. The kind that creates a foundation for speed rather than a monument to complexity.

The 85% Failure Rate

But wait, you might say. Surely all these complex architectures exist for a reason. Surely the industry’s best minds have converged on data mesh and lakehouses because they work.

In 2015, Gartner estimated that most big data projects would fail. By 2017, analyst Nick Heudecker admitted they’d been “too conservative.” The actual failure rate was higher—in the 80% range. Most data migrations fail or exceed their budget and scope.

Eighty percent. In what other engineering discipline would we accept failure rates like that? In what other field would we keep advocating for the same approaches that fail four out of five times?

The UK’s National Health Service attempted to centralise patient records in one of the largest data projects ever undertaken. They flushed $15 billion down the drain. Fifteen billion dollars. And they got nothing.

Lyft spent years migrating to Hive. By the time they finished in 2020, the entire industry had moved on. They ended up stuck on an outdated system, having invested countless engineering hours chasing an architecture that was already obsolete.

These aren’t exceptions. They’re the pattern. Complex architectures fail. Simple foundations, built well, endure.

The Philosophy of Worse

This tension isn’t new. Richard Gabriel identified it in 1989 with his famous essay “Worse is Better.”

The core idea is counterintuitive: “Implementation simplicity has highest priority.” Not correctness. Not completeness. Not elegance. Simplicity.

Gabriel argued that “it is better to get half of the right thing available so that it spreads like a virus. Once people are hooked on it, take the time to improve it to 90% of the right thing.”

The four principles he outlined still apply:

Simplicity: The system should be simple above all else.
Correctness: It’s slightly better to be simple than correct.
Consistency: Can be sacrificed for simplicity.
Completeness: Can be sacrificed in favour of any other quality.

This feels wrong to engineers. We’re trained to build things correctly. We’re judged on technical sophistication. Our peers admire elegant architectures and comprehensive solutions.

But Gabriel was describing how Unix beat more sophisticated operating systems. How C beat more powerful languages. How TCP/IP beat more theoretically correct protocols. The simpler, “worse” solutions won because they shipped, spread, and evolved.

John Gall formalised the pattern in 1975: “A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”

Don’t architect a lakehouse from scratch. Start with a simple pipeline that works. Evolve it as you understand your actual requirements. That’s not compromise—that’s how successful systems are built.

The Foundation That Enables Speed

Here’s where the duct tape philosophy goes wrong: it assumes all architecture is overhead.

I believe strongly in dimensional modeling. Not because it’s elegant—though it is. Not because it’s theoretically pure—though it can be. But because when a proper dimensional model exists, I can answer the general manager’s Friday afternoon question in fifteen minutes instead of four hours.

The conformed dimension that took a week to build properly? It’s saved me hundreds of hours of query writing since. The fact table with clean grain definitions? It means I’m not debugging data quality issues at 11 PM.

This is what separates the foundational pragmatist from the pure duct tape engineer. Sarah could answer Friday’s question. But she’d have to re-derive the customer definition from scratch every time. She’d spend hours reconciling data that a conformed dimension would have reconciled once. She could produce 3 reports but they might differ in definitions of what a customer is.

The architecture astronaut spends six months building infrastructure nobody uses. The duct tape engineer hacks together solutions that become unmaintainable. The foundational pragmatist invests two weeks in dimensional modeling and creates the conditions for rapid, reliable delivery forever after.

The question isn’t whether to invest in architecture. It’s which architecture and when. A solid dimensional model? Almost always worth it. A lakehouse for your 50GB dataset? Probably not. Data contracts and federated governance for a team of three? You’re solving problems you don’t have.

That team I mentioned—the one answering questions in minutes—they weren’t magicians. They’d done the foundational work. They had clean dimensions. They had documented grain. They had conformed definitions.

Their duct tape stuck better because they’d prepared the surface.

The Maturity Gradient

There’s a progression that the conference talks never mention:

Monthly reports → Weekly reports → Daily pipelines → Micro-batches → Streaming

Each step leftward costs exponentially more in complexity. Each step rightward gives you exponentially more time to get the quality right.

When I build something new, I start as far right as the business can tolerate. Monthly reporting sounds unsexy, but it gives you thirty days to catch quality issues before anyone sees bad data. You have time to validate, to cross-check, to build confidence.

Once monthly is bulletproof, moving to weekly is straightforward. You’ve already solved the hard problems—the business logic, the edge cases, the data quality rules. Now you’re just increasing frequency. Daily follows. Micro-batch follows.

Can you skip straight to real-time? Sure. I’ve seen it done. I’ve also seen the pain involved—debugging race conditions, explaining to stakeholders why the “real-time” dashboard showed impossible numbers, rebuilding pipelines from scratch because the streaming-first design couldn’t handle a schema change.

The streaming-first approach means you’re solving data quality, business logic, edge cases, AND distributed systems problems simultaneously. That’s a recipe for the 85% failure rate we talked about.

Ship the monthly report. Prove the logic. Harden the quality. Then earn your way to daily.

Duct Tape With a Sunset Date

Here’s what separates strategic pragmatism from technical negligence: the hardening plan.

Every quick solution I ship comes with an implicit contract. Yes, I’ll wrangle this data together to plug the compliance gap by Friday. Yes, I’ll build that dashboard for the legal team using queries that would make a data modeler weep. But written somewhere—in Jira, in a Confluence page, in my own notes—is the plan to do it properly.

The CFO’s urgent analysis gets a spreadsheet today. It gets a proper pipeline next quarter.

This isn’t compromise—it’s sequencing. The business gets value immediately. The team gets breathing room. And the quick solution becomes a specification for the permanent one. You’ve proven the logic works. You’ve validated the business need. Now you can invest in building it right, with far less risk than starting from scratch.

The failure mode isn’t duct tape itself. It’s duct tape without a sunset date. It’s the “temporary” query that’s still running five years later because nobody remembered to harden it. It’s technical debt accumulated without a repayment plan.

I’ve used this approach to protect my teams from massive pieces of work that would have consumed them for months. The compliance hole that needed plugging immediately? Duct tape with a ticket. The legal team’s urgent risk analysis? Duct tape with a documented plan. The high-value project that couldn’t wait for proper architecture? Duct tape with a sunset date.

The duct tape isn’t the problem. Forgetting to remove it is.

Sarah’s Legacy

Every organisation has Bob’s query.

You know the one. Bob wrote it three years ago. Bob left eighteen months ago. The query is still running every Tuesday morning, feeding a report that twelve people apparently depend on, documented nowhere, understood by no one.

Here’s the uncomfortable truth: Bob’s query is what Sarah’s Friday afternoon heroics look like three years later.

Sarah shipped. She solved the problem. The CEO got their answer. But she didn’t write the ticket. She didn’t document the logic. She didn’t create a sunset date. She moved on to the next fire, and the quick fix quietly became load-bearing infrastructure.

The person who inherited Sarah’s work doesn’t think she’s a hero. They think she’s the reason they’re debugging mystery scripts at 11 PM instead of building something new.

This is what duct tape looks like when the sunset date gets forgotten. It’s technical debt, pure and simple. A liability waiting to break at the worst possible moment.

But here’s the reframe: Sarah’s legacy is also an opportunity.

That query is a perfect project for a junior engineer. The risk is low—it’s already working, so they have a baseline to compare against. The scope is contained—one query, one output, one business purpose. They can pull it apart, understand the logic, document what they find, and rebuild it as a proper pipeline.

They learn data modeling. They learn testing. They learn documentation. They learn how to interview stakeholders about requirements that were never written down. They harden their skills in a safe environment while simultaneously paying down technical debt. This is what builds up Junior to Senior levels.

Every piece of Sarah’s legacy is both a problem to solve and a training ground to offer. The mature data engineer sees both. They create the ticket, assign the junior, and turn a liability into a development opportunity.

The debt gets paid. The junior grows. The organisation gets a documented, tested, maintainable pipeline instead of a mystery script.

That’s pragmatism with a long-term view.

The Pragmatist’s Manifesto

Here’s what the foundational pragmatist understands:

Shipping is a feature. The most important feature. Your data pipeline doesn’t exist until it’s serving production queries. Your data model doesn’t matter until it’s powering real decisions. “Your product must have it.”

The pipeline that ships insights beats the pipeline that ships nothing. A “50%-good solution that people actually have solves more problems and survives longer than a 99% solution that nobody has because it’s in your lab where you’re endlessly polishing the damn thing.”

But foundations compound. A solid dimensional model isn’t architecture astronaut behaviour—it’s the investment that makes all future delivery faster and more reliable. Build the foundation once, benefit forever.

Directionally correct beats precisely wrong. The dashboard that’s 80% accurate and delivered today creates more value than the perfectly governed data product delivered never.

Start with CSV, graduate to Parquet, earn your way to streaming. You don’t need real-time until you do. You don’t need a lakehouse until you do. You don’t need Kafka until you do. And you probably don’t.

Every shortcut needs a sunset. Ship the hack. Document the plan. Pay down the debt. Quick fixes without a hardening plan aren’t pragmatism—they’re negligence with extra steps.

Jamie Zawinski said it best: “At the end of the day, ship the fucking thing! It’s great to rewrite your code and make it cleaner and by the third time it’ll actually be pretty. But that’s not the point—you’re not here to write code; you’re here to ship products.”

For data engineers, the adaptation is obvious: You’re not here to build architectures. You’re here to ship insights. But the insights you ship today shouldn’t sabotage the insights you need to ship tomorrow.

The Caveat

Here’s where I need to be careful.

Pragmatic shortcuts aren’t an excuse for incompetence. “Duct tape programmers must be talented enough to pull it off.”

Sarah wasn’t shipping ugly code because she didn’t know better. She was making conscious trade-offs. She understood the implications of pulling from production databases. She knew how to write cleaner code—she just knew when not to bother.

Her failure wasn’t the duct tape. It was stopping there.

There’s a difference between strategic simplicity and reckless shortcuts. The foundational pragmatist understands the trade-offs. They know what they’re sacrificing and why it’s worth it. They can explain their decisions and defend them. And they write the ticket for the hardening work.

Netflix actually needs Kafka. Meta actually needs petabyte-scale systems. Google actually needs distributed query engines.

The question is whether your company does.

Most don’t. The data supports this overwhelmingly. But knowing when you’re the exception requires understanding the fundamentals. You can’t break the rules effectively until you know what the rules are.

So yes, learn the proper patterns. Understand data modeling. Know why slowly changing dimensions exist. Study the medallion architecture. Read about data mesh.

Then make conscious decisions about when to use them. Not because a blog post said so. Not because a vendor promised ROI. Not because that’s what the architecture astronaut wants.

Because they solve a problem you actually have.

The Engineer Who Holds All Three

Here’s what I’ve learned after years of watching these patterns play out: the best data engineers understand all three modes and know when each applies.

I read every blog post about data mesh. I attend the conferences. I evaluate new tools. I think constantly about how we might evolve our architecture. That’s not being an astronaut—that’s staying informed.

When my team faces a Friday afternoon crisis, I can be Sarah. I can hack together the solution, ship the answer, buy us breathing room. That’s not recklessness—that’s responsiveness.

But unlike Sarah, I write the ticket. I document the sunset date. And when we have the space, I advocate for the dimensional modeling that will make the next crisis easier to handle.

The architecture astronaut’s failure is pursuing elegance without urgency. The duct tape engineer’s failure is urgency without investment in foundations. The pure pragmatist ships fast on solid ground and pays down their debts.

I’ve watched teams spend years on data mesh with nothing to show for it. I’ve inherited Sarah’s legacy code and cursed her name at midnight. I’ve also seen teams with solid data models answer any question in minutes, year after year.

The difference wasn’t the sophistication of their approach—it was whether their investments actually compounded into capability, and whether their shortcuts came with expiration dates.

So yes, ship the dashboard. Answer the executive’s question. Plug the compliance hole today.

But write down the hardening plan. Build the dimensional model when you can. Start with monthly, earn your way to daily. Invest in the foundations that make future speed possible.

And when the junior engineer asks why Sarah’s query is still running after five years, hand them the ticket to fix it.

That’s the job.

That Tuesday Morning When I Finally Fixed Our Ten-Minute Queries

Chris Hillman — Fri, 16 Jan 2026 09:00:00 +1100

The Ten-Minute Query

I’m sitting at my laptop on a Tuesday morning, waiting. The progress bar on my screen says ‘Query running… 4 minutes, 37 seconds.’ I lean back in my chair and let out this long sigh that probably says more than I intended.

My manager walks past my desk. She glances at my screen, and I can see that look—the one that says she already knows what I’m about to tell her. I didn’t need to explain.

“Still pulling that customer report?”

“Yeah.”

She shakes her head. “We need to fix this.”

It was an ad-hoc piece of work that we’d been running for months. But I just nod.

Here’s what was happening: every time someone needed to know ‘how many orders did customer X place?’ or ‘which products sold best last month?’, we’d query this massive table. Three million rows. Every single customer’s name, address, email—repeated on every order row. The system was scanning through all that redundant data every single time.

And then, two weeks ago, it got worse. Someone ran a data import script that had a bug. It replaced half our customer names with NULL. By the time we caught it, twelve reports had already gone out with blank customer names. I had to send apology emails. My team and I had to stay until midnight fixing it.

I thought, there has to be a better way to do this. People have been building data systems for decades—someone must have figured this out.

And they have. His name is Ralph Kimball, and his dimensional modeling methodology has been the gold standard for data warehouses since the 1990s.

The Problem I Couldn’t Unsee

Let me show you what our data looked like.

order_id | date | customer_name | address | email | product | amount
---------|------------|---------------|--------------|--------------|---------|-------
1 | 2024-11-15 | John Smith | 123 Main St | john@... | Pizza | 25
2 | 2024-11-15 | John Smith | 123 Main St | john@... | Soda | 3
3 | 2024-11-15 | Jane Doe | 456 Oak Ave | jane@... | Pizza | 25
4 | 2024-11-16 | John Smith | 123 Main St | john@... | Burger | 12

See what’s happening? John Smith’s address and email are stored three times. Multiply that by thousands of customers and millions of orders, and you’ve got:

Every query scanning way more data than necessary
Storage costs that make your CFO unhappy
One bug corrupting your entire history
Query times that make you question your career choices

Every single query was like asking someone to find your keys by searching every room in your house, including the ones you never use. Sure, they’ll find the keys eventually. But there’s a better way.

The Kitchen That Changed Everything

Let me paint you a hypothetical scenario. Imagine it’s a Saturday, and you’re standing in your kitchen after just moving house. You can’t find anything. Everything’s still in boxes.

You’re rummaging through the third box, looking for a spatula. You’re pulling out random stuff—a whisk, some Tupperware lids, a pizza cutter, yesterday’s takeout containers, a bag of flour. Everything mixed together. This is chaos.

Now imagine a friend comes over. They take one look at your kitchen and say, “What is this?”

“It’s… I haven’t unpacked yet.”

“You’ve been here a week.” They walk over to the mess. “Here’s what you should do. The stuff that rarely changes—your plates, your pots, your cutting boards—those go in the cabinets. Organised, labelled, easy to find. The stuff that comes and goes constantly—the groceries, the leftovers, what you’re cooking today—that goes in the fridge and on the counter. Don’t mix the permanent stuff with the temporary stuff.”

Two hours later, the kitchen makes sense. You can find anything in seconds. And here’s the wild part—it actually takes up less space because you’re not duplicating storage for the same items.

That’s dimensional modeling.

When your data is like those boxes, everything’s mixed together. Customer information sitting right next to order information, repeated over and over. Every time you need something, you’re digging through all the boxes.

Dimensional modeling says: let’s organise this. Put stable descriptive data—things that rarely change—in one place. We call these dimension tables. Put the events that happen constantly—the transactions, the updates, the measurements—in another place. We call these fact tables. Link them with keys.

Here’s what that transformation looks like:

BEFORE (Messy Boxes - everything mixed together):

order_id | customer_name | address | product | amount
1 | John Smith | 123 Main St | Pizza | 25
2 | John Smith | 123 Main St | Soda | 3

↑ John’s details repeated every single order!

AFTER (Organised Kitchen): DIM_CUSTOMER (Cabinets - stable stuff, stored ONCE):

customer_key | customer_id | name | address | email
1001 | C-123 | John Smith | 123 Main St | john@...
1002 | C-456 | Jane Doe | 456 Oak Ave | jane@...

DIM_DATE (Calendar on the wall - one row per day):

date_key | full_date | day_name | month | quarter | year | is_weekend
20241115 | 2024-11-15 | Friday | Nov | Q4 | 2024 | N
20241116 | 2024-11-16 | Saturday | Nov | Q4 | 2024 | Y

FACT_ORDERS (Fridge/counter - events that happen constantly):

order_key | date_key | customer_key | product_key | amount
1 | 20241115 | 1001 | 5 | 25
2 | 20241115 | 1001 | 8 | 3

Customer details stored one time. Not a thousand times. When you need to answer ‘how much did we sell?’, you scan the compact fact table. When you need ‘which customers bought?’, you JOIN to get the descriptive details.

Notice something important in that structure? The dimension tables have two types of keys:

customer_key (1001, 1002) — This is a surrogate key. A meaningless integer we generate ourselves.
customer_id (C-123, C-456) — This is the natural key. The ID from the source system.

Why both? Because source systems are chaotic. They recycle IDs. They change formats. They get merged in acquisitions. The surrogate key is our key—stable, predictable, and under our control. Every dimension table gets a surrogate key as its primary key.

And that date dimension? It’s “the most important dimension.” Every fact table should have one. It lets you slice data by day of week, month, quarter, fiscal period, holidays—without calculating these on every query.

The core principle: things that rarely change go in dedicated storage, organised and easy to find. Things that happen constantly go in their own space, linked back by keys. Same rule applies to data:

Dimensions = Stable descriptive stuff (customers, products, locations, dates)
Facts = Events that happen constantly (orders, clicks, updates, measurements)

The Conference Room I’ll Never Forget

I’m sitting in a conference room. Fluorescent lights buzzing. That weird hum they make when it’s too quiet—I’m wondering why we haven’t changed to LEDs. My manager is standing at the whiteboard, and she’s pissed.

“Someone,” she says, not looking at anyone in particular, “pushed bad data to production last night.”

I feel my stomach drop. The data pipeline my team had been working on.

“The revenue report that went to the board this morning showed every transaction as zero dollars. Zero. The board emailed me asking if the company made any money last quarter.”

Oh god. Oh no. I sank down in my chair a little.

Turns out it wasn’t actually our release—it was someone else’s. But in that moment, sitting there, I realised something crucial: we had no safety net. Data went straight from transformation to production. No checks. No validation. If you made a mistake, everyone saw it immediately.

That conference room moment stuck with me. I started researching how other teams prevented this kind of disaster. That’s when I learned about Write-Audit-Publish. And honestly? It’s probably saved me from that same sick feeling countless times since.

Here’s the idea:

WRITE — You finish your transformation. Instead of putting the data straight into production, you write it to a staging branch. Think of it like a draft folder for emails. It’s there, but it’s not official yet.

AUDIT — This is the crucial part. You run every check you can think of:

Did I lose any rows?
Are there any NULL values where there shouldn’t be?
Do all the foreign keys match up?
Does the math make sense?
Do surrogate keys have duplicates?

It’s like re-reading an important email before you hit send. You check for typos, you verify the recipients, you make sure the attachment is right.

Important: In the future when you find incidents, bugs, or problems—add them to this AUDIT layer. Ensure you don’t make the same mistakes twice.

PUBLISH — Only if everything passes, you merge the branch to production. If anything fails, you stop. You fix it. You try again. The bad data never reaches production. Never reaches that board meeting.

The traditional way to implement this involves copying data between folders or tables. But there’s a better approach using Apache Iceberg’s branching feature. Instead of physically copying data, you create a branch—like in Git. Your audit runs against the branch. If it passes, you merge the branch to main. If it fails, you drop the branch. No data was ever at risk.

Critical design decision: WAP should happen at each table level, not just at the end of the pipeline. Each dimension job does its own Write-Audit-Publish cycle. Each fact job does its own. This way:

Failures are isolated to the specific table that failed
You catch issues early, not at the very end
Each job is self-contained and independently testable
Downstream jobs read from published (main) tables, not staging branches

Let me show you what this looks like when an audit catches something:

AUDIT PHASE (dim_system):

Row count check.......................... PASS
 Source: 1,198 rows
 Target: 1,198 rows

Primary key uniqueness................... FAIL
 Found 13 duplicate system_keys in dim_system!

AUDIT FAILED — BRANCH NOT MERGED

Data remains in staging branch.
Production table unchanged.
Fix the duplicates and re-run the job.

Look at that. The audit caught duplicate surrogate keys before they reached production. Nobody sees this except you. You fix the problem. You run it again. Then it passes, and the branch merges to main.

This is Write-Audit-Publish. It’s the difference between sending an email with a typo versus catching it first. Between serving undercooked food versus checking with a thermometer. Between publishing bad data versus validating before it matters.

That conference room moment? Doesn’t happen with this pattern.

Building This Thing for Real

Alright, enough theory. Let me show you how to actually build this in AWS.

We’re going to use real (game) space exploration data—systems, planets, updates. It’s way more interesting than order data, and the patterns work exactly the same.

In this section:

Architecture Overview
Step 1: Setting Up S3 Buckets
Step 2: Creating the Iceberg Tables
The Glue Jobs (with Per-Table WAP)
Step Functions Workflow
Dependency Triggers with S3 Events

Architecture Overview

Here’s our architecture:

 +------------------------------------------+
 | AWS Step Functions Workflow |
 +------------------------------------------+
 |
 +----------------------------+----------------------------+
 | | |
 v v v
 +------------+ +------------+ +------------+
 | Glue Job: | | Glue Job: | | Glue Job: |
 | dim_date | | dim_system | | dim_planet |
 | (WAP) | | (WAP) | | (WAP) |
 +------------+ +------------+ +------------+
 | | |
 +----------------------------+----------------------------+
 |
 (Wait for all dimensions)
 |
 v
 +------------------------+
 | Glue Job: |
 | fact_system_updates |
 | (WAP + Incremental) |
 +------------------------+

Data Flow:

Landing Layer	Staging Layer (PSA)	Semantic Layer
Raw 1:1 from source	Iceberg + data_date partitioning	Production dimensional model
`systems/`	`stg_systems/`	`dim_date`
`planets/`	`stg_planets/`	`dim_system`
`updates/`	`stg_updates/`	`dim_planet`
		`fact_system_updates`

Let me explain the three layers:

Landing Layer — Raw data, 1:1 from source systems. No transformations except what’s needed to land it. Contains only the latest fetch. If sourcing fails, this layer fails. Get the data close to you as quickly as possible (into your ecosystem). Typically all the files will be string or varchar to mitigate data type issues from a source system.

Staging Layer (Persistent Staging Area) — This is where we convert data types and prepare for dimensional modeling. Critically, we partition by data_date—the date the source data was fetched. This gives us:

Historical reprocessing capability (reload any past date)
Debugging (what did the data look like on January 15th?)
Dependency triggers (when a partition is written, trigger downstream jobs)

Semantic Layer — Production-ready dimensional model. This is where business users query. Tables are partitioned appropriately for query performance.

Also notice the job structure: each job does its own WAP cycle. Dimensions build in parallel, each publishing to main when their audits pass. Fact tables wait for all dimensions to publish, then do their own WAP cycle. This ensures:

Fact tables read from published dimensions (main branch), not staging branches
Foreign key lookups use production-ready surrogate keys
Each job is independently testable

Step 1: Setting Up S3 Buckets

I’m in the S3 console. Click “Create bucket”. I’m calling mine space-data-pipeline-demo.

Important settings:

Region: ap-southeast-2 (Sydney—pick one close to you)
Block all public access: Yes
Versioning: Enabled
Default encryption: Enabled

Inside this bucket:

space-data-pipeline-demo/
├── landing/ ← Raw 1:1 from source (latest only)
│ ├── systems/
│ ├── planets/
│ └── updates/
├── staging/ ← PSA with data_date partitioning
│ ├── stg_systems/
│ ├── stg_planets/
│ └── stg_updates/
└── semantic/ ← Production dimensional model
 ├── dim_date/
 ├── dim_system/
 ├── dim_planet/
 └── fact_system_updates/

Step 2: Creating the Iceberg Tables

Here’s the DDL for our dimensional model. Notice the partitioning strategies and SCD Type 2 columns.

Click to expand: dim_date DDL

This is a reference dimension—typically loaded once with all dates, then rarely updated. Partitioned by year for query performance.

CREATE TABLE space_exploration.dim_date (
 date_key INT,
 full_date DATE,
 day_of_week STRING,
 day_of_month INT,
 day_of_year INT,
 week_of_year INT,
 month_number INT,
 month_name STRING,
 quarter INT,
 year INT,
 is_weekend STRING,
 is_holiday STRING,
 fiscal_quarter INT,
 fiscal_year INT,
 -- Audit columns
 load_timestamp TIMESTAMP,
 source_system STRING
)
PARTITIONED BY (year)
LOCATION 's3://space-data-pipeline-demo/semantic/dim_date/'
TBLPROPERTIES (
 'table_type' = 'ICEBERG',
 'format' = 'parquet'
);

Click to expand: dim_system DDL (SCD Type 2)

Tracks history with effective_date/expiration_date. Partitioned by effective_year for efficient history queries.

CREATE TABLE space_exploration.dim_system (
 system_key INT, -- Surrogate key (generated)
 system_id STRING, -- Natural key (from source)
 system_name STRING,
 sector STRING,
 main_star_type STRING,
 coordinates_x DOUBLE,
 coordinates_y DOUBLE,
 coordinates_z DOUBLE,
 discovery_date DATE,
 exploration_status STRING,
 -- SCD Type 2 columns
 effective_date DATE,
 expiration_date DATE,
 current_flag STRING, -- 'Y' or 'N'
 -- Audit columns
 load_timestamp TIMESTAMP,
 source_system STRING,
 -- Partition column
 effective_year INT
)
PARTITIONED BY (effective_year)
LOCATION 's3://space-data-pipeline-demo/semantic/dim_system/'
TBLPROPERTIES (
 'table_type' = 'ICEBERG',
 'format' = 'parquet'
);

Click to expand: dim_planet DDL (SCD Type 2)

Foreign key to dim_system uses the surrogate key. Partitioned by effective_year.

CREATE TABLE space_exploration.dim_planet (
 planet_key INT, -- Surrogate key
 planet_id STRING, -- Natural key
 system_key INT, -- FK to dim_system (surrogate!)
 planet_name STRING,
 planet_type STRING,
 orbital_period DOUBLE,
 has_atmosphere STRING,
 habitability_score DOUBLE,
 -- SCD Type 2 columns
 effective_date DATE,
 expiration_date DATE,
 current_flag STRING,
 -- Audit columns
 load_timestamp TIMESTAMP,
 source_system STRING,
 -- Partition column
 effective_year INT
)
PARTITIONED BY (effective_year)
LOCATION 's3://space-data-pipeline-demo/semantic/dim_planet/'
TBLPROPERTIES (
 'table_type' = 'ICEBERG',
 'format' = 'parquet'
);

Click to expand: fact_system_updates DDL

Incremental fact table partitioned by date_year. Loaded with delete-then-insert pattern.

CREATE TABLE space_exploration.fact_system_updates (
 update_key BIGINT, -- Surrogate key
 date_key INT, -- FK to dim_date
 system_key INT, -- FK to dim_system (current surrogate!)
 update_timestamp TIMESTAMP, -- Original precision preserved
 status STRING,
 ships_present INT, -- Additive fact
 resource_level DOUBLE, -- Additive fact
 -- Audit columns
 load_timestamp TIMESTAMP,
 source_system STRING,
 data_date DATE, -- Source data date (for reprocessing)
 -- Partition column
 date_year INT
)
PARTITIONED BY (date_year)
LOCATION 's3://space-data-pipeline-demo/semantic/fact_system_updates/'
TBLPROPERTIES (
 'table_type' = 'ICEBERG',
 'format' = 'parquet'
);

Click to expand: Staging Tables DDL (Persistent Staging Area)

Partitioned by data_date for historical reprocessing.

CREATE TABLE space_exploration.stg_systems (
 system_id STRING,
 system_name STRING,
 sector STRING,
 main_star_type STRING,
 coordinates_x DOUBLE,
 coordinates_y DOUBLE,
 coordinates_z DOUBLE,
 discovery_date DATE,
 exploration_status STRING,
 -- PSA columns
 data_date DATE,
 load_timestamp TIMESTAMP
)
PARTITIONED BY (data_date)
LOCATION 's3://space-data-pipeline-demo/staging/stg_systems/'
TBLPROPERTIES ('table_type' = 'ICEBERG', 'format' = 'parquet');

CREATE TABLE space_exploration.stg_planets (
 planet_id STRING,
 system_id STRING,
 planet_name STRING,
 planet_type STRING,
 orbital_period DOUBLE,
 has_atmosphere STRING,
 habitability_score DOUBLE,
 -- PSA columns
 data_date DATE,
 load_timestamp TIMESTAMP
)
PARTITIONED BY (data_date)
LOCATION 's3://space-data-pipeline-demo/staging/stg_planets/'
TBLPROPERTIES ('table_type' = 'ICEBERG', 'format' = 'parquet');

CREATE TABLE space_exploration.stg_updates (
 system_id STRING,
 update_timestamp TIMESTAMP,
 status STRING,
 ships_present INT,
 resource_level DOUBLE,
 -- PSA columns
 data_date DATE,
 load_timestamp TIMESTAMP
)
PARTITIONED BY (data_date)
LOCATION 's3://space-data-pipeline-demo/staging/stg_updates/'
TBLPROPERTIES ('table_type' = 'ICEBERG', 'format' = 'parquet');

Key design decisions:

dim_date: Partitioned by year. This is a reference dimension—typically loaded once with all dates from 1900-01-01 to 9999-12-31, then rarely updated.
dim_system / dim_planet: SCD Type 2 with effective_date, expiration_date, current_flag. Partitioned by effective_year for efficient history queries. Updated daily.
fact_system_updates: Partitioned by date_year. Loaded incrementally using delete-then-insert pattern.
Staging tables (PSA): Partitioned by data_date. This lets us reprocess any historical date and provides dependency triggers.

The Heart of the System: Orchestrated Glue Jobs with Per-Table WAP

Each job does its own Write-Audit-Publish cycle. Let me walk you through each one.

Job 1: Build Date Dimension (Reference Dimension)

The date dimension is special—it’s a reference dimension. We generate it ourselves with all calendar attributes, typically as a one-time load. You’d only rerun this if you need to extend the date range or add new attributes (like new holidays).

Click to expand: build_dim_date Glue Job

This job generates a complete date dimension from 2000-01-01 to 2050-12-31 (for demo purposes—production would use a wider range). It creates date attributes like day_of_week, quarter, fiscal_year, and is_weekend, then runs WAP to validate before publishing.

# Glue Job: build_dim_date
# This is typically a ONE-TIME RUN to populate all dates
# Rerun only when extending date range or adding attributes

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *
from pyspark.sql.types import *
from datetime import datetime, timedelta

# Initialize
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

print("="*60)
print("Building Date Dimension (Reference Dimension)")
print("This is typically a ONE-TIME load")
print("="*60)

# Generate comprehensive date range
# Standard practice: 1900-01-01 to 9999-12-31
start_date = datetime(1900, 1, 1)
end_date = datetime(9999, 12, 31)

# For demo purposes, let's use a more reasonable range
# In production, you'd use the full range
start_date = datetime(2000, 1, 1)
end_date = datetime(2050, 12, 31)

print(f"Generating dates from {start_date.date()} to {end_date.date()}")

date_list = []
current = start_date
while current <= end_date:
 date_list.append((current,))
 current += timedelta(days=1)

date_df = spark.createDataFrame(date_list, ["full_date"])

# Build dimension with rich attributes
dim_date = date_df.select(
 # Surrogate key: YYYYMMDD format
 (year("full_date") * 10000 +
 month("full_date") * 100 +
 dayofmonth("full_date")).cast("int").alias("date_key"),

 col("full_date"),
 date_format("full_date", "EEEE").alias("day_of_week"),
 dayofmonth("full_date").alias("day_of_month"),
 dayofyear("full_date").alias("day_of_year"),
 weekofyear("full_date").alias("week_of_year"),
 month("full_date").alias("month_number"),
 date_format("full_date", "MMMM").alias("month_name"),
 quarter("full_date").alias("quarter"),
 year("full_date").alias("year"),

 # Business-friendly flags
 when(dayofweek("full_date").isin([1, 7]), "Y")
 .otherwise("N").alias("is_weekend"),
 lit("N").alias("is_holiday"), # Populate from holiday calendar

 # Fiscal calendar (assuming fiscal year = calendar year)
 quarter("full_date").alias("fiscal_quarter"),
 year("full_date").alias("fiscal_year"),

 # Audit columns
 current_timestamp().alias("load_timestamp"),
 lit("GENERATED").alias("source_system")
)

row_count = dim_date.count()
print(f"Generated {row_count:,} date rows")

# ============================================
# WRITE PHASE: Write to staging branch
# ============================================
print("\n--- WRITE PHASE ---")
print("Writing to staging branch...")

spark.sql("ALTER TABLE space_exploration.dim_date CREATE BRANCH IF NOT EXISTS staging")

dim_date.writeTo("space_exploration.dim_date") \
 .option("branch", "staging") \
 .overwritePartitions()

print("Written to staging branch")

# ============================================
# AUDIT PHASE: Validate before publishing
# ============================================
print("\n--- AUDIT PHASE ---")

staging_df = spark.read \
 .option("branch", "staging") \
 .table("space_exploration.dim_date")

audit_failures = []

# Audit 1: Row count
staged_count = staging_df.count()
print(f"Audit 1 - Row count: {staged_count:,}")
if staged_count == 0:
 audit_failures.append("FAIL: dim_date is empty")

# Audit 2: Primary key uniqueness
unique_keys = staging_df.select("date_key").distinct().count()
if staged_count != unique_keys:
 duplicates = staged_count - unique_keys
 audit_failures.append(f"FAIL: {duplicates} duplicate date_keys")
 print(f"Audit 2 - PK uniqueness: FAIL ({duplicates} duplicates)")
else:
 print(f"Audit 2 - PK uniqueness: PASS (all {unique_keys:,} unique)")

# Audit 3: No NULLs in critical fields
null_count = staging_df.filter(
 col("date_key").isNull() | col("full_date").isNull()
).count()
if null_count > 0:
 audit_failures.append(f"FAIL: {null_count} NULL values in critical fields")
 print(f"Audit 3 - NULL check: FAIL ({null_count} NULLs)")
else:
 print("Audit 3 - NULL check: PASS")

# Audit 4: Date range coverage
date_range = staging_df.agg(
 min("full_date").alias("min_date"),
 max("full_date").alias("max_date")
).collect()[0]
print(f"Audit 4 - Date range: {date_range['min_date']} to {date_range['max_date']}")

# ============================================
# PUBLISH PHASE: Merge to main if audits pass
# ============================================
print("\n--- PUBLISH PHASE ---")

if len(audit_failures) > 0:
 print("\nAUDIT FAILED:")
 for failure in audit_failures:
 print(f" {failure}")
 print("\nBranch NOT merged. Production unchanged.")
 spark.sql("ALTER TABLE space_exploration.dim_date DROP BRANCH staging")
 raise Exception(f"Audit failed with {len(audit_failures)} errors")
else:
 print("\nALL AUDITS PASSED")
 print("Merging staging branch to main...")

 # Fast-forward main to staging
 spark.sql("""
 CALL system.fast_forward(
 table => 'space_exploration.dim_date',
 branch => 'main',
 to => 'staging'
 )
 """)

 # Clean up staging branch
 spark.sql("ALTER TABLE space_exploration.dim_date DROP BRANCH staging")

 print(f"PUBLISHED: dim_date ({staged_count:,} rows)")

print("\n" + "="*60)
print("dim_date job complete")
print("="*60)

job.commit()

Job 2: Build System Dimension (SCD Type 2)

This dimension tracks history. When a system’s attributes change (like exploration_status), we expire the old row and create a new one.

Click to expand: build_dim_system Glue Job

This job implements SCD Type 2 change detection. It compares incoming data against the current dimension, identifies NEW/CHANGED/UNCHANGED records, expires old versions, and creates new versions with updated surrogate keys.

# Glue Job: build_dim_system
# SCD Type 2 implementation with effective/expiration dates

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *
from pyspark.sql.window import Window

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'DATA_DATE'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

data_date = args['DATA_DATE'] # e.g., '2026-01-17'
print("="*60)
print(f"Building System Dimension (SCD Type 2)")
print(f"Data Date: {data_date}")
print("="*60)

# ============================================
# LOAD: Read from Persistent Staging Area
# ============================================
print("\n--- LOAD PHASE ---")

# Read today's staged data (already landed in PSA)
source_df = spark.read.table("space_exploration.stg_systems") \
 .filter(col("data_date") == data_date)

source_count = source_df.count()
print(f"Source rows from PSA (data_date={data_date}): {source_count:,}")

if source_count == 0:
 print("No data for this date. Exiting.")
 job.commit()
 sys.exit(0)

# Read current production dimension (for SCD2 comparison)
try:
 current_dim = spark.read.table("space_exploration.dim_system") \
 .filter(col("current_flag") == "Y")
 current_count = current_dim.count()
 print(f"Current dimension rows (current_flag='Y'): {current_count:,}")
 has_existing_data = current_count > 0
except:
 print("No existing dimension data (first load)")
 has_existing_data = False
 current_dim = None

# ============================================
# TRANSFORM: SCD Type 2 Logic
# ============================================
print("\n--- TRANSFORM PHASE ---")

if not has_existing_data:
 # First load: all rows are new
 print("First load - all rows are INSERT")

 window_spec = Window.orderBy("system_id")

 new_dim = source_df.select(
 row_number().over(window_spec).alias("system_key"),
 col("system_id"),
 col("system_name"),
 col("sector"),
 col("main_star_type"),
 col("coordinates_x"),
 col("coordinates_y"),
 col("coordinates_z"),
 col("discovery_date"),
 col("exploration_status"),
 lit(data_date).cast("date").alias("effective_date"),
 lit("9999-12-31").cast("date").alias("expiration_date"),
 lit("Y").alias("current_flag"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 year(lit(data_date)).alias("effective_year")
 ).distinct()

else:
 # Incremental load: detect changes for SCD2
 print("Incremental load - detecting changes for SCD2")

 # Define which columns to track for changes
 tracked_columns = [
 "system_name", "sector", "main_star_type",
 "coordinates_x", "coordinates_y", "coordinates_z",
 "exploration_status"
 ]

 # Join source with current to detect changes
 comparison = source_df.alias("src").join(
 current_dim.alias("curr"),
 col("src.system_id") == col("curr.system_id"),
 "full_outer"
 )

 # Identify: NEW (no match in current), CHANGED (values differ), UNCHANGED
 change_detection = comparison.withColumn(
 "change_type",
 when(col("curr.system_id").isNull(), "NEW")
 .when(col("src.system_id").isNull(), "DELETED")
 .when(
 (col("src.system_name") != col("curr.system_name")) |
 (col("src.sector") != col("curr.sector")) |
 (col("src.main_star_type") != col("curr.main_star_type")) |
 (col("src.exploration_status") != col("curr.exploration_status")) |
 (abs(col("src.coordinates_x") - col("curr.coordinates_x")) > 0.001) |
 (abs(col("src.coordinates_y") - col("curr.coordinates_y")) > 0.001) |
 (abs(col("src.coordinates_z") - col("curr.coordinates_z")) > 0.001),
 "CHANGED"
 )
 .otherwise("UNCHANGED")
 )

 # Count changes
 change_counts = change_detection.groupBy("change_type").count().collect()
 for row in change_counts:
 print(f" {row['change_type']}: {row['count']:,}")

 # Get max surrogate key for new key generation
 max_key = current_dim.agg(max("system_key")).collect()[0][0] or 0

 # Build new dimension rows for NEW and CHANGED records
 new_records = change_detection.filter(col("change_type").isin(["NEW", "CHANGED"]))

 window_spec = Window.orderBy("src.system_id")

 new_rows = new_records.select(
 (row_number().over(window_spec) + max_key).alias("system_key"),
 col("src.system_id").alias("system_id"),
 col("src.system_name").alias("system_name"),
 col("src.sector").alias("sector"),
 col("src.main_star_type").alias("main_star_type"),
 col("src.coordinates_x").alias("coordinates_x"),
 col("src.coordinates_y").alias("coordinates_y"),
 col("src.coordinates_z").alias("coordinates_z"),
 col("src.discovery_date").alias("discovery_date"),
 col("src.exploration_status").alias("exploration_status"),
 lit(data_date).cast("date").alias("effective_date"),
 lit("9999-12-31").cast("date").alias("expiration_date"),
 lit("Y").alias("current_flag"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 year(lit(data_date)).alias("effective_year")
 )

 # Build expired rows (for CHANGED records, expire the old version)
 changed_ids = change_detection.filter(col("change_type") == "CHANGED") \
 .select(col("curr.system_id").alias("system_id"))

 expired_rows = current_dim.join(changed_ids, "system_id") \
 .withColumn("expiration_date", lit(data_date).cast("date") - 1) \
 .withColumn("current_flag", lit("N"))

 # Unchanged rows stay as-is
 unchanged_rows = current_dim.join(
 change_detection.filter(col("change_type") == "UNCHANGED")
 .select(col("curr.system_id").alias("system_id")),
 "system_id"
 )

 # Historical rows (already expired) - read from full table
 historical_rows = spark.read.table("space_exploration.dim_system") \
 .filter(col("current_flag") == "N")

 # Combine all
 new_dim = new_rows \
 .unionByName(expired_rows, allowMissingColumns=True) \
 .unionByName(unchanged_rows, allowMissingColumns=True) \
 .unionByName(historical_rows, allowMissingColumns=True)

transform_count = new_dim.count()
print(f"Total dimension rows after SCD2: {transform_count:,}")

# ============================================
# WRITE PHASE: Write to staging branch
# ============================================
print("\n--- WRITE PHASE ---")

spark.sql("ALTER TABLE space_exploration.dim_system CREATE BRANCH IF NOT EXISTS staging")

new_dim.writeTo("space_exploration.dim_system") \
 .option("branch", "staging") \
 .overwritePartitions()

print("Written to staging branch")

# ============================================
# AUDIT PHASE
# ============================================
print("\n--- AUDIT PHASE ---")

staging_df = spark.read \
 .option("branch", "staging") \
 .table("space_exploration.dim_system")

audit_failures = []

# Audit 1: Row count
staged_count = staging_df.count()
print(f"Audit 1 - Row count: {staged_count:,}")
if staged_count == 0:
 audit_failures.append("FAIL: dim_system is empty")

# Audit 2: Surrogate key uniqueness
unique_keys = staging_df.select("system_key").distinct().count()
if staged_count != unique_keys:
 duplicates = staged_count - unique_keys
 audit_failures.append(f"FAIL: {duplicates} duplicate system_keys")
 print(f"Audit 2 - PK uniqueness: FAIL ({duplicates} duplicates)")
else:
 print(f"Audit 2 - PK uniqueness: PASS")

# Audit 3: Only one current row per natural key
current_rows = staging_df.filter(col("current_flag") == "Y")
current_count = current_rows.count()
unique_natural_keys = current_rows.select("system_id").distinct().count()
if current_count != unique_natural_keys:
 audit_failures.append(f"FAIL: Multiple current rows for same system_id")
 print(f"Audit 3 - Single current: FAIL")
else:
 print(f"Audit 3 - Single current: PASS ({current_count:,} current rows)")

# Audit 4: No NULLs in critical fields
null_count = staging_df.filter(
 col("system_key").isNull() |
 col("system_id").isNull() |
 col("effective_date").isNull() |
 col("current_flag").isNull()
).count()
if null_count > 0:
 audit_failures.append(f"FAIL: {null_count} NULL values in critical fields")
 print(f"Audit 4 - NULL check: FAIL")
else:
 print("Audit 4 - NULL check: PASS")

# Audit 5: Effective/expiration date logic
invalid_dates = staging_df.filter(
 col("effective_date") > col("expiration_date")
).count()
if invalid_dates > 0:
 audit_failures.append(f"FAIL: {invalid_dates} rows with effective > expiration")
 print(f"Audit 5 - Date logic: FAIL")
else:
 print("Audit 5 - Date logic: PASS")

# ============================================
# PUBLISH PHASE
# ============================================
print("\n--- PUBLISH PHASE ---")

if len(audit_failures) > 0:
 print("\nAUDIT FAILED:")
 for failure in audit_failures:
 print(f" {failure}")
 print("\nBranch NOT merged. Production unchanged.")
 spark.sql("ALTER TABLE space_exploration.dim_system DROP BRANCH staging")
 raise Exception(f"Audit failed with {len(audit_failures)} errors")
else:
 print("\nALL AUDITS PASSED")
 print("Merging staging branch to main...")

 spark.sql("""
 CALL system.fast_forward(
 table => 'space_exploration.dim_system',
 branch => 'main',
 to => 'staging'
 )
 """)

 spark.sql("ALTER TABLE space_exploration.dim_system DROP BRANCH staging")

 print(f"PUBLISHED: dim_system ({staged_count:,} rows)")

job.commit()

Job 3: Build Planet Dimension (SCD Type 2)

Similar pattern to dimsystem, but with a foreign key lookup to the _published dim_system.

Click to expand: build_dim_planet Glue Job

This job demonstrates FK lookup to a published dimension. It reads from the main branch of dim_system (not staging), ensuring it uses production-ready surrogate keys. The FK integrity audit validates this relationship.

# Glue Job: build_dim_planet
# SCD Type 2 with FK lookup to published dim_system

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *
from pyspark.sql.window import Window

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'DATA_DATE'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

data_date = args['DATA_DATE']
print("="*60)
print(f"Building Planet Dimension (SCD Type 2)")
print(f"Data Date: {data_date}")
print("="*60)

# ============================================
# LOAD PHASE
# ============================================
print("\n--- LOAD PHASE ---")

source_df = spark.read.table("space_exploration.stg_planets") \
 .filter(col("data_date") == data_date)

source_count = source_df.count()
print(f"Source rows: {source_count:,}")

if source_count == 0:
 print("No data. Exiting.")
 job.commit()
 sys.exit(0)

# CRITICAL: Read from PUBLISHED dim_system (main branch, not staging)
# This is why dim_planet must run AFTER dim_system publishes
dim_system = spark.read.table("space_exploration.dim_system") \
 .filter(col("current_flag") == "Y") \
 .select("system_key", "system_id")

print(f"dim_system lookup rows: {dim_system.count():,}")

# ============================================
# TRANSFORM PHASE
# ============================================
print("\n--- TRANSFORM PHASE ---")

# Check for existing data
try:
 current_dim = spark.read.table("space_exploration.dim_planet") \
 .filter(col("current_flag") == "Y")
 has_existing = current_dim.count() > 0
 max_key = current_dim.agg(max("planet_key")).collect()[0][0] or 0
except:
 has_existing = False
 max_key = 0

# Join source with dim_system to get surrogate key
source_with_sk = source_df.join(
 dim_system,
 source_df.system_id == dim_system.system_id,
 "left"
).select(
 source_df["*"],
 dim_system.system_key
)

# Track orphaned planets (no matching system)
orphaned = source_with_sk.filter(col("system_key").isNull()).count()
if orphaned > 0:
 print(f"WARNING: {orphaned} planets have no matching system!")

# SCD2 logic (simplified for first load demonstration)
window_spec = Window.orderBy("planet_id")

new_dim = source_with_sk.select(
 (row_number().over(window_spec) + max_key).alias("planet_key"),
 col("planet_id"),
 col("system_key"),
 col("planet_name"),
 col("planet_type"),
 col("orbital_period"),
 col("has_atmosphere"),
 col("habitability_score"),
 lit(data_date).cast("date").alias("effective_date"),
 lit("9999-12-31").cast("date").alias("expiration_date"),
 lit("Y").alias("current_flag"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 year(lit(data_date)).alias("effective_year")
).distinct()

# (Full SCD2 change detection would follow same pattern as dim_system)

transform_count = new_dim.count()
print(f"Dimension rows: {transform_count:,}")

# ============================================
# WRITE PHASE
# ============================================
print("\n--- WRITE PHASE ---")

spark.sql("ALTER TABLE space_exploration.dim_planet CREATE BRANCH IF NOT EXISTS staging")

new_dim.writeTo("space_exploration.dim_planet") \
 .option("branch", "staging") \
 .overwritePartitions()

print("Written to staging branch")

# ============================================
# AUDIT PHASE
# ============================================
print("\n--- AUDIT PHASE ---")

staging_df = spark.read \
 .option("branch", "staging") \
 .table("space_exploration.dim_planet")

audit_failures = []

# Audit 1: PK uniqueness
staged_count = staging_df.count()
unique_keys = staging_df.select("planet_key").distinct().count()
if staged_count != unique_keys:
 audit_failures.append(f"FAIL: Duplicate planet_keys")
else:
 print(f"Audit 1 - PK uniqueness: PASS")

# Audit 2: FK integrity to dim_system (published!)
published_system_keys = spark.read.table("space_exploration.dim_system") \
 .select("system_key").distinct()

orphaned_fk = staging_df \
 .filter(col("system_key").isNotNull()) \
 .select("system_key").distinct() \
 .join(published_system_keys, "system_key", "left_anti") \
 .count()

if orphaned_fk > 0:
 audit_failures.append(f"FAIL: {orphaned_fk} orphaned system_keys")
 print(f"Audit 2 - FK integrity: FAIL")
else:
 print("Audit 2 - FK integrity: PASS")

# Audit 3: NULL check
null_count = staging_df.filter(
 col("planet_key").isNull() | col("planet_id").isNull()
).count()
if null_count > 0:
 audit_failures.append(f"FAIL: {null_count} NULLs in critical fields")
else:
 print("Audit 3 - NULL check: PASS")

# ============================================
# PUBLISH PHASE
# ============================================
print("\n--- PUBLISH PHASE ---")

if len(audit_failures) > 0:
 print("\nAUDIT FAILED:")
 for f in audit_failures:
 print(f" {f}")
 spark.sql("ALTER TABLE space_exploration.dim_planet DROP BRANCH staging")
 raise Exception(f"Audit failed")
else:
 print("\nALL AUDITS PASSED")
 spark.sql("""
 CALL system.fast_forward(
 table => 'space_exploration.dim_planet',
 branch => 'main',
 to => 'staging'
 )
 """)
 spark.sql("ALTER TABLE space_exploration.dim_planet DROP BRANCH staging")
 print(f"PUBLISHED: dim_planet ({staged_count:,} rows)")

job.commit()

Job 4: Build Fact Table (Incremental with Delete-Insert)

This is the critical job. It reads from published dimensions (main branch), uses delete-then-insert pattern for the date partition, and includes left anti-join as a defensive check for missing records.

Click to expand: build_fact_system_updates Glue Job

This job loads facts incrementally by date. For a given data_date, it deletes existing records and inserts new ones. It validates FK integrity against all published dimensions before merging to main.

# Glue Job: build_fact_system_updates
# Incremental load with delete-then-insert + left anti-join defensive coding

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *
from pyspark.sql.window import Window

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'DATA_DATE'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

data_date = args['DATA_DATE']
print("="*60)
print(f"Building Fact Table: System Updates (Incremental)")
print(f"Data Date: {data_date}")
print("="*60)

# ============================================
# LOAD PHASE
# ============================================
print("\n--- LOAD PHASE ---")

# Read from PSA for this data_date
source_df = spark.read.table("space_exploration.stg_updates") \
 .filter(col("data_date") == data_date)

source_count = source_df.count()
print(f"Source rows: {source_count:,}")

if source_count == 0:
 print("No data. Exiting.")
 job.commit()
 sys.exit(0)

# Read PUBLISHED dimensions (main branch)
# Fact tables MUST read from published dimensions, not staging branches
dim_system = spark.read.table("space_exploration.dim_system") \
 .filter(col("current_flag") == "Y") \
 .select("system_key", "system_id")

dim_date = spark.read.table("space_exploration.dim_date") \
 .select("date_key", "full_date", "year")

print(f"dim_system rows: {dim_system.count():,}")
print(f"dim_date rows: {dim_date.count():,}")

# Read existing fact table for incremental logic
try:
 existing_fact = spark.read.table("space_exploration.fact_system_updates")
 existing_count = existing_fact.count()
 print(f"Existing fact rows: {existing_count:,}")
 max_key = existing_fact.agg(max("update_key")).collect()[0][0] or 0
except:
 print("No existing fact data (first load)")
 existing_fact = None
 max_key = 0

# ============================================
# TRANSFORM PHASE
# ============================================
print("\n--- TRANSFORM PHASE ---")

# Add date column for joining
source_with_date = source_df.withColumn(
 "update_date",
 to_date("update_timestamp")
)

# Lookup surrogate keys from published dimensions
fact_transformed = source_with_date \
 .join(dim_system, source_with_date.system_id == dim_system.system_id, "left") \
 .join(dim_date, source_with_date.update_date == dim_date.full_date, "left") \
 .select(
 col("date_key"),
 dim_system.system_key,
 source_with_date.update_timestamp,
 col("status"),
 col("ships_present"),
 col("resource_level"),
 lit(data_date).cast("date").alias("data_date"),
 dim_date.year.alias("date_year")
 )

# Track orphaned facts
orphaned_systems = fact_transformed.filter(col("system_key").isNull()).count()
orphaned_dates = fact_transformed.filter(col("date_key").isNull()).count()

if orphaned_systems > 0:
 print(f"WARNING: {orphaned_systems} facts have no matching system")
if orphaned_dates > 0:
 print(f"WARNING: {orphaned_dates} facts have no matching date")

# ============================================
# INCREMENTAL LOGIC: Delete-then-Insert + Left Anti-Join
# ============================================
print("\n--- INCREMENTAL LOGIC ---")

# Extract the date partition we're loading
date_partition_year = int(data_date[:4]) # e.g., 2026

if existing_fact is not None:
 # Step 1: Get existing records for this data_date
 existing_for_date = existing_fact.filter(col("data_date") == data_date)
 existing_for_date_count = existing_for_date.count()
 print(f"Existing records for data_date={data_date}: {existing_for_date_count:,}")

 # Step 2: Records NOT in this date (keep as-is)
 records_to_keep = existing_fact.filter(col("data_date") != data_date)
 keep_count = records_to_keep.count()
 print(f"Records to keep (other dates): {keep_count:,}")

 # Step 3: New records from source (delete-then-insert for this date)
 # Generate new surrogate keys starting after max existing key
 window_spec = Window.orderBy("update_timestamp", "system_key")

 new_records = fact_transformed.select(
 (row_number().over(window_spec) + max_key).alias("update_key"),
 col("date_key"),
 col("system_key"),
 col("update_timestamp"),
 col("status"),
 col("ships_present"),
 col("resource_level"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 col("data_date"),
 col("date_year")
 )

 new_count = new_records.count()
 print(f"New records for data_date={data_date}: {new_count:,}")

 # Step 4: Left anti-join defensive check
 # Find any records in destination that should exist but don't
 # (This catches edge cases where source has records not in our transform)
 # For this example, we're doing a full replace for the date, so this is informational

 # Step 5: Combine: kept records + new records (effectively delete-insert)
 final_fact = records_to_keep.unionByName(new_records)

else:
 # First load: all records are new
 window_spec = Window.orderBy("update_timestamp", "system_key")

 final_fact = fact_transformed.select(
 row_number().over(window_spec).alias("update_key"),
 col("date_key"),
 col("system_key"),
 col("update_timestamp"),
 col("status"),
 col("ships_present"),
 col("resource_level"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 col("data_date"),
 col("date_year")
 )

transform_count = final_fact.count()
print(f"Final fact rows: {transform_count:,}")

# ============================================
# WRITE PHASE
# ============================================
print("\n--- WRITE PHASE ---")

spark.sql("ALTER TABLE space_exploration.fact_system_updates CREATE BRANCH IF NOT EXISTS staging")

final_fact.writeTo("space_exploration.fact_system_updates") \
 .option("branch", "staging") \
 .overwritePartitions()

print("Written to staging branch")

# ============================================
# AUDIT PHASE
# ============================================
print("\n--- AUDIT PHASE ---")

staging_df = spark.read \
 .option("branch", "staging") \
 .table("space_exploration.fact_system_updates")

audit_failures = []

# Audit 1: Row count (source to target for this date)
staged_for_date = staging_df.filter(col("data_date") == data_date).count()
print(f"Audit 1 - Row count for {data_date}: Source={source_count:,}, Target={staged_for_date:,}")
if source_count != staged_for_date:
 # This might be OK if we filtered orphans, but flag for review
 print(f" Note: Difference of {source_count - staged_for_date} (likely orphaned records)")

# Audit 2: PK uniqueness
staged_count = staging_df.count()
unique_keys = staging_df.select("update_key").distinct().count()
if staged_count != unique_keys:
 audit_failures.append(f"FAIL: Duplicate update_keys")
 print(f"Audit 2 - PK uniqueness: FAIL")
else:
 print(f"Audit 2 - PK uniqueness: PASS ({staged_count:,} unique)")

# Audit 3: FK integrity to dim_system
published_system_keys = spark.read.table("space_exploration.dim_system") \
 .select("system_key").distinct()

orphaned_fk_system = staging_df \
 .filter(col("system_key").isNotNull()) \
 .select("system_key").distinct() \
 .join(published_system_keys, "system_key", "left_anti") \
 .count()

if orphaned_fk_system > 0:
 audit_failures.append(f"FAIL: {orphaned_fk_system} orphaned system_keys")
 print(f"Audit 3 - FK to dim_system: FAIL")
else:
 print("Audit 3 - FK to dim_system: PASS")

# Audit 4: FK integrity to dim_date
published_date_keys = spark.read.table("space_exploration.dim_date") \
 .select("date_key").distinct()

orphaned_fk_date = staging_df \
 .filter(col("date_key").isNotNull()) \
 .select("date_key").distinct() \
 .join(published_date_keys, "date_key", "left_anti") \
 .count()

if orphaned_fk_date > 0:
 audit_failures.append(f"FAIL: {orphaned_fk_date} orphaned date_keys")
 print(f"Audit 4 - FK to dim_date: FAIL")
else:
 print("Audit 4 - FK to dim_date: PASS")

# Audit 5: No NULLs in mandatory FKs
# (We allow NULLs for orphaned records, but track them)
null_system = staging_df.filter(col("system_key").isNull()).count()
null_date = staging_df.filter(col("date_key").isNull()).count()
print(f"Audit 5 - NULL FKs: {null_system} null system_key, {null_date} null date_key")

# Audit 6: Value range checks
invalid_resources = staging_df.filter(
 (col("resource_level") < 0) | (col("resource_level") > 1)
).count()
if invalid_resources > 0:
 audit_failures.append(f"FAIL: {invalid_resources} invalid resource_level values")
 print(f"Audit 6 - Value ranges: FAIL")
else:
 print("Audit 6 - Value ranges: PASS")

# Audit 7: No future timestamps
future_ts = staging_df.filter(col("update_timestamp") > current_timestamp()).count()
if future_ts > 0:
 audit_failures.append(f"FAIL: {future_ts} future timestamps")
 print(f"Audit 7 - Timestamps: FAIL")
else:
 print("Audit 7 - Timestamps: PASS")

# ============================================
# PUBLISH PHASE
# ============================================
print("\n--- PUBLISH PHASE ---")

if len(audit_failures) > 0:
 print("\nAUDIT FAILED:")
 for f in audit_failures:
 print(f" {f}")
 spark.sql("ALTER TABLE space_exploration.fact_system_updates DROP BRANCH staging")
 raise Exception(f"Audit failed with {len(audit_failures)} errors")
else:
 print("\nALL AUDITS PASSED")
 spark.sql("""
 CALL system.fast_forward(
 table => 'space_exploration.fact_system_updates',
 branch => 'main',
 to => 'staging'
 )
 """)
 spark.sql("ALTER TABLE space_exploration.fact_system_updates DROP BRANCH staging")
 print(f"PUBLISHED: fact_system_updates ({staged_count:,} rows)")

job.commit()

Step Functions Workflow

The Step Functions workflow orchestrates everything. Here’s the execution flow:

Stage 1 (Parallel): dim_date and dim_system run simultaneously—neither depends on the other
Stage 2: dim_planet runs after Stage 1 completes (needs dim_system’s surrogate keys)
Stage 3: fact_system_updates runs last (needs all dimension surrogate keys)

If any job fails, the workflow stops and sends an SNS notification.

Click to expand: Step Functions JSON Definition

{
 "Comment": "Dimensional Modeling Pipeline - Per-Table WAP Pattern",
 "StartAt": "Parallel Dimensions Stage 1",
 "States": {
 "Parallel Dimensions Stage 1": {
 "Type": "Parallel",
 "Comment": "dim_date and dim_system have no dependencies",
 "Branches": [
 {
 "StartAt": "Build dim_date (WAP)",
 "States": {
 "Build dim_date (WAP)": {
 "Type": "Task",
 "Resource": "arn:aws:states:::glue:startJobRun.sync",
 "Parameters": {
 "JobName": "build_dim_date"
 },
 "End": true
 }
 }
 },
 {
 "StartAt": "Build dim_system (WAP)",
 "States": {
 "Build dim_system (WAP)": {
 "Type": "Task",
 "Resource": "arn:aws:states:::glue:startJobRun.sync",
 "Parameters": {
 "JobName": "build_dim_system",
 "Arguments": {
 "--DATA_DATE.$": "$.data_date"
 }
 },
 "End": true
 }
 }
 }
 ],
 "Next": "Build dim_planet (WAP)",
 "Catch": [
 {
 "ErrorEquals": ["States.ALL"],
 "Next": "Notify Failure",
 "ResultPath": "$.error"
 }
 ]
 },
 "Build dim_planet (WAP)": {
 "Type": "Task",
 "Comment": "Depends on dim_system being published",
 "Resource": "arn:aws:states:::glue:startJobRun.sync",
 "Parameters": {
 "JobName": "build_dim_planet",
 "Arguments": {
 "--DATA_DATE.$": "$.data_date"
 }
 },
 "Next": "Build fact_system_updates (WAP)",
 "Catch": [
 {
 "ErrorEquals": ["States.ALL"],
 "Next": "Notify Failure",
 "ResultPath": "$.error"
 }
 ]
 },
 "Build fact_system_updates (WAP)": {
 "Type": "Task",
 "Comment": "Depends on ALL dimensions being published",
 "Resource": "arn:aws:states:::glue:startJobRun.sync",
 "Parameters": {
 "JobName": "build_fact_system_updates",
 "Arguments": {
 "--DATA_DATE.$": "$.data_date"
 }
 },
 "Next": "Pipeline Succeeded",
 "Catch": [
 {
 "ErrorEquals": ["States.ALL"],
 "Next": "Notify Failure",
 "ResultPath": "$.error"
 }
 ]
 },
 "Notify Failure": {
 "Type": "Task",
 "Resource": "arn:aws:states:::sns:publish",
 "Parameters": {
 "TopicArn": "arn:aws:sns:ap-southeast-2:ACCOUNT:pipeline-alerts",
 "Message.$": "States.Format('Pipeline failed: {}', $.error.Cause)",
 "Subject": "Data Pipeline Failed"
 },
 "Next": "Pipeline Failed"
 },
 "Pipeline Succeeded": {
 "Type": "Succeed"
 },
 "Pipeline Failed": {
 "Type": "Fail",
 "Error": "PipelineError",
 "Cause": "One or more jobs failed - check individual job logs"
 }
 }
}

Start the workflow with input:

{
 "data_date": "2026-01-17"
}

This data_date propagates to each job, enabling:

PSA partition reads (WHERE data_date = '2026-01-17')
Historical reprocessing (re-run with a past date)
Dependency triggers (S3 event on partition write triggers workflow)

Dependency Triggers with S3 Events

You can improve this further by using partition writes as triggers. Here’s how to set that up:

Option 1: S3 Event → EventBridge → Step Functions

When a file lands in landing/updates/, trigger the pipeline:

# EventBridge Rule
{
 "source": ["aws.s3"],
 "detail-type": ["Object Created"],
 "detail":
 {
 "bucket": { "name": ["space-data-pipeline-demo"] },
 "object": { "key": [{ "prefix": "landing/updates/" }] },
 },
}

Option 2: Glue Trigger on Partition

Configure Glue to watch for new partitions:

# Check if partition exists before running
partition_path = f"s3://space-data-pipeline-demo/staging/stg_updates/data_date={data_date}/"

import boto3
s3 = boto3.client('s3')

try:
 s3.head_object(
 Bucket='space-data-pipeline-demo',
 Key=f'staging/stg_updates/data_date={data_date}/_SUCCESS'
 )
 print(f"Partition exists - proceeding")
except:
 print(f"Partition not ready - exiting")
 sys.exit(0)

This lets you build dependency chains based on partition availability.

The Moment of Truth: Querying

Now let’s query the properly partitioned dimensional model:

-- Query leverages date partition pruning
SELECT
 dt.year,
 dt.quarter,
 d.main_star_type,
 COUNT(*) as updates,
 SUM(f.ships_present) as total_ships,
 ROUND(AVG(f.resource_level), 3) as avg_resources
FROM
 space_exploration.fact_system_updates f

 LEFT JOIN space_exploration.dim_system d
 ON f.system_key = d.system_key
 AND d.current_flag = 'Y' -- Only current dimension records!

 LEFT JOIN space_exploration.dim_date dt
 ON f.date_key = dt.date_key
WHERE
 dt.year = 2026 -- Partition pruning!
 AND dt.quarter = 1
GROUP BY
 dt.year, dt.quarter, d.main_star_type
ORDER BY
 total_ships DESC;

Notice d.current_flag = 'Y' — this is how you query SCD Type 2 dimensions for current state. For point-in-time analysis:

-- Point-in-time analysis: What was system status on Jan 15, 2026?
SELECT
 d.system_name,
 d.exploration_status,
 d.effective_date,
 d.expiration_date
FROM
 space_exploration.dim_system d
WHERE
 d.system_id = 'SOL'
 AND DATE '2026-01-15' BETWEEN d.effective_date AND d.expiration_date;

A Few Things Worth Remembering

WAP at each table, not just at the end. Each job should do its own Write-Audit-Publish cycle. Failures are isolated. Downstream jobs read from published (main) tables.

Landing is raw, Staging is persistent. Landing = 1:1 from source, latest only. Staging (PSA) = partitioned by data_date for historical reprocessing.

dim_date is a reference dimension. Load it once with a wide date range. Partition by year. Rarely needs updates.

SCD Type 2 for dimensions that change. Add effective_date, expiration_date, current_flag. Partition by effective year. Query with current_flag = 'Y' for current state.

Incremental facts with delete-then-insert. For a given date partition: delete existing, insert new. Use left anti-join as defensive check for missing records.

Partition strategically:

dim_date: by year
dim_system/planet: by effective_year (SCD2)
fact: by date_year

Use data_date for dependencies. When stg_updates/data_date=2026-01-17/ is written, trigger the downstream pipeline. This makes dependencies explicit and reprocessing easy.

That ten-minute query? Down to 1.2 seconds with partition pruning.

That conference room moment? Each job’s WAP cycle catches issues before they propagate.

These patterns work. They’ve saved me from embarrassment, late nights, and angry emails. They’ll do the same for you.

The 2026 Data Engineering Strategy Nobody's Writing (But Everyone Needs)

Chris Hillman — Thu, 15 Jan 2026 09:00:00 +1100

What if I told you the biggest threat to your data platform isn’t technology—it’s that we’ve stopped building the next generation of engineers who’ll run it?

Not the latest database that promises to solve everything. Not whether you picked the right orchestrator.

The real crisis is that we’ve systematically broken our talent pipeline. And in 2026, that decision is going to start costing us in ways that no amount of tooling can fix.

But let’s back up. Because if you’re in planning mode right now—building roadmaps, setting budgets, arguing about which technologies to bet on—you’re probably focused on the wrong things. Everyone is.

Here’s what I believe actually matters for 2026—and what will determine whether you’re thriving or scrambling by 2030.

We’re Creating the Next Talent Crisis—On Purpose

This was a conversation:

“We need more engineers. Senior people who can hit the ground running.”

“Agreed. Let’s post the role at six years minimum experience.”

“Why aren’t we getting qualified candidates?”

“Must be a talent shortage. Everyone wants data engineers these days.”

It’s not a shortage. It’s a manufactured crisis.

Entry-level data engineering positions represent just 2% of job postings. Two percent. Meanwhile, roles requiring 6+ years of experience make up nearly 20% of openings. We’ve created an impossible paradox: the industry demands experienced engineers while systematically refusing to create them.

And here’s the thing—data engineering skills aren’t taught in computer science programs. You don’t learn dimensional modeling or pipeline orchestration or data quality frameworks in a classroom. These are skills built through experience, through making mistakes, through having a senior engineer look at your code and say “okay, but what happens when this table has a million rows instead of a thousand?”

I’ve watched this play out across every team I’ve worked with. Companies want plug-and-play professionals, but they won’t invest in training. Fresh graduates realize quickly that “junior data engineer” jobs are about as common as unicorn sightings. The roles exist, technically. They’re just not being hired for.

The retention numbers tell the same story from the other end. Surveys show that 95% of current data engineers report experiencing burnout. Seventy percent are likely to seek new jobs within 12 months. Four out of five are considering leaving the career entirely.

So we’re not hiring juniors, and we’re burning out the seniors we have.

The irony is that hiring juniors actually helps senior engineers. Juniors handle the repetitive stuff—the data quality checks you’ve written a hundred times, the standard ETL patterns you could code in your sleep, the documentation that needs doing but never gets prioritized. This frees up senior time for architecture decisions, complex problem-solving, and the high-value work that actually requires experience.

But more importantly, juniors represent the institutional knowledge that walks out the door when that burned-out senior finally quits. And right now, we’re choosing short-term convenience over long-term sustainability.

When I started in this field, there were 10 candidates per data engineering job listing. Now? You’re lucky to get 2.5. For context, web developers see 10 candidates per job. Marketing managers? Try 53.

This isn’t a future problem. It’s a 2026 problem. And by 2030, projections suggest 10.5 million unfilled data and analytics positions globally.

Your 2030 self is going to be really mad about this decision.

AI Changes Everything (Except What Actually Matters)

And this is where many teams think AI will save them.

A junior engineer on a team needed to build a data quality framework for their pipeline. In 2020, this would have meant three hours of my time walking through the approach, another two hours of their time implementing it, and probably a day of debugging edge cases.

Instead, she spent 20 minutes with Claude. Described the problem, got three different approaches, validated the logic, tested the implementations, and asked for feedback on edge cases I hadn’t even thought to mention. By the time she pinged me, she had working code and specific questions about production deployment.

This is the reality of AI in data engineering in 2026. It’s not replacing engineers. It’s fundamentally changing what “junior” means.

GitHub Copilot has over 15 million users now. Ninety percent of Fortune 100 companies have adopted it. Developers report 51% faster coding speed—and that matches what I see in practice. The text-to-SQL market has matured to where straightforward queries just work. Tools like Vanna.AI and the native integrations in Databricks and Snowflake handle the routine stuff competently.

But here’s what the adoption stats don’t tell you: AI is a force multiplier for experience, not a replacement for it.

I’ve seen junior engineers use AI to cover gaps in their understanding—generating code they can’t debug, creating pipelines they can’t explain, building on top of patterns they don’t actually grasp. And I’ve seen senior engineers use it to remove friction and accelerate decisions they already know how to make.

The difference matters. A lot.

Some organizations now require 85% test coverage for AI-assisted code versus 70% for human-written code. They flag high-AI-content pull requests for additional security review. A few even implement “AI-free Fridays” to prevent skill atrophy. Because the risk isn’t that AI writes bad code—it’s that engineers stop learning how to recognize when it does.

The Spider 2.0 benchmark—a standard test for enterprise-level SQL generation—is instructive here. GPT-4 solves only 6% of enterprise-level SQL questions. Six percent. Not because the models are bad, but because enterprise reality is messy. Tables with a thousand columns, sparse data that doesn’t match the schema documentation, business logic buried in five years of accumulated stored procedures that nobody fully understands anymore.

AI is phenomenal at solving clean, well-defined problems. Enterprise data engineering is rarely clean or well-defined.

The Hive Mind Advantage (You Can Actually Use)

Here’s something that doesn’t get talked about enough: when an AI learns something—like how to write performant SQL, or recognize a common anti-pattern—every instance of that AI learns it simultaneously.

Compare that to human learning. If I figure out a better approach to incremental loads, I share it with my team. Maybe it spreads to 5-10 people. Maybe someone writes a blog post and it reaches a few hundred. But it takes years for a best practice to become industry standard.

AI doesn’t have that limitation. When Claude or GPT gets better at something, every developer using it gets better at it on the same day.

This is where the real opportunity lies for 2026: not fighting against AI or using it as a crutch, but deliberately leveraging that collective learning capability:

Peer review assistance: AI can spot patterns across your entire codebase that no single engineer could hold in their head. It’s seen millions of pipelines fail in similar ways—use that knowledge.
Best practice enforcement: Standards that would require constant human vigilance can be automated. Style guides, naming conventions, common performance pitfalls—AI catches these at scale.
Accelerated learning: Juniors can ask “why is this pattern better?” and get explanations based on millions of examples (specific to your industry, or tapped into real work examples), not just what their one mentor happens to know. The AI has seen every permutation of every mistake.

The engineers who thrive won’t be the ones competing against AI. They’ll be the ones who figured out how to use that hive-mind capability to augment their own judgment.

So yes, AI should be in your 2026 strategy. But if you’re counting on it to solve your junior engineer problem? You’re in for an expensive surprise. Because AI might help juniors be productive faster, but it doesn’t teach them judgment and critical thinking. And in data engineering, this is one of those soft skills that separates the engineers who ship reliable systems from the ones who generate incidents.

The question isn’t “should we use AI”—of course you should. The question is “how do we use AI without creating a generation of engineers who can’t function without it?”

I don’t have a perfect answer to that. But I know it starts with still hiring and training juniors, even though it’s harder and slower than we’d like.

The $180,000 Mistake Hiding in Your Architecture

Here’s a story that’s going to sound familiar.

I got asked to review a friends data platform that was spending a lot of compute. The team was frustrated. “We’re growing fast, costs keep climbing, and we can’t figure out where the money’s going.”

So I looked at their queries. Ninety percent of them scanned less than 100MB of data.

Not gigabytes. Not terabytes. Megabytes. The kind of data that fits comfortably in your laptop’s memory.

They were running a distributed cloud data warehouse—with all the complexity, cost, and operational overhead that entails—to process files that could run on a Raspberry Pi.

This is the emperor-has-no-clothes moment of 2026: most “big data” isn’t actually big.

Jordan Tigani figured this out a few years ago. He’s the former Google BigQuery founding engineer, so he’s not some random person with an axe to grind. He looked at actual BigQuery usage patterns and found that 90% of queries process less than 100MB. Among instances with supposedly “big data,” 98% of queries scan less than 1TB.

His conclusion: “The data cataclysm that had been predicted hasn’t come to pass. Data sizes may have gotten marginally larger, but hardware has gotten bigger at an even faster rate.”

DuckDB is the practical realization of this insight. It’s an embedded columnar database that runs inside Python, R, or just the command line. No server required. No cluster management. When it runs out of memory, it automatically spills to disk. It natively handles Parquet, CSV, and JSON.

For most data engineering work, this means you can develop locally without cloud dependencies, test rapidly in CI/CD without warehouse costs, and handle production workloads up to hundreds of gigabytes on a single machine.

The cost implications are striking. After moving appropriate workloads to DuckDB, teams regularly see 80-90% cost reduction. One team processed pipeline work that was taking 8 hours and got it down to 8 minutes. On cheaper infrastructure.

Polars complements this story nicely—a Rust-based DataFrame library that gives you programmatic workflows where DuckDB gives you SQL. Together, they represent a genuine paradigm shift from “scale out everything” to “right-size your architecture.”

You still need distributed systems for genuinely big data, for complex concurrent workloads, for petabyte-scale scanning. But the skill that matters isn’t knowing DuckDB or Polars—tools change. The skill is architectural judgment: knowing when a single powerful machine beats a distributed cluster. Knowing when to optimize for cost versus scale versus simplicity.

This is what separates senior engineers from junior ones. And it’s why AI can’t replace architectural judgment—the models are trained on a world where “big data” meant Hadoop clusters and distributed everything. They don’t know that world is ending.

What Your 2026 Planning Should Actually Address

If you’re in planning meetings right now, here’s what’s probably on your list:

Which AI tools to adopt
Whether to migrate to a new data warehouse
Cost optimization initiatives
Team headcount requests

And here’s what should be on your list but probably isn’t:

First: How are we creating the next generation of engineers?

Not “when we have time” or “once we’re fully staffed.” Now. Budget for at least one junior engineer per three seniors. Build mentorship into performance reviews. Dedicate senior time to teaching, and treat that as valuable work, not a distraction from “real” work.

Your 2027 self will thank you. Your 2030 self will wonder why this wasn’t obvious to everyone.

Second: Where are we using distributed systems unnecessarily?

Audit your infrastructure. Find the places where you’re using Spark to process files that fit in memory, or Snowflake for queries that scan megabytes. Not to eliminate distributed systems—they’re essential for genuinely big data—but to right-size your architecture.

Every workload that moves from a distributed system to a well-tuned single machine is potentially 80-90% cost savings. That adds up fast.

Third: How are we integrating AI without creating skill atrophy?

Use AI. Absolutely use it. But have a plan for ensuring your team still knows how to think without it. Maybe that’s higher test coverage requirements for AI-assisted code. Maybe it’s regular code reviews focused on understanding, not just correctness. Maybe it’s explicit training on recognizing when AI is confidently wrong.

Whatever it is, don’t just adopt AI and hope for the best.

Fourth: What are we doing about retention?

If 95% of data engineers are burned out and 70% are job-hunting, what makes you think yours are different? This isn’t about ping-pong tables or free snacks. It’s about sustainable workloads, clear career progression, meaningful work, and not being on-call for systems that could be more reliable with better architecture.

Every senior engineer who leaves takes years of institutional knowledge with them. In a world where we’re not hiring juniors to replace them, that knowledge is gone permanently.

The 2030 Horizon: Why This Matters Now

Everything I’ve described isn’t just about surviving 2026. It’s about positioning for where this industry is headed.

By 2030, projections suggest demand for data professionals will grow by 36%. The global AI market will hit approximately $2 trillion. And those 10.5 million unfilled positions I mentioned earlier? That’s the scale of the gap we’re creating right now, with every junior engineer we don’t hire and every senior we burn out.

The consensus from industry analysts is clear: AI will transform data engineering workflows, but it won’t replace the humans who design, implement, and maintain those systems. What will change is the nature of the work.

Data engineers will transition from technical executors to strategic leaders. The routine tasks—basic ETL, straightforward data cleansing, simple pipeline monitoring—will increasingly be handled by AI assistants. What remains is the work that requires judgment: understanding business context, making architectural trade-offs, ensuring data quality for AI applications, and building systems that are reliable at scale.

Here’s the paradox that most people miss: AI applications are hungry for huge, well-prepared datasets. Every chatbot, every recommendation engine, every automated decision system needs data pipelines feeding it. The same AI revolution that automates some of our work multiplies the demand for the infrastructure we build.

The question is whether you’ll have the team to build it.

Organizations that invest in talent pipelines today—hiring juniors, retaining seniors, building mentorship cultures—will have the workforce to capture this opportunity. Organizations that don’t will be competing for an ever-shrinking pool of experienced engineers, paying premium salaries for people who could have been developed internally.

The technology decisions you make in 2026 are reversible. Switch databases, change orchestrators, adopt new tools—these are tactical choices you can adjust. But the talent decisions compound. Every junior you don’t hire today is a mid-level engineer you won’t have in 2028 and a senior you won’t have in 2030.

What Actually Matters

The data engineering landscape in 2026 is simultaneously more accessible and more demanding than ever before. AI tools, managed services, and mature open source have lowered the barriers to building data systems. But the systems we need to build are more complex, the scale challenges are more varied, and the skills required to make good architectural decisions have broadened significantly.

The teams that thrive in this environment—and position themselves for 2030—won’t be the ones with the fanciest tools or the biggest budgets. They’ll be the ones who:

Build sustainable talent pipelines instead of fighting for a shrinking pool of senior engineers
Right-size their architectures instead of defaulting to distributed systems for everything
Use AI as a force multiplier—including leveraging its collective learning capability—instead of treating it as a crutch
Create environments where people want to stay instead of burning through talent

None of this is easy. But it’s necessary.

Because here’s the thing: the talent pipeline crisis, the cost optimization opportunities, the AI integration challenges—these aren’t going away. They’re getting worse. The organizations that address them proactively in 2026 will have a massive advantage. The ones that don’t will spend 2027 wondering why they can’t hire anyone, why their costs keep climbing, and why their best engineers keep leaving.

The choice is yours. But choose soon.

The biggest risk to your data platform in 2026 isn’t technology. It’s whether you’ll have anyone left who knows how to run it.

And by 2030, everyone will wish they’d acted now.

The Guerrilla Guide to Data Engineering Interviews

Chris Hillman — Sun, 11 Jan 2026 09:00:00 +1100

The Scenario That Changes Everything

Picture this: You’re sitting in an interview room—or more likely these days, staring at a Zoom window with your carefully curated bookshelf background—and the interviewer asks you about data quality.

“Tell me about your experience with data quality,” they say.

You have two choices.

Choice A: “Data quality is really important in data engineering. It involves ensuring data is accurate, complete, consistent, and timely. I believe strongly in implementing data quality checks throughout the pipeline.”

Choice B: “Last year, we were bleeding money. Our data wasn’t running on time, it wasn’t consistent we had a table that had a 12% duplicate rate—we only discovered this when the Head of department noticed and our spend per customer was mysteriously 12% higher than industry benchmarks. I implemented row-level assertions on the incoming CDC stream that caught duplicates before they hit the merge, then backfilled three months of historical data by running a deduplication job that prioritized records based on source system hierarchy and last-updated timestamps. Took two weeks. Cut the duplicate rate to 0.3%.”

One of these answers gets you hired. The other gets you a polite “we’ll be in touch.”

Here’s the thing about data engineering interviews: they’re not testing whether you know things. Any idiot with ChatGPT can tell you what a slowly changing dimension is. What interviewers desperately want to know is whether you’ve actually built things. Have you stood in the wreckage of a failed pipeline at 2 AM and figured out what went wrong? Have you stared at two conflicting requirements from two department heads and architected a solution that made both of them reasonably happy?

This guide is about demonstrating that you’ve been in the trenches. Because that’s what separates the candidates who get offers from the candidates who get “we decided to move forward with other applicants.”

The Two Things That Actually Matter

After nearly two decades of interviewing data engineers—and being interviewed plenty of times myself—I’ve come to believe that good data engineering candidates have exactly two qualities:

Smart, and
Get things done

That’s it. That’s the whole list.

Sound overly simplistic? It’s not. These two qualities encompass everything an interviewer is actually trying to assess. Technical knowledge? That’s a component of smart. Communication skills? Part of getting things done. Problem-solving ability? Both.

The beauty of this framework is that it helps you understand what interviewers are really looking for beneath all those questions about window functions and star schemas.

People who are Smart but don’t Get Things Done often have impressive credentials but struggle to ship anything. They’re the ones who want to spend three weeks “properly” designing a data model for a proof-of-concept that needs to be done by Friday. They’ll tell you why your approach has theoretical limitations but won’t propose a practical alternative. In interviews, they can answer technical questions perfectly but give vague, abstract answers when asked about projects they’ve completed.

People who Get Things Done but aren’t Smart will build things that barely work, create technical debt that takes months to unwind, and make decisions that seem reasonable in the moment but reveal fundamental misunderstandings later. In interviews, they can talk enthusiastically about all the pipelines they’ve built but can’t explain why they made specific technical choices.

The magic happens when you have both. Smart people who get things done understand the theory, recognize when it matters and when it doesn’t, and ship production-quality solutions that their colleagues can maintain.

Your job in an interview is to demonstrate that you’re one of these people.

Show Me What You’ve Built

The single most important piece of advice I can give you is this: Come to every interview with a mental catalog of specific implementations you’ve delivered.

Not concepts you understand. Not technologies you’ve used. Not certificates you have attained. Actual problems you’ve solved, with specific details about what you did and why.

I’ve seen brilliant engineers fumble interviews because when asked “Tell me about a time you implemented a backfill strategy,” they gave a generic answer about what backfills are. Meanwhile, mediocre engineers who happened to prepare good stories about their work sailed through.

The stories matter. Let me show you why.

The Backfill Story

Every data engineer has done backfills. But here’s how most candidates talk about them:

“Yes, I’ve done backfills. We had to reload historical data when we changed the schema. I wrote a script to process the data in batches.”

Okay. That tells me nothing about your judgment, your problem-solving, or your ability to handle complexity.

Here’s what I want to hear:

“Our analytics team realized they needed two years of historical data for a new churn model, but our pipeline had only been running for six months. The source system had the data, but it was in a different format—they’d migrated from Oracle to Postgres about 18 months prior. So I had three different data formats to reconcile: the current CDC stream, the Postgres historical data, and the Oracle archives.

I built a unified transformation layer that normalized all three formats, then created a DAG that processed the Oracle data first—about 400 million records—in weekly chunks to avoid overwhelming the warehouse. The tricky part was handling the transition period where we had data in both Oracle and Postgres with potential duplicates. I used a combination of source system timestamps and record hashes to deduplicate across the boundary.

The whole backfill took about four days to run, but the prep work took two weeks. The churn model team was able to start their training three weeks after the initial request.”

See the difference? The second answer demonstrates:

Handling messy real-world complexity
Breaking down a large problem into manageable pieces
Making thoughtful tradeoffs (weekly chunks to avoid warehouse load)
Understanding the business context (enabling the churn model team)
Concrete numbers (400 million records, three weeks total)

That’s what gets you hired.

The Technical Questions They Actually Ask

Let’s get practical. Based on my experience on both sides of the table, here are the technical areas that come up most frequently in data engineering interviews—and more importantly, what interviewers are really trying to learn from each question.

Data Modeling: Don’t Just Know It, Defend It

When someone asks about your approach to data modeling, they’re not testing whether you can define a star schema. They’re trying to understand your judgment.

“We need to model customer order data. Walk me through your approach.”

The wrong answer starts rattling off dimensional modeling terminology. The right answer asks questions:

“Before I design anything, I need to understand a few things. What are the primary queries this data needs to support? Are we optimizing for dashboard performance, ad-hoc analysis, or ML feature generation? What’s the data volume and growth rate? How frequently will it be updated? And who are the consumers—analysts writing SQL, a BI tool like Tableau, or data scientists in notebooks?”

Only after understanding the requirements do you start talking about your design choices. And when you do, explain why:

“For this use case, I’d go with a denormalized wide table rather than a traditional star schema. Your analysts are primarily doing ad-hoc analysis in notebooks, and they’ve told you query simplicity matters more than storage efficiency. A star schema would mean teaching everyone to join fact to dimension tables correctly—and in my experience, that’s where most analytical errors come from. The denormalized approach trades some storage cost for query simplicity and reduces the chance of incorrect joins.”

This shows you understand that data modeling isn’t about following rules—it’s about making tradeoffs based on specific requirements.

SCD Type 2: The Implementation Details Matter

Every data engineer can explain what SCD Type 2 is. Few can explain how to actually implement it efficiently at scale.

If you’re working with Delta Lake or Iceberg, this is where things get interesting:

“Our customer dimension has about 50 million records and changes maybe 100,000 times per day. We’re on Delta Lake. How would you implement SCD Type 2?”

Here’s where I want to see you’ve actually done this:

“With Delta Lake, I’d use the MERGE statement with a match condition on the natural key, but here’s where it gets nuanced. You can’t just do a simple merge because you need to both update existing records (set the end date) and insert new records (the current version) in the same operation.

The approach I’ve used is to structure the merge to match on natural key AND where the record is current (end_date is null), then on MATCHED and when there’s an actual change in the tracked columns, update the end_date to yesterday, and separately insert the new record. But that requires two passes—or you can use the multi-action MERGE syntax if your Delta version supports it.

The gotcha is handling late-arriving data. If you get yesterday’s changes after you’ve already processed today’s changes, your end_dates can get out of order. We solved this by including a logical sequence number in our CDC stream and processing changes in order within each merge batch.”

This answer shows implementation experience, awareness of edge cases, and practical problem-solving.

Merge vs. Delete-Insert: Know When to Use Each

This comes up constantly, and the wrong answer is “I always use MERGE” or “I always use DELETE-INSERT.” The right answer is “it depends, and here’s how I decide.”

MERGE is better when:

You need atomic upsert behavior
You’re implementing SCD Type 2
The update volume is small relative to table size
Your warehouse has efficient MERGE support (most modern ones do)

DELETE-INSERT is better when:

You’re doing full partition replacement
The “update” volume is close to 100% of the existing data
You want simpler logic that’s easier to debug
You need to handle deletes from the source system

Here’s how I’d explain it in an interview:

“In my experience, DELETE-INSERT is actually more common in analytics pipelines than people expect. When we refresh a daily partition, we don’t mess around trying to figure out what changed—we drop the partition and reload it. It’s simpler, it’s idempotent, and if something goes wrong, we just run it again.

MERGE is what we use for dimensions where we need SCD Type 2 behavior, or for fact tables where we’re getting late-arriving facts that need to be upserted into historical partitions. The key decision point is: do I actually need to identify and handle individual changes, or can I just reload the whole thing? Nine times out of ten, the answer is reload.”

This shows practical wisdom about tradeoffs, not textbook regurgitation.

Problem-Solving Questions: Your Chance to Shine

The questions I love most as an interviewer—and the ones you should love as a candidate—are the open-ended problem-solving questions. These are where you demonstrate that you’re smart and get things done.

“You’ve been asked to build a pipeline that ingests data from a new source system. Walk me through how you’d approach it.”

This is a goldmine of an opportunity. Here’s how a strong candidate handles it:

Step 1: Ask clarifying questions (shows you understand context matters)

“What’s the source system? Is it pushing data to us, or do we need to pull it? What’s the volume and velocity? What’s the latency requirement—does it need to be near-real-time, or is daily batch okay? Who are the consumers, and what questions are they trying to answer? Is there existing documentation, or do we need to do discovery?”

Step 2: Outline your approach at a high level (shows you can structure complex problems)

“Assuming this is a reasonably standard scenario—a REST API that we need to poll daily—here’s my approach. First, I’d spend time understanding the data. What entities are exposed? What are the relationships? What’s the grain? Are there any gotchas like soft deletes or non-standard timestamp formats?

Then I’d design a landing zone—probably a staging schema with minimal transformation, just enough to make the data queryable. From there, I’d build the transformation layer to reshape it into our target schema, applying data quality checks along the way.

Finally, I’d implement monitoring: row counts, schema change detection, and anomaly alerts for unexpected patterns.”

Step 3: Dive into specific challenges (shows depth of experience)

“The piece that often trips people up is handling incremental loads with APIs that don’t support proper change tracking. If the API doesn’t give you a reliable modified_timestamp, you might need to do full loads and then diff against your existing data—which gets expensive fast. I’ve handled this by implementing a hash-based change detection system where we store a hash of each record and only process records whose hash has changed.”

Step 4: Discuss what could go wrong (shows you’ve been burned before)

“The things I’d watch out for: API rate limits that could cause us to fall behind, schema changes from the source system that break our transformations, and timestamp timezone issues—I’ve been bitten by that one more times than I’d like to admit.”

This kind of answer demonstrates everything an interviewer is looking for: structured thinking, technical depth, practical experience, and awareness of edge cases.

The Incident Management Question

At some point, you’ll be asked about handling production incidents. This is where your war stories come in handy.

“Tell me about a time you had to debug a production data issue.”

Here’s the thing: everyone has these stories. The difference between a good answer and a great answer is how you structure it.

The Great Answer Structure:

The alert: How did you find out something was wrong?
The triage: How did you assess the severity and scope?
The investigation: How did you identify the root cause?
The fix: What did you do to resolve it?
The prevention: What did you do to prevent it from happening again?

Here’s an example:

“We got an alert at 9 PM on a Friday—because of course it was Friday—that our customer metrics dashboard was showing a 30% drop in active users. At first, I thought it might be a real business event, but the drop was too sudden and too large.

I started with the basics: checked the pipeline run logs, all green. Checked row counts in the metrics table, normal. Checked upstream tables, normal. Nothing obviously broken.

Then I looked at the actual data. Our active user count was filtering by ’last_activity_date >= current_date - 7’. Turns out, a schema change in the source system had changed the last_activity_date column from a timestamp to a date-with-timezone, and our transformation was truncating it incorrectly. Users in certain timezones were getting their dates shifted by a day, which pushed them outside the 7-day window.

The immediate fix was straightforward—I adjusted the timezone handling in the transformation and backfilled the last two weeks of data. Took about an hour.

But the prevention was the interesting part. I added explicit timezone assertions to our data quality framework. Any time we ingest a timestamp column, we now validate that the timezone handling matches our expectations. We’ve caught three similar issues since then before they hit production.”

This answer hits all the marks: structured approach, technical depth, practical resolution, and systemic improvement.

The AI and LLM Question

If you’re interviewing in 2026, you will be asked about AI. How you answer tells the interviewer a lot about whether you’re keeping current.

“How have you used AI or LLMs in your data engineering work?”

The weak answer is “I’ve experimented with ChatGPT for writing queries.”

The strong answer shows you’ve actually integrated AI into production workflows:

“We’ve implemented LLM-assisted data classification in our pipeline. We have a data catalog with about 3,000 tables, and maintaining accurate tags for sensitivity, domain, and data type was a nightmare—the manual process was always out of date.

I built a service that uses Claude’s API to analyze table schemas and sample data, then suggests classifications. It’s not fully automated—we have a human review step—but it reduced the time to classify a new table from 20 minutes of manual analysis to about 2 minutes of review.

The tricky part was prompt engineering. The first version was too aggressive with PII classification—it was flagging everything with ‘ID’ in the name as personally identifiable. We refined the prompts to include context about our specific data domain and examples of what we consider PII versus internal identifiers.”

This shows practical application, awareness of limitations (human review), and iteration based on real-world feedback.

Building Experience in Areas You Don’t Know

Here’s a question I hear all the time: “What if I don’t have experience with [Delta Lake / Iceberg / dbt / whatever technology they’re asking about]?”

First, be honest. Don’t claim experience you don’t have—good interviewers will expose this within two follow-up questions.

Second, go build something. You can spin up a personal project in a weekend that gives you legitimate hands-on experience.

Want to learn Delta Lake? Create a databricks community edition account, load some public dataset, and implement a basic SCD Type 2 merge. Want to learn dbt? Fork the jaffle_shop demo project and extend it with some real transformations. Want to understand data quality at scale? Implement Great Expectations on one of your side projects.

The beauty of data engineering is that the tools are mostly free for small-scale learning. You don’t need a $50,000/month Snowflake instance to learn Snowflake—you need the free trial and a CSV file.

When you interview, you can honestly say: “I haven’t used Delta Lake in production, but I’ve built a personal project that implements [specific thing]. Here’s what I learned, and here’s how I’d apply it at scale.”

That’s infinitely better than “I’ve heard of Delta Lake but haven’t used it.”

The Questions You Should Ask Them

At the end of every interview, you’ll be asked “Do you have any questions for us?” This isn’t a formality—it’s a real opportunity to demonstrate your sophistication as a data engineer.

Bad questions:

“What’s the work-life balance like?” (Important, but not what shows you’re a great candidate)
“What technologies do you use?” (You should have researched this already)
“What’s the career growth path?” (Reasonable, but generic)

Great questions:

“What’s the biggest data quality challenge you’re facing right now, and what’s your current approach to solving it?”
“Walk me through your deployment process for a new data pipeline—from development to production.”
“How do you handle schema evolution in your source systems? Is it a significant pain point?”
“What’s the split between building new pipelines versus maintaining existing ones?”
“If I joined, what would success look like in the first 90 days?”

These questions show you understand real data engineering challenges and you’re thinking about how you’d actually do the job, not just whether you want it.

The Meta-Game: Time Management During Interviews

Here’s something nobody tells you about interviews: time management matters as much as your answers.

If you spend 15 minutes on the first whiteboard question and only have 5 minutes for the next two, you’ve failed—even if your first answer was brilliant. Interviewers have a checklist of things they need to assess, and if you don’t give them enough data points, you’ll get a “no hire” simply because they couldn’t tell.

The Rule of Thirds:

For a 60-minute technical interview:

First 20 minutes: The introductory/behavioral question. Don’t ramble. Get your point across clearly and move on.
Middle 30 minutes: The core technical assessment. This is where you prove your skills. Spend your time wisely here.
Final 10 minutes: Wrap-up, your questions, selling you on the role.

If you find yourself 25 minutes into a detailed answer about your architecture philosophy and haven’t written any code yet, you’re in trouble. Learn to recognize when you’ve made your point and need to move on.

Breaking Down Problems:

When faced with a complex technical question, don’t start writing code immediately. Spend 2-3 minutes structuring your approach out loud:

“Okay, let me break this down. There are three main pieces here: the ingestion, the transformation, and the output. For the ingestion, I need to handle [X]. For transformation, the key challenge is [Y]. Let me start with the ingestion piece, since that’s the foundation.”

This shows structured thinking and gives the interviewer a roadmap of what you’re going to do. If you run out of time, they at least know you understood the full scope.

The Uncomfortable Truth About Interviews

I’m going to tell you something that might be unsettling: interviews are imperfect signals.

Smart, talented people fail interviews all the time. Mediocre people sometimes pass them. The best interviewers are wrong about 20-30% of the time.

This is why you shouldn’t let a failed interview devastate you, and you shouldn’t let a passed interview make you complacent. The skills that make you successful in interviews are related to—but not identical to—the skills that make you successful in the job.

What you can control:

Your preparation
Your catalog of project stories
Your ability to explain technical concepts clearly
Your structured approach to problem-solving
Your honesty about what you know and don’t know

What you can’t control:

Whether the interviewer is having a bad day
Whether they’re biased toward a technology you don’t know
Whether they’ve already decided to hire an internal candidate
Whether you remind them of someone they didn’t like in their last job

Do everything you can to control the controllables. Then recognize that sometimes it just doesn’t work out, and that’s not a reflection of your worth as a data engineer.

Your Pre-Interview Checklist

Before any data engineering interview, make sure you can speak confidently about:

At least three implementation stories covering different areas: one about pipeline building, one about problem-solving/debugging, one about data modeling or architecture decisions.

Your approach to data quality: What checks you implement, how you handle failures, how you balance coverage with performance.

Tradeoff decisions: When to use batch vs. streaming, when to denormalize vs. normalize, when to merge vs. delete-insert, when to build vs. buy.

Your technical fundamentals: Window functions in SQL, how you handle slowly changing dimensions, your experience with your primary orchestration tool (Airflow, Dagster, dbt).

Technology-specific depth: Whatever platforms they use (Databricks, Snowflake, BigQuery), have at least surface-level familiarity and one or two interesting insights.

Questions for them: At least three substantive questions that show you’re thinking about the actual work.

The Final Word

Here’s what I’ve learned after interviewing hundreds of data engineers: the candidates who get hired aren’t necessarily the ones who know the most. They’re the ones who can demonstrate that they’ve taken on hard problems and found practical solutions.

Knowledge without execution is academic. Execution without knowledge is dangerous. The combination is rare and valuable.

Your job in an interview is to prove you have both.

So go catalog your victories. Remember the gnarly problems you solved, the architectural decisions you made, the production fires you put out. Write them down if you need to. Practice telling those stories until they flow naturally.

Because in the end, data engineering interviews aren’t about reciting definitions or demonstrating mastery of every possible technology. They’re about convincing a group of smart people that if they hire you, things will get built, problems will get solved, and the pipelines will run.

Everything else is details.

Why Your Ideas Die in Planning Meetings

Chris Hillman — Wed, 07 Jan 2026 09:00:00 +1100

The silence that kills good ideas

One morning, I sat in yet another meeting where we just spent two weeks backfilling a table then we found it was riddled with issues with the data. Even if we resolve the issue, it would then be another 2 weeks to backfill the data, there has to be a better way.

“So, what do we think? Give me your best ideas for tackling this.”

Silence.

Two senior engineers stared at their laptops. One analyst started typing what looked suspiciously like Microsoft Teams messages. I knew for a fact that at least two people in that room had brilliant, half-formed thoughts about the problem. I’d heard them discussing it the day before.

But in this moment? Nothing.

This pattern repeats in data teams everywhere, and it’s not because people lack ideas. It’s because the way we ask for ideas practically guarantees we’ll never hear the good ones. The research on collaborative ideation reveals something that should fundamentally change how leaders run their teams: the difference between “give me your best ideas” and “give me all your ideas” isn’t just semantic—it’s the difference between innovation and stagnation.

What psychological safety actually means for data teams

Amy Edmondson’s research at Harvard Business School established that psychological safety—“a shared belief that the team is safe for interpersonal risk taking”—is the essential precondition for teams to function effectively. But here’s what made her findings particularly striking: in a study of 51 work teams, she discovered that team confidence alone didn’t predict learning behaviors when you controlled for safety.

People need to feel that speaking up won’t damage their standing before their capability matters.

This has profound implications.

Think about your last sprint planning meeting. Who did most of the talking? If you’re like most organizations, it was probably the most senior person in the room—the staff engineer, the principal data scientist, or the engineering manager. Everyone else contributed sporadically, if at all.

But, that senior person doesn’t have all the answers. The analyst who joined three months ago might have spotted a pattern in the data that the ten-year veteran missed because they’re looking at it with fresh eyes. The data engineer who’s quiet in meetings might have implemented the exact solution we need at their previous company. The junior scientist might have read a recent paper that completely reframes the problem.

You’re not hearing any of this because they’re doing what Edmondson calls the “interpersonal risk calculus”—silently weighing whether the perceived benefits of speaking up outweigh the risks of looking incompetent, being rejected, or appearing ignorant.

When safety is low, self-protective filtering dominates. Only “safe,” conventional ideas get shared. The unconventional, half-formed, potentially breakthrough thoughts stay locked in people’s heads.

Kerry Patterson’s research on crucial conversations provides a useful framework for understanding what’s happening here. He calls it the Pool of Shared Meaning—the collective understanding that emerges when people openly share their views. When everyone contributes their complete thinking, the pool is full and decisions are stronger. When safety is low and people self-censor, the pool stays shallow.

In data teams, this shows up constantly. Someone notices that our data quality problems stem from misaligned incentives between product and engineering, but sharing that observation feels risky—it implies criticism of another team. Someone else has experience with a similar architecture problem at their previous company, but they’re only three months in and worry about seeming presumptuous. A third person has an unconventional idea that might actually work, but it would require explaining a complex technical concept and they’re not sure they can articulate it clearly under pressure.

Each person performs their own silent risk calculus and decides to withhold. The pool stays shallow. The meeting ends with only the safest, most obvious ideas on the table. And the best solution—which probably required combining insights from all three of those people—never emerges.

I’ve seen this play out. An engineer might notice that the root cause of data quality issues isn’t technical at all—it’s that product managers keep requesting “quick one-off reports” that bypass all our governance processes. But suggesting that we need to have a difficult conversation with product leadership? That requires tremendous psychological safety. Easier to just say “we should add more validation checks to the pipeline” and move on.

Why “give me your best ideas” is the worst possible question

Here’s something that surprised me when I first encountered the research: asking people for their “best” ideas actually produces worse outcomes than asking for “all” ideas.

It seems counterintuitive. Surely asking for best ideas filters out the noise and gets straight to quality, right?

Wrong.

When you ask for “best ideas,” you trigger what psychologists call evaluation apprehension. People must judge their own ideas before sharing them, which activates fear of judgment. In studies, participants in high-apprehension conditions generated significantly fewer ideas (about 21 ideas on average) compared to low-apprehension conditions (about 33 ideas)—and explored fewer idea categories entirely.

The request “give me your best ideas” does several harmful things simultaneously:

First, it forces people to evaluate while they’re generating. This is cognitively expensive. Part of their mental resources are devoted to thinking up solutions, while another part is busy playing judge and jury on those solutions before they’re even fully formed.

Second, it signals that quality judgment happens before sharing, which means people assume incomplete or uncertain ideas aren’t welcome. The very thoughts that might serve as stepping stones to breakthrough insights get filtered out before they reach the group.

Third, it creates a high-stakes environment. If you’re only sharing your “best” ideas, you’re putting your judgment on display. What if the group disagrees? What if someone senior thinks your “best” idea is actually terrible?

There’s a psychological mechanism at work here that Daniel Pink calls the Sawyer Effect, from Mark Twain’s observation that “work consists of whatever a body is obliged to do, and play consists of whatever a body is not obliged to do.” Pink’s research on motivation shows that external evaluation can transform inherently interesting tasks into drudgery—turning play into work.

When you ask for “best ideas,” you’re asking people to apply external evaluation to their own thinking. What was potentially an enjoyable creative exercise—“how might we solve this interesting problem?"—becomes a performance task with judgment attached. The intrinsic motivation to explore solutions gets crowded out by the extrinsic pressure to look competent.

This is particularly damaging for the kind of creative problem-solving data teams do daily. We’re not assembly line workers performing routine tasks—we’re solving novel problems that require genuine creativity. The Federal Reserve actually studied this: they found that offering higher rewards for cognitive tasks didn’t improve performance. In fact, those offered the highest bonuses performed the worst. When stakes are high and judgment is involved, external pressure actively harms the kind of thinking we need.

I seen this play out various times. The team lead asked everyone to “bring your best approach for data migration from our monolithic data warehouse to a lake house architecture.” People showed up to the next meeting with carefully prepared ideas, each advocating for their single preferred solution. Majority of them are the same, but it doesn’t look at the problem differently.

The meeting turned into positional warfare. People defended their chosen approach, pointed out flaws in alternatives, and nobody budged. We spent three weeks in analysis paralysis before a decision to end the stalemate occured.

Contrast that with, instead of asking for “best approaches,” I asked the team to generate every possible way we could solve the problem, no matter how impractical or half-baked. “Give me all your ideas—the good, the bad, the weird, the incomplete.”

We filled three whiteboards. Someone suggested new tech. Someone proposed a datamesh approach. One engineer threw out “what if we just stayed with the data warehouse and optimized it better?” Another suggested a serverless approach using AWS Glue and Athena. Someone jokingly said “rewrite everything with heavy partitioning.”

That last “joke” actually sparked a serious conversation. Turned out our issues weren’t really about warehouse versus lake house at all—they were about query patterns and data organization. We ended up with a solution that kept our existing warehouse but completely restructured how we modeled data.

We never would have arrived at that solution if people had been filtering for their “best” ideas before speaking.

Quantity unlocks quality through cognitive mechanisms you can actually use

The counterintuitive finding that quantity breeds quality is now well-established in cognitive science, and it has immediate practical implications for how data teams approach problem-solving.

The mechanism works like this: when you start generating ideas, your brain naturally accesses the most available associations first—which are typically the most obvious and conventional solutions. The really original ideas live in more remote semantic connections that only get accessed after you’ve exhausted the obvious options.

Research by Nijstad and Stroebe explains this through their Search for Ideas in Associative Memory (SIAM) model. Idea generation happens in two stages: first, a search cue activates a knowledge structure in long-term memory; then, features of that structure combine to produce specific ideas. The crucial insight is that initial ideas access what’s most readily available. Later ideas, after depleting these obvious options, require exploring more unusual mental paths.

This produces what researchers call the serial order effect: as people generate more ideas, later ideas become increasingly original.

I use this principle deliberately now when working with teams. Here’s a practical example:

When we were trying to improve our data backfill mechanism, I asked the team to come up with 15 different approaches for how we could optimise or improve how we backfill tables with data. The idea was to short fuse and increase the ability to understand issues, before they reached production. Not “a few good approaches”—specifically 15.

The first five were predictable: create a cyclic job that runs over each date, another was, load a week of data then run tests.

Ideas six through ten got more interesting: create a insert only table to be able to batch load data hisotircal data in one go, create synthetic test data that exercises a longer time span of data.

Ideas eleven through fifteen got genuinely creative: To build on the above, loading data as insert only, but then compressing timelines, and virtually end dating the records.

We ended up implementing a combination where we rebuilt the history doing that insert spine of data, and then compressing the timeline (where previous and current records row hash is identical)

What was amazing here is, not only did running this script take less time, 1hr to execute. However the compute cost of running 1 query versus throusands of queries cost alot less in compute as well.

This works because high quantity targets provide psychological permission. When the goal is 15 ideas, it legitimizes including uncertain, incomplete, or seemingly foolish contributions. Nobody expects all 15 to be brilliant—which paradoxically enables some of them to be exactly that.

The IKEA effect explains why imposed solutions fail

Here’s something you already know intuitively: people are far more committed to solutions they helped create than solutions handed to them. What you might not know is just how powerful this effect is, or the psychological mechanisms that drive it.

Research calls this the IKEA effect—consumers place disproportionately high value on products they partially created, even when those products are objectively inferior to professionally made alternatives. In studies, people valued self-assembled furniture about 63% higher than identical pre-assembled items.

The mechanism is what psychologists call psychological ownership. When you invest effort in creating something, it becomes part of your extended self. The idea is no longer “the company’s data quality initiative”—it’s “our data quality initiative that we designed together.”

This explains patterns I see repeatedly in organizations:

A new senior leaders comes in and mandates that all teams must adopt a particular framework. Teams comply minimally, work around it when possible, and resist at every opportunity. Two years later, the senior leaders leaves, and the framework dies immediately.

Contrast that with a different organization where the data team spent three months collaboratively designing their own framework. They held working sessions. They debated principles. They argued about where to draw boundaries between centralized control and team autonomy. The resulting framework wasn’t objectively better than the mandated one—arguably it was less sophisticated. But adoption was near-universal because people felt ownership of it.

People don’t just accept ideas they helped create—they champion them.

What actually works: practical patterns for data teams

After reviewing all this research and reflecting on what I’ve observed across multiple data teams, certain patterns emerge consistently.

Create explicit psychological safety through leader behavior. This isn’t about being nice or avoiding difficult conversations. It’s about modeling specific behaviors: openly acknowledging your own mistakes, asking for team input in group settings and responding positively even to challenging questions, treating failures as learning opportunities rather than blame opportunities.

In practice, this means: when a data pipeline fails, your first question is “what can we learn?” not “who’s responsible?” When someone junior proposes an idea that won’t work, you explore why they thought it might before explaining the problems. When you make a decision that turns out wrong, you say so explicitly rather than quietly changing direction.

Structure ideation sessions to separate generation from evaluation. Set high quantity targets, ask for “all ideas” not “best ideas,” use anonymous contribution methods when appropriate, actively suppress evaluation during the generation phase—even subtle head shakes or skeptical looks kill psychological safety.

Manage hierarchy flexibly across phases. Here’s something that took me too long to understand: power dynamics exist in every meeting whether we acknowledge them or not. Priya Parker’s research on effective gatherings makes this point forcefully—ignoring power dynamics doesn’t make them disappear, it just means they operate invisibly and unmanaged.

In data teams, hierarchy comes from multiple sources. There’s the obvious organizational hierarchy—staff engineers outrank senior engineers who outrank mid-level engineers. But there’s also expertise hierarchy (the person who built the system has more authority than the person who joined yesterday), social capital (some people have more trust and relationships), and communication skill (articulate people get more airtime regardless of title).

When we pretend everyone’s voice carries equal weight, we’re not creating equality—we’re just refusing to acknowledge reality. The senior architect’s skeptical look still kills ideas. The staff engineer’s casual “I don’t think that’ll work” still shuts down exploration. The difference is whether we’re managing these dynamics intentionally or letting them operate invisibly.

Managing hierarchy flexibly means acknowledging power exists and deliberately adjusting it for different phases of work. During idea generation, we want to minimize hierarchy’s influence. During evaluation and decision-making, we can leverage senior judgment appropriately. But we have to be explicit about the shift.

Alison Wood Brooks’ research on conversation dynamics identifies specific behaviors that high-status members must practice to create genuine safety in group settings. The problem isn’t just that senior people dominate airtime—though they do—it’s that high status actually inhibits perspective-taking. When you’re senior in a group, you naturally become less attentive to others’ viewpoints and less careful with your language.

This means senior engineers and managers need to work actively against their default behaviors. Brooks recommends:

Acknowledge junior members by name before they speak and after they contribute. “Sarah, I’m curious what you think about this” and “That point Sarah made earlier about partitioning strategies is worth exploring” both signal that contributions from less senior people matter.

Be vulnerable about your own failures early in the discussion. When a principal engineer says “I tried this approach last year and completely misjudged the performance implications,” it reduces everyone else’s fear of proposing imperfect ideas. Vulnerability from the top of the hierarchy creates permission for uncertainty throughout the group.

Use genuinely inclusive eye contact. Don’t just look at other senior people or the most vocal contributors. Make eye contact with the quiet analyst, the new hire, the contractor who hasn’t spoken yet. Your attention signals whose contributions you value.

Sometimes just get out of the way. Stop talking. Give others space. If you’re the most senior person and you’ve spoken three times while others have spoken once or not at all, you’re probably dominating even if you don’t feel like you are.

I’ve started doing something deliberate in architecture discussions: I ask a question to frame the problem, then I explicitly say “I’m going to listen for the first 15 minutes and not share my thoughts yet.” It feels uncomfortable—I usually have opinions and I’m used to sharing them. But the quality of the solutions that emerge when I’m not steering the conversation is consistently better than what I would have proposed initially.

Build genuine ownership through participatory design. Don’t ask for input on your pre-decided solution—that’s performative participation. Create actual joint solution development where the outcome isn’t predetermined. Be willing to implement approaches that differ from what you would have chosen if people genuinely helped design them and they’ll work in your context.

Develop champions, don’t just appoint them. Champions need belief in the innovation, organizational support, time to help others, experience implementing what they’re championing, and self-efficacy in leading change. You can’t just tap someone’s shoulder and expect them to magically become an effective change agent.

Run pathfinder projects before mandates. Let teams experiment with new approaches in low-risk contexts. Make results visible to others. Create opportunities for interested teams to try things without committing to wholesale adoption. NASA’s approach to Model-Based Systems Engineering provides the template: pathfinders demonstrate value, reduce perceived risk, and create internal proof points.

Respect the evidence-based culture of technical teams. Engineers want to see proof that something works before investing effort. Don’t appeal to authority or best practices—run experiments, collect data, show concrete outcomes. When adopting a new tool, the question “has anyone actually measured the improvement?” should have a data-driven answer.

I’ve used these patterns to implement changes ranging from adopting dbt to building data contracts to migrating cloud platforms to establishing data quality frameworks. They work consistently—not because they’re revolutionary, but because they align with how humans actually engage with ideas and change.

The fundamental insight is this: process shapes ownership, ownership shapes commitment, and commitment shapes outcomes.

The compounding effects you actually see

Here’s what I’ve observed when teams get this right versus when they don’t.

High psychological safety teams surface problems early before they become critical, explore diverse solution approaches, help each other debug issues without fear of judgment, openly discuss failures and what was learned, share knowledge freely across team boundaries, try new tools and approaches without waiting for mandates, and actively participate in architectural decisions.

Low psychological safety teams hide problems until they explode, implement the first obvious solution without exploring alternatives, avoid asking for help, repeat the same mistakes, hoard knowledge as job security, wait for explicit direction before trying anything new, and defer all decisions to the most senior person.

The performance gap is enormous. Google’s Project Aristotle found that teams with low psychological safety show measurable declines in delivery throughput, recovery time, and change failure rates.

In data teams specifically, I’ve seen the impacts manifest as: technical debt accumulation (engineers fear reporting that shortcuts were taken), narrow solution spaces (teams explore fewer alternatives because suggesting unconventional approaches feels risky), knowledge silos (learning isn’t shared because it provides competitive advantage within the team), and review avoidance (discomfort with feedback prevents thorough code reviews and pair programming).

The mechanisms reinforce each other. Psychological safety enables idea sharing. Quantity targets reduce remaining evaluation apprehension. Removing hierarchy during generation eliminates authority-triggered self-censorship. Participation creates ownership. Ownership drives championing. Champions spread adoption. Successful adoption reinforces psychological safety for the next change.

Each mechanism matters independently, but the compounding effects explain why some teams achieve dramatically better outcomes than others.

I watched this play out when comparing two data teams within the same company. Both had similar talent, similar tools, similar organizational challenges. But one team had a leader who’d internalized these principles, and one hadn’t.

The first team ran weekly “all ideas” sessions when facing problems. They explicitly managed hierarchy—senior people sometimes sat out of initial brainstorming. They built solutions collaboratively. They developed internal champions who helped spread new practices. When they adopted new tools, it was because teams had experimented and demonstrated value.

The second team operated top-down. The lead made architectural decisions and communicated them. People implemented what was asked. Adoption of new practices happened when mandated. Problems surfaced late because people were reluctant to admit issues.

After 18 months, the performance gap was stark. The first team delivered more features, had higher data quality metrics, lower on-call burden, better retention, and higher engagement scores. The second team struggled with all of these, and turnover was high.

The difference wasn’t the people—it was the process for how ideas were generated, evaluated, and implemented.

What this means for your team Monday morning

You don’t need to transform your entire organization overnight. You can start applying these principles immediately in small ways.

Next time you’re planning a significant change—adopting a new tool, migrating architecture, implementing new processes—try this:

Before announcing the approach, bring the team together and ask them to generate solutions. Set a specific quantity target high enough to make people uncomfortable. Ask for “all ideas” not “best ideas.” Explicitly suspend judgment during generation.

If there’s significant hierarchy in the room, manage it. Have senior people contribute last, or run initial brainstorming without them present. Try anonymous collection methods.

Take the ideas seriously. Not every idea will be implementable, but engage with them genuinely. Build the final solution collaboratively from the ideas generated, not from your pre-decided approach with some team input sprinkled on top.

Identify potential champions—people who believe in the solution (because they helped create it), have experience with the domain, are willing to help others, and are trusted by peers. Support them with time, resources, and organizational backing.

Run pathfinder projects before mandates. Let interested teams try the approach in low-risk contexts. Make results visible. Create proof points.

You’ll likely be surprised by both the quality of solutions generated and the enthusiasm for implementing them.

The research suggests not that technical teams need different approaches than other teams, but that they may be especially well-suited to participatory methods when those methods are implemented authentically rather than performatively. Evidence-based culture, distributed expertise, and autonomy expectations all align with genuine collaborative ideation.

But the key word is genuine. Teams can smell performative participation from a mile away. If you’re asking for input but have already decided the answer, people know. If you’re creating the appearance of collaboration while maintaining top-down control, it’s worse than just being explicitly directive.

The fundamental shift is from “how do I get people to adopt my solution?” to “how do I create conditions where we develop better solutions together?”

That shift is uncomfortable for many leaders. It requires genuine sharing of control, tolerance for messiness in the process, and comfort with outcomes that differ from what you would have chosen alone.

But if you want your data team to generate better ideas, adopt changes more willingly, and sustain improvements longer—the research is clear about what works.

The question isn’t whether these approaches are effective. The question is whether you’re willing to use them.

The Science of Conversation for people who hate small talk

Chris Hillman — Sun, 04 Jan 2026 09:00:00 +1100

One morning, I watched a data engineer struggle with using AI for thirty minutes, trying to debug a DBT job. The problem wasn’t the LLM’s capabilities—it was how the engineer framed the question. No context about what they’d already tried. No explanation of the expected versus actual output. Just “fix this code” followed by a massive code dump.

This same engineer had similar struggles with stakeholders. Presentations that assumed too much context. Emails that buried the ask. Meetings where they answered questions nobody asked.

Here’s what I realized: the framework for good human conversation is the same framework for effective AI interaction. Whether you’re prompting an LLM or presenting to executives, the mechanics are identical—you’re coordinating with another entity to exchange information and achieve shared goals.

Harvard professor Alison Wood Brooks spent years analyzing thousands of conversations, discovering that communication follows predictable, learnable patterns. Her research culminated in the TALK framework—a systematic approach to conversation that works whether you’re talking to Claude, Chat GPT or your CEO, or a date you’re not sure about. For technical professionals who recognize communication matters but don’t feel it’s their natural strength, this is a game-changer.

The research is clear: 70% of career advancement for engineers stems from communication skills, not technical prowess. Yet we spend hours learning data optimization and zero minutes learning conversation science. Brooks’ findings offer evidence-based techniques that play to analytical strengths—asking better questions beats being charismatic, preparation trumps improvisation, and introverts have hidden advantages.

This isn’t about becoming an extrovert. It’s about understanding how conversation actually works and deploying specific, research-backed strategies. The same strategies that make you better at prompting AI make you better at managing stakeholders, building teams, and advancing your career.

T Is for Topics: Preparation Isn’t Performance

Most people balk at preparing conversation topics. It feels unnatural, performative, like you’re planning spontaneity out of existence. Brooks found that 53% of people think topic prep is unnecessary, while 50% believe it will decrease their enjoyment of conversations.

This resistance stems from what Brooks calls the “Myth of Naturalness”—the belief that charismatic people just know what to say. We see someone excel at conversation and assume it’s effortless for them, when in reality they’ve accumulated years of deliberate practice. Same pattern we see in technical skills, but we forget to apply the logic.

Here’s the data: before conversations, 27% of people spend at least five minutes deciding what to wear, while only 18% think about what they’ll talk about. We optimize our appearance and neglect the substance. Then we wonder why conversations feel awkward or shallow.

The System 1 vs System 2 Problem

During conversation, we rely on “system 1 thinking”—fast, instinctive, automatic processing. This guides us toward whatever’s top of mind: the weather, that random person standing nearby, the fact that someone just ordered appetizers. We talk about what’s accessible rather than what’s meaningful.

The topics most accessible to us aren’t always best for connection. We drift into safe small talk rather than searching for mutually rewarding subjects. We prove how smart we are instead of learning what our partner knows. This is natural egocentrism—optimizing for ourselves rather than the coordination game.

Topic preparation engages “system 2 thinking”—slower, deliberative, logical processing. Before the conversation, when you have mental space, brainstorm 3-5 topics aligned with your goals. This isn’t scripting—it’s cognitive offloading. By reducing information-processing requirements beforehand, you free mental space during the conversation to actually listen and respond.

What This Looks Like for Data Engineers

Before that stakeholder meeting, don’t just prepare your technical update. Prepare topics:

“I’m curious what drove the timeline pressure you mentioned last week”
“What would success look like from your team’s perspective?”
“How does this fit with the Q2 roadmap priorities?”

Before the networking event, prepare genuine curiosities:

“What made you move from engineering into product management?”
“What’s the hardest part of scaling data infrastructure at your company?”
“How did your team approach the migration to Snowflake?”

For your 1:1 with your manager, beyond status updates:

“Where do you see the biggest risks in our current architecture?”
“What capabilities do you wish the data team had?”
“How can I better support the team’s strategic goals?”

Brooks uses a “topic pyramid” framework—base level is weather and small talk, middle level involves experiences and perspectives, top level consists of topics only you and your partner could discuss in that moment. The mistake most people make: getting stuck at the base. Small talk isn’t the problem; lingering there too long is.

Prepare topics that climb the pyramid. Instead of “How was your weekend?” (base level), try “I’ve been curious what made you interested in data quality originally” (middle to top level). You’re not abandoning spontaneity—you’re creating scaffolding that lets meaningful conversation emerge.

How This Applies to LLM Communication

The exact same principle applies when prompting AI. Most people use system 1 thinking—whatever question first comes to mind. “Debug this code.” “Summarize this document.” “Write me a function.”

Better results come from system 2 prep. Before opening Claude, ask yourself:

What’s my actual goal here?
What context does Claude need?
What have I already tried?
What specific output format do I want?
What constraints matter?

Compare these prompts:

System 1: “Fix this SQL query”

System 2: “I’m getting duplicate rows in this SQL query that joins orders to customers. I expected one row per order, but I’m getting multiple. Here’s the query, sample input data (remove sensitive data/mask it), and current output. Can you identify why duplicates are appearing and suggest a fix?”

The second provides context (what’s wrong), expected behavior (one row per order), actual behavior (multiple rows), and supporting information (query, data, output). This is topic preparation for AI—giving the LLM the topics it needs to help you effectively.

A Is for Asking: Follow-Up Questions Are Your Superpower

Brooks’ team analyzed 900+ speed dates and conversations among 398 strangers, classifying every question asked. They identified four types: introductory (“What’s your name?”), mirror (“I’m good. How are you?”), topic-switching, and follow-up questions.

The finding that should change how you approach every conversation: follow-up questions drive almost all positive effects on likeability.

The numbers are striking. People in the top third of question-askers got the most second dates. Asking just one more follow-up question per date across 20 dates resulted in one additional second date agreement. Mirror questions—the polite reciprocations we default to—produced no measurable increase in likeability.

Why Follow-Ups Work

Follow-up questions signal three things simultaneously: you were actually listening, you care about what the person said, and you’re thoughtful enough to connect their previous statement to a deeper inquiry. They make people feel heard and validated.

When a product manager explains a feature request, the instinct is to immediately propose technical solutions. That’s a shift response—redirecting attention to your expertise. A follow-up question maintains focus: “What’s driving the timeline pressure you mentioned?” This signals engagement far more effectively than demonstrating knowledge.

Brooks warns against “ZQs”—people who leave conversations having asked zero questions. Professional matchmaker Rachel Greenwald says it directly: “Zero questions means zero second dates.” But even when we ask questions, we vastly overestimate how many.

In one study, negotiators estimated that over 50% of their turns included a question. In reality, less than 10% did—less than one-fifth of what they thought. We’ve found the same pattern in friend conversations and first dates. In the chaos of conversation, we don’t notice the link between asking questions and being liked, so we don’t get clear feedback on what’s working.

The Four Question Types (And Which Matter Most)

Introductory questions (“What’s your name?” “How are you?”) are necessary social ritual. Important but don’t linger.

Mirror questions (“I’m good. How are you?”) reciprocate what was just asked. They’re polite but add little value—they’re “low-hanging improvisational fruit” that feel effortless but don’t increase likeability. Use them when you genuinely want to hear the answer, not out of reflex.

Topic-switching questions initiate noticeably new subjects. Instead of mirroring “How was your day?” with your own day summary, try: “Did you see that new data governance framework GitHub just released?” This climbs the topic pyramid rather than floundering at the base.

Follow-up questions are the superheroes. They keep you on the present subject, probing deeper on something your partner said. “You mentioned the migration was stressful—what made it complicated?” “How did your team approach that problem?” “What would you do differently next time?”

In Brooks’ studies, when people were prompted to ask more questions, the extra questions were mainly follow-ups. This means the rewards of asking more questions were driven almost entirely by follow-up questions. They unlock deeper learning, feel personal and validating, and show you’ve actually been heard.

The Never-Ending Follow-Up Exercise

Brooks challenges her students to have conversations using only follow-up questions. They can make statements for smoothness, but must always respond with a follow-up. The goal: see how easy and powerful sustained curiosity is.

Students are surprised by three things:

How easily they can maintain never-ending follow-ups
How much they learn about their partner
How engaging it is—for everyone involved

Great conversationalists strike the right balance between topic-switching questions (to hop to new islands) and follow-up questions (to explore islands once you’re there). Typically, asking mostly follow-up questions doesn’t mean staying on the same topic—they lead your partner through exciting drift, opening new worlds even as they spring from what was previously said.

For Data Engineers: Learning from Masters

Terry Gross, Oprah Winfrey, and Barbara Walters are follow-up masters. They don’t need specialized expertise to discuss any topic because they can ask follow-up questions. It’s the ultimate improv tool, especially when you don’t understand or know what to say.

Specific patterns to emulate:

Emotional probing : “How did that make you feel?” When a stakeholder says “the migration was challenging,” instead of proposing solutions: “What was most frustrating about the process?”

Past-present-future : “If you had to do it again, would you choose Snowflake?” “Are you happy with the decision now?” “What will you tell the next team considering this migration?” This breaks people out of present-myopia and reveals new insights.

Journalistic curiosity : When in doubt, just keep asking. The key to good conversation isn’t knowing—it’s learning. Be interested in your partner, not interesting yourself.

When a colleague mentions they’re struggling with data quality, resist immediately showcasing your expertise. Instead:

“What specific quality issues are you seeing?”
“How is this affecting downstream teams?”
“What have you tried so far?”
“What would ideal state look like?”

You’ll learn more, build better rapport, and position yourself as someone who actually listens—a rarity in technical organizations.

The Boomerasking Trap

Brooks identifies a destructive pattern called “boomerasking”—asking questions purely to talk about yourself. “Have you won any contests lately?” followed immediately by “I won the Pick a Present contest!” The question goes out and returns like a boomerang.

In technical contexts, this manifests as:

“How did your team handle the migration?” → Immediate pivot to “Here’s how we did ours”
“What challenges are you facing?” → Instant “Let me tell you about our challenges”

Sociologist Charles Derber distinguishes between shift responses (redirecting to yourself) and support responses (maintaining focus on your partner):

Shift: “Yeah, I had the same issue on my last project—let me tell you what happened” Support: “What’s making the timeline feel tight?”

For technical professionals, the instinct to demonstrate expertise through personal examples creates precisely the wrong impression. Three types of support responses build trust:

Background acknowledgments: “Seriously?” “Oh wow!”
Supportive assertions: “That sounds really difficult”
Supportive questions: “What happened next?” “How did that affect the team?”

How This Applies to LLM Communication

The best LLM interactions are conversations, not commands. Follow-up with Claude based on responses:

Initial: “Can you help me optimize this SQL query?”

Claude responds with a solution

Follow-up: “Why did you choose a CTE instead of a subquery here? What are the performance implications?”

Claude explains

Follow-up: “In what scenarios would the subquery approach be better?”

This creates a learning conversation rather than a one-shot transaction. You’re not just getting code—you’re understanding the reasoning, which makes you better at writing queries independently.

Same pattern works in code review. Instead of “Why did you do it this way?” (which sounds accusatory), try:

“Can you walk me through your thinking here?”
“What were you optimizing for?”
“Did you consider [alternative]? What made you choose this approach?”

Follow-up questions signal curiosity rather than criticism, making people more willing to explain and learn.

L Is for Levity: Energy Management, Not Entertainment

Brooks introduces levity using an emotion chart with two dimensions: arousal (high vs low energy) and valence (pleasure vs displeasure). When conversations drift into the lower-left quadrant (sad, bored, disengaged), people disengage. Minds wander. Eye contact drops. Words become neutral or negative. Pauses lengthen.

In the upper-right quadrant (excited, engaged, joyous), the opposite happens. People lean forward, make eye contact, give back-channel feedback (“yeah,” “uh-huh”), respond quickly, even interrupt—because they’re excited to know what happens next.

Levity is any conversational move—playful, funny, unexpected, warm—that infuses positive energy. It’s the yeast that turns dense dough into fluffy bread, the helium keeping a balloon aloft. It moves conversations from disengagement to engagement.

Why Engineers Underuse Levity

Research shows people overestimate how often humor goes poorly and underestimate the upside. At age 23, 84% of people report smiling and laughing boosts mood, draws people closer, and increases perceptions of competence.

Brooks’ research found that managers randomly assigned to make one joke in one conversation were over 9% more likely to be voted into leadership positions. Bosses with any sense of humor are 27% more motivating and admired than those with none, and their employees are 15% more engaged.

The individual risk that one-off jokes will flop is much smaller than the aggregated risk of a lifetime of boring, disengaged conversation. But our worries aren’t totally unfounded—the trick is getting it right.

The Benign Violation Theory

Psychologists Peter McGraw and Caleb Warren suggest humor works when things are neither too benign (boring) nor too violating (scary, aggressive, inappropriate)—but somewhere between.

Simply walking downstairs: benign. Falling and breaking your arm: violation. Pretending to fall, landing in a silly pose: benign violation.

The challenge: the sweet spot moves depending on context, relationship, and individual humor styles. What seems funny from sweet-Olivia might seem mundane from sarcastic-Jimmy.

Affiliative vs Aggressive Humor

Affiliative humor brings people together, makes everyone feel included, increases psychological safety.

Aggressive humor involves put-downs or insults, decreases psychological safety, fragments people into categories.

When in doubt, be gentle. It’s less costly to flop for being too benign than too aggressive. Failed gentle humor means people don’t laugh and move on. Failed aggressive humor means hurt feelings that take a long time to heal.

Self-Deprecation: The Technical Professional’s Secret Weapon

Former congressman Ric Keller traces his political career start to a joke at age 34. After sitting through hours of serious speakers, he opened: “I feel like Elizabeth Taylor’s seventh husband on his wedding night. Technically, I know what I’m supposed to do. But at this point, I don’t know how to make it interesting.”

Cheesy? Sure. But it got a huge laugh from a crowd hungry for a fresh voice. Self-deprecation works best when leaders and high-status people use it—it makes them seem more approachable and human.

For technical professionals presenting to executives:

“This dashboard has so many charts I’m not sure if it’s a Tableau workbook or modern art”
“I’ve explained this architecture so many times I dream in flowcharts”
“Our data pipeline is held together with duct tape and prayers—but it’s documented duct tape and prayers”

Self-deprecation shows vulnerability while maintaining competence. You’re acknowledging imperfection without undermining your expertise.

What Levity Isn’t

Don’t make fun of stakeholders, team members, or anyone with lower status. “Punching down” kills psychological safety and trust. The New Yorker cartoon wisdom—“If you can’t say something nice, say something clever but devastating”—might work for comedy, but it’s terrible for professional relationships.

If you’re choosing between gentle-but-cheesy and clever-but-devastating, choose the former. Everyone likes to laugh, but nobody likes to be laughed at.

For Data Engineers: Practical Levity

Levity isn’t about being a comedian. It’s about energy management—keeping conversations in the engaged quadrant. Specific tactics:

In presentations: Start with a light observation. “I know the last thing anyone wants Monday morning is another architecture diagram, but I promise this one has colors.” Not hilarious, but it acknowledges shared reality and signals you’re human.

In meetings: When discussions get too heavy, inject perspective. “We’ve been debating this for 30 minutes. I propose we decide by seeing who can juggle the most coffee cups.” Gets a laugh, resets energy, helps people realize they’re overthinking.

In Slack: Use gentle humor to soften messages. Instead of “This code has bugs,” try “Found some unexpected features in the code—I think the bugs are multiplying via asexual reproduction.”

In documentation: “Note: This regex looks like someone sat on the keyboard. I promise it makes sense.”

The goal isn’t entertainment—it’s keeping people engaged when discussing complex topics. Small moments of levity prevent cognitive fatigue and make information more memorable.

How This Applies to LLM Communication

LLMs respond to tone. Compare:

No levity: “Explain dependency injection”

With levity: “Explain dependency injection like I’m a curious golden retriever who somehow learned to code but still gets confused by fancy patterns”

The second isn’t just more fun—it actually produces better results because it gives Claude clear framing about complexity level and explanation style. You’re using levity to establish shared context.

K Is for Kindness: Putting Others’ Conversational Needs First

Brooks’ final maxim is the most challenging: trying your utmost to put others’ conversational needs first. Not always possible, not always optimal, but reaching for it relentlessly gives you the best chance of being the partner you mean to be.

The Anderson Cooper and Stephen Colbert conversation about grief demonstrates this beautifully. Both men lost fathers and brothers tragically young. When Cooper interviews Colbert on CNN—no live audience, just two people in director’s chairs—they create one of the most powerful public conversations on grief you’ll watch.

What makes it work isn’t just the topics (grief, loss, mothers) or the questions (14 questions in 20 minutes) or even the levity (warm laughter every 2-3 minutes). It’s their invisible but palpable focus on the other person—their words, their struggle, their life, their needs.

Cooper prepared extensively—read previous Colbert interviews, can quote his words, has notes and questions ready. Colbert is self-deprecating, allows Cooper to lead, responds to vulnerability with vulnerability. When one offers a thought, the other builds on it or discloses in return.

Ten minutes in, Cooper says: “I wish I had, like, a scar. A Bond villain scar running down my eye and face” to signal “I’m not the person I was meant to be.”

Colbert shifts in his seat. Takes off his glasses. Smiles. “But you’re entirely the person you were meant to be.”

This is kindness in conversation—gently challenging a thought that might not serve Cooper, even when it creates tension. Cooper asks, voice breaking, “Do you really believe that?” Colbert pauses, maintains unwavering gaze, replies simply: “Yes.”

Kindness doesn’t mean agreeing with everything. It means figuring out what your partner needs and helping them get it—encouragement, hard feedback, new ideas, a laugh, a sounding board, challenging questions.

The Egocentrism Problem

Psychologist Jean Piaget showed that around age seven, children begin recognizing others have different perspectives. Before that: extreme egocentrism. Hands over eyes during hide-and-seek? They think you can’t see them. “That’s mine!” isn’t compelling justification—it’s self-centered assertion.

Here’s the devastating fact: we don’t shed egocentrism after age seven. It operates constantly in the background, sabotaging our ability to converse effectively.

Egocentrism makes us choose topics we like (assuming our partner will like them because we do). It makes us ask questions we find interesting rather than ones they’d be excited to answer. It makes us create humor that we think is funny without considering whether it serves them.

Researcher Boaz Keysar demonstrated this brilliantly. He had pairs sit back-to-back as speaker and listener. Speakers imagined scenarios (“suspect your friend has been planning a surprise”) then delivered lines (“What have you been up to?”) with any inflection. Listeners chose the intended meaning from four options.

Results: Listeners thought they correctly identified meaning 85% of the time. Speakers thought listeners understood 70% of the time. Actual accuracy: 44%.

Even more striking: Keysar repeated this with people who didn’t speak the same language. Chinese speakers, English listeners. American listeners believed they understood 65% of the time. Chinese speakers believed they’d been understood 50% of the time. Actual accuracy: 32%.

George Bernard Shaw: “The single biggest problem with communication is the illusion that it has taken place.”

We can’t prioritize others’ conversational needs if we don’t understand them, especially when we mistakenly believe we do. Kindness requires continuously checking that illusion.

For Data Engineers: Practical Kindness

In technical discussions, kindness means matching your partner’s level:

With junior engineers: Don’t overwhelm with advanced concepts. Ask “Does this make sense?” and actually wait for response. Explain the “why” behind decisions, not just the “what.”

With senior leaders: Don’t bury them in technical details. Start with business impact, then offer technical depth as needed. Ask “What level of detail would be helpful?”

With non-technical stakeholders: Use their domain language, not yours. Instead of “We need to denormalize the schema to improve query performance,” try “We need to restructure how we store data so reports run faster. Think of it like rearranging a library—same books, faster to find what you need.”

In difficult conversations, kindness means creating safety:

When delivering bad news: “I need to tell you something that’s going to be disappointing” (prepares them) vs. abrupt revelation.

When receiving feedback: “I appreciate you bringing this up—help me understand what you’re seeing” vs. defensive response.

When disagreeing: “I see it differently—can I explain my thinking?” vs. “That’s wrong.”

In team meetings, kindness means ensuring everyone’s heard:

Notice who hasn’t spoken: “Jamie, you’ve been quiet—what’s your take on this?”

Build on others’ ideas: “Building on what Alex said…” vs. introducing competing idea.

Redirect credit: “That was Sarah’s insight—Sarah, can you explain how you approached it?”

The Hardest Part

Always putting others’ needs first is unrealistic—and sometimes not optimal. You have legitimate needs on the low-relational side of the conversational compass. The advice isn’t to be selfless martyr.

The advice is to try relentlessly. To continuously ask yourself: “What does this person need from this conversation?” Then help them get it.

Topics and questions provide substance. Levity keeps people engaged. Kindness creates space where meaningful exchange can flourish—where people feel respected and valued. It’s the master class that takes other skills to elite level.

How This Applies to LLM Communication

Yes, kindness applies to AI conversation. Not because LLMs have feelings, but because kindness frameworks produce better results.

Provide context about what you need:

Bad: “Write me a function”

Good: “I need a function that parses JSON from an API response. I’m fairly comfortable with Python but new to this API, so I’d appreciate comments explaining the approach.”

Acknowledge what’s working:

“That solution worked perfectly for the first case, but when I tried it with nested JSON, I got an error. Here’s what I’m seeing…”

Be specific about confusion:

Instead of “This doesn’t make sense,” try “I understand the first two steps, but I’m confused about why you’re using a set instead of a list in step 3. Can you explain the reasoning?”

You’re not being polite to a machine—you’re practicing communication patterns that work with humans too. Every interaction is practice for the coordination game.

The Through-Line: System Design for Human Connection

Brooks’ research points to a fundamental reframe: “The key to good conversation isn’t knowing, but learning. It’s about being interested in your partner, not interesting yourself.”

This should resonate with analytical professionals. You don’t need charisma or extroversion. You need to deploy genuine curiosity—something most technical people already possess in abundance for technical problems—toward the humans you work with.

The TALK framework provides structure:

T - Topics: Prepare 3-5 topics before important conversations. System 2 thinking frees mental space for system 1 responsiveness.

A - Asking: Ask more follow-up questions. They signal listening, care, and thoughtfulness. Master conversationalists don’t need specialized expertise—they have specialized curiosity.

L - Levity: Manage energy, not entertainment. Small moments of playfulness prevent disengagement and make information memorable.

K - Kindness: Continuously try to understand and meet others’ conversational needs. Check your egocentrism at the door.

These aren’t difficult techniques: brainstorm topics beforehand, ask more follow-up questions, inject occasional levity, focus on your partner rather than yourself. What makes them powerful is consistent application.

The Career Impact

Communication skills ranked #2 among the most important factors when hiring engineering project managers—higher than technical proficiency at #10. Executive presence accounts for 26% of what it takes to get promoted. Google found their most productive employees weren’t those with highest technical skills but those with developed soft skills who could turn expertise into influence.

Research shows conversation is a learnable skill with predictable patterns. For data engineers willing to treat communication with the same analytical rigor we bring to technical problems, the evidence suggests significant returns—career advancement, team performance, stakeholder trust.

You don’t need to become someone different. You need to become more strategically curious. The research shows exactly how.

Implementation Checklist

Before your next stakeholder meeting:

Brainstorm 3-5 questions based on their likely concerns
Prepare topics that climb beyond status updates
Plan one moment of levity (even just acknowledging shared reality)
Identify what they need from the conversation

During the conversation:

Ask at least one follow-up question for every major point
Notice when you’re shift-responding vs support-responding
Watch for lower-left quadrant (disengagement) and inject energy
Check: am I making this about me or about them?

After the conversation:

What topics worked? Which fell flat?
Did I ask enough questions?
Was there authentic laughter?
Did they seem to get what they needed?

Start with one conversation. Apply one element of TALK. Notice the difference. Build from there.

The same framework that makes you better at technical writing makes you better at stakeholder management. The same curiosity that drives your debugging makes you better at understanding people. You already have the skills—you just need to deploy them systematically.

When AI Stops Treating Us Like People: The Twin Crises Nobody Wants to Discuss

Chris Hillman — Fri, 12 Dec 2025 09:00:00 +1100

The moment it stops being theoretical

A friend was telling me about their recent job interviews. Not the questions they struggled with or the technical challenge they bombed—that’s normal interview anxiety. No, they were frustrated because they’d spent forty-five minutes talking to an AI chatbot that analyzed their “enthusiasm for remote work” and tracked their eye movements while asking whether they’d “misrepresent themselves 3.8x more than average candidates.”

The speed with which I would exit an AI interview would rival even the most seasoned teenager’s alt-tab skills.

But here’s the thing—this isn’t just another complaint about awkward technology or corporate efficiency theater. This moment, repeated thousands of times daily across hiring processes, represents something more fundamental breaking down. AI was supposed to eliminate the boring parts of work, the repetitive drudgery that didn’t require human judgment. Instead, it’s eliminating the human parts of work—the connection, the authenticity, the actual assessment of whether someone can do the job.

And that pattern—AI deployed to dehumanize rather than enhance, to reduce costs rather than create value—isn’t limited to hiring. It’s the story of a $560 billion investment that’s generated just $35 billion in revenue. It’s the story of public trust collapsing from cautious optimism to outright skepticism. It’s the story of two interconnected crises that could reshape the entire technology industry if either one fails.

What in the dystopian hellscape is going on

Let’s stay with that hiring example for a moment, because it crystallizes the problem. Companies now use AI to screen resumes, conduct first-round interviews, analyze voice patterns for “culture fit,” track eye movements for “engagement signals,” and evaluate “authenticity” through speech pattern analysis. The justification? Candidates are supposedly committing fraud “3.8x higher rates”—though higher than what, when, or where remains conveniently unspecified.

So AI is good when companies use it to dehumanize candidates, but it’s bad when candidates use it to stay competitive in that same dehumanizing process. Got it.

The fundamental issue, is that “AI was supposed to do the parts of the jobs humans found boring or repetitive.” Taking meeting notes. Generating documentation. Processing expense reports. The stuff that makes you think “surely a computer could do this.” Instead, we’re using it for the parts that desperately need human judgment—assessing whether someone will thrive in a role, whether they’ll mesh with a team, whether their particular combination of skills and experience makes them the right fit for this specific challenge.

If I’m hiring a human for a role, I don’t care about grading their every word or analyzing their eye movement patterns. Doing so is fundamentally counterproductive since it encourages candidates to be as boring and uninteresting as possible while maintaining perfect eye contact and regurgitating perfect phrases, lest they don’t get the job.

Honestly? I don’t think I want to hang out with the people who AI likes anyway.

But this pattern—deploying AI in ways that optimize for the wrong things—appears everywhere once you start looking. McDonald’s AI drive-thru adding 260 Chicken McNuggets to orders. Klarna claiming AI “did the work of 700 agents” before quietly reversing course when quality tanked. Google’s AI Overview suggesting you add glue to pizza. These aren’t bugs. They’re symptoms of a systematic misapplication of technology by companies that haven’t figured out what problems AI actually solves.

The numbers that should terrify you

OpenAI, the company most synonymous with the AI revolution, lost $5 billion in 2024 on revenue of just $3.7 billion. Let that sink in. They’re spending $2.25 for every dollar they earn. The projected loss for 2025? $14.4 billion. Between 2023 and 2028, the company expects cumulative losses of $44 billion before potentially—possibly, theoretically—reaching cash-flow positive status in 2029 or 2030.

Despite these staggering losses, OpenAI’s valuation rocketed from $157 billion in October 2024 to somewhere between $300-500 billion today. That’s a tripling in twelve months with no profitability in sight. These aren’t valuations based on business fundamentals. They’re bets that future revenue will somehow, miraculously, justify present spending.

Anthropic tells the same story with different numbers. Revenue jumped from $1 billion annually in 2024 to a projected $9 billion by end of 2025—impressive growth, right? Except they’re burning through $2.7-3 billion annually and don’t expect to break even until 2028. The valuation inflated tenfold in eighteen months, from $18.4 billion to $183 billion. Again: not a sustainable business, but a bet on a future that may never materialize.

The cost structure explains why profitability remains this elusive unicorn. Every ChatGPT query costs real money to compute. Every Claude conversation. Every AI-generated image. Inference—running trained models—accounts for 80-90% of total AI lifetime expenses. Unlike traditional software where marginal costs approach zero as you scale, AI scales in reverse. More users mean more costs, linearly and predictably.

Cursor, the AI coding assistant, reportedly sends 100% of its revenue to Anthropic just for model access. Perplexity AI spent 164% of revenue on compute providers in 2024. When your biggest customers operate at negative margins, when the companies building businesses on your platform lose money on every transaction, the entire ecosystem becomes brittle.

The hyperscalers—Microsoft, Google, Amazon, Meta—are betting their futures on AI infrastructure with combined 2025 capital expenditures projected at $325-380 billion, up 46% from 2024. These spending levels now exceed 1% of U.S. GDP quarterly. Microsoft committed $88.7 billion for fiscal 2025. Amazon projected $105-125 billion. Yet Meta’s CFO acknowledged that “genAI work is not going to be a meaningful driver of revenue this year or next year.”

Read that again. Companies are spending hundreds of billions building infrastructure for revenue that their own financial officers admit won’t materialize for years. The gap between investment and returns has never been wider in technology history.

When the guy who called 2008 places a billion-dollar bet

Michael Burry isn’t some permabear shouting about every market downturn. He’s the investor who correctly identified the 2008 subprime mortgage crisis before it happened, who placed massive bets against the housing market when everyone called him crazy, who was proven devastatingly right when the entire financial system nearly collapsed.

In early November 2025, Burry disclosed regulatory filings showing over $1 billion in put options against AI leaders: $912 million against Palantir (66% of his portfolio) and $187 million against Nvidia. His thesis is explicit and detailed, laid out in his Substack newsletter “Cassandra Unchained.”

“I am not claiming Nvidia is Enron. It is clearly Cisco,” Burry wrote. He’s identifying Nvidia as today’s equivalent of Cisco during the dot-com bubble—the essential hardware supplier powering a massive capital investment cycle that will eventually see overbuilt supply meet far less demand than expected.

But his most damning observation concerns accounting practices. Burry projects that tech companies are understating depreciation by $176 billion between 2026-2028 by extending the useful life of AI computing equipment from realistic 2-3 year product cycles to artificially long periods. This accounting treatment inflates reported earnings—Oracle by 26.9%, Meta by 20.8%—creating the illusion of profitability where losses actually exist.

His criticism of “circular financing” cuts deeper. OpenAI owns 10% of AMD while Nvidia invested $100 billion in OpenAI. Microsoft is both a major OpenAI shareholder and one of Nvidia’s largest customers (representing roughly 20% of Nvidia revenue). CoreWeave, funded by Nvidia, serves OpenAI’s compute needs. “True end demand is ridiculously small,” Burry observed. “Almost all customers are funded by their dealers.”

When an investor who successfully predicted the last major financial crisis places billion-dollar bets against your industry and explicitly compares your accounting practices to pre-2008 mortgage fraud, that’s not speculation. That’s pattern recognition.

The trust collapse nobody wants to acknowledge

Here’s the part where we connect the financial instability back to that AI interview experience. Because these crises aren’t separate—they’re feeding each other in ways that threaten the entire AI business model.

According to Pew Research, only 17% of Americans believe AI will have a positive impact on the U.S. over the next 20 years. A full 51% are more concerned than excited about AI’s increased use in daily life. These figures represent dramatic worsening since 2021, when optimism still outweighed concern.

The gap between expert opinion and public sentiment has become a chasm. While 56% of AI experts see positive impact ahead, only 17% of the general public agrees. While 73% of experts expect positive employment effects, only 23% of the public shares that view. Either the experts are completely disconnected from reality, or the public has accurately identified risks the industry refuses to acknowledge.

The trust erosion isn’t abstract. It’s rooted in lived experience. That AI interview that made you feel reduced to “an n-dimensional matrix.” The AI-generated customer service response that completely missed your problem. The hiring algorithm that rejected your application before a human ever saw it. The chatbot that confidently gave you wrong information. The image generator that couldn’t distinguish historical accuracy from Reddit jokes.

That “authenticity is a limited resource that is diminishing every day.” Every time someone encounters AI deployed to reduce costs rather than add value, deployed to eliminate human judgment rather than enhance it, that person becomes incrementally more skeptical. Not of technology in general—of this specific application pattern where efficiency trumps everything else.

Consumer behavior reflects this skepticism in ways that directly threaten revenue. A Washington State University study found that products described as “AI-powered” are consistently less popular with consumers. Gartner surveys show 64% of customers prefer companies NOT use AI for customer service, with 53% willing to switch to competitors that don’t. Only 3% of smartphone owners are willing to pay extra for AI features.

Three percent.

When you’re building a business model that depends on massive consumer adoption and widespread willingness to pay premium prices, 3% willingness to pay is a death sentence. It means the revenue models underpinning all those billion-dollar valuations are built on fundamentally wrong assumptions about demand.

The enterprise adoption crisis

Maybe consumer skepticism doesn’t matter, you might argue. Maybe the real money is in enterprise. Except enterprise adoption is stalling for the same reasons, just with bigger price tags attached to the disappointment.

The MIT NANDA study found that 95% of enterprise AI pilots yield zero measurable business return despite $30-40 billion in GenAI investment. BCG reports 74% of companies haven’t seen tangible value from AI initiatives. McKinsey’s surveys show 80%+ of organizations haven’t realized significant bottom-line AI impact. Gartner predicts 30% of GenAI projects will be abandoned by end of 2025 due to poor data quality, escalating costs, or unclear value.

Monte Carlo’s 2024 survey of data professionals reveals the pressure cooker: 100% feel pressure from leadership to implement GenAI strategy, yet 90% believe their leaders do NOT have realistic expectations for what’s technically feasible or capable of driving business value. Only 12% of companies have dedicated AI teams—meaning 84% rely on existing data teams to figure it out.

This is the reality on the ground for data professionals. Leadership demands AI transformation. Budgets flow to AI initiatives. But the actual work of implementation falls to teams who know the technology isn’t mature enough, the data isn’t clean enough, the use cases aren’t well-defined enough. The pressure to deploy anyway—to show “innovation,” to justify spending, to keep up with competitors—leads to implementations that fail quietly, generate minimal value, and burn credibility.

The case studies pile up. McDonald’s ended its three-year AI drive-thru partnership with IBM in June 2024 after viral videos showed the system adding 260 Chicken McNuggets to orders and repeatedly misinterpreting requests. Klarna initially claimed AI was “doing the work of 700 agents” after cutting 22% of its workforce, then reversed course with a “major hiring initiative” in May 2025. The CEO admitted AI was actually “doing work of 700 really bad agents”—quality decline cost customers, and the rollback cost approximately $15 million with no net gain.

IBM’s Watson Health, once touted as transformative for cancer diagnosis, never reached production use, ran over budget, and was eventually sold for approximately $1 billion after multi-billion dollar investment. The Humane Ai Pin and Rabbit R1 represent hardware category failures, with slashed prices following weak sales and critical reviews noting the products solve “problems that don’t exist.”

Each failure compounds the trust deficit. Each over-promised and under-delivered implementation makes the next pitch harder. Each news story about AI gone wrong creates more skeptics who start questioning whether this technology delivers value or just disrupts things that were working fine.

The 2008 playbook, now in tech

The parallels between current AI investment patterns and pre-2008 financial crisis have moved from concerning to alarming. Deutsche Bank is now using Synthetic Risk Transfers (SRTs) to hedge AI infrastructure loans—structures that analysts say “strongly resemble CDOs and CDSs that amplified the 2008 financial crisis.” Tech companies shift AI spending into Special Purpose Vehicles (SPVs) to obscure true costs, echoing Enron’s infamous off-balance-sheet accounting.

The leverage buildup mirrors pre-crisis patterns. Goldman Sachs reports hyperscalers took on $121 billion in debt over the past year—a 300%+ increase from typical levels. Meta raised $29 billion in debt for a single Louisiana data center. Oracle operates at a 500% debt-to-equity ratio to compete in AI data centers. Morgan Stanley estimates Big Tech will spend $3 trillion on AI infrastructure through 2028, with cash flows covering only half.

The dot-com parallels are equally sobering. In the late 1990s, 80+ million miles of fiber optic cable were laid based on inflated demand projections; 85-95% remained unused (“dark fiber”) years after the bubble burst. Today’s equivalent is the massive AI data center buildout—Meta constructing a facility covering a “significant part of Manhattan”—predicated on demand growth that may never materialize. The $500 billion Stargate Project promises unprecedented AI infrastructure investment for enterprise AI adoption that’s already stalling.

The NASDAQ took 15 years to recover from its 2000 peak. Cisco shares peaked at $80 and still haven’t returned to that level twenty-five years later. Those aren’t cautionary tales from ancient history—they’re previews of what happens when infrastructure investment radically outpaces actual demand.

What the insiders are saying

Perhaps most telling are the warnings from industry insiders themselves. Sam Altman, OpenAI’s CEO, acknowledges that investors are “overexcited about AI” and predicts “people will overinvest and lose money during this phase.” Google CEO Sundar Pichai warns of “elements of irrationality” and says “no company is going to be immune” if the bubble bursts. Jeff Bezos called the current environment “kind of an industrial bubble.”

When the people building and funding AI companies publicly warn about bubble dynamics, when they’re literally saying investors will lose money, that’s not bearish speculation. That’s reality acknowledgment.

Ray Dalio’s proprietary bubble indicator sits at approximately 80% of historical bubble peaks, with his AI chatbot estimating a 65-75% probability of meaningful correction in AI equities by end of 2026. Morgan Stanley’s Lisa Shalett warned of a “Cisco moment” within 24 months, describing the market as a “one-note narrative” entirely dependent on AI capital expenditures.

The timing predictions are getting specific. Motley Fool analysts suggest the bubble could burst in 2025 due to unsustainable valuations. Ruchir Sharma of Rockefeller International predicts 2026 if interest rates rise. BeyondTrust researchers argue “artificial inflation of AI has already peaked in 2024.” A November 2025 Nature analysis noted that “many financial analysts now agree there is an ‘AI bubble,’ some speculate it could finally burst in the next few months.”

Preparing for what comes next

Whether the bubble bursts catastrophically or deflates gradually, data professionals can take specific steps to build resilient careers. The skills that remain human-essential include critical thinking and complex decision-making, cross-contextual reasoning (connecting insights across domains), ethical judgment in novel situations, relationship building and stakeholder management, and what researchers call “AI discernment”—knowing when to trust AI versus human judgment.

Technical preparation should emphasize both breadth and depth. Core competencies remain essential: probability, statistics, Python, SQL. But add ML fundamentals including embeddings and model monitoring. Develop cloud platform expertise. Build governance and ethics capabilities that will be increasingly valuable as regulation tightens. Document measurable business impact, not just technical achievements—when budgets tighten, demonstrated ROI becomes the difference between retention and reduction.

For data leaders, the most important preparation may be honest communication. Setting realistic expectations with executive leadership about what AI can deliver—and on what timeline—protects both the organization and the team. The companies that navigate the potential correction successfully will be those treating AI as a tool for specific problems rather than a magic solution for every challenge.

What happens when authenticity runs out

I desperately want to be surrounded by real people with real hobbies and passions. I don’t care if a candidate loves remote work, I care about their humanity. I don’t care if a candidate tripped over their words a few times, I care about their authenticity.

This isn’t nostalgia for pre-AI times. It’s recognition that authenticity, human connection, genuine judgment—these aren’t inefficiencies to optimize away. They’re the core value proposition. They’re what makes work meaningful, what makes organizations effective, what makes products worth buying.

When companies deploy AI to eliminate these elements—to reduce humans to vectors, to automate judgment, to replace connection with efficiency—they’re not just failing to capture value. They’re actively destroying it. The companies that will survive aren’t those with the most sophisticated AI. They’re those that figured out how to use AI to enhance human capability rather than replace it, to preserve authenticity rather than eliminate it.

The crisis isn’t coming. It’s already here, building pressure daily in the gap between investment and returns, between promises and delivery, between what AI can actually do and what we’ve been told it will become. The question isn’t whether there will be a reckoning. The question is whether you’ll be positioned to navigate it when it arrives—and whether the companies you work for will still treat you like a person when they do.

The Invisible PR You're Building Right Now

Chris Hillman — Mon, 08 Dec 2025 09:00:00 +1100

The Email That Changed Everything

I received a meeting invite labeled “Check In” and it was to discuss news about the restructure. In a few months, my role would be gone. I was still processing it, sitting with that particular brand of numbness that comes when your career gets upended. In the following weeks, my inbox started filling up. LinkedIn messages. Texts. Emails from people I’d worked with years ago.

They weren’t condolences. They were stories.

“You probably don’t remember, but five years ago you sat with me for an hour and helped me understand why my pipeline kept failing. You didn’t have to do that—I wasn’t even on your team.”

“The way you ran those workshops on values and purpose changed how I lead. I still use it now.”

“You once told me I was ready for a promotion before I believed it myself. That conversation changed my career trajectory.”

Message after message. Hundreds of them. People I’d mentored, challenged, collaborated with, sometimes clashed with. And sitting there, reading through them, I realized something:

I’d been building a reputation without knowing it. Every small interaction, every moment I thought was insignificant—they’d all been compounding silently in the background.

This is what I’ve come to think of as invisible PR.

The Marble Jar of Trust

Brené Brown talks about trust as a marble jar. Every time someone shows up for you, follows through, or demonstrates integrity in a small way, they put a marble in your jar. Every time they let you down or act inconsistently, they take marbles out. Over time, the jar fills or empties based on these micro-transactions.

But here’s what Brené doesn’t emphasize enough: you can’t see the jar. You don’t know how many marbles you have until you need them. And you definitely don’t know which specific moments put marbles in or took them out.

That’s the invisible part.

We spend so much time thinking about the big moments—the presentations to executives, the crucial project deliveries, the performance reviews. But your reputation isn’t built there. It’s built in the margins. In the small interactions that feel inconsequential in the moment but echo for years.

Let me show you what I mean.

Jenny’s Birthday Cake

About four years ago, someone on one of the teams I worked with was leaving. Her name was Jenny (the name is fictional for this purpose), quiet, did solid work but never made a lot of noise. The team was busy, everyone had deadlines, and honestly, the person leaving wasn’t particularly senior or well-connected. It would have been easy to let them slip out with just a generic goodbye email.

Jenny organized a celebration lunch. She bought a cake, got people to sign stories in a card, and made sure the person felt genuinely appreciated on their last day. It took her maybe two hours of effort and fifty dollars of her own money.

I watched her do this and filed it away. I didn’t say anything at the time—it wasn’t about praise. It was just data about who Jenny was when nobody was watching.

Fast forward two years. A senior analyst position opened up, and I had two candidates who were roughly equal on technical skills. One was Jenny. The other had a slightly stronger resume on paper. But I kept thinking about that birthday cake. Not the cake itself, but what it revealed: Jenny showed up for people. She created culture. She understood that teams are built in the small moments, not just the big deliveries.

Jenny got the promotion. She has no idea that birthday cake was part of the decision. And that’s exactly the point—she didn’t do it for recognition. She did it because that’s who she is. That’s invisible PR at work.

When It Goes Wrong

But here’s the thing about invisible PR: it works both ways. And it compounds in both directions.

Jenny spent two hours and fifty dollars building trust. But you can destroy trust in thirty seconds. And the withdrawal compounds faster than the deposit ever did.

I’ve watched this play out dozens of times over the years. The person who consistently doesn’t respond to requests from other teams. The leader who’s sharp with people when they’re under pressure. The colleague who’s helpful to senior stakeholders but dismissive to juniors. The teammate who commits to something in a meeting and then goes silent.

None of these are career-ending moments. But they’re reputation-defining ones.

Because here’s what happens: someone has that negative experience, and they tell a story. Not maliciously—just factually. “Oh, that team? Yeah, I tried working with them once and…” or “Be careful with that person, they tend to…” And that story spreads. It compounds. Five people hear it. Then fifty. Then it becomes the thing people “know” about you before they’ve ever met you.

I’ve seen this up close with my own work. I do this exercise with new team members called personal maps. It’s simple: sit down with someone for 30-45 minutes and just understand their story. Where did they grow up? Where did they go to school? What did they study? What are their hobbies? What matters to them outside of work?

It’s not an interrogation—it’s an invitation to be seen. And it’s remarkably effective at building trust quickly with new teams. I’ve done personal maps with over 200 people at this point. Some have been surface-level, some have gone deep and emotional. But they’ve all been valuable.

Only one ever went badly.

The person became immediately defensive. They didn’t want to share anything personal. They questioned why I was asking. They made it clear they thought the whole exercise was invasive and pointless. I could feel the tension in the room, so I stopped. “No problem,” I said. “We don’t have to do this.”

I’ve told that story dozens of times—not to shame, but because people are curious if it has always worked. It taught me something important about consent and boundaries in team-building. Not everyone wants to connect the same way, and that’s okay.

But think about how that interaction cascades. People will hear the story and form an impression of someone who treats connection as invasion. That’s their invisible PR working against them.

Now, here’s the critical distinction: if this was a one-off, if other people had completely different experiences, the story would fade. But if people hear it and think, “Oh yeah, I had a similar experience with them…"—if the pattern repeats—then you’ve got a reputation problem.

That’s the asymmetry of invisible PR. Positive interactions need repetition and consistency to build trust. Negative interactions? They only need a pattern. Two or three similar stories, and suddenly that’s who you are in other people’s minds.

You can’t control whether someone tells your story. But you can control whether the stories align or contradict each other.

The Hard Truth About Micro-Transactions

Here’s what I’ve come to understand: every single interaction you have is either a deposit or a withdrawal in someone’s invisible ledger of you. You don’t get to choose which interactions matter. You don’t get to designate some as “important” and others as “throwaway.”

The junior person you’re curt with in the hallway might be the hiring manager in three years. The colleague whose email you ignore might be your only advocate when you’re facing redundancy. The person whose birthday cake you buy might become your strongest reference when you’re job hunting.

Or flip it around: the person you snap at under stress might tell that story for a decade. The email you send when you’re annoyed might define how an entire team sees you. The meeting where you’re dismissive might be the only interaction someone has with you before making a judgment about your character.

You can’t control how people remember you. But you can control how you show up.

This connects directly to the research on trust and psychological safety. Amy Edmondson’s work on psychological safety shows that teams perform better when people feel safe to take interpersonal risks—to speak up, to ask questions, to admit mistakes. But that safety isn’t created through policy or pronouncements. It’s created through thousands of small interactions where people experience acceptance rather than judgment.

Your invisible PR is the sum of how you show up in these moments when you think nobody’s keeping score.

Plot twist: everyone’s keeping score. They just don’t tell you.

The Offer I Make Every New Team

Whenever I inherit a new team or start leading a new group, I do something that makes some people uncomfortable. In our first team meeting, I tell them this:

“I know you don’t know me yet. I know you’re taking a risk trusting me as your leader. So here’s what I’m going to offer: if you want to talk to anyone I’ve worked with in the past—anyone—I will personally help you get in touch with them. I will give you names, email addresses, LinkedIn profiles. You can ask them anything you want about what it’s like to work with me.”

Some people think this is weird. Some think it’s a power move. But it’s neither. It’s confidence born from consistency.

I can make this offer because I trust my invisible PR. I know that if you talk to people I’ve led, you’ll hear stories about me being demanding, about me challenging people, about me pushing them outside their comfort zones. Not everyone loved working with me—I’m not trying to be everyone’s friend. But you’ll also hear about me showing up, following through, investing in their growth, and treating them with respect even when we disagreed.

That’s the invisible PR I’ve built. And it’s portable. It travels with me from team to team, role to role. It’s why, when I got the redundancy news, my first thought wasn’t panic—it was curiosity about what would come next. Because I knew the reputation I’d built would carry me through.

Inconvenience vs. Problem

This is where we get to the distinction I’ve been thinking about a lot lately: the difference between an inconvenience and a problem.

Losing your job is objectively bad news. But whether it’s an inconvenience or an existential problem depends entirely on the invisible PR you’ve built. If you’ve spent years burning bridges, ignoring relationships, and treating people as transactional resources, that job loss becomes a crisis. Your network is thin. Your references are weak. Your reputation precedes you in the wrong way.

But if you’ve been making deposits—showing up for people, building trust, creating value even when it didn’t directly benefit you—that same job loss becomes navigable. Your network activates. People reach out. Opportunities surface. The transition becomes an inconvenience you’re managing rather than a catastrophe you’re surviving.

I’m living this right now. Yes, it’s disruptive and stressful. But I’m also flooded with support, with referrals, with people wanting to help. Not because I asked for it, but because I’ve been building invisible PR for years without realizing it. Even to the extent of people wanting to know where I go, as they want to work with me wherever I am.

The variations in work and life won’t always go your way. You’ll face setbacks, restructures, health issues, relationship challenges. Whether these remain inconveniences or escalate into existential problems often depends less on the event itself and more on the reservoir of trust you’ve built.

Connection is the buffer. Community is the shock absorber. And both are built through invisible PR—one small interaction at a time.

The Work That Matters Most

Your reputation is being built right now. Not in the big presentation next week or the annual review in three months. Right now. In how you respond to the next email. In whether you remember to thank someone. In your tone on the next Teams message. In whether you show up for the small things when nobody’s watching.

You’re making deposits or withdrawals with every single interaction. The jar is filling or emptying. And you won’t know which moments mattered most until much later—maybe years later, maybe when you’re sitting at your desk reading messages from people whose marbles you earned without knowing it.

Jenny didn’t know the birthday cake would matter. The person who attacked me over email didn’t know that interaction would define how I see their entire team. We never know in the moment. That’s why the only rational strategy is to treat every moment as if it matters.

Because it does.

Your invisible PR is working right now, compounding in ways you can’t see. The only question is: which direction is it compounding?

When the moment comes that you need to draw on your reputation—and that moment always comes eventually—will you find a full marble jar or an empty one?

Start filling it now. One small interaction at a time.