Dimensional Modelling on Ghost in the data

Ghost Skills: Teaching AI Agents to Think Like Data Engineers

Chris Hillman — Sun, 14 Jun 2026 09:00:00 +1000

Another week, another skills repo on the GitHub trending page. I know. There are roughly seventeen of them now, all promising to turn your AI coding agent from a confident intern into a slightly-less-confident intern. Most of them are great. Most of them are also built by solo devs, for solo devs, on solo-dev codebases that fit comfortably in a context window.

Which is fine, if that’s your world.

Less fine if your world involves a Snowflake warehouse with four tables that could be the source of truth for “customer”, an SCD2 someone half-built in 2021 and quietly walked away from, and a dbt project where stg_users_final_v3_actually_use_this is, somehow, the one you’re meant to use. (Don’t laugh. You’ve seen worse.)

I’ve been using AI agents day-to-day for a while now — mostly as a peer reviewer, a second pair of eyes, a thinking partner that doesn’t sigh when I ask it to look at the same query three times. They’re genuinely useful.

So I built something Ghost Skills — a collection of data-engineering methodology skills for AI coding agents. Before I get into what’s in it, let me explain why I bothered.

The problem isn’t the model. It’s that data work has rules nobody told it about.

Your AI agent is, by default, an extremely bright graduate on their first day. It knows Python. It knows SQL. It can write a beautifully-commented for loop and explain window functions to you in five different ways. It will happily generate a fact table, a dbt model, an Airflow DAG, all of it looking very professional.

What it doesn’t know is everything that actually matters.

It doesn’t know that the customer_id in that source isn’t really unique because of the 2022 CRM migration. It doesn’t know that Roger pushed a release on Friday afternoon (because of course he did) and the platform’s been wheezing through a seven-year backfill ever since. It doesn’t know that this dimension changes slowly and the audit team will lose their minds if you flatten the history. It doesn’t know that the business definition of “active customer” changed in March, but only in the marketing data mart, and only sometimes.

These aren’t model intelligence problems. The model is plenty smart. They’re context problems — and the context is exactly the stuff that takes a data engineer two years to learn and roughly thirty minutes to forget when they leave the company.

What agents are missing isn’t capability. It’s methodology. It’s the durable craft of data engineering — the part that doesn’t change when you swap Snowflake for BigQuery, or dbt for SQLMesh, or your warehouse for a “lakehouse” that’s somehow priced like a warehouse anyway.

That’s the gap this is attempting to fill.

Skills, briefly, for anyone who hasn’t drunk this particular Kool-Aid yet

If you’ve not been living inside Claude Code or Cursor for the last six months: a skill is a folder of instructions you give an AI coding agent. “When you build a fact table, declare the grain first.” “When you write tests, follow this severity model.” “When something breaks, here’s the comms workflow.” You version it, you commit it, you keep it in the repo. The agent reads it before doing the thing, and re-reads it next session because agents have the long-term memory of a goldfish.

It’s a deceptively powerful pattern. Instead of pasting the same prompt every session (and forgetting half of it, and writing it slightly differently this time), the standard lives in the repo. New team member joins? They get the skills. Agent updates next month? It still reads the skills.

The catch — and we’ll come back to this — is that skills tell an agent how you do something. They don’t tell it what you’ve built, why you built it that way, or which of your seven utility tables is the one that’s still maintained.

For solo devs, that’s fine. There is no “seven utility tables”. For data teams at any kind of scale, it’s a real limit. More on that at the end.

Introducing Ghost Skills

Right. The repo: github.com/ghostinthedata-info/skills. Built mostly for myself, polished up for sharing.

30-second setup:

npx skills@latest add ghostinthedata-info/skills

Pick the skills you want, pick your agent (Claude Code, Codex, Cursor — whatever you’re using), and run /setup-ghost-skills. It’ll ask you three questions: warehouse dialect, transform tooling, and how your domain docs are laid out. Then it writes itself a configuration block into your CLAUDE.md or AGENTS.md and gets out of your way.

The agent reads it next session. And the one after that. And the one after that. The same standards, every time, without you re-explaining them.

That’s it. That’s the whole thing.

What’s in the catalogue

The skills group into four areas, mapped to roughly how data work actually happens.

Discovery is everything you should do before you build anything, and frequently don’t. profile-data runs the baseline checks on a new dataset — row counts, cardinality, null analysis, key uniqueness — so the agent isn’t generating models against assumptions that fall apart in week two. gather-requirements pins down grain, sources, consumers, freshness SLAs, and acceptance criteria one question at a time, instead of letting the agent guess. refine-context stress-tests a plan against your documented domain model and writes the decisions back to CONTEXT.md and ADRs as they crystallise.

Modelling is the methodology your agent should already be applying and usually isn’t. dimensional-modeling walks the four-step process. fact-table-design forces grain declaration and measure classification before a single line of SQL gets generated. keys covers business, natural, surrogate, composite, and durable keys — and the anti-patterns that bite you eighteen months later when the source system gets re-platformed. slowly-changing-dimensions covers SCD types 0 through 7, including my Healing Tables approach to deterministic, path-independent SCD2.

Quality is where defensive engineering lives. test-data produces a test plan you can actually defend in a code review — uniqueness, referential integrity, nulls, accepted values, freshness, volume variance — and maps each test to a severity level. performance-tuning follows the measure-first philosophy: find the critical path, then fix partition pruning, incremental processing, and the phantom dependencies that are silently doubling your costs. spark-performance handles the distributed case — partition counts, shuffle minimisation, skew handling, broadcasting small tables.

Operations is what should happen when things go pop. incident-comms gives the agent the severity classification, notification workflow, update cadence, and post-incident review template — the “Don’t Go Dark” pattern in skill form. pipeline-design encodes idempotency, reproducibility, and defensive engineering as defaults instead of afterthoughts. data-as-a-product brings data mesh thinking into the agent — domain ownership, discoverability, SLAs, federated governance. data-security-classification covers the four A’s, PII/PHI handling, and least privilege.

There’s a setup skill (setup-ghost-skills) that runs once per repo and handles the configuration. Everything else, you opt into.

Why tool-agnostic, even though tool-specific would sell better

The temptation when building something like this is to specialise. Snowflake-only. dbt-only. Make every skill assume your exact stack and produce immediately runnable code.

I deliberately didn’t do that, and the tradeoff is worth naming honestly. Generic skills are less immediately useful than highly specific ones. A skill that knows your exact dbt project structure, your column naming conventions, your team’s snake_case-vs-camelCase argument that’s been going on since 2022 — that’s more powerful, day one.

But it’s also yours. It’s not shareable. And the bits that are genuinely universal — declare the grain before you build the fact, a business key isn’t a real business key until you’ve checked it’s unique, test before publish — those don’t change between platforms. They didn’t change when we moved from Teradata to Snowflake. They won’t change when we move from Snowflake to whatever everyone’s furiously rebranding next year.

Ghost Skills is the base layer. Your repo’s CONTEXT.md is your specialisation layer. They’re meant to work together, not replace each other. Fork it. Add a snowflake-cost-optimisation skill on top. Open a PR if you want to share it back. (Or don’t. Do what you like.)

Skills help. They don’t solve everything. Let’s not pretend.

The honest thing to say at this point is that skills are not a silver bullet, and anyone telling you otherwise is selling you a course.

Skills tell an agent how to do things. They don’t tell it what’s there or why. An agent that knows the Kimball four-step process will still make the wrong grain declaration if it doesn’t understand the business process it’s modelling. A skill can encode what SCD2 is; it can’t replace a conversation with the stakeholder who actually needs the history. A skill can encode the Write-Audit-Publish pattern; it can’t tell the agent which of your fourteen “users” tables is the one to run it against.

The other thing skills can’t do is fix a stale skill. The conventions you encode today are the conventions you knew today. If your team’s standards evolve — and they should — your skills need to evolve with them. A stale skill is worse than no skill, because it gives the agent a confident wrong answer instead of asking a question.

So treat them as a starting point. Use them. Fork them. Argue with them in a code review. Update them when your standards shift. The agent will keep reading whatever’s in the repo, faithfully, every session — which means the skills are only ever as good as the last person who maintained them.

The repo is at github.com/ghostinthedata-info/skills. Fork it for your cloud-specific layer. Open issues if something’s wrong. Open a PR if you’ve got methodology worth encoding and you don’t want to keep it to yourself.

And if you build something better on top of it, please tell me.

Healing Tables: When Day-by-Day Backfills Become a Slow-Motion Disaster

Chris Hillman — Sat, 07 Feb 2026 09:00:00 +1100

It was 2 AM on a Saturday when I realised we’d been loading data wrong for six months.

The situation: a customer dimension with three years of history needed to be backfilled after a source system migration. The previous team’s approach was straightforward—run the daily incremental process 1,095 times, once for each day of history. They estimated three weeks to complete.

What they hadn’t accounted for was how errors compound. By the time I looked at the data, we had 47,000 records with overlapping date ranges, 12,000 timeline gaps where customers seemed to vanish and reappear, and an unknowable number of missed changes from when source systems updated the same record multiple times in a single day.

The dimension wasn’t just wrong. It was unfixably wrong using traditional methods. Every incremental run had layered new errors on top of old ones, creating a Jenga tower of data quality issues that couldn’t be untangled without starting over.

That night, I started building what I now call a Healing Table—a dimension that can be completely rebuilt from source data at any point, “healing” whatever accumulated inconsistencies have crept in over months or years of incremental loads.

This isn’t just a technique for disaster recovery. It’s a fundamentally different approach to SCD Type 2 that separates change detection from period construction, processes entire historical ranges in a single pass, and produces dimensions that are deterministically reproducible from source data.

Here’s how it works.

Why Day-by-Day Backfills Fail

The traditional approach to backfilling historical SCD2 dimensions follows a pattern most data engineers know well: take your daily incremental process, wrap it in a loop, and execute it once for each historical date. Simple, reuses existing code, and catastrophically fragile.

The problems start immediately.

Errors compound across iterations. If your change detection logic misses an edge case on January 3rd, that incorrect record becomes the baseline for January 4th’s comparison. The error doesn’t stay contained—it propagates forward through every subsequent run. By the time you discover the issue months later, you can’t simply fix January 3rd because hundreds of downstream records depend on that incorrect state.

Performance degrades non-linearly. Each daily run needs to compare incoming records against the existing dimension. As the dimension grows through backfill, comparison costs increase. A process that takes 5 minutes for a single day’s delta might take 45 minutes per day when the target table contains three years of history. That three-week estimate becomes three months.

Source system quirks multiply. Real source systems don’t change data once per day at a predictable time. They update records multiple times, delete and recreate rows, and occasionally backdate changes. Day-by-day processing either misses these patterns entirely or handles them inconsistently across runs.

Recovery requires complete rebuild anyway. When—not if—something goes wrong, fixing it requires blowing away the dimension and starting over. But if you’re going to rebuild from scratch regardless, why not design for that from the beginning?

The Healing Tables framework embraces this reality. Instead of trying to prevent rebuilds, it makes them fast, reliable, and deterministic.

The Six-Step Framework

Healing Tables work by separating two concerns that traditional SCD2 implementations conflate: change detection (identifying when attributes changed) and period construction (building valid time slices with proper start and end dates).

Traditional approaches detect changes and construct periods simultaneously, comparing incoming records to existing dimension state. This creates tight coupling between current dimension contents and processing logic—if the dimension is wrong, future processing will also be wrong.

Healing Tables decouple these concerns through a six-step pipeline that operates entirely on source data, constructing the dimension from scratch without reference to any existing target state:

Effectivity Table Creation — Extract all change points from sources
Time Slice Generation — Build date ranges with proper valid_from/valid_to
Source Table Joining — Conform attributes from multiple sources
Hash Computation — Enable efficient change detection
Row Compression — Eliminate consecutive identical states
Validation Testing — Verify temporal integrity before loading

Let me walk through each step.

Step 1: Building the Effectivity Table

The Effectivity Table captures every moment when any tracked attribute changed for any business key. Think of it as a timeline of “something happened here” markers that we’ll later fill in with actual attribute values.

For sources with explicit timestamps, extraction looks like this:

-- Extract all change points from source
WITH source_ordered AS (
 SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,credit_limit
 ,updated_at AS change_timestamp
 ,ROW_NUMBER() OVER (
 PARTITION BY customer_id
 ORDER BY updated_at
 ) AS version_sequence
 FROM
 raw_customers
 WHERE
 updated_at IS NOT NULL
),

-- Detect actual changes using LAG comparison
change_detection AS (
 SELECT
 *
 ,LAG(customer_name) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ) AS prev_name
 ,LAG(customer_status) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ) AS prev_status
 ,LAG(credit_limit) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ) AS prev_credit_limit
 FROM
 source_ordered
)

-- Keep only first record OR records where attributes actually changed
SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,credit_limit
 ,change_timestamp
FROM
 change_detection
WHERE
 version_sequence = 1
 OR customer_name IS DISTINCT FROM prev_name
 OR customer_status IS DISTINCT FROM prev_status
 OR credit_limit IS DISTINCT FROM prev_credit_limit

The critical insight here is filtering to actual changes only. Raw source tables often contain records where nothing meaningful changed—perhaps a batch process touched every row, or a status field was updated and then immediately reverted. Including these non-changes creates unnecessary dimension versions that complicate queries and waste storage.

When working with daily snapshot sources (where you don’t have change timestamps, just daily extracts), the pattern shifts to detecting differences between consecutive snapshots:

WITH daily_ordered AS (
 SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,snapshot_date
 ,LAG(customer_name) OVER (
 PARTITION BY customer_id
 ORDER BY snapshot_date
 ) AS prev_name
 ,LAG(customer_status) OVER (
 PARTITION BY customer_id
 ORDER BY snapshot_date
 ) AS prev_status
 FROM
 daily_customer_snapshots
)

SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,snapshot_date AS effective_from
FROM
 daily_ordered
WHERE
 prev_name IS NULL -- First occurrence
 OR customer_name IS DISTINCT FROM prev_name
 OR customer_status IS DISTINCT FROM prev_status

The IS DISTINCT FROM operator handles NULL comparisons correctly—a common source of bugs in change detection logic. Standard != treats NULL as unknown, meaning NULL != 'value' returns NULL rather than TRUE. IS DISTINCT FROM treats NULL as a concrete value, giving us the comparison semantics we actually want.

Step 2: Generating Time Slices

Time Slices transform point-in-time change events into date ranges with valid_from and valid_to columns. The LEAD window function does the heavy lifting, calculating when each version expires based on when the next version begins:

WITH effectivity_data AS (
 SELECT
 customer_id, customer_name, customer_status, change_timestamp
 FROM
 effectivity_table
)

SELECT
 customer_id
 ,customer_name
 ,customer_status
 ,change_timestamp AS valid_from
 -- Calculate valid_to from next version's start date
 ,COALESCE(
 LEAD(change_timestamp) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ),
 DATE '9999-12-31' -- High date for current records
 ) AS valid_to
 -- Flag current record for easy filtering
 ,CASE
 WHEN LEAD(change_timestamp) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ) IS NULL THEN TRUE
 ELSE FALSE
 END AS is_current
FROM
 effectivity_data

A few implementation decisions deserve attention here.

High dates versus NULLs for current records. Using 9999-12-31 for current record end dates simplifies BETWEEN queries but can confuse some BI tools that try to calculate date differences. Using NULL for valid_to has clearer semantic meaning but requires IS NULL predicates everywhere. My recommendation: use the high date for the column value, but also include an explicit is_current boolean flag. This gives query authors flexibility without forcing them to remember magic date values.

Interval conventions matter. Use left-closed, right-open intervals where valid_from <= date < valid_to. This ensures every point in time maps to exactly one dimension version without ambiguity at boundaries. The alternative—inclusive on both ends—creates situations where midnight timestamps belong to two periods simultaneously.

Point-in-time lookup pattern:

SELECT d.*
FROM
 dim_customer d
WHERE
 @lookup_date >= d.valid_from
 AND @lookup_date < d.valid_to

Step 3: Joining Multiple Sources

When dimension attributes come from multiple systems, you need to conform those sources before SCD2 processing begins. The temptation is to load directly from multiple sources into a single target—don’t do this. You’ll create conflicting versions where the same timestamp has different attribute combinations depending on which source processed first.

The solution is a Unified Timeline that collects all change points from all sources, then joins each source back using as-of-date logic:

-- Step 1: Collect ALL change points from ALL sources
WITH unified_change_points AS (
 SELECT customer_id, change_timestamp FROM crm_customers
 UNION
 SELECT customer_id, change_timestamp FROM erp_customers
 UNION
 SELECT customer_id, change_timestamp FROM web_profiles
),

-- Step 2: Build time slices from unified timeline
unified_time_slices AS (
 SELECT
 customer_id
 ,change_timestamp AS valid_from
 ,COALESCE(
 LEAD(change_timestamp) OVER (
 PARTITION BY customer_id
 ORDER BY change_timestamp
 ),
 TIMESTAMP '9999-12-31 23:59:59'
 ) AS valid_to
 FROM
 unified_change_points
),

-- Step 3: Join each source back using as-of logic
final_dimension AS (
 SELECT
 ts.customer_id
 ,ts.valid_from
 ,ts.valid_to
 -- CRM attributes (authoritative for name)
 ,crm.customer_name
 -- ERP attributes (authoritative for financial data)
 ,erp.credit_limit
 ,erp.payment_terms
 -- Web attributes (authoritative for contact preferences)
 ,web.email_opt_in
 ,web.preferred_contact_method
 FROM
 unified_time_slices ts

 LEFT JOIN crm_customers crm
 ON ts.customer_id = crm.customer_id
 AND crm.change_timestamp = (
 SELECT MAX(change_timestamp)
 FROM crm_customers
 WHERE customer_id = ts.customer_id
 AND change_timestamp <= ts.valid_from
 )

 LEFT JOIN erp_customers erp
 ON ts.customer_id = erp.customer_id
 AND erp.change_timestamp = (
 SELECT MAX(change_timestamp)
 FROM erp_customers
 WHERE customer_id = ts.customer_id
 AND change_timestamp <= ts.valid_from
 )

 LEFT JOIN web_profiles web
 ON ts.customer_id = web.customer_id
 AND web.change_timestamp = (
 SELECT MAX(change_timestamp)
 FROM web_profiles
 WHERE customer_id = ts.customer_id
 AND change_timestamp <= ts.valid_from
 )
)

SELECT * FROM final_dimension

Notice the correlated subqueries for each join—we’re finding the most recent version from each source that was effective at each unified timeline point. This handles cases where one source updates frequently while another updates rarely.

Document attribute ownership explicitly. When CRM and ERP both have a customer name field, which wins? When web profile email conflicts with CRM email, which is authoritative? These decisions belong in documentation and code comments, not in tribal knowledge. Build a table:

Attribute	Primary Source	Fallback	Resolution Rule
Customer Name	CRM	ERP	CRM always wins
Credit Limit	ERP	None	ERP authoritative
Email	Web	CRM	Most recent wins

Then implement those rules explicitly in your SQL using COALESCE or CASE expressions.

Step 4: Hash Computation

Hash-based change detection replaces expensive multi-column comparisons with single-column hash comparisons. The Healing Tables framework uses a two-hash strategy: one hash for business key identification, another for attribute change detection.

key_hash identifies the business entity. Hash the natural key columns together:

-- Trino/Starburst syntax
lower(to_hex(sha256(to_utf8(
 COALESCE(CAST(source_system AS VARCHAR), '') || '|' ||
 COALESCE(CAST(customer_id AS VARCHAR), '')
)))) AS key_hash

row_hash detects when any tracked attribute changes. Hash all business attributes excluding keys and metadata:

lower(to_hex(sha256(to_utf8(
 COALESCE(customer_name, '^^NULL^^') || '|' ||
 COALESCE(customer_status, '^^NULL^^') || '|' ||
 COALESCE(CAST(credit_limit AS VARCHAR), '^^NULL^^') || '|' ||
 COALESCE(CAST(birth_date AS VARCHAR), '^^NULL^^')
)))) AS row_hash

The ^^NULL^^ placeholder distinguishes NULL values from empty strings—a critical detail that trips up many implementations. Without this, ('John', NULL, 'Active') and ('John', '', 'Active') produce identical hashes despite having semantically different data.

Why not use the built-in CONCAT_WS? Because CONCAT_WS skips NULL values entirely. The inputs ('A', NULL, 'B') and ('A', 'B') produce identical output, creating false positive matches where none exist. Always wrap columns in COALESCE before concatenation.

Column ordering best practice: concatenate in alphabetical order. This creates deterministic, reproducible hashes that survive schema evolution—adding a new column doesn’t change the hash of existing columns.

Platform-specific hash functions vary:

Platform	Function	Syntax
Snowflake	SHA2	`SHA2(expression, 256)`
BigQuery	SHA256 + TO_HEX	`TO_HEX(SHA256(expression))`
Trino/Starburst	sha256 + to_hex	`to_hex(sha256(to_utf8(expression)))`
SQL Server	HASHBYTES	`HASHBYTES('SHA2_256', expression)`

Step 5: Row Compression

Row compression eliminates consecutive duplicate states—adjacent time periods where all attributes are identical. This happens more often than you’d expect: source systems frequently touch records without changing meaningful data, or changes get reverted within the same processing window.

The technique is a classic Islands and Gaps pattern. Assign a group identifier to runs of consecutive identical rows, then collapse each group into a single time slice:

-- Step 1: Detect group boundaries
WITH boundary_detection AS (
 SELECT
 *
 ,CASE
 -- New group if: periods don't connect OR attributes changed
 WHEN LAG(valid_to) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) = valid_from
 AND row_hash = LAG(row_hash) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 )
 THEN 0 -- Continuation of previous group
 ELSE 1 -- Start of new group
 END AS is_group_start
 FROM
 dimension_with_hashes
),

-- Step 2: Assign group IDs via running sum
group_assignment AS (
 SELECT
 *
 ,SUM(is_group_start) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ROWS UNBOUNDED PRECEDING
 ) AS group_id
 FROM
 boundary_detection
)

-- Step 3: Collapse each group to single row
SELECT
 customer_id
 ,MIN(valid_from) AS valid_from
 ,MAX(valid_to) AS valid_to
 ,MAX(CASE WHEN is_current THEN 1 ELSE 0 END) = 1 AS is_current
 ,MAX(customer_name) AS customer_name -- All identical within group
 ,MAX(customer_status) AS customer_status
 ,MAX(row_hash) AS row_hash
 ,MAX(key_hash) AS key_hash
 ,COUNT(*) AS compressed_row_count -- Audit trail
FROM
 group_assignment
GROUP BY
 customer_id, group_id
ORDER BY
 customer_id, valid_from

Two conditions must both be true for rows to compress:

Temporal contiguity: Previous row’s valid_to equals current row’s valid_from
Attribute identity: row_hash values match

The contiguity check prevents incorrectly merging records that happen to have identical attributes but represent separate valid periods. If a customer had credit limit 10000 from January to March, then credit limit 5000 from March to June, then credit limit 10000 again from June to December—those are three distinct business states, not one, even though the first and third have identical attribute values.

Before compression (6 rows):

customer_id	valid_from	valid_to	credit_limit
1002	2020-01-01	2020-01-05	40000
1002	2020-01-05	2020-01-09	40000
1002	2020-01-09	2020-01-11	30000
1002	2022-01-01	2022-03-01	30000
1002	2022-03-01	2022-05-01	30000
1002	2022-05-01	2022-06-01	30000

After compression (3 rows):

customer_id	valid_from	valid_to	credit_limit
1002	2020-01-01	2020-01-09	40000
1002	2020-01-09	2020-01-11	30000
1002	2022-01-01	2022-06-01	30000

The gap between 2020-01-11 and 2022-01-01 correctly prevents merging the two “30000” groups—they represent distinct validity periods separated by time when the customer didn’t exist in source data.

Step 6: Validation Testing

Production SCD2 dimensions require comprehensive validation to ensure temporal integrity. These tests should run after every load and block pipeline completion on failure. Silent failures in dimension loading corrupt downstream fact tables in ways that are extremely difficult to diagnose later.

Test 1: Single current record per key

Each business key must have exactly one active (current) record:

-- CRITICAL: Fails if any key has multiple current records
SELECT
 customer_id
 ,COUNT(*) AS current_record_count
FROM
 dim_customer
WHERE
 is_current = TRUE
GROUP BY
 customer_id
HAVING
 COUNT(*) <> 1

-- Expected: Zero rows returned

Test 2: No overlapping date ranges

Time periods for the same key must not overlap. This is where I learned a painful lesson: the obvious LEAD-based check is insufficient.

-- INSUFFICIENT - Only checks adjacent records
WITH overlap_check AS (
 SELECT
 customer_id
 ,valid_from
 ,valid_to
 ,LEAD(valid_from) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) AS next_valid_from
 FROM
 dim_customer
)
SELECT *
FROM
 overlap_check
WHERE
 next_valid_from <= valid_to

This misses overlaps between non-adjacent records. Records at positions 1 and 3 could overlap even if record 2 doesn’t overlap with either. The comprehensive check requires a self-join:

-- COMPREHENSIVE - Checks all pairs
SELECT a.*, b.*
FROM
 dim_customer a

 INNER JOIN dim_customer b
 ON a.customer_id = b.customer_id
 AND a.dbt_scd_id != b.dbt_scd_id -- Different records
WHERE
 a.valid_from < b.valid_to
 AND a.valid_to > b.valid_from

-- Expected: Zero rows returned

Test 3: No timeline gaps

For dimensions requiring continuous history, adjacent periods must connect:

WITH gap_check AS (
 SELECT
 customer_id
 ,valid_from
 ,LAG(valid_to) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) AS prev_valid_to
 FROM
 dim_customer
)
SELECT
 customer_id
 ,prev_valid_to AS gap_start
 ,valid_from AS gap_end
FROM
 gap_check
WHERE
 prev_valid_to IS NOT NULL
 AND
 prev_valid_to <> valid_from

-- Expected: Zero rows (or documented acceptable gaps)

Test 4: Valid date ordering

Every record must have valid_from < valid_to:

SELECT *
FROM
 dim_customer
WHERE
 valid_from >= valid_to

-- Expected: Zero rows

Test 5: No consecutive duplicate versions

After compression, no adjacent rows should have identical attributes:

WITH version_check AS (
 SELECT
 customer_id
 ,valid_from
 ,row_hash
 ,LAG(row_hash) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) AS prev_row_hash
 ,LAG(valid_to) OVER (
 PARTITION BY customer_id
 ORDER BY valid_from
 ) AS prev_valid_to
 FROM
 dim_customer
)
SELECT *
FROM
 version_check
WHERE
 row_hash = prev_row_hash
 AND valid_from = prev_valid_to -- Contiguous AND identical

-- Expected: Zero rows

In dbt, implement these as reusable tests:

# schema.yml
models:
 - name: dim_customer
 tests:
 - dbt_utils.unique_combination_of_columns:
 combination_of_columns:
 - customer_id
 - valid_from
 columns:
 - name: customer_id
 tests:
 - unique:
 config:
 where: "is_current = TRUE"
 - name: valid_from
 tests:
 - not_null
 - name: valid_to
 tests:
 - not_null

The Self-Healing Property

The name “Healing Tables” comes from a property that emerges from this design: dimensions built this way can recover from accumulated data quality issues simply by reprocessing source data.

Traditional incremental SCD2 implementations are path-dependent—the current state depends on the exact sequence of historical loads. If Tuesday’s load had a bug, and Wednesday through Friday ran correctly, you can’t just re-run Tuesday. Those later loads built on Tuesday’s incorrect state.

Healing Tables are path-independent. The dimension produced depends only on source data contents, not on the history of load operations. Re-run any date range with corrected logic, and the result is identical to what you’d get from a full rebuild.

This property transforms how teams think about data quality issues:

Bug in change detection logic? Fix it, reprocess affected date range, done.
Source system correction applied retroactively? Reprocess includes those corrections automatically.
New attribute added to dimension? Reprocess rebuilds all historical versions with the new column.
Suspected data corruption? Rebuild and compare—if results differ, you’ve found your problem.

The dimension becomes reproducible in the same way good software builds are reproducible. Given the same inputs and logic, you always get the same outputs. This isn’t just convenient for debugging—it’s a fundamental quality characteristic that enables reliable data pipelines.

When to Use This Approach

Healing Tables aren’t appropriate for every situation. They work best when:

You have access to complete source history. The pattern assumes you can extract all historical changes, not just deltas. If your source system only provides “current state” with no history, you’ll need a different approach.
Source data is contiguous. The timeline construction assumes no gaps in source timestamps. If customers can appear, disappear, and reappear in source data, you’ll need additional logic to handle these discontinuities.
Rebuild time is acceptable. Processing years of history takes longer than processing a single day’s changes. For dimensions with billions of historical rows, full rebuilds might be impractical even with optimisation.
Your sources don’t have hard deletes. The pattern assumes deletions are logical (flagged) rather than physical. If source systems physically remove rows, you’ll lose that history.

Traditional incremental approaches remain appropriate for:

Very high-volume dimensions where full rebuilds are impractical
Real-time or near-real-time requirements
Situations where source history is unavailable
Simple dimensions with stable, well-understood source systems

The choice isn’t binary. Many teams use Healing Tables for periodic “refresh” operations while running standard incremental processes daily. If the daily process accumulates issues, the weekly or monthly heal repairs them.

Wrapping Up

That 2 AM debugging session taught me something important: the traditional approach to SCD2—incremental changes applied to existing dimension state—creates tight coupling between historical load operations and current data quality. Every bug, every edge case, every source system quirk gets baked into the dimension permanently.

Healing Tables break that coupling. By extracting change points from source data, constructing time slices independently, compressing duplicate states, and validating results before loading, you create dimensions that are deterministically reproducible from source data.

The six-step framework isn’t complicated, but it requires discipline:

Build the Effectivity Table with actual changes only
Generate Time Slices with proper valid_from/valid_to boundaries
Join multiple sources using unified timeline logic
Compute hashes for efficient change detection
Compress consecutive identical states
Validate temporal integrity before loading

Implement those steps correctly, and you’ll have dimensions that can heal themselves from whatever accumulated issues have crept in over months or years of operation.

The next time someone asks you to backfill three years of history by running the daily process 1,095 times, you’ll have a better answer.

Want the full implementation? The Healing Tables SCD2 Kit packages this six-step framework as production SQL for Snowflake, BigQuery, Trino and SQL Server, plus dbt macros and the temporal-integrity tests above, ready to drop into your project. If you’d rather have an agent apply it, the Ghost Skills library includes a data-modelling skill that follows the same method.

That Tuesday Morning When I Finally Fixed Our Ten-Minute Queries

Chris Hillman — Fri, 16 Jan 2026 09:00:00 +1100

The Ten-Minute Query

I’m sitting at my laptop on a Tuesday morning, waiting. The progress bar on my screen says ‘Query running… 4 minutes, 37 seconds.’ I lean back in my chair and let out this long sigh that probably says more than I intended.

My manager walks past my desk. She glances at my screen, and I can see that look—the one that says she already knows what I’m about to tell her. I didn’t need to explain.

“Still pulling that customer report?”

“Yeah.”

She shakes her head. “We need to fix this.”

It was an ad-hoc piece of work that we’d been running for months. But I just nod.

Here’s what was happening: every time someone needed to know ‘how many orders did customer X place?’ or ‘which products sold best last month?’, we’d query this massive table. Three million rows. Every single customer’s name, address, email—repeated on every order row. The system was scanning through all that redundant data every single time.

And then, two weeks ago, it got worse. Someone ran a data import script that had a bug. It replaced half our customer names with NULL. By the time we caught it, twelve reports had already gone out with blank customer names. I had to send apology emails. My team and I had to stay until midnight fixing it.

I thought, there has to be a better way to do this. People have been building data systems for decades—someone must have figured this out.

And they have. His name is Ralph Kimball, and his dimensional modelling methodology has been the gold standard for data warehouses since the 1990s.

The Problem I Couldn’t Unsee

Let me show you what our data looked like.

order_id | date | customer_name | address | email | product | amount
---------|------------|---------------|--------------|--------------|---------|-------
1 | 2024-11-15 | John Smith | 123 Main St | john@... | Pizza | 25
2 | 2024-11-15 | John Smith | 123 Main St | john@... | Soda | 3
3 | 2024-11-15 | Jane Doe | 456 Oak Ave | jane@... | Pizza | 25
4 | 2024-11-16 | John Smith | 123 Main St | john@... | Burger | 12

See what’s happening? John Smith’s address and email are stored three times. Multiply that by thousands of customers and millions of orders, and you’ve got:

Every query scanning way more data than necessary
Storage costs that make your CFO unhappy
One bug corrupting your entire history
Query times that make you question your career choices

Every single query was like asking someone to find your keys by searching every room in your house, including the ones you never use. Sure, they’ll find the keys eventually. But there’s a better way.

The Kitchen That Changed Everything

Let me paint you a hypothetical scenario. Imagine it’s a Saturday, and you’re standing in your kitchen after just moving house. You can’t find anything. Everything’s still in boxes.

You’re rummaging through the third box, looking for a spatula. You’re pulling out random stuff—a whisk, some Tupperware lids, a pizza cutter, yesterday’s takeout containers, a bag of flour. Everything mixed together. This is chaos.

Now imagine a friend comes over. They take one look at your kitchen and say, “What is this?”

“It’s… I haven’t unpacked yet.”

“You’ve been here a week.” They walk over to the mess. “Here’s what you should do. The stuff that rarely changes—your plates, your pots, your cutting boards—those go in the cabinets. Organised, labelled, easy to find. The stuff that comes and goes constantly—the groceries, the leftovers, what you’re cooking today—that goes in the fridge and on the counter. Don’t mix the permanent stuff with the temporary stuff.”

Two hours later, the kitchen makes sense. You can find anything in seconds. And here’s the wild part—it actually takes up less space because you’re not duplicating storage for the same items.

That’s dimensional modelling.

When your data is like those boxes, everything’s mixed together. Customer information sitting right next to order information, repeated over and over. Every time you need something, you’re digging through all the boxes.

Dimensional modelling says: let’s organise this. Put stable descriptive data—things that rarely change—in one place. We call these dimension tables. Put the events that happen constantly—the transactions, the updates, the measurements—in another place. We call these fact tables. Link them with keys.

Here’s what that transformation looks like:

BEFORE (Messy Boxes - everything mixed together):

order_id | customer_name | address | product | amount
1 | John Smith | 123 Main St | Pizza | 25
2 | John Smith | 123 Main St | Soda | 3

↑ John’s details repeated every single order!

AFTER (Organised Kitchen): DIM_CUSTOMER (Cabinets - stable stuff, stored ONCE):

customer_key | customer_id | name | address | email
1001 | C-123 | John Smith | 123 Main St | john@...
1002 | C-456 | Jane Doe | 456 Oak Ave | jane@...

DIM_DATE (Calendar on the wall - one row per day):

date_key | full_date | day_name | month | quarter | year | is_weekend
20241115 | 2024-11-15 | Friday | Nov | Q4 | 2024 | N
20241116 | 2024-11-16 | Saturday | Nov | Q4 | 2024 | Y

FACT_ORDERS (Fridge/counter - events that happen constantly):

order_key | date_key | customer_key | product_key | amount
1 | 20241115 | 1001 | 5 | 25
2 | 20241115 | 1001 | 8 | 3

Customer details stored one time. Not a thousand times. When you need to answer ‘how much did we sell?’, you scan the compact fact table. When you need ‘which customers bought?’, you JOIN to get the descriptive details.

Notice something important in that structure? The dimension tables have two types of keys:

customer_key (1001, 1002) — This is a surrogate key. A meaningless integer we generate ourselves.
customer_id (C-123, C-456) — This is the natural key. The ID from the source system.

Why both? Because source systems are chaotic. They recycle IDs. They change formats. They get merged in acquisitions. The surrogate key is our key—stable, predictable, and under our control. Every dimension table gets a surrogate key as its primary key.

And that date dimension? It’s “the most important dimension.” Every fact table should have one. It lets you slice data by day of week, month, quarter, fiscal period, holidays—without calculating these on every query.

The core principle: things that rarely change go in dedicated storage, organised and easy to find. Things that happen constantly go in their own space, linked back by keys. Same rule applies to data:

Dimensions = Stable descriptive stuff (customers, products, locations, dates)
Facts = Events that happen constantly (orders, clicks, updates, measurements)

The Conference Room I’ll Never Forget

I’m sitting in a conference room. Fluorescent lights buzzing. That weird hum they make when it’s too quiet—I’m wondering why we haven’t changed to LEDs. My manager is standing at the whiteboard, and she’s pissed.

“Someone,” she says, not looking at anyone in particular, “pushed bad data to production last night.”

I feel my stomach drop. The data pipeline my team had been working on.

“The revenue report that went to the board this morning showed every transaction as zero dollars. Zero. The board emailed me asking if the company made any money last quarter.”

Oh god. Oh no. I sank down in my chair a little.

Turns out it wasn’t actually our release—it was someone else’s. But in that moment, sitting there, I realised something crucial: we had no safety net. Data went straight from transformation to production. No checks. No validation. If you made a mistake, everyone saw it immediately.

That conference room moment stuck with me. I started researching how other teams prevented this kind of disaster. That’s when I learned about Write-Audit-Publish. And honestly? It’s probably saved me from that same sick feeling countless times since.

Here’s the idea:

WRITE — You finish your transformation. Instead of putting the data straight into production, you write it to a staging branch. Think of it like a draft folder for emails. It’s there, but it’s not official yet.

AUDIT — This is the crucial part. You run every check you can think of:

Did I lose any rows?
Are there any NULL values where there shouldn’t be?
Do all the foreign keys match up?
Does the math make sense?
Do surrogate keys have duplicates?

It’s like re-reading an important email before you hit send. You check for typos, you verify the recipients, you make sure the attachment is right.

Important: In the future when you find incidents, bugs, or problems—add them to this AUDIT layer. Ensure you don’t make the same mistakes twice.

PUBLISH — Only if everything passes, you merge the branch to production. If anything fails, you stop. You fix it. You try again. The bad data never reaches production. Never reaches that board meeting.

The traditional way to implement this involves copying data between folders or tables. But there’s a better approach using Apache Iceberg’s branching feature. Instead of physically copying data, you create a branch—like in Git. Your audit runs against the branch. If it passes, you merge the branch to main. If it fails, you drop the branch. No data was ever at risk.

Critical design decision: WAP should happen at each table level, not just at the end of the pipeline. Each dimension job does its own Write-Audit-Publish cycle. Each fact job does its own. This way:

Failures are isolated to the specific table that failed
You catch issues early, not at the very end
Each job is self-contained and independently testable
Downstream jobs read from published (main) tables, not staging branches

Let me show you what this looks like when an audit catches something:

AUDIT PHASE (dim_system):

Row count check.......................... PASS
 Source: 1,198 rows
 Target: 1,198 rows

Primary key uniqueness................... FAIL
 Found 13 duplicate system_keys in dim_system!

AUDIT FAILED — BRANCH NOT MERGED

Data remains in staging branch.
Production table unchanged.
Fix the duplicates and re-run the job.

Look at that. The audit caught duplicate surrogate keys before they reached production. Nobody sees this except you. You fix the problem. You run it again. Then it passes, and the branch merges to main.

This is Write-Audit-Publish. It’s the difference between sending an email with a typo versus catching it first. Between serving undercooked food versus checking with a thermometer. Between publishing bad data versus validating before it matters.

That conference room moment? Doesn’t happen with this pattern.

Building This Thing for Real

Alright, enough theory. Let me show you how to actually build this in AWS.

We’re going to use real (game) space exploration data—systems, planets, updates. It’s way more interesting than order data, and the patterns work exactly the same.

In this section:

Architecture Overview
Step 1: Setting Up S3 Buckets
Step 2: Creating the Iceberg Tables
The Glue Jobs (with Per-Table WAP)
Step Functions Workflow
Dependency Triggers with S3 Events

Architecture Overview

Here’s our architecture:

 +------------------------------------------+
 | AWS Step Functions Workflow |
 +------------------------------------------+
 |
 +----------------------------+----------------------------+
 | | |
 v v v
 +------------+ +------------+ +------------+
 | Glue Job: | | Glue Job: | | Glue Job: |
 | dim_date | | dim_system | | dim_planet |
 | (WAP) | | (WAP) | | (WAP) |
 +------------+ +------------+ +------------+
 | | |
 +----------------------------+----------------------------+
 |
 (Wait for all dimensions)
 |
 v
 +------------------------+
 | Glue Job: |
 | fact_system_updates |
 | (WAP + Incremental) |
 +------------------------+

Data Flow:

Landing Layer	Staging Layer (PSA)	Semantic Layer
Raw 1:1 from source	Iceberg + data_date partitioning	Production dimensional model
`systems/`	`stg_systems/`	`dim_date`
`planets/`	`stg_planets/`	`dim_system`
`updates/`	`stg_updates/`	`dim_planet`
		`fact_system_updates`

Let me explain the three layers:

Landing Layer — Raw data, 1:1 from source systems. No transformations except what’s needed to land it. Contains only the latest fetch. If sourcing fails, this layer fails. Get the data close to you as quickly as possible (into your ecosystem). Typically all the files will be string or varchar to mitigate data type issues from a source system.

Staging Layer (Persistent Staging Area) — This is where we convert data types and prepare for dimensional modelling. Critically, we partition by data_date—the date the source data was fetched. This gives us:

Historical reprocessing capability (reload any past date)
Debugging (what did the data look like on January 15th?)
Dependency triggers (when a partition is written, trigger downstream jobs)

Semantic Layer — Production-ready dimensional model. This is where business users query. Tables are partitioned appropriately for query performance.

Also notice the job structure: each job does its own WAP cycle. Dimensions build in parallel, each publishing to main when their audits pass. Fact tables wait for all dimensions to publish, then do their own WAP cycle. This ensures:

Fact tables read from published dimensions (main branch), not staging branches
Foreign key lookups use production-ready surrogate keys
Each job is independently testable

Step 1: Setting Up S3 Buckets

I’m in the S3 console. Click “Create bucket”. I’m calling mine space-data-pipeline-demo.

Important settings:

Region: ap-southeast-2 (Sydney—pick one close to you)
Block all public access: Yes
Versioning: Enabled
Default encryption: Enabled

Inside this bucket:

space-data-pipeline-demo/
├── landing/ ← Raw 1:1 from source (latest only)
│ ├── systems/
│ ├── planets/
│ └── updates/
├── staging/ ← PSA with data_date partitioning
│ ├── stg_systems/
│ ├── stg_planets/
│ └── stg_updates/
└── semantic/ ← Production dimensional model
 ├── dim_date/
 ├── dim_system/
 ├── dim_planet/
 └── fact_system_updates/

Step 2: Creating the Iceberg Tables

Here’s the DDL for our dimensional model. Notice the partitioning strategies and SCD Type 2 columns.

Click to expand: dim_date DDL

This is a reference dimension—typically loaded once with all dates, then rarely updated. Partitioned by year for query performance.

CREATE TABLE space_exploration.dim_date (
 date_key INT,
 full_date DATE,
 day_of_week STRING,
 day_of_month INT,
 day_of_year INT,
 week_of_year INT,
 month_number INT,
 month_name STRING,
 quarter INT,
 year INT,
 is_weekend STRING,
 is_holiday STRING,
 fiscal_quarter INT,
 fiscal_year INT,
 -- Audit columns
 load_timestamp TIMESTAMP,
 source_system STRING
)
PARTITIONED BY (year)
LOCATION 's3://space-data-pipeline-demo/semantic/dim_date/'
TBLPROPERTIES (
 'table_type' = 'ICEBERG',
 'format' = 'parquet'
);

Click to expand: dim_system DDL (SCD Type 2)

Tracks history with effective_date/expiration_date. Partitioned by effective_year for efficient history queries.

CREATE TABLE space_exploration.dim_system (
 system_key INT, -- Surrogate key (generated)
 system_id STRING, -- Natural key (from source)
 system_name STRING,
 sector STRING,
 main_star_type STRING,
 coordinates_x DOUBLE,
 coordinates_y DOUBLE,
 coordinates_z DOUBLE,
 discovery_date DATE,
 exploration_status STRING,
 -- SCD Type 2 columns
 effective_date DATE,
 expiration_date DATE,
 current_flag STRING, -- 'Y' or 'N'
 -- Audit columns
 load_timestamp TIMESTAMP,
 source_system STRING,
 -- Partition column
 effective_year INT
)
PARTITIONED BY (effective_year)
LOCATION 's3://space-data-pipeline-demo/semantic/dim_system/'
TBLPROPERTIES (
 'table_type' = 'ICEBERG',
 'format' = 'parquet'
);

Click to expand: dim_planet DDL (SCD Type 2)

Foreign key to dim_system uses the surrogate key. Partitioned by effective_year.

CREATE TABLE space_exploration.dim_planet (
 planet_key INT, -- Surrogate key
 planet_id STRING, -- Natural key
 system_key INT, -- FK to dim_system (surrogate!)
 planet_name STRING,
 planet_type STRING,
 orbital_period DOUBLE,
 has_atmosphere STRING,
 habitability_score DOUBLE,
 -- SCD Type 2 columns
 effective_date DATE,
 expiration_date DATE,
 current_flag STRING,
 -- Audit columns
 load_timestamp TIMESTAMP,
 source_system STRING,
 -- Partition column
 effective_year INT
)
PARTITIONED BY (effective_year)
LOCATION 's3://space-data-pipeline-demo/semantic/dim_planet/'
TBLPROPERTIES (
 'table_type' = 'ICEBERG',
 'format' = 'parquet'
);

Click to expand: fact_system_updates DDL

Incremental fact table partitioned by date_year. Loaded with delete-then-insert pattern.

CREATE TABLE space_exploration.fact_system_updates (
 update_key BIGINT, -- Surrogate key
 date_key INT, -- FK to dim_date
 system_key INT, -- FK to dim_system (current surrogate!)
 update_timestamp TIMESTAMP, -- Original precision preserved
 status STRING,
 ships_present INT, -- Additive fact
 resource_level DOUBLE, -- Additive fact
 -- Audit columns
 load_timestamp TIMESTAMP,
 source_system STRING,
 data_date DATE, -- Source data date (for reprocessing)
 -- Partition column
 date_year INT
)
PARTITIONED BY (date_year)
LOCATION 's3://space-data-pipeline-demo/semantic/fact_system_updates/'
TBLPROPERTIES (
 'table_type' = 'ICEBERG',
 'format' = 'parquet'
);

Click to expand: Staging Tables DDL (Persistent Staging Area)

Partitioned by data_date for historical reprocessing.

CREATE TABLE space_exploration.stg_systems (
 system_id STRING,
 system_name STRING,
 sector STRING,
 main_star_type STRING,
 coordinates_x DOUBLE,
 coordinates_y DOUBLE,
 coordinates_z DOUBLE,
 discovery_date DATE,
 exploration_status STRING,
 -- PSA columns
 data_date DATE,
 load_timestamp TIMESTAMP
)
PARTITIONED BY (data_date)
LOCATION 's3://space-data-pipeline-demo/staging/stg_systems/'
TBLPROPERTIES ('table_type' = 'ICEBERG', 'format' = 'parquet');

CREATE TABLE space_exploration.stg_planets (
 planet_id STRING,
 system_id STRING,
 planet_name STRING,
 planet_type STRING,
 orbital_period DOUBLE,
 has_atmosphere STRING,
 habitability_score DOUBLE,
 -- PSA columns
 data_date DATE,
 load_timestamp TIMESTAMP
)
PARTITIONED BY (data_date)
LOCATION 's3://space-data-pipeline-demo/staging/stg_planets/'
TBLPROPERTIES ('table_type' = 'ICEBERG', 'format' = 'parquet');

CREATE TABLE space_exploration.stg_updates (
 system_id STRING,
 update_timestamp TIMESTAMP,
 status STRING,
 ships_present INT,
 resource_level DOUBLE,
 -- PSA columns
 data_date DATE,
 load_timestamp TIMESTAMP
)
PARTITIONED BY (data_date)
LOCATION 's3://space-data-pipeline-demo/staging/stg_updates/'
TBLPROPERTIES ('table_type' = 'ICEBERG', 'format' = 'parquet');

Key design decisions:

dim_date: Partitioned by year. This is a reference dimension—typically loaded once with all dates from 1900-01-01 to 9999-12-31, then rarely updated.
dim_system / dim_planet: SCD Type 2 with effective_date, expiration_date, current_flag. Partitioned by effective_year for efficient history queries. Updated daily.
fact_system_updates: Partitioned by date_year. Loaded incrementally using delete-then-insert pattern.
Staging tables (PSA): Partitioned by data_date. This lets us reprocess any historical date and provides dependency triggers.

The Heart of the System: Orchestrated Glue Jobs with Per-Table WAP

Each job does its own Write-Audit-Publish cycle. Let me walk you through each one.

Job 1: Build Date Dimension (Reference Dimension)

The date dimension is special—it’s a reference dimension. We generate it ourselves with all calendar attributes, typically as a one-time load. You’d only rerun this if you need to extend the date range or add new attributes (like new holidays).

Click to expand: build_dim_date Glue Job

This job generates a complete date dimension from 2000-01-01 to 2050-12-31 (for demo purposes—production would use a wider range). It creates date attributes like day_of_week, quarter, fiscal_year, and is_weekend, then runs WAP to validate before publishing.

# Glue Job: build_dim_date
# This is typically a ONE-TIME RUN to populate all dates
# Rerun only when extending date range or adding attributes

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *
from pyspark.sql.types import *
from datetime import datetime, timedelta

# Initialize
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

print("="*60)
print("Building Date Dimension (Reference Dimension)")
print("This is typically a ONE-TIME load")
print("="*60)

# Generate comprehensive date range
# Standard practice: 1900-01-01 to 9999-12-31
start_date = datetime(1900, 1, 1)
end_date = datetime(9999, 12, 31)

# For demo purposes, let's use a more reasonable range
# In production, you'd use the full range
start_date = datetime(2000, 1, 1)
end_date = datetime(2050, 12, 31)

print(f"Generating dates from {start_date.date()} to {end_date.date()}")

date_list = []
current = start_date
while current <= end_date:
 date_list.append((current,))
 current += timedelta(days=1)

date_df = spark.createDataFrame(date_list, ["full_date"])

# Build dimension with rich attributes
dim_date = date_df.select(
 # Surrogate key: YYYYMMDD format
 (year("full_date") * 10000 +
 month("full_date") * 100 +
 dayofmonth("full_date")).cast("int").alias("date_key"),

 col("full_date"),
 date_format("full_date", "EEEE").alias("day_of_week"),
 dayofmonth("full_date").alias("day_of_month"),
 dayofyear("full_date").alias("day_of_year"),
 weekofyear("full_date").alias("week_of_year"),
 month("full_date").alias("month_number"),
 date_format("full_date", "MMMM").alias("month_name"),
 quarter("full_date").alias("quarter"),
 year("full_date").alias("year"),

 # Business-friendly flags
 when(dayofweek("full_date").isin([1, 7]), "Y")
 .otherwise("N").alias("is_weekend"),
 lit("N").alias("is_holiday"), # Populate from holiday calendar

 # Fiscal calendar (assuming fiscal year = calendar year)
 quarter("full_date").alias("fiscal_quarter"),
 year("full_date").alias("fiscal_year"),

 # Audit columns
 current_timestamp().alias("load_timestamp"),
 lit("GENERATED").alias("source_system")
)

row_count = dim_date.count()
print(f"Generated {row_count:,} date rows")

# ============================================
# WRITE PHASE: Write to staging branch
# ============================================
print("\n--- WRITE PHASE ---")
print("Writing to staging branch...")

spark.sql("ALTER TABLE space_exploration.dim_date CREATE BRANCH IF NOT EXISTS staging")

dim_date.writeTo("space_exploration.dim_date") \
 .option("branch", "staging") \
 .overwritePartitions()

print("Written to staging branch")

# ============================================
# AUDIT PHASE: Validate before publishing
# ============================================
print("\n--- AUDIT PHASE ---")

staging_df = spark.read \
 .option("branch", "staging") \
 .table("space_exploration.dim_date")

audit_failures = []

# Audit 1: Row count
staged_count = staging_df.count()
print(f"Audit 1 - Row count: {staged_count:,}")
if staged_count == 0:
 audit_failures.append("FAIL: dim_date is empty")

# Audit 2: Primary key uniqueness
unique_keys = staging_df.select("date_key").distinct().count()
if staged_count != unique_keys:
 duplicates = staged_count - unique_keys
 audit_failures.append(f"FAIL: {duplicates} duplicate date_keys")
 print(f"Audit 2 - PK uniqueness: FAIL ({duplicates} duplicates)")
else:
 print(f"Audit 2 - PK uniqueness: PASS (all {unique_keys:,} unique)")

# Audit 3: No NULLs in critical fields
null_count = staging_df.filter(
 col("date_key").isNull() | col("full_date").isNull()
).count()
if null_count > 0:
 audit_failures.append(f"FAIL: {null_count} NULL values in critical fields")
 print(f"Audit 3 - NULL check: FAIL ({null_count} NULLs)")
else:
 print("Audit 3 - NULL check: PASS")

# Audit 4: Date range coverage
date_range = staging_df.agg(
 min("full_date").alias("min_date"),
 max("full_date").alias("max_date")
).collect()[0]
print(f"Audit 4 - Date range: {date_range['min_date']} to {date_range['max_date']}")

# ============================================
# PUBLISH PHASE: Merge to main if audits pass
# ============================================
print("\n--- PUBLISH PHASE ---")

if len(audit_failures) > 0:
 print("\nAUDIT FAILED:")
 for failure in audit_failures:
 print(f" {failure}")
 print("\nBranch NOT merged. Production unchanged.")
 spark.sql("ALTER TABLE space_exploration.dim_date DROP BRANCH staging")
 raise Exception(f"Audit failed with {len(audit_failures)} errors")
else:
 print("\nALL AUDITS PASSED")
 print("Merging staging branch to main...")

 # Fast-forward main to staging
 spark.sql("""
 CALL system.fast_forward(
 table => 'space_exploration.dim_date',
 branch => 'main',
 to => 'staging'
 )
 """)

 # Clean up staging branch
 spark.sql("ALTER TABLE space_exploration.dim_date DROP BRANCH staging")

 print(f"PUBLISHED: dim_date ({staged_count:,} rows)")

print("\n" + "="*60)
print("dim_date job complete")
print("="*60)

job.commit()

Job 2: Build System Dimension (SCD Type 2)

This dimension tracks history. When a system’s attributes change (like exploration_status), we expire the old row and create a new one.

Click to expand: build_dim_system Glue Job

This job implements SCD Type 2 change detection. It compares incoming data against the current dimension, identifies NEW/CHANGED/UNCHANGED records, expires old versions, and creates new versions with updated surrogate keys.

# Glue Job: build_dim_system
# SCD Type 2 implementation with effective/expiration dates

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *
from pyspark.sql.window import Window

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'DATA_DATE'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

data_date = args['DATA_DATE'] # e.g., '2026-01-17'
print("="*60)
print(f"Building System Dimension (SCD Type 2)")
print(f"Data Date: {data_date}")
print("="*60)

# ============================================
# LOAD: Read from Persistent Staging Area
# ============================================
print("\n--- LOAD PHASE ---")

# Read today's staged data (already landed in PSA)
source_df = spark.read.table("space_exploration.stg_systems") \
 .filter(col("data_date") == data_date)

source_count = source_df.count()
print(f"Source rows from PSA (data_date={data_date}): {source_count:,}")

if source_count == 0:
 print("No data for this date. Exiting.")
 job.commit()
 sys.exit(0)

# Read current production dimension (for SCD2 comparison)
try:
 current_dim = spark.read.table("space_exploration.dim_system") \
 .filter(col("current_flag") == "Y")
 current_count = current_dim.count()
 print(f"Current dimension rows (current_flag='Y'): {current_count:,}")
 has_existing_data = current_count > 0
except:
 print("No existing dimension data (first load)")
 has_existing_data = False
 current_dim = None

# ============================================
# TRANSFORM: SCD Type 2 Logic
# ============================================
print("\n--- TRANSFORM PHASE ---")

if not has_existing_data:
 # First load: all rows are new
 print("First load - all rows are INSERT")

 window_spec = Window.orderBy("system_id")

 new_dim = source_df.select(
 row_number().over(window_spec).alias("system_key"),
 col("system_id"),
 col("system_name"),
 col("sector"),
 col("main_star_type"),
 col("coordinates_x"),
 col("coordinates_y"),
 col("coordinates_z"),
 col("discovery_date"),
 col("exploration_status"),
 lit(data_date).cast("date").alias("effective_date"),
 lit("9999-12-31").cast("date").alias("expiration_date"),
 lit("Y").alias("current_flag"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 year(lit(data_date)).alias("effective_year")
 ).distinct()

else:
 # Incremental load: detect changes for SCD2
 print("Incremental load - detecting changes for SCD2")

 # Define which columns to track for changes
 tracked_columns = [
 "system_name", "sector", "main_star_type",
 "coordinates_x", "coordinates_y", "coordinates_z",
 "exploration_status"
 ]

 # Join source with current to detect changes
 comparison = source_df.alias("src").join(
 current_dim.alias("curr"),
 col("src.system_id") == col("curr.system_id"),
 "full_outer"
 )

 # Identify: NEW (no match in current), CHANGED (values differ), UNCHANGED
 change_detection = comparison.withColumn(
 "change_type",
 when(col("curr.system_id").isNull(), "NEW")
 .when(col("src.system_id").isNull(), "DELETED")
 .when(
 (col("src.system_name") != col("curr.system_name")) |
 (col("src.sector") != col("curr.sector")) |
 (col("src.main_star_type") != col("curr.main_star_type")) |
 (col("src.exploration_status") != col("curr.exploration_status")) |
 (abs(col("src.coordinates_x") - col("curr.coordinates_x")) > 0.001) |
 (abs(col("src.coordinates_y") - col("curr.coordinates_y")) > 0.001) |
 (abs(col("src.coordinates_z") - col("curr.coordinates_z")) > 0.001),
 "CHANGED"
 )
 .otherwise("UNCHANGED")
 )

 # Count changes
 change_counts = change_detection.groupBy("change_type").count().collect()
 for row in change_counts:
 print(f" {row['change_type']}: {row['count']:,}")

 # Get max surrogate key for new key generation
 max_key = current_dim.agg(max("system_key")).collect()[0][0] or 0

 # Build new dimension rows for NEW and CHANGED records
 new_records = change_detection.filter(col("change_type").isin(["NEW", "CHANGED"]))

 window_spec = Window.orderBy("src.system_id")

 new_rows = new_records.select(
 (row_number().over(window_spec) + max_key).alias("system_key"),
 col("src.system_id").alias("system_id"),
 col("src.system_name").alias("system_name"),
 col("src.sector").alias("sector"),
 col("src.main_star_type").alias("main_star_type"),
 col("src.coordinates_x").alias("coordinates_x"),
 col("src.coordinates_y").alias("coordinates_y"),
 col("src.coordinates_z").alias("coordinates_z"),
 col("src.discovery_date").alias("discovery_date"),
 col("src.exploration_status").alias("exploration_status"),
 lit(data_date).cast("date").alias("effective_date"),
 lit("9999-12-31").cast("date").alias("expiration_date"),
 lit("Y").alias("current_flag"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 year(lit(data_date)).alias("effective_year")
 )

 # Build expired rows (for CHANGED records, expire the old version)
 changed_ids = change_detection.filter(col("change_type") == "CHANGED") \
 .select(col("curr.system_id").alias("system_id"))

 expired_rows = current_dim.join(changed_ids, "system_id") \
 .withColumn("expiration_date", lit(data_date).cast("date") - 1) \
 .withColumn("current_flag", lit("N"))

 # Unchanged rows stay as-is
 unchanged_rows = current_dim.join(
 change_detection.filter(col("change_type") == "UNCHANGED")
 .select(col("curr.system_id").alias("system_id")),
 "system_id"
 )

 # Historical rows (already expired) - read from full table
 historical_rows = spark.read.table("space_exploration.dim_system") \
 .filter(col("current_flag") == "N")

 # Combine all
 new_dim = new_rows \
 .unionByName(expired_rows, allowMissingColumns=True) \
 .unionByName(unchanged_rows, allowMissingColumns=True) \
 .unionByName(historical_rows, allowMissingColumns=True)

transform_count = new_dim.count()
print(f"Total dimension rows after SCD2: {transform_count:,}")

# ============================================
# WRITE PHASE: Write to staging branch
# ============================================
print("\n--- WRITE PHASE ---")

spark.sql("ALTER TABLE space_exploration.dim_system CREATE BRANCH IF NOT EXISTS staging")

new_dim.writeTo("space_exploration.dim_system") \
 .option("branch", "staging") \
 .overwritePartitions()

print("Written to staging branch")

# ============================================
# AUDIT PHASE
# ============================================
print("\n--- AUDIT PHASE ---")

staging_df = spark.read \
 .option("branch", "staging") \
 .table("space_exploration.dim_system")

audit_failures = []

# Audit 1: Row count
staged_count = staging_df.count()
print(f"Audit 1 - Row count: {staged_count:,}")
if staged_count == 0:
 audit_failures.append("FAIL: dim_system is empty")

# Audit 2: Surrogate key uniqueness
unique_keys = staging_df.select("system_key").distinct().count()
if staged_count != unique_keys:
 duplicates = staged_count - unique_keys
 audit_failures.append(f"FAIL: {duplicates} duplicate system_keys")
 print(f"Audit 2 - PK uniqueness: FAIL ({duplicates} duplicates)")
else:
 print(f"Audit 2 - PK uniqueness: PASS")

# Audit 3: Only one current row per natural key
current_rows = staging_df.filter(col("current_flag") == "Y")
current_count = current_rows.count()
unique_natural_keys = current_rows.select("system_id").distinct().count()
if current_count != unique_natural_keys:
 audit_failures.append(f"FAIL: Multiple current rows for same system_id")
 print(f"Audit 3 - Single current: FAIL")
else:
 print(f"Audit 3 - Single current: PASS ({current_count:,} current rows)")

# Audit 4: No NULLs in critical fields
null_count = staging_df.filter(
 col("system_key").isNull() |
 col("system_id").isNull() |
 col("effective_date").isNull() |
 col("current_flag").isNull()
).count()
if null_count > 0:
 audit_failures.append(f"FAIL: {null_count} NULL values in critical fields")
 print(f"Audit 4 - NULL check: FAIL")
else:
 print("Audit 4 - NULL check: PASS")

# Audit 5: Effective/expiration date logic
invalid_dates = staging_df.filter(
 col("effective_date") > col("expiration_date")
).count()
if invalid_dates > 0:
 audit_failures.append(f"FAIL: {invalid_dates} rows with effective > expiration")
 print(f"Audit 5 - Date logic: FAIL")
else:
 print("Audit 5 - Date logic: PASS")

# ============================================
# PUBLISH PHASE
# ============================================
print("\n--- PUBLISH PHASE ---")

if len(audit_failures) > 0:
 print("\nAUDIT FAILED:")
 for failure in audit_failures:
 print(f" {failure}")
 print("\nBranch NOT merged. Production unchanged.")
 spark.sql("ALTER TABLE space_exploration.dim_system DROP BRANCH staging")
 raise Exception(f"Audit failed with {len(audit_failures)} errors")
else:
 print("\nALL AUDITS PASSED")
 print("Merging staging branch to main...")

 spark.sql("""
 CALL system.fast_forward(
 table => 'space_exploration.dim_system',
 branch => 'main',
 to => 'staging'
 )
 """)

 spark.sql("ALTER TABLE space_exploration.dim_system DROP BRANCH staging")

 print(f"PUBLISHED: dim_system ({staged_count:,} rows)")

job.commit()

Job 3: Build Planet Dimension (SCD Type 2)

Similar pattern to dimsystem, but with a foreign key lookup to the _published dim_system.

Click to expand: build_dim_planet Glue Job

This job demonstrates FK lookup to a published dimension. It reads from the main branch of dim_system (not staging), ensuring it uses production-ready surrogate keys. The FK integrity audit validates this relationship.

# Glue Job: build_dim_planet
# SCD Type 2 with FK lookup to published dim_system

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *
from pyspark.sql.window import Window

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'DATA_DATE'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

data_date = args['DATA_DATE']
print("="*60)
print(f"Building Planet Dimension (SCD Type 2)")
print(f"Data Date: {data_date}")
print("="*60)

# ============================================
# LOAD PHASE
# ============================================
print("\n--- LOAD PHASE ---")

source_df = spark.read.table("space_exploration.stg_planets") \
 .filter(col("data_date") == data_date)

source_count = source_df.count()
print(f"Source rows: {source_count:,}")

if source_count == 0:
 print("No data. Exiting.")
 job.commit()
 sys.exit(0)

# CRITICAL: Read from PUBLISHED dim_system (main branch, not staging)
# This is why dim_planet must run AFTER dim_system publishes
dim_system = spark.read.table("space_exploration.dim_system") \
 .filter(col("current_flag") == "Y") \
 .select("system_key", "system_id")

print(f"dim_system lookup rows: {dim_system.count():,}")

# ============================================
# TRANSFORM PHASE
# ============================================
print("\n--- TRANSFORM PHASE ---")

# Check for existing data
try:
 current_dim = spark.read.table("space_exploration.dim_planet") \
 .filter(col("current_flag") == "Y")
 has_existing = current_dim.count() > 0
 max_key = current_dim.agg(max("planet_key")).collect()[0][0] or 0
except:
 has_existing = False
 max_key = 0

# Join source with dim_system to get surrogate key
source_with_sk = source_df.join(
 dim_system,
 source_df.system_id == dim_system.system_id,
 "left"
).select(
 source_df["*"],
 dim_system.system_key
)

# Track orphaned planets (no matching system)
orphaned = source_with_sk.filter(col("system_key").isNull()).count()
if orphaned > 0:
 print(f"WARNING: {orphaned} planets have no matching system!")

# SCD2 logic (simplified for first load demonstration)
window_spec = Window.orderBy("planet_id")

new_dim = source_with_sk.select(
 (row_number().over(window_spec) + max_key).alias("planet_key"),
 col("planet_id"),
 col("system_key"),
 col("planet_name"),
 col("planet_type"),
 col("orbital_period"),
 col("has_atmosphere"),
 col("habitability_score"),
 lit(data_date).cast("date").alias("effective_date"),
 lit("9999-12-31").cast("date").alias("expiration_date"),
 lit("Y").alias("current_flag"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 year(lit(data_date)).alias("effective_year")
).distinct()

# (Full SCD2 change detection would follow same pattern as dim_system)

transform_count = new_dim.count()
print(f"Dimension rows: {transform_count:,}")

# ============================================
# WRITE PHASE
# ============================================
print("\n--- WRITE PHASE ---")

spark.sql("ALTER TABLE space_exploration.dim_planet CREATE BRANCH IF NOT EXISTS staging")

new_dim.writeTo("space_exploration.dim_planet") \
 .option("branch", "staging") \
 .overwritePartitions()

print("Written to staging branch")

# ============================================
# AUDIT PHASE
# ============================================
print("\n--- AUDIT PHASE ---")

staging_df = spark.read \
 .option("branch", "staging") \
 .table("space_exploration.dim_planet")

audit_failures = []

# Audit 1: PK uniqueness
staged_count = staging_df.count()
unique_keys = staging_df.select("planet_key").distinct().count()
if staged_count != unique_keys:
 audit_failures.append(f"FAIL: Duplicate planet_keys")
else:
 print(f"Audit 1 - PK uniqueness: PASS")

# Audit 2: FK integrity to dim_system (published!)
published_system_keys = spark.read.table("space_exploration.dim_system") \
 .select("system_key").distinct()

orphaned_fk = staging_df \
 .filter(col("system_key").isNotNull()) \
 .select("system_key").distinct() \
 .join(published_system_keys, "system_key", "left_anti") \
 .count()

if orphaned_fk > 0:
 audit_failures.append(f"FAIL: {orphaned_fk} orphaned system_keys")
 print(f"Audit 2 - FK integrity: FAIL")
else:
 print("Audit 2 - FK integrity: PASS")

# Audit 3: NULL check
null_count = staging_df.filter(
 col("planet_key").isNull() | col("planet_id").isNull()
).count()
if null_count > 0:
 audit_failures.append(f"FAIL: {null_count} NULLs in critical fields")
else:
 print("Audit 3 - NULL check: PASS")

# ============================================
# PUBLISH PHASE
# ============================================
print("\n--- PUBLISH PHASE ---")

if len(audit_failures) > 0:
 print("\nAUDIT FAILED:")
 for f in audit_failures:
 print(f" {f}")
 spark.sql("ALTER TABLE space_exploration.dim_planet DROP BRANCH staging")
 raise Exception(f"Audit failed")
else:
 print("\nALL AUDITS PASSED")
 spark.sql("""
 CALL system.fast_forward(
 table => 'space_exploration.dim_planet',
 branch => 'main',
 to => 'staging'
 )
 """)
 spark.sql("ALTER TABLE space_exploration.dim_planet DROP BRANCH staging")
 print(f"PUBLISHED: dim_planet ({staged_count:,} rows)")

job.commit()

Job 4: Build Fact Table (Incremental with Delete-Insert)

This is the critical job. It reads from published dimensions (main branch), uses delete-then-insert pattern for the date partition, and includes left anti-join as a defensive check for missing records.

Click to expand: build_fact_system_updates Glue Job

This job loads facts incrementally by date. For a given data_date, it deletes existing records and inserts new ones. It validates FK integrity against all published dimensions before merging to main.

# Glue Job: build_fact_system_updates
# Incremental load with delete-then-insert + left anti-join defensive coding

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *
from pyspark.sql.window import Window

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'DATA_DATE'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

data_date = args['DATA_DATE']
print("="*60)
print(f"Building Fact Table: System Updates (Incremental)")
print(f"Data Date: {data_date}")
print("="*60)

# ============================================
# LOAD PHASE
# ============================================
print("\n--- LOAD PHASE ---")

# Read from PSA for this data_date
source_df = spark.read.table("space_exploration.stg_updates") \
 .filter(col("data_date") == data_date)

source_count = source_df.count()
print(f"Source rows: {source_count:,}")

if source_count == 0:
 print("No data. Exiting.")
 job.commit()
 sys.exit(0)

# Read PUBLISHED dimensions (main branch)
# Fact tables MUST read from published dimensions, not staging branches
dim_system = spark.read.table("space_exploration.dim_system") \
 .filter(col("current_flag") == "Y") \
 .select("system_key", "system_id")

dim_date = spark.read.table("space_exploration.dim_date") \
 .select("date_key", "full_date", "year")

print(f"dim_system rows: {dim_system.count():,}")
print(f"dim_date rows: {dim_date.count():,}")

# Read existing fact table for incremental logic
try:
 existing_fact = spark.read.table("space_exploration.fact_system_updates")
 existing_count = existing_fact.count()
 print(f"Existing fact rows: {existing_count:,}")
 max_key = existing_fact.agg(max("update_key")).collect()[0][0] or 0
except:
 print("No existing fact data (first load)")
 existing_fact = None
 max_key = 0

# ============================================
# TRANSFORM PHASE
# ============================================
print("\n--- TRANSFORM PHASE ---")

# Add date column for joining
source_with_date = source_df.withColumn(
 "update_date",
 to_date("update_timestamp")
)

# Lookup surrogate keys from published dimensions
fact_transformed = source_with_date \
 .join(dim_system, source_with_date.system_id == dim_system.system_id, "left") \
 .join(dim_date, source_with_date.update_date == dim_date.full_date, "left") \
 .select(
 col("date_key"),
 dim_system.system_key,
 source_with_date.update_timestamp,
 col("status"),
 col("ships_present"),
 col("resource_level"),
 lit(data_date).cast("date").alias("data_date"),
 dim_date.year.alias("date_year")
 )

# Track orphaned facts
orphaned_systems = fact_transformed.filter(col("system_key").isNull()).count()
orphaned_dates = fact_transformed.filter(col("date_key").isNull()).count()

if orphaned_systems > 0:
 print(f"WARNING: {orphaned_systems} facts have no matching system")
if orphaned_dates > 0:
 print(f"WARNING: {orphaned_dates} facts have no matching date")

# ============================================
# INCREMENTAL LOGIC: Delete-then-Insert + Left Anti-Join
# ============================================
print("\n--- INCREMENTAL LOGIC ---")

# Extract the date partition we're loading
date_partition_year = int(data_date[:4]) # e.g., 2026

if existing_fact is not None:
 # Step 1: Get existing records for this data_date
 existing_for_date = existing_fact.filter(col("data_date") == data_date)
 existing_for_date_count = existing_for_date.count()
 print(f"Existing records for data_date={data_date}: {existing_for_date_count:,}")

 # Step 2: Records NOT in this date (keep as-is)
 records_to_keep = existing_fact.filter(col("data_date") != data_date)
 keep_count = records_to_keep.count()
 print(f"Records to keep (other dates): {keep_count:,}")

 # Step 3: New records from source (delete-then-insert for this date)
 # Generate new surrogate keys starting after max existing key
 window_spec = Window.orderBy("update_timestamp", "system_key")

 new_records = fact_transformed.select(
 (row_number().over(window_spec) + max_key).alias("update_key"),
 col("date_key"),
 col("system_key"),
 col("update_timestamp"),
 col("status"),
 col("ships_present"),
 col("resource_level"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 col("data_date"),
 col("date_year")
 )

 new_count = new_records.count()
 print(f"New records for data_date={data_date}: {new_count:,}")

 # Step 4: Left anti-join defensive check
 # Find any records in destination that should exist but don't
 # (This catches edge cases where source has records not in our transform)
 # For this example, we're doing a full replace for the date, so this is informational

 # Step 5: Combine: kept records + new records (effectively delete-insert)
 final_fact = records_to_keep.unionByName(new_records)

else:
 # First load: all records are new
 window_spec = Window.orderBy("update_timestamp", "system_key")

 final_fact = fact_transformed.select(
 row_number().over(window_spec).alias("update_key"),
 col("date_key"),
 col("system_key"),
 col("update_timestamp"),
 col("status"),
 col("ships_present"),
 col("resource_level"),
 current_timestamp().alias("load_timestamp"),
 lit("ELITE_DANGEROUS").alias("source_system"),
 col("data_date"),
 col("date_year")
 )

transform_count = final_fact.count()
print(f"Final fact rows: {transform_count:,}")

# ============================================
# WRITE PHASE
# ============================================
print("\n--- WRITE PHASE ---")

spark.sql("ALTER TABLE space_exploration.fact_system_updates CREATE BRANCH IF NOT EXISTS staging")

final_fact.writeTo("space_exploration.fact_system_updates") \
 .option("branch", "staging") \
 .overwritePartitions()

print("Written to staging branch")

# ============================================
# AUDIT PHASE
# ============================================
print("\n--- AUDIT PHASE ---")

staging_df = spark.read \
 .option("branch", "staging") \
 .table("space_exploration.fact_system_updates")

audit_failures = []

# Audit 1: Row count (source to target for this date)
staged_for_date = staging_df.filter(col("data_date") == data_date).count()
print(f"Audit 1 - Row count for {data_date}: Source={source_count:,}, Target={staged_for_date:,}")
if source_count != staged_for_date:
 # This might be OK if we filtered orphans, but flag for review
 print(f" Note: Difference of {source_count - staged_for_date} (likely orphaned records)")

# Audit 2: PK uniqueness
staged_count = staging_df.count()
unique_keys = staging_df.select("update_key").distinct().count()
if staged_count != unique_keys:
 audit_failures.append(f"FAIL: Duplicate update_keys")
 print(f"Audit 2 - PK uniqueness: FAIL")
else:
 print(f"Audit 2 - PK uniqueness: PASS ({staged_count:,} unique)")

# Audit 3: FK integrity to dim_system
published_system_keys = spark.read.table("space_exploration.dim_system") \
 .select("system_key").distinct()

orphaned_fk_system = staging_df \
 .filter(col("system_key").isNotNull()) \
 .select("system_key").distinct() \
 .join(published_system_keys, "system_key", "left_anti") \
 .count()

if orphaned_fk_system > 0:
 audit_failures.append(f"FAIL: {orphaned_fk_system} orphaned system_keys")
 print(f"Audit 3 - FK to dim_system: FAIL")
else:
 print("Audit 3 - FK to dim_system: PASS")

# Audit 4: FK integrity to dim_date
published_date_keys = spark.read.table("space_exploration.dim_date") \
 .select("date_key").distinct()

orphaned_fk_date = staging_df \
 .filter(col("date_key").isNotNull()) \
 .select("date_key").distinct() \
 .join(published_date_keys, "date_key", "left_anti") \
 .count()

if orphaned_fk_date > 0:
 audit_failures.append(f"FAIL: {orphaned_fk_date} orphaned date_keys")
 print(f"Audit 4 - FK to dim_date: FAIL")
else:
 print("Audit 4 - FK to dim_date: PASS")

# Audit 5: No NULLs in mandatory FKs
# (We allow NULLs for orphaned records, but track them)
null_system = staging_df.filter(col("system_key").isNull()).count()
null_date = staging_df.filter(col("date_key").isNull()).count()
print(f"Audit 5 - NULL FKs: {null_system} null system_key, {null_date} null date_key")

# Audit 6: Value range checks
invalid_resources = staging_df.filter(
 (col("resource_level") < 0) | (col("resource_level") > 1)
).count()
if invalid_resources > 0:
 audit_failures.append(f"FAIL: {invalid_resources} invalid resource_level values")
 print(f"Audit 6 - Value ranges: FAIL")
else:
 print("Audit 6 - Value ranges: PASS")

# Audit 7: No future timestamps
future_ts = staging_df.filter(col("update_timestamp") > current_timestamp()).count()
if future_ts > 0:
 audit_failures.append(f"FAIL: {future_ts} future timestamps")
 print(f"Audit 7 - Timestamps: FAIL")
else:
 print("Audit 7 - Timestamps: PASS")

# ============================================
# PUBLISH PHASE
# ============================================
print("\n--- PUBLISH PHASE ---")

if len(audit_failures) > 0:
 print("\nAUDIT FAILED:")
 for f in audit_failures:
 print(f" {f}")
 spark.sql("ALTER TABLE space_exploration.fact_system_updates DROP BRANCH staging")
 raise Exception(f"Audit failed with {len(audit_failures)} errors")
else:
 print("\nALL AUDITS PASSED")
 spark.sql("""
 CALL system.fast_forward(
 table => 'space_exploration.fact_system_updates',
 branch => 'main',
 to => 'staging'
 )
 """)
 spark.sql("ALTER TABLE space_exploration.fact_system_updates DROP BRANCH staging")
 print(f"PUBLISHED: fact_system_updates ({staged_count:,} rows)")

job.commit()

Step Functions Workflow

The Step Functions workflow orchestrates everything. Here’s the execution flow:

Stage 1 (Parallel): dim_date and dim_system run simultaneously—neither depends on the other
Stage 2: dim_planet runs after Stage 1 completes (needs dim_system’s surrogate keys)
Stage 3: fact_system_updates runs last (needs all dimension surrogate keys)

If any job fails, the workflow stops and sends an SNS notification.

Click to expand: Step Functions JSON Definition

{
 "Comment": "Dimensional Modeling Pipeline - Per-Table WAP Pattern",
 "StartAt": "Parallel Dimensions Stage 1",
 "States": {
 "Parallel Dimensions Stage 1": {
 "Type": "Parallel",
 "Comment": "dim_date and dim_system have no dependencies",
 "Branches": [
 {
 "StartAt": "Build dim_date (WAP)",
 "States": {
 "Build dim_date (WAP)": {
 "Type": "Task",
 "Resource": "arn:aws:states:::glue:startJobRun.sync",
 "Parameters": {
 "JobName": "build_dim_date"
 },
 "End": true
 }
 }
 },
 {
 "StartAt": "Build dim_system (WAP)",
 "States": {
 "Build dim_system (WAP)": {
 "Type": "Task",
 "Resource": "arn:aws:states:::glue:startJobRun.sync",
 "Parameters": {
 "JobName": "build_dim_system",
 "Arguments": {
 "--DATA_DATE.$": "$.data_date"
 }
 },
 "End": true
 }
 }
 }
 ],
 "Next": "Build dim_planet (WAP)",
 "Catch": [
 {
 "ErrorEquals": ["States.ALL"],
 "Next": "Notify Failure",
 "ResultPath": "$.error"
 }
 ]
 },
 "Build dim_planet (WAP)": {
 "Type": "Task",
 "Comment": "Depends on dim_system being published",
 "Resource": "arn:aws:states:::glue:startJobRun.sync",
 "Parameters": {
 "JobName": "build_dim_planet",
 "Arguments": {
 "--DATA_DATE.$": "$.data_date"
 }
 },
 "Next": "Build fact_system_updates (WAP)",
 "Catch": [
 {
 "ErrorEquals": ["States.ALL"],
 "Next": "Notify Failure",
 "ResultPath": "$.error"
 }
 ]
 },
 "Build fact_system_updates (WAP)": {
 "Type": "Task",
 "Comment": "Depends on ALL dimensions being published",
 "Resource": "arn:aws:states:::glue:startJobRun.sync",
 "Parameters": {
 "JobName": "build_fact_system_updates",
 "Arguments": {
 "--DATA_DATE.$": "$.data_date"
 }
 },
 "Next": "Pipeline Succeeded",
 "Catch": [
 {
 "ErrorEquals": ["States.ALL"],
 "Next": "Notify Failure",
 "ResultPath": "$.error"
 }
 ]
 },
 "Notify Failure": {
 "Type": "Task",
 "Resource": "arn:aws:states:::sns:publish",
 "Parameters": {
 "TopicArn": "arn:aws:sns:ap-southeast-2:ACCOUNT:pipeline-alerts",
 "Message.$": "States.Format('Pipeline failed: {}', $.error.Cause)",
 "Subject": "Data Pipeline Failed"
 },
 "Next": "Pipeline Failed"
 },
 "Pipeline Succeeded": {
 "Type": "Succeed"
 },
 "Pipeline Failed": {
 "Type": "Fail",
 "Error": "PipelineError",
 "Cause": "One or more jobs failed - check individual job logs"
 }
 }
}

Start the workflow with input:

{
 "data_date": "2026-01-17"
}

This data_date propagates to each job, enabling:

PSA partition reads (WHERE data_date = '2026-01-17')
Historical reprocessing (re-run with a past date)
Dependency triggers (S3 event on partition write triggers workflow)

Dependency Triggers with S3 Events

You can improve this further by using partition writes as triggers. Here’s how to set that up:

Option 1: S3 Event → EventBridge → Step Functions

When a file lands in landing/updates/, trigger the pipeline:

# EventBridge Rule
{
 "source": ["aws.s3"],
 "detail-type": ["Object Created"],
 "detail":
 {
 "bucket": { "name": ["space-data-pipeline-demo"] },
 "object": { "key": [{ "prefix": "landing/updates/" }] },
 },
}

Option 2: Glue Trigger on Partition

Configure Glue to watch for new partitions:

# Check if partition exists before running
partition_path = f"s3://space-data-pipeline-demo/staging/stg_updates/data_date={data_date}/"

import boto3
s3 = boto3.client('s3')

try:
 s3.head_object(
 Bucket='space-data-pipeline-demo',
 Key=f'staging/stg_updates/data_date={data_date}/_SUCCESS'
 )
 print(f"Partition exists - proceeding")
except:
 print(f"Partition not ready - exiting")
 sys.exit(0)

This lets you build dependency chains based on partition availability.

The Moment of Truth: Querying

Now let’s query the properly partitioned dimensional model:

-- Query leverages date partition pruning
SELECT
 dt.year,
 dt.quarter,
 d.main_star_type,
 COUNT(*) as updates,
 SUM(f.ships_present) as total_ships,
 ROUND(AVG(f.resource_level), 3) as avg_resources
FROM
 space_exploration.fact_system_updates f

 LEFT JOIN space_exploration.dim_system d
 ON f.system_key = d.system_key
 AND d.current_flag = 'Y' -- Only current dimension records!

 LEFT JOIN space_exploration.dim_date dt
 ON f.date_key = dt.date_key
WHERE
 dt.year = 2026 -- Partition pruning!
 AND dt.quarter = 1
GROUP BY
 dt.year, dt.quarter, d.main_star_type
ORDER BY
 total_ships DESC;

Notice d.current_flag = 'Y' — this is how you query SCD Type 2 dimensions for current state. For point-in-time analysis:

-- Point-in-time analysis: What was system status on Jan 15, 2026?
SELECT
 d.system_name,
 d.exploration_status,
 d.effective_date,
 d.expiration_date
FROM
 space_exploration.dim_system d
WHERE
 d.system_id = 'SOL'
 AND DATE '2026-01-15' BETWEEN d.effective_date AND d.expiration_date;

A Few Things Worth Remembering

WAP at each table, not just at the end. Each job should do its own Write-Audit-Publish cycle. Failures are isolated. Downstream jobs read from published (main) tables.

Landing is raw, Staging is persistent. Landing = 1:1 from source, latest only. Staging (PSA) = partitioned by data_date for historical reprocessing.

dim_date is a reference dimension. Load it once with a wide date range. Partition by year. Rarely needs updates.

SCD Type 2 for dimensions that change. Add effective_date, expiration_date, current_flag. Partition by effective year. Query with current_flag = 'Y' for current state.

Incremental facts with delete-then-insert. For a given date partition: delete existing, insert new. Use left anti-join as defensive check for missing records.

Partition strategically:

dim_date: by year
dim_system/planet: by effective_year (SCD2)
fact: by date_year

Use data_date for dependencies. When stg_updates/data_date=2026-01-17/ is written, trigger the downstream pipeline. This makes dependencies explicit and reprocessing easy.

That ten-minute query? Down to 1.2 seconds with partition pruning.

That conference room moment? Each job’s WAP cycle catches issues before they propagate.

These patterns work. They’ve saved me from embarrassment, late nights, and angry emails. They’ll do the same for you.

Why Dimensional Modelling Isn't Dead—It's Just Getting Started

Chris Hillman — Fri, 07 Nov 2025 09:00:00 +1100

The Great Data Modelling Debate Nobody Asked For

Another meeting where someone confidently declared, “We don’t need data modelling anymore—just dump everything in the data lake and let analysts figure it out.”

I’ve heard variations of this statement for years now, in meetings or at conferences. The pitch is always the same: traditional data warehousing is dead, dimensional modelling is a relic from the 90s, and modern big data tools have made structured modelling obsolete. Schema-on-read is the future. Agility over architecture.

You know what’s fascinating? The people making these arguments usually end up drowning in a data swamp they can’t navigate.

Here’s the thing nobody wants to admit: dimensional modelling isn’t dead—it’s more widely used than at any other point in history. Ralph Kimball’s methodology from 1996 now powers over 90% of enterprise data warehouses, delivering measurable ROI of 401% over three years. Companies from Uber (managing 100+ petabytes) to Netflix, Airbnb, and Spotify continue betting on dimensional approaches because they deliver what actually matters: fast queries, consistent metrics, and analytics that business users can understand.

The debate isn’t whether to model data. It’s how to adapt proven principles to modern platforms.

Let me show you why.

The Fundamentals That Refuse to Die

Ralph Kimball introduced dimensional modelling to the data warehousing world in 1996 with “The Data Warehouse Toolkit,” which became the definitive guide for organising analytical data. The methodology centres on a deceptively simple four-step process: select the business process, declare the grain, identify the dimensions, and identify the facts.

This framework emerged from Kimball’s background in human-computer interaction at Xerox PARC, where he helped design the Star Workstation—the first commercial product using mice, icons, and windows. His insight wasn’t just about database structure. It was about how humans naturally think about business measurements.

The brilliance lies in a fundamental division: separate measurements from context. Facts are numeric measurements taken repeatedly from business processes—sales amounts, quantities, durations. Dimensions provide the “who, what, where, when, why, and how” context surrounding those measurements. This creates star schemas where a central fact table connects to surrounding dimension tables, resembling a star when visualised.

The denormalised structure deliberately relaxes traditional database normalisation rules to optimise for analytical queries rather than transactional updates. And that’s where the magic happens.

Star schemas deliver concrete performance advantages. According to the official Kimball methodology documentation, the benefits include simpler queries compared to highly normalised schemas, simplified business reporting logic, query performance gains for read-only applications, and fast aggregations. The flat, denormalised dimension tables are perfect targets for bitmap indexes, while star-join optimisations built into modern database engines can process these patterns efficiently.

Kimball explicitly warns against snowflaking dimensions into third normal form because it “destroys the ability to use bitmap indexes and increases the user-perceived complexity of the design.” When you’re trying to answer business questions quickly, that complexity tax adds up fast.

The Integration Secret: Conformed Dimensions

Here’s where dimensional modelling gets really powerful—and where most data lakes completely fall apart.

Conformed dimensions represent the integration backbone of enterprise data warehousing. When dimension tables share identical column names and domain contents across different fact tables, they enable “drilling across”—combining information from separate business processes in unified reports.

Think about it this way: a date dimension conforms across sales, inventory, and customer service. A product dimension spans sales, procurement, and manufacturing. As the Kimball Group states: “This is the essence of integration in an enterprise DW/BI system. Conformed dimensions, defined once in collaboration with the business’s data governance representatives, are reused across fact tables; they deliver both analytic consistency and reduced future development costs because the wheel is not repeatedly re-created.”

I can’t tell you how many teams I’ve seen struggle because they have five different “customer” definitions, three different ways to calculate “revenue,” and no standard date dimension. Every analyst builds their own version. Every report uses slightly different logic. Nobody trusts the numbers because they never match.

Conformed dimensions solve this. Once. Properly. Forever.

Slowly Changing Dimensions: The Time Machine Nobody Talks About

One of the most elegant solutions in dimensional modelling addresses a tricky problem: how do you track attributes that change over time?

Slowly Changing Dimensions (SCDs) handle this challenge through several type patterns:

Type 0: Preserves original values that never change
Type 1: Overwrites old values, destroying history but remaining simple
Type 2: Adds new rows with new surrogate keys whenever attributes change, preserving complete historical accuracy through row effective dates, expiration dates, and current row indicators
Type 3: Adds columns for alternate reality comparisons
Types 4-7: Hybrid approaches for rapidly changing attributes or dual historical/current perspectives

Type 2 is the most common approach, and for good reason. It enables time-variant analysis while maintaining referential integrity between facts and dimensions. You can answer questions like “what was this customer’s address when they placed that order three years ago?” without complex joins or temporal logic in every query.

Try doing that efficiently in a schema-on-read data lake without proper modelling. I’ll wait.

The Performance Numbers That Matter

Let’s talk about actual performance, because that’s where the rubber meets the road.

Apache Doris demonstrated 31x performance improvement on Star Schema Benchmark (SSB) queries between versions. StarRocks proved 4.75x faster than Apache Druid and 1.87x faster than ClickHouse on 100GB datasets. Most queries complete under 200 milliseconds, with Apache Kylin achieving O(1) constant-time query latency regardless of data size.

Academic research on distributed dimensional data warehouses shows materialised views can reduce query processing time by 89-98% with only 13% disk space overhead.

But performance isn’t just about milliseconds. McKinsey Global Institute research reveals the business impact: retailers using big data to the full have potential to increase operating margins by more than 60%. Organisations leveraging data-driven approaches are 23 times more likely to acquire customers, 6 times more likely to retain customers, and 19 times more likely to be profitable.

These aren’t small companies or theoretical studies. McKinsey examined five domains globally including healthcare, retail, manufacturing, and public sector organisations.

The Hidden Tax of Bad Data

Here’s what nobody tells you about skipping data modelling: you pay for it anyway, just later and more expensively.

Gartner research shows organisations lose an average of $12.9 million annually due to poor data quality. IBM estimates US businesses lose $3.1 trillion collectively. The Wakefield Research survey found over 50% of respondents indicated 25% or more of their revenue was affected by data quality issues.

The 1x10x100 Rule quantifies escalating remediation costs: fixing a data quality issue at point of entry costs 1x, fixing after it propagates within systems costs 10x, and fixing after it reaches end-users and decision-making costs 100x.

Let me give you some real-world disasters:

Unity Software lost $110 million in revenue and $4.2 billion in market cap in 2022 from ingesting bad data from a large customer, causing 37% drop in share price and loss of investor confidence.

Samsung Securities suffered a $300 million “fat-finger” error when an employee mistakenly paid 1,000 shares instead of 1,000 won per share in dividends, issuing $105 billion worth of shares before correction.

Equifax sent lenders inaccurate credit scores on millions of customers due to coding issues, dropping stock 5% and triggering class-action lawsuits.

JPMorgan Chase’s “London Whale” incident lost $6.2 billion, exacerbated by errors in risk models relying on flawed data.

NASA’s Mars Climate Orbiter burned $125 million due to unit mismatches between metric and imperial measurements.

These aren’t abstract warnings. They’re real money disappearing from real balance sheets.

The Productivity Drain Nobody Measures

Want to know where your data team’s time actually goes?

Data scientists and analysts spend 60% of their time cleaning and organising data according to Forbes analysis, with 19% more spent searching for information. Anaconda’s State of Data Science surveys consistently show 39-45% of time on data preparation tasks rather than actual analysis.

An IDC report found 44% of data workers’ time is wasted on unsuccessful activities, with 33% (12.5 hours weekly) spent preparing information and 15% searching for data. For a team of 10 analysts, this represents 8,900+ hours of lost productivity annually.

Think about that. Your highly skilled analysts—the people you hired to generate insights—are spending half their time just trying to figure out what the data means and how to structure it for analysis.

Dimensional modelling solves this because the structure itself communicates meaning. An analyst looks at a star schema and immediately understands: here are the measurements, here’s the context, here’s how they relate. The modelling work gets done once by experts, then leveraged by everyone. It follows that DRY mantra of programming for years - Don’t Repeat Yourself.

The Inconsistent Metrics Problem

This is the silent killer of data team credibility.

When three different tables represent the same business entity with different SQL logic, two analysts answering identical questions produce different results. Stakeholders lose confidence. Executive meetings devolve into debates where one department reports growing customer base while another reports increased churn.

Simple metrics like Daily Active Users or Revenue get calculated differently by different teams. One defines DAU as logins, another as specific actions, a third as time spent—each producing wildly different results.

Years back I was in a meeting where 2 GM’s literally argue about whether revenue is up or down because they’re looking at reports built on different data models (different teams data) with different business logic. The data team ends up spending hours reconciling numbers instead of generating insights.

Conformed dimensions and standardised fact tables eliminate this problem. There’s one product dimension. One customer dimension. One date dimension. One revenue fact. When everyone queries the same well-modelled tables, everyone gets the same answers.

Real Companies, Real Scale, Real Results

Let’s look at organisations actually doing this at scale.

Uber’s Global Data Warehouse

Uber’s data lake consists of “foundational fact, dimension, and aggregate tables developed using dimensional data modelling techniques” accessible to engineers and data scientists in a self-serve manner.

The scale is staggering:

100+ petabytes of data in HDFS
100,000 vcores in compute cluster
100,000 Presto queries per day
10,000 Spark jobs per day
20,000 Hive queries per day

Their incremental ETL approach using Apache Hudi delivered 50% decrease in pipeline run time and 60% decrease in SLA, supporting minute-level latency for petabyte-scale data.

As Uber’s engineering blog states directly: “The data lake consists of foundational fact, dimension, and aggregate tables developed using dimensional data modelling techniques that can be accessed by engineers and data scientists in a self-serve manner to power data engineering, data science, machine learning, and reporting across Uber.”

Netflix’s Unified Data Architecture

Netflix integrates dimensional models for operational reporting while managing data from 500+ microservices globally. Their data warehouse built on Apache Iceberg tables uses GraphQL integration with knowledge-graph-based design for consistent data modelling.

This powers Sphere, Netflix’s self-service operational reporting tool for business users, handling multiple petabytes of data. The architecture demonstrates dimensional modelling works seamlessly with modern streaming and microservices architectures.

Spotify’s Event Processing

Spotify processes 500 billion events per day (70TB compressed daily) running 20,000 batch data pipelines across 1,000+ repositories. Their Google Cloud Platform implementation uses BigQuery for dimensional models supporting podcast analytics and business reporting.

The company attributes 75% year-over-year ad revenue growth to data-driven insights enabled by their dimensional data architecture.

Airbnb, Facebook, and Lyft’s Snapshot Approach

These companies pioneered snapshot-based dimensional modelling, a technique developed by Maxime Beauchemin (senior Lyft data engineer) that modernizes slowly changing dimensions for cloud environments.

Rather than complex SCD Type 2 logic, they create daily or weekly table partitions as snapshots of dimensional data, leveraging cheap cloud storage. Beauchemin’s philosophy: “Compute is cheap. Storage is cheap. Engineering time is expensive.”

This pragmatic adaptation of Kimball principles for cloud-native environments exemplifies how dimensional modelling evolves without abandoning core principles.

The Healthcare and Finance Proof Points

Want proof dimensional modelling works in regulated environments with strict compliance requirements?

Piedmont Healthcare manages 15 TB of data from 16 hospitals and 1,400 physician practices serving 2+ million patient visits annually. Their migration to Exasol with dimensional models reduced dashboard refresh time from 10 minutes to seconds while delivering 20 new metrics per month.

Kaiser Permanente’s HealthConnect integrates millions of patient records using dimensional models, enabling reduced hospital stays through early risk identification and improved chronic disease management. These implementations prove dimensional modelling satisfies stringent HIPAA and regulatory requirements while delivering performance.

JPMorgan Chase uses sophisticated dimensional data warehouse infrastructure for risk management, regulatory compliance, and personalized banking experiences.

Target Corporation’s Guest Data Platform integrates dimensional models for unified customer views, powering highly successful personalized marketing campaigns and optimised store layouts.

The ROI Study That Ends the Debate

An International Data Corporation study of 62 organisations documented average ROI of 401% over three years, excluding failed projects and extreme outliers.

Common metrics show:

Payback periods of 1-3 years
Positive net present value in 75%+ of implementations
15-30% reduction in operational costs
10-25% increase in data-driven revenue opportunities

These aren’t theoretical benefits. They represent measured outcomes from actual enterprise implementations.

Meanwhile, industry adoption rates paint a clear picture: 90%+ of enterprises globally utilise data warehousing infrastructure, with 58% of deployments cloud-based. The data warehousing market reached $34.5-35 billion in 2024, projected to grow to $75-93 billion by 2033.

Joe Reis, prominent data engineering consultant, observes: “Kimball is more in use than any other point in time due to being the default way to model data in Self-Service Business Intelligence applications (BI): Power BI and Tableau. And BI is 10 times bigger in usage than data engineering.”

Modern Platforms Embrace Dimensional Modelling

Here’s where the “dimensional modelling is dead” crowd gets it completely wrong. Modern cloud platforms haven’t replaced dimensional modelling—they’ve made it easier to implement.

Snowflake’s Explicit Recommendation

Snowflake’s official blog states: “When using Snowflake, store the raw data history in a structured or variant format, clean and fit the data using the third normal form or the Data Vault model, and it makes sense to store the final consumable data in the Kimball dimensional data model.”

The company notes each data model has advantages and “storing intermediate step results has significant architectural advantages.” Snowflake’s columnar storage, automatic query optimisation for star joins, and separation of compute and storage complement dimensional designs perfectly.

Databricks’ Lakehouse Approach

Databricks’ official guidance emphasises: “A large number of our customers are migrating their legacy data warehouses to Databricks Lakehouse… As we help customers in the field, we find that many are looking for best practices around proper data modelling and physical data model implementations in Databricks. For Data Warehousing, Analytics-friendly modelling styles like Star-schema and Data Vault are quite popular.”

The lakehouse medallion architecture (Bronze/Silver/Gold layers) typically implements dimensional models in the Gold layer for business consumption. Databricks supports identity columns for surrogate keys, primary/foreign key constraints, Z-order clustering for performance, and column-level data quality constraints—all features that facilitate dimensional modelling.

BigQuery’s Optimisation

BigQuery’s architecture optimises for dimensional models. The platform’s columnar storage, partitioning, clustering, and primary/foreign key hints enable query optimisers to efficiently execute dimensional model queries. Type 2 dimension tables at the same grain as fact tables can be flattened into physical tables, while the logical model maintains clear dimensional structure.

Microsoft Fabric’s Position

Microsoft Fabric’s official documentation states unequivocally: “A star schema design is optimised for analytic query workloads. For this reason, it’s considered a prerequisite for enterprise Power BI semantic models.”

The documentation emphasises that while specific circumstances might suggest alternatives, “the theory of dimensional modelling is still relevant. That theory helps analysts create intuitive and efficient models.”

The Fortune 500 Case Study Nobody Talks About

Dustin Dorsey (Principal Data Architect at Onyx) presented at the Open Data Science Conference in April 2024, busting modern myths about dimensional modelling.

He demonstrated a Fortune 500 company using dbt and Databricks that was “drowning in complexity” with one massive dbt model containing over 1,000 lines of code.

After implementing dimensional modelling:

21 modular dbt models replaced the monolith
179 tests replaced 2
6 measurable columns replaced 1
250+ accessible attributes became available

“Analysts went from reading 1,000 lines of code to writing 20.”

Performance remained comparable but with significantly better governance, scalability, and business usability.

This is the pattern I see repeatedly: teams start with the promise of agility and flexibility, build increasingly complex models to handle reality, then eventually refactor back to proper dimensional modelling after the pain becomes unbearable.

Why not start with the proven approach?

The Technical Debt Time Bomb

McKinsey research shows organisations spend 20-40% of technology estate value managing technical debt. Stripe’s engineering study found teams spend an average of 33% of their time managing technical debt—time that could drive innovation and growth.

Database migration horror stories illustrate accumulated debt coming due. Datometry analysis shows 300-500% cost overruns are more common than expected in data warehouse migrations, with consulting fees for large migrations reaching tens of millions, sometimes exceeding $100 million for Fortune 100 projects.

Hidden costs include:

Upgrading all ETL, BI, and reporting systems simultaneously
Business disruptions lasting weeks or months
Compliance re-certification for regulated workloads
Schema mismatch analysis and transformation
Data cleansing for decades of dirty data

One mid-sized e-commerce company with approximately 50 million users attempted MySQL to MongoDB migration with fundamentally broken data modelling—trying to recreate relational tables in a document database.

Product catalog queries requiring 15+ joins saw response times balloon from 50 milliseconds to 8 seconds. A Black Friday migration created data inconsistencies. AWS costs exploded from expected $15,000 monthly to $47,000 the first month.

This is what happens when you skip proper modelling and accumulate technical debt.

The Hybrid Approach: Using the Right Tool

Modern consensus has emerged around a polyglot data modelling strategy: use multiple modelling techniques appropriately.

Data Vault or normalised models in integration layers (Silver) provide auditability and flexibility for source system changes
Dimensional models in presentation layers (Gold) provide query performance and business user accessibility
Raw data in Bronze layers enables exploratory analysis

This approach uses the right model for the right use case rather than forcing a single methodology everywhere.

As Margy Ross, co-author with Ralph Kimball and president of DecisionWorks Consulting with 35+ years teaching dimensional modelling to over 10,000 students, addressed directly in 2017:

“The short answer is ‘yes.’ The need to focus on business process measurement events, plus grain, dimensions and facts, is as important as ever. Dimensional modelling has helped countless organisations across every industry make better business decisions which should be the true measure of DW/BI success. While it’s fun to try something new, why dismiss a proven, valuable technique that has provided positive return on investment?”

What Actually Works in 2025

The organisations thriving in modern data environments aren’t those that abandoned modelling for raw data lakes. They’re the ones that adapted dimensional modelling principles to new platforms, combining proven analytical design patterns with modern engineering practices.

Uber’s 100+ petabyte dimensional data lake, Netflix’s unified data architecture, and Spotify’s 500 billion daily events all demonstrate dimensional modelling scales to modern big data volumes.

For teams building modern data platforms, the question isn’t whether to use dimensional modelling—it’s how to implement Kimball principles with tools like dbt, cloud warehouses like Snowflake and Databricks, and lakehouse architectures.

Start with business process identification and careful grain declaration. Build conformed dimensions that enable integration across subject areas. Implement fact tables that capture measurements at atomic grain. Use modern tools like dbt for modular, testable transformations. Leverage cloud platform features for performance and scale. Test rigorously and document thoroughly.

The methodology Ralph Kimball introduced nearly three decades ago continues delivering value because it addresses fundamental challenges in organising data for analytics—challenges that persist regardless of underlying technology.

The evidence is overwhelming. The ROI is measurable. The alternatives are expensive.

Dimensional modelling isn’t dead. It’s just getting started.

References

Analytics8. “Data Modelling in the Modern Data Stack.” Analytics8 Blog. https://www.analytics8.com/blog/data-modeling-in-the-modern-data-stack/
Gartner Research. “The State of Data Quality 2024.” Gartner.
DBSync. “The Cost of Poor Data Quality.” DBSync Research.
McKinsey & Company. “Big Data: The Next Frontier for Innovation, Competition, and Productivity.” McKinsey Global Institute.
Uber Engineering. “Building Uber’s Data Lake.” Uber Engineering Blog.
GeeksforGeeks. “Dimensional Modelling in Data Warehouse.” GeeksforGeeks.
Data-Sleek. “The Four-Step Dimensional Design Process.” Data-Sleek Blog.
McKinsey & Company. “Analytics Comes of Age.” McKinsey Research.