<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Platforms &amp; Cloud on Ghost in the data</title><link>https://ghostinthedata.info/categories/platforms--cloud/</link><description>Ghost in the data</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>Ghost in the data</copyright><lastBuildDate>Sat, 18 Jul 2026 09:00:00 +1000</lastBuildDate><atom:link href="https://ghostinthedata.info/categories/platforms--cloud/index.xml" rel="self" type="application/rss+xml"/><item><title>AWS MWAA: The Practitioner's Unvarnished Guide</title><link>https://ghostinthedata.info/posts/2026/2026-07-18-mwaa-unvarnished-guide/</link><pubDate>Sat, 18 Jul 2026 09:00:00 +1000</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-07-18-mwaa-unvarnished-guide/</guid><author>Chris Hillman</author><description>Everything data engineers need to know about AWS MWAA — real costs, operational gotchas, and head-to-head comparisons with Astronomer, Dagster, Prefect, Step Functions, and self-hosted Airflow.</description><content:encoded>&lt;p>I remember the first time I sat down properly with the Step Functions architecture our team inherited. Someone had designed it thoughtfully — real engineering thought had gone into it when it was built.&lt;/p>
&lt;p>But what I was looking at in the console was a map of state machines and Lambda functions that had grown well past the point where anyone could hold it in their head.&lt;/p>
&lt;p>The first production issue we had to debug on that setup took most of a working day. Not because the problem was complicated — it wasn&amp;rsquo;t — but because finding it meant jumping between CloudWatch log groups, correlating timestamps, building a mental picture of which Lambda had received what input and where the chain had broken. Every time I thought I had it, another log group. Another timestamp comparison. Another dead end.&lt;/p>
&lt;p>That&amp;rsquo;s when I suggested to the team that we test out Airflow. I remember the first time something failed on the new setup. I clicked into the task, read the log attached to that specific run, saw the error, fixed it, and restarted the DAG. Four minutes. The CDK stacks for the old architecture ran to thousands of lines. The equivalent DAG was fifty to two hundred. Same logic. Made legible.&lt;/p>
&lt;p>The difference wasn&amp;rsquo;t performance — Lambda and Airflow are comparable where it counts. It was visibility. One place to look. That&amp;rsquo;s what orchestration is actually about: not execution speed, not infrastructure elegance, but whether you can understand what&amp;rsquo;s happening in your data platform — at 9am in a post-mortem, or at 2am when something breaks and you need to know why.&lt;/p>
&lt;p>AWS MWAA — Managed Workflows for Apache Airflow — is Amazon&amp;rsquo;s answer to this problem. And it&amp;rsquo;s genuinely good at some things. But the gap between what the marketing page promises and what you&amp;rsquo;ll actually experience in production is wider than most managed services I&amp;rsquo;ve worked with. The $365/month sticker price? That&amp;rsquo;s just the beginning. The &amp;ldquo;managed&amp;rdquo; part? It manages less than you&amp;rsquo;d think.&lt;/p>
&lt;p>This guide covers everything I wish someone had told me before committing to MWAA — the real architecture, the actual costs (including the ones AWS conveniently leaves off the pricing page), the operational gotchas that&amp;rsquo;ll bite you in production, and honest head-to-head comparisons with every serious alternative: Astronomer Astro, Dagster Cloud, Prefect Cloud, Step Functions, and the self-hosted approach that might be smarter than you&amp;rsquo;d expect.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="how-mwaa-actually-works-under-the-hood">How MWAA Actually Works Under the Hood&lt;/h3>
&lt;br>
&lt;p>MWAA runs on a &lt;strong>dual-VPC architecture&lt;/strong> that&amp;rsquo;s important to understand because it explains most of the service&amp;rsquo;s constraints.&lt;/p>
&lt;p>AWS provisions Fargate containers for your schedulers, workers, and web server inside an AWS-managed ECS cluster. These containers connect to your VPC through Elastic Network Interfaces injected into your private subnets. A single-tenant &lt;strong>Aurora PostgreSQL&lt;/strong> instance stores the Airflow metadata — fully managed by AWS, but completely inaccessible to you. No connection string. No direct SQL queries. No custom dashboards built against the metadata database. If you&amp;rsquo;ve ever relied on querying &lt;code>task_instance&lt;/code> or &lt;code>dag_run&lt;/code> tables directly for custom monitoring, MWAA takes that away.&lt;/p>
&lt;p>The networking requirements are specific and unforgiving. You need two private subnets in different Availability Zones, a NAT gateway (or VPC endpoints) for internet access, and a self-referencing security group. The VPC you choose at creation is permanent — change your mind later and you&amp;rsquo;re building a new environment from scratch. Choose private webserver mode (which most production teams do for security), and accessing the Airflow UI requires a VPN or bastion host setup.&lt;/p>
&lt;p>DAGs live in an S3 bucket with versioning enabled. MWAA syncs files to the workers every 30 seconds, though new DAG files take roughly five minutes to be recognised (controlled by &lt;code>dag_dir_list_interval&lt;/code>). Updates to &lt;code>requirements.txt&lt;/code> or &lt;code>plugins.zip&lt;/code> trigger a full environment update — and that means 20 to 40 minutes of waiting while MWAA reprovisions everything. More on that particular joy later.&lt;/p>
&lt;p>A few setup details that&amp;rsquo;ll save you time: the S3 bucket must be in the same region as your MWAA environment and must have block public access enabled. Your &lt;code>dags/&lt;/code> folder sits at the root of the bucket, with &lt;code>requirements.txt&lt;/code> alongside it (not inside the &lt;code>dags/&lt;/code> folder). The &lt;code>plugins.zip&lt;/code> file follows a strict structure — if your custom operator lives at &lt;code>plugins/operators/my_operator.py&lt;/code>, the zip must mirror that path exactly. Get the structure wrong and the import will silently fail with no useful error message until you dig into the DAGProcessing logs.&lt;/p>
&lt;p>For IAM, MWAA needs an execution role with access to your S3 bucket, CloudWatch Logs, SQS (for the Celery broker), and whatever AWS services your DAGs interact with — Glue, EMR, Redshift, Lambda, whatever. The principle of least privilege matters here, but AWS&amp;rsquo;s own documentation starts with fairly broad permissions and leaves tightening to you. In practice, most teams start broad and restrict after stabilising.&lt;/p>
&lt;br>
&lt;p>&lt;strong>Environment classes&lt;/strong> span six tiers since the 2024 additions, from micro through 2xlarge:&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Class&lt;/th>
 &lt;th>vCPU / Memory per Worker&lt;/th>
 &lt;th>Concurrent Tasks/Worker&lt;/th>
 &lt;th>Approx. DAG Capacity&lt;/th>
 &lt;th>Base Cost/hr&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>mw1.micro&lt;/td>
 &lt;td>Combined scheduler/worker: 1 vCPU, 3 GB&lt;/td>
 &lt;td>3&lt;/td>
 &lt;td>~25&lt;/td>
 &lt;td>~$0.05&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>mw1.small&lt;/td>
 &lt;td>1 vCPU, 2 GB&lt;/td>
 &lt;td>5&lt;/td>
 &lt;td>~50&lt;/td>
 &lt;td>$0.49&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>mw1.medium&lt;/td>
 &lt;td>2 vCPU, 4 GB&lt;/td>
 &lt;td>10&lt;/td>
 &lt;td>~250&lt;/td>
 &lt;td>~$0.74&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>mw1.large&lt;/td>
 &lt;td>4 vCPU, 8 GB&lt;/td>
 &lt;td>20&lt;/td>
 &lt;td>~1,000&lt;/td>
 &lt;td>$0.99&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>mw1.xlarge&lt;/td>
 &lt;td>8 vCPU, 24 GB&lt;/td>
 &lt;td>40&lt;/td>
 &lt;td>~2,000&lt;/td>
 &lt;td>~$1.49&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>mw1.2xlarge&lt;/td>
 &lt;td>16 vCPU, 48 GB&lt;/td>
 &lt;td>80&lt;/td>
 &lt;td>~4,000&lt;/td>
 &lt;td>~$1.99&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>The micro class (November 2024) is a genuine game-changer for dev/test — at roughly $37/month, you can finally have a development MWAA environment without the $365+ price floor. The catch: it combines the scheduler and worker into a single container with no autoscaling. Fine for testing DAGs, not for production.&lt;/p>
&lt;p>Worker autoscaling follows a straightforward formula: &lt;code>(running + queued tasks) / (tasks per worker) = required workers&lt;/code>. Scale-down triggers after running and queued tasks hit zero for more than two minutes, and takes two to five minutes to complete. A significant autoscaling bug — where tasks could be assigned to workers being decommissioned — persisted until May 2025. If you&amp;rsquo;re running a version from before that fix, you&amp;rsquo;ll want to upgrade.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-true-cost-of-mwaa-its-not-365-a-month">The True Cost of MWAA (It&amp;rsquo;s Not $365 a Month)&lt;/h3>
&lt;br>
&lt;p>The AWS pricing page says mw1.small costs $0.49/hr, which works out to $364.56/month. That number is misleading. Every production MWAA deployment carries infrastructure costs that the headline figure conveniently excludes.&lt;/p>
&lt;p>&lt;strong>NAT gateways are the number one hidden cost.&lt;/strong> The standard architecture requires two NAT gateways (one per AZ), costing $0.045/hr each — that&amp;rsquo;s $65.70/month before a single byte of data moves through them. Add $0.045/GB for data processing. One team documented a $12K monthly bill where 87% was unused NAT gateway costs from misconfigured routing. This is not an MWAA-specific problem, but MWAA forces you into it because of the VPC requirement.&lt;/p>
&lt;p>&lt;strong>CloudWatch metrics are the second surprise.&lt;/strong> MWAA auto-publishes per-DAG and per-task metrics, and one practitioner documented $272/month for custom metrics alone — 907 metrics at $0.30 per metric. Unless you actively restrict what gets published using &lt;code>metrics.statsd_allow_list&lt;/code>, you&amp;rsquo;ll see this line item growing steadily as you add more DAGs.&lt;/p>
&lt;p>&lt;strong>CloudWatch Logs&lt;/strong> at INFO level with active workflows adds $50–300/month in ingestion and storage charges, depending on how chatty your DAGs are.&lt;/p>
&lt;p>Here&amp;rsquo;s what a small team running a production mw1.small environment actually pays:&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Component&lt;/th>
 &lt;th>Monthly Cost&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>mw1.small environment (24/7)&lt;/td>
 &lt;td>$364.56&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>2 NAT gateways (hourly)&lt;/td>
 &lt;td>$65.70&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>NAT data processing (~20 GB)&lt;/td>
 &lt;td>$0.90&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>CloudWatch metrics&lt;/td>
 &lt;td>$30–100&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>CloudWatch Logs (~2 GB)&lt;/td>
 &lt;td>$1.00&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Elastic IPs (2)&lt;/td>
 &lt;td>$7.30&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>S3 + metadata storage&lt;/td>
 &lt;td>$1.50&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>Total&lt;/strong>&lt;/td>
 &lt;td>&lt;strong>$470–540&lt;/strong>&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>Scale up to mw1.large with additional workers running six hours a day, extra schedulers, and full logging, and you&amp;rsquo;re looking at $2,150–2,300/month all-in. AWS&amp;rsquo;s own pricing example for a comparable setup quotes $1,047/month — but conveniently excludes NAT, CloudWatch, and networking overhead.&lt;/p>
&lt;p>Here&amp;rsquo;s that breakdown for a heavier workload — say a mid-size data team running 200+ DAGs with burst periods:&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Component&lt;/th>
 &lt;th>Monthly Cost&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>mw1.large environment (24/7)&lt;/td>
 &lt;td>$733.68&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>19 additional workers (6 hrs/day avg)&lt;/td>
 &lt;td>$672.00&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>5 schedulers&lt;/td>
 &lt;td>$164.00&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>2 NAT gateways (hourly)&lt;/td>
 &lt;td>$65.70&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>NAT data processing (~100 GB)&lt;/td>
 &lt;td>$4.50&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>CloudWatch metrics (400+)&lt;/td>
 &lt;td>$120.00&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>CloudWatch Logs (~10 GB)&lt;/td>
 &lt;td>$5.03&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Elastic IPs, S3, misc.&lt;/td>
 &lt;td>$10.00&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>Total&lt;/strong>&lt;/td>
 &lt;td>&lt;strong>~$1,775–2,300&lt;/strong>&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>The range depends heavily on how well you control CloudWatch metric sprawl and how long your burst periods actually last. The point is: you need to model the &lt;em>real&lt;/em> infrastructure cost, not just the MWAA line item.&lt;/p>
&lt;br>
&lt;p>&lt;strong>Cost optimisation levers do exist.&lt;/strong> Replace NAT gateways with VPC endpoints — the S3 Gateway Endpoint is free, and interface endpoints process data at $0.01/GB versus NAT&amp;rsquo;s $0.045/GB. Restrict CloudWatch metrics with &lt;code>metrics.statsd_allow_list = scheduler,executor&lt;/code> or disable custom metrics entirely. Set log retention to 30–90 days instead of indefinite. For dev/test environments, implement pause/resume schedules — one team reported 70% savings by running MWAA only during business hours.&lt;/p>
&lt;p>The November 2025 launch of &lt;strong>MWAA Serverless&lt;/strong> fundamentally changes the cost equation for intermittent workloads. At $0.08/hr per task (billed per-second, minimum one minute), 2,000 tasks averaging one to two minutes costs roughly $4/month — versus $365+ for the always-on equivalent. The trade-off is substantial though: no Airflow UI, YAML-based workflow definitions, and only 80-odd AWS operators supported (no PythonOperator, no BashOperator). It&amp;rsquo;s better understood as a different product targeting a different use case than as a cheaper MWAA.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-gotchas">The Gotchas&lt;/h3>
&lt;br>
&lt;p>Practitioners consistently flag the same pain points, and understanding them before committing is worth more than any architecture diagram.&lt;/p>
&lt;p>&lt;strong>Dependency management is the highest-risk operational area.&lt;/strong> A bad &lt;code>requirements.txt&lt;/code> can crash-loop your entire environment for hours. Not minutes — hours. The constraint statement is mandatory since Airflow 2.7.2, and omitting it risks catastrophic dependency conflicts. If pip install exceeds ten minutes, the Fargate task times out and rolls back — leaving your environment stuck in an update cycle. Certain packages can break MWAA&amp;rsquo;s CloudWatch logging by overriding the &lt;code>watchtower&lt;/code> library. The only safe approach: always test with the &lt;code>aws-mwaa-local-runner&lt;/code> Docker tool before deploying dependencies to production.&lt;/p>
&lt;p>&lt;strong>Environment update times frustrate everyone.&lt;/strong> Creating an environment takes 25–30 minutes. Updating requirements or plugins: 20–40 minutes. Version upgrades: up to two hours with the environment unavailable during the process. The May 2025 &amp;ldquo;graceful updates&amp;rdquo; feature helps — it replaces components without interrupting running tasks — but the wall-clock time for the update itself hasn&amp;rsquo;t changed meaningfully.&lt;/p>
&lt;p>&lt;strong>Observability is CloudWatch-only, and navigating it is painful.&lt;/strong> MWAA creates five separate CloudWatch log groups per environment: DAGProcessing, Scheduler, Task, WebServer, and Worker. Correlating a single pipeline failure requires jumping between log groups, and CloudWatch Logs Insights queries aren&amp;rsquo;t intuitive for Airflow-style troubleshooting. You can&amp;rsquo;t redirect StatsD metrics to external tools like Datadog or Prometheus — MWAA overrides the &lt;code>statsd_host&lt;/code> configuration. External monitoring integration requires CloudWatch Logs subscriptions via Kinesis Data Firehose, adding both complexity and cost.&lt;/p>
&lt;p>If you&amp;rsquo;ve spent time in the trenches of AWS logs — and maybe this is just me — it can be a genuine nightmare to piece together what happened across services. MWAA should improve this, and in some ways it does (the Airflow UI&amp;rsquo;s task logs are excellent). But the moment you need infrastructure-level debugging, you&amp;rsquo;re back in CloudWatch hell.&lt;/p>
&lt;p>&lt;strong>The 12-hour task execution limit is a hard wall.&lt;/strong> Tasks exceeding 12 hours are killed and get stuck in the SQS queue for another 12 hours due to SQS timeout settings. The CeleryExecutor-only constraint means no per-task Docker images and no pod-level resource isolation. If you need tasks that run longer than 12 hours, you&amp;rsquo;ll need to offload compute to ECS Fargate or Lambda, using MWAA purely for orchestration.&lt;/p>
&lt;p>&lt;strong>Secrets Manager integration works but generates surprising API volume.&lt;/strong> Without configuring &lt;code>connections_lookup_pattern&lt;/code>, Airflow attempts to look up every connection in Secrets Manager — including non-existent ones. One practitioner reported 100,000+ error API calls per day per MWAA instance. Secrets stored in Secrets Manager don&amp;rsquo;t appear in the Airflow UI either, which only shows connections from its own backend.&lt;/p>
&lt;p>Other gotchas worth noting: you can&amp;rsquo;t remove &lt;code>plugins.zip&lt;/code> or &lt;code>requirements.txt&lt;/code> once added (you can only point to empty files), the VPC can&amp;rsquo;t be changed after creation, frequent DAG updates to the S3 folder can break the MWAA installation, and if VPC endpoints are accidentally deleted the environment is broken and must be entirely recreated.&lt;/p>
&lt;p>One more that catches people off guard: &lt;strong>Airflow configuration overrides&lt;/strong>. MWAA lets you set most Airflow configuration options through the console or API, but several are explicitly blocked — &lt;code>statsd_host&lt;/code>, &lt;code>statsd_port&lt;/code>, &lt;code>broker_url&lt;/code>, &lt;code>result_backend&lt;/code>, and anything related to the metadata database connection. These are locked down because MWAA manages those components. If your Airflow experience includes tuning Celery broker settings or metadata database connection pools, you&amp;rsquo;ll need to adjust your expectations about what &amp;ldquo;managed&amp;rdquo; means here. It means AWS decides those values, and some of them (particularly the default Celery broker configuration) aren&amp;rsquo;t optimal for all workload patterns.&lt;/p>
&lt;p>The practical impact: if you&amp;rsquo;ve tuned self-hosted Airflow for specific performance characteristics — aggressive task polling, custom broker acknowledgement settings, metadata database query timeouts — MWAA won&amp;rsquo;t let you replicate that tuning. For most teams this doesn&amp;rsquo;t matter. For teams at scale with specific performance requirements, it can be a dealbreaker.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="head-to-head-astronomer-astro">Head-to-Head: Astronomer Astro&lt;/h3>
&lt;br>
&lt;p>Astronomer&amp;rsquo;s Astro platform runs Apache Airflow underneath, so your DAGs, operators, and muscle memory all transfer. The differences are in the operational layer — and they&amp;rsquo;re significant.&lt;/p>
&lt;p>&lt;strong>Workers scale to zero.&lt;/strong> When no tasks are running, you pay nothing for compute. MWAA charges for the environment 24/7 regardless. Astro&amp;rsquo;s base pricing starts at $0.35/hr for the control plane, with worker costs at $0.13/hr that only accrue when tasks actually execute.&lt;/p>
&lt;p>&lt;strong>The developer experience gap is substantial.&lt;/strong> Astro CLI provides a full local Airflow environment with &lt;code>astro dev start&lt;/code> that mirrors production exactly — same Airflow version, same Python version, same provider packages. MWAA has no equivalent. The &lt;code>aws-mwaa-local-runner&lt;/code> Docker tool is helpful but doesn&amp;rsquo;t perfectly replicate the MWAA environment. DAG deployment via &lt;code>astro deploy&lt;/code> takes 5–10 minutes through a proper CI/CD pipeline; MWAA&amp;rsquo;s S3 upload plus sync cycle is slower and less integrated into developer workflows.&lt;/p>
&lt;p>&lt;strong>Astro supports the KubernetesExecutor natively&lt;/strong>, meaning you get per-task Docker images, custom resource limits, and the thin orchestrator pattern described below — without managing Kubernetes yourself. MWAA is locked to CeleryExecutor with no workaround.&lt;/p>
&lt;p>Campspot, which migrated from MWAA to Astro, reported that a critical nightly job went from over two hours to two to three minutes. Another company, Black Crow AI, recouped roughly 20% more engineering time after leaving MWAA, citing zombie task issues and the absence of Airflow-specific support.&lt;/p>
&lt;p>Astro&amp;rsquo;s &lt;strong>Observe&lt;/strong> feature provides built-in pipeline lineage, SLA tracking, and data quality monitoring — capabilities absent from MWAA. Support comes from actual Airflow core committers, versus MWAA&amp;rsquo;s generic AWS support channels.&lt;/p>
&lt;p>The migration story from MWAA to Astro is worth paying attention to because it reveals what teams actually struggle with. Campspot completed their migration in a two-week sprint — their DAGs transferred with minimal modification because both platforms run Airflow. The performance improvement came not from different code, but from better infrastructure: KubernetesExecutor allowing parallel task execution in isolated pods versus CeleryExecutor bottlenecking through shared workers. Black Crow AI&amp;rsquo;s experience was similar — the DAGs were the same, but the operational layer (scaling, monitoring, deployment speed) was categorically better.&lt;/p>
&lt;p>That said, Astronomer is a startup, not AWS. If your organisation&amp;rsquo;s procurement process requires a vendor that&amp;rsquo;s been around for 20+ years, has SOC 2 Type II (Astronomer does have this, to be fair), and won&amp;rsquo;t disappear if funding dries up — that&amp;rsquo;s a valid concern. Astronomer raised significant funding and is growing, but the risk calculus is different from buying from AWS.&lt;/p>
&lt;p>The trade-offs: Astro&amp;rsquo;s dedicated clusters cost $2.40/hr ($1,782/month at the cluster level), and networking costs (NAT, PrivateLink) are passed through from your cloud provider. For teams deeply embedded in AWS wanting consolidated billing and native IAM integration without multi-cloud requirements, MWAA&amp;rsquo;s tight ecosystem coupling has genuine value. But if developer experience and operational flexibility are priorities, Astro is meaningfully ahead.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="head-to-head-dagster-cloud">Head-to-Head: Dagster Cloud&lt;/h3>
&lt;br>
&lt;p>Dagster represents the most fundamental philosophical departure from the Airflow model — and by extension, from MWAA.&lt;/p>
&lt;p>Where Airflow (and MWAA) asks &amp;ldquo;which tasks ran successfully?&amp;rdquo;, Dagster asks &amp;ldquo;which data assets are fresh, and why?&amp;rdquo; The difference sounds academic until you&amp;rsquo;re debugging a pipeline at 7am and you want to know not just which task failed, but which downstream datasets are now stale and which business reports can&amp;rsquo;t be trusted.&lt;/p>
&lt;p>&lt;strong>Software-Defined Assets&lt;/strong> are Dagster&amp;rsquo;s core abstraction. Each asset declares its dependencies, its materialisation logic, and its freshness expectations. The framework automatically builds a dependency graph, tracks lineage at the column level, and can tell you in real time which assets are stale, which are being materialised, and which are healthy. This is observability that MWAA simply can&amp;rsquo;t match — you&amp;rsquo;d need to bolt on separate lineage tools, metadata platforms, and custom monitoring to approximate what Dagster provides natively.&lt;/p>
&lt;p>The developer experience is excellent. &lt;code>dagster dev&lt;/code> starts a local instance immediately without Docker. Branch deployments create isolated staging environments from pull requests — push a feature branch, get a dedicated Dagster instance for testing. dbt integration maps models directly into the asset graph, so your dbt transformations are first-class citizens alongside your Python pipelines.&lt;/p>
&lt;br>
&lt;p>Dagster+ pricing uses a credit-based model: one credit equals one asset materialisation. The Solo tier starts at $10/month, Starter at $100/month. For infrequent batch workloads under 30,000 materialisations per month, Dagster is dramatically cheaper than MWAA. But credits can scale unpredictably — one practitioner calculated that an 8-operation job running every five minutes consumed 69,120 credits per month, costing $2,464 at Solo rates. For high-frequency, multi-step jobs, MWAA&amp;rsquo;s predictable hourly pricing may actually be more economical.&lt;/p>
&lt;p>The ecosystem trade-off is real. Airflow has 1,600+ operators and a community of 80,000+ organisations built over a decade. Dagster&amp;rsquo;s integration library is growing rapidly but isn&amp;rsquo;t as broad. If you need a provider for an obscure SaaS API or legacy system, Airflow almost certainly has one; Dagster might not. For teams starting fresh without Airflow baggage, Dagster&amp;rsquo;s asset-centric model is compelling. For teams with hundreds of existing DAGs, the migration cost is substantial.&lt;/p>
&lt;p>There&amp;rsquo;s also a philosophical difference worth naming. Airflow thinks in terms of &lt;em>schedules and tasks&lt;/em> — &amp;ldquo;run this DAG at 6am, execute these tasks in order.&amp;rdquo; Dagster thinks in terms of &lt;em>data freshness and assets&lt;/em> — &amp;ldquo;these datasets should be no more than 2 hours stale, materialise them as needed.&amp;rdquo; Both models work, but they lead to fundamentally different approaches to monitoring, alerting, and debugging. If your team&amp;rsquo;s primary question is &amp;ldquo;did the 6am job run?&amp;rdquo; then Airflow&amp;rsquo;s model fits naturally. If your question is &amp;ldquo;is the executive dashboard showing current data, and if not, what&amp;rsquo;s stale?&amp;rdquo; then Dagster&amp;rsquo;s model is more direct.&lt;/p>
&lt;p>The deployment model also differs meaningfully. Dagster+ offers both &lt;strong>Serverless&lt;/strong> (fully managed, AWS-hosted) and &lt;strong>Hybrid&lt;/strong> (Dagster manages the control plane, your infrastructure runs the compute). The Hybrid model gives you the thin orchestrator pattern with professional management of the scheduling layer — a compelling middle ground between fully managed and fully self-hosted.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="head-to-head-prefect-cloud">Head-to-Head: Prefect Cloud&lt;/h3>
&lt;br>
&lt;p>Prefect takes the most Python-native approach to orchestration. Add &lt;code>@flow&lt;/code> and &lt;code>@task&lt;/code> decorators to regular Python functions. No DSL, no custom operators, no XCom for passing data between tasks — just Python.&lt;/p>
&lt;p>This matters most for teams where the data engineers are primarily Python developers who find Airflow&amp;rsquo;s operator model cumbersome. Dynamic workflows, event-driven triggers, and runtime conditional logic are all first-class features in Prefect. Need to loop over a list of files and process each one as a separate task? That&amp;rsquo;s a Python for-loop, not an Airflow dynamic task mapping exercise.&lt;/p>
&lt;p>Prefect Cloud&amp;rsquo;s free Hobby tier (2 users, 5 deployments) lets small teams start at $0 — versus MWAA&amp;rsquo;s $365+ minimum. The Starter tier at $100/month includes bring-your-own-compute, meaning you still pay separately for the infrastructure where your flows actually run. Prefect Cloud is a control plane, not a compute platform — similar to the thin orchestrator pattern, but with Prefect managing the scheduling and monitoring layer instead of Airflow.&lt;/p>
&lt;p>The honest limitation: Prefect&amp;rsquo;s community is a fraction of Airflow&amp;rsquo;s. Airflow pulls 30 million monthly downloads; Prefect sits around 1.8 million weekly. That gap means fewer Stack Overflow answers, fewer blog posts solving your exact problem, and fewer engineers who already know the tool when you&amp;rsquo;re hiring. For teams where hiring velocity matters, Airflow&amp;rsquo;s ubiquity is a genuine competitive advantage — and MWAA inherits that.&lt;/p>
&lt;p>Where Prefect genuinely shines is the rapid-prototyping-to-production pipeline. You can convert an existing Python script into an orchestrated workflow by adding decorators — no rewriting into operator patterns, no separating logic from configuration, no learning a new abstraction layer. One engineering team compared this directly to their Airflow experience and found that Prefect flows took roughly half the time to develop and deploy. The trade-off is less structure — Airflow&amp;rsquo;s opinionated DAG model forces a discipline that Prefect&amp;rsquo;s flexibility doesn&amp;rsquo;t enforce.&lt;/p>
&lt;p>Prefect Cloud&amp;rsquo;s Pro tier ($500/month) adds RBAC, audit logs, and custom retention policies. The Enterprise tier (custom pricing) adds SSO, dedicated infrastructure, and priority support. For context, an MWAA mw1.small environment costs $470–540/month all-in — roughly equivalent to Prefect Pro before you add your own compute infrastructure. The total cost comparison depends entirely on how much compute your flows need and where it runs.&lt;/p>
&lt;p>One more thing worth noting: Prefect 2 (the current generation) was a ground-up rewrite that broke compatibility with Prefect 1. If you&amp;rsquo;re evaluating Prefect, you&amp;rsquo;re evaluating a framework that made the decision to start over once already. That took conviction, and the result is a significantly better product — but it&amp;rsquo;s worth understanding that the ecosystem is younger than the company&amp;rsquo;s founding date suggests.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="head-to-head-aws-step-functions">Head-to-Head: AWS Step Functions&lt;/h3>
&lt;br>
&lt;p>Step Functions is the right answer more often than most Airflow advocates want to admit.&lt;/p>
&lt;p>For a 10-step workflow running once daily (~300 state transitions per month), Step Functions costs nothing within the free tier. MWAA&amp;rsquo;s minimum is $365+. Even at medium scale — 100 executions per day — costs stay under $1/month for standard workflows. The Distributed Map mode can process up to 10,000 parallel S3 objects; one AWS demo processed 560,000 CSV files in 100 seconds.&lt;/p>
&lt;p>Step Functions also runs natively serverless with zero infrastructure management. No VPC, no NAT gateways, no &lt;code>requirements.txt&lt;/code> crashes, no 20-minute environment updates. For teams building AWS-native event-driven architectures — S3 events triggering Glue jobs, Lambda processing, DynamoDB writes — Step Functions integrates with over 220 AWS services directly in the workflow definition.&lt;/p>
&lt;p>The limitations are real though. The 256KB payload limit between states constrains data-heavy handoffs. JSON-based Amazon States Language for workflow definitions is verbose and hard to debug. There&amp;rsquo;s no built-in scheduling (you need EventBridge), no backfill capability, no equivalent of Airflow&amp;rsquo;s &lt;code>catchup=True&lt;/code> for replaying historical runs. The 25,000-event history limit per standard execution bites data engineering workloads — though Express Workflows (5-minute maximum duration, $1 per million requests) eliminate that constraint for short-lived jobs.&lt;/p>
&lt;p>For complex data pipelines with interdependencies, branching logic, retries, and backfill requirements, MWAA is genuinely the better tool. For linear workflows, event-driven processing, and microservice orchestration, Step Functions wins on cost, simplicity, and operational overhead.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="beyond-named-platforms-a-pattern-worth-considering">Beyond Named Platforms: A Pattern Worth Considering&lt;/h3>
&lt;br>
&lt;p>Beyond named platforms, there&amp;rsquo;s an architectural pattern that deserves consideration alongside them — one that doesn&amp;rsquo;t come with a vendor attached.&lt;/p>
&lt;p>The idea is simple: &lt;strong>run Airflow as a lightweight scheduler and UI on minimal infrastructure, and delegate all actual compute to Kubernetes pods or ECS tasks that spin up on demand.&lt;/strong>&lt;/p>
&lt;p>In this model, your Airflow installation is deliberately &amp;ldquo;dumb.&amp;rdquo; A small EC2 instance or a modest EKS deployment runs just the webserver, scheduler, and metadata database. Workers don&amp;rsquo;t exist in the traditional sense. Instead, every task in your DAG uses the &lt;strong>KubernetesExecutor&lt;/strong> (or &lt;code>EcsRunTaskOperator&lt;/code>) to launch a purpose-built container that does the actual work, then terminates.&lt;/p>
&lt;p>Here&amp;rsquo;s why this matters:&lt;/p>
&lt;p>&lt;strong>Per-task isolation.&lt;/strong> Each task runs in its own container with its own Docker image, its own dependencies, its own resource limits. Your dbt task runs a slim Python image with just dbt-core. Your Spark submission task runs an image with PySpark and your JAR files. Your ML training task gets a GPU-enabled image with PyTorch. No dependency conflicts. No shared memory pressure. No crash-loop risk from a bad &lt;code>requirements.txt&lt;/code> — because each task manages its own dependencies independently.&lt;/p>
&lt;p>&lt;strong>Scale to zero.&lt;/strong> When nothing is running, you&amp;rsquo;re paying for the scheduler and webserver — maybe $50–80/month on a small EC2 instance or ECS task. When a hundred tasks fire simultaneously, Kubernetes spins up a hundred pods, each with exactly the resources that task needs. When they finish, those pods terminate and you stop paying. MWAA, by contrast, charges for workers 24/7 regardless of whether tasks are running.&lt;/p>
&lt;p>&lt;strong>No CeleryExecutor limitations.&lt;/strong> The KubernetesExecutor gives you everything MWAA&amp;rsquo;s CeleryExecutor can&amp;rsquo;t: custom Docker images per task, no 12-hour execution limit (Kubernetes pods can run indefinitely), pod-level resource requests and limits, and the ability to use Spot instances for worker nodes at 50–70% cost savings.&lt;/p>
&lt;p>&lt;strong>The cold-start trade-off.&lt;/strong> Spinning up a Kubernetes pod takes 15–30 seconds. For batch workloads that run hourly or daily, this is negligible. For real-time or sub-minute scheduling, it&amp;rsquo;s not ideal — but those workloads probably shouldn&amp;rsquo;t be in Airflow anyway.&lt;/p>
&lt;p>&lt;strong>What does this actually look like in practice?&lt;/strong> Your DAG file stays clean and declarative. Instead of using &lt;code>PythonOperator&lt;/code> to run transformations inside the Airflow worker, you use &lt;code>KubernetesPodOperator&lt;/code> pointing at a Docker image that contains your transformation logic and its dependencies:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>transform_sales &lt;span style="color:#f92672">=&lt;/span> KubernetesPodOperator(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> task_id&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;transform_sales_data&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> image&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;your-ecr-repo/sales-transform:v2.3&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;airflow-tasks&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> resources&lt;span style="color:#f92672">=&lt;/span>{&lt;span style="color:#e6db74">&amp;#34;request_cpu&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;2&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;request_memory&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;4Gi&amp;#34;&lt;/span>},
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> node_selector&lt;span style="color:#f92672">=&lt;/span>{&lt;span style="color:#e6db74">&amp;#34;lifecycle&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;spot&amp;#34;&lt;/span>}, &lt;span style="color:#75715e"># 50-70% cost savings&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> is_delete_operator_pod&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Each task declares exactly what it needs. Your dbt task runs a slim image. Your Spark submission gets a heavier one. Your ML training task gets GPU resources. And because each pod is ephemeral, a dependency conflict in one task can never affect another — the isolation is complete.&lt;/p>
&lt;br>
&lt;p>So is this just a slower version of what MWAA does? No — it&amp;rsquo;s architecturally different in important ways.&lt;/p>
&lt;p>MWAA uses CeleryExecutor with always-on Fargate workers. Tasks run inside those workers, sharing the same Python environment, the same dependencies, the same memory. The workers exist whether tasks are queued or not. When you update &lt;code>requirements.txt&lt;/code>, every worker gets rebuilt (hence the 20–40 minute update times).&lt;/p>
&lt;p>The thin orchestrator pattern puts compute &lt;em>outside&lt;/em> the scheduler entirely. Airflow becomes a pure control plane. The data plane — where actual work happens — is ephemeral and independently deployable. You can update a task&amp;rsquo;s Docker image in seconds without touching the Airflow environment at all.&lt;/p>
&lt;p>The self-hosted overhead is real though. You&amp;rsquo;re managing the Airflow installation yourself: upgrades, database administration (or using Amazon RDS for the metadata database), security patching, and monitoring setup. Budget 0.25–0.5 FTE for a small-to-medium deployment. At $150K/year fully loaded, even 0.25 FTE equals $3,125/month in engineering time — which can exceed MWAA&amp;rsquo;s premium for teams that don&amp;rsquo;t already have Kubernetes expertise.&lt;/p>
&lt;p>This pattern works best when your team already runs Kubernetes for other workloads (so the infrastructure cost is shared), you need per-task dependency isolation, or you&amp;rsquo;re running compute-heavy tasks where Spot instance pricing makes a material difference.&lt;/p>
&lt;p>One team running 80 pipelines on self-hosted ECS reported $400–500/month total compute costs. A comparable MWAA setup would run $600–900/month. But the engineering time to maintain the self-hosted setup is the real cost comparison — and it varies enormously depending on team capability.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="when-mwaa-is--and-isnt--the-right-choice">When MWAA Is — and Isn&amp;rsquo;t — the Right Choice&lt;/h3>
&lt;br>
&lt;p>MWAA earns its place when three conditions align: &lt;strong>deep AWS commitment&lt;/strong> (your stack is S3, Redshift, Glue, EMR), &lt;strong>existing Airflow investment&lt;/strong> (DAGs, team expertise, operator familiarity), and a &lt;strong>preference for managed infrastructure&lt;/strong> over platform engineering. Teams running 50–500 DAGs on AWS with two or more data engineers and no multi-cloud requirements will find MWAA productive and cost-effective relative to the operational burden of self-hosting.&lt;/p>
&lt;p>Here&amp;rsquo;s where it falls short:&lt;/p>
&lt;p>&lt;strong>Intermittent or light workloads&lt;/strong> — unless you&amp;rsquo;re willing to accept MWAA Serverless&amp;rsquo;s operator restrictions, the always-on cost floor is too high. Step Functions or Prefect&amp;rsquo;s free tier are dramatically cheaper.&lt;/p>
&lt;p>&lt;strong>Multi-cloud requirements&lt;/strong> — MWAA is AWS-only. Astronomer or self-hosted Airflow provides portability across clouds.&lt;/p>
&lt;p>&lt;strong>Asset-centric observability&lt;/strong> — if data lineage, freshness tracking, and column-level quality monitoring are priorities, Dagster Cloud is purpose-built for this. Bolting these capabilities onto MWAA requires multiple additional tools and significant integration work.&lt;/p>
&lt;p>&lt;strong>KubernetesExecutor or per-task isolation&lt;/strong> — MWAA is locked to CeleryExecutor. Teams needing custom Docker images per task or pod-level resource isolation should look at Astronomer, the thin orchestrator pattern on self-hosted, or Dagster Cloud.&lt;/p>
&lt;p>&lt;strong>Rapid Airflow version adoption&lt;/strong> — MWAA lags upstream releases by 2–6 months. Astronomer typically offers day-zero support for new versions.&lt;/p>
&lt;p>&lt;strong>Cost sensitivity at scale with multiple environments&lt;/strong> — per-team MWAA environments compound quickly. Three teams each needing their own environment is $1,100+/month before any work happens. Astronomer&amp;rsquo;s scale-to-zero workers or self-hosted with Spot instances become significantly cheaper at this scale.&lt;/p>
&lt;br>
&lt;p>For teams genuinely evaluating from scratch, here&amp;rsquo;s the landscape at a glance:&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Factor&lt;/th>
 &lt;th>MWAA&lt;/th>
 &lt;th>Astronomer Astro&lt;/th>
 &lt;th>Dagster Cloud&lt;/th>
 &lt;th>Prefect Cloud&lt;/th>
 &lt;th>Step Functions&lt;/th>
 &lt;th>Self-Hosted + K8s&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>Minimum monthly cost&lt;/td>
 &lt;td>~$470&lt;/td>
 &lt;td>~$260 (scale-to-zero)&lt;/td>
 &lt;td>$10&lt;/td>
 &lt;td>$0 (Hobby)&lt;/td>
 &lt;td>$0 (free tier)&lt;/td>
 &lt;td>~$80 (scheduler only)&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Heavy workload cost&lt;/td>
 &lt;td>$1,800–2,300&lt;/td>
 &lt;td>$1,200–2,000&lt;/td>
 &lt;td>Variable (credits)&lt;/td>
 &lt;td>$500+ plus compute&lt;/td>
 &lt;td>&amp;lt;$50&lt;/td>
 &lt;td>$400–600 plus eng. time&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Setup complexity&lt;/td>
 &lt;td>Medium&lt;/td>
 &lt;td>Low&lt;/td>
 &lt;td>Low&lt;/td>
 &lt;td>Low&lt;/td>
 &lt;td>Low&lt;/td>
 &lt;td>High&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Ongoing ops burden&lt;/td>
 &lt;td>Low&lt;/td>
 &lt;td>Low&lt;/td>
 &lt;td>Low&lt;/td>
 &lt;td>Low–Medium&lt;/td>
 &lt;td>None&lt;/td>
 &lt;td>High&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Airflow compatibility&lt;/td>
 &lt;td>Full&lt;/td>
 &lt;td>Full&lt;/td>
 &lt;td>None (different paradigm)&lt;/td>
 &lt;td>None (different paradigm)&lt;/td>
 &lt;td>N/A&lt;/td>
 &lt;td>Full&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>KubernetesExecutor&lt;/td>
 &lt;td>No (Celery only)&lt;/td>
 &lt;td>Yes&lt;/td>
 &lt;td>N/A&lt;/td>
 &lt;td>N/A&lt;/td>
 &lt;td>N/A&lt;/td>
 &lt;td>Yes&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Observability&lt;/td>
 &lt;td>CloudWatch only&lt;/td>
 &lt;td>Built-in Observe&lt;/td>
 &lt;td>Native asset tracking&lt;/td>
 &lt;td>Built-in dashboard&lt;/td>
 &lt;td>CloudWatch/X-Ray&lt;/td>
 &lt;td>Whatever you build&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Multi-cloud&lt;/td>
 &lt;td>No&lt;/td>
 &lt;td>Yes&lt;/td>
 &lt;td>Yes&lt;/td>
 &lt;td>Yes&lt;/td>
 &lt;td>No&lt;/td>
 &lt;td>Yes&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Ecosystem breadth&lt;/td>
 &lt;td>Largest (Airflow)&lt;/td>
 &lt;td>Largest (Airflow)&lt;/td>
 &lt;td>Growing&lt;/td>
 &lt;td>Moderate&lt;/td>
 &lt;td>AWS-native (220+ services)&lt;/td>
 &lt;td>Largest (Airflow)&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>Here&amp;rsquo;s how I&amp;rsquo;d frame the decision:&lt;/p>
&lt;p>&lt;strong>Choose MWAA&lt;/strong> if you&amp;rsquo;re on AWS, your team knows Airflow, you want AWS to handle the scheduler and metadata database, and your workloads are substantial enough to justify the cost floor. It&amp;rsquo;s a solid, mature service that has improved dramatically in 2024–2025.&lt;/p>
&lt;p>&lt;strong>Choose Astronomer Astro&lt;/strong> if developer experience, KubernetesExecutor support, and fast Airflow upgrades matter more than AWS-native billing integration. It&amp;rsquo;s MWAA but better in almost every operational dimension — at a comparable or lower cost for active workloads.&lt;/p>
&lt;p>&lt;strong>Choose Dagster Cloud&lt;/strong> if you&amp;rsquo;re starting fresh, prioritise observability and data quality, and your team is comfortable learning a new paradigm. The asset-centric model is genuinely superior for understanding the health of your data platform.&lt;/p>
&lt;p>&lt;strong>Choose Prefect Cloud&lt;/strong> if your team is Python-first, dislikes Airflow&amp;rsquo;s operator model, and values dynamic workflows with minimal boilerplate. The free tier makes experimentation trivial.&lt;/p>
&lt;p>&lt;strong>Choose the thin orchestrator pattern&lt;/strong> (self-hosted Airflow + Kubernetes/ECS compute) if you already run Kubernetes, need per-task isolation, want maximum cost control, and have the engineering capacity to maintain the Airflow installation.&lt;/p>
&lt;p>&lt;strong>Choose Step Functions&lt;/strong> if your workflows are linear, event-driven, AWS-native, and don&amp;rsquo;t require backfill or complex scheduling. For many data engineering use cases, Step Functions plus EventBridge is simpler, cheaper, and less to maintain than any Airflow deployment.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-20242025-updates-that-changed-the-equation">The 2024–2025 Updates That Changed the Equation&lt;/h3>
&lt;br>
&lt;p>MWAA&amp;rsquo;s pace of development accelerated dramatically in this period, and several updates addressed the community&amp;rsquo;s loudest complaints.&lt;/p>
&lt;p>&lt;strong>Environment class expansion&lt;/strong> in April 2024 (mw1.xlarge and mw1.2xlarge) and November 2024 (mw1.micro) gave teams both the headroom for heavy workloads and the affordable entry point for dev/test that had been missing since launch. The micro class at ~$37/month is particularly valuable — before its introduction, every dev environment cost the same $365+ as production.&lt;/p>
&lt;p>Airflow versions progressed from 2.8 through 2.9 and 2.10, with &lt;strong>Airflow 3.0 landing on MWAA in October 2025&lt;/strong>. The 3.0 release brought a redesigned React UI, event-driven scheduling via Assets (Airflow&amp;rsquo;s answer to Dagster&amp;rsquo;s Software-Defined Assets), a Task SDK with least-privilege execution, and Python 3.12 support. The Asset concept is significant — it means Airflow is converging toward the same asset-centric thinking that Dagster pioneered.&lt;/p>
&lt;p>&lt;strong>MWAA Serverless&lt;/strong> (November 2025) was the most consequential launch — true pay-per-execution pricing that directly addresses the always-on cost complaint. But as mentioned earlier, the limitations (no Airflow UI, YAML-only definitions, no PythonOperator) make it a different product for a different use case, not a cheaper version of MWAA.&lt;/p>
&lt;p>&lt;strong>Graceful updates&lt;/strong> (May 2025) eliminated the other major pain point: environments can now be updated without interrupting running tasks. The SigV4 REST API (October 2024) simplified programmatic access by supporting standard AWS credentials instead of login token management.&lt;/p>
&lt;p>Community sentiment sits at mixed-positive. One engineering team&amp;rsquo;s May 2025 evaluation captured the tension well: Airflow remains the industry standard with a proven track record, but the improvements in Airflow 3 are following features that Dagster and Prefect already have. Apache Airflow pulled 320 million downloads in 2024, dwarfing competitors — but Dagster&amp;rsquo;s growth trajectory and commit activity signal a genuine shift in the market.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="closing-thoughts">Closing Thoughts&lt;/h3>
&lt;br>
&lt;p>The orchestration landscape is converging. Airflow 3.0 adopted asset-aware scheduling — a concept Dagster pioneered. Prefect&amp;rsquo;s event-driven model influenced Airflow&amp;rsquo;s trigger and asset-watcher features. Dagster is building broader operator support. They&amp;rsquo;re all moving toward the same destination from different starting points.&lt;/p>
&lt;p>That convergence means the &amp;ldquo;right&amp;rdquo; choice increasingly depends not on feature parity but on team expertise, existing investment, and cloud strategy. And honestly? The worst decision you can make is choosing MWAA by default because you&amp;rsquo;re on AWS. Run the numbers. Evaluate the developer experience gaps. Try spinning up a Dagster Cloud or Prefect Cloud instance alongside your MWAA evaluation — both have free tiers, and the comparison will be illuminating.&lt;/p>
&lt;p>I think about that first debugging session on the inherited Step Functions setup often. Hours to find something that should have taken minutes. The platform we eventually chose didn&amp;rsquo;t change the underlying compute or the logic of the pipelines — it changed whether we could &lt;em>see&lt;/em> what was happening. Pick the orchestrator that gives your team that visibility. The one they&amp;rsquo;ll trust when something breaks at an inconvenient hour. That&amp;rsquo;s the one that wins.&lt;/p>
&lt;p>&lt;br>&lt;br>&lt;/p></content:encoded><category>Data Engineering</category><category>Platforms &amp; Cloud</category><category>AWS</category><category>MWAA</category><category>Apache Airflow</category><category>Astronomer</category><category>Dagster</category><category>Prefect</category><category>Step Functions</category><category>Orchestration</category><category>Data Pipelines</category></item><item><title>Your Data Platform Costs More Than It Should</title><link>https://ghostinthedata.info/posts/2026/2026-04-25-cost-management/</link><pubDate>Sat, 25 Apr 2026 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-04-25-cost-management/</guid><author>Chris Hillman</author><description>A practical guide to understanding, measuring, and reducing your Snowflake and AWS data platform costs — starting with the habits that actually move the needle.</description><content:encoded>&lt;p>Let me tell you about the moment I stopped treating cloud costs as someone else&amp;rsquo;s problem.&lt;/p>
&lt;p>We were three months into a Snowflake migration. Everything was humming. Pipelines were green, dashboards were fast, the analytics team was happier than I&amp;rsquo;d seen them before. I felt good about the work we&amp;rsquo;d done.&lt;/p>
&lt;p>Then finance forwarded me the invoice.&lt;/p>
&lt;p>The number wasn&amp;rsquo;t catastrophic. But it was significantly higher than what we&amp;rsquo;d budgeted, and when I started digging, I couldn&amp;rsquo;t explain where most of it was going. I knew we had warehouses running. I knew we had pipelines executing. But I couldn&amp;rsquo;t tell you which warehouse was responsible for what cost, which pipelines were the expensive ones, or whether the money was well spent. I had built a platform I was proud of — and I had no idea what it actually cost to operate.&lt;/p>
&lt;p>That&amp;rsquo;s the moment that changed how I think about data engineering. Not because of the dollar amount, but because of the realisation underneath it: &lt;strong>I had built something I couldn&amp;rsquo;t explain to the people paying for it.&lt;/strong> And if I couldn&amp;rsquo;t explain it, I couldn&amp;rsquo;t defend it. And if I couldn&amp;rsquo;t defend it, someone else would make the decisions for me — someone who didn&amp;rsquo;t understand why the platform mattered.&lt;/p>
&lt;br>
&lt;p>I&amp;rsquo;m telling you this because cost management is one of those things that sounds like a finance problem until you experience the consequences firsthand. It&amp;rsquo;s not about being cheap. It&amp;rsquo;s about being intentional. It&amp;rsquo;s about knowing that every credit you spend is buying something valuable — and being able to prove it when someone asks.&lt;/p>
&lt;p>The data engineers who understand their costs don&amp;rsquo;t just save money. They earn trust. They get budget for the projects that matter. They sleep better because they&amp;rsquo;ve eliminated the waste that eventually becomes someone else&amp;rsquo;s excuse to cut headcount or freeze hiring.&lt;/p>
&lt;p>This article is about building that understanding. Not with a vendor&amp;rsquo;s optimisation tool or a consultant&amp;rsquo;s audit — but with the habits, queries, and mental models that let you own your platform&amp;rsquo;s economics from the inside. Everything here is grounded in Snowflake and AWS, with specific code you can run today.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="you-cant-optimise-what-you-cant-see">You can&amp;rsquo;t optimise what you can&amp;rsquo;t see&lt;/h3>
&lt;br>
&lt;p>Before you touch a single warehouse configuration, you need to answer one question: &lt;strong>where is the money going?&lt;/strong>&lt;/p>
&lt;p>Most teams skip this step. They read a blog post about auto-suspend settings, change a few defaults, and call it optimisation. That&amp;rsquo;s like going on a diet by switching to diet soda while eating three pizzas a day. The soda wasn&amp;rsquo;t the problem.&lt;/p>
&lt;p>Here&amp;rsquo;s the query I run first on every Snowflake environment I touch. It tells you which warehouses are consuming the most credits over the last 30 days:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> warehouse_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used) &lt;span style="color:#66d9ef">AS&lt;/span> total_credits,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used) &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#ae81ff">3&lt;/span>.&lt;span style="color:#ae81ff">00&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> estimated_cost_usd, &lt;span style="color:#75715e">-- adjust your credit price
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">DISTINCT&lt;/span> DATE_TRUNC(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, start_time)) &lt;span style="color:#66d9ef">AS&lt;/span> active_days,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROUND(&lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used) &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">DISTINCT&lt;/span> DATE_TRUNC(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, start_time)), &lt;span style="color:#ae81ff">2&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> credits_per_day
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> snowflake.account_usage.warehouse_metering_history
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> start_time &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">30&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> warehouse_name
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> total_credits &lt;span style="color:#66d9ef">DESC&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Run that. Right now. I&amp;rsquo;ll wait.&lt;/p>
&lt;p>If you&amp;rsquo;re like most teams, 70–80% of your credits come from two or three warehouses. That&amp;rsquo;s your starting point. Not everything — just the expensive stuff.&lt;/p>
&lt;br>
&lt;p>Now do the same thing for queries. This one finds your top 20 most expensive queries by bytes scanned:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> query_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> query_text,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> warehouse_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> user_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> total_elapsed_time &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#ae81ff">1000&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> elapsed_seconds,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> bytes_scanned &lt;span style="color:#f92672">/&lt;/span> POWER(&lt;span style="color:#ae81ff">1024&lt;/span>, &lt;span style="color:#ae81ff">3&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> gb_scanned,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> partitions_scanned,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> partitions_total,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROUND(partitions_scanned &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#66d9ef">NULLIF&lt;/span>(partitions_total, &lt;span style="color:#ae81ff">0&lt;/span>) &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#ae81ff">100&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> pct_partitions_scanned
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> snowflake.account_usage.query_history
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> start_time &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">7&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> bytes_scanned &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> bytes_scanned &lt;span style="color:#66d9ef">DESC&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LIMIT&lt;/span> &lt;span style="color:#ae81ff">20&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Pay attention to that &lt;code>pct_partitions_scanned&lt;/code> column. If you see queries scanning 90–100% of a table&amp;rsquo;s partitions, those queries aren&amp;rsquo;t benefiting from clustering or partition pruning. That&amp;rsquo;s where the big wins hide.&lt;/p>
&lt;p>This 15-minute exercise — top warehouses, top queries — tells you more about your cost profile than any dashboard. It&amp;rsquo;s the equivalent of checking your bank statement before creating a budget. Obvious in hindsight. Almost nobody does it.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-idle-tax-is-your-single-biggest-waste">The idle tax is your single biggest waste&lt;/h3>
&lt;br>
&lt;p>Here&amp;rsquo;s what nobody tells you about Snowflake billing: you pay for compute in 60-second minimums. Every time a warehouse resumes from suspension, you&amp;rsquo;re billed for at least one minute — regardless of whether the query takes 2 seconds or 58 seconds.&lt;/p>
&lt;p>This matters more than it sounds. Picture a BI tool like Metabase or Tableau hitting your warehouse with 20 small metadata queries over 15 minutes. If your warehouse auto-suspends after 5 minutes (Snowflake&amp;rsquo;s default for many setups), it might suspend and resume multiple times during that window. Each resume triggers another 60-second charge.&lt;/p>
&lt;p>Twenty queries that take 3 seconds each? That&amp;rsquo;s 60 seconds of actual compute. But if the warehouse suspends and resumes 4 times, you&amp;rsquo;re billed for 240 seconds. A 4x overhead.&lt;/p>
&lt;p>The fix is straightforward but requires thought:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- For transformation warehouses (predictable, bursty workloads)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">ALTER&lt;/span> WAREHOUSE transform_wh
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SET&lt;/span> AUTO_SUSPEND &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">60&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> AUTO_RESUME &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">TRUE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> WAREHOUSE_SIZE &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;MEDIUM&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- For BI/dashboard warehouses (frequent small queries)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">ALTER&lt;/span> WAREHOUSE bi_wh
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SET&lt;/span> AUTO_SUSPEND &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">300&lt;/span> &lt;span style="color:#75715e">-- 5 min keeps it warm between dashboard interactions
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> AUTO_RESUME &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">TRUE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> WAREHOUSE_SIZE &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;SMALL&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- For ad-hoc/analyst warehouses (unpredictable, intermittent)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">ALTER&lt;/span> WAREHOUSE adhoc_wh
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SET&lt;/span> AUTO_SUSPEND &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">60&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> AUTO_RESUME &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">TRUE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> WAREHOUSE_SIZE &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;XSMALL&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The principle: &lt;strong>transformation warehouses should suspend aggressively&lt;/strong> because they run in defined windows with gaps between runs. BI warehouses should stay warm a bit longer because dashboard users generate clusters of queries with short pauses in between. Ad-hoc warehouses should be small and aggressive — analysts can tolerate a 1–2 second resume delay.&lt;/p>
&lt;p>Here&amp;rsquo;s how to find warehouses that are running but idle — the silent cost killer:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> warehouse_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used) &lt;span style="color:#66d9ef">AS&lt;/span> total_credits,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used_compute) &lt;span style="color:#66d9ef">AS&lt;/span> compute_credits,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used_cloud_services) &lt;span style="color:#66d9ef">AS&lt;/span> cloud_credits,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROUND(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (&lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used) &lt;span style="color:#f92672">-&lt;/span> &lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used_compute))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#66d9ef">NULLIF&lt;/span>(&lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used), &lt;span style="color:#ae81ff">0&lt;/span>) &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#ae81ff">100&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ) &lt;span style="color:#66d9ef">AS&lt;/span> pct_idle_cost
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> snowflake.account_usage.warehouse_metering_history
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> start_time &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">30&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> warehouse_name
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">HAVING&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> pct_idle_cost &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">20&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> total_credits &lt;span style="color:#66d9ef">DESC&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>If &lt;code>pct_idle_cost&lt;/code> is above 20% on any warehouse, you&amp;rsquo;re burning money on idle time. Tighten the auto-suspend, or investigate what&amp;rsquo;s keeping the warehouse awake between queries.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-most-expensive-query-is-the-one-that-scans-everything">The most expensive query is the one that scans everything&lt;/h3>
&lt;br>
&lt;p>Snowflake charges you for the compute time your queries consume, and nothing drives compute time like full table scans. The two most common culprits are queries that wrap filter columns in functions, and queries that select more columns than they need.&lt;/p>
&lt;p>Here&amp;rsquo;s what I mean. This query looks innocent:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Expensive: function on the filter column disables pruning
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_id, customer_id, total_amount
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> analytics.fct_orders
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> DATE(order_timestamp) &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;2026-03-01&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>But it&amp;rsquo;s quietly terrible. Wrapping &lt;code>order_timestamp&lt;/code> in &lt;code>DATE()&lt;/code> forces Snowflake to evaluate every row before filtering. The query planner can&amp;rsquo;t use micro-partition metadata to skip irrelevant partitions.&lt;/p>
&lt;p>This version does the same thing, but lets Snowflake prune:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Cheap: range filter on raw column enables pruning
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_id, customer_id, total_amount
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> analytics.fct_orders
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_timestamp &lt;span style="color:#f92672">&amp;gt;=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;2026-03-01&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> order_timestamp &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#e6db74">&amp;#39;2026-03-02&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The difference can be enormous on large tables — I&amp;rsquo;ve seen this single change reduce bytes scanned by 95% on billion-row fact tables.&lt;/p>
&lt;p>The other silent killer is &lt;code>SELECT *&lt;/code> in intermediate transformations. In a dbt project, I often find staging models that look like this:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- stg_orders.sql (before)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> &lt;span style="color:#66d9ef">source&lt;/span>(&lt;span style="color:#e6db74">&amp;#39;raw&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;orders&amp;#39;&lt;/span>) &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> _loaded_at &lt;span style="color:#f92672">&amp;gt;&lt;/span> (&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#66d9ef">MAX&lt;/span>(_loaded_at) &lt;span style="color:#66d9ef">FROM&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> this &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>That &lt;code>SELECT *&lt;/code> pulls every column from the source — including columns nobody downstream ever uses. In a columnar store like Snowflake, you only pay to scan the columns you reference. Trimming to the columns you actually need is free performance:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- stg_orders.sql (after)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_timestamp,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> total_amount,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> status,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> _loaded_at
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> &lt;span style="color:#66d9ef">source&lt;/span>(&lt;span style="color:#e6db74">&amp;#39;raw&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;orders&amp;#39;&lt;/span>) &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> _loaded_at &lt;span style="color:#f92672">&amp;gt;&lt;/span> (&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#66d9ef">MAX&lt;/span>(_loaded_at) &lt;span style="color:#66d9ef">FROM&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> this &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This isn&amp;rsquo;t premature optimisation. It&amp;rsquo;s hygiene. Every &lt;code>SELECT *&lt;/code> in your transformation layer is a small tax you pay on every single run.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="full-refreshes-are-the-most-expensive-default-in-data-engineering">Full refreshes are the most expensive default in data engineering&lt;/h3>
&lt;br>
&lt;p>If you&amp;rsquo;re using dbt — and most teams on Snowflake are — the default materialisation is either &lt;code>view&lt;/code> or &lt;code>table&lt;/code>. Both have the same problem at scale: they rebuild everything, every time.&lt;/p>
&lt;p>A table materialisation on a 500-million-row fact table means Snowflake reads, transforms, and writes 500 million rows every run. Even if only 50,000 rows changed since yesterday. That&amp;rsquo;s a 10,000x overhead.&lt;/p>
&lt;p>Switching to incremental materialisation is the single highest-impact cost change most teams can make:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- fct_orders.sql
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> config(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> materialized&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;incremental&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> unique_key&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;order_id&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> incremental_strategy&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;merge&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> on_schema_change&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;append_new_columns&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_timestamp,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> total_amount,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> status,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> updated_at
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> &lt;span style="color:#66d9ef">ref&lt;/span>(&lt;span style="color:#e6db74">&amp;#39;stg_orders&amp;#39;&lt;/span>) &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#960050;background-color:#1e0010">{&lt;/span>&lt;span style="color:#f92672">%&lt;/span> &lt;span style="color:#66d9ef">if&lt;/span> is_incremental() &lt;span style="color:#f92672">%&lt;/span>&lt;span style="color:#960050;background-color:#1e0010">}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> updated_at &lt;span style="color:#f92672">&amp;gt;&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;hour&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">3&lt;/span>, &lt;span style="color:#66d9ef">MAX&lt;/span>(updated_at))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> this &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#960050;background-color:#1e0010">{&lt;/span>&lt;span style="color:#f92672">%&lt;/span> endif &lt;span style="color:#f92672">%&lt;/span>&lt;span style="color:#960050;background-color:#1e0010">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A few things worth noting here. The &lt;code>DATEADD('hour', -3, ...)&lt;/code> creates a 3-hour lookback window. This catches late-arriving data and handles clock skew between source systems. Without it, you&amp;rsquo;ll miss rows that arrive slightly out of order — and your numbers will silently drift.&lt;/p>
&lt;p>The &lt;code>unique_key&lt;/code> with &lt;code>incremental_strategy='merge'&lt;/code> means Snowflake will update existing rows and insert new ones. This is essential for tables where source records get modified after initial load (order status changes, for example).&lt;/p>
&lt;p>The rule of thumb: &lt;strong>if a table has more than a million rows and less than 20% changes per run, make it incremental.&lt;/strong> The cost savings are usually 80–95% on that model&amp;rsquo;s compute.&lt;/p>
&lt;p>But — and this is important — schedule a periodic full refresh to correct any drift. I typically set up a weekly &lt;code>dbt build --full-refresh --select fct_orders&lt;/code> via a separate Airflow DAG or GitHub Actions workflow. Belt and suspenders.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-hidden-costs-your-snowflake-dashboard-wont-show-you">The hidden costs your Snowflake dashboard won&amp;rsquo;t show you&lt;/h3>
&lt;br>
&lt;p>Most teams monitor warehouse credits because that&amp;rsquo;s what the Snowflake UI makes visible. But there are cost vectors that don&amp;rsquo;t show up in the obvious places.&lt;/p>
&lt;p>&lt;strong>Cloud services credits&lt;/strong> accrue when queries use Snowflake&amp;rsquo;s coordination layer — query compilation, metadata operations, result set caching. Normally this is covered by a 10% &amp;ldquo;free&amp;rdquo; adjustment against your compute credits. But if you have BI tools making thousands of small metadata queries, cloud services can exceed that adjustment and start costing real money.&lt;/p>
&lt;p>Find them:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> DATE_TRUNC(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, usage_date) &lt;span style="color:#66d9ef">AS&lt;/span> &lt;span style="color:#66d9ef">day&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used) &lt;span style="color:#66d9ef">AS&lt;/span> total_credits,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(credits_adjustment_cloud_services) &lt;span style="color:#66d9ef">AS&lt;/span> cloud_services_adjustment,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">SUM&lt;/span>(credits_adjustment_cloud_services) &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#66d9ef">ABS&lt;/span>(&lt;span style="color:#66d9ef">SUM&lt;/span>(credits_adjustment_cloud_services))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> excess_cloud_services_cost
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> snowflake.account_usage.metering_daily_history
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> usage_date &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">30&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">day&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">day&lt;/span> &lt;span style="color:#66d9ef">DESC&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Serverless feature credits&lt;/strong> — Snowpipe, automatic clustering, materialised view maintenance, search optimisation — all consume credits outside your warehouse billing. They don&amp;rsquo;t show up in &lt;code>warehouse_metering_history&lt;/code>. You need to check separately:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Snowpipe costs
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> pipe_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used) &lt;span style="color:#66d9ef">AS&lt;/span> total_credits
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> snowflake.account_usage.pipe_usage_history
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> start_time &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">30&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> pipe_name
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> total_credits &lt;span style="color:#66d9ef">DESC&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Automatic clustering costs
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">table_name&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used) &lt;span style="color:#66d9ef">AS&lt;/span> total_credits
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> snowflake.account_usage.automatic_clustering_history
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> start_time &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">30&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">table_name&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> total_credits &lt;span style="color:#66d9ef">DESC&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>On the AWS side, the sneaky costs tend to be data transfer. Moving data between S3 regions, or from S3 out to the internet, adds up quietly. If your Snowflake account is in &lt;code>us-east-1&lt;/code> but your S3 landing zone is in &lt;code>ap-southeast-2&lt;/code>, every byte of ingestion carries a cross-region transfer charge. Check your AWS Cost Explorer with the &amp;ldquo;Data Transfer&amp;rdquo; service filter — the numbers are often surprising.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="tag-everything-or-youll-optimise-blind">Tag everything or you&amp;rsquo;ll optimise blind&lt;/h3>
&lt;br>
&lt;p>You cannot reduce costs you can&amp;rsquo;t attribute. The simplest and most underused tool in Snowflake is the query tag — a piece of metadata you attach to every query that tells you &lt;em>what&lt;/em> generated it.&lt;/p>
&lt;p>If you&amp;rsquo;re using dbt, this takes about five minutes to set up. Create a macro:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- macros/set_query_tag.sql
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#960050;background-color:#1e0010">{&lt;/span>&lt;span style="color:#f92672">%&lt;/span> macro set_query_tag() &lt;span style="color:#f92672">%&lt;/span>&lt;span style="color:#960050;background-color:#1e0010">}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">{&lt;/span>&lt;span style="color:#f92672">%&lt;/span> &lt;span style="color:#66d9ef">set&lt;/span> query_tag &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">{&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;dbt_model&amp;#34;&lt;/span>: model.name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;dbt_schema&amp;#34;&lt;/span>: model.&lt;span style="color:#66d9ef">schema&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;dbt_materialized&amp;#34;&lt;/span>: model.config.materialized,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;dbt_invocation_id&amp;#34;&lt;/span>: invocation_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;environment&amp;#34;&lt;/span>: target.name
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">}&lt;/span> &lt;span style="color:#f92672">%&lt;/span>&lt;span style="color:#960050;background-color:#1e0010">}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">{&lt;/span>&lt;span style="color:#f92672">%&lt;/span> &lt;span style="color:#66d9ef">do&lt;/span> run_query(&lt;span style="color:#e6db74">&amp;#34;ALTER SESSION SET QUERY_TAG = &amp;#39;{}&amp;#39;&amp;#34;&lt;/span>.format(query_tag &lt;span style="color:#f92672">|&lt;/span> tojson)) &lt;span style="color:#f92672">%&lt;/span>&lt;span style="color:#960050;background-color:#1e0010">}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#960050;background-color:#1e0010">{&lt;/span>&lt;span style="color:#f92672">%&lt;/span> endmacro &lt;span style="color:#f92672">%&lt;/span>&lt;span style="color:#960050;background-color:#1e0010">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Then add it as a pre-hook in your &lt;code>dbt_project.yml&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># dbt_project.yml&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">models&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">your_project&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">+pre-hook&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;{{ set_query_tag() }}&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Now every query dbt runs is tagged with the model name, materialisation type, and environment. You can query cost by model:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> PARSE_JSON(query_tag):dbt_model::STRING &lt;span style="color:#66d9ef">AS&lt;/span> model_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> PARSE_JSON(query_tag):dbt_materialized::STRING &lt;span style="color:#66d9ef">AS&lt;/span> materialization,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> query_count,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(total_elapsed_time) &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#ae81ff">1000&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> total_seconds,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(bytes_scanned) &lt;span style="color:#f92672">/&lt;/span> POWER(&lt;span style="color:#ae81ff">1024&lt;/span>, &lt;span style="color:#ae81ff">4&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> tb_scanned
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> snowflake.account_usage.query_history
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> query_tag &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> TRY_PARSE_JSON(query_tag) &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> start_time &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">7&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> model_name, materialization
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> tb_scanned &lt;span style="color:#66d9ef">DESC&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LIMIT&lt;/span> &lt;span style="color:#ae81ff">20&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This is how you find the one dbt model that&amp;rsquo;s responsible for 40% of your bill. Without tags, it&amp;rsquo;s a guessing game.&lt;/p>
&lt;p>For non-dbt workloads — Airflow tasks, Lambda functions, BI tools — set query tags at the session level in your connection configuration. In Python:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> snowflake.connector
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> json
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>conn &lt;span style="color:#f92672">=&lt;/span> snowflake&lt;span style="color:#f92672">.&lt;/span>connector&lt;span style="color:#f92672">.&lt;/span>connect(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> account&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;your_account&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> user&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;your_user&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> password&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;your_password&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> warehouse&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;transform_wh&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> session_parameters&lt;span style="color:#f92672">=&lt;/span>{
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;QUERY_TAG&amp;#39;&lt;/span>: json&lt;span style="color:#f92672">.&lt;/span>dumps({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;pipeline&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;customer_ingestion&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;task&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;load_raw_customers&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;environment&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;production&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The goal is simple: &lt;strong>every query that runs on your platform should be attributable to a team, a pipeline, or a tool.&lt;/strong> Start with the big consumers and work outward.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="kill-the-zombies">Kill the zombies&lt;/h3>
&lt;br>
&lt;p>Every data platform accumulates dead weight. Tables nobody queries. Pipelines that run faithfully every morning, transforming data that no dashboard, no analyst, and no model has touched in months.&lt;/p>
&lt;p>These zombies cost you twice: once in compute (the pipeline that refreshes them) and once in storage (the data that sits there). Finding them is straightforward:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Tables with zero reads in the last 90 days
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> t.table_schema,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> t.&lt;span style="color:#66d9ef">table_name&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> t.&lt;span style="color:#66d9ef">row_count&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> t.bytes &lt;span style="color:#f92672">/&lt;/span> POWER(&lt;span style="color:#ae81ff">1024&lt;/span>, &lt;span style="color:#ae81ff">3&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> size_gb,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> t.last_altered,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">MAX&lt;/span>(ah.query_start_time) &lt;span style="color:#66d9ef">AS&lt;/span> last_queried
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> snowflake.account_usage.tables t
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">LEFT&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> snowflake.account_usage.access_history ah
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ON&lt;/span> ah.base_objects_accessed &lt;span style="color:#66d9ef">LIKE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;%&amp;#39;&lt;/span> &lt;span style="color:#f92672">||&lt;/span> t.&lt;span style="color:#66d9ef">table_name&lt;/span> &lt;span style="color:#f92672">||&lt;/span> &lt;span style="color:#e6db74">&amp;#39;%&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> ah.query_start_time &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">90&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> t.table_schema &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">IN&lt;/span> (&lt;span style="color:#e6db74">&amp;#39;INFORMATION_SCHEMA&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> t.deleted &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> t.&lt;span style="color:#66d9ef">row_count&lt;/span> &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> t.table_schema, t.&lt;span style="color:#66d9ef">table_name&lt;/span>, t.&lt;span style="color:#66d9ef">row_count&lt;/span>, t.bytes, t.last_altered
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">HAVING&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> last_queried &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> size_gb &lt;span style="color:#66d9ef">DESC&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>On the AWS side, check your S3 storage for orphaned data. Landing zones accumulate raw files that were loaded months ago and never cleaned up. A simple lifecycle policy handles this:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-json" data-lang="json">&lt;span style="display:flex;">&lt;span>{
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Rules&amp;#34;&lt;/span>: [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;ID&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Archive raw data after 90 days&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Status&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Enabled&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Filter&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Prefix&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;raw-landing/&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Transitions&amp;#34;&lt;/span>: [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Days&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">90&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;StorageClass&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;GLACIER_INSTANT_RETRIEVAL&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Expiration&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Days&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">365&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The cultural shift matters as much as the tooling. Make it a quarterly habit: pull the zombie table list, review it with the team, and deprecate what nobody uses. If someone screams, you can always restore from Time Travel. But in my experience, nobody screams. The data was already dead — you&amp;rsquo;re just acknowledging it.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="cost-anomalies-will-find-you--or-you-can-find-them-first">Cost anomalies will find you — or you can find them first&lt;/h3>
&lt;br>
&lt;p>The scariest cost event isn&amp;rsquo;t the gradual creep. It&amp;rsquo;s the single bad query or misconfigured pipeline that doubles your weekly bill overnight. I&amp;rsquo;ve seen a single Snowflake query with a missing WHERE clause scan an entire 2TB table repeatedly inside a loop — burning through hundreds of credits in an hour.&lt;/p>
&lt;p>On Snowflake, set up resource monitors as a basic guardrail:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create a resource monitor with alerts and hard stop
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">REPLACE&lt;/span> RESOURCE MONITOR monthly_budget
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WITH&lt;/span> CREDIT_QUOTA &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">5000&lt;/span> &lt;span style="color:#75715e">-- adjust to your monthly budget
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> FREQUENCY &lt;span style="color:#f92672">=&lt;/span> MONTHLY
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> START_TIMESTAMP &lt;span style="color:#f92672">=&lt;/span> IMMEDIATELY
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> TRIGGERS
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ON&lt;/span> &lt;span style="color:#ae81ff">75&lt;/span> PERCENT &lt;span style="color:#66d9ef">DO&lt;/span> &lt;span style="color:#66d9ef">NOTIFY&lt;/span> &lt;span style="color:#75715e">-- email alert at 75%
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">ON&lt;/span> &lt;span style="color:#ae81ff">90&lt;/span> PERCENT &lt;span style="color:#66d9ef">DO&lt;/span> &lt;span style="color:#66d9ef">NOTIFY&lt;/span> &lt;span style="color:#75715e">-- email alert at 90%
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">ON&lt;/span> &lt;span style="color:#ae81ff">100&lt;/span> PERCENT &lt;span style="color:#66d9ef">DO&lt;/span> SUSPEND; &lt;span style="color:#75715e">-- hard stop at 100%
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Apply it to a warehouse
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">ALTER&lt;/span> WAREHOUSE transform_wh
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SET&lt;/span> RESOURCE_MONITOR &lt;span style="color:#f92672">=&lt;/span> monthly_budget;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>On AWS, enable Cost Anomaly Detection in the AWS Cost Management console. It uses ML to detect unusual spending patterns and sends alerts via SNS. For Snowflake-specific monitoring, a lightweight approach is a scheduled task that checks daily credit consumption against a rolling average:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create a simple anomaly detection view
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">REPLACE&lt;/span> &lt;span style="color:#66d9ef">VIEW&lt;/span> monitoring.daily_cost_anomalies &lt;span style="color:#66d9ef">AS&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WITH&lt;/span> daily_usage &lt;span style="color:#66d9ef">AS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> DATE_TRUNC(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, start_time) &lt;span style="color:#66d9ef">AS&lt;/span> usage_date,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> warehouse_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(credits_used) &lt;span style="color:#66d9ef">AS&lt;/span> daily_credits
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> snowflake.account_usage.warehouse_metering_history
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> start_time &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">60&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> usage_date, warehouse_name
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>averages &lt;span style="color:#66d9ef">AS&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> warehouse_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AVG&lt;/span>(daily_credits) &lt;span style="color:#66d9ef">AS&lt;/span> avg_daily_credits,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> STDDEV(daily_credits) &lt;span style="color:#66d9ef">AS&lt;/span> stddev_daily_credits
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> daily_usage
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> usage_date &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> warehouse_name
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> du.usage_date,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> du.warehouse_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> du.daily_credits,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> a.avg_daily_credits,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROUND((du.daily_credits &lt;span style="color:#f92672">-&lt;/span> a.avg_daily_credits) &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#66d9ef">NULLIF&lt;/span>(a.stddev_daily_credits, &lt;span style="color:#ae81ff">0&lt;/span>), &lt;span style="color:#ae81ff">2&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> z_score
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> daily_usage du
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">JOIN&lt;/span> averages a
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ON&lt;/span> du.warehouse_name &lt;span style="color:#f92672">=&lt;/span> a.warehouse_name
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> du.usage_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> (du.daily_credits &lt;span style="color:#f92672">-&lt;/span> a.avg_daily_credits) &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#66d9ef">NULLIF&lt;/span>(a.stddev_daily_credits, &lt;span style="color:#ae81ff">0&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">2&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> z_score &lt;span style="color:#66d9ef">DESC&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A z-score above 2 means today&amp;rsquo;s spend is more than two standard deviations above the 60-day average. That&amp;rsquo;s worth investigating. Pipe this into a Teams or Slack alert via an AWS Lambda function and you&amp;rsquo;ve got same-day cost anomaly detection for the price of a few lines of SQL.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="what-your-data-model-costs-you">What your data model costs you&lt;/h3>
&lt;br>
&lt;p>This one&amp;rsquo;s subtle but powerful. The way you model your data directly affects how much compute you burn on every query.&lt;/p>
&lt;p>In columnar warehouses like Snowflake, wide denormalised tables query faster than star schemas with multiple JOINs — because each JOIN has overhead, and Snowflake is optimised for scanning columns from flat structures. But wide tables cost more to &lt;em>maintain&lt;/em> because updating a single dimension attribute means rewriting many rows in the fact table.&lt;/p>
&lt;p>The practical approach is a layered architecture:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Staging models&lt;/strong> (views or ephemeral): Minimal transformation, column selection, type casting. Cheap to run, no storage cost.&lt;/li>
&lt;li>&lt;strong>Intermediate models&lt;/strong> (tables or incremental): Business logic, deduplication, SCD handling. Materialised because they&amp;rsquo;re referenced by multiple downstream models.&lt;/li>
&lt;li>&lt;strong>Mart models&lt;/strong> (incremental or table): Wide, denormalised, optimised for consumer queries. These are what analysts and BI tools actually hit.&lt;/li>
&lt;/ul>
&lt;p>The cost trap is materialising too much too early. Every table materialisation means Snowflake stores and maintains that data. Every &lt;code>dbt run&lt;/code> rebuilds it. If an intermediate model is only referenced by one downstream model, make it ephemeral or a view — let Snowflake inline it at query time.&lt;/p>
&lt;p>Check your dbt DAG for this pattern: a staging model materialised as a table, referenced by a single intermediate model, which is also materialised as a table, referenced by a single mart. That&amp;rsquo;s three materialisations where one (the mart) would suffice. The staging and intermediate models can be views or ephemeral — the compute happens once when the mart builds, not three separate times.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="coming-back-to-why-this-matters">Coming back to why this matters&lt;/h3>
&lt;br>
&lt;p>I started this article with a story about not being able to explain my own platform&amp;rsquo;s costs. That wasn&amp;rsquo;t a technical failure. It was a leadership failure. I&amp;rsquo;d built something valuable and then neglected the part that kept it funded.&lt;/p>
&lt;p>The data engineers I respect most aren&amp;rsquo;t the ones who build the most elegant pipelines. They&amp;rsquo;re the ones who can walk into a budget conversation and say: &amp;ldquo;Here&amp;rsquo;s what we spend. Here&amp;rsquo;s what we get for it. Here&amp;rsquo;s what I&amp;rsquo;d cut, and here&amp;rsquo;s what I&amp;rsquo;d invest more in.&amp;rdquo; That&amp;rsquo;s the kind of clarity that earns trust — and trust is what keeps data teams alive when the inevitable cost-cutting conversations happen.&lt;/p>
&lt;p>You don&amp;rsquo;t need a FinOps certification or an expensive monitoring tool to get there. You need the queries in this article, a quarterly habit of reviewing them, and the willingness to kill the zombies nobody wants to admit are dead.&lt;/p>
&lt;p>Start with the 15-minute audit. Tag your queries. Set up a resource monitor. Do those three things this week and you&amp;rsquo;ll know more about your platform&amp;rsquo;s economics than most data teams learn in a year.&lt;/p>
&lt;p>The cheapest query is the one you never need to run. But the most valuable skill is knowing which queries are worth paying for.&lt;/p>
&lt;p>&lt;br>&lt;br>&lt;/p></content:encoded><category>Data Engineering</category><category>Platforms &amp; Cloud</category><category>Snowflake</category><category>AWS</category><category>Cost Optimisation</category><category>FinOps</category><category>dbt</category><category>Data Platform</category></item><item><title>Why Your Pipeline Finishes Later Every Month</title><link>https://ghostinthedata.info/posts/2026/2026-04-18-pipeline-optimization/</link><pubDate>Sat, 18 Apr 2026 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-04-18-pipeline-optimization/</guid><author>Chris Hillman</author><description>A practical guide to diagnosing pipeline bottlenecks, fixing unnecessary dependencies, and getting data to consumers faster — with Snowflake and AWS patterns you can apply today.</description><content:encoded>&lt;p>Let me tell you about a graph that changed how I think about data engineering.&lt;/p>
&lt;p>A junior engineer on my team — let&amp;rsquo;s call her Priya — had been tracking something nobody asked her to track. Every morning for two months, she&amp;rsquo;d noted the timestamp when our main analytics pipeline completed. She wasn&amp;rsquo;t trying to make a point. She was just curious, because the finance team kept mentioning their dashboards weren&amp;rsquo;t ready when they arrived at 8 AM anymore.&lt;/p>
&lt;p>One afternoon she pulled me aside and showed me a scatter plot on her laptop. Pipeline completion time, plotted daily over several months. The trend was unmistakable: a slow, steady drift to the right. What used to finish at 5:47 AM was now finishing at 7:23 AM. And the slope wasn&amp;rsquo;t flattening.&lt;/p>
&lt;p>&amp;ldquo;If this keeps going,&amp;rdquo; she said, &amp;ldquo;we&amp;rsquo;ll miss the 9 AM SLA in about six weeks.&amp;rdquo;&lt;/p>
&lt;p>She was right. And nobody else on the team — including me — had noticed. We were watching for failures. Green DAGs, clean logs, no alerts. But the pipeline wasn&amp;rsquo;t failing. It was &lt;em>slowing&lt;/em>. And slow is harder to see than broken, because slow doesn&amp;rsquo;t trigger an alert. Slow just quietly erodes trust until one day someone in finance builds their own spreadsheet and stops asking you for anything.&lt;/p>
&lt;br>
&lt;p>That&amp;rsquo;s the moment I understood that pipeline health isn&amp;rsquo;t about pass/fail. It&amp;rsquo;s about &lt;em>trajectory&lt;/em>. A pipeline that runs successfully but takes 5% longer every month is a ticking clock. And the people who notice first aren&amp;rsquo;t the engineers watching the DAG — they&amp;rsquo;re the consumers waiting for their data.&lt;/p>
&lt;p>I&amp;rsquo;m telling you this because pipeline optimisation sounds like a performance engineering problem, and it is. But underneath the technical work, it&amp;rsquo;s really about a commitment: the commitment to deliver data when you said you would, every single day. That&amp;rsquo;s what builds trust between data teams and the rest of the organisation. Not fancy architectures. Not real-time everything. Just showing up on time, reliably.&lt;/p>
&lt;p>This article is about diagnosing why your pipelines get slower, identifying the bottlenecks that actually matter, and fixing the patterns that cause data to arrive later than it should. Everything is grounded in Snowflake, AWS, Airflow, and dbt — with specific patterns you can apply immediately.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="measure-the-pipeline-in-stages-not-as-a-single-number">Measure the pipeline in stages, not as a single number&lt;/h3>
&lt;br>
&lt;p>When a pipeline is slow, the instinct is to look at the longest-running task. That&amp;rsquo;s often the wrong place to start.&lt;/p>
&lt;p>A pipeline is a chain of stages: extract, transform, load, test. The total runtime is the sum of all stages on the critical path — the longest chain of dependent tasks. But the &lt;em>bottleneck&lt;/em> might not be the longest task. It might be a 30-second task that blocks five parallel branches from starting.&lt;/p>
&lt;p>Before you optimise anything, instrument your pipeline to measure each stage independently. In Airflow, the task instance metadata already captures this.&lt;/p>
&lt;p>Two numbers matter here: &lt;code>duration_seconds&lt;/code> (how long the task actually ran) and &lt;code>queue_wait_seconds&lt;/code> (how long it waited before it could run). If queue wait is high, your problem isn&amp;rsquo;t the task — it&amp;rsquo;s resource contention. Too many tasks competing for too few Airflow worker slots, or too many queries competing for the same Snowflake warehouse.&lt;/p>
&lt;p>For dbt runs specifically, the &lt;code>run_results.json&lt;/code> that dbt generates after every invocation is a goldmine:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> json
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">with&lt;/span> open(&lt;span style="color:#e6db74">&amp;#39;target/run_results.json&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> results &lt;span style="color:#f92672">=&lt;/span> json&lt;span style="color:#f92672">.&lt;/span>load(f)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Find your slowest models&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>models &lt;span style="color:#f92672">=&lt;/span> sorted(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> [r &lt;span style="color:#66d9ef">for&lt;/span> r &lt;span style="color:#f92672">in&lt;/span> results[&lt;span style="color:#e6db74">&amp;#39;results&amp;#39;&lt;/span>] &lt;span style="color:#66d9ef">if&lt;/span> r[&lt;span style="color:#e6db74">&amp;#39;status&amp;#39;&lt;/span>] &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#39;success&amp;#39;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> key&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">lambda&lt;/span> x: x[&lt;span style="color:#e6db74">&amp;#39;execution_time&amp;#39;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> reverse&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">for&lt;/span> m &lt;span style="color:#f92672">in&lt;/span> models[:&lt;span style="color:#ae81ff">10&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>m[&lt;span style="color:#e6db74">&amp;#39;unique_id&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">60s&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>m[&lt;span style="color:#e6db74">&amp;#39;execution_time&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">8.1f&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">s&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Run that after your next &lt;code>dbt build&lt;/code>. You&amp;rsquo;ll immediately see which models dominate your pipeline runtime. In my experience, 3–5 models account for 60–80% of total execution time. Those are the only models worth optimising.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="your-dag-probably-has-dependencies-it-doesnt-need">Your DAG probably has dependencies it doesn&amp;rsquo;t need&lt;/h3>
&lt;br>
&lt;p>This is the most common structural problem in data pipelines, and it comes in two forms. One is obvious once you know to look for it. The other is sneaky, because it starts as a completely reasonable engineering decision.&lt;/p>
&lt;p>&lt;strong>The first form: phantom dependencies.&lt;/strong>&lt;/p>
&lt;p>An engineer builds Model C and declares it depends on Model B — because Model B produces a table that Model C reads. Fair enough. But six months later, someone refactors Model C and it no longer reads from Model B. The &lt;code>ref()&lt;/code> gets removed from the SQL. But the dependency in the Airflow DAG? That stays. Or in dbt, someone adds a &lt;code>ref()&lt;/code> to a model they don&amp;rsquo;t actually need, just to &amp;ldquo;make sure it runs first.&amp;rdquo;&lt;/p>
&lt;p>The result is a DAG with phantom dependencies — tasks that wait for other tasks to complete even though they don&amp;rsquo;t use those tasks&amp;rsquo; outputs. Every phantom dependency adds serial wait time to your pipeline.&lt;/p>
&lt;p>&lt;strong>The second form: dependency monsters.&lt;/strong>&lt;/p>
&lt;p>This one is trickier, because it starts with good intentions.&lt;/p>
&lt;p>The customer team needs three new enrichment attributes on &lt;code>dim_customers&lt;/code> — regional segment codes sourced from a CRM export, tenure tier derived from a subscription history table, and a propensity score from a data science model. All reasonable requests. Each one approved and added without much ceremony.&lt;/p>
&lt;p>But &lt;code>dim_customers&lt;/code> is upstream of 40+ models across your DAG. Adding those three attributes means pulling in the CRM extract, joining the subscription history table — which has its own upstream dependencies — and waiting on the propensity score model to complete before any of those 40 downstream models can start. Your pipeline used to finish at 9 AM. Eighteen months and a dozen enrichment requests later, it finishes at 3 PM.&lt;/p>
&lt;p>Each addition was individually justified. Nobody modelled what they&amp;rsquo;d cost collectively. That&amp;rsquo;s how you build a dependency monster.&lt;/p>
&lt;p>The fix isn&amp;rsquo;t to refuse enrichment requests — it&amp;rsquo;s to stop embedding enrichment into your core spine model. Keep &lt;code>dim_customers&lt;/code> lean: identifiers, names, status, the attributes that nearly every consumer genuinely needs. Build a separate &lt;code>dim_customers_extended&lt;/code> model that joins in the expensive enrichment for the consumers who actually need it. Most of your downstream models will never touch the propensity score. There&amp;rsquo;s no reason to make them wait for it.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- dim_customers.sql: lean spine — runs fast, unblocks the DAG
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> email,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> status,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> created_at
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> &lt;span style="color:#66d9ef">ref&lt;/span>(&lt;span style="color:#e6db74">&amp;#39;stg_customers&amp;#39;&lt;/span>) &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- dim_customers_extended.sql: enrichment for consumers who need it
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- runs independently, doesn&amp;#39;t block the main pipeline
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">c&lt;/span>.customer_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">c&lt;/span>.customer_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">c&lt;/span>.status,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> crm.regional_segment,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sub.tenure_tier,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ps.propensity_score
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> &lt;span style="color:#66d9ef">ref&lt;/span>(&lt;span style="color:#e6db74">&amp;#39;dim_customers&amp;#39;&lt;/span>) &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span> &lt;span style="color:#66d9ef">c&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">LEFT&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> &lt;span style="color:#66d9ef">ref&lt;/span>(&lt;span style="color:#e6db74">&amp;#39;stg_crm_segments&amp;#39;&lt;/span>) &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span> crm
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">on&lt;/span> crm.customer_id &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">c&lt;/span>.customer_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">LEFT&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> &lt;span style="color:#66d9ef">ref&lt;/span>(&lt;span style="color:#e6db74">&amp;#39;int_subscription_tenure&amp;#39;&lt;/span>) &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span> sub
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">on&lt;/span> sub.customer_id &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">c&lt;/span>.customer_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">LEFT&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> &lt;span style="color:#66d9ef">ref&lt;/span>(&lt;span style="color:#e6db74">&amp;#39;ml_propensity_scores&amp;#39;&lt;/span>) &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span> ps
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">on&lt;/span> ps.customer_id &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">c&lt;/span>.customer_id
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Now &lt;code>dim_customers_extended&lt;/code> and its expensive upstream dependencies sit on their own branch of the DAG. Consumers who need the enrichment reference the extended model. The rest of the pipeline is unblocked.&lt;/p>
&lt;p>A useful governance rule of thumb before adding a field to a core model: if fewer than half your consumers will ever query that attribute, it probably doesn&amp;rsquo;t belong there. The team who needs it should own the enrichment themselves, as a downstream model they maintain.&lt;/p>
&lt;br>
&lt;p>&lt;strong>Auditing for both problems&lt;/strong>&lt;/p>
&lt;p>In dbt, the &lt;code>dbt_project_evaluator&lt;/code> package surfaces structural issues systematically. Install it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># packages.yml&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">packages&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">package&lt;/span>: &lt;span style="color:#ae81ff">dbt-labs/dbt_project_evaluator&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">version&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;&amp;gt;=0.8.0 &amp;lt;1.0.0&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Then run:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>dbt build --select package:dbt_project_evaluator
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>It will flag models with fan-out issues (one model referenced by too many downstream models), unnecessary dependencies, and models that could be parallelised but aren&amp;rsquo;t. It won&amp;rsquo;t catch dependency monsters directly — those require a human asking &amp;ldquo;does every consumer of this model actually use every attribute in it?&amp;rdquo; — but it&amp;rsquo;s a good starting point for the structural audit.&lt;/p>
&lt;p>For Airflow DAGs, the visual DAG view tells you a lot at a glance. A healthy DAG looks like a tree with wide parallel branches. An unhealthy DAG looks like a chain — or worse, a funnel where everything converges through a single overloaded task before it can branch out again. That funnel shape is the visual signature of a dependency monster: many expensive upstreams flowing into one hub model, with dozens of downstream models queued behind it.&lt;/p>
&lt;p>If your DAG looks like a chain, ask this question for every dependency edge: &lt;strong>does downstream task B actually read data produced by upstream task A?&lt;/strong> If the answer is no, remove the dependency. Let them run in parallel.&lt;/p>
&lt;p>I once audited a DAG with 47 tasks running in a strict serial chain. After removing phantom dependencies and restructuring, 31 of those tasks could run in parallel. The pipeline went from 2 hours 15 minutes to 38 minutes. Same tasks, same compute, same data. Just fewer unnecessary wait states.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-critical-path-is-the-only-thing-worth-optimising">The critical path is the only thing worth optimising&lt;/h3>
&lt;br>
&lt;p>Here&amp;rsquo;s a concept from project management that applies directly to pipeline engineering: the &lt;strong>critical path&lt;/strong>.&lt;/p>
&lt;p>The critical path is the longest chain of dependent tasks through your DAG. It determines your pipeline&amp;rsquo;s minimum possible runtime. Every other path through the DAG has &amp;ldquo;float&amp;rdquo; — slack time where tasks can be delayed without affecting the overall completion time.&lt;/p>
&lt;p>This means: &lt;strong>optimising a task that isn&amp;rsquo;t on the critical path has zero impact on your pipeline&amp;rsquo;s end-to-end runtime.&lt;/strong> You could make a non-critical task 10x faster and your pipeline would finish at exactly the same time.&lt;/p>
&lt;p>Finding the critical path requires knowing each task&amp;rsquo;s duration and dependency structure. For a dbt project, you can approximate it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Using dbt&amp;#39;s run results stored in Snowflake (if you log them)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Or query your orchestrator&amp;#39;s task instance table
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">WITH&lt;/span> task_durations &lt;span style="color:#66d9ef">AS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> task_id &lt;span style="color:#66d9ef">AS&lt;/span> model_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AVG&lt;/span>(&lt;span style="color:#66d9ef">EXTRACT&lt;/span>(EPOCH &lt;span style="color:#66d9ef">FROM&lt;/span> (end_date &lt;span style="color:#f92672">-&lt;/span> start_date))) &lt;span style="color:#66d9ef">AS&lt;/span> avg_duration
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> task_instance
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> dag_id &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;dbt_daily&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> &lt;span style="color:#66d9ef">state&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;success&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> execution_date &lt;span style="color:#f92672">&amp;gt;=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span> &lt;span style="color:#f92672">-&lt;/span> INTERVAL &lt;span style="color:#e6db74">&amp;#39;7 days&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> task_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> model_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROUND(avg_duration, &lt;span style="color:#ae81ff">1&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> avg_seconds,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROUND(avg_duration &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#ae81ff">60&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> avg_minutes
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> task_durations
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> avg_duration &lt;span style="color:#66d9ef">DESC&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LIMIT&lt;/span> &lt;span style="color:#ae81ff">20&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The longest tasks are &lt;em>candidates&lt;/em> for the critical path, but only if they&amp;rsquo;re on the dependency chain that determines total runtime. A 20-minute model that runs in parallel with a 45-minute model isn&amp;rsquo;t the bottleneck — the 45-minute model is.&lt;/p>
&lt;p>Once you&amp;rsquo;ve identified the critical path, you have three options for shortening it: make the slow tasks faster (query optimisation, incremental models), reduce the number of tasks on the path (remove unnecessary dependencies), or parallelise sequential tasks (split a monolithic model into independent pieces).&lt;/p>
&lt;hr>
&lt;h3 id="shifting-right-diagnose-it-before-it-breaks-your-sla">Shifting right: diagnose it before it breaks your SLA&lt;/h3>
&lt;br>
&lt;p>Remember Priya&amp;rsquo;s scatter plot? That pattern — pipeline completion drifting later and later — has a name: &lt;strong>shifting right&lt;/strong>. And it has predictable causes.&lt;/p>
&lt;p>Track it with a simple query against your orchestrator&amp;rsquo;s metadata:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Airflow: track pipeline completion time drift
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> execution_date::DATE &lt;span style="color:#66d9ef">AS&lt;/span> run_date,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">MAX&lt;/span>(end_date)::TIME &lt;span style="color:#66d9ef">AS&lt;/span> completion_time,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">EXTRACT&lt;/span>(EPOCH &lt;span style="color:#66d9ef">FROM&lt;/span> (&lt;span style="color:#66d9ef">MAX&lt;/span>(end_date) &lt;span style="color:#f92672">-&lt;/span> &lt;span style="color:#66d9ef">MIN&lt;/span>(start_date))) &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#ae81ff">60&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> total_runtime_minutes
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> task_instance
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dag_id &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;daily_analytics&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> &lt;span style="color:#66d9ef">state&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;success&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> execution_date &lt;span style="color:#f92672">&amp;gt;=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span> &lt;span style="color:#f92672">-&lt;/span> INTERVAL &lt;span style="color:#e6db74">&amp;#39;60 days&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> run_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> run_date;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Plot &lt;code>completion_time&lt;/code> over &lt;code>run_date&lt;/code>. If it trends upward, you&amp;rsquo;re shifting right. The four root causes, in order of how often I see them:&lt;/p>
&lt;p>&lt;strong>1. Data volume growth.&lt;/strong> Your table had 10 million rows when you built the model. Now it has 200 million. That JOIN that took 8 seconds now takes 3 minutes. This is the most common cause and the easiest to fix — switch to incremental materialisation and the growth stops mattering.&lt;/p>
&lt;p>&lt;strong>2. Resource contention.&lt;/strong> You&amp;rsquo;ve added more pipelines and they all run in the same window, competing for the same Snowflake warehouse. In Snowflake, you can see this directly:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Find warehouse queueing (queries waiting for compute)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> warehouse_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> DATE_TRUNC(&lt;span style="color:#e6db74">&amp;#39;hour&amp;#39;&lt;/span>, start_time) &lt;span style="color:#66d9ef">AS&lt;/span> hour,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> total_queries,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> queued_overload_time &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> queued_queries,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AVG&lt;/span>(queued_overload_time) &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#ae81ff">1000&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> avg_queue_seconds
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> snowflake.account_usage.query_history
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> start_time &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">7&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> warehouse_name &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;TRANSFORM_WH&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> warehouse_name, hour
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">HAVING&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> queued_queries &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> hour;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>If &lt;code>queued_queries&lt;/code> is non-zero during your pipeline window, queries are waiting for warehouse compute. Either scale up the warehouse during that window, split workloads across dedicated warehouses, or stagger pipeline start times to reduce concurrency.&lt;/p>
&lt;p>&lt;strong>3. Dependency chain lengthening.&lt;/strong> Someone added a new intermediate model between your staging and mart layers. That model takes 4 minutes. But because it&amp;rsquo;s on the critical path, the entire pipeline now finishes 4 minutes later. This is death by a thousand cuts — each addition is small, but they accumulate.&lt;/p>
&lt;p>&lt;strong>4. Upstream source delays.&lt;/strong> Your pipeline starts at 4 AM because the source system&amp;rsquo;s extract used to land in S3 by 3:45 AM. But the source system has grown too, and now the extract doesn&amp;rsquo;t land until 4:30 AM. Your pipeline sensors wait, and everything shifts right.&lt;/p>
&lt;p>For upstream delays in an AWS environment, replace time-based scheduling with event-driven triggers. Instead of scheduling your Airflow DAG at 4 AM and hoping the data is there, trigger it when the data actually arrives:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Airflow DAG: trigger on S3 file landing using AWS sensor&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> airflow.providers.amazon.aws.sensors.s3 &lt;span style="color:#f92672">import&lt;/span> S3KeySensor
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> airflow &lt;span style="color:#f92672">import&lt;/span> DAG
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> datetime &lt;span style="color:#f92672">import&lt;/span> datetime
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">with&lt;/span> DAG(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;daily_analytics&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> start_date&lt;span style="color:#f92672">=&lt;/span>datetime(&lt;span style="color:#ae81ff">2026&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> schedule_interval&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">None&lt;/span>, &lt;span style="color:#75715e"># triggered externally, not on a schedule&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> catchup&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">False&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>) &lt;span style="color:#66d9ef">as&lt;/span> dag:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> wait_for_source &lt;span style="color:#f92672">=&lt;/span> S3KeySensor(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> task_id&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;wait_for_orders_extract&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> bucket_name&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;your-data-lake&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> bucket_key&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;raw/orders/dt={{ ds }}/orders.parquet&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> aws_conn_id&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;aws_default&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> mode&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;reschedule&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># frees the worker slot while waiting&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> poke_interval&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">300&lt;/span>, &lt;span style="color:#75715e"># check every 5 minutes&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> timeout&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">7200&lt;/span>, &lt;span style="color:#75715e"># fail after 2 hours if file never arrives&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>mode='reschedule'&lt;/code> is critical. Without it, the sensor occupies a worker slot for the entire time it&amp;rsquo;s waiting. With &lt;code>reschedule&lt;/code>, it checks, releases the slot, and checks again later. This prevents the classic deadlock where all your worker slots are consumed by sensors and no actual work can run.&lt;/p>
&lt;p>Even better: use S3 event notifications to trigger a Lambda function that kicks off the DAG via Airflow&amp;rsquo;s REST API. Zero polling, zero wasted slots:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Lambda function triggered by S3 PutObject event&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> boto3
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> requests
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> os
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">handler&lt;/span>(event, context):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> bucket &lt;span style="color:#f92672">=&lt;/span> event[&lt;span style="color:#e6db74">&amp;#39;Records&amp;#39;&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;s3&amp;#39;&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;bucket&amp;#39;&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;name&amp;#39;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> key &lt;span style="color:#f92672">=&lt;/span> event[&lt;span style="color:#e6db74">&amp;#39;Records&amp;#39;&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;s3&amp;#39;&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;object&amp;#39;&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;key&amp;#39;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Trigger Airflow DAG via REST API&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> airflow_url &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ[&lt;span style="color:#e6db74">&amp;#39;AIRFLOW_API_URL&amp;#39;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> response &lt;span style="color:#f92672">=&lt;/span> requests&lt;span style="color:#f92672">.&lt;/span>post(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>airflow_url&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">/api/v1/dags/daily_analytics/dagRuns&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> json&lt;span style="color:#f92672">=&lt;/span>{
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;conf&amp;#34;&lt;/span>: {&lt;span style="color:#e6db74">&amp;#34;source_bucket&amp;#34;&lt;/span>: bucket, &lt;span style="color:#e6db74">&amp;#34;source_key&amp;#34;&lt;/span>: key}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> auth&lt;span style="color:#f92672">=&lt;/span>(os&lt;span style="color:#f92672">.&lt;/span>environ[&lt;span style="color:#e6db74">&amp;#39;AIRFLOW_USER&amp;#39;&lt;/span>], os&lt;span style="color:#f92672">.&lt;/span>environ[&lt;span style="color:#e6db74">&amp;#39;AIRFLOW_PASSWORD&amp;#39;&lt;/span>]),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> headers&lt;span style="color:#f92672">=&lt;/span>{&lt;span style="color:#e6db74">&amp;#34;Content-Type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;application/json&amp;#34;&lt;/span>}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> {&lt;span style="color:#e6db74">&amp;#34;statusCode&amp;#34;&lt;/span>: response&lt;span style="color:#f92672">.&lt;/span>status_code}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This eliminates the &amp;ldquo;safety margin&amp;rdquo; scheduling pattern entirely. Your pipeline runs as soon as the data is available — not 30 minutes after you &lt;em>hope&lt;/em> it will be available.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="partition-and-cluster-to-eliminate-full-table-scans">Partition and cluster to eliminate full table scans&lt;/h3>
&lt;br>
&lt;p>The single most effective performance optimisation in Snowflake is making sure your queries only read the data they need. Snowflake&amp;rsquo;s micro-partition pruning does this automatically — but only if your data is physically organised in a way that aligns with your query patterns.&lt;/p>
&lt;p>Clustering keys tell Snowflake how to organise data within micro-partitions. If you consistently filter on &lt;code>order_date&lt;/code>, clustering on that column means queries with a &lt;code>WHERE order_date = '2026-03-01'&lt;/code> clause scan a tiny fraction of the table instead of the whole thing.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Add clustering to a large fact table
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">ALTER&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> analytics.fct_orders
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CLUSTER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> (order_date, customer_segment);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Choose clustering keys based on how the table is actually queried, not how it&amp;rsquo;s loaded. The best candidates are columns that appear in WHERE clauses, JOIN conditions, and range filters. Limit yourself to 2–3 keys — more than that and Snowflake can&amp;rsquo;t maintain effective clustering.&lt;/p>
&lt;p>Check whether your existing tables benefit from clustering:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Check clustering depth (lower is better, 0 is perfect)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">table_name&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> clustering_key,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> total_constant_partition_count,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> total_partition_count,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> average_overlaps,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> average_depth
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> snowflake.account_usage.table_storage_metrics
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> table_catalog &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;YOUR_DATABASE&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> clustering_key &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> active_bytes &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> average_depth &lt;span style="color:#66d9ef">DESC&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>An &lt;code>average_depth&lt;/code> above 5 means your clustering isn&amp;rsquo;t effective — queries are still scanning more partitions than they should. Either the clustering key doesn&amp;rsquo;t match query patterns, or the table has had so many small writes that the clustering has degraded. Running &lt;code>ALTER TABLE ... RECLUSTER&lt;/code> manually or relying on automatic clustering (which costs credits) addresses the latter.&lt;/p>
&lt;p>On the AWS side, if you&amp;rsquo;re using S3 as a data lake with Parquet or Iceberg, the equivalent is &lt;strong>partition layout&lt;/strong>. Partition your S3 data by the columns you filter most:&lt;/p>



&lt;div class="goat svg-container ">
 
 &lt;svg
 xmlns="http://www.w3.org/2000/svg"
 font-family="Menlo,Lucida Console,monospace"
 
 viewBox="0 0 312 121"
 >
 &lt;g transform='translate(8,16)'>
&lt;polygon points='176.000000,80.000000 164.000000,74.400002 164.000000,85.599998' fill='currentColor' transform='rotate(90.000000, 168.000000, 80.000000)'>&lt;/polygon>
&lt;text text-anchor='middle' x='0' y='4' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='8' y='4' fill='currentColor' style='font-size:1em'>3&lt;/text>
&lt;text text-anchor='middle' x='16' y='4' fill='currentColor' style='font-size:1em'>:&lt;/text>
&lt;text text-anchor='middle' x='24' y='4' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='32' y='4' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='32' y='20' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='40' y='4' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='40' y='20' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='48' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='48' y='20' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='56' y='4' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='56' y='20' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='64' y='4' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='64' y='20' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='64' y='36' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='72' y='4' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='72' y='20' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='72' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='80' y='4' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='80' y='20' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='80' y='36' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='88' y='4' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='88' y='36' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='96' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='96' y='36' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='96' y='52' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='104' y='4' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='104' y='36' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='104' y='52' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='112' y='4' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='112' y='36' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='112' y='52' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='120' y='4' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='120' y='36' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='120' y='52' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='128' y='4' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='128' y='36' fill='currentColor' style='font-size:1em'>6&lt;/text>
&lt;text text-anchor='middle' x='128' y='52' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='128' y='68' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='136' y='4' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='136' y='36' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='136' y='52' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='136' y='68' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='144' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='144' y='52' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='144' y='68' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='152' y='4' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='152' y='52' fill='currentColor' style='font-size:1em'>3&lt;/text>
&lt;text text-anchor='middle' x='152' y='68' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='160' y='52' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='160' y='68' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='160' y='84' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='160' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='168' y='68' fill='currentColor' style='font-size:1em'>5&lt;/text>
&lt;text text-anchor='middle' x='168' y='100' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='176' y='68' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='176' y='84' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='176' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='184' y='84' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='184' y='100' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='192' y='84' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='192' y='100' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='200' y='84' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='200' y='100' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='208' y='84' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='208' y='100' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='216' y='84' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='216' y='100' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='224' y='84' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='224' y='100' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='232' y='84' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='232' y='100' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='240' y='84' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='240' y='100' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='248' y='84' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='248' y='100' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='256' y='84' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='256' y='100' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='264' y='84' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='264' y='100' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='272' y='84' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='272' y='100' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='280' y='84' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='280' y='100' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='288' y='84' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='288' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='296' y='84' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='296' y='100' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;/g>

 &lt;/svg>
 
&lt;/div>
&lt;p>When Snowflake external tables or AWS Athena query this structure with a date filter, they skip entire directories. A query for March 15th reads two files instead of scanning the entire &lt;code>events/&lt;/code> prefix. The savings compound with data volume — at a billion rows, proper partitioning can reduce query times from minutes to seconds.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="make-every-task-idempotent-or-debugging-becomes-a-nightmare">Make every task idempotent or debugging becomes a nightmare&lt;/h3>
&lt;br>
&lt;p>Here&amp;rsquo;s a pattern I see constantly: a pipeline fails midway through, the engineer reruns it, and now there are duplicate rows in the target table. Because the first run inserted half the data before failing, and the rerun inserted &lt;em>all&lt;/em> the data — including the half that already existed.&lt;/p>
&lt;p>Idempotency means a task produces the same result whether it runs once or ten times. This isn&amp;rsquo;t just a nice-to-have — it&amp;rsquo;s the foundation that makes everything else in pipeline engineering possible. Without it, you can&amp;rsquo;t safely retry. You can&amp;rsquo;t backfill. You can&amp;rsquo;t debug.&lt;/p>
&lt;p>In dbt, incremental models with a &lt;code>unique_key&lt;/code> are idempotent by default — the MERGE statement handles duplicates:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> config(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> materialized&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;incremental&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> unique_key&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;event_id&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> incremental_strategy&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;merge&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> event_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> user_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> event_type,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> event_timestamp,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> properties
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> &lt;span style="color:#66d9ef">ref&lt;/span>(&lt;span style="color:#e6db74">&amp;#39;stg_events&amp;#39;&lt;/span>) &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#960050;background-color:#1e0010">{&lt;/span>&lt;span style="color:#f92672">%&lt;/span> &lt;span style="color:#66d9ef">if&lt;/span> is_incremental() &lt;span style="color:#f92672">%&lt;/span>&lt;span style="color:#960050;background-color:#1e0010">}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> event_timestamp &lt;span style="color:#f92672">&amp;gt;=&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">3&lt;/span>, &lt;span style="color:#66d9ef">MAX&lt;/span>(event_timestamp))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> this &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#960050;background-color:#1e0010">{&lt;/span>&lt;span style="color:#f92672">%&lt;/span> endif &lt;span style="color:#f92672">%&lt;/span>&lt;span style="color:#960050;background-color:#1e0010">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>For non-dbt loads — say, a Lambda function loading data from an API into Snowflake — use MERGE instead of INSERT:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>MERGE &lt;span style="color:#66d9ef">INTO&lt;/span> raw.api_customers &lt;span style="color:#66d9ef">AS&lt;/span> target
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">USING&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">$&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>:id::STRING &lt;span style="color:#66d9ef">AS&lt;/span> customer_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">$&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>:name::STRING &lt;span style="color:#66d9ef">AS&lt;/span> customer_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">$&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>:email::STRING &lt;span style="color:#66d9ef">AS&lt;/span> email,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">$&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>:updated_at::TIMESTAMP_NTZ &lt;span style="color:#66d9ef">AS&lt;/span> updated_at
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> &lt;span style="color:#f92672">@&lt;/span>raw.s3_stage&lt;span style="color:#f92672">/&lt;/span>customers&lt;span style="color:#f92672">/&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (FILE_FORMAT &lt;span style="color:#f92672">=&amp;gt;&lt;/span> &lt;span style="color:#e6db74">&amp;#39;json_format&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>) &lt;span style="color:#66d9ef">AS&lt;/span> &lt;span style="color:#66d9ef">source&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ON&lt;/span> target.customer_id &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">source&lt;/span>.customer_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> MATCHED &lt;span style="color:#66d9ef">AND&lt;/span> &lt;span style="color:#66d9ef">source&lt;/span>.updated_at &lt;span style="color:#f92672">&amp;gt;&lt;/span> target.updated_at &lt;span style="color:#66d9ef">THEN&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">UPDATE&lt;/span> &lt;span style="color:#66d9ef">SET&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_name &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">source&lt;/span>.customer_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> email &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">source&lt;/span>.email,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> updated_at &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">source&lt;/span>.updated_at
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> MATCHED &lt;span style="color:#66d9ef">THEN&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">INSERT&lt;/span> (customer_id, customer_name, email, updated_at)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">VALUES&lt;/span> (&lt;span style="color:#66d9ef">source&lt;/span>.customer_id, &lt;span style="color:#66d9ef">source&lt;/span>.customer_name, &lt;span style="color:#66d9ef">source&lt;/span>.email, &lt;span style="color:#66d9ef">source&lt;/span>.updated_at);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>WHEN MATCHED AND source.updated_at &amp;gt; target.updated_at&lt;/code> condition prevents overwriting newer data with older data during reruns. This matters when you&amp;rsquo;re replaying historical loads or running overlapping backfills.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="most-we-need-real-time-requests-actually-need-we-need-faster-batch">Most &amp;ldquo;we need real-time&amp;rdquo; requests actually need &amp;ldquo;we need faster batch&amp;rdquo;&lt;/h3>
&lt;br>
&lt;p>Before you reach for Kafka, Kinesis, or any streaming infrastructure, have this conversation with your stakeholders: &amp;ldquo;When you say real-time, what do you actually mean?&amp;rdquo;&lt;/p>
&lt;p>In my experience, the answers fall into three categories:&lt;/p>
&lt;p>&lt;strong>&amp;ldquo;I want data from today, not yesterday.&amp;rdquo;&lt;/strong> That&amp;rsquo;s daily batch with a morning refresh. You already have this. Maybe you need to move the refresh earlier.&lt;/p>
&lt;p>&lt;strong>&amp;ldquo;I want data that&amp;rsquo;s at most an hour old.&amp;rdquo;&lt;/strong> That&amp;rsquo;s micro-batch — running your existing pipeline every 15–60 minutes instead of once a day. No new infrastructure required. Your existing SQL, dbt, and Airflow tools work unchanged at shorter intervals.&lt;/p>
&lt;p>&lt;strong>&amp;ldquo;I need to see events within seconds of them happening.&amp;rdquo;&lt;/strong> &lt;em>This&lt;/em> is actual real-time. And it&amp;rsquo;s genuinely rare. Fraud detection, safety monitoring, real-time bidding — these need streaming. Your internal sales dashboard almost certainly does not.&lt;/p>
&lt;p>For the micro-batch pattern in Airflow, it&amp;rsquo;s a scheduling change:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">with&lt;/span> DAG(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;micro_batch_analytics&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> schedule_interval&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;*/15 * * * *&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># every 15 minutes&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> catchup&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">False&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> max_active_runs&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#75715e"># prevent overlap&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>) &lt;span style="color:#66d9ef">as&lt;/span> dag:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># ...tasks here&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>max_active_runs=1&lt;/code> is important. Without it, if a run takes longer than 15 minutes, the next run starts before the previous one finishes. They compete for the same warehouse, both run slower, and you&amp;rsquo;ve created the resource contention problem from the shifting-right section.&lt;/p>
&lt;p>In Snowflake, pair this with Snowpipe for continuous S3 ingestion:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create a pipe for continuous loading from S3
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">REPLACE&lt;/span> PIPE raw.orders_pipe
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> AUTO_INGEST &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">TRUE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AS&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COPY&lt;/span> &lt;span style="color:#66d9ef">INTO&lt;/span> raw.orders_stream
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> &lt;span style="color:#f92672">@&lt;/span>raw.s3_orders_stage
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> FILE_FORMAT &lt;span style="color:#f92672">=&lt;/span> (&lt;span style="color:#66d9ef">TYPE&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PARQUET&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> MATCH_BY_COLUMN_NAME &lt;span style="color:#f92672">=&lt;/span> CASE_INSENSITIVE;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Snowpipe loads files within minutes of them landing in S3. Your micro-batch dbt pipeline then transforms whatever arrived since the last run. The combination delivers data freshness measured in minutes — not seconds, but close enough for the vast majority of business use cases — without any streaming infrastructure.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="measure-time-to-consumer-not-just-pipeline-runtime">Measure time-to-consumer, not just pipeline runtime&lt;/h3>
&lt;br>
&lt;p>Pipeline runtime tells you how long your DAG takes. Time-to-consumer tells you how long a business event takes to reach a human decision-maker. These are different numbers, and the second one is what your stakeholders actually care about.&lt;/p>
&lt;p>Time-to-consumer decomposes into stages:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Source extraction delay&lt;/strong>: Time between the event occurring and the data landing in your lake&lt;/li>
&lt;li>&lt;strong>Pipeline processing time&lt;/strong>: Your DAG runtime (the part you control most directly)&lt;/li>
&lt;li>&lt;strong>Warehouse serving time&lt;/strong>: Query execution time when a dashboard or report reads the data&lt;/li>
&lt;li>&lt;strong>Cache/refresh delay&lt;/strong>: How often the BI tool refreshes its cache&lt;/li>
&lt;/ul>
&lt;p>Track it by stamping timestamps at each handoff:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Build a freshness tracking model in dbt
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- mart_data_freshness.sql
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;fct_orders&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> model_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">MAX&lt;/span>(order_timestamp) &lt;span style="color:#66d9ef">AS&lt;/span> latest_source_event,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">MAX&lt;/span>(_loaded_at) &lt;span style="color:#66d9ef">AS&lt;/span> latest_load_time,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>() &lt;span style="color:#66d9ef">AS&lt;/span> measured_at,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> TIMESTAMPDIFF(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;minute&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">MAX&lt;/span>(order_timestamp),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ) &lt;span style="color:#66d9ef">AS&lt;/span> minutes_since_latest_event,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> TIMESTAMPDIFF(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;minute&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">MAX&lt;/span>(_loaded_at),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ) &lt;span style="color:#66d9ef">AS&lt;/span> minutes_since_latest_load
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> &lt;span style="color:#66d9ef">ref&lt;/span>(&lt;span style="color:#e6db74">&amp;#39;fct_orders&amp;#39;&lt;/span>) &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Schedule this model to run after every pipeline completion. Over time, you&amp;rsquo;ll see whether &lt;code>minutes_since_latest_event&lt;/code> is stable, improving, or drifting. If it&amp;rsquo;s drifting, you can pinpoint which stage is responsible by comparing the event timestamp, load timestamp, and measurement timestamp.&lt;/p>
&lt;p>In dbt, you can also use source freshness tests to alert when upstream data is stale:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># models/staging/_sources.yml&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">sources&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">raw&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">database&lt;/span>: &lt;span style="color:#ae81ff">raw_db&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">schema&lt;/span>: &lt;span style="color:#ae81ff">public&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">loaded_at_field&lt;/span>: &lt;span style="color:#ae81ff">_loaded_at&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">freshness&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">warn_after&lt;/span>: { &lt;span style="color:#f92672">count: 2, period&lt;/span>: &lt;span style="color:#ae81ff">hour }&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">error_after&lt;/span>: { &lt;span style="color:#f92672">count: 6, period&lt;/span>: &lt;span style="color:#ae81ff">hour }&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">tables&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">orders&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">customers&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">events&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Running &lt;code>dbt source freshness&lt;/code> checks whether the upstream data is as fresh as you expect. If &lt;code>raw.orders&lt;/code> hasn&amp;rsquo;t received new rows in 6 hours, the test fails — and that&amp;rsquo;s a signal that the source system&amp;rsquo;s extract is delayed, not that your pipeline is broken.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-view-on-view-problem-the-compound-interest-of-bad-performance">The view-on-view problem: the compound interest of bad performance&lt;/h3>
&lt;br>
&lt;p>This is the architectural anti-pattern I see most often in Snowflake environments, and it&amp;rsquo;s one of those problems that starts small and compounds quietly.&lt;/p>
&lt;p>A view in Snowflake doesn&amp;rsquo;t store any data — it re-executes its SQL every time it&amp;rsquo;s queried. That&amp;rsquo;s fine for a simple view. But when View C references View B, which references View A, which scans a large table — every query against View C triggers the entire chain from scratch.&lt;/p>
&lt;p>I&amp;rsquo;ve seen environments where analysts querying a &amp;ldquo;simple&amp;rdquo; dashboard view were unknowingly triggering five layers of nested views, each with its own JOINs and aggregations. The query took 4 minutes. After materialising the two most expensive intermediate layers as tables (refreshed daily), the same query took 3 seconds.&lt;/p>
&lt;p>The fix is to audit your materialisation strategy:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Find views that reference other views (potential nesting)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> referencing_object_name &lt;span style="color:#66d9ef">AS&lt;/span> downstream_view,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> referenced_object_name &lt;span style="color:#66d9ef">AS&lt;/span> upstream_object,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> referenced_object_type
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> snowflake.account_usage.object_dependencies
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> referencing_object_type &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;VIEW&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> referenced_object_type &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;VIEW&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> downstream_view;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>If this query returns results, you have view-on-view nesting. Not all of it is bad — a simple renaming view on top of a complex view is fine. But if you find three or more layers, or if any of the intermediate views involve heavy JOINs or aggregations, materialise the expensive layers as tables.&lt;/p>
&lt;p>In dbt terms, the decision framework is:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Ephemeral&lt;/strong>: Simple CTEs, column renaming, type casting. Zero cost.&lt;/li>
&lt;li>&lt;strong>View&lt;/strong>: Light transformations queried infrequently by a few consumers.&lt;/li>
&lt;li>&lt;strong>Table&lt;/strong>: Complex transformations queried frequently. Rebuilt every run.&lt;/li>
&lt;li>&lt;strong>Incremental&lt;/strong>: Large tables where only a fraction of rows change. The right choice for most fact tables.&lt;/li>
&lt;/ul>
&lt;p>If a model takes more than 30 seconds to build and is queried more than once between builds, it should be a table or incremental — not a view.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="coming-back-to-priyas-scatter-plot">Coming back to Priya&amp;rsquo;s scatter plot&lt;/h3>
&lt;br>
&lt;p>Six weeks after Priya showed me that graph, we&amp;rsquo;d fixed the three root causes: an intermediate model that had grown from 5 million to 180 million rows without being switched to incremental, a phantom dependency chain that added 12 minutes of serial wait time, and a warehouse that was being shared between transformation and BI queries during the morning refresh window.&lt;/p>
&lt;p>Pipeline completion moved from 7:23 AM back to 5:15 AM. The finance team got their dashboards before their morning coffee. And Priya? She built a monitoring dashboard that tracked completion time drift automatically — the thing I should have built from the start.&lt;/p>
&lt;p>Here&amp;rsquo;s what I learned from that experience, and what I hope you take from this article: &lt;strong>pipeline performance isn&amp;rsquo;t a problem you solve once.&lt;/strong> It&amp;rsquo;s a trajectory you manage. Data grows. Teams add models. Source systems change. The pipeline that&amp;rsquo;s fast today will be slow in six months unless someone is watching the trend line.&lt;/p>
&lt;p>The engineers who build reliable data platforms aren&amp;rsquo;t the ones who build the fastest pipeline on day one. They&amp;rsquo;re the ones who notice when completion time drifts by 3 minutes per week — and fix it before anyone else has to ask.&lt;/p>
&lt;p>Track your completion times. Audit your dependencies quarterly. Profile your critical path after every significant change. And teach your team that a green DAG doesn&amp;rsquo;t mean a healthy pipeline. Sometimes green just means it hasn&amp;rsquo;t failed &lt;em>yet&lt;/em>.&lt;/p>
&lt;p>&lt;br>&lt;br>&lt;/p></content:encoded><category>Data Engineering</category><category>Platforms &amp; Cloud</category><category>Snowflake</category><category>AWS</category><category>Apache Airflow</category><category>Performance Optimisation</category><category>dbt</category><category>Data Freshness</category></item><item><title>Stop Building Salesforce Integrations From Scratch</title><link>https://ghostinthedata.info/posts/2026/2026-04-04-snowflake-openflow/</link><pubDate>Sat, 04 Apr 2026 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-04-04-snowflake-openflow/</guid><author>Chris Hillman</author><description>A hands-on guide to Snowflake's OpenFlow Salesforce connector — why managed connectors beat custom code, how to set one up step by step, and the schema evolution feature that makes it all worth it.</description><content:encoded>&lt;p>Let me tell you about Marcus.&lt;/p>
&lt;/br>
&lt;p>Marcus was on a team I led a few years back. Sharp, motivated, the kind of engineer who actually read documentation before writing code. When the business asked us to get Salesforce data into our warehouse, Marcus volunteered. He&amp;rsquo;d done API work before. He figured a few weeks, tops.&lt;/p>
&lt;p>He scoped it carefully. Built a Python service that authenticated via OAuth, pulled Account, Contact, and Opportunity objects through the Bulk API, flattened the nested JSON into relational tables, handled pagination, managed rate limits. Wrote solid tests. Documented everything. The kind of work you&amp;rsquo;d point to in a code review and say &lt;em>this is how it&amp;rsquo;s done&lt;/em>.&lt;/p>
&lt;p>It worked beautifully. For about three months.&lt;/p>
&lt;/br>
&lt;p>Then our Salesforce admin added a custom field to the Opportunity object — &lt;code>Renewal_Likelihood__c&lt;/code> — and nobody told the data team. The pipeline didn&amp;rsquo;t fail. That&amp;rsquo;s the insidious part. It kept running, kept landing data. It just quietly dropped the new field on the floor.&lt;/p>
&lt;p>When Marcus tracked it down, he added the field. Then he realised there were &lt;em>eleven&lt;/em> other custom fields that had been added since go-live that the pipeline was silently ignoring. And the compound address fields — &lt;code>BillingAddress&lt;/code>, &lt;code>ShippingAddress&lt;/code> — had never worked with the Bulk API in the first place. He&amp;rsquo;d been extracting the component parts (&lt;code>BillingStreet&lt;/code>, &lt;code>BillingCity&lt;/code>) as a workaround, but the workaround had a bug that truncated postal codes for international addresses.&lt;/p>
&lt;p>Marcus spent three weeks patching all of it. Three weeks of a talented engineer doing work that a managed connector handles automatically.&lt;/p>
&lt;/br>
&lt;p>When I watched a good engineer spend the best part of a month on a problem that shouldn&amp;rsquo;t exist. Marcus wasn&amp;rsquo;t learning anything. He wasn&amp;rsquo;t growing. He was hand-coding schema detection logic for the fourth time because Salesforce ships three major releases a year and our sales ops team adds custom fields like they&amp;rsquo;re decorating a Christmas tree.&lt;/p>
&lt;p>I&amp;rsquo;m telling you this because &lt;strong>Snowflake OpenFlow&lt;/strong> is one of those things that sounds like &amp;ldquo;just another integration tool&amp;rdquo; until you&amp;rsquo;ve lived through the alternative. What I want to show you today is how to set it up with Salesforce — step by step, with enough detail that you could do it tomorrow — and more importantly, &lt;em>why&lt;/em> the schema evolution feature alone justifies the switch from custom code.&lt;/p>
&lt;p>If you&amp;rsquo;ve ever maintained a custom Salesforce integration, you already know why this matters.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-problem-nobody-warns-you-about">The Problem Nobody Warns You About&lt;/h3>
&lt;/br>
&lt;p>Here&amp;rsquo;s what the &amp;ldquo;just build it yourself&amp;rdquo; crowd doesn&amp;rsquo;t tell you: the Salesforce API isn&amp;rsquo;t one API. It&amp;rsquo;s an ecosystem of overlapping interfaces, each with its own quirks, limits, and failure modes.&lt;/p>
&lt;p>The REST API handles small, synchronous requests well — but try pulling a million Opportunity records through it and you&amp;rsquo;ll burn through your rate limits before lunch. The Bulk API 2.0 is designed for volume — it can handle up to 150 million records in a 24-hour window — but it doesn&amp;rsquo;t support compound fields like &lt;code>BillingAddress&lt;/code> or &lt;code>MailingAddress&lt;/code>. Those silently return nothing. Not an error. Nothing.&lt;/p>
&lt;p>Then there&amp;rsquo;s SOQL, Salesforce&amp;rsquo;s proprietary query language, which looks enough like SQL to trick you into thinking you understand it. Until you hit the 2,000-record offset limit, or try to resolve a polymorphic relationship where a &lt;code>WhoId&lt;/code> field on a Task could reference either a Contact or a Lead depending on the record. That&amp;rsquo;s not a data modelling problem — that&amp;rsquo;s a &amp;ldquo;your pipeline logic needs to branch based on the referenced object type&amp;rdquo; problem.&lt;/p>
&lt;p>And rate limits. Enterprise Edition gives you 100,000 API calls per 24 hours, plus 1,000 per user licence. Sounds generous until you remember that your BI tool, marketing automation platform, customer support system, and your data pipeline are all drinking from the same well.&lt;/p>
&lt;/br>
&lt;p>But schema evolution is the silent killer.&lt;/p>
&lt;p>Every Salesforce org is different because every business is different. Your Salesforce admin creates custom objects and fields to match how your company actually works. That&amp;rsquo;s the point — it&amp;rsquo;s a customisation platform. But every custom field added to Salesforce is a potential break point for any integration that doesn&amp;rsquo;t handle schema changes automatically.&lt;/p>
&lt;p>And those changes happen constantly. Research suggests schemas in enterprise SaaS tools change roughly every three days on average. In Salesforce specifically, between admin-driven customisation and Salesforce&amp;rsquo;s own three-times-a-year release cycle (Spring, Summer, Winter), the schema you built against last quarter may not be the schema you&amp;rsquo;re ingesting from today.&lt;/p>
&lt;p>This is the trap. Building the initial integration isn&amp;rsquo;t hard. Any competent engineer can get Salesforce data into a warehouse. &lt;strong>Keeping it working&lt;/strong> — across schema changes, API version deprecations, authentication token rotations, and rate limit adjustments — is where it eats your life.&lt;/p>
&lt;p>I&amp;rsquo;ve seen teams eliminate two full days of monthly engineering maintenance by migrating off custom connectors. Two days, every month, spent on work that a managed tool does automatically. That&amp;rsquo;s not engineering — it&amp;rsquo;s janitorial work wearing an engineering costume.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="what-is-snowflake-openflow">What Is Snowflake OpenFlow?&lt;/h3>
&lt;/br>
&lt;p>OpenFlow is Snowflake&amp;rsquo;s native, fully managed data integration service. It&amp;rsquo;s not a partnership or a marketplace app — it&amp;rsquo;s a first-party feature built into the platform. If you&amp;rsquo;re already running Snowflake, OpenFlow runs inside your existing environment.&lt;/p>
&lt;p>The backstory matters. In November 2024, Snowflake acquired a company called &lt;strong>Datavolo&lt;/strong> for roughly $110 million. Datavolo was founded by Joe Witt, who co-created Apache NiFi back when it was an NSA project. That NiFi heritage is the foundation of OpenFlow — curated, versioned connector definitions powered by NiFi&amp;rsquo;s flow engine, wrapped in a managed Snowflake experience.&lt;/p>
&lt;p>Architecturally, it splits into two pieces. The &lt;strong>control plane&lt;/strong> lives inside Snowflake and gives you pipeline management, scheduling, and monitoring through the Snowsight UI. The &lt;strong>data plane&lt;/strong> handles the actual work and can run in two modes: &lt;strong>Snowflake Deployments&lt;/strong> (running on Snowpark Container Services within Snowflake&amp;rsquo;s own infrastructure) or &lt;strong>BYOC&lt;/strong> (Bring Your Own Cloud, running as a Kubernetes cluster in your VPC). For most teams starting out, the SPCS option is simpler — it went generally available in November 2025 on AWS and Azure.&lt;/p>
&lt;p>Data flows into Snowflake primarily through Snowpipe Streaming for the initial load, with merge queries handling incremental updates. The result is structured tables in your Snowflake database — not staged JSON files, not semi-structured VARIANT columns. Actual, queryable tables with proper column types.&lt;/p>
&lt;/br>
&lt;p>The connector catalogue currently covers about 20 curated sources: databases (PostgreSQL, MySQL, SQL Server, Oracle), SaaS platforms (Salesforce, Workday, Jira, LinkedIn Ads, Meta Ads, Google Ads), cloud storage (Google Drive, SharePoint, Box), and streaming sources (Kafka, Kinesis). It&amp;rsquo;s not Fivetran&amp;rsquo;s 500+ connector library, and nobody&amp;rsquo;s pretending it is. But the sources it does cover are the ones that account for the vast majority of enterprise data movement.&lt;/p>
&lt;p>One important caveat up front: the &lt;strong>Salesforce Bulk API connector is still in preview&lt;/strong> as of March 2026. It works. Preview in Snowflake terms means the feature is functional but not yet covered by production support SLAs. Keep that in your mental model as we walk through the setup.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="setting-up-the-salesforce-connector-a-walkthrough">Setting Up the Salesforce Connector: A Walkthrough&lt;/h3>
&lt;/br>
&lt;p>This section is the step-by-step. I&amp;rsquo;m going to walk through the full setup — Salesforce configuration, Snowflake preparation, and connector deployment — with enough detail that you could follow along on your own org.&lt;/p>
&lt;/br>
&lt;h4 id="phase-1-salesforce-configuration">Phase 1: Salesforce Configuration&lt;/h4>
&lt;p>The connector uses JWT Bearer Flow for authentication, which means you need an RSA key pair and a Connected App configured in Salesforce.&lt;/p>
&lt;p>&lt;strong>Generate your RSA key pair.&lt;/strong> Open a terminal and run:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Generate 2048-bit RSA private key&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>openssl genrsa -out salesforce_private_key.pem &lt;span style="color:#ae81ff">2048&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Generate the corresponding public certificate (valid 1 year)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>openssl req -new -x509 -key salesforce_private_key.pem &lt;span style="color:#ae81ff">\
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#ae81ff">&lt;/span> -out salesforce_certificate.pem -days &lt;span style="color:#ae81ff">365&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Keep that private key safe. You&amp;rsquo;ll need it when configuring the connector in Snowflake.&lt;/p>
&lt;p>&lt;strong>Create the External Client App in Salesforce.&lt;/strong> Navigate to Setup → Apps → App Manager → New External Client App. Enable OAuth and add two scopes: &lt;code>api&lt;/code> and &lt;code>refresh_token&lt;/code> (sometimes labelled &lt;code>offline_access&lt;/code>). Enable the JWT Bearer Flow toggle and upload your &lt;code>salesforce_certificate.pem&lt;/code> public certificate. Save the app and record the &lt;strong>Consumer Key&lt;/strong> and &lt;strong>Consumer Secret&lt;/strong> that Salesforce generates.&lt;/p>
&lt;p>&lt;strong>Configure access.&lt;/strong> On the app&amp;rsquo;s Manage settings, set the Permitted Users policy to &amp;ldquo;Admin approved users are pre-authorised.&amp;rdquo; Then assign the appropriate user profiles or permission sets. The user account the connector will authenticate as needs read access to every object you intend to sync.&lt;/p>
&lt;/br>
&lt;h4 id="phase-2-snowflake-preparation">Phase 2: Snowflake Preparation&lt;/h4>
&lt;p>Before deploying the connector, you need a database, schema, role, and warehouse ready for it.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create a dedicated database for Salesforce data
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">DATABASE&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> SALESFORCE_RAW;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">SCHEMA&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> SALESFORCE_RAW.OPENFLOW;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create a connector-specific role
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">ROLE&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> OPENFLOW_SF_CONNECTOR;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Grant the role permissions to write to the destination
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">GRANT&lt;/span> &lt;span style="color:#66d9ef">USAGE&lt;/span> &lt;span style="color:#66d9ef">ON&lt;/span> &lt;span style="color:#66d9ef">DATABASE&lt;/span> SALESFORCE_RAW &lt;span style="color:#66d9ef">TO&lt;/span> &lt;span style="color:#66d9ef">ROLE&lt;/span> OPENFLOW_SF_CONNECTOR;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GRANT&lt;/span> &lt;span style="color:#66d9ef">USAGE&lt;/span> &lt;span style="color:#66d9ef">ON&lt;/span> &lt;span style="color:#66d9ef">SCHEMA&lt;/span> SALESFORCE_RAW.OPENFLOW &lt;span style="color:#66d9ef">TO&lt;/span> &lt;span style="color:#66d9ef">ROLE&lt;/span> OPENFLOW_SF_CONNECTOR;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GRANT&lt;/span> &lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> &lt;span style="color:#66d9ef">ON&lt;/span> &lt;span style="color:#66d9ef">SCHEMA&lt;/span> SALESFORCE_RAW.OPENFLOW &lt;span style="color:#66d9ef">TO&lt;/span> &lt;span style="color:#66d9ef">ROLE&lt;/span> OPENFLOW_SF_CONNECTOR;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create a warehouse for merge operations
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> WAREHOUSE &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> OPENFLOW_WH
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> WAREHOUSE_SIZE &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;SMALL&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> AUTO_SUSPEND &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">60&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> AUTO_RESUME &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">TRUE&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GRANT&lt;/span> &lt;span style="color:#66d9ef">USAGE&lt;/span> &lt;span style="color:#66d9ef">ON&lt;/span> WAREHOUSE OPENFLOW_WH &lt;span style="color:#66d9ef">TO&lt;/span> &lt;span style="color:#66d9ef">ROLE&lt;/span> OPENFLOW_SF_CONNECTOR;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GRANT&lt;/span> OPERATE &lt;span style="color:#66d9ef">ON&lt;/span> WAREHOUSE OPENFLOW_WH &lt;span style="color:#66d9ef">TO&lt;/span> &lt;span style="color:#66d9ef">ROLE&lt;/span> OPENFLOW_SF_CONNECTOR;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>For SPCS deployments, you also need a &lt;strong>network rule&lt;/strong> allowing the connector to reach your Salesforce instance. OpenFlow runs inside Snowflake&amp;rsquo;s container environment, and by default it can&amp;rsquo;t reach external endpoints without explicit permission:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Allow egress to your Salesforce instance
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> NETWORK &lt;span style="color:#66d9ef">RULE&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> OPENFLOW_SF_EGRESS
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">TYPE&lt;/span> &lt;span style="color:#f92672">=&lt;/span> HOST_PORT
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">MODE&lt;/span> &lt;span style="color:#f92672">=&lt;/span> EGRESS
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> VALUE_LIST &lt;span style="color:#f92672">=&lt;/span> (&lt;span style="color:#e6db74">&amp;#39;yourinstance.salesforce.com:443&amp;#39;&lt;/span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Wrap it in an external access integration
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">EXTERNAL&lt;/span> &lt;span style="color:#66d9ef">ACCESS&lt;/span> INTEGRATION &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> OPENFLOW_SF_ACCESS
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ALLOWED_NETWORK_RULES &lt;span style="color:#f92672">=&lt;/span> (OPENFLOW_SF_EGRESS)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ENABLED &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">TRUE&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Replace &lt;code>yourinstance.salesforce.com&lt;/code> with your actual Salesforce instance hostname. If you&amp;rsquo;re not sure what that is, log in to Salesforce and look at the URL bar — it&amp;rsquo;s the domain before &lt;code>.salesforce.com&lt;/code>.&lt;/p>
&lt;/br>
&lt;h4 id="phase-3-deploying-the-connector">Phase 3: Deploying the Connector&lt;/h4>
&lt;p>Now the fun part. Navigate to &lt;strong>Snowsight → Data → Openflow&lt;/strong> in the left sidebar. Find the &amp;ldquo;Openflow connector for Salesforce Bulk API&amp;rdquo; tile and add it to your runtime.&lt;/p>
&lt;p>This opens the NiFi canvas — a visual flow editor where the connector&amp;rsquo;s pre-built process groups appear. Right-click the Salesforce process group and select &lt;strong>Configure Parameters&lt;/strong>. Here&amp;rsquo;s where you&amp;rsquo;ll enter:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Salesforce Instance URL&lt;/strong> — e.g., &lt;code>https://yourinstance.salesforce.com&lt;/code>&lt;/li>
&lt;li>&lt;strong>Consumer Key&lt;/strong> — from the Connected App you created&lt;/li>
&lt;li>&lt;strong>Consumer Secret&lt;/strong> — same source&lt;/li>
&lt;li>&lt;strong>Private Key&lt;/strong> — the contents of &lt;code>salesforce_private_key.pem&lt;/code> (the full PEM text, headers included)&lt;/li>
&lt;li>&lt;strong>Username&lt;/strong> — the Salesforce user the connector authenticates as&lt;/li>
&lt;li>&lt;strong>Filter&lt;/strong> — the objects to sync, by API name&lt;/li>
&lt;/ul>
&lt;p>The &lt;strong>Filter&lt;/strong> parameter is where you define what to pull. List standard objects by name (&lt;code>Account&lt;/code>, &lt;code>Contact&lt;/code>, &lt;code>Opportunity&lt;/code>, &lt;code>Lead&lt;/code>, &lt;code>Task&lt;/code>, &lt;code>Event&lt;/code>) and custom objects using their API name with the &lt;code>__c&lt;/code> suffix:&lt;/p>



&lt;div class="goat svg-container ">
 
 &lt;svg
 xmlns="http://www.w3.org/2000/svg"
 font-family="Menlo,Lucida Console,monospace"
 
 viewBox="0 0 720 25"
 >
 &lt;g transform='translate(8,16)'>
&lt;text text-anchor='middle' x='0' y='4' fill='currentColor' style='font-size:1em'>A&lt;/text>
&lt;text text-anchor='middle' x='8' y='4' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='16' y='4' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='24' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='32' y='4' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='40' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='48' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='56' y='4' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='72' y='4' fill='currentColor' style='font-size:1em'>C&lt;/text>
&lt;text text-anchor='middle' x='80' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='88' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='96' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='104' y='4' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='112' y='4' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='120' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='128' y='4' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='144' y='4' fill='currentColor' style='font-size:1em'>O&lt;/text>
&lt;text text-anchor='middle' x='152' y='4' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='160' y='4' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='168' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='176' y='4' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='184' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='192' y='4' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='200' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='208' y='4' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='216' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='224' y='4' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='232' y='4' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='248' y='4' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='256' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='264' y='4' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='272' y='4' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='280' y='4' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='296' y='4' fill='currentColor' style='font-size:1em'>T&lt;/text>
&lt;text text-anchor='middle' x='304' y='4' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='312' y='4' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='320' y='4' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='328' y='4' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='344' y='4' fill='currentColor' style='font-size:1em'>E&lt;/text>
&lt;text text-anchor='middle' x='352' y='4' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='360' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='368' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='376' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='384' y='4' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='400' y='4' fill='currentColor' style='font-size:1em'>C&lt;/text>
&lt;text text-anchor='middle' x='408' y='4' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='416' y='4' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='424' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='432' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='440' y='4' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='448' y='4' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='456' y='4' fill='currentColor' style='font-size:1em'>P&lt;/text>
&lt;text text-anchor='middle' x='464' y='4' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='472' y='4' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='480' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='488' y='4' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='496' y='4' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='504' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='512' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='520' y='4' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='528' y='4' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='536' y='4' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='544' y='4' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='560' y='4' fill='currentColor' style='font-size:1em'>R&lt;/text>
&lt;text text-anchor='middle' x='568' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='576' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='584' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='592' y='4' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='600' y='4' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='608' y='4' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='616' y='4' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='624' y='4' fill='currentColor' style='font-size:1em'>T&lt;/text>
&lt;text text-anchor='middle' x='632' y='4' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='640' y='4' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='648' y='4' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='656' y='4' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='664' y='4' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='672' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='680' y='4' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='688' y='4' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='696' y='4' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='704' y='4' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;/g>

 &lt;/svg>
 
&lt;/div>
&lt;p>There&amp;rsquo;s also a &lt;strong>Special Objects Filter&lt;/strong> for objects that require different handling (like &lt;code>User&lt;/code> or &lt;code>RecordType&lt;/code>).&lt;/p>
&lt;p>Once the parameters are configured, enable all controller services in the process group, then start the connector. The initial sync will pull a full snapshot of every object in your filter list. Subsequent runs perform incremental syncs using Salesforce&amp;rsquo;s &lt;code>SystemModstamp&lt;/code> field to identify changed records.&lt;/p>
&lt;/br>
&lt;h4 id="what-you-get">What You Get&lt;/h4>
&lt;p>Once the connector runs, you&amp;rsquo;ll find one table per Salesforce object in &lt;code>SALESFORCE_RAW.OPENFLOW&lt;/code>. The tables are &lt;strong>flattened and structured&lt;/strong> — every Salesforce field becomes a column with an appropriate Snowflake data type. No VARIANT columns to parse. No nested JSON to unpack.&lt;/p>
&lt;p>An &lt;code>Account&lt;/code> table will have columns like &lt;code>ID&lt;/code>, &lt;code>NAME&lt;/code>, &lt;code>INDUSTRY&lt;/code>, &lt;code>ANNUALREVENUE&lt;/code>, &lt;code>BILLINGSTREET&lt;/code>, &lt;code>BILLINGCITY&lt;/code>, &lt;code>BILLINGSTATE&lt;/code>, &lt;code>BILLINGPOSTALCODE&lt;/code>, &lt;code>CREATEDDATE&lt;/code>, &lt;code>LASTMODIFIEDDATE&lt;/code>, and so on. Custom fields like &lt;code>Renewal_Likelihood__c&lt;/code> appear as &lt;code>RENEWAL_LIKELIHOOD__C&lt;/code>.&lt;/p>
&lt;p>Two metadata columns are added automatically: &lt;code>ISDELETED&lt;/code> (tracking Salesforce soft deletes) and the system timestamp fields. This is important — &lt;strong>hard deletes in Salesforce are not reflected&lt;/strong>. If a record is permanently deleted in Salesforce, it won&amp;rsquo;t disappear from your Snowflake table. You&amp;rsquo;ll need to handle that in your downstream dbt models if it matters for your use case.&lt;/p>
&lt;/br>
&lt;h4 id="a-note-on-multiple-sync-frequencies">A Note on Multiple Sync Frequencies&lt;/h4>
&lt;p>Not every Salesforce object changes at the same pace. Your &lt;code>Opportunity&lt;/code> table might need hourly syncs while &lt;code>Account&lt;/code> is fine with daily. OpenFlow handles this by letting you deploy &lt;strong>multiple connector instances in the same runtime&lt;/strong> — each with its own filter list and schedule — at no additional compute cost beyond the shared runtime.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="schema-evolution-where-openflow-earns-its-keep">Schema Evolution: Where OpenFlow Earns Its Keep&lt;/h3>
&lt;/br>
&lt;p>This is the feature that matters most, and the one that would have saved Marcus three weeks.&lt;/p>
&lt;p>When someone adds a new custom field to a Salesforce object — say the sales ops team creates &lt;code>Deal_Confidence_Score__c&lt;/code> on the Opportunity object — the OpenFlow connector &lt;strong>automatically detects the new field and adds a corresponding column&lt;/strong> to the Snowflake destination table on the next sync. No configuration change. No redeployment. No Teams message at 7am asking the data team to &amp;ldquo;add that new field we made yesterday.&amp;rdquo;&lt;/p>
&lt;p>The column appears, data starts flowing into it, and your dbt models can reference it whenever you&amp;rsquo;re ready.&lt;/p>
&lt;/br>
&lt;p>For field renames, OpenFlow takes a conservative approach: the old column stays in place (with stale data from the last sync before the rename), and a new column is created under the new field name. This means you&amp;rsquo;ll have both &lt;code>Old_Field_Name__c&lt;/code> and &lt;code>New_Field_Name__c&lt;/code> in your table for a period. That&amp;rsquo;s actually useful for audit purposes — you can see exactly when the rename happened by comparing timestamps — but it does mean your downstream queries need updating.&lt;/p>
&lt;p>Here&amp;rsquo;s the honest gap: &lt;strong>new custom objects are not auto-discovered&lt;/strong>. If someone creates an entirely new custom object in Salesforce, you have to manually add it to the connector&amp;rsquo;s Filter parameter. Schema evolution handles field-level changes automatically, but object-level discovery is still a manual step.&lt;/p>
&lt;p>And field type changes — say someone changes a text field to a picklist, or a number to a formula — aren&amp;rsquo;t comprehensively documented in the current OpenFlow materials. In practice, this is a rare enough occurrence that it hasn&amp;rsquo;t been a problem in my experience, but it&amp;rsquo;s worth knowing the edge case exists.&lt;/p>
&lt;/br>
&lt;p>Compare this to what Marcus had to maintain. His custom integration used a hardcoded list of fields per object. Every time a field was added, renamed, or retyped, someone had to update the Python code, test it, deploy it, and verify the data. Usually that &amp;ldquo;someone&amp;rdquo; was Marcus, usually at an inconvenient time, and usually because nobody told him the change was coming.&lt;/p>
&lt;p>The value of automatic schema evolution isn&amp;rsquo;t technical elegance. It&amp;rsquo;s that &lt;strong>your engineers stop spending time on schema babysitting and start spending it on work that actually matters&lt;/strong> — building models, improving data quality, answering the questions the business is actually asking.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="a-word-about-fivetran">A Word About Fivetran&lt;/h3>
&lt;/br>
&lt;p>I&amp;rsquo;d be doing you a disservice if I wrote about managed Salesforce connectors without mentioning Fivetran. For most of the last decade, Fivetran has been the gold standard here, and their Salesforce connector is genuinely excellent.&lt;/p>
&lt;p>Fivetran&amp;rsquo;s schema evolution handling is more mature and more configurable than OpenFlow&amp;rsquo;s current offering. They offer three schema change policies: &lt;code>ALLOW_ALL&lt;/code> (sync everything automatically, including new tables), &lt;code>ALLOW_COLUMNS&lt;/code> (auto-add new columns but not new tables), and &lt;code>BLOCK_ALL&lt;/code> (manual opt-in for everything). They also generate a &lt;code>LOG&lt;/code> table that records every schema change event — &lt;code>create_table&lt;/code>, &lt;code>alter_table&lt;/code>, &lt;code>drop_table&lt;/code> — giving you an audit trail that&amp;rsquo;s useful for debugging and compliance. Field type widening (like &lt;code>INT&lt;/code> to &lt;code>LONG&lt;/code>) happens automatically; narrowing changes are blocked to prevent data loss.&lt;/p>
&lt;p>Their pricing model is fundamentally different from OpenFlow. Fivetran charges per Monthly Active Row (MAR) — the count of distinct rows synced per connection per month. A unique primary key counts once regardless of how many times it&amp;rsquo;s updated. Approximate pricing clusters around $500 per million MAR on the Standard plan, with a $12,000 annual minimum commitment. Since early 2025, MAR is calculated per connection rather than account-wide, which eliminated the bulk discount that previously benefited multi-connector setups.&lt;/p>
&lt;/br>
&lt;p>OpenFlow, by contrast, charges for compute — SPCS credits for the container runtime, Snowpipe Streaming costs for ingestion, and warehouse costs for merge operations. There&amp;rsquo;s no per-row fee and no separate licence. It&amp;rsquo;s included with Snowflake. The trade-off is that your management compute pool runs continuously while a deployment exists, which means you&amp;rsquo;re paying a base cost even when no data is flowing. One estimate I&amp;rsquo;ve seen puts idle costs at roughly $10 per day, though this varies with configuration.&lt;/p>
&lt;p>For a single large Salesforce org, OpenFlow&amp;rsquo;s compute-based model may end up cheaper than Fivetran&amp;rsquo;s per-row pricing — especially if your Salesforce data doesn&amp;rsquo;t churn heavily. For environments where you need 30+ connectors across different source types, Fivetran&amp;rsquo;s breadth and maturity is hard to argue with. They have 500+ connectors. OpenFlow has about 20.&lt;/p>
&lt;p>The honest framing: &lt;strong>OpenFlow is the better choice if you&amp;rsquo;re already deeply invested in Snowflake, value platform consolidation, and your primary sources are in the current connector catalogue.&lt;/strong> Fivetran is the safer choice if you need broad connector coverage, battle-tested reliability, and you&amp;rsquo;d rather pay a premium for someone else to handle the ops.&lt;/p>
&lt;p>They&amp;rsquo;re not mutually exclusive, either. I&amp;rsquo;ve seen teams run Fivetran for the long tail of small connectors while using OpenFlow for their highest-volume sources where compute-based pricing wins.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-zero-copy-alternative">The Zero-Copy Alternative&lt;/h3>
&lt;/br>
&lt;p>There&amp;rsquo;s a third path I haven&amp;rsquo;t mentioned yet, and it&amp;rsquo;s worth understanding even if it&amp;rsquo;s not right for every team: &lt;strong>zero-copy data federation&lt;/strong>.&lt;/p>
&lt;p>Salesforce Data Cloud offers a zero-copy integration with Snowflake that flips the entire model on its head. Instead of extracting data from Salesforce and loading it into your warehouse, zero-copy gives you direct read access to Salesforce data &lt;em>without moving it at all&lt;/em>. Salesforce&amp;rsquo;s query pushdown engine handles the work — when you query the federated data, Salesforce pushes the query to the source, filters and aggregates there, and returns only the results you need. No pipelines to build. No schemas to manage. No sync schedules to configure.&lt;/p>
&lt;p>It works bidirectionally, too. You can share data from Snowflake back into Salesforce Data Cloud, letting sales reps see warehouse-enriched signals directly on Account and Opportunity pages without anyone exporting a CSV or building a reverse ETL pipeline. The integration runs on Apache Iceberg under the hood, which at least means the underlying format is open.&lt;/p>
&lt;/br>
&lt;p>For handling new custom objects — the gap I flagged with OpenFlow — zero-copy sidesteps the problem entirely. There&amp;rsquo;s nothing to discover because there&amp;rsquo;s nothing to sync. The data stays where it is, and you access it in place. As your Salesforce org evolves, the federated view evolves with it.&lt;/p>
&lt;p>That&amp;rsquo;s genuinely appealing. For proof-of-concept work, rapid prototyping, or use cases where you need real-time access to Salesforce data without building infrastructure, zero-copy is fast and effective.&lt;/p>
&lt;/br>
&lt;p>But here&amp;rsquo;s where I get cautious.&lt;/p>
&lt;p>&lt;strong>It comes at a premium.&lt;/strong> Salesforce Data Cloud runs on a consumption credit model — credits are purchased in bundles (roughly $500 per 100,000 credits at list price), and every action burns them. Zero-copy federation queries consume about 70 credits per million records, which sounds reasonable until you&amp;rsquo;re running it at scale across multiple business units. The pricing model has simplified since late 2025 — Salesforce consolidated multiple credit types into a single fungible credit and made ingestion from core Salesforce products free — but it&amp;rsquo;s still a consumption model with real costs that compound quickly if you&amp;rsquo;re not watching.&lt;/p>
&lt;p>More importantly, &lt;strong>zero-copy creates a deep platform dependency&lt;/strong>. Your data access is mediated entirely through Salesforce and Snowflake&amp;rsquo;s partnership. If Salesforce changes their terms, adjusts their pricing (which they&amp;rsquo;ve done repeatedly), or if you decide to move off Snowflake to Databricks or BigQuery, that zero-copy integration doesn&amp;rsquo;t come with you. You&amp;rsquo;d need to build actual data movement infrastructure — the thing you avoided — under time pressure and with no existing pipeline to fall back on.&lt;/p>
&lt;/br>
&lt;p>I value keeping infrastructure portable. Not because I&amp;rsquo;m paranoid about vendor lock-in — I&amp;rsquo;m pragmatic about it. I&amp;rsquo;ve been in this industry long enough to see what happens when a new CEO arrives and wants to renegotiate every vendor contract, or when a cloud provider changes their pricing structure overnight, or when the company you depend on gets acquired and the roadmap shifts. Having your data physically in your own warehouse, in formats you control, with pipelines you can redirect to a different destination — that&amp;rsquo;s insurance. Not glamorous insurance, but the kind you&amp;rsquo;re grateful for when you need it.&lt;/p>
&lt;p>The balance is real, though. You always have to weigh portability against the time you spend building and maintaining pipelines. If your team is drowning in pipeline maintenance, zero-copy might buy you breathing room while you stand up proper infrastructure. Just go in with your eyes open about what you&amp;rsquo;re trading for that convenience.&lt;/p>
&lt;/br>
&lt;p>There&amp;rsquo;s a more fundamental concern, though, and it&amp;rsquo;s the one that won&amp;rsquo;t appear in any vendor comparison chart.&lt;/p>
&lt;p>&lt;strong>Zero-copy makes it dangerously easy to skip the data warehouse entirely.&lt;/strong> When you give analysts and report builders direct read access to raw Salesforce data, the temptation to build dashboards straight from it is enormous. And for a quick proof of concept or a one-off analysis, fine. But the moment those PoC dashboards become &amp;ldquo;the dashboard the VP checks every Monday,&amp;rdquo; you&amp;rsquo;ve got a problem.&lt;/p>
&lt;p>Raw source data doesn&amp;rsquo;t tell you which &lt;code>CustomerNo&lt;/code> field to use — the one from the legacy migration, the one from the 2023 CRM consolidation, or the one from the current integration. It doesn&amp;rsquo;t encode the business rule that says &amp;ldquo;Closed Won&amp;rdquo; opportunities in APAC exclude training deals under $5,000 because those are tracked separately. It doesn&amp;rsquo;t flag that the &lt;code>Last_Activity_Date&lt;/code> field is unreliable for accounts owned by the partner team because they log activities in a different system.&lt;/p>
&lt;/br>
&lt;p>That knowledge lives in your data warehouse layer. It lives in the dbt models that your team built over months of conversations with stakeholders, through change management cycles, through debugging sessions where someone finally said &amp;ldquo;oh, we stopped using that field in Q3 because&amp;hellip;&amp;rdquo; That&amp;rsquo;s not transformation logic — it&amp;rsquo;s &lt;em>institutional memory encoded as code&lt;/em>. It connects raw data across domains and gives the business a full landscape picture instead of a narrow, single-source view. It catches the behavioural anomalies unique to your organisation — the sales rep who bulk-updates 500 records every Friday afternoon, the integration that occasionally double-fires during DST transitions, the custom object that gets repurposed every time the business restructures.&lt;/p>
&lt;p>So by all means, evaluate zero-copy for the use cases where it shines. Real-time signals on Account pages for sales reps? Great fit. Federated access for a specific analytics team that knows the source data intimately? Reasonable. But don&amp;rsquo;t let it replace the warehouse layer. That layer is where the hard-won understanding of your data lives, and it&amp;rsquo;s harder to recreate than any pipeline.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-honest-trade-offs">The Honest Trade-Offs&lt;/h3>
&lt;/br>
&lt;p>I&amp;rsquo;ve been positive about OpenFlow because I think it solves a real problem well. But I&amp;rsquo;d be violating my own writing principles if I didn&amp;rsquo;t lay out the limitations clearly.&lt;/p>
&lt;p>&lt;strong>The Salesforce connector is still in preview.&lt;/strong> It works, but it&amp;rsquo;s not covered by production support SLAs. If you&amp;rsquo;re running a mission-critical pipeline that your CFO stares at every Monday morning, that matters.&lt;/p>
&lt;p>&lt;strong>Custom Salesforce domains aren&amp;rsquo;t supported.&lt;/strong> If your org uses a vanity URL (like &lt;code>mycompany.my.salesforce.com&lt;/code> with a custom domain), check the documentation carefully before committing.&lt;/p>
&lt;p>&lt;strong>Hard deletes aren&amp;rsquo;t tracked.&lt;/strong> The &lt;code>ISDELETED&lt;/code> column catches soft deletes, but records that are permanently purged from Salesforce will persist in your Snowflake tables indefinitely. You&amp;rsquo;ll need a reconciliation process if that matters for your use case.&lt;/p>
&lt;p>&lt;strong>Certain field types are silently dropped.&lt;/strong> &lt;code>location&lt;/code>, &lt;code>address&lt;/code> (compound), and &lt;code>base64&lt;/code> fields are not synced. The connector doesn&amp;rsquo;t error on these — it just ignores them. This is the kind of thing that bites you three months in when someone asks why geographic data isn&amp;rsquo;t in the warehouse.&lt;/p>
&lt;p>&lt;strong>Formula fields require full refresh.&lt;/strong> Because formula field values are computed server-side and don&amp;rsquo;t update &lt;code>SystemModstamp&lt;/code>, they can&amp;rsquo;t be synced incrementally. You need a separate connector instance running full refreshes to capture formula field changes. That&amp;rsquo;s not a bug — it&amp;rsquo;s how Salesforce works — but it&amp;rsquo;s a gotcha if you&amp;rsquo;re not expecting it.&lt;/p>
&lt;p>&lt;strong>No relationship traversal.&lt;/strong> You can&amp;rsquo;t configure the connector to follow lookups and pull related objects automatically. Each object is pulled independently. Joining them is your job in the transformation layer, which is where it belongs anyway if you&amp;rsquo;re running dbt.&lt;/p>
&lt;/br>
&lt;p>None of these are dealbreakers. But every one of them is the kind of thing that bites an engineer at 4pm on a Friday if they weren&amp;rsquo;t expecting it. Now you&amp;rsquo;re expecting it.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="getting-started">Getting Started&lt;/h3>
&lt;/br>
&lt;p>If you want to try this on your own Salesforce org, here&amp;rsquo;s my recommended order of operations:&lt;/p>
&lt;p>Start with a &lt;strong>non-production Salesforce sandbox&lt;/strong> and a Snowflake trial account if you don&amp;rsquo;t have a dev environment handy. Pick two or three standard objects — &lt;code>Account&lt;/code>, &lt;code>Contact&lt;/code>, &lt;code>Opportunity&lt;/code> — and one custom object if your org has them. Run through the three-phase setup I described above. Get data flowing, verify the table structures, and then add a custom field to one of the objects in Salesforce. Wait for the next sync. Watch the column appear automatically in Snowflake.&lt;/p>
&lt;p>That&amp;rsquo;s the moment it clicks. That&amp;rsquo;s the moment you stop thinking about managed connectors as a luxury and start thinking about custom API code as technical debt you&amp;rsquo;re choosing to carry.&lt;/p>
&lt;p>Once you&amp;rsquo;ve validated the core flow, layer on your dbt models. OpenFlow lands raw data — it doesn&amp;rsquo;t transform it. Your staging models handle naming conventions, type casting, soft delete filtering, and the join logic that stitches related objects together. This is the same pattern you&amp;rsquo;d use with Fivetran or any other EL tool: land it raw, transform it in the warehouse.&lt;/p>
&lt;/br>
&lt;p>For teams already running Fivetran for Salesforce, I&amp;rsquo;m not suggesting you rip it out tomorrow. If it&amp;rsquo;s working and the cost is acceptable, that&amp;rsquo;s a solved problem. But the next time you&amp;rsquo;re adding a new high-volume source that&amp;rsquo;s in OpenFlow&amp;rsquo;s catalogue — especially if you&amp;rsquo;re already paying for Snowflake — it&amp;rsquo;s worth running the numbers. You might find that native integration with compute-based pricing is the better deal.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-real-why">The Real Why&lt;/h3>
&lt;/br>
&lt;p>I started this article with Marcus&amp;rsquo;s story because it illustrates something I feel strongly about: &lt;strong>your engineers&amp;rsquo; time is the most expensive resource in your data organisation, and spending it on solved problems is a leadership failure&lt;/strong>.&lt;/p>
&lt;p>Building a custom Salesforce integration isn&amp;rsquo;t impressive. It was impressive in 2016, when the tooling didn&amp;rsquo;t exist. Today, it&amp;rsquo;s a choice to take on maintenance burden that a managed service handles automatically. It&amp;rsquo;s choosing to babysit schema changes instead of building the models and analyses that actually move the business forward.&lt;/p>
&lt;p>Marcus is a senior engineer now. He builds dimensional models and designs data products that directly influence how the sales team allocates resources. He doesn&amp;rsquo;t write API integration code anymore. Not because he can&amp;rsquo;t — because his time is worth more than that.&lt;/p>
&lt;/br>
&lt;p>But here&amp;rsquo;s what I keep coming back to. The tools keep getting better — OpenFlow, Fivetran, zero-copy, and yes, AI-assisted pipeline generation. Every year there&amp;rsquo;s a new thing that promises to automate another piece of the data engineering workflow. And many of them genuinely do.&lt;/p>
&lt;p>What they don&amp;rsquo;t automate is the part that actually matters.&lt;/p>
&lt;p>No tool — not OpenFlow, not Fivetran, not an AI agent — is going to sit in a room with your head of sales and ask &amp;ldquo;what keeps you up at night?&amp;rdquo; No connector is going to notice that the way your company defines &amp;ldquo;active customer&amp;rdquo; has quietly drifted from what the CRM tracks. No pipeline is going to push back and say &amp;ldquo;we could build that dashboard, but the metric you&amp;rsquo;re asking for doesn&amp;rsquo;t answer the question you actually have.&amp;rdquo;&lt;/p>
&lt;/br>
&lt;p>That&amp;rsquo;s the connective tissue between raw data and business value. It&amp;rsquo;s a person who understands the domain, who&amp;rsquo;s been present through the change management cycles, who&amp;rsquo;s built relationships with the stakeholders and knows which &lt;code>CustomerNo&lt;/code> field to use because they were in the room when the decision was made three years ago. It&amp;rsquo;s someone who asks clarifying questions instead of presuming they already know the answer.&lt;/p>
&lt;p>AI will keep getting better at the mechanical parts of data engineering. Schema detection, pipeline code generation, anomaly detection — those are well-defined problems that automation is suited for. But the assumption that any tool can skip the human step — the clarification, the context, the judgment about what&amp;rsquo;s actually worth building — that&amp;rsquo;s where I&amp;rsquo;ve seen projects go sideways. Not because the technology failed, but because nobody asked the right questions before building.&lt;/p>
&lt;/br>
&lt;p>OpenFlow isn&amp;rsquo;t perfect. The connector catalogue is young, the Salesforce connector is in preview, and Fivetran still wins on breadth and battle-hardened maturity. But OpenFlow represents something important: the data platform taking responsibility for data movement, not just data storage and compute. It&amp;rsquo;s Snowflake saying &amp;ldquo;we&amp;rsquo;ll handle getting the data in — you focus on making it useful.&amp;rdquo;&lt;/p>
&lt;p>That last part — &lt;em>making it useful&lt;/em> — is still your job. And it&amp;rsquo;s the part that no tool can do for you. The best thing a managed connector gives you isn&amp;rsquo;t fewer lines of code. It&amp;rsquo;s time back. Time to spend on the work that actually requires a human: understanding the business, building the right models, and making sure the data tells an accurate story.&lt;/p>
&lt;p>The next time someone on your team says &amp;ldquo;I&amp;rsquo;ll just build a quick Salesforce integration&amp;rdquo; — send them this article. And the next time someone says &amp;ldquo;AI can just handle the data pipeline&amp;rdquo; — ask them who&amp;rsquo;s going to sit with the stakeholders and figure out what the pipeline should actually deliver.&lt;/p>
&lt;p>That&amp;rsquo;s still you. Make sure you have the time for it.&lt;/p>
&lt;p>&lt;br>&lt;br>&lt;/p></content:encoded><category>Data Engineering</category><category>Platforms &amp; Cloud</category><category>Data Engineering</category><category>Snowflake</category><category>OpenFlow</category><category>Salesforce</category><category>API Integration</category><category>Schema Evolution</category><category>Fivetran</category><category>Data Pipelines</category></item><item><title>That Tuesday Morning When I Finally Fixed Our Ten-Minute Queries</title><link>https://ghostinthedata.info/posts/2026/2026-01-16-aws-dimensional-modelling/</link><pubDate>Fri, 16 Jan 2026 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-01-16-aws-dimensional-modelling/</guid><author>Chris Hillman</author><description>How Kimball dimensional modelling, proper job orchestration, and Write-Audit-Publish patterns transformed our data pipeline from a nightmare into something that actually works</description><content:encoded>&lt;h3 id="the-ten-minute-query">The Ten-Minute Query&lt;/h3>
&lt;hr>
&lt;p>I&amp;rsquo;m sitting at my laptop on a Tuesday morning, waiting. The progress bar on my screen says &amp;lsquo;Query running&amp;hellip; 4 minutes, 37 seconds.&amp;rsquo; I lean back in my chair and let out this long sigh that probably says more than I intended.&lt;/p>
&lt;p>My manager walks past my desk. She glances at my screen, and I can see that look—the one that says she already knows what I&amp;rsquo;m about to tell her. I didn&amp;rsquo;t need to explain.&lt;/p>
&lt;p>&amp;ldquo;Still pulling that customer report?&amp;rdquo;&lt;/p>
&lt;p>&amp;ldquo;Yeah.&amp;rdquo;&lt;/p>
&lt;p>She shakes her head. &amp;ldquo;We need to fix this.&amp;rdquo;&lt;/p>
&lt;p>It was an ad-hoc piece of work that we&amp;rsquo;d been running for months. But I just nod.&lt;/p>
&lt;p>Here&amp;rsquo;s what was happening: every time someone needed to know &amp;lsquo;how many orders did customer X place?&amp;rsquo; or &amp;lsquo;which products sold best last month?&amp;rsquo;, we&amp;rsquo;d query this massive table. Three million rows. Every single customer&amp;rsquo;s name, address, email—repeated on every order row. The system was scanning through all that redundant data every single time.&lt;/p>
&lt;p>And then, two weeks ago, it got worse. Someone ran a data import script that had a bug. It replaced half our customer names with NULL. By the time we caught it, twelve reports had already gone out with blank customer names. I had to send apology emails. My team and I had to stay until midnight fixing it.&lt;/p>
&lt;p>I thought, &lt;em>there has to be a better way to do this&lt;/em>. People have been building data systems for decades—someone must have figured this out.&lt;/p>
&lt;p>And they have. His name is Ralph Kimball, and his dimensional modelling methodology has been the gold standard for data warehouses since the 1990s.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-problem-i-couldnt-unsee">The Problem I Couldn&amp;rsquo;t Unsee&lt;/h3>
&lt;hr>
&lt;p>Let me show you what our data looked like.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-text" data-lang="text">&lt;span style="display:flex;">&lt;span>order_id | date | customer_name | address | email | product | amount
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>---------|------------|---------------|--------------|--------------|---------|-------
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>1 | 2024-11-15 | John Smith | 123 Main St | john@... | Pizza | 25
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>2 | 2024-11-15 | John Smith | 123 Main St | john@... | Soda | 3
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>3 | 2024-11-15 | Jane Doe | 456 Oak Ave | jane@... | Pizza | 25
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>4 | 2024-11-16 | John Smith | 123 Main St | john@... | Burger | 12
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>See what&amp;rsquo;s happening? John Smith&amp;rsquo;s address and email are stored three times. Multiply that by thousands of customers and millions of orders, and you&amp;rsquo;ve got:&lt;/p>
&lt;ul>
&lt;li>Every query scanning way more data than necessary&lt;/li>
&lt;li>Storage costs that make your CFO unhappy&lt;/li>
&lt;li>One bug corrupting your entire history&lt;/li>
&lt;li>Query times that make you question your career choices&lt;/li>
&lt;/ul>
&lt;p>Every single query was like asking someone to find your keys by searching every room in your house, including the ones you never use. Sure, they&amp;rsquo;ll find the keys eventually. But there&amp;rsquo;s a better way.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-kitchen-that-changed-everything">The Kitchen That Changed Everything&lt;/h3>
&lt;hr>
&lt;p>Let me paint you a hypothetical scenario. Imagine it&amp;rsquo;s a Saturday, and you&amp;rsquo;re standing in your kitchen after just moving house. You can&amp;rsquo;t find anything. Everything&amp;rsquo;s still in boxes.&lt;/p>
&lt;p>You&amp;rsquo;re rummaging through the third box, looking for a spatula. You&amp;rsquo;re pulling out random stuff—a whisk, some Tupperware lids, a pizza cutter, yesterday&amp;rsquo;s takeout containers, a bag of flour. Everything mixed together. This is chaos.&lt;/p>
&lt;p>Now imagine a friend comes over. They take one look at your kitchen and say, &amp;ldquo;What is this?&amp;rdquo;&lt;/p>
&lt;p>&amp;ldquo;It&amp;rsquo;s&amp;hellip; I haven&amp;rsquo;t unpacked yet.&amp;rdquo;&lt;/p>
&lt;p>&amp;ldquo;You&amp;rsquo;ve been here a week.&amp;rdquo; They walk over to the mess. &amp;ldquo;Here&amp;rsquo;s what you should do. The stuff that rarely changes—your plates, your pots, your cutting boards—those go in the cabinets. Organised, labelled, easy to find. The stuff that comes and goes constantly—the groceries, the leftovers, what you&amp;rsquo;re cooking today—that goes in the fridge and on the counter. Don&amp;rsquo;t mix the permanent stuff with the temporary stuff.&amp;rdquo;&lt;/p>
&lt;p>Two hours later, the kitchen makes sense. You can find anything in seconds. And here&amp;rsquo;s the wild part—it actually takes up less space because you&amp;rsquo;re not duplicating storage for the same items.&lt;/p>
&lt;p>That&amp;rsquo;s dimensional modelling.&lt;/p>
&lt;p>When your data is like those boxes, everything&amp;rsquo;s mixed together. Customer information sitting right next to order information, repeated over and over. Every time you need something, you&amp;rsquo;re digging through all the boxes.&lt;/p>
&lt;p>Dimensional modelling says: let&amp;rsquo;s organise this. Put stable descriptive data—things that rarely change—in one place. We call these &lt;strong>dimension tables&lt;/strong>. Put the events that happen constantly—the transactions, the updates, the measurements—in another place. We call these &lt;strong>fact tables&lt;/strong>. Link them with keys.&lt;/p>
&lt;p>Here&amp;rsquo;s what that transformation looks like:&lt;/p>
&lt;p>BEFORE (Messy Boxes - everything mixed together):&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-text" data-lang="text">&lt;span style="display:flex;">&lt;span>order_id | customer_name | address | product | amount
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>1 | John Smith | 123 Main St | Pizza | 25
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>2 | John Smith | 123 Main St | Soda | 3
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>↑ John&amp;rsquo;s details repeated every single order!&lt;/p>
&lt;p>AFTER (Organised Kitchen):
DIM_CUSTOMER (Cabinets - stable stuff, stored ONCE):&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-text" data-lang="text">&lt;span style="display:flex;">&lt;span>customer_key | customer_id | name | address | email
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>1001 | C-123 | John Smith | 123 Main St | john@...
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>1002 | C-456 | Jane Doe | 456 Oak Ave | jane@...
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>DIM_DATE (Calendar on the wall - one row per day):&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-text" data-lang="text">&lt;span style="display:flex;">&lt;span>date_key | full_date | day_name | month | quarter | year | is_weekend
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>20241115 | 2024-11-15 | Friday | Nov | Q4 | 2024 | N
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>20241116 | 2024-11-16 | Saturday | Nov | Q4 | 2024 | Y
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>FACT_ORDERS (Fridge/counter - events that happen constantly):&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-text" data-lang="text">&lt;span style="display:flex;">&lt;span>order_key | date_key | customer_key | product_key | amount
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>1 | 20241115 | 1001 | 5 | 25
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>2 | 20241115 | 1001 | 8 | 3
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Customer details stored one time. Not a thousand times. When you need to answer &amp;lsquo;how much did we sell?&amp;rsquo;, you scan the compact fact table. When you need &amp;lsquo;which customers bought?&amp;rsquo;, you JOIN to get the descriptive details.&lt;/p>
&lt;p>Notice something important in that structure? The dimension tables have two types of keys:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>customer_key&lt;/strong> (1001, 1002) — This is a &lt;em>surrogate key&lt;/em>. A meaningless integer we generate ourselves.&lt;/li>
&lt;li>&lt;strong>customer_id&lt;/strong> (C-123, C-456) — This is the &lt;em>natural key&lt;/em>. The ID from the source system.&lt;/li>
&lt;/ul>
&lt;p>Why both? Because source systems are chaotic. They recycle IDs. They change formats. They get merged in acquisitions. The surrogate key is &lt;em>our&lt;/em> key—stable, predictable, and under our control. Every dimension table gets a surrogate key as its primary key.&lt;/p>
&lt;p>And that date dimension? It&amp;rsquo;s &amp;ldquo;the most important dimension.&amp;rdquo; Every fact table should have one. It lets you slice data by day of week, month, quarter, fiscal period, holidays—without calculating these on every query.&lt;/p>
&lt;p>The core principle: things that rarely change go in dedicated storage, organised and easy to find. Things that happen constantly go in their own space, linked back by keys. Same rule applies to data:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Dimensions&lt;/strong> = Stable descriptive stuff (customers, products, locations, dates)&lt;/li>
&lt;li>&lt;strong>Facts&lt;/strong> = Events that happen constantly (orders, clicks, updates, measurements)&lt;/li>
&lt;/ul>
&lt;/br>
&lt;/br>
&lt;h3 id="the-conference-room-ill-never-forget">The Conference Room I&amp;rsquo;ll Never Forget&lt;/h3>
&lt;hr>
&lt;p>I&amp;rsquo;m sitting in a conference room. Fluorescent lights buzzing. That weird hum they make when it&amp;rsquo;s too quiet—I&amp;rsquo;m wondering why we haven&amp;rsquo;t changed to LEDs. My manager is standing at the whiteboard, and she&amp;rsquo;s pissed.&lt;/p>
&lt;p>&amp;ldquo;Someone,&amp;rdquo; she says, not looking at anyone in particular, &amp;ldquo;pushed bad data to production last night.&amp;rdquo;&lt;/p>
&lt;p>I feel my stomach drop. The data pipeline my team had been working on.&lt;/p>
&lt;p>&amp;ldquo;The revenue report that went to the board this morning showed every transaction as zero dollars. Zero. The board emailed me asking if the company made any money last quarter.&amp;rdquo;&lt;/p>
&lt;p>Oh god. Oh no. I sank down in my chair a little.&lt;/p>
&lt;p>Turns out it wasn&amp;rsquo;t actually our release—it was someone else&amp;rsquo;s. But in that moment, sitting there, I realised something crucial: we had no safety net. Data went straight from transformation to production. No checks. No validation. If you made a mistake, everyone saw it immediately.&lt;/p>
&lt;p>That conference room moment stuck with me. I started researching how other teams prevented this kind of disaster. That&amp;rsquo;s when I learned about Write-Audit-Publish. And honestly? It&amp;rsquo;s probably saved me from that same sick feeling countless times since.&lt;/p>
&lt;p>Here&amp;rsquo;s the idea:&lt;/p>
&lt;p>&lt;strong>WRITE&lt;/strong> — You finish your transformation. Instead of putting the data straight into production, you write it to a staging branch. Think of it like a draft folder for emails. It&amp;rsquo;s there, but it&amp;rsquo;s not official yet.&lt;/p>
&lt;p>&lt;strong>AUDIT&lt;/strong> — This is the crucial part. You run every check you can think of:&lt;/p>
&lt;ul>
&lt;li>Did I lose any rows?&lt;/li>
&lt;li>Are there any NULL values where there shouldn&amp;rsquo;t be?&lt;/li>
&lt;li>Do all the foreign keys match up?&lt;/li>
&lt;li>Does the math make sense?&lt;/li>
&lt;li>Do surrogate keys have duplicates?&lt;/li>
&lt;/ul>
&lt;p>It&amp;rsquo;s like re-reading an important email before you hit send. You check for typos, you verify the recipients, you make sure the attachment is right.&lt;/p>
&lt;p>&lt;strong>Important:&lt;/strong> In the future when you find incidents, bugs, or problems—add them to this AUDIT layer. Ensure you don&amp;rsquo;t make the same mistakes twice.&lt;/p>
&lt;p>&lt;strong>PUBLISH&lt;/strong> — Only if everything passes, you merge the branch to production. If anything fails, you stop. You fix it. You try again. The bad data never reaches production. Never reaches that board meeting.&lt;/p>
&lt;p>The traditional way to implement this involves copying data between folders or tables. But there&amp;rsquo;s a better approach using Apache Iceberg&amp;rsquo;s branching feature. Instead of physically copying data, you create a branch—like in Git. Your audit runs against the branch. If it passes, you merge the branch to main. If it fails, you drop the branch. No data was ever at risk.&lt;/p>
&lt;p>&lt;strong>Critical design decision:&lt;/strong> WAP should happen at &lt;em>each table level&lt;/em>, not just at the end of the pipeline. Each dimension job does its own Write-Audit-Publish cycle. Each fact job does its own. This way:&lt;/p>
&lt;ul>
&lt;li>Failures are isolated to the specific table that failed&lt;/li>
&lt;li>You catch issues early, not at the very end&lt;/li>
&lt;li>Each job is self-contained and independently testable&lt;/li>
&lt;li>Downstream jobs read from published (main) tables, not staging branches&lt;/li>
&lt;/ul>
&lt;p>Let me show you what this looks like when an audit catches something:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-text" data-lang="text">&lt;span style="display:flex;">&lt;span>AUDIT PHASE (dim_system):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>Row count check.......................... PASS
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> Source: 1,198 rows
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> Target: 1,198 rows
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>Primary key uniqueness................... FAIL
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> Found 13 duplicate system_keys in dim_system!
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>AUDIT FAILED — BRANCH NOT MERGED
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>Data remains in staging branch.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>Production table unchanged.
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>Fix the duplicates and re-run the job.
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Look at that. The audit caught duplicate surrogate keys before they reached production. Nobody sees this except you. You fix the problem. You run it again. Then it passes, and the branch merges to main.&lt;/p>
&lt;p>This is Write-Audit-Publish. It&amp;rsquo;s the difference between sending an email with a typo versus catching it first. Between serving undercooked food versus checking with a thermometer. Between publishing bad data versus validating before it matters.&lt;/p>
&lt;p>That conference room moment? Doesn&amp;rsquo;t happen with this pattern.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="building-this-thing-for-real">Building This Thing for Real&lt;/h3>
&lt;hr>
&lt;p>Alright, enough theory. Let me show you how to actually build this in AWS.&lt;/p>
&lt;p>We&amp;rsquo;re going to use real (game) space exploration data—systems, planets, updates. It&amp;rsquo;s way more interesting than order data, and the patterns work exactly the same.&lt;/p>
&lt;p>&lt;strong>In this section:&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>&lt;a href="#architecture-overview">Architecture Overview&lt;/a>&lt;/li>
&lt;li>&lt;a href="#step-1-setting-up-s3-buckets">Step 1: Setting Up S3 Buckets&lt;/a>&lt;/li>
&lt;li>&lt;a href="#step-2-creating-the-iceberg-tables">Step 2: Creating the Iceberg Tables&lt;/a>&lt;/li>
&lt;li>&lt;a href="#the-heart-of-the-system-orchestrated-glue-jobs-with-per-table-wap">The Glue Jobs (with Per-Table WAP)&lt;/a>&lt;/li>
&lt;li>&lt;a href="#step-functions-workflow">Step Functions Workflow&lt;/a>&lt;/li>
&lt;li>&lt;a href="#dependency-triggers-with-s3-events">Dependency Triggers with S3 Events&lt;/a>&lt;/li>
&lt;/ul>
&lt;h4 id="architecture-overview">Architecture Overview&lt;/h4>
&lt;p>Here&amp;rsquo;s our architecture:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-text" data-lang="text">&lt;span style="display:flex;">&lt;span> +------------------------------------------+
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> | AWS Step Functions Workflow |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> +------------------------------------------+
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> +----------------------------+----------------------------+
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> | | |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> v v v
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> +------------+ +------------+ +------------+
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> | Glue Job: | | Glue Job: | | Glue Job: |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> | dim_date | | dim_system | | dim_planet |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> | (WAP) | | (WAP) | | (WAP) |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> +------------+ +------------+ +------------+
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> | | |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> +----------------------------+----------------------------+
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (Wait for all dimensions)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> v
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> +------------------------+
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> | Glue Job: |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> | fact_system_updates |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> | (WAP + Incremental) |
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> +------------------------+
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Data Flow:&lt;/strong>&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Landing Layer&lt;/th>
 &lt;th>Staging Layer (PSA)&lt;/th>
 &lt;th>Semantic Layer&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>Raw 1:1 from source&lt;/td>
 &lt;td>Iceberg + data_date partitioning&lt;/td>
 &lt;td>Production dimensional model&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;code>systems/&lt;/code>&lt;/td>
 &lt;td>&lt;code>stg_systems/&lt;/code>&lt;/td>
 &lt;td>&lt;code>dim_date&lt;/code>&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;code>planets/&lt;/code>&lt;/td>
 &lt;td>&lt;code>stg_planets/&lt;/code>&lt;/td>
 &lt;td>&lt;code>dim_system&lt;/code>&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;code>updates/&lt;/code>&lt;/td>
 &lt;td>&lt;code>stg_updates/&lt;/code>&lt;/td>
 &lt;td>&lt;code>dim_planet&lt;/code>&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;/td>
 &lt;td>&lt;/td>
 &lt;td>&lt;code>fact_system_updates&lt;/code>&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>Let me explain the three layers:&lt;/p>
&lt;p>&lt;strong>Landing Layer&lt;/strong> — Raw data, 1:1 from source systems. No transformations except what&amp;rsquo;s needed to land it. Contains only the &lt;em>latest&lt;/em> fetch. If sourcing fails, this layer fails. Get the data close to you as quickly as possible (into your ecosystem). Typically all the files will be string or varchar to mitigate data type issues from a source system.&lt;/p>
&lt;p>&lt;strong>Staging Layer (Persistent Staging Area)&lt;/strong> — This is where we convert data types and prepare for dimensional modelling. Critically, we partition by &lt;code>data_date&lt;/code>—the date the source data was fetched. This gives us:&lt;/p>
&lt;ul>
&lt;li>Historical reprocessing capability (reload any past date)&lt;/li>
&lt;li>Debugging (what did the data look like on January 15th?)&lt;/li>
&lt;li>Dependency triggers (when a partition is written, trigger downstream jobs)&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Semantic Layer&lt;/strong> — Production-ready dimensional model. This is where business users query. Tables are partitioned appropriately for query performance.&lt;/p>
&lt;p>Also notice the job structure: &lt;strong>each job does its own WAP cycle&lt;/strong>. Dimensions build in parallel, each publishing to main when their audits pass. Fact tables wait for all dimensions to publish, then do their own WAP cycle. This ensures:&lt;/p>
&lt;ul>
&lt;li>Fact tables read from &lt;em>published&lt;/em> dimensions (main branch), not staging branches&lt;/li>
&lt;li>Foreign key lookups use production-ready surrogate keys&lt;/li>
&lt;li>Each job is independently testable&lt;/li>
&lt;/ul>
&lt;h4 id="step-1-setting-up-s3-buckets">Step 1: Setting Up S3 Buckets&lt;/h4>
&lt;p>I&amp;rsquo;m in the S3 console. Click &amp;ldquo;Create bucket&amp;rdquo;. I&amp;rsquo;m calling mine &lt;code>space-data-pipeline-demo&lt;/code>.&lt;/p>
&lt;p>Important settings:&lt;/p>
&lt;ul>
&lt;li>Region: ap-southeast-2 (Sydney—pick one close to you)&lt;/li>
&lt;li>Block all public access: Yes&lt;/li>
&lt;li>Versioning: Enabled&lt;/li>
&lt;li>Default encryption: Enabled&lt;/li>
&lt;/ul>
&lt;p>Inside this bucket:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-text" data-lang="text">&lt;span style="display:flex;">&lt;span>space-data-pipeline-demo/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>├── landing/ ← Raw 1:1 from source (latest only)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>│ ├── systems/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>│ ├── planets/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>│ └── updates/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>├── staging/ ← PSA with data_date partitioning
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>│ ├── stg_systems/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>│ ├── stg_planets/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>│ └── stg_updates/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>└── semantic/ ← Production dimensional model
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ├── dim_date/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ├── dim_system/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ├── dim_planet/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> └── fact_system_updates/
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="step-2-creating-the-iceberg-tables">Step 2: Creating the Iceberg Tables&lt;/h4>
&lt;p>Here&amp;rsquo;s the DDL for our dimensional model. Notice the partitioning strategies and SCD Type 2 columns.&lt;/p>
&lt;details>
&lt;summary>&lt;strong>Click to expand: dim_date DDL&lt;/strong>&lt;/summary>
&lt;p>This is a reference dimension—typically loaded once with all dates, then rarely updated. Partitioned by year for query performance.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> space_exploration.dim_date (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> date_key INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> full_date DATE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> day_of_week STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> day_of_month INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> day_of_year INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> week_of_year INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> month_number INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> month_name STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> quarter INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">year&lt;/span> INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> is_weekend STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> is_holiday STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> fiscal_quarter INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> fiscal_year INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Audit columns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> load_timestamp &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> source_system STRING
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PARTITIONED &lt;span style="color:#66d9ef">BY&lt;/span> (&lt;span style="color:#66d9ef">year&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LOCATION&lt;/span> &lt;span style="color:#e6db74">&amp;#39;s3://space-data-pipeline-demo/semantic/dim_date/&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>TBLPROPERTIES (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;table_type&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;ICEBERG&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;format&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;parquet&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/details>
&lt;details>
&lt;summary>&lt;strong>Click to expand: dim_system DDL (SCD Type 2)&lt;/strong>&lt;/summary>
&lt;p>Tracks history with effective_date/expiration_date. Partitioned by effective_year for efficient history queries.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> space_exploration.dim_system (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> system_key INT, &lt;span style="color:#75715e">-- Surrogate key (generated)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> system_id STRING, &lt;span style="color:#75715e">-- Natural key (from source)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> system_name STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sector STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> main_star_type STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> coordinates_x DOUBLE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> coordinates_y DOUBLE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> coordinates_z DOUBLE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> discovery_date DATE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> exploration_status STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- SCD Type 2 columns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> effective_date DATE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> expiration_date DATE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_flag STRING, &lt;span style="color:#75715e">-- &amp;#39;Y&amp;#39; or &amp;#39;N&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#75715e">-- Audit columns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> load_timestamp &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> source_system STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Partition column
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> effective_year INT
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PARTITIONED &lt;span style="color:#66d9ef">BY&lt;/span> (effective_year)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LOCATION&lt;/span> &lt;span style="color:#e6db74">&amp;#39;s3://space-data-pipeline-demo/semantic/dim_system/&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>TBLPROPERTIES (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;table_type&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;ICEBERG&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;format&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;parquet&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/details>
&lt;details>
&lt;summary>&lt;strong>Click to expand: dim_planet DDL (SCD Type 2)&lt;/strong>&lt;/summary>
&lt;p>Foreign key to dim_system uses the surrogate key. Partitioned by effective_year.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> space_exploration.dim_planet (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> planet_key INT, &lt;span style="color:#75715e">-- Surrogate key
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> planet_id STRING, &lt;span style="color:#75715e">-- Natural key
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> system_key INT, &lt;span style="color:#75715e">-- FK to dim_system (surrogate!)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> planet_name STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> planet_type STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> orbital_period DOUBLE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> has_atmosphere STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> habitability_score DOUBLE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- SCD Type 2 columns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> effective_date DATE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> expiration_date DATE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_flag STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Audit columns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> load_timestamp &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> source_system STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Partition column
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> effective_year INT
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PARTITIONED &lt;span style="color:#66d9ef">BY&lt;/span> (effective_year)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LOCATION&lt;/span> &lt;span style="color:#e6db74">&amp;#39;s3://space-data-pipeline-demo/semantic/dim_planet/&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>TBLPROPERTIES (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;table_type&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;ICEBERG&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;format&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;parquet&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/details>
&lt;details>
&lt;summary>&lt;strong>Click to expand: fact_system_updates DDL&lt;/strong>&lt;/summary>
&lt;p>Incremental fact table partitioned by date_year. Loaded with delete-then-insert pattern.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> space_exploration.fact_system_updates (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> update_key BIGINT, &lt;span style="color:#75715e">-- Surrogate key
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> date_key INT, &lt;span style="color:#75715e">-- FK to dim_date
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> system_key INT, &lt;span style="color:#75715e">-- FK to dim_system (current surrogate!)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> update_timestamp &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span>, &lt;span style="color:#75715e">-- Original precision preserved
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> status STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ships_present INT, &lt;span style="color:#75715e">-- Additive fact
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> resource_level DOUBLE, &lt;span style="color:#75715e">-- Additive fact
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#75715e">-- Audit columns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> load_timestamp &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> source_system STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> data_date DATE, &lt;span style="color:#75715e">-- Source data date (for reprocessing)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#75715e">-- Partition column
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> date_year INT
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PARTITIONED &lt;span style="color:#66d9ef">BY&lt;/span> (date_year)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LOCATION&lt;/span> &lt;span style="color:#e6db74">&amp;#39;s3://space-data-pipeline-demo/semantic/fact_system_updates/&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>TBLPROPERTIES (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;table_type&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;ICEBERG&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;format&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;parquet&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/details>
&lt;details>
&lt;summary>&lt;strong>Click to expand: Staging Tables DDL (Persistent Staging Area)&lt;/strong>&lt;/summary>
&lt;p>Partitioned by data_date for historical reprocessing.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> space_exploration.stg_systems (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> system_id STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> system_name STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sector STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> main_star_type STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> coordinates_x DOUBLE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> coordinates_y DOUBLE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> coordinates_z DOUBLE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> discovery_date DATE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> exploration_status STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- PSA columns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> data_date DATE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> load_timestamp &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PARTITIONED &lt;span style="color:#66d9ef">BY&lt;/span> (data_date)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LOCATION&lt;/span> &lt;span style="color:#e6db74">&amp;#39;s3://space-data-pipeline-demo/staging/stg_systems/&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>TBLPROPERTIES (&lt;span style="color:#e6db74">&amp;#39;table_type&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;ICEBERG&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;format&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;parquet&amp;#39;&lt;/span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> space_exploration.stg_planets (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> planet_id STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> system_id STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> planet_name STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> planet_type STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> orbital_period DOUBLE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> has_atmosphere STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> habitability_score DOUBLE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- PSA columns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> data_date DATE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> load_timestamp &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PARTITIONED &lt;span style="color:#66d9ef">BY&lt;/span> (data_date)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LOCATION&lt;/span> &lt;span style="color:#e6db74">&amp;#39;s3://space-data-pipeline-demo/staging/stg_planets/&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>TBLPROPERTIES (&lt;span style="color:#e6db74">&amp;#39;table_type&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;ICEBERG&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;format&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;parquet&amp;#39;&lt;/span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> space_exploration.stg_updates (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> system_id STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> update_timestamp &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> status STRING,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ships_present INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> resource_level DOUBLE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- PSA columns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> data_date DATE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> load_timestamp &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PARTITIONED &lt;span style="color:#66d9ef">BY&lt;/span> (data_date)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LOCATION&lt;/span> &lt;span style="color:#e6db74">&amp;#39;s3://space-data-pipeline-demo/staging/stg_updates/&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>TBLPROPERTIES (&lt;span style="color:#e6db74">&amp;#39;table_type&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;ICEBERG&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;format&amp;#39;&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;parquet&amp;#39;&lt;/span>);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/details>
&lt;p>&lt;strong>Key design decisions:&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>
&lt;p>&lt;strong>dim_date&lt;/strong>: Partitioned by &lt;code>year&lt;/code>. This is a &lt;em>reference dimension&lt;/em>—typically loaded once with all dates from 1900-01-01 to 9999-12-31, then rarely updated.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>dim_system / dim_planet&lt;/strong>: SCD Type 2 with &lt;code>effective_date&lt;/code>, &lt;code>expiration_date&lt;/code>, &lt;code>current_flag&lt;/code>. Partitioned by &lt;code>effective_year&lt;/code> for efficient history queries. Updated daily.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>fact_system_updates&lt;/strong>: Partitioned by &lt;code>date_year&lt;/code>. Loaded incrementally using delete-then-insert pattern.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Staging tables (PSA)&lt;/strong>: Partitioned by &lt;code>data_date&lt;/code>. This lets us reprocess any historical date and provides dependency triggers.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;/br>
&lt;/br>
&lt;h3 id="the-heart-of-the-system-orchestrated-glue-jobs-with-per-table-wap">The Heart of the System: Orchestrated Glue Jobs with Per-Table WAP&lt;/h3>
&lt;hr>
&lt;p>Each job does its own Write-Audit-Publish cycle. Let me walk you through each one.&lt;/p>
&lt;hr>
&lt;h4 id="job-1-build-date-dimension-reference-dimension">Job 1: Build Date Dimension (Reference Dimension)&lt;/h4>
&lt;p>The date dimension is special—it&amp;rsquo;s a &lt;em>reference dimension&lt;/em>. We generate it ourselves with all calendar attributes, typically as a one-time load. You&amp;rsquo;d only rerun this if you need to extend the date range or add new attributes (like new holidays).&lt;/p>
&lt;details>
&lt;summary>&lt;strong>Click to expand: build_dim_date Glue Job&lt;/strong>&lt;/summary>
&lt;p>This job generates a complete date dimension from 2000-01-01 to 2050-12-31 (for demo purposes—production would use a wider range). It creates date attributes like day_of_week, quarter, fiscal_year, and is_weekend, then runs WAP to validate before publishing.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Glue Job: build_dim_date&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># This is typically a ONE-TIME RUN to populate all dates&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Rerun only when extending date range or adding attributes&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> sys
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.transforms &lt;span style="color:#f92672">import&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.utils &lt;span style="color:#f92672">import&lt;/span> getResolvedOptions
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pyspark.context &lt;span style="color:#f92672">import&lt;/span> SparkContext
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.context &lt;span style="color:#f92672">import&lt;/span> GlueContext
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.job &lt;span style="color:#f92672">import&lt;/span> Job
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pyspark.sql.functions &lt;span style="color:#f92672">import&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pyspark.sql.types &lt;span style="color:#f92672">import&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> datetime &lt;span style="color:#f92672">import&lt;/span> datetime, timedelta
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Initialize&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>args &lt;span style="color:#f92672">=&lt;/span> getResolvedOptions(sys&lt;span style="color:#f92672">.&lt;/span>argv, [&lt;span style="color:#e6db74">&amp;#39;JOB_NAME&amp;#39;&lt;/span>])
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sc &lt;span style="color:#f92672">=&lt;/span> SparkContext()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>glueContext &lt;span style="color:#f92672">=&lt;/span> GlueContext(sc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>spark &lt;span style="color:#f92672">=&lt;/span> glueContext&lt;span style="color:#f92672">.&lt;/span>spark_session
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>job &lt;span style="color:#f92672">=&lt;/span> Job(glueContext)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>job&lt;span style="color:#f92672">.&lt;/span>init(args[&lt;span style="color:#e6db74">&amp;#39;JOB_NAME&amp;#39;&lt;/span>], args)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;=&amp;#34;&lt;/span>&lt;span style="color:#f92672">*&lt;/span>&lt;span style="color:#ae81ff">60&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;Building Date Dimension (Reference Dimension)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;This is typically a ONE-TIME load&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;=&amp;#34;&lt;/span>&lt;span style="color:#f92672">*&lt;/span>&lt;span style="color:#ae81ff">60&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Generate comprehensive date range&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Standard practice: 1900-01-01 to 9999-12-31&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>start_date &lt;span style="color:#f92672">=&lt;/span> datetime(&lt;span style="color:#ae81ff">1900&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>end_date &lt;span style="color:#f92672">=&lt;/span> datetime(&lt;span style="color:#ae81ff">9999&lt;/span>, &lt;span style="color:#ae81ff">12&lt;/span>, &lt;span style="color:#ae81ff">31&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># For demo purposes, let&amp;#39;s use a more reasonable range&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># In production, you&amp;#39;d use the full range&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>start_date &lt;span style="color:#f92672">=&lt;/span> datetime(&lt;span style="color:#ae81ff">2000&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>end_date &lt;span style="color:#f92672">=&lt;/span> datetime(&lt;span style="color:#ae81ff">2050&lt;/span>, &lt;span style="color:#ae81ff">12&lt;/span>, &lt;span style="color:#ae81ff">31&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Generating dates from &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>start_date&lt;span style="color:#f92672">.&lt;/span>date()&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> to &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>end_date&lt;span style="color:#f92672">.&lt;/span>date()&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>date_list &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>current &lt;span style="color:#f92672">=&lt;/span> start_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">while&lt;/span> current &lt;span style="color:#f92672">&amp;lt;=&lt;/span> end_date:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> date_list&lt;span style="color:#f92672">.&lt;/span>append((current,))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current &lt;span style="color:#f92672">+=&lt;/span> timedelta(days&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>date_df &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>createDataFrame(date_list, [&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>])
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Build dimension with rich attributes&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>dim_date &lt;span style="color:#f92672">=&lt;/span> date_df&lt;span style="color:#f92672">.&lt;/span>select(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Surrogate key: YYYYMMDD format&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (year(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>) &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#ae81ff">10000&lt;/span> &lt;span style="color:#f92672">+&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> month(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>) &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#ae81ff">100&lt;/span> &lt;span style="color:#f92672">+&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dayofmonth(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>))&lt;span style="color:#f92672">.&lt;/span>cast(&lt;span style="color:#e6db74">&amp;#34;int&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> date_format(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;EEEE&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;day_of_week&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dayofmonth(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;day_of_month&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dayofyear(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;day_of_year&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> weekofyear(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;week_of_year&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> month(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;month_number&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> date_format(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;MMMM&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;month_name&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> quarter(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;quarter&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> year(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;year&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Business-friendly flags&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> when(dayofweek(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isin([&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">7&lt;/span>]), &lt;span style="color:#e6db74">&amp;#34;Y&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>otherwise(&lt;span style="color:#e6db74">&amp;#34;N&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;is_weekend&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;N&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;is_holiday&amp;#34;&lt;/span>), &lt;span style="color:#75715e"># Populate from holiday calendar&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Fiscal calendar (assuming fiscal year = calendar year)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> quarter(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;fiscal_quarter&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> year(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;fiscal_year&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Audit columns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_timestamp()&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;load_timestamp&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;GENERATED&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;source_system&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>row_count &lt;span style="color:#f92672">=&lt;/span> dim_date&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Generated &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>row_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> date rows&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># WRITE PHASE: Write to staging branch&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- WRITE PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;Writing to staging branch...&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;ALTER TABLE space_exploration.dim_date CREATE BRANCH IF NOT EXISTS staging&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>dim_date&lt;span style="color:#f92672">.&lt;/span>writeTo(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_date&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>option(&lt;span style="color:#e6db74">&amp;#34;branch&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;staging&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>overwritePartitions()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;Written to staging branch&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># AUDIT PHASE: Validate before publishing&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- AUDIT PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>staging_df &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>option(&lt;span style="color:#e6db74">&amp;#34;branch&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;staging&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_date&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>audit_failures &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 1: Row count&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>staged_count &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 1 - Row count: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>staged_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> staged_count &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;FAIL: dim_date is empty&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 2: Primary key uniqueness&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>unique_keys &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>distinct()&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> staged_count &lt;span style="color:#f92672">!=&lt;/span> unique_keys:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> duplicates &lt;span style="color:#f92672">=&lt;/span> staged_count &lt;span style="color:#f92672">-&lt;/span> unique_keys
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>duplicates&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> duplicate date_keys&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 2 - PK uniqueness: FAIL (&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>duplicates&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> duplicates)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 2 - PK uniqueness: PASS (all &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>unique_keys&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> unique)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 3: No NULLs in critical fields&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>null_count &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>filter(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull() &lt;span style="color:#f92672">|&lt;/span> col(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> null_count &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>null_count&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> NULL values in critical fields&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 3 - NULL check: FAIL (&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>null_count&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> NULLs)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Audit 3 - NULL check: PASS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 4: Date range coverage&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>date_range &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>agg(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> min(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;min_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> max(&lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;max_date&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)&lt;span style="color:#f92672">.&lt;/span>collect()[&lt;span style="color:#ae81ff">0&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 4 - Date range: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>date_range[&lt;span style="color:#e6db74">&amp;#39;min_date&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> to &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>date_range[&lt;span style="color:#e6db74">&amp;#39;max_date&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># PUBLISH PHASE: Merge to main if audits pass&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- PUBLISH PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> len(audit_failures) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">AUDIT FAILED:&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> failure &lt;span style="color:#f92672">in&lt;/span> audit_failures:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34; &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>failure&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">Branch NOT merged. Production unchanged.&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;ALTER TABLE space_exploration.dim_date DROP BRANCH staging&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">raise&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span>(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit failed with &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(audit_failures)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> errors&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">ALL AUDITS PASSED&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Merging staging branch to main...&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Fast-forward main to staging&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> CALL system.fast_forward(
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> table =&amp;gt; &amp;#39;space_exploration.dim_date&amp;#39;,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> branch =&amp;gt; &amp;#39;main&amp;#39;,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> to =&amp;gt; &amp;#39;staging&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> )
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Clean up staging branch&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;ALTER TABLE space_exploration.dim_date DROP BRANCH staging&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;PUBLISHED: dim_date (&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>staged_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> rows)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span> &lt;span style="color:#f92672">+&lt;/span> &lt;span style="color:#e6db74">&amp;#34;=&amp;#34;&lt;/span>&lt;span style="color:#f92672">*&lt;/span>&lt;span style="color:#ae81ff">60&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;dim_date job complete&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;=&amp;#34;&lt;/span>&lt;span style="color:#f92672">*&lt;/span>&lt;span style="color:#ae81ff">60&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>job&lt;span style="color:#f92672">.&lt;/span>commit()
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/details>
&lt;hr>
&lt;h4 id="job-2-build-system-dimension-scd-type-2">Job 2: Build System Dimension (SCD Type 2)&lt;/h4>
&lt;p>This dimension tracks history. When a system&amp;rsquo;s attributes change (like &lt;code>exploration_status&lt;/code>), we expire the old row and create a new one.&lt;/p>
&lt;details>
&lt;summary>&lt;strong>Click to expand: build_dim_system Glue Job&lt;/strong>&lt;/summary>
&lt;p>This job implements SCD Type 2 change detection. It compares incoming data against the current dimension, identifies NEW/CHANGED/UNCHANGED records, expires old versions, and creates new versions with updated surrogate keys.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Glue Job: build_dim_system&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># SCD Type 2 implementation with effective/expiration dates&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> sys
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.transforms &lt;span style="color:#f92672">import&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.utils &lt;span style="color:#f92672">import&lt;/span> getResolvedOptions
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pyspark.context &lt;span style="color:#f92672">import&lt;/span> SparkContext
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.context &lt;span style="color:#f92672">import&lt;/span> GlueContext
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.job &lt;span style="color:#f92672">import&lt;/span> Job
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pyspark.sql.functions &lt;span style="color:#f92672">import&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pyspark.sql.window &lt;span style="color:#f92672">import&lt;/span> Window
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>args &lt;span style="color:#f92672">=&lt;/span> getResolvedOptions(sys&lt;span style="color:#f92672">.&lt;/span>argv, [&lt;span style="color:#e6db74">&amp;#39;JOB_NAME&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;DATA_DATE&amp;#39;&lt;/span>])
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sc &lt;span style="color:#f92672">=&lt;/span> SparkContext()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>glueContext &lt;span style="color:#f92672">=&lt;/span> GlueContext(sc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>spark &lt;span style="color:#f92672">=&lt;/span> glueContext&lt;span style="color:#f92672">.&lt;/span>spark_session
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>job &lt;span style="color:#f92672">=&lt;/span> Job(glueContext)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>job&lt;span style="color:#f92672">.&lt;/span>init(args[&lt;span style="color:#e6db74">&amp;#39;JOB_NAME&amp;#39;&lt;/span>], args)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>data_date &lt;span style="color:#f92672">=&lt;/span> args[&lt;span style="color:#e6db74">&amp;#39;DATA_DATE&amp;#39;&lt;/span>] &lt;span style="color:#75715e"># e.g., &amp;#39;2026-01-17&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;=&amp;#34;&lt;/span>&lt;span style="color:#f92672">*&lt;/span>&lt;span style="color:#ae81ff">60&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Building System Dimension (SCD Type 2)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Data Date: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>data_date&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;=&amp;#34;&lt;/span>&lt;span style="color:#f92672">*&lt;/span>&lt;span style="color:#ae81ff">60&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># LOAD: Read from Persistent Staging Area&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- LOAD PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Read today&amp;#39;s staged data (already landed in PSA)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>source_df &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.stg_systems&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;data_date&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> data_date)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>source_count &lt;span style="color:#f92672">=&lt;/span> source_df&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Source rows from PSA (data_date=&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>data_date&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">): &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>source_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> source_count &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;No data for this date. Exiting.&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> job&lt;span style="color:#f92672">.&lt;/span>commit()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sys&lt;span style="color:#f92672">.&lt;/span>exit(&lt;span style="color:#ae81ff">0&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Read current production dimension (for SCD2 comparison)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_dim &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_system&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;current_flag&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;Y&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_count &lt;span style="color:#f92672">=&lt;/span> current_dim&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Current dimension rows (current_flag=&amp;#39;Y&amp;#39;): &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>current_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> has_existing_data &lt;span style="color:#f92672">=&lt;/span> current_count &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">except&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;No existing dimension data (first load)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> has_existing_data &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">False&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_dim &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">None&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># TRANSFORM: SCD Type 2 Logic&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- TRANSFORM PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#f92672">not&lt;/span> has_existing_data:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># First load: all rows are new&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;First load - all rows are INSERT&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> window_spec &lt;span style="color:#f92672">=&lt;/span> Window&lt;span style="color:#f92672">.&lt;/span>orderBy(&lt;span style="color:#e6db74">&amp;#34;system_id&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> new_dim &lt;span style="color:#f92672">=&lt;/span> source_df&lt;span style="color:#f92672">.&lt;/span>select(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> row_number()&lt;span style="color:#f92672">.&lt;/span>over(window_spec)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;system_id&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;system_name&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;sector&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;main_star_type&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;coordinates_x&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;coordinates_y&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;coordinates_z&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;discovery_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;exploration_status&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(data_date)&lt;span style="color:#f92672">.&lt;/span>cast(&lt;span style="color:#e6db74">&amp;#34;date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;effective_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;9999-12-31&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>cast(&lt;span style="color:#e6db74">&amp;#34;date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;expiration_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;Y&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;current_flag&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_timestamp()&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;load_timestamp&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;ELITE_DANGEROUS&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;source_system&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> year(lit(data_date))&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;effective_year&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )&lt;span style="color:#f92672">.&lt;/span>distinct()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Incremental load: detect changes for SCD2&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Incremental load - detecting changes for SCD2&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Define which columns to track for changes&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> tracked_columns &lt;span style="color:#f92672">=&lt;/span> [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;system_name&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;sector&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;main_star_type&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;coordinates_x&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;coordinates_y&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;coordinates_z&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;exploration_status&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Join source with current to detect changes&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> comparison &lt;span style="color:#f92672">=&lt;/span> source_df&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;src&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>join(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_dim&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;curr&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;src.system_id&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> col(&lt;span style="color:#e6db74">&amp;#34;curr.system_id&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;full_outer&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Identify: NEW (no match in current), CHANGED (values differ), UNCHANGED&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> change_detection &lt;span style="color:#f92672">=&lt;/span> comparison&lt;span style="color:#f92672">.&lt;/span>withColumn(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;change_type&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> when(col(&lt;span style="color:#e6db74">&amp;#34;curr.system_id&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull(), &lt;span style="color:#e6db74">&amp;#34;NEW&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>when(col(&lt;span style="color:#e6db74">&amp;#34;src.system_id&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull(), &lt;span style="color:#e6db74">&amp;#34;DELETED&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>when(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (col(&lt;span style="color:#e6db74">&amp;#34;src.system_name&amp;#34;&lt;/span>) &lt;span style="color:#f92672">!=&lt;/span> col(&lt;span style="color:#e6db74">&amp;#34;curr.system_name&amp;#34;&lt;/span>)) &lt;span style="color:#f92672">|&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (col(&lt;span style="color:#e6db74">&amp;#34;src.sector&amp;#34;&lt;/span>) &lt;span style="color:#f92672">!=&lt;/span> col(&lt;span style="color:#e6db74">&amp;#34;curr.sector&amp;#34;&lt;/span>)) &lt;span style="color:#f92672">|&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (col(&lt;span style="color:#e6db74">&amp;#34;src.main_star_type&amp;#34;&lt;/span>) &lt;span style="color:#f92672">!=&lt;/span> col(&lt;span style="color:#e6db74">&amp;#34;curr.main_star_type&amp;#34;&lt;/span>)) &lt;span style="color:#f92672">|&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (col(&lt;span style="color:#e6db74">&amp;#34;src.exploration_status&amp;#34;&lt;/span>) &lt;span style="color:#f92672">!=&lt;/span> col(&lt;span style="color:#e6db74">&amp;#34;curr.exploration_status&amp;#34;&lt;/span>)) &lt;span style="color:#f92672">|&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (abs(col(&lt;span style="color:#e6db74">&amp;#34;src.coordinates_x&amp;#34;&lt;/span>) &lt;span style="color:#f92672">-&lt;/span> col(&lt;span style="color:#e6db74">&amp;#34;curr.coordinates_x&amp;#34;&lt;/span>)) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0.001&lt;/span>) &lt;span style="color:#f92672">|&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (abs(col(&lt;span style="color:#e6db74">&amp;#34;src.coordinates_y&amp;#34;&lt;/span>) &lt;span style="color:#f92672">-&lt;/span> col(&lt;span style="color:#e6db74">&amp;#34;curr.coordinates_y&amp;#34;&lt;/span>)) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0.001&lt;/span>) &lt;span style="color:#f92672">|&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (abs(col(&lt;span style="color:#e6db74">&amp;#34;src.coordinates_z&amp;#34;&lt;/span>) &lt;span style="color:#f92672">-&lt;/span> col(&lt;span style="color:#e6db74">&amp;#34;curr.coordinates_z&amp;#34;&lt;/span>)) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0.001&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;CHANGED&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>otherwise(&lt;span style="color:#e6db74">&amp;#34;UNCHANGED&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Count changes&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> change_counts &lt;span style="color:#f92672">=&lt;/span> change_detection&lt;span style="color:#f92672">.&lt;/span>groupBy(&lt;span style="color:#e6db74">&amp;#34;change_type&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>count()&lt;span style="color:#f92672">.&lt;/span>collect()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> row &lt;span style="color:#f92672">in&lt;/span> change_counts:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34; &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>row[&lt;span style="color:#e6db74">&amp;#39;change_type&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>row[&lt;span style="color:#e6db74">&amp;#39;count&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get max surrogate key for new key generation&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> max_key &lt;span style="color:#f92672">=&lt;/span> current_dim&lt;span style="color:#f92672">.&lt;/span>agg(max(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>))&lt;span style="color:#f92672">.&lt;/span>collect()[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>] &lt;span style="color:#f92672">or&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Build new dimension rows for NEW and CHANGED records&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> new_records &lt;span style="color:#f92672">=&lt;/span> change_detection&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;change_type&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isin([&lt;span style="color:#e6db74">&amp;#34;NEW&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;CHANGED&amp;#34;&lt;/span>]))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> window_spec &lt;span style="color:#f92672">=&lt;/span> Window&lt;span style="color:#f92672">.&lt;/span>orderBy(&lt;span style="color:#e6db74">&amp;#34;src.system_id&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> new_rows &lt;span style="color:#f92672">=&lt;/span> new_records&lt;span style="color:#f92672">.&lt;/span>select(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (row_number()&lt;span style="color:#f92672">.&lt;/span>over(window_spec) &lt;span style="color:#f92672">+&lt;/span> max_key)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;src.system_id&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;system_id&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;src.system_name&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;system_name&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;src.sector&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;sector&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;src.main_star_type&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;main_star_type&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;src.coordinates_x&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;coordinates_x&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;src.coordinates_y&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;coordinates_y&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;src.coordinates_z&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;coordinates_z&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;src.discovery_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;discovery_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;src.exploration_status&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;exploration_status&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(data_date)&lt;span style="color:#f92672">.&lt;/span>cast(&lt;span style="color:#e6db74">&amp;#34;date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;effective_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;9999-12-31&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>cast(&lt;span style="color:#e6db74">&amp;#34;date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;expiration_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;Y&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;current_flag&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_timestamp()&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;load_timestamp&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;ELITE_DANGEROUS&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;source_system&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> year(lit(data_date))&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;effective_year&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Build expired rows (for CHANGED records, expire the old version)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> changed_ids &lt;span style="color:#f92672">=&lt;/span> change_detection&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;change_type&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;CHANGED&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>select(col(&lt;span style="color:#e6db74">&amp;#34;curr.system_id&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;system_id&amp;#34;&lt;/span>))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> expired_rows &lt;span style="color:#f92672">=&lt;/span> current_dim&lt;span style="color:#f92672">.&lt;/span>join(changed_ids, &lt;span style="color:#e6db74">&amp;#34;system_id&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>withColumn(&lt;span style="color:#e6db74">&amp;#34;expiration_date&amp;#34;&lt;/span>, lit(data_date)&lt;span style="color:#f92672">.&lt;/span>cast(&lt;span style="color:#e6db74">&amp;#34;date&amp;#34;&lt;/span>) &lt;span style="color:#f92672">-&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>withColumn(&lt;span style="color:#e6db74">&amp;#34;current_flag&amp;#34;&lt;/span>, lit(&lt;span style="color:#e6db74">&amp;#34;N&amp;#34;&lt;/span>))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Unchanged rows stay as-is&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> unchanged_rows &lt;span style="color:#f92672">=&lt;/span> current_dim&lt;span style="color:#f92672">.&lt;/span>join(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> change_detection&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;change_type&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;UNCHANGED&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>select(col(&lt;span style="color:#e6db74">&amp;#34;curr.system_id&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;system_id&amp;#34;&lt;/span>)),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;system_id&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Historical rows (already expired) - read from full table&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> historical_rows &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_system&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;current_flag&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;N&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Combine all&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> new_dim &lt;span style="color:#f92672">=&lt;/span> new_rows \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>unionByName(expired_rows, allowMissingColumns&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>unionByName(unchanged_rows, allowMissingColumns&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>unionByName(historical_rows, allowMissingColumns&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>transform_count &lt;span style="color:#f92672">=&lt;/span> new_dim&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Total dimension rows after SCD2: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>transform_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># WRITE PHASE: Write to staging branch&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- WRITE PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;ALTER TABLE space_exploration.dim_system CREATE BRANCH IF NOT EXISTS staging&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>new_dim&lt;span style="color:#f92672">.&lt;/span>writeTo(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_system&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>option(&lt;span style="color:#e6db74">&amp;#34;branch&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;staging&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>overwritePartitions()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;Written to staging branch&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># AUDIT PHASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- AUDIT PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>staging_df &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>option(&lt;span style="color:#e6db74">&amp;#34;branch&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;staging&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_system&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>audit_failures &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 1: Row count&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>staged_count &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 1 - Row count: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>staged_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> staged_count &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;FAIL: dim_system is empty&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 2: Surrogate key uniqueness&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>unique_keys &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>distinct()&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> staged_count &lt;span style="color:#f92672">!=&lt;/span> unique_keys:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> duplicates &lt;span style="color:#f92672">=&lt;/span> staged_count &lt;span style="color:#f92672">-&lt;/span> unique_keys
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>duplicates&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> duplicate system_keys&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 2 - PK uniqueness: FAIL (&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>duplicates&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> duplicates)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 2 - PK uniqueness: PASS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 3: Only one current row per natural key&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>current_rows &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;current_flag&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;Y&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>current_count &lt;span style="color:#f92672">=&lt;/span> current_rows&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>unique_natural_keys &lt;span style="color:#f92672">=&lt;/span> current_rows&lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;system_id&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>distinct()&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> current_count &lt;span style="color:#f92672">!=&lt;/span> unique_natural_keys:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: Multiple current rows for same system_id&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 3 - Single current: FAIL&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 3 - Single current: PASS (&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>current_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> current rows)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 4: No NULLs in critical fields&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>null_count &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>filter(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull() &lt;span style="color:#f92672">|&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;system_id&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull() &lt;span style="color:#f92672">|&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;effective_date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull() &lt;span style="color:#f92672">|&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;current_flag&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> null_count &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>null_count&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> NULL values in critical fields&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 4 - NULL check: FAIL&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Audit 4 - NULL check: PASS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 5: Effective/expiration date logic&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>invalid_dates &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>filter(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;effective_date&amp;#34;&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> col(&lt;span style="color:#e6db74">&amp;#34;expiration_date&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> invalid_dates &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>invalid_dates&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> rows with effective &amp;gt; expiration&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 5 - Date logic: FAIL&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Audit 5 - Date logic: PASS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># PUBLISH PHASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- PUBLISH PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> len(audit_failures) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">AUDIT FAILED:&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> failure &lt;span style="color:#f92672">in&lt;/span> audit_failures:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34; &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>failure&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">Branch NOT merged. Production unchanged.&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;ALTER TABLE space_exploration.dim_system DROP BRANCH staging&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">raise&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span>(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit failed with &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(audit_failures)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> errors&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">ALL AUDITS PASSED&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Merging staging branch to main...&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> CALL system.fast_forward(
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> table =&amp;gt; &amp;#39;space_exploration.dim_system&amp;#39;,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> branch =&amp;gt; &amp;#39;main&amp;#39;,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> to =&amp;gt; &amp;#39;staging&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> )
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;ALTER TABLE space_exploration.dim_system DROP BRANCH staging&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;PUBLISHED: dim_system (&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>staged_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> rows)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>job&lt;span style="color:#f92672">.&lt;/span>commit()
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/details>
&lt;hr>
&lt;h4 id="job-3-build-planet-dimension-scd-type-2">Job 3: Build Planet Dimension (SCD Type 2)&lt;/h4>
&lt;p>Similar pattern to dim&lt;em>system, but with a foreign key lookup to the _published&lt;/em> dim_system.&lt;/p>
&lt;details>
&lt;summary>&lt;strong>Click to expand: build_dim_planet Glue Job&lt;/strong>&lt;/summary>
&lt;p>This job demonstrates FK lookup to a published dimension. It reads from the main branch of dim_system (not staging), ensuring it uses production-ready surrogate keys. The FK integrity audit validates this relationship.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Glue Job: build_dim_planet&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># SCD Type 2 with FK lookup to published dim_system&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> sys
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.transforms &lt;span style="color:#f92672">import&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.utils &lt;span style="color:#f92672">import&lt;/span> getResolvedOptions
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pyspark.context &lt;span style="color:#f92672">import&lt;/span> SparkContext
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.context &lt;span style="color:#f92672">import&lt;/span> GlueContext
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.job &lt;span style="color:#f92672">import&lt;/span> Job
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pyspark.sql.functions &lt;span style="color:#f92672">import&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pyspark.sql.window &lt;span style="color:#f92672">import&lt;/span> Window
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>args &lt;span style="color:#f92672">=&lt;/span> getResolvedOptions(sys&lt;span style="color:#f92672">.&lt;/span>argv, [&lt;span style="color:#e6db74">&amp;#39;JOB_NAME&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;DATA_DATE&amp;#39;&lt;/span>])
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sc &lt;span style="color:#f92672">=&lt;/span> SparkContext()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>glueContext &lt;span style="color:#f92672">=&lt;/span> GlueContext(sc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>spark &lt;span style="color:#f92672">=&lt;/span> glueContext&lt;span style="color:#f92672">.&lt;/span>spark_session
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>job &lt;span style="color:#f92672">=&lt;/span> Job(glueContext)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>job&lt;span style="color:#f92672">.&lt;/span>init(args[&lt;span style="color:#e6db74">&amp;#39;JOB_NAME&amp;#39;&lt;/span>], args)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>data_date &lt;span style="color:#f92672">=&lt;/span> args[&lt;span style="color:#e6db74">&amp;#39;DATA_DATE&amp;#39;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;=&amp;#34;&lt;/span>&lt;span style="color:#f92672">*&lt;/span>&lt;span style="color:#ae81ff">60&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Building Planet Dimension (SCD Type 2)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Data Date: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>data_date&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;=&amp;#34;&lt;/span>&lt;span style="color:#f92672">*&lt;/span>&lt;span style="color:#ae81ff">60&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># LOAD PHASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- LOAD PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>source_df &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.stg_planets&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;data_date&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> data_date)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>source_count &lt;span style="color:#f92672">=&lt;/span> source_df&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Source rows: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>source_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> source_count &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;No data. Exiting.&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> job&lt;span style="color:#f92672">.&lt;/span>commit()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sys&lt;span style="color:#f92672">.&lt;/span>exit(&lt;span style="color:#ae81ff">0&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># CRITICAL: Read from PUBLISHED dim_system (main branch, not staging)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># This is why dim_planet must run AFTER dim_system publishes&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>dim_system &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_system&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;current_flag&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;Y&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;system_id&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dim_system lookup rows: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>dim_system&lt;span style="color:#f92672">.&lt;/span>count()&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># TRANSFORM PHASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- TRANSFORM PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Check for existing data&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_dim &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_planet&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;current_flag&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;Y&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> has_existing &lt;span style="color:#f92672">=&lt;/span> current_dim&lt;span style="color:#f92672">.&lt;/span>count() &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> max_key &lt;span style="color:#f92672">=&lt;/span> current_dim&lt;span style="color:#f92672">.&lt;/span>agg(max(&lt;span style="color:#e6db74">&amp;#34;planet_key&amp;#34;&lt;/span>))&lt;span style="color:#f92672">.&lt;/span>collect()[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>] &lt;span style="color:#f92672">or&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">except&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> has_existing &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">False&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> max_key &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Join source with dim_system to get surrogate key&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>source_with_sk &lt;span style="color:#f92672">=&lt;/span> source_df&lt;span style="color:#f92672">.&lt;/span>join(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dim_system,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> source_df&lt;span style="color:#f92672">.&lt;/span>system_id &lt;span style="color:#f92672">==&lt;/span> dim_system&lt;span style="color:#f92672">.&lt;/span>system_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;left&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)&lt;span style="color:#f92672">.&lt;/span>select(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> source_df[&lt;span style="color:#e6db74">&amp;#34;*&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dim_system&lt;span style="color:#f92672">.&lt;/span>system_key
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Track orphaned planets (no matching system)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>orphaned &lt;span style="color:#f92672">=&lt;/span> source_with_sk&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull())&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> orphaned &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;WARNING: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>orphaned&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> planets have no matching system!&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># SCD2 logic (simplified for first load demonstration)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>window_spec &lt;span style="color:#f92672">=&lt;/span> Window&lt;span style="color:#f92672">.&lt;/span>orderBy(&lt;span style="color:#e6db74">&amp;#34;planet_id&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>new_dim &lt;span style="color:#f92672">=&lt;/span> source_with_sk&lt;span style="color:#f92672">.&lt;/span>select(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (row_number()&lt;span style="color:#f92672">.&lt;/span>over(window_spec) &lt;span style="color:#f92672">+&lt;/span> max_key)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;planet_key&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;planet_id&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;planet_name&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;planet_type&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;orbital_period&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;has_atmosphere&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;habitability_score&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(data_date)&lt;span style="color:#f92672">.&lt;/span>cast(&lt;span style="color:#e6db74">&amp;#34;date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;effective_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;9999-12-31&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>cast(&lt;span style="color:#e6db74">&amp;#34;date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;expiration_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;Y&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;current_flag&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_timestamp()&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;load_timestamp&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;ELITE_DANGEROUS&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;source_system&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> year(lit(data_date))&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;effective_year&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)&lt;span style="color:#f92672">.&lt;/span>distinct()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># (Full SCD2 change detection would follow same pattern as dim_system)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>transform_count &lt;span style="color:#f92672">=&lt;/span> new_dim&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Dimension rows: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>transform_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># WRITE PHASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- WRITE PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;ALTER TABLE space_exploration.dim_planet CREATE BRANCH IF NOT EXISTS staging&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>new_dim&lt;span style="color:#f92672">.&lt;/span>writeTo(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_planet&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>option(&lt;span style="color:#e6db74">&amp;#34;branch&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;staging&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>overwritePartitions()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;Written to staging branch&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># AUDIT PHASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- AUDIT PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>staging_df &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>option(&lt;span style="color:#e6db74">&amp;#34;branch&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;staging&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_planet&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>audit_failures &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 1: PK uniqueness&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>staged_count &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>unique_keys &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;planet_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>distinct()&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> staged_count &lt;span style="color:#f92672">!=&lt;/span> unique_keys:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: Duplicate planet_keys&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 1 - PK uniqueness: PASS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 2: FK integrity to dim_system (published!)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>published_system_keys &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_system&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>distinct()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>orphaned_fk &lt;span style="color:#f92672">=&lt;/span> staging_df \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNotNull()) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>distinct() \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>join(published_system_keys, &lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;left_anti&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> orphaned_fk &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>orphaned_fk&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> orphaned system_keys&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 2 - FK integrity: FAIL&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Audit 2 - FK integrity: PASS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 3: NULL check&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>null_count &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>filter(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;planet_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull() &lt;span style="color:#f92672">|&lt;/span> col(&lt;span style="color:#e6db74">&amp;#34;planet_id&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> null_count &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>null_count&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> NULLs in critical fields&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Audit 3 - NULL check: PASS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># PUBLISH PHASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- PUBLISH PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> len(audit_failures) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">AUDIT FAILED:&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> f &lt;span style="color:#f92672">in&lt;/span> audit_failures:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34; &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>f&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;ALTER TABLE space_exploration.dim_planet DROP BRANCH staging&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">raise&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span>(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit failed&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">ALL AUDITS PASSED&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> CALL system.fast_forward(
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> table =&amp;gt; &amp;#39;space_exploration.dim_planet&amp;#39;,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> branch =&amp;gt; &amp;#39;main&amp;#39;,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> to =&amp;gt; &amp;#39;staging&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> )
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;ALTER TABLE space_exploration.dim_planet DROP BRANCH staging&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;PUBLISHED: dim_planet (&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>staged_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> rows)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>job&lt;span style="color:#f92672">.&lt;/span>commit()
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/details>
&lt;hr>
&lt;h4 id="job-4-build-fact-table-incremental-with-delete-insert">Job 4: Build Fact Table (Incremental with Delete-Insert)&lt;/h4>
&lt;p>This is the critical job. It reads from &lt;em>published&lt;/em> dimensions (main branch), uses delete-then-insert pattern for the date partition, and includes left anti-join as a defensive check for missing records.&lt;/p>
&lt;details>
&lt;summary>&lt;strong>Click to expand: build_fact_system_updates Glue Job&lt;/strong>&lt;/summary>
&lt;p>This job loads facts incrementally by date. For a given data_date, it deletes existing records and inserts new ones. It validates FK integrity against all published dimensions before merging to main.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Glue Job: build_fact_system_updates&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Incremental load with delete-then-insert + left anti-join defensive coding&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> sys
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.transforms &lt;span style="color:#f92672">import&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.utils &lt;span style="color:#f92672">import&lt;/span> getResolvedOptions
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pyspark.context &lt;span style="color:#f92672">import&lt;/span> SparkContext
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.context &lt;span style="color:#f92672">import&lt;/span> GlueContext
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> awsglue.job &lt;span style="color:#f92672">import&lt;/span> Job
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pyspark.sql.functions &lt;span style="color:#f92672">import&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pyspark.sql.window &lt;span style="color:#f92672">import&lt;/span> Window
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>args &lt;span style="color:#f92672">=&lt;/span> getResolvedOptions(sys&lt;span style="color:#f92672">.&lt;/span>argv, [&lt;span style="color:#e6db74">&amp;#39;JOB_NAME&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;DATA_DATE&amp;#39;&lt;/span>])
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sc &lt;span style="color:#f92672">=&lt;/span> SparkContext()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>glueContext &lt;span style="color:#f92672">=&lt;/span> GlueContext(sc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>spark &lt;span style="color:#f92672">=&lt;/span> glueContext&lt;span style="color:#f92672">.&lt;/span>spark_session
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>job &lt;span style="color:#f92672">=&lt;/span> Job(glueContext)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>job&lt;span style="color:#f92672">.&lt;/span>init(args[&lt;span style="color:#e6db74">&amp;#39;JOB_NAME&amp;#39;&lt;/span>], args)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>data_date &lt;span style="color:#f92672">=&lt;/span> args[&lt;span style="color:#e6db74">&amp;#39;DATA_DATE&amp;#39;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;=&amp;#34;&lt;/span>&lt;span style="color:#f92672">*&lt;/span>&lt;span style="color:#ae81ff">60&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Building Fact Table: System Updates (Incremental)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Data Date: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>data_date&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;=&amp;#34;&lt;/span>&lt;span style="color:#f92672">*&lt;/span>&lt;span style="color:#ae81ff">60&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># LOAD PHASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- LOAD PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Read from PSA for this data_date&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>source_df &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.stg_updates&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;data_date&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> data_date)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>source_count &lt;span style="color:#f92672">=&lt;/span> source_df&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Source rows: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>source_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> source_count &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;No data. Exiting.&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> job&lt;span style="color:#f92672">.&lt;/span>commit()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sys&lt;span style="color:#f92672">.&lt;/span>exit(&lt;span style="color:#ae81ff">0&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Read PUBLISHED dimensions (main branch)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Fact tables MUST read from published dimensions, not staging branches&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>dim_system &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_system&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;current_flag&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;Y&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;system_id&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>dim_date &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_date&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;full_date&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;year&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dim_system rows: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>dim_system&lt;span style="color:#f92672">.&lt;/span>count()&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dim_date rows: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>dim_date&lt;span style="color:#f92672">.&lt;/span>count()&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Read existing fact table for incremental logic&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> existing_fact &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.fact_system_updates&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> existing_count &lt;span style="color:#f92672">=&lt;/span> existing_fact&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Existing fact rows: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>existing_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> max_key &lt;span style="color:#f92672">=&lt;/span> existing_fact&lt;span style="color:#f92672">.&lt;/span>agg(max(&lt;span style="color:#e6db74">&amp;#34;update_key&amp;#34;&lt;/span>))&lt;span style="color:#f92672">.&lt;/span>collect()[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>] &lt;span style="color:#f92672">or&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">except&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;No existing fact data (first load)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> existing_fact &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">None&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> max_key &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># TRANSFORM PHASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- TRANSFORM PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Add date column for joining&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>source_with_date &lt;span style="color:#f92672">=&lt;/span> source_df&lt;span style="color:#f92672">.&lt;/span>withColumn(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;update_date&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> to_date(&lt;span style="color:#e6db74">&amp;#34;update_timestamp&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Lookup surrogate keys from published dimensions&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>fact_transformed &lt;span style="color:#f92672">=&lt;/span> source_with_date \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>join(dim_system, source_with_date&lt;span style="color:#f92672">.&lt;/span>system_id &lt;span style="color:#f92672">==&lt;/span> dim_system&lt;span style="color:#f92672">.&lt;/span>system_id, &lt;span style="color:#e6db74">&amp;#34;left&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>join(dim_date, source_with_date&lt;span style="color:#f92672">.&lt;/span>update_date &lt;span style="color:#f92672">==&lt;/span> dim_date&lt;span style="color:#f92672">.&lt;/span>full_date, &lt;span style="color:#e6db74">&amp;#34;left&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>select(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dim_system&lt;span style="color:#f92672">.&lt;/span>system_key,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> source_with_date&lt;span style="color:#f92672">.&lt;/span>update_timestamp,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;status&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;ships_present&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;resource_level&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(data_date)&lt;span style="color:#f92672">.&lt;/span>cast(&lt;span style="color:#e6db74">&amp;#34;date&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;data_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dim_date&lt;span style="color:#f92672">.&lt;/span>year&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;date_year&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Track orphaned facts&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>orphaned_systems &lt;span style="color:#f92672">=&lt;/span> fact_transformed&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull())&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>orphaned_dates &lt;span style="color:#f92672">=&lt;/span> fact_transformed&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull())&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> orphaned_systems &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;WARNING: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>orphaned_systems&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> facts have no matching system&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> orphaned_dates &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;WARNING: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>orphaned_dates&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> facts have no matching date&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># INCREMENTAL LOGIC: Delete-then-Insert + Left Anti-Join&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- INCREMENTAL LOGIC ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Extract the date partition we&amp;#39;re loading&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>date_partition_year &lt;span style="color:#f92672">=&lt;/span> int(data_date[:&lt;span style="color:#ae81ff">4&lt;/span>]) &lt;span style="color:#75715e"># e.g., 2026&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> existing_fact &lt;span style="color:#f92672">is&lt;/span> &lt;span style="color:#f92672">not&lt;/span> &lt;span style="color:#66d9ef">None&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Step 1: Get existing records for this data_date&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> existing_for_date &lt;span style="color:#f92672">=&lt;/span> existing_fact&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;data_date&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> data_date)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> existing_for_date_count &lt;span style="color:#f92672">=&lt;/span> existing_for_date&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Existing records for data_date=&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>data_date&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>existing_for_date_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Step 2: Records NOT in this date (keep as-is)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> records_to_keep &lt;span style="color:#f92672">=&lt;/span> existing_fact&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;data_date&amp;#34;&lt;/span>) &lt;span style="color:#f92672">!=&lt;/span> data_date)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> keep_count &lt;span style="color:#f92672">=&lt;/span> records_to_keep&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Records to keep (other dates): &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>keep_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Step 3: New records from source (delete-then-insert for this date)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Generate new surrogate keys starting after max existing key&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> window_spec &lt;span style="color:#f92672">=&lt;/span> Window&lt;span style="color:#f92672">.&lt;/span>orderBy(&lt;span style="color:#e6db74">&amp;#34;update_timestamp&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> new_records &lt;span style="color:#f92672">=&lt;/span> fact_transformed&lt;span style="color:#f92672">.&lt;/span>select(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (row_number()&lt;span style="color:#f92672">.&lt;/span>over(window_spec) &lt;span style="color:#f92672">+&lt;/span> max_key)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;update_key&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;update_timestamp&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;status&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;ships_present&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;resource_level&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_timestamp()&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;load_timestamp&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;ELITE_DANGEROUS&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;source_system&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;data_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;date_year&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> new_count &lt;span style="color:#f92672">=&lt;/span> new_records&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;New records for data_date=&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>data_date&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>new_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Step 4: Left anti-join defensive check&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Find any records in destination that should exist but don&amp;#39;t&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># (This catches edge cases where source has records not in our transform)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># For this example, we&amp;#39;re doing a full replace for the date, so this is informational&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Step 5: Combine: kept records + new records (effectively delete-insert)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> final_fact &lt;span style="color:#f92672">=&lt;/span> records_to_keep&lt;span style="color:#f92672">.&lt;/span>unionByName(new_records)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># First load: all records are new&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> window_spec &lt;span style="color:#f92672">=&lt;/span> Window&lt;span style="color:#f92672">.&lt;/span>orderBy(&lt;span style="color:#e6db74">&amp;#34;update_timestamp&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> final_fact &lt;span style="color:#f92672">=&lt;/span> fact_transformed&lt;span style="color:#f92672">.&lt;/span>select(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> row_number()&lt;span style="color:#f92672">.&lt;/span>over(window_spec)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;update_key&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;update_timestamp&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;status&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;ships_present&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;resource_level&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> current_timestamp()&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;load_timestamp&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lit(&lt;span style="color:#e6db74">&amp;#34;ELITE_DANGEROUS&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>alias(&lt;span style="color:#e6db74">&amp;#34;source_system&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;data_date&amp;#34;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col(&lt;span style="color:#e6db74">&amp;#34;date_year&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>transform_count &lt;span style="color:#f92672">=&lt;/span> final_fact&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Final fact rows: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>transform_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># WRITE PHASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- WRITE PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;ALTER TABLE space_exploration.fact_system_updates CREATE BRANCH IF NOT EXISTS staging&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>final_fact&lt;span style="color:#f92672">.&lt;/span>writeTo(&lt;span style="color:#e6db74">&amp;#34;space_exploration.fact_system_updates&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>option(&lt;span style="color:#e6db74">&amp;#34;branch&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;staging&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>overwritePartitions()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;Written to staging branch&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># AUDIT PHASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- AUDIT PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>staging_df &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>option(&lt;span style="color:#e6db74">&amp;#34;branch&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;staging&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.fact_system_updates&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>audit_failures &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 1: Row count (source to target for this date)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>staged_for_date &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;data_date&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> data_date)&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 1 - Row count for &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>data_date&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">: Source=&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>source_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">, Target=&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>staged_for_date&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> source_count &lt;span style="color:#f92672">!=&lt;/span> staged_for_date:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># This might be OK if we filtered orphans, but flag for review&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34; Note: Difference of &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>source_count &lt;span style="color:#f92672">-&lt;/span> staged_for_date&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> (likely orphaned records)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 2: PK uniqueness&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>staged_count &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>unique_keys &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;update_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>distinct()&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> staged_count &lt;span style="color:#f92672">!=&lt;/span> unique_keys:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: Duplicate update_keys&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 2 - PK uniqueness: FAIL&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 2 - PK uniqueness: PASS (&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>staged_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> unique)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 3: FK integrity to dim_system&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>published_system_keys &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_system&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>distinct()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>orphaned_fk_system &lt;span style="color:#f92672">=&lt;/span> staging_df \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNotNull()) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>distinct() \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>join(published_system_keys, &lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;left_anti&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> orphaned_fk_system &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>orphaned_fk_system&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> orphaned system_keys&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 3 - FK to dim_system: FAIL&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Audit 3 - FK to dim_system: PASS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 4: FK integrity to dim_date&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>published_date_keys &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>table(&lt;span style="color:#e6db74">&amp;#34;space_exploration.dim_date&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>distinct()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>orphaned_fk_date &lt;span style="color:#f92672">=&lt;/span> staging_df \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNotNull()) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>select(&lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>distinct() \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>join(published_date_keys, &lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;left_anti&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> orphaned_fk_date &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>orphaned_fk_date&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> orphaned date_keys&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 4 - FK to dim_date: FAIL&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Audit 4 - FK to dim_date: PASS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 5: No NULLs in mandatory FKs&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># (We allow NULLs for orphaned records, but track them)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>null_system &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;system_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull())&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>null_date &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;date_key&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isNull())&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 5 - NULL FKs: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>null_system&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> null system_key, &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>null_date&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> null date_key&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 6: Value range checks&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>invalid_resources &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>filter(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (col(&lt;span style="color:#e6db74">&amp;#34;resource_level&amp;#34;&lt;/span>) &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>) &lt;span style="color:#f92672">|&lt;/span> (col(&lt;span style="color:#e6db74">&amp;#34;resource_level&amp;#34;&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> invalid_resources &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>invalid_resources&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> invalid resource_level values&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 6 - Value ranges: FAIL&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Audit 6 - Value ranges: PASS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Audit 7: No future timestamps&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>future_ts &lt;span style="color:#f92672">=&lt;/span> staging_df&lt;span style="color:#f92672">.&lt;/span>filter(col(&lt;span style="color:#e6db74">&amp;#34;update_timestamp&amp;#34;&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> current_timestamp())&lt;span style="color:#f92672">.&lt;/span>count()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> future_ts &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> audit_failures&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;FAIL: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>future_ts&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> future timestamps&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit 7 - Timestamps: FAIL&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Audit 7 - Timestamps: PASS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># PUBLISH PHASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># ============================================&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">--- PUBLISH PHASE ---&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> len(audit_failures) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">AUDIT FAILED:&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> f &lt;span style="color:#f92672">in&lt;/span> audit_failures:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34; &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>f&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;ALTER TABLE space_exploration.fact_system_updates DROP BRANCH staging&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">raise&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span>(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Audit failed with &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(audit_failures)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> errors&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">ALL AUDITS PASSED&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> CALL system.fast_forward(
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> table =&amp;gt; &amp;#39;space_exploration.fact_system_updates&amp;#39;,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> branch =&amp;gt; &amp;#39;main&amp;#39;,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> to =&amp;gt; &amp;#39;staging&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> )
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> spark&lt;span style="color:#f92672">.&lt;/span>sql(&lt;span style="color:#e6db74">&amp;#34;ALTER TABLE space_exploration.fact_system_updates DROP BRANCH staging&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;PUBLISHED: fact_system_updates (&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>staged_count&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">,&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> rows)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>job&lt;span style="color:#f92672">.&lt;/span>commit()
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/details>
&lt;hr>
&lt;h3 id="step-functions-workflow">Step Functions Workflow&lt;/h3>
&lt;hr>
&lt;p>The Step Functions workflow orchestrates everything. Here&amp;rsquo;s the execution flow:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Stage 1 (Parallel):&lt;/strong> dim_date and dim_system run simultaneously—neither depends on the other&lt;/li>
&lt;li>&lt;strong>Stage 2:&lt;/strong> dim_planet runs after Stage 1 completes (needs dim_system&amp;rsquo;s surrogate keys)&lt;/li>
&lt;li>&lt;strong>Stage 3:&lt;/strong> fact_system_updates runs last (needs all dimension surrogate keys)&lt;/li>
&lt;/ol>
&lt;p>If any job fails, the workflow stops and sends an SNS notification.&lt;/p>
&lt;details>
&lt;summary>&lt;strong>Click to expand: Step Functions JSON Definition&lt;/strong>&lt;/summary>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-json" data-lang="json">&lt;span style="display:flex;">&lt;span>{
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Comment&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Dimensional Modeling Pipeline - Per-Table WAP Pattern&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;StartAt&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Parallel Dimensions Stage 1&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;States&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Parallel Dimensions Stage 1&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Parallel&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Comment&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;dim_date and dim_system have no dependencies&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Branches&amp;#34;&lt;/span>: [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;StartAt&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Build dim_date (WAP)&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;States&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Build dim_date (WAP)&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Task&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Resource&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;arn:aws:states:::glue:startJobRun.sync&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Parameters&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;JobName&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;build_dim_date&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;End&amp;#34;&lt;/span>: &lt;span style="color:#66d9ef">true&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;StartAt&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Build dim_system (WAP)&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;States&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Build dim_system (WAP)&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Task&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Resource&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;arn:aws:states:::glue:startJobRun.sync&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Parameters&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;JobName&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;build_dim_system&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Arguments&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;--DATA_DATE.$&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;$.data_date&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;End&amp;#34;&lt;/span>: &lt;span style="color:#66d9ef">true&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Next&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Build dim_planet (WAP)&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Catch&amp;#34;&lt;/span>: [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;ErrorEquals&amp;#34;&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;States.ALL&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Next&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Notify Failure&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;ResultPath&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;$.error&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Build dim_planet (WAP)&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Task&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Comment&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Depends on dim_system being published&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Resource&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;arn:aws:states:::glue:startJobRun.sync&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Parameters&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;JobName&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;build_dim_planet&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Arguments&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;--DATA_DATE.$&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;$.data_date&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Next&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Build fact_system_updates (WAP)&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Catch&amp;#34;&lt;/span>: [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;ErrorEquals&amp;#34;&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;States.ALL&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Next&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Notify Failure&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;ResultPath&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;$.error&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Build fact_system_updates (WAP)&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Task&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Comment&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Depends on ALL dimensions being published&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Resource&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;arn:aws:states:::glue:startJobRun.sync&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Parameters&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;JobName&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;build_fact_system_updates&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Arguments&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;--DATA_DATE.$&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;$.data_date&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Next&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Pipeline Succeeded&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Catch&amp;#34;&lt;/span>: [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;ErrorEquals&amp;#34;&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;States.ALL&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Next&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Notify Failure&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;ResultPath&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;$.error&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Notify Failure&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Task&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Resource&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;arn:aws:states:::sns:publish&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Parameters&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;TopicArn&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;arn:aws:sns:ap-southeast-2:ACCOUNT:pipeline-alerts&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Message.$&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;States.Format(&amp;#39;Pipeline failed: {}&amp;#39;, $.error.Cause)&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Subject&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Data Pipeline Failed&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Next&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Pipeline Failed&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Pipeline Succeeded&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Succeed&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Pipeline Failed&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Fail&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Error&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;PipelineError&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Cause&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;One or more jobs failed - check individual job logs&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/details>
&lt;p>Start the workflow with input:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-json" data-lang="json">&lt;span style="display:flex;">&lt;span>{
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;data_date&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;2026-01-17&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This &lt;code>data_date&lt;/code> propagates to each job, enabling:&lt;/p>
&lt;ul>
&lt;li>PSA partition reads (&lt;code>WHERE data_date = '2026-01-17'&lt;/code>)&lt;/li>
&lt;li>Historical reprocessing (re-run with a past date)&lt;/li>
&lt;li>Dependency triggers (S3 event on partition write triggers workflow)&lt;/li>
&lt;/ul>
&lt;/br>
&lt;/br>
&lt;h3 id="dependency-triggers-with-s3-events">Dependency Triggers with S3 Events&lt;/h3>
&lt;hr>
&lt;p>You can improve this further by using partition writes as triggers. Here&amp;rsquo;s how to set that up:&lt;/p>
&lt;p>&lt;strong>Option 1: S3 Event → EventBridge → Step Functions&lt;/strong>&lt;/p>
&lt;p>When a file lands in &lt;code>landing/updates/&lt;/code>, trigger the pipeline:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># EventBridge Rule&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>{
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;source&amp;#34;: &lt;/span>[&lt;span style="color:#e6db74">&amp;#34;aws.s3&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;detail-type&amp;#34;: &lt;/span>[&lt;span style="color:#e6db74">&amp;#34;Object Created&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;detail&amp;#34;&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;bucket&amp;#34;: { &amp;#34;name&amp;#34;: &lt;/span>[&lt;span style="color:#e6db74">&amp;#34;space-data-pipeline-demo&amp;#34;&lt;/span>] },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;object&amp;#34;: { &amp;#34;key&amp;#34;: [{ &amp;#34;prefix&amp;#34;: &lt;/span>&lt;span style="color:#e6db74">&amp;#34;landing/updates/&amp;#34;&lt;/span> }] },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Option 2: Glue Trigger on Partition&lt;/strong>&lt;/p>
&lt;p>Configure Glue to watch for new partitions:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Check if partition exists before running&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>partition_path &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;s3://space-data-pipeline-demo/staging/stg_updates/data_date=&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>data_date&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">/&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> boto3
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>s3 &lt;span style="color:#f92672">=&lt;/span> boto3&lt;span style="color:#f92672">.&lt;/span>client(&lt;span style="color:#e6db74">&amp;#39;s3&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> s3&lt;span style="color:#f92672">.&lt;/span>head_object(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> Bucket&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;space-data-pipeline-demo&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> Key&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#39;staging/stg_updates/data_date=&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>data_date&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">/_SUCCESS&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Partition exists - proceeding&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">except&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Partition not ready - exiting&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sys&lt;span style="color:#f92672">.&lt;/span>exit(&lt;span style="color:#ae81ff">0&lt;/span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This lets you build dependency chains based on partition availability.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-moment-of-truth-querying">The Moment of Truth: Querying&lt;/h3>
&lt;hr>
&lt;p>Now let&amp;rsquo;s query the properly partitioned dimensional model:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Query leverages date partition pruning
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dt.&lt;span style="color:#66d9ef">year&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dt.quarter,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> d.main_star_type,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> updates,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(f.ships_present) &lt;span style="color:#66d9ef">as&lt;/span> total_ships,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROUND(&lt;span style="color:#66d9ef">AVG&lt;/span>(f.resource_level), &lt;span style="color:#ae81ff">3&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> avg_resources
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> space_exploration.fact_system_updates f
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">LEFT&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> space_exploration.dim_system d
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ON&lt;/span> f.system_key &lt;span style="color:#f92672">=&lt;/span> d.system_key
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> d.current_flag &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;Y&amp;#39;&lt;/span> &lt;span style="color:#75715e">-- Only current dimension records!
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">LEFT&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> space_exploration.dim_date dt
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ON&lt;/span> f.date_key &lt;span style="color:#f92672">=&lt;/span> dt.date_key
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dt.&lt;span style="color:#66d9ef">year&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">2026&lt;/span> &lt;span style="color:#75715e">-- Partition pruning!
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">AND&lt;/span> dt.quarter &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dt.&lt;span style="color:#66d9ef">year&lt;/span>, dt.quarter, d.main_star_type
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> total_ships &lt;span style="color:#66d9ef">DESC&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Notice &lt;code>d.current_flag = 'Y'&lt;/code> — this is how you query SCD Type 2 dimensions for current state. For point-in-time analysis:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Point-in-time analysis: What was system status on Jan 15, 2026?
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> d.system_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> d.exploration_status,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> d.effective_date,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> d.expiration_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> space_exploration.dim_system d
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> d.system_id &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;SOL&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> DATE &lt;span style="color:#e6db74">&amp;#39;2026-01-15&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">BETWEEN&lt;/span> d.effective_date &lt;span style="color:#66d9ef">AND&lt;/span> d.expiration_date;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;/br>
&lt;h3 id="a-few-things-worth-remembering">A Few Things Worth Remembering&lt;/h3>
&lt;hr>
&lt;p>&lt;strong>WAP at each table, not just at the end.&lt;/strong> Each job should do its own Write-Audit-Publish cycle. Failures are isolated. Downstream jobs read from published (main) tables.&lt;/p>
&lt;p>&lt;strong>Landing is raw, Staging is persistent.&lt;/strong> Landing = 1:1 from source, latest only. Staging (PSA) = partitioned by &lt;code>data_date&lt;/code> for historical reprocessing.&lt;/p>
&lt;p>&lt;strong>dim_date is a reference dimension.&lt;/strong> Load it once with a wide date range. Partition by year. Rarely needs updates.&lt;/p>
&lt;p>&lt;strong>SCD Type 2 for dimensions that change.&lt;/strong> Add &lt;code>effective_date&lt;/code>, &lt;code>expiration_date&lt;/code>, &lt;code>current_flag&lt;/code>. Partition by effective year. Query with &lt;code>current_flag = 'Y'&lt;/code> for current state.&lt;/p>
&lt;p>&lt;strong>Incremental facts with delete-then-insert.&lt;/strong> For a given date partition: delete existing, insert new. Use left anti-join as defensive check for missing records.&lt;/p>
&lt;p>&lt;strong>Partition strategically:&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>dim_date: by year&lt;/li>
&lt;li>dim_system/planet: by effective_year (SCD2)&lt;/li>
&lt;li>fact: by date_year&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Use data_date for dependencies.&lt;/strong> When &lt;code>stg_updates/data_date=2026-01-17/&lt;/code> is written, trigger the downstream pipeline. This makes dependencies explicit and reprocessing easy.&lt;/p>
&lt;hr>
&lt;p>That ten-minute query? Down to 1.2 seconds with partition pruning.&lt;/p>
&lt;p>That conference room moment? Each job&amp;rsquo;s WAP cycle catches issues before they propagate.&lt;/p>
&lt;p>These patterns work. They&amp;rsquo;ve saved me from embarrassment, late nights, and angry emails. They&amp;rsquo;ll do the same for you.&lt;/p></content:encoded><category>Data Engineering</category><category>Platforms &amp; Cloud</category><category>AWS Glue</category><category>Dimensional Modelling</category><category>Kimball</category><category>Data Quality</category><category>ETL</category><category>Write-Audit-Publish</category><category>Apache Iceberg</category><category>Step Functions</category><category>SCD</category></item><item><title>Building Your First AWS Data Pipeline: A Guide for Data Professionals Who've Never Touched Cloud Infrastructure</title><link>https://ghostinthedata.info/posts/2025/2025-11-26-aws-sourcing/</link><pubDate>Wed, 26 Nov 2025 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2025/2025-11-26-aws-sourcing/</guid><author>Chris Hillman</author><description>Learn how to build a production-ready data ingestion pipeline using S3, Lambda, and Athena—without managing a single server</description><content:encoded>&lt;h3 id="the-spreadsheet-that-changed-everything">The spreadsheet that changed everything&lt;/h3>
&lt;hr>
&lt;p>Here&amp;rsquo;s a story that might sound familiar. You&amp;rsquo;re pulling data from an API—maybe daily sales numbers, maybe customer interactions, maybe something else entirely. Every morning, you open your laptop, run a Python script, save the CSV somewhere, and get on with your actual work. It takes maybe five minutes, but it&amp;rsquo;s five minutes you can&amp;rsquo;t forget about. Miss a day and you&amp;rsquo;ve got a gap in your data. Go on vacation? Better hope someone remembers to run your script.&lt;/p>
&lt;p>I watched a colleague do this for six months before he finally asked: &amp;ldquo;There&amp;rsquo;s got to be a better way, right?&amp;rdquo;&lt;/p>
&lt;p>Yeah. There absolutely is. What if you could build a system that downloads data automatically, stores it properly, and makes it queryable—all without you managing servers or remembering to run scripts? What if that system cost you about $2 per month and scaled to handle gigabytes or terabytes without breaking a sweat?&lt;/p>
&lt;p>That&amp;rsquo;s what we&amp;rsquo;re building today using AWS. By the end of this guide, you&amp;rsquo;ll understand how professional data teams actually structure their cloud infrastructure, not just theoretically but hands-on. You&amp;rsquo;ll have built a working data pipeline that ingests data from an API, stores it following industry standards, and lets you query it using SQL as if it were sitting in a database.&lt;/p>
&lt;p>No prior AWS experience needed. No server management required. Just follow along, and in about an hour, you&amp;rsquo;ll have infrastructure you could deploy in production. Let&amp;rsquo;s get started.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="why-the-cloud-matters-more-than-you-think">Why the cloud matters more than you think&lt;/h3>
&lt;hr>
&lt;p>Before we write any code or click anything in AWS, let&amp;rsquo;s talk about why this approach fundamentally changes how data work happens.&lt;/p>
&lt;p>Traditional data infrastructure requires servers. You buy physical machines (or rent virtual ones), install databases, configure networking, manage security patches, monitor disk space, worry about backups, handle scaling when data grows. It&amp;rsquo;s a full-time job—literally. Companies hire entire teams just to keep the infrastructure running.&lt;/p>
&lt;p>Cloud infrastructure flips this model. Instead of managing servers, you use services. Need storage? Use S3. Need to run code? Use Lambda. Need to query data? Use Athena. Each service handles its own scaling, security, backups, and maintenance. You just use them.&lt;/p>
&lt;p>The economics are compelling. Research from 451 Research found that companies using cloud services reduce infrastructure costs by an average of 30-40% compared to on-premises solutions. But cost isn&amp;rsquo;t even the biggest advantage. The real benefit is speed. What used to take weeks of procurement, setup, and configuration now takes minutes.&lt;/p>
&lt;p>For data professionals specifically, this matters because you can focus on extracting value from data instead of maintaining infrastructure. You&amp;rsquo;re not a systems administrator. You&amp;rsquo;re someone who needs reliable data storage, automated processing, and easy querying. AWS provides exactly that.&lt;/p>
&lt;p>Here&amp;rsquo;s what we&amp;rsquo;re building today: a serverless data pipeline. &amp;ldquo;Serverless&amp;rdquo; doesn&amp;rsquo;t mean there are no servers—there are servers, you just don&amp;rsquo;t manage them. It&amp;rsquo;s like how you use electricity without maintaining a power plant. You use the service, AWS handles the infrastructure.&lt;/p>
&lt;p>The specific pipeline we&amp;rsquo;ll build follows a pattern so common in industry that once you recognise it, you&amp;rsquo;ll see it everywhere. API data flows into Lambda functions running on a schedule. Lambda stores raw data in S3, organised by date. AWS Glue catalogs the data structure. Athena queries the data using SQL. This exact architecture powers data pipelines at companies ranging from startups to Fortune 500s.&lt;/p>
&lt;p>Let&amp;rsquo;s break down each component.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="s3-understanding-the-foundation-of-modern-data-storage">S3: Understanding the foundation of modern data storage&lt;/h3>
&lt;hr>
&lt;p>Amazon S3 (Simple Storage Service) is probably the most important AWS service to understand. Not because it&amp;rsquo;s complex—it&amp;rsquo;s actually remarkably simple—but because it&amp;rsquo;s foundational. Nearly every data pipeline on AWS touches S3 at some point.&lt;/p>
&lt;p>Think of S3 as an infinite file system. You create buckets (essentially folders), upload files, organise them into more folders, and retrieve them when needed. That&amp;rsquo;s it. But the elegance is in what S3 handles automatically:&lt;/p>
&lt;p>&lt;strong>Durability&lt;/strong>: S3 provides 99.999999999% durability (that&amp;rsquo;s eleven nines). In practical terms, if you store 10 million files in S3, you can expect to lose one file every 10,000 years. You&amp;rsquo;re more likely to lose data from your laptop than from S3.&lt;/p>
&lt;p>&lt;strong>Scalability&lt;/strong>: S3 scales automatically. Store one file or one billion files—S3 doesn&amp;rsquo;t care. No capacity planning required. No running out of disk space. Just upload data and S3 expands to accommodate it.&lt;/p>
&lt;p>&lt;strong>Availability&lt;/strong>: S3 is designed for 99.99% availability, meaning it&amp;rsquo;s accessible 99.99% of the time. For most data pipelines, brief unavailability during off-peak hours doesn&amp;rsquo;t matter, but if you need higher availability, S3 offers different storage classes with varying guarantees.&lt;/p>
&lt;p>&lt;strong>Cost efficiency&lt;/strong>: S3 charges approximately $0.023 per GB per month. That&amp;rsquo;s about $23 to store a terabyte. Compare that to buying and maintaining physical storage, and the economics are compelling.&lt;/p>
&lt;p>But the real power of S3 comes from how other AWS services integrate with it. Lambda can write to S3. Glue can catalog S3 data. Athena can query S3 directly. EMR can process S3 files. Redshift can load from S3. The entire AWS ecosystem orbits around S3 as the central data store.&lt;/p>
&lt;p>This is why data teams use the term &amp;ldquo;data lake&amp;rdquo;—S3 serves as the repository where all raw data lands before being processed, transformed, or analysed. The lake metaphor fits because you dump everything in, organise it later, and fish out what you need when you need it.&lt;/p>
&lt;p>Let&amp;rsquo;s create your first bucket and establish the organisational patterns professionals use.&lt;/p>
&lt;/br>
&lt;h4 id="creating-your-s3-bucket-the-details-that-actually-matter">Creating your S3 bucket: The details that actually matter&lt;/h4>
&lt;div style="max-width: 600px; margin: 0 auto;">
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-11-26-aws-sourcing/images/001-s3.jpg" alt="s3">&lt;/p>
&lt;/div>
&lt;p>Open the AWS Console and search for S3. You&amp;rsquo;ll see a dashboard showing existing buckets (probably none if this is your first time). Click &amp;ldquo;Create bucket.&amp;rdquo;&lt;/p>
&lt;div style="max-width: 300px; margin: 0 auto;">
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-11-26-aws-sourcing/images/002-CreateBucket.jpg" alt="CreateBucket">&lt;/p>
&lt;/div>
&lt;p>The first critical decision: bucket name. Here&amp;rsquo;s what matters:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Globally unique&lt;/strong>: Every S3 bucket name across all AWS accounts worldwide must be unique. Not just unique to you—unique period. This is because bucket names become part of URLs: &lt;code>https://your-bucket-name.s3.amazonaws.com/&lt;/code>&lt;/li>
&lt;li>&lt;strong>Naming conventions&lt;/strong>: Use lowercase letters, numbers, and hyphens. No underscores, no uppercase, no spaces. Keep it descriptive but concise.&lt;/li>
&lt;/ul>
&lt;p>I&amp;rsquo;m naming mine &lt;code>spansh-data-pipeline-ch-2025&lt;/code>. The date suffix helps ensure uniqueness. In production environments, teams often use patterns like &lt;code>company-team-purpose-environment-date&lt;/code> (e.g., &lt;code>acme-analytics-sales-prod-2025&lt;/code>).&lt;/p>
&lt;p>&lt;strong>Region selection&lt;/strong>: Buckets exist in specific AWS regions. This matters for two reasons:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Latency&lt;/strong>: Data access is faster when the bucket is geographically closer to the service accessing it&lt;/li>
&lt;li>&lt;strong>Compliance&lt;/strong>: Some regulations require data to stay in specific geographic regions&lt;/li>
&lt;/ol>
&lt;p>Choose the region closest to where you or your services will access the data. I&amp;rsquo;m using &lt;code>us-east-1&lt;/code> because it&amp;rsquo;s AWS&amp;rsquo;s original region, tends to get new features first, and often has the most competitive pricing.&lt;/p>
&lt;p>&lt;strong>Block Public Access&lt;/strong>: See those four checkboxes about blocking public access? Keep them all checked. By default, S3 blocks all public access to your data, which is exactly what you want. The number of data breaches caused by accidentally public S3 buckets is staggering. Capital One&amp;rsquo;s 2019 breach exposed data of 100 million people partly due to misconfigured S3 permissions. Don&amp;rsquo;t be a statistic. Block public access.&lt;/p>
&lt;p>&lt;strong>Versioning&lt;/strong>: Enable this. Versioning means S3 keeps every version of every file you upload. If you overwrite a file, the old version remains accessible. If you delete a file, S3 keeps it (marked as deleted) and you can restore it. This is your insurance policy against mistakes. The cost is minimal—you only pay for the storage of old versions—but the protection is invaluable.&lt;/p>
&lt;p>Click through the rest of the settings (the defaults are fine), and create your bucket.&lt;/p>
&lt;/br>
&lt;h4 id="organising-your-data-the-partition-pattern-that-scales">Organising your data: The partition pattern that scales&lt;/h4>
&lt;p>You&amp;rsquo;ve got an empty bucket. Before adding any data, let&amp;rsquo;s establish an organisational structure that will serve you whether you&amp;rsquo;re storing megabytes or petabytes.&lt;/p>
&lt;p>The pattern we&amp;rsquo;re using is called partitioning, and it looks like this:&lt;/p>



&lt;div class="goat svg-container ">
 
 &lt;svg
 xmlns="http://www.w3.org/2000/svg"
 font-family="Menlo,Lucida Console,monospace"
 
 viewBox="0 0 216 121"
 >
 &lt;g transform='translate(8,16)'>
&lt;text text-anchor='middle' x='0' y='4' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='8' y='4' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='16' y='4' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='16' y='20' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='24' y='4' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='24' y='20' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='32' y='4' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='32' y='20' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='32' y='36' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='40' y='4' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='40' y='20' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='40' y='36' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='48' y='4' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='48' y='36' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='48' y='52' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='56' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='56' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='56' y='52' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='64' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='64' y='36' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='64' y='52' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='64' y='68' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='72' y='4' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='72' y='36' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='72' y='52' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='72' y='68' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='80' y='36' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='80' y='52' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='80' y='68' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='80' y='84' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='88' y='52' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='88' y='68' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='88' y='84' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='96' y='52' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='96' y='68' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='96' y='84' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='96' y='100' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='104' y='52' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='104' y='68' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='104' y='84' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='104' y='100' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='112' y='52' fill='currentColor' style='font-size:1em'>5&lt;/text>
&lt;text text-anchor='middle' x='112' y='68' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='112' y='84' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='112' y='100' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='120' y='52' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='120' y='68' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='120' y='84' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='120' y='100' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='128' y='68' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='128' y='84' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='128' y='100' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='136' y='100' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='144' y='100' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='152' y='100' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='160' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='168' y='100' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='176' y='100' fill='currentColor' style='font-size:1em'>j&lt;/text>
&lt;text text-anchor='middle' x='184' y='100' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='192' y='100' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='200' y='100' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;/g>

 &lt;/svg>
 
&lt;/div>
&lt;p>This structure is deliberate. Let me explain why each piece matters:&lt;/p>
&lt;p>&lt;strong>raw/&lt;/strong>: This is your persistent staging layer. Data lands here first, exactly as received from the source. Never modify raw data. Never delete raw data. If something goes wrong downstream—a transformation bug, a bad query, a misunderstood requirement—you can always return to the raw data and start over. This folder is sacred.&lt;/p>
&lt;p>&lt;strong>spansh/&lt;/strong>: The data source. If you&amp;rsquo;re ingesting from multiple sources, you&amp;rsquo;ll have multiple folders here: &lt;code>salesforce/&lt;/code>, &lt;code>stripe/&lt;/code>, &lt;code>google-analytics/&lt;/code>, whatever. This keeps sources separated and makes permissions management easier.&lt;/p>
&lt;p>&lt;strong>year=YYYY/month=MM/day=DD/&lt;/strong>: This is partition scheme, and it&amp;rsquo;s where the magic happens. When you query this data later using Athena, you can filter by date without scanning every file ever created. Instead of &amp;ldquo;search all files for records from January 22,&amp;rdquo; you tell Athena &amp;ldquo;only look in the year=2025/month=01/day=22 folder.&amp;rdquo; This makes queries faster and cheaper (Athena charges by data scanned).&lt;/p>
&lt;p>The specific format—&lt;code>year=2025&lt;/code> rather than just &lt;code>2025&lt;/code>—is Hive-style partitioning. It&amp;rsquo;s a standard format that AWS Glue and Athena recognise automatically. When Glue crawls this data, it understands that &lt;code>year&lt;/code>, &lt;code>month&lt;/code>, and &lt;code>day&lt;/code> are partition columns and configures the table schema accordingly.&lt;/p>
&lt;p>Alternative structures exist: &lt;code>year/month/day&lt;/code>, &lt;code>yyyy-mm-dd&lt;/code>, &lt;code>source/date/hour&lt;/code>. The choice depends on query patterns. Daily partitions work for most use cases. If you&amp;rsquo;re ingesting data hourly, add an &lt;code>hour=HH&lt;/code> level. If data arrives weekly, daily partitions might create too many folders. Adjust based on your access patterns.&lt;/p>
&lt;p>Partitioning strategy is only half the battle—file size within those partitions determines whether your queries run smoothly or become expensive disasters.&lt;/p>
&lt;p>The sweet spot for S3 files in data lake architectures is typically &lt;strong>128MB to 1GB&lt;/strong>. Why? Because of how distributed query engines like Athena actually work.&lt;/p>
&lt;p>When Athena reads your data, it splits work across multiple parallel workers. Each worker grabs a chunk of data to process. If your partition contains one massive 5GB file, you get limited parallelism—maybe only a few workers can operate simultaneously, and some might run out of memory trying to process such large chunks. Your query takes longer, costs more, and might even fail.&lt;/p>
&lt;p>But here&amp;rsquo;s the twist: having thousands of tiny files is worse. Much worse.&lt;/p>
&lt;p>Say you&amp;rsquo;re ingesting data every few minutes and each file is 5MB. After a month, you have 8,000 files in a single partition. When Athena queries that partition, it needs to:&lt;/p>
&lt;ul>
&lt;li>List all 8,000 files (API calls to S3)&lt;/li>
&lt;li>Open each file (more API calls)&lt;/li>
&lt;li>Read the metadata from each file&lt;/li>
&lt;li>Coordinate across thousands of small tasks&lt;/li>
&lt;/ul>
&lt;p>This creates what&amp;rsquo;s known as the &amp;ldquo;small files problem.&amp;rdquo; You&amp;rsquo;re drowning in metadata operations. The query engine spends more time managing files than actually processing data. You get some parallelism, sure, but the overhead destroys any benefit. Queries are slow despite having multiple files. Costs are higher because you&amp;rsquo;re paying for all those S3 API calls and Athena has to scan metadata for thousands of tiny files.&lt;/p>
&lt;p>Real example: I&amp;rsquo;ve seen cases where a partition with 10,000 files totaling 50GB took longer to query than a partition with 50 files totaling 500GB. The metadata overhead from thousands of small files was the killer.&lt;/p>
&lt;p>The solution? Batch and compact your data before writing to S3. If your Lambda function runs every 5 minutes and produces small files, you have two options:&lt;/p>
&lt;p>&lt;strong>Option 1&lt;/strong>: Buffer data in memory and only write to S3 once you&amp;rsquo;ve accumulated 100MB+ of data. This requires tracking state across Lambda invocations, which adds complexity.&lt;/p>
&lt;p>&lt;strong>Option 2&lt;/strong>: Write small files to a temporary location, then run a separate compaction process (maybe hourly) that reads all small files for a partition and writes them out as properly-sized files. This is cleaner and more reliable.&lt;/p>
&lt;p>For the data we&amp;rsquo;re working with, files are naturally around 100-200MB compressed, which is perfect. But if you&amp;rsquo;re adapting this pattern for high-frequency data ingestion—like streaming sensor data or click events—pay attention to file sizes. You&amp;rsquo;ll save yourself headaches and money.&lt;/p>
&lt;p>Create these folders in your bucket now (through the console, just click &amp;ldquo;Create folder&amp;rdquo; and build the hierarchy). Once you have the structure in place, we&amp;rsquo;re ready to add the automation.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="lambda-the-robot-that-never-sleeps">Lambda: The robot that never sleeps&lt;/h3>
&lt;hr>
&lt;p>Lambda is AWS&amp;rsquo;s serverless compute service, and if you&amp;rsquo;ve never worked with serverless before, the concept might feel strange at first. But stick with me—once you get it, you&amp;rsquo;ll wonder how you ever lived without it.&lt;/p>
&lt;p>Traditional computing requires servers. You spin up a virtual machine (EC2 in AWS terms), install your code, run it, and the server sits there consuming resources (and costing money) whether it&amp;rsquo;s doing work or not. Even if your code runs for 5 minutes per day, you&amp;rsquo;re paying for 24 hours of server time.&lt;/p>
&lt;p>Lambda inverts this. You write code, upload it to Lambda, specify a trigger, and Lambda runs your code only when triggered. The code executes, completes, and Lambda terminates the execution environment. You pay only for the compute time you actually use, measured in milliseconds.&lt;/p>
&lt;p>For scheduled tasks—like pulling data from an API once every day—this is perfect. Your code runs once daily, downloads data, saves it to S3, and stops. Total execution time: maybe 10-30 seconds. Total cost: fractions of a cent.&lt;/p>
&lt;p>The constraints you need to know:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>15-minute maximum runtime&lt;/strong>: Lambda kills any function running longer than 15 minutes. For data ingestion from APIs, this is rarely an issue. For processing multi-gigabyte files, you&amp;rsquo;d need a different approach.&lt;/li>
&lt;li>&lt;strong>10GB maximum memory&lt;/strong>: Memory allocation ranges from 128MB to 10GB. More memory also means more CPU power, but for API calls, even 512MB is usually plenty.&lt;/li>
&lt;li>&lt;strong>512MB temporary storage&lt;/strong>: Lambda provides &lt;code>/tmp&lt;/code> storage during execution, but it&amp;rsquo;s cleared between runs. Don&amp;rsquo;t try to store data in Lambda itself—always write to S3.&lt;/li>
&lt;/ul>
&lt;p>These limitations make Lambda unsuitable for certain tasks (long-running batch processing, stateful applications, extremely large data processing). But for scheduled API ingestion? Lambda is ideal.&lt;/p>
&lt;p>Let&amp;rsquo;s build our function.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h4 id="building-the-lambda-function-that-does-the-actual-work">Building the Lambda function that does the actual work&lt;/h4>
&lt;div style="max-width: 600px; margin: 0 auto;">
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-11-26-aws-sourcing/images/003-lambda.jpg" alt="lambda">&lt;/p>
&lt;/div>
&lt;p>Navigate to Lambda in the AWS Console. Click &amp;ldquo;Create function.&amp;rdquo; You&amp;rsquo;ll see three options:&lt;/p>
&lt;div style="max-width: 300px; margin: 0 auto;">
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-11-26-aws-sourcing/images/004-CreateLambda.jpg" alt="Create Lambda">&lt;/p>
&lt;/div>
&lt;ol>
&lt;li>Author from scratch (we&amp;rsquo;ll use this)&lt;/li>
&lt;li>Use a blueprint (pre-built templates)&lt;/li>
&lt;li>Container image (for more complex deployments)&lt;/li>
&lt;/ol>
&lt;p>Choose &amp;ldquo;Author from scratch&amp;rdquo; and provide:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Function name&lt;/strong>: &lt;code>spansh-data-ingestion&lt;/code> (descriptive, hyphenated, no spaces)&lt;/li>
&lt;li>&lt;strong>Runtime&lt;/strong>: Python 3.11 (or latest available—Lambda supports multiple languages, but Python is excellent for data work)&lt;/li>
&lt;li>&lt;strong>Architecture&lt;/strong>: x86_64 (the default—arm64 is cheaper but x86_64 has better compatibility)&lt;/li>
&lt;/ul>
&lt;div style="max-width: 600px; margin: 0 auto;">
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-11-26-aws-sourcing/images/005-CreateFunction.jpg" alt="Create Function">&lt;/p>
&lt;/div>
&lt;p>For permissions, Lambda needs to assume a role that grants it access to other AWS services. Click &amp;ldquo;Create a new role with basic Lambda permissions.&amp;rdquo; This creates an IAM role with permissions to write logs to CloudWatch (for debugging). We&amp;rsquo;ll add S3 permissions shortly.&lt;/p>
&lt;p>Click &amp;ldquo;Create function&amp;rdquo; and AWS sets up your Lambda environment.&lt;/p>
&lt;p>You&amp;rsquo;ll land on the function configuration page. The centre shows a code editor with a default template. Delete that template—we&amp;rsquo;re writing our own.&lt;/p>
&lt;div style="max-width: 600px; margin: 0 auto;">
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-11-26-aws-sourcing/images/006-PythonCode.jpg" alt="Code">&lt;/p>
&lt;/div>
&lt;p>Here&amp;rsquo;s the code we&amp;rsquo;re implementing:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> json
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> boto3
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> urllib3
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> datetime &lt;span style="color:#f92672">import&lt;/span> datetime
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">lambda_handler&lt;/span>(event, context):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Downloads data from SPANSH API and saves to S3.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> This function is triggered by EventBridge on a schedule.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Configuration - in production, use environment variables&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> BUCKET_NAME &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;spansh-data-pipeline-ch-2025&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> API_URL &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;https://spansh.co.uk/api/systems&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Initialize AWS clients&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> s3 &lt;span style="color:#f92672">=&lt;/span> boto3&lt;span style="color:#f92672">.&lt;/span>client(&lt;span style="color:#e6db74">&amp;#39;s3&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> http &lt;span style="color:#f92672">=&lt;/span> urllib3&lt;span style="color:#f92672">.&lt;/span>PoolManager()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Download data from API&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Fetching data from &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>API_URL&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> response &lt;span style="color:#f92672">=&lt;/span> http&lt;span style="color:#f92672">.&lt;/span>request(&lt;span style="color:#e6db74">&amp;#39;GET&amp;#39;&lt;/span>, API_URL)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> response&lt;span style="color:#f92672">.&lt;/span>status &lt;span style="color:#f92672">!=&lt;/span> &lt;span style="color:#ae81ff">200&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">raise&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span>(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;API request failed with status &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>response&lt;span style="color:#f92672">.&lt;/span>status&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Parse response&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> data &lt;span style="color:#f92672">=&lt;/span> json&lt;span style="color:#f92672">.&lt;/span>loads(response&lt;span style="color:#f92672">.&lt;/span>data&lt;span style="color:#f92672">.&lt;/span>decode(&lt;span style="color:#e6db74">&amp;#39;utf-8&amp;#39;&lt;/span>))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Successfully retrieved &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(data)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> records&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Generate S3 key with current date partitioning&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> now &lt;span style="color:#f92672">=&lt;/span> datetime&lt;span style="color:#f92672">.&lt;/span>utcnow()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> year &lt;span style="color:#f92672">=&lt;/span> now&lt;span style="color:#f92672">.&lt;/span>strftime(&lt;span style="color:#e6db74">&amp;#39;%Y&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> month &lt;span style="color:#f92672">=&lt;/span> now&lt;span style="color:#f92672">.&lt;/span>strftime(&lt;span style="color:#e6db74">&amp;#39;%m&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> day &lt;span style="color:#f92672">=&lt;/span> now&lt;span style="color:#f92672">.&lt;/span>strftime(&lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#e6db74">%d&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> timestamp &lt;span style="color:#f92672">=&lt;/span> now&lt;span style="color:#f92672">.&lt;/span>strftime(&lt;span style="color:#e6db74">&amp;#39;%Y%m&lt;/span>&lt;span style="color:#e6db74">%d&lt;/span>&lt;span style="color:#e6db74">_%H%M%S&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> s3_key &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;raw/spansh/year=&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>year&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">/month=&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>month&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">/day=&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>day&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">/spansh_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>timestamp&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">.json&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Upload to S3&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Uploading to s3://&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>BUCKET_NAME&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">/&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>s3_key&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> s3&lt;span style="color:#f92672">.&lt;/span>put_object(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> Bucket&lt;span style="color:#f92672">=&lt;/span>BUCKET_NAME,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> Key&lt;span style="color:#f92672">=&lt;/span>s3_key,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> Body&lt;span style="color:#f92672">=&lt;/span>json&lt;span style="color:#f92672">.&lt;/span>dumps(data, indent&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">2&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ContentType&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;application/json&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Upload successful&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;statusCode&amp;#39;&lt;/span>: &lt;span style="color:#ae81ff">200&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;body&amp;#39;&lt;/span>: json&lt;span style="color:#f92672">.&lt;/span>dumps({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;message&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;Data ingestion successful&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;records&amp;#39;&lt;/span>: len(data),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;s3_location&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;s3://&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>BUCKET_NAME&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">/&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>s3_key&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;statusCode&amp;#39;&lt;/span>: &lt;span style="color:#ae81ff">500&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;body&amp;#39;&lt;/span>: json&lt;span style="color:#f92672">.&lt;/span>dumps({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;message&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;Data ingestion failed&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;error&amp;#39;&lt;/span>: str(e)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Let&amp;rsquo;s break down what this code does, because understanding each piece helps you adapt it for your own APIs:&lt;/p>
&lt;p>&lt;strong>Import statements&lt;/strong>: We&amp;rsquo;re using &lt;code>boto3&lt;/code> (AWS&amp;rsquo;s Python SDK for interacting with AWS services), &lt;code>urllib3&lt;/code> (for HTTP requests—simpler than &lt;code>requests&lt;/code> and doesn&amp;rsquo;t require external dependencies), and &lt;code>datetime&lt;/code> (for generating date-based paths).&lt;/p>
&lt;p>&lt;strong>lambda_handler function&lt;/strong>: This is the entry point. Lambda calls this function when triggered. The parameters &lt;code>event&lt;/code> and &lt;code>context&lt;/code> contain information about the trigger and execution environment (we&amp;rsquo;re not using them here, but they&amp;rsquo;re standard).&lt;/p>
&lt;p>&lt;strong>Configuration&lt;/strong>: The bucket name and API URL are hardcoded here for simplicity. In production, you&amp;rsquo;d use environment variables (Lambda lets you set these in the function configuration) or AWS Systems Manager Parameter Store for more secure configuration management.&lt;/p>
&lt;p>&lt;strong>AWS client initialisation&lt;/strong>: &lt;code>boto3.client('s3')&lt;/code> creates an S3 client for uploading files. Lambda automatically provides credentials for the IAM role attached to the function—no need to manage API keys.&lt;/p>
&lt;p>&lt;strong>API request&lt;/strong>: We fetch data from the SPANSH API. If the request fails, we raise an exception, which triggers the error handling block.&lt;/p>
&lt;p>&lt;strong>Partition path generation&lt;/strong>: This is crucial. We generate the S3 key using the current date, creating that &lt;code>year=YYYY/month=MM/day=DD&lt;/code> structure we discussed. The timestamp in the filename ensures multiple uploads per day don&amp;rsquo;t overwrite each other.&lt;/p>
&lt;p>&lt;strong>S3 upload&lt;/strong>: &lt;code>s3.put_object()&lt;/code> writes the JSON data to S3. We&amp;rsquo;re formatting the JSON with indentation (&lt;code>indent=2&lt;/code>) to make it human-readable—useful for debugging. In production with massive files, you&amp;rsquo;d skip this to reduce file size.&lt;/p>
&lt;p>&lt;strong>Error handling&lt;/strong>: The try/except block catches any failures and returns an appropriate error message. Importantly, we&amp;rsquo;re using &lt;code>print()&lt;/code> statements throughout. These logs appear in CloudWatch, AWS&amp;rsquo;s logging service, which is invaluable for debugging.&lt;/p>
&lt;p>&lt;strong>Return value&lt;/strong>: Lambda functions should return a response. Here we&amp;rsquo;re returning HTTP-style status codes and messages, though since we&amp;rsquo;re running on a schedule (not via API Gateway), nothing actually receives this response. Still, it&amp;rsquo;s good practice—makes the function reusable if you later expose it as an API.&lt;/p>
&lt;p>Copy this code into the Lambda editor and click &amp;ldquo;Deploy&amp;rdquo; to save.&lt;/p>
&lt;p>But we&amp;rsquo;re not done yet. This function will fail if you test it right now. Why? Permissions.&lt;/p>
&lt;/br>
&lt;h4 id="giving-lambda-permission-to-write-to-s3">Giving Lambda permission to write to S3&lt;/h4>
&lt;p>Lambda runs with the permissions granted by its IAM role. By default, that role can only write logs. It cannot access S3. We need to fix that.&lt;/p>
&lt;div style="max-width: 600px; margin: 0 auto;">
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-11-26-aws-sourcing/images/007-IAM.jpg" alt="iam">&lt;/p>
&lt;/div>
&lt;p>In your Lambda function page, click the &amp;ldquo;Configuration&amp;rdquo; tab, then &amp;ldquo;Permissions&amp;rdquo; in the left sidebar. You&amp;rsquo;ll see the execution role Lambda is using. Click the role name—it opens in a new tab showing the IAM console.&lt;/p>
&lt;div style="max-width: 400px; margin: 0 auto;">
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-11-26-aws-sourcing/images/008-IAM-AttachPolicies.jpg" alt="policies">&lt;/p>
&lt;/div>
&lt;p>You&amp;rsquo;ll see an &amp;ldquo;Add permissions&amp;rdquo; button. Click it and choose &amp;ldquo;Attach policies.&amp;rdquo; Search for &lt;code>AmazonS3FullAccess&lt;/code> and attach it.&lt;/p>
&lt;p>&lt;strong>Wait—full access? Isn&amp;rsquo;t that dangerous?&lt;/strong>&lt;/p>
&lt;p>Yes. In production, you&amp;rsquo;d follow the principle of least privilege: grant only the specific permissions needed (in this case, &lt;code>s3:PutObject&lt;/code> on the specific bucket and path). But for learning and prototyping, full S3 access is easier. Just remember to tighten permissions before production deployment.&lt;/p>
&lt;p>The proper policy would look like this:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-json" data-lang="json">&lt;span style="display:flex;">&lt;span>{
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Version&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;2012-10-17&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Statement&amp;#34;&lt;/span>: [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Effect&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Allow&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Action&amp;#34;&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;s3:PutObject&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;Resource&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;arn:aws:s3:::spansh-data-pipeline-ch-2025/raw/*&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This grants permission to write objects (files) only within the &lt;code>raw/&lt;/code> prefix of your specific bucket. Nothing else. If you&amp;rsquo;re comfortable with IAM policies, create this custom policy instead of using the full access policy.&lt;/p>
&lt;p>Now Lambda can write to S3. Let&amp;rsquo;s test it.&lt;/p>
&lt;/br>
&lt;h4 id="testing-lambda-before-scheduling-it">Testing Lambda before scheduling it&lt;/h4>
&lt;p>In the Lambda console, click the &amp;ldquo;Test&amp;rdquo; tab. Lambda provides a test event feature that simulates triggers. Since our function doesn&amp;rsquo;t actually use the event data, the default test event works fine. Click &amp;ldquo;Test.&amp;rdquo;&lt;/p>
&lt;p>Watch the execution results panel. If everything works, you&amp;rsquo;ll see:&lt;/p>
&lt;ul>
&lt;li>Execution successful&lt;/li>
&lt;li>Logs showing each print statement&lt;/li>
&lt;li>The return value with status 200 and the S3 location&lt;/li>
&lt;/ul>
&lt;p>If it fails, read the error message carefully. Common issues:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>&amp;ldquo;Access Denied&amp;rdquo;&lt;/strong>: IAM permissions aren&amp;rsquo;t configured correctly&lt;/li>
&lt;li>&lt;strong>&amp;ldquo;Unable to import module&amp;rdquo;&lt;/strong>: Code syntax error or missing dependency&lt;/li>
&lt;li>&lt;strong>&amp;ldquo;Timeout&amp;rdquo;&lt;/strong>: Function took longer than the configured timeout (default is 3 seconds—we need to increase that)&lt;/li>
&lt;/ul>
&lt;p>Speaking of timeout: Go to Configuration → General configuration → Edit. Set the timeout to 60 seconds (1 minute). This gives Lambda enough time to download data and upload to S3. The default 3 seconds isn&amp;rsquo;t enough for API calls.&lt;/p>
&lt;p>Once your test succeeds, check S3. Navigate to your bucket and drill into the folder structure. You should see a JSON file in the correct date-partitioned folder. Open it to verify the data looks correct.&lt;/p>
&lt;p>Congratulations—you&amp;rsquo;ve manually triggered your first Lambda function to ingest data into S3. Now let&amp;rsquo;s make it run automatically.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="eventbridge-the-clock-that-triggers-your-pipeline">EventBridge: The clock that triggers your pipeline&lt;/h3>
&lt;hr>
&lt;p>Your Lambda function works perfectly when you manually test it. But the whole point is automation—having it run on a schedule without you clicking anything.&lt;/p>
&lt;p>Enter Amazon EventBridge (formerly called CloudWatch Events—you might see that name in older documentation). EventBridge is a serverless event bus that routes events from various sources to various targets. We&amp;rsquo;re using its scheduler feature to trigger Lambda on a daily schedule.&lt;/p>
&lt;p>EventBridge schedules are written using cron expressions or rate expressions, the same syntax used in Unix systems for decades. If you&amp;rsquo;ve ever scheduled anything on a Linux server, you already know this syntax.&lt;/p>
&lt;p>Let&amp;rsquo;s create the schedule.&lt;/p>
&lt;/br>
&lt;h4 id="creating-the-eventbridge-schedule">Creating the EventBridge schedule&lt;/h4>
&lt;div style="max-width: 600px; margin: 0 auto;">
&lt;p>&lt;img src="images/009-EventBridge.jpg" alt="Event Bridge">&lt;/p>
&lt;/div>
&lt;p>In the AWS Console, search for &amp;ldquo;EventBridge&amp;rdquo; and select &amp;ldquo;EventBridge Scheduler&amp;rdquo; (not &amp;ldquo;EventBridge Rules&amp;rdquo;—Scheduler is the newer, simplified interface).&lt;/p>
&lt;div style="max-width: 300px; margin: 0 auto;">
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-11-26-aws-sourcing/images/010-EventBridgeScheduler.jpg" alt="Scheduler">&lt;/p>
&lt;/div>
&lt;p>Click &amp;ldquo;Create schedule.&amp;rdquo; You&amp;rsquo;ll walk through a wizard:&lt;/p>
&lt;p>&lt;strong>Schedule name&lt;/strong>: &lt;code>spansh-daily-ingestion&lt;/code> (descriptive, indicates what it does)&lt;/p>
&lt;p>&lt;strong>Schedule pattern&lt;/strong>: Choose &amp;ldquo;Recurring schedule&amp;rdquo;&lt;/p>
&lt;p>For the cron expression, you have options:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Rate expressions&lt;/strong>: Simple intervals like &amp;ldquo;every 1 day&amp;rdquo; or &amp;ldquo;every 6 hours&amp;rdquo;&lt;/li>
&lt;li>&lt;strong>Cron expressions&lt;/strong>: More precise control like &amp;ldquo;every day at 3 AM UTC&amp;rdquo;&lt;/li>
&lt;/ul>
&lt;p>Let&amp;rsquo;s use a cron expression for precision. The syntax is:&lt;/p>



&lt;div class="goat svg-container ">
 
 &lt;svg
 xmlns="http://www.w3.org/2000/svg"
 font-family="Menlo,Lucida Console,monospace"
 
 viewBox="0 0 448 25"
 >
 &lt;g transform='translate(8,16)'>
&lt;text text-anchor='middle' x='0' y='4' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='8' y='4' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='16' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='24' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='32' y='4' fill='currentColor' style='font-size:1em'>(&lt;/text>
&lt;text text-anchor='middle' x='40' y='4' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='48' y='4' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='56' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='64' y='4' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='72' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='80' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='88' y='4' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='104' y='4' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='112' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='120' y='4' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='128' y='4' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='136' y='4' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='152' y='4' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='160' y='4' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='168' y='4' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='176' y='4' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='184' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='192' y='4' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='200' y='4' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='208' y='4' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='216' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='224' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='232' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='240' y='4' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='256' y='4' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='264' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='272' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='280' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='288' y='4' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='304' y='4' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='312' y='4' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='320' y='4' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='328' y='4' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='336' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='344' y='4' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='352' y='4' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='360' y='4' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='368' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='376' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='384' y='4' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='400' y='4' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='408' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='416' y='4' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='424' y='4' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='432' y='4' fill='currentColor' style='font-size:1em'>)&lt;/text>
&lt;/g>

 &lt;/svg>
 
&lt;/div>
&lt;p>For daily at 9 AM UTC:&lt;/p>



&lt;div class="goat svg-container ">
 
 &lt;svg
 xmlns="http://www.w3.org/2000/svg"
 font-family="Menlo,Lucida Console,monospace"
 
 viewBox="0 0 144 25"
 >
 &lt;g transform='translate(8,16)'>
&lt;circle cx='72' cy='0' r='6' stroke='currentColor' fill='currentColor'>&lt;/circle>
&lt;circle cx='88' cy='0' r='6' stroke='currentColor' fill='currentColor'>&lt;/circle>
&lt;circle cx='120' cy='0' r='6' stroke='currentColor' fill='currentColor'>&lt;/circle>
&lt;text text-anchor='middle' x='0' y='4' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='8' y='4' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='16' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='24' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='32' y='4' fill='currentColor' style='font-size:1em'>(&lt;/text>
&lt;text text-anchor='middle' x='40' y='4' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='56' y='4' fill='currentColor' style='font-size:1em'>9&lt;/text>
&lt;text text-anchor='middle' x='104' y='4' fill='currentColor' style='font-size:1em'>?&lt;/text>
&lt;/g>

 &lt;/svg>
 
&lt;/div>
&lt;p>What this means:&lt;/p>
&lt;ul>
&lt;li>&lt;code>0&lt;/code>: Minute 0 (on the hour)&lt;/li>
&lt;li>&lt;code>9&lt;/code>: Hour 9 (9 AM)&lt;/li>
&lt;li>&lt;code>*&lt;/code>: Any day of month&lt;/li>
&lt;li>&lt;code>*&lt;/code>: Any month&lt;/li>
&lt;li>&lt;code>?&lt;/code>: Any day of week (? is required when * is used for day-of-month)&lt;/li>
&lt;li>&lt;code>*&lt;/code>: Any year&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Important&lt;/strong>: EventBridge uses UTC time. If you want 9 AM in your local timezone, adjust accordingly. For example, if you&amp;rsquo;re in US Eastern Time (UTC-5), 9 AM EST is 2 PM UTC, so use &lt;code>cron(0 14 * * ? *)&lt;/code>.&lt;/p>
&lt;p>&lt;strong>Flexible time window&lt;/strong>: You can add a window where the schedule might execute (e.g., &amp;ldquo;anytime within 1 hour&amp;rdquo;). This helps AWS optimise resource usage. For data ingestion, exact timing usually doesn&amp;rsquo;t matter, so a 15-minute window is fine.&lt;/p>
&lt;p>&lt;strong>Timezone&lt;/strong>: Choose your timezone (AWS adjusts the cron expression automatically).&lt;/p>
&lt;p>Click &amp;ldquo;Next.&amp;rdquo;&lt;/p>
&lt;p>&lt;strong>Target&lt;/strong>: Select &amp;ldquo;AWS Lambda&amp;rdquo; and choose your &lt;code>spansh-data-ingestion&lt;/code> function from the dropdown.&lt;/p>
&lt;p>&lt;strong>Payload&lt;/strong>: Leave empty—our function doesn&amp;rsquo;t use event data.&lt;/p>
&lt;p>Click through the remaining settings (defaults are fine) and create the schedule.&lt;/p>
&lt;p>You&amp;rsquo;ve now built a complete automated data pipeline. Every day at your scheduled time, EventBridge triggers Lambda, which downloads data from the API and stores it in S3 in a properly partitioned structure.&lt;/p>
&lt;p>But we&amp;rsquo;re not quite done. The data is in S3, but querying it directly is cumbersome. We need to make this data accessible via SQL.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="glue-teaching-aws-about-your-data-structure">Glue: Teaching AWS about your data structure&lt;/h3>
&lt;hr>
&lt;p>You&amp;rsquo;ve got JSON files sitting in S3, partitioned by date. To query this data using SQL, AWS needs to understand the structure: what columns exist, what data types they have, how the data is partitioned.&lt;/p>
&lt;p>That&amp;rsquo;s where AWS Glue comes in—specifically, the Glue Data Catalog.&lt;/p>
&lt;p>Think of the Glue Data Catalog as a metadata repository. It stores schemas (table definitions) that describe data in S3. Once Glue has cataloged your data, services like Athena can query it using SQL without you manually defining the schema every time.&lt;/p>
&lt;p>Glue offers two approaches for cataloging data:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Manual schema definition&lt;/strong>: You write code or use the console to explicitly define the schema&lt;/li>
&lt;li>&lt;strong>Glue Crawlers&lt;/strong>: Automated processes that scan your S3 data, infer the schema, and create tables automatically&lt;/li>
&lt;/ol>
&lt;p>For most use cases, crawlers are easier. They&amp;rsquo;re like database schema discovery tools—point them at your data, and they figure out the structure.&lt;/p>
&lt;p>Let&amp;rsquo;s create a crawler for our data.&lt;/p>
&lt;/br>
&lt;h4 id="creating-your-first-glue-crawler">Creating your first Glue Crawler&lt;/h4>
&lt;p>Navigate to AWS Glue in the console. On the left sidebar, click &amp;ldquo;Crawlers,&amp;rdquo; then &amp;ldquo;Create crawler.&amp;rdquo;&lt;/p>
&lt;p>&lt;strong>Crawler name&lt;/strong>: &lt;code>spansh-raw-data-crawler&lt;/code> (descriptive and indicates it&amp;rsquo;s crawling raw data)&lt;/p>
&lt;p>&lt;strong>Data source&lt;/strong>: Click &amp;ldquo;Add a data source&amp;rdquo; and configure:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Data source&lt;/strong>: S3&lt;/li>
&lt;li>&lt;strong>S3 path&lt;/strong>: Browse to your bucket and select the &lt;code>raw/spansh/&lt;/code> prefix (not the full date-partitioned path—we want the crawler to discover all partitions)&lt;/li>
&lt;li>&lt;strong>Subsequent crawler runs&lt;/strong>: Choose &amp;ldquo;Crawl all sub-folders&amp;rdquo; so the crawler discovers new partitions as they&amp;rsquo;re added&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>IAM role&lt;/strong>: Glue needs permissions to read S3 and write to the Glue Data Catalog. You can either create a new role or use an existing one. If creating new, AWS automatically configures the necessary permissions.&lt;/p>
&lt;p>&lt;strong>Target database&lt;/strong>: Glue organises tables into databases (similar to database schemas in traditional RDBMS). Create a new database called &lt;code>spansh_data&lt;/code>.&lt;/p>
&lt;p>&lt;strong>Crawler schedule&lt;/strong>: You can schedule crawlers to run periodically (similar to how we scheduled Lambda). For now, choose &amp;ldquo;On demand&amp;rdquo;—we&amp;rsquo;ll manually trigger it after Lambda runs to create the initial table, then consider scheduling it later.&lt;/p>
&lt;p>&lt;strong>Configuration options&lt;/strong>: Leave the defaults. The crawler will automatically detect schema changes and add new partitions.&lt;/p>
&lt;p>Create the crawler and click &amp;ldquo;Run crawler.&amp;rdquo;&lt;/p>
&lt;p>The crawler takes a minute or two to complete. It scans your S3 data, infers the JSON structure, detects the date partitions, and creates a table in the Glue Data Catalog.&lt;/p>
&lt;p>Once complete, navigate to &amp;ldquo;Databases&amp;rdquo; in the Glue console, click &lt;code>spansh_data&lt;/code>, then &amp;ldquo;Tables.&amp;rdquo; You should see a table created by the crawler. Click it to view the schema.&lt;/p>
&lt;p>You&amp;rsquo;ll see:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Column names and types&lt;/strong>: Inferred from your JSON structure&lt;/li>
&lt;li>&lt;strong>Partition keys&lt;/strong>: &lt;code>year&lt;/code>, &lt;code>month&lt;/code>, &lt;code>day&lt;/code> detected from the folder structure&lt;/li>
&lt;li>&lt;strong>Table properties&lt;/strong>: Location in S3, format (JSON), and other metadata&lt;/li>
&lt;/ul>
&lt;p>This table definition is what Athena will use to query your S3 data.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="athena-querying-data-without-loading-a-database">Athena: Querying data without loading a database&lt;/h3>
&lt;hr>
&lt;p>This is where everything comes together. You have data in S3, organised properly, cataloged by Glue. Now you want to query it using SQL. Enter Amazon Athena.&lt;/p>
&lt;p>Athena is a query engine that reads data directly from S3 using the schemas defined in Glue. There&amp;rsquo;s no database to set up, no data to load, no indexes to maintain. You write SQL, Athena executes it against your S3 files, and returns results.&lt;/p>
&lt;p>The underlying technology is Presto/Trino, an open-source distributed SQL query engine originally developed at Facebook. AWS packages Presto as a managed service, handles all the infrastructure, and charges you only for the data scanned—approximately $5 per terabyte queried.&lt;/p>
&lt;p>This pricing model has important implications: write efficient queries. Use partition filters. Limit the data scanned. We&amp;rsquo;ll cover optimisation shortly, but first, let&amp;rsquo;s run some queries.&lt;/p>
&lt;/br>
&lt;h4 id="setting-up-athena-and-running-your-first-query">Setting up Athena and running your first query&lt;/h4>
&lt;p>Navigate to Amazon Athena in the console. The first time you open Athena, you&amp;rsquo;ll need to configure a location for query results. Athena saves query output to S3 (because where else would it go?).&lt;/p>
&lt;p>Click &amp;ldquo;Settings&amp;rdquo; and specify an S3 path: &lt;code>s3://spansh-data-pipeline-ch-2025/athena-results/&lt;/code>&lt;/p>
&lt;p>Create this folder in your bucket if it doesn&amp;rsquo;t exist. Athena will write query results here.&lt;/p>
&lt;p>Return to the query editor. In the left sidebar, you&amp;rsquo;ll see:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Data source&lt;/strong>: AwsDataCatalog (Glue)&lt;/li>
&lt;li>&lt;strong>Database&lt;/strong>: &lt;code>spansh_data&lt;/code>&lt;/li>
&lt;li>&lt;strong>Tables&lt;/strong>: Your crawler-created table&lt;/li>
&lt;/ul>
&lt;p>Click the three dots next to your table name and select &amp;ldquo;Preview table.&amp;rdquo; Athena generates and runs a &lt;code>SELECT * LIMIT 10&lt;/code> query automatically.&lt;/p>
&lt;p>Within seconds, you see results. Data sitting in S3, queried using SQL, no database setup required. That&amp;rsquo;s Athena.&lt;/p>
&lt;p>Let&amp;rsquo;s write some more interesting queries.&lt;/p>
&lt;/br>
&lt;h4 id="useful-queries-for-validating-your-pipeline">Useful queries for validating your pipeline&lt;/h4>
&lt;p>&lt;strong>Check total record count&lt;/strong>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> total_records
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> spansh_data.your_table_name;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This scans all data to count records. Notice how Athena shows &amp;ldquo;Data scanned&amp;rdquo; at the bottom of the results—that&amp;rsquo;s what you&amp;rsquo;re paying for.&lt;/p>
&lt;p>&lt;strong>Check records by date partition&lt;/strong>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#66d9ef">year&lt;/span>, &lt;span style="color:#66d9ef">month&lt;/span>, &lt;span style="color:#66d9ef">day&lt;/span>, &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> records
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> spansh_data.your_table_name
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> &lt;span style="color:#66d9ef">year&lt;/span>, &lt;span style="color:#66d9ef">month&lt;/span>, &lt;span style="color:#66d9ef">day&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> &lt;span style="color:#66d9ef">year&lt;/span>, &lt;span style="color:#66d9ef">month&lt;/span>, &lt;span style="color:#66d9ef">day&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This shows how many records exist in each partition—useful for verifying your ingestion is running daily.&lt;/p>
&lt;p>&lt;strong>Query specific date range&lt;/strong>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> spansh_data.your_table_name
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> &lt;span style="color:#66d9ef">year&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;2025&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> &lt;span style="color:#66d9ef">month&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;01&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> &lt;span style="color:#66d9ef">day&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;22&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LIMIT&lt;/span> &lt;span style="color:#ae81ff">100&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Notice we&amp;rsquo;re filtering on partition columns. Athena only scans files in that specific partition, making the query fast and cheap.&lt;/p>
&lt;p>&lt;strong>Without partition filters&lt;/strong>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> spansh_data.your_table_name
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> some_column &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;some_value&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LIMIT&lt;/span> &lt;span style="color:#ae81ff">100&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This scans ALL files across ALL partitions looking for matching records. Expensive and slow. Always filter on partitions when possible.&lt;/p>
&lt;h4 id="keeping-partitions-updated">Keeping partitions updated&lt;/h4>
&lt;p>Here&amp;rsquo;s a gotcha: when Lambda adds new files to S3 in new date partitions, Glue doesn&amp;rsquo;t automatically know about them. Your table still only knows about partitions that existed when the crawler last ran.&lt;/p>
&lt;p>You have three options:&lt;/p>
&lt;p>&lt;strong>1. Run the crawler daily&lt;/strong>: Schedule the Glue crawler to run after your Lambda function completes (e.g., Lambda runs at 9 AM, crawler runs at 10 AM). The crawler discovers new partitions and updates the table.&lt;/p>
&lt;p>&lt;strong>2. Manual partition updates&lt;/strong>: Run this command in Athena after new data arrives:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>MSCK REPAIR &lt;span style="color:#66d9ef">TABLE&lt;/span> spansh_data.your_table_name;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This scans S3 for new partitions and updates the Glue catalog without running a full crawler.&lt;/p>
&lt;p>&lt;strong>3. Add partitions explicitly&lt;/strong>: If you know the partition path, add it directly:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ALTER&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> spansh_data.your_table_name
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ADD&lt;/span> PARTITION (&lt;span style="color:#66d9ef">year&lt;/span>&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;2025&amp;#39;&lt;/span>, &lt;span style="color:#66d9ef">month&lt;/span>&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;01&amp;#39;&lt;/span>, &lt;span style="color:#66d9ef">day&lt;/span>&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;23&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LOCATION&lt;/span> &lt;span style="color:#e6db74">&amp;#39;s3://spansh-data-pipeline-ch-2025/raw/spansh/year=2025/month=01/day=23/&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>For daily data ingestion, option 1 (scheduled crawler) is most reliable. Configure your EventBridge schedule to run the crawler shortly after Lambda completes.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="understanding-costs-what-youre-actually-paying-for">Understanding costs: What you&amp;rsquo;re actually paying for&lt;/h3>
&lt;hr>
&lt;p>Let&amp;rsquo;s talk money, because surprisingly, this entire setup costs almost nothing to run.&lt;/p>
&lt;p>&lt;strong>S3 storage&lt;/strong>: $0.023 per GB per month. If you&amp;rsquo;re storing 10GB of data, that&amp;rsquo;s $0.23 per month.&lt;/p>
&lt;p>&lt;strong>Lambda execution&lt;/strong>: First 1 million requests per month are free, then $0.20 per million requests. Each request gets 400,000 GB-seconds of compute free, then $0.0000166667 for every GB-second thereafter. If your Lambda function uses 512MB and runs for 30 seconds daily, that&amp;rsquo;s about 15GB-seconds per month—well within the free tier.&lt;/p>
&lt;p>&lt;strong>Glue crawler&lt;/strong>: $0.44 per hour (prorated per second). Running a crawler for 2 minutes daily costs about $0.30 per month.&lt;/p>
&lt;p>&lt;strong>Athena queries&lt;/strong>: $5 per TB scanned. If you scan 1GB per month in queries, that&amp;rsquo;s $0.005.&lt;/p>
&lt;p>&lt;strong>Total monthly cost&lt;/strong>: Less than $1 for small-scale usage. Even at moderate scale (100GB storage, daily ingestion, regular querying), you&amp;rsquo;re looking at $5-10 per month.&lt;/p>
&lt;p>Compare that to running a dedicated database server (RDS starting at $25/month minimum) or an EC2 instance for scheduled tasks ($10-50/month depending on size). The serverless approach is dramatically cheaper for intermittent workloads.&lt;/p>
&lt;p>The bigger savings aren&amp;rsquo;t even the dollars—it&amp;rsquo;s the time. No patching servers, no managing backups, no worrying about scaling, no 2 AM pages when something crashes. You write code, configure services, and it runs.&lt;/p>
&lt;p>That said, costs can spiral if you&amp;rsquo;re not careful:&lt;/p>
&lt;p>&lt;strong>Common mistakes that inflate costs&lt;/strong>:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Running queries without LIMIT clauses&lt;/strong>: Testing a query on production data without &lt;code>LIMIT&lt;/code> can scan terabytes accidentally&lt;/li>
&lt;li>&lt;strong>Not using partition filters&lt;/strong>: Scanning all data when you only need recent data&lt;/li>
&lt;li>&lt;strong>Over-running crawlers&lt;/strong>: Scheduling crawlers hourly when daily would suffice&lt;/li>
&lt;li>&lt;strong>Not compressing data&lt;/strong>: JSON is inefficient; gzipping files reduces storage and query costs by 70-80%&lt;/li>
&lt;li>&lt;strong>Leaving Lambda triggers enabled during development&lt;/strong>: If you&amp;rsquo;re testing and forget to disable the schedule, it runs daily even when you don&amp;rsquo;t want it&lt;/li>
&lt;/ol>
&lt;p>Set up billing alerts in AWS (under AWS Budgets) to notify you if costs exceed thresholds. For this pipeline, alert at $10/month—if you hit that, something&amp;rsquo;s misconfigured.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="when-this-pattern-works-and-when-it-doesnt">When this pattern works (and when it doesn&amp;rsquo;t)&lt;/h3>
&lt;hr>
&lt;p>The architecture we&amp;rsquo;ve built—Lambda for ingestion, S3 for storage, Glue for cataloging, Athena for querying—is incredibly versatile. But it&amp;rsquo;s not appropriate for every scenario.&lt;/p>
&lt;p>&lt;strong>This pattern excels when&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Data arrives in batches (hourly, daily, weekly) rather than as a continuous stream&lt;/li>
&lt;li>You need historical data retention without database costs&lt;/li>
&lt;li>Multiple teams need access to the same data&lt;/li>
&lt;li>Data volumes range from gigabytes to petabytes&lt;/li>
&lt;li>Query patterns are analytical (aggregations, scans) rather than transactional (point lookups, updates)&lt;/li>
&lt;li>You&amp;rsquo;re okay with eventual consistency rather than immediate data availability&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>This pattern struggles when&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>You need real-time data (use Kinesis Streams instead)&lt;/li>
&lt;li>You require sub-second query response times (use a database or cache)&lt;/li>
&lt;li>Data undergoes frequent updates or deletes (S3 is append-only in practice)&lt;/li>
&lt;li>You need ACID transactions (use RDS or DynamoDB)&lt;/li>
&lt;li>File sizes are extremely large per record (&amp;gt;10GB) and Lambda isn&amp;rsquo;t sufficient (use EMR or Glue ETL jobs)&lt;/li>
&lt;/ul>
&lt;p>Most data engineering work falls into the &amp;ldquo;batch processing&amp;rdquo; category that this pattern handles well. Customer data, transaction logs, application events, API data, log files—these all fit the model.&lt;/p>
&lt;p>Real-world examples from industry:&lt;/p>
&lt;p>&lt;strong>Netflix&lt;/strong> processes billions of events daily using a similar architecture, ingesting data into S3, cataloging with Glue, and querying with Athena and other analytics engines.&lt;/p>
&lt;p>&lt;strong>Airbnb&lt;/strong> uses S3 as the foundation of their data lake, storing raw logs and datasets that teams query using Presto (which Athena is built on).&lt;/p>
&lt;p>&lt;strong>Uber&lt;/strong> built their big data platform on S3, processing petabytes of trip data using variations of this exact pattern.&lt;/p>
&lt;p>These companies operate at massive scale, but the fundamentals remain the same as what you&amp;rsquo;ve built today. The services scale seamlessly—what works for 1GB works for 1PB.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="what-comes-next-building-on-your-foundation">What comes next: Building on your foundation&lt;/h3>
&lt;hr>
&lt;p>You&amp;rsquo;ve built a working data pipeline. Data flows from an API into Lambda, gets stored in partitioned S3 folders, cataloged by Glue, and queried using SQL in Athena. That&amp;rsquo;s a real accomplishment. But data pipelines are rarely finished—they evolve based on requirements.&lt;/p>
&lt;p>Here are logical next steps, ordered from easiest to most complex:&lt;/p>
&lt;p>&lt;strong>Error handling and monitoring&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Add CloudWatch alarms to alert when Lambda fails&lt;/li>
&lt;li>Configure SNS (Simple Notification Service) to email you on errors&lt;/li>
&lt;li>Implement retry logic in Lambda for transient failures&lt;/li>
&lt;li>Add detailed logging for debugging&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Data transformation&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Create a second Lambda function that transforms raw data (cleaning, normalising, enriching)&lt;/li>
&lt;li>Store transformed data in &lt;code>processed/&lt;/code> folder with its own partitions&lt;/li>
&lt;li>Use Glue ETL jobs for more complex transformations&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Data quality checks&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Validate data completeness (did we get the expected number of records?)&lt;/li>
&lt;li>Check data freshness (is today&amp;rsquo;s data present?)&lt;/li>
&lt;li>Detect schema changes (have column names or types changed?)&lt;/li>
&lt;li>Implement automated alerts when quality issues arise&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Incremental loading&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Modify Lambda to only download changed records instead of full datasets&lt;/li>
&lt;li>Implement change data capture (CDC) patterns&lt;/li>
&lt;li>Track state between runs using DynamoDB&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Multiple data sources&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Replicate this pattern for additional APIs&lt;/li>
&lt;li>Create separate Lambda functions for each source&lt;/li>
&lt;li>Organise by source in S3 (&lt;code>raw/source1/&lt;/code>, &lt;code>raw/source2/&lt;/code>)&lt;/li>
&lt;li>Consolidate in processed layer for unified analytics&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Advanced querying&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Create Athena views for commonly-used queries&lt;/li>
&lt;li>Implement incremental materialisation (saving query results for reuse)&lt;/li>
&lt;li>Experiment with other query engines (Presto, Spark on EMR)&lt;/li>
&lt;li>Connect Athena to BI tools like Tableau or PowerBI&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Cost optimisation&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Convert JSON to Parquet format (columnar storage reduces costs by 70-90%)&lt;/li>
&lt;li>Implement S3 Lifecycle Policies to move old data to cheaper storage classes&lt;/li>
&lt;li>Use Athena&amp;rsquo;s partition projection instead of crawlers&lt;/li>
&lt;li>Compress files with gzip or Snappy&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Infrastructure as code&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Define your entire pipeline using AWS CloudFormation or Terraform&lt;/li>
&lt;li>Version control your infrastructure definitions&lt;/li>
&lt;li>Enable reproducible deployments across environments (dev/staging/prod)&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>CI/CD pipelines&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Implement automated testing for Lambda functions&lt;/li>
&lt;li>Deploy code changes through proper release pipelines&lt;/li>
&lt;li>Use AWS CodePipeline for continuous deployment&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Alternative Triggers - push&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>Files land in S3 unpredictably&lt;/li>
&lt;li>Real-time or near-real-time processing required&lt;/li>
&lt;li>Cost optimisation matters (avoiding unnecessary invocations)&lt;/li>
&lt;/ul>
&lt;p>Start with monitoring and error handling—these protect your pipeline. Then expand to transformation and quality checks. The rest depends on your specific needs.&lt;/p>
&lt;p>The beauty of this architecture is modularity. Add components incrementally without redesigning the entire system. That&amp;rsquo;s why it&amp;rsquo;s so common in industry—it scales both technically and organisationally.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-bigger-picture-what-youve-actually-learned">The bigger picture: What you&amp;rsquo;ve actually learned&lt;/h3>
&lt;hr>
&lt;p>If you followed along and built this pipeline, you&amp;rsquo;ve gained more than just AWS skills. You&amp;rsquo;ve learned fundamental patterns that transfer across technologies and cloud providers.&lt;/p>
&lt;p>&lt;strong>The concept of data lakes&lt;/strong>: Storing raw data in cheap object storage, organising by partitions, processing separately from storage. Azure uses Blob Storage with Data Lake Gen2. Google Cloud uses Cloud Storage. The implementation varies, but the pattern persists.&lt;/p>
&lt;p>&lt;strong>Serverless computing&lt;/strong>: Running code without managing servers. Azure Functions and Google Cloud Functions work nearly identically to Lambda. Once you understand serverless, you understand it everywhere.&lt;/p>
&lt;p>&lt;strong>Schema-on-read&lt;/strong>: Defining structure when querying data rather than when storing it. This flexibility is why data lakes outcompete traditional databases for analytical workloads. Databricks, Snowflake, BigQuery—they all embrace this model.&lt;/p>
&lt;p>&lt;strong>Event-driven architecture&lt;/strong>: Triggers, functions, and actions. This pattern appears in modern application development, microservices, and data engineering. Understanding EventBridge helps you understand Kubernetes CronJobs, Apache Airflow, and every other scheduler.&lt;/p>
&lt;p>&lt;strong>Infrastructure as code principles&lt;/strong>: Even though we used the console today, production environments define everything as code. The services you configured—S3 buckets, Lambda functions, Glue tables—can all be defined in CloudFormation, Terraform, or AWS CDK. That&amp;rsquo;s how real data teams operate.&lt;/p>
&lt;p>&lt;strong>Cost-conscious engineering&lt;/strong>: Understanding how services charge (storage vs compute vs data scanned) shapes design decisions. This mindset transfers to any cloud platform and many on-premises tools.&lt;/p>
&lt;p>These aren&amp;rsquo;t AWS-specific skills. They&amp;rsquo;re data engineering fundamentals wrapped in AWS packaging. Learn them on AWS, apply them anywhere.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="some-final-practical-wisdom">Some final practical wisdom&lt;/h3>
&lt;hr>
&lt;p>Before you close this guide and start building your own pipelines, let me share a few lessons learned from watching people implement this pattern—both successfully and unsuccessfully.&lt;/p>
&lt;p>&lt;strong>Start simple, evolve deliberately&lt;/strong>: The temptation exists to immediately add complexity—transformations, validations, multiple sources, fancy orchestration. Resist this. Build the basic pipeline first, verify it works, then add features one at a time. Most failed data projects collapse under premature complexity.&lt;/p>
&lt;p>&lt;strong>Documentation matters more than you think&lt;/strong>: Six months from now, you won&amp;rsquo;t remember why you partitioned a certain way or what that Lambda environment variable does. Comment your code. Document your decisions. Your future self will thank you.&lt;/p>
&lt;p>&lt;strong>Test with small data first&lt;/strong>: Don&amp;rsquo;t point your Lambda function at an API endpoint that returns 100GB of data for your first test. Start with APIs that return kilobytes. Verify everything works, then scale up.&lt;/p>
&lt;p>&lt;strong>Monitoring is not optional&lt;/strong>: In production, things fail. APIs go down. AWS services have brief outages. Lambda functions hit timeouts. Without proper monitoring, you won&amp;rsquo;t know until someone asks why the data&amp;rsquo;s missing. Set up CloudWatch alarms early.&lt;/p>
&lt;p>&lt;strong>Costs are usually predictable, but spikes happen&lt;/strong>: That $2/month pipeline can suddenly spike to $500 if you accidentally run a full-table scan on petabytes of data. Billing alerts save you from unpleasant surprises.&lt;/p>
&lt;p>&lt;strong>The AWS ecosystem is vast—don&amp;rsquo;t try to learn everything&lt;/strong>: We touched on maybe 5% of available AWS services today. That&amp;rsquo;s fine. Master these core services first. Expand as needs arise. Nobody knows all of AWS—everyone specialises.&lt;/p>
&lt;p>&lt;strong>Community resources beat documentation&lt;/strong>: AWS documentation is comprehensive but dry. When you&amp;rsquo;re stuck, search for blog posts, Stack Overflow answers, and YouTube tutorials. The community&amp;rsquo;s collective wisdom often explains concepts better than official docs.&lt;/p>
&lt;p>&lt;strong>Free tier is generous, use it&lt;/strong>: AWS provides 12 months of free tier for new accounts, including Lambda, S3, and many other services. Take advantage of it for learning and experimentation.&lt;/p>
&lt;/br>
&lt;/br></content:encoded><category>Data Engineering</category><category>Platforms &amp; Cloud</category><category>AWS</category><category>Data Pipelines</category><category>Lambda</category><category>S3</category><category>Athena</category><category>Cloud Computing</category><category>Data Ingestion</category></item><item><title>Delta-lake - Z-Ordering, Z-Cube, Liquid Clustering and Partitions</title><link>https://ghostinthedata.info/posts/2024/2024-01-14-delta-lake-zordering-zcub-liquidclustering-and-partitions/</link><pubDate>Sun, 14 Jan 2024 00:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2024/2024-01-14-delta-lake-zordering-zcub-liquidclustering-and-partitions/</guid><author>Chris Hillman</author><description>Introduction Ever feel like your data lake is more of a data swamp, swallowing queries whole and spitting out eternity? You’re not alone. Managing massive datasets can be a Herculean task, especially when it comes to squeezing out those precious milliseconds of query performance. But fear not, data warriors, for Delta Lake has hidden treasures waiting to be unearthed: Z-ordering, Z-cube, and liquid clustering.
Partition Pruning: The OG Hero Before we dive into these exotic beasts, let’s pay homage to the OG hero of data organisation: partition pruning. Imagine your data lake as a meticulously organised library, with each book (partition) shelved by a specific topic (partition column). When a query saunters in, it doesn’t have to wander through every aisle. It simply heads straight for the relevant section, drastically reducing the time it takes to find what it needs. That’s the magic of partition pruning!</description><content:encoded>&lt;h3 id="introduction">Introduction&lt;/h3>
&lt;hr>
&lt;p>Ever feel like your data lake is more of a data swamp, swallowing queries whole and spitting out eternity? You&amp;rsquo;re not alone. Managing massive datasets can be a Herculean task, especially when it comes to squeezing out those precious milliseconds of query performance. But fear not, data warriors, for Delta Lake has hidden treasures waiting to be unearthed: Z-ordering, Z-cube, and liquid clustering.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="partition-pruning-the-og-hero">Partition Pruning: The OG Hero&lt;/h3>
&lt;hr>
&lt;p>Before we dive into these exotic beasts, let&amp;rsquo;s pay homage to the OG hero of data organisation: partition pruning. Imagine your data lake as a meticulously organised library, with each book (partition) shelved by a specific topic (partition column). When a query saunters in, it doesn&amp;rsquo;t have to wander through every aisle. It simply heads straight for the relevant section, drastically reducing the time it takes to find what it needs. That&amp;rsquo;s the magic of partition pruning!&lt;/p>
&lt;img src="https://ghostinthedata.info/posts/2024/2024-01-14-delta-lake-zordering-zcub-liquidclustering-and-partitions/images/Files%20Large.jpeg"
 
 alt="Large Files"
 
 
 
 
 
 class="center"
 
>

&lt;/br>
&lt;/br>
&lt;h3 id="z-ordering-the-spatial-sensei">Z-Ordering: The Spatial Sensei&lt;/h3>
&lt;hr>
&lt;p>But what happens when your library&amp;rsquo;s shelves are spread across multiple buildings? Z-ordering enters the scene, a master of spatial organisation. It takes related information, like books on similar subjects, and arranges them close to each other, even across different buildings. This means your queries can zip between relevant data like a parkour ninja, saving precious time.&lt;/p>
&lt;p>Example - PySpark&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-PYTHON" data-lang="PYTHON">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> pyspark.sql.functions &lt;span style="color:#66d9ef">as&lt;/span> F 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> delta.tables &lt;span style="color:#f92672">import&lt;/span> &lt;span style="color:#f92672">*&lt;/span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Create a SparkSession spark = SparkSession.builder.appName(&amp;#34;ZOrderingExample&amp;#34;).getOrCreate() &lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Read the dataset from a Delta table (or load from a source like CSV) data = spark.read.format(&amp;#34;delta&amp;#34;).load(&amp;#34;/path/to/your/delta/table&amp;#34;) &lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Apply Z-ordering on the &amp;#34;date&amp;#34; and &amp;#34;user_id&amp;#34; columns zOrderedData = data.write.format(&amp;#34;delta&amp;#34;) \ &lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">.&lt;/span>mode(&lt;span style="color:#e6db74">&amp;#34;overwrite&amp;#34;&lt;/span>) \ 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">.&lt;/span>option(&lt;span style="color:#e6db74">&amp;#34;ZOrderBy&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;date, user_id&amp;#34;&lt;/span>) \ 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">.&lt;/span>save(&lt;span style="color:#e6db74">&amp;#34;/path/to/zOrderedDataset&amp;#34;&lt;/span>) 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Query the Z-ordered table, filtering by date and user_id query = zOrderedData.filter(F.col(&amp;#34;date&amp;#34;) == &amp;#34;2024-01-15&amp;#34; and F.col(&amp;#34;user_id&amp;#34;) &amp;gt; 1000) query.show() &lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># To optimize an existing Delta table with Z-ordering: &lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>optimize(spark, &lt;span style="color:#e6db74">&amp;#34;/path/to/zOrderedDataset&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;ZORDER BY date, user_id&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Example - SQL&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-SQL" data-lang="SQL">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create a Z-ordered Delta table:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> z_ordered_table
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">USING&lt;/span> DELTA
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PARTITIONED &lt;span style="color:#66d9ef">BY&lt;/span> (date)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>ZORDER &lt;span style="color:#66d9ef">BY&lt;/span> (user_id)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">AS&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> original_table;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Optimize an existing Delta table with Z-ordering:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">ALTER&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> original_table OPTIMIZE ZORDER &lt;span style="color:#66d9ef">BY&lt;/span> (date, user_id);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;/br>
&lt;h3 id="z-cube-the-multi-dimensional-mastermind">Z-Cube: The Multi-Dimensional Mastermind&lt;/h3>
&lt;hr>
&lt;p>Not all libraries are simple one-dimensional rows, though. Some house intricate collections with multiple dimensions, like a museum showcasing art from different eras and regions. Z-cube, the multi-dimensional mastermind, comes to the rescue here. It maps your data onto a multi-dimensional grid, ensuring that art pieces from similar eras and regions are always close neighbours. Your complex queries can now navigate this grid with ease, finding the treasures they seek in a flash.&lt;/p>
&lt;p>Example - PySpark&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-PYTHON" data-lang="PYTHON">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> pyspark.sql.functions &lt;span style="color:#66d9ef">as&lt;/span> F
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> delta.tables &lt;span style="color:#f92672">import&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Create a SparkSession&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>spark &lt;span style="color:#f92672">=&lt;/span> SparkSession&lt;span style="color:#f92672">.&lt;/span>builder&lt;span style="color:#f92672">.&lt;/span>appName(&lt;span style="color:#e6db74">&amp;#34;ZCubeExample&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>getOrCreate()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Read the dataset&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>data &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>format(&lt;span style="color:#e6db74">&amp;#34;delta&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>load(&lt;span style="color:#e6db74">&amp;#34;/path/to/your/delta/table&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Apply Z-Cube on the &amp;#34;category&amp;#34;, &amp;#34;brand&amp;#34;, and &amp;#34;price&amp;#34; columns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>zCubeData &lt;span style="color:#f92672">=&lt;/span> data&lt;span style="color:#f92672">.&lt;/span>write&lt;span style="color:#f92672">.&lt;/span>format(&lt;span style="color:#e6db74">&amp;#34;delta&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>mode(&lt;span style="color:#e6db74">&amp;#34;overwrite&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>option(&lt;span style="color:#e6db74">&amp;#34;ZCube&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;category, brand, price&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>save(&lt;span style="color:#e6db74">&amp;#34;/path/to/zCubeDataset&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Query the Z-Cube table, filtering across multiple dimensions&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>query &lt;span style="color:#f92672">=&lt;/span> zCubeData&lt;span style="color:#f92672">.&lt;/span>filter(F&lt;span style="color:#f92672">.&lt;/span>col(&lt;span style="color:#e6db74">&amp;#34;category&amp;#34;&lt;/span>) &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;Electronics&amp;#34;&lt;/span> &lt;span style="color:#f92672">and&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> F&lt;span style="color:#f92672">.&lt;/span>col(&lt;span style="color:#e6db74">&amp;#34;brand&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>isin([&lt;span style="color:#e6db74">&amp;#34;Apple&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;Samsung&amp;#34;&lt;/span>]) &lt;span style="color:#f92672">and&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> F&lt;span style="color:#f92672">.&lt;/span>col(&lt;span style="color:#e6db74">&amp;#34;price&amp;#34;&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">500&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>query&lt;span style="color:#f92672">.&lt;/span>show()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># To optimize an existing Delta table with Z-Cube:&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>optimize(spark, &lt;span style="color:#e6db74">&amp;#34;/path/to/zCubeDataset&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;ZCube BY category, brand, price&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Example - SQL&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-SQL" data-lang="SQL">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create a Z-Cube Delta table:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> z_cube_table
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">USING&lt;/span> DELTA
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PARTITIONED &lt;span style="color:#66d9ef">BY&lt;/span> (category)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>ZORDER &lt;span style="color:#66d9ef">BY&lt;/span> (brand, price)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">AS&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> original_table;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Optimize an existing Delta table with Z-Cube:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">ALTER&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> original_table OPTIMIZE ZORDER &lt;span style="color:#66d9ef">BY&lt;/span> (category, brand, price);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;/br>
&lt;h3 id="liquid-clustering-the-evolving-adaptor">Liquid Clustering: The Evolving Adaptor&lt;/h3>
&lt;hr>
&lt;p>But what if the library&amp;rsquo;s collection keeps growing and changing? Enter liquid clustering, the ever-adapting chameleon of data organisation. It continuously analyses your data, grouping similar items together and reshuffling them as new information arrives. Think of it as a librarian constantly rearranging shelves to keep up with the latest acquisitions. This dynamic approach is perfect for datasets in constant flux, ensuring queries always have the most efficient path to their targets.&lt;/p>
&lt;p>Example - PySpark&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-PYTHON" data-lang="PYTHON">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> pyspark.sql.functions &lt;span style="color:#66d9ef">as&lt;/span> F
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> delta.tables &lt;span style="color:#f92672">import&lt;/span> &lt;span style="color:#f92672">*&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Create a SparkSession&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>spark &lt;span style="color:#f92672">=&lt;/span> SparkSession&lt;span style="color:#f92672">.&lt;/span>builder&lt;span style="color:#f92672">.&lt;/span>appName(&lt;span style="color:#e6db74">&amp;#34;LiquidClusteringExample&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>getOrCreate()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Read the dataset&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>data &lt;span style="color:#f92672">=&lt;/span> spark&lt;span style="color:#f92672">.&lt;/span>read&lt;span style="color:#f92672">.&lt;/span>format(&lt;span style="color:#e6db74">&amp;#34;delta&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>load(&lt;span style="color:#e6db74">&amp;#34;/path/to/your/delta/table&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Enable liquid clustering on the desired columns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>data&lt;span style="color:#f92672">.&lt;/span>write&lt;span style="color:#f92672">.&lt;/span>format(&lt;span style="color:#e6db74">&amp;#34;delta&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>option(&lt;span style="color:#e6db74">&amp;#34;clustering&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;user_id, sentiment&amp;#34;&lt;/span>) \
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">.&lt;/span>save(&lt;span style="color:#e6db74">&amp;#34;/path/to/liquidClusteredDataset&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># To optimize an existing Delta table with liquid clustering:&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>optimize(spark, &lt;span style="color:#e6db74">&amp;#34;/path/to/liquidClusteredDataset&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;clustering user_id, sentiment&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Note: Limited to Databricks runtime as of Jan 2024&lt;/p>
&lt;p>Choosing Your Weapon: The Data Lake Dojo&lt;/p>
&lt;p>So, which data organisation warrior should you choose? Don&amp;rsquo;t worry, we&amp;rsquo;ve got your back! Z-ordering shines when you have a few key filter columns and predictable data patterns. Z-cube thrives in multi-dimensional data realms with complex queries. And liquid clustering adapts like a ninja to ever-changing datasets.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h4 id="further-reading-and-resources">Further Reading and Resources&lt;/h4>
&lt;h5 id="z-ordering">Z-Ordering:&lt;/h5>
&lt;ul>
&lt;li>&lt;a href="https://docs.databricks.com/en/delta/tutorial.html" target="_blank" rel="noopener">Delta Lake Optimisation Guide&lt;/a>: Provides detailed explanations of Z-ordering, including benefits, limitations, and configuration options.&lt;/li>
&lt;li>&lt;a href="https://arxiv.org/abs/2204.08453" target="_blank" rel="noopener">Space-Filling Curves for Data Organisation&lt;/a>: Research paper delving into the mathematical aspects of Z-curves and their applications in data organisation.&lt;/li>
&lt;/ul>
&lt;h5 id="z-cube">Z-Cube:&lt;/h5>
&lt;ul>
&lt;li>&lt;a href="https://docs.databricks.com/en/delta/tutorial.html" target="_blank" rel="noopener">Delta Lake Summit 2023 Presentation&lt;/a>: Slide deck includes detailed explanations and diagrams of Z-cube.&lt;/li>
&lt;li>&lt;a href="https://docs.databricks.com/en/delta/data-skipping.html" target="_blank" rel="noopener">Multi-Dimensional Data Indexing Using Space-Filling Curves&lt;/a>: Databricks blog post with technical discussion and illustrations of Z-cube.&lt;/li>
&lt;/ul>
&lt;h5 id="liquid-clustering">Liquid Clustering:&lt;/h5>
&lt;ul>
&lt;li>&lt;a href="https://docs.databricks.com/en/delta/tutorial.html" target="_blank" rel="noopener">Delta Lake 2.0: What&amp;rsquo;s New and Exciting&lt;/a>: Databricks blog post introducing liquid clustering with conceptual explanations and demos.&lt;/li>
&lt;li>&lt;a href="https://scalablehuman.com/2022/07/24/what-is-dynamic-partitioning/" target="_blank" rel="noopener">Dynamic Data Partitioning for Evolving Datasets&lt;/a>: TechTarget article discussing concepts similar to liquid clustering with illustrative diagrams.&lt;/li>
&lt;/ul></content:encoded><category>Platforms &amp; Cloud</category><category>Delta Lake</category></item><item><title>Create a Minecraft (PE)Bedrock Server in GCP</title><link>https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp/</link><pubDate>Sun, 15 Jan 2023 00:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp/</guid><author>Chris Hillman</author><description>Introduction My eldest daughter, plays with her friends a lot with the Minecraft Pocket Edition (PE). The PE Edition, really limits it to WiiU, xbox or ipad. This is the most connivent gaming version for them as they all have iPads.
Occasionally the world they are all playing on gets corrupted and she looses all the work they have done. There is ways to recover it, but its rather time consuming.</description><content:encoded>&lt;h3 id="introduction">Introduction&lt;/h3>
&lt;hr>
&lt;p>My eldest daughter, plays with her friends a lot with the Minecraft Pocket Edition (PE). The PE Edition, really limits it to WiiU, xbox or ipad. This is the most connivent gaming version for them as they all have iPads.&lt;/p>
&lt;p>Occasionally the world they are all playing on gets corrupted and she looses all the work they have done. There is ways to recover it, but its rather time consuming.&lt;/p>
&lt;p>So thought I would look into making a server for them. This means we have more control over the backup process. I am not a GCP expert, but this was a great little process to get familiar with some of the functionality that comes with GCP.&lt;/p>
&lt;h3 id="create-a-new-project">Create a new Project&lt;/h3>
&lt;hr>
&lt;p>Navigate to &lt;a href="https://console.cloud.google.com/home/dashboard" target="_blank" rel="noopener">Google Cloud Console&lt;/a>&lt;/p>
&lt;p>In the left side, there will be a drop down, and you can navigate here to Create a &lt;a href="https://console.cloud.google.com/projectcreate" target="_blank" rel="noopener">new project&lt;/a>.&lt;/p>
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/GCPProject.PNG"
 
 alt="GCP Project"
 
 
 
 
 
 class="center"
 
>

&lt;p>Google Cloud Console — Create New ProjectI named this project &lt;strong>Minecraft&lt;/strong>.&lt;/p>
&lt;p>The &lt;strong>ProjectID&lt;/strong> that it generates is important for a future step, so you can copy and paste that to a notepad, or write it down.&lt;/p>
&lt;p>On the left side there will be 3 horizontal lines, which we will refer to as the GCP Console, its the quick navigation to all the googles functionality.&lt;/p>
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/GCPProjectMenu.PNG"
 
 alt="GCP Project - Menu"
 
 
 
 
 
 class="center"
 
>

&lt;h3 id="create-the-vm-virtual-machine">Create the VM (Virtual Machine)&lt;/h3>
&lt;hr>
&lt;p>In the GCP console, click &lt;a href="https://console.cloud.google.com/compute/instances" target="_blank" rel="noopener">Compute Engine &amp;gt; VM Instances&lt;/a>&lt;/p>
&lt;p>Click create instance&lt;/p>
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/GCPVMInstances.PNG"
 
 alt="GCP VM Instances"
 
 
 
 
 
 class="center"
 
>

&lt;p>We want to the below options to be input:
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/GCPVMInstancesConfig.PNG"
 
 alt="VM Configuration"
 
 
 
 
 
 class="center"
 
>
&lt;/p>
&lt;p>Under &lt;strong>Management, security, disks, networking, sole tenancy&lt;/strong>.&lt;/p>
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/NetworkSecurityDisks.PNG"
 
 alt="Network Security and Disks"
 
 
 
 
 
 class="center"
 
>

&lt;p>Click first on &lt;strong>Disks&lt;/strong>
Click &lt;strong>Add new Disk&lt;/strong>&lt;/p>
&lt;p>We want to the below options to be input:
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/DiskConfig.PNG"
 
 alt="Disk Configuration"
 
 
 
 
 
 class="center"
 
>
&lt;/p>
&lt;p>Click &lt;strong>Done&lt;/strong>
Click now on &lt;strong>Networking&lt;/strong>&lt;/p>
&lt;p>We want to the below options to be input:
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/Networking.PNG"
 
 alt="Networking"
 
 
 
 
 
 class="center"
 
>
&lt;/p>
&lt;p>Click &lt;strong>Done&lt;/strong>
Click &lt;strong>Create&lt;/strong>&lt;/p>
&lt;h3 id="setup-the-firewall">Setup the firewall&lt;/h3>
&lt;hr>
&lt;p>In the GCP console, Click on &lt;a href="https://console.cloud.google.com/networking/firewalls" target="_blank" rel="noopener">VPC Network &amp;gt; Firewall rules&lt;/a>&lt;/p>
&lt;p>Click on &lt;strong>create firewall rule&lt;/strong>
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/Firewall.PNG"
 
 alt="Firewall"
 
 
 
 
 
 class="center"
 
>
&lt;/p>
&lt;p>We want to the below options to be input:
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/FirewallConfig.PNG"
 
 alt="Firewall Configuration"
 
 
 
 
 
 class="center"
 
>
&lt;/p>
&lt;p>Click &lt;strong>Create&lt;/strong>&lt;/p>
&lt;h3 id="setup-the-vm">Setup the VM&lt;/h3>
&lt;hr>
&lt;p>This disk is attached to the compute instance, but its not mounted.&lt;/p>
&lt;p>Click on &lt;strong>SSH&lt;/strong> to pop out a terminal and connect to your compute instance
Create a &lt;strong>directory&lt;/strong> that will become the mount point for the disk&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>sudo mkdir -p /home/minecraft
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Then we need to format the disk run. The disk name may vary&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>fdisk -ls
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>in my case it was /dev/sda1&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>sudo mkfs.ext4 -F -E lazy_itable_init&lt;span style="color:#f92672">=&lt;/span>0,lazy_journal_init&lt;span style="color:#f92672">=&lt;/span>0,discard /dev/sda1
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>To mount the disk run the following&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>sudo mount -o discard,defaults /dev/sda1 /home/minecraft
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Update the OS repositories on the VM, runing the below&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>sudo apt-get update &lt;span style="color:#f92672">&amp;amp;&amp;amp;&lt;/span> sudo apt-get upgrade &lt;span style="color:#f92672">&amp;amp;&amp;amp;&lt;/span> sudo apt-get dist-upgrade
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>now install unzip, wget, screen, htop, headless JRE&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>sudo apt-get install -y default-jre-headless
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo apt-get install zip unzip
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo apt-get install wget
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo apt-get install screen
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo apt-get install htop
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Now lets install the bedrock server (which is specific for Pocket Edition)&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>cd /home/minecraft
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo mkdir downloads
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo wget -O downloads/version.html https://minecraft.net/en-us/download/server/bedrock/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo DownloadURL&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">$(&lt;/span>grep -o &lt;span style="color:#e6db74">&amp;#39;https://minecraft.azureedge.net/bin-linux/[^&amp;#34;]*&amp;#39;&lt;/span> downloads/version.html&lt;span style="color:#66d9ef">)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo DownloadFile&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">$(&lt;/span>echo &lt;span style="color:#e6db74">&amp;#34;&lt;/span>$DownloadURL&lt;span style="color:#e6db74">&amp;#34;&lt;/span>| sed &lt;span style="color:#e6db74">&amp;#39;s#.*/##&amp;#39;&lt;/span>&lt;span style="color:#66d9ef">)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo wget -O &lt;span style="color:#e6db74">&amp;#34;downloads/&lt;/span>$DownloadFile&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&lt;/span>$DownloadURL&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo unzip -o &lt;span style="color:#e6db74">&amp;#34;downloads/&lt;/span>$DownloadFile&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Run the &lt;strong>server&lt;/strong>&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>sudo LD_LIBRARY_PATH&lt;span style="color:#f92672">=&lt;/span>. ./bedrock_server
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="create-backups">Create Backups&lt;/h3>
&lt;hr>
&lt;p>In the GCP console, Click on &lt;a href="https://console.cloud.google.com/compute/instances" target="_blank" rel="noopener">Compute Engine &amp;gt; VM Instances&lt;/a>&lt;/p>
&lt;p>We already have the &lt;strong>mc-server&lt;/strong> created, so click on &lt;strong>SSH&lt;/strong>
We want to create a global unique bucket name&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>export YOUR_BUCKET_NAME&lt;span style="color:#f92672">=&lt;/span>&amp;lt;Enter your bucket name here&amp;gt;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>gsutil mb gs://$YOUR_BUCKET_NAME-minecraft-backup
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Now we want to create a backup shell script.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>cd /home/minecraft
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo nano /home/minecraft/backup.sh
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The below code will enable you to save the world, to your backup bucket&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#e6db74">&amp;#39;/save-all\n/save-off\n&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>/usr/bin/gsutil -m cp -R /home/minecraft/worlds/* gs://&lt;span style="color:#e6db74">${&lt;/span>YOUR_BUCKET_NAME&lt;span style="color:#e6db74">}&lt;/span>-minecraft-backup/world
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">&amp;#39;/save-on\n&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>/usr/bin/gsutil -m cp -R /home/minecraft/server.properties gs://&lt;span style="color:#e6db74">${&lt;/span>YOUR_BUCKET_NAME&lt;span style="color:#e6db74">}&lt;/span>-minecraft-backup/config/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>/usr/bin/gsutil -m cp -R /home/minecraft/permissions.json gs://&lt;span style="color:#e6db74">${&lt;/span>YOUR_BUCKET_NAME&lt;span style="color:#e6db74">}&lt;/span>-minecraft-backup/config/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>/usr/bin/gsutil -m cp -R /home/minecraft/whitelist.json gs://&lt;span style="color:#e6db74">${&lt;/span>YOUR_BUCKET_NAME&lt;span style="color:#e6db74">}&lt;/span>-minecraft-backup/config/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>/usr/bin/gsutil -m cp -R /home/minecraft/valid_known_packs.json gs://&lt;span style="color:#e6db74">${&lt;/span>YOUR_BUCKET_NAME&lt;span style="color:#e6db74">}&lt;/span>-minecraft-backup/config/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>/usr/bin/gsutil -m cp -R /home/minecraft/*.sh gs://&lt;span style="color:#e6db74">${&lt;/span>YOUR_BUCKET_NAME&lt;span style="color:#e6db74">}&lt;/span>-minecraft-backup/config/
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Press &lt;strong>Ctrl O&lt;/strong>, then &lt;strong>Enter&lt;/strong> to &lt;strong>Save&lt;/strong>, then &lt;strong>Ctrl X&lt;/strong> to &lt;strong>exit&lt;/strong>&lt;/p>
&lt;p>We can then allow the script to execute&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>sudo chmod &lt;span style="color:#ae81ff">755&lt;/span> /home/minecraft/backup.sh
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>We can test the script is working by running&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>. /home/minecraft/backup.sh
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Then verify it in the Storage by navigating here
&lt;a href="https://console.cloud.google.com/storage/browser" target="_blank" rel="noopener">https://console.cloud.google.com/storage/browser&lt;/a>&lt;/p>
&lt;p>Some other utility scripts I created are here;&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>gsutil -m cp -R &lt;span style="color:#e6db74">&amp;#34;gs://&lt;/span>&lt;span style="color:#e6db74">${&lt;/span>YOUR_BUCKET_NAME&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">-minecraft-backup/config/*&amp;#34;&amp;#34;/home/minecraft&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>gsutil -m cp -R &lt;span style="color:#e6db74">&amp;#34;gs://&lt;/span>&lt;span style="color:#e6db74">${&lt;/span>YOUR_BUCKET_NAME&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">-minecraft-backup/worlds/*&amp;#34;&amp;#34;/home/minecraft/worlds/&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Change directory to server directory&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo cd /home/minecraft
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Retrieve latest version of Minecraft Bedrock dedicated server&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>echo &lt;span style="color:#e6db74">&amp;#34;Checking for the latest version of Minecraft Bedrock server ...&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Test internet connectivity first&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>wget --quiet http://www.minecraft.net/ -O /dev/null
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#f92672">[&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&lt;/span>$?&lt;span style="color:#e6db74">&amp;#34;&lt;/span> !&lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#f92672">]&lt;/span>; &lt;span style="color:#66d9ef">then&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> echo &lt;span style="color:#e6db74">&amp;#34;Unable to connect to update website (internet connection may be down). Skipping update ...&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Download server index.html to check latest version&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sudo wget -O downloads/version.html https://minecraft.net/en-us/download/server/bedrock/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sudo DownloadURL&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">$(&lt;/span>grep -o &lt;span style="color:#e6db74">&amp;#39;https://minecraft.azureedge.net/bin-linux/[^&amp;#34;]*&amp;#39;&lt;/span> downloads/version.html&lt;span style="color:#66d9ef">)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sudo DownloadFile&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">$(&lt;/span>echo &lt;span style="color:#e6db74">&amp;#34;&lt;/span>$DownloadURL&lt;span style="color:#e6db74">&amp;#34;&lt;/span> | sed &lt;span style="color:#e6db74">&amp;#39;s#.*/##&amp;#39;&lt;/span>&lt;span style="color:#66d9ef">)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Download latest version of Minecraft Bedrock dedicated server if a new one is available&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#f92672">[&lt;/span> -f &lt;span style="color:#e6db74">&amp;#34;downloads/&lt;/span>$DownloadFile&lt;span style="color:#e6db74">&amp;#34;&lt;/span> &lt;span style="color:#f92672">]&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">then&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> echo &lt;span style="color:#e6db74">&amp;#34;Minecraft Bedrock server is up to date...&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">else&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> echo &lt;span style="color:#e6db74">&amp;#34;New version &lt;/span>$DownloadFile&lt;span style="color:#e6db74"> is available. Updating Minecraft Bedrock server ...&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sudo wget -O &lt;span style="color:#e6db74">&amp;#34;downloads/&lt;/span>$DownloadFile&lt;span style="color:#e6db74">&amp;#34;&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&lt;/span>$DownloadURL&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sudo unzip -o &lt;span style="color:#e6db74">&amp;#34;downloads/&lt;/span>$DownloadFile&lt;span style="color:#e6db74">&amp;#34;&lt;/span> -x &lt;span style="color:#e6db74">&amp;#34;*server.properties*&amp;#34;&lt;/span> &lt;span style="color:#e6db74">&amp;#34;*permissions.json*&amp;#34;&lt;/span> &lt;span style="color:#e6db74">&amp;#34;*whitelist.json*&amp;#34;&lt;/span> &lt;span style="color:#e6db74">&amp;#34;*valid_known_packs.json*&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">fi&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="start-up-scripts-on-server">Start up scripts on server&lt;/h3>
&lt;hr>
&lt;p>Click on &lt;strong>mc-server&lt;/strong>
Click &lt;strong>edit&lt;/strong>&lt;/p>
&lt;p>Custom metadata add a new key/value pair&lt;/p>
&lt;p>Key:
&lt;code>startup-script&lt;/code>&lt;/p>
&lt;p>Value:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">#!/bin/bash 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Minecraft Bedrock server startup script using screen&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo -i
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo mount /dev/sda1 /home/minecraft
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Check if server is already started&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> screen -list | grep -q &lt;span style="color:#e6db74">&amp;#34;servername&amp;#34;&lt;/span>; &lt;span style="color:#66d9ef">then&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> echo &lt;span style="color:#e6db74">&amp;#34;Server is already started! Press screen -r servername to open it&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> exit &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Change directory to server directory&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>cd /home/minecraft
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Retrieve latest version of Minecraft Bedrock dedicated server&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>echo &lt;span style="color:#e6db74">&amp;#34;Checking for the latest version of Minecraft Bedrock server ...&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Test internet connectivity first&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>wget --quiet http://www.minecraft.net/ -O /dev/null
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#f92672">[&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&lt;/span>$?&lt;span style="color:#e6db74">&amp;#34;&lt;/span> !&lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#f92672">]&lt;/span>; &lt;span style="color:#66d9ef">then&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> echo &lt;span style="color:#e6db74">&amp;#34;Unable to connect to update website (internet connection may be down). Skipping update ...&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Download server index.html to check latest version&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> wget -O downloads/version.html https://minecraft.net/en-us/download/server/bedrock/
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> DownloadURL&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">$(&lt;/span>grep -o &lt;span style="color:#e6db74">&amp;#39;https://minecraft.azureedge.net/bin-linux/[^&amp;#34;]*&amp;#39;&lt;/span> downloads/version.html&lt;span style="color:#66d9ef">)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> DownloadFile&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">$(&lt;/span>echo &lt;span style="color:#e6db74">&amp;#34;&lt;/span>$DownloadURL&lt;span style="color:#e6db74">&amp;#34;&lt;/span> | sed &lt;span style="color:#e6db74">&amp;#39;s#.*/##&amp;#39;&lt;/span>&lt;span style="color:#66d9ef">)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Download latest version of Minecraft Bedrock dedicated server if a new one is available&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#f92672">[&lt;/span> -f &lt;span style="color:#e6db74">&amp;#34;downloads/&lt;/span>$DownloadFile&lt;span style="color:#e6db74">&amp;#34;&lt;/span> &lt;span style="color:#f92672">]&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">then&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> echo &lt;span style="color:#e6db74">&amp;#34;Minecraft Bedrock server is up to date...&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">else&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> echo &lt;span style="color:#e6db74">&amp;#34;New version &lt;/span>$DownloadFile&lt;span style="color:#e6db74"> is available. Updating Minecraft Bedrock server ...&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> wget -O &lt;span style="color:#e6db74">&amp;#34;downloads/&lt;/span>$DownloadFile&lt;span style="color:#e6db74">&amp;#34;&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&lt;/span>$DownloadURL&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> unzip -o &lt;span style="color:#e6db74">&amp;#34;downloads/&lt;/span>$DownloadFile&lt;span style="color:#e6db74">&amp;#34;&lt;/span> -x &lt;span style="color:#e6db74">&amp;#34;*server.properties*&amp;#34;&lt;/span> &lt;span style="color:#e6db74">&amp;#34;*permissions.json*&amp;#34;&lt;/span> &lt;span style="color:#e6db74">&amp;#34;*whitelist.json*&amp;#34;&lt;/span> &lt;span style="color:#e6db74">&amp;#34;*valid_known_packs.json*&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>echo &lt;span style="color:#e6db74">&amp;#34;Starting Minecraft server. To view window type screen -r servername&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>echo &lt;span style="color:#e6db74">&amp;#34;To minimise the window and let the server run in the background, press Ctrl+A then Ctrl+D&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>screen -dmS servername /bin/bash -c &lt;span style="color:#e6db74">&amp;#34;LD_LIBRARY_PATH=. ./bedrock_server&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Key:
&lt;code>shutdown-script&lt;/code>&lt;/p>
&lt;p>Value:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#75715e"># Minecraft Server stop script - primarily called by minecraft service but can be ran manually&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sudo -i
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>/home/minecraft/backup.sh
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>screen -Rd servername -X stuff &lt;span style="color:#e6db74">&amp;#34;stop&lt;/span>&lt;span style="color:#66d9ef">$(&lt;/span>printf &lt;span style="color:#e6db74">&amp;#39;\r&amp;#39;&lt;/span>&lt;span style="color:#66d9ef">)&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>screen -S servername -X quit
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Click &lt;strong>Save&lt;/strong>&lt;/p>
&lt;h3 id="create-auto-startshutdown-instance">Create Auto Start/Shutdown Instance&lt;/h3>
&lt;hr>
&lt;p>In the GCP console, Click on Cloud Scheduler&lt;/p>
&lt;p>Click on &lt;strong>Create Job&lt;/strong>
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/CloudScheduler.PNG"
 
 alt="Cloud Scheduler"
 
 
 
 
 
 class="center"
 
>
&lt;/p>
&lt;p>For my server, I want it to start at 2pm, and close by 10pm.&lt;/p>
&lt;p>We want to the below options to be input:
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/StartServerConfig.PNG"
 
 alt="Start Server Configuration"
 
 
 
 
 
 class="center"
 
>
&lt;/p>
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/StopServerConfig.PNG"
 
 alt="Stop Server Configuration"
 
 
 
 
 
 class="center"
 
>

&lt;p>In the GCP console, Click on Cloud Functions
Click on &lt;strong>Create Functions&lt;/strong>
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/CreateFunctions.PNG"
 
 alt="Create Functions"
 
 
 
 
 
 class="center"
 
>
&lt;/p>
&lt;p>We want to the below options to be input:
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/FunctionsConfig.PNG"
 
 alt="Functions Configuration"
 
 
 
 
 
 class="center"
 
>
&lt;/p>
&lt;p>Code:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> googleapiclient &lt;span style="color:#f92672">import&lt;/span> discovery
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">start_server_pubsub&lt;/span>(event, context):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> service &lt;span style="color:#f92672">=&lt;/span> discovery&lt;span style="color:#f92672">.&lt;/span>build(&lt;span style="color:#e6db74">&amp;#39;compute&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;v1&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#39;VM Instance starting&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Project ID for this request.&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> project &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;minecraft-xxxxxx&amp;#39;&lt;/span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># The name of the zone for this request.&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> zone &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;australia-southeast1-a&amp;#39;&lt;/span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Name of the instance resource to start.&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> instance &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;mc-server&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> request &lt;span style="color:#f92672">=&lt;/span> service&lt;span style="color:#f92672">.&lt;/span>instances()&lt;span style="color:#f92672">.&lt;/span>start(project&lt;span style="color:#f92672">=&lt;/span>project, zone&lt;span style="color:#f92672">=&lt;/span>zone, instance&lt;span style="color:#f92672">=&lt;/span>instance)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> response &lt;span style="color:#f92672">=&lt;/span> request&lt;span style="color:#f92672">.&lt;/span>execute()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#39;VM Instance started&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Click on &lt;strong>Create Functions&lt;/strong>
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp//images/CloudFunctions.PNG"
 
 alt="Cloud Functions"
 
 
 
 
 
 class="center"
 
>
&lt;/p>
&lt;p>We want to the below options to be input:
&lt;img src="https://ghostinthedata.info/posts/2023/2023-01-15-create-a-minecraft-bedrock-server-on-gcp/images/CloudFunctionsConfig.PNG"
 
 alt="Cloud Functions Configuration"
 
 
 
 
 
 class="center"
 
>
&lt;/p>
&lt;p>Code:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> googleapiclient &lt;span style="color:#f92672">import&lt;/span> discovery
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">stop_server_pubsub&lt;/span>(event, context):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> service &lt;span style="color:#f92672">=&lt;/span> discovery&lt;span style="color:#f92672">.&lt;/span>build(&lt;span style="color:#e6db74">&amp;#39;compute&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;v1&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#39;VM Instance stopping&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Project ID for this request.&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> project &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;minecraft-xxxxxx&amp;#39;&lt;/span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># The name of the zone for this request.&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> zone &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;australia-southeast1-a&amp;#39;&lt;/span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Name of the instance resource to start.&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> instance &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;mc-server&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> request &lt;span style="color:#f92672">=&lt;/span> service&lt;span style="color:#f92672">.&lt;/span>instances()&lt;span style="color:#f92672">.&lt;/span>stop(project&lt;span style="color:#f92672">=&lt;/span>project, zone&lt;span style="color:#f92672">=&lt;/span>zone, instance&lt;span style="color:#f92672">=&lt;/span>instance)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> response &lt;span style="color:#f92672">=&lt;/span> request&lt;span style="color:#f92672">.&lt;/span>execute()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#39;VM Instance stopped&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>There is likely lots of improvements that can be made on this, so if you do have any recommendations let me know.&lt;/p></content:encoded><category>Life</category><category>Platforms &amp; Cloud</category><category>GCP</category><category>Minecraft</category><category>Cloud Gaming</category><category>Server Setup</category><category>Family Gaming</category><category>Bedrock Edition</category></item></channel></rss>